This blog entry is a cheat sheet for various Avamar commands and options required during deployment or day-to-day operations.
Disable ConnectEMC
To disable ConnectEMC service a password is needed, use: DISABLECONNECTEMCOK
License Installation
If during deployment a new license needs to be installed, you can use the following steps:
# vi /usr/local/avamar/etc/license.xml
Paste the XML provided to you by Dell EMC and save it. Perform the following command to enable it
# avmaint license /usr/local/avamar/etc/license.xml --ava
Duplicate Names
If, for example you have registered a client as a Virtual Machine (VM), you cannot register this VM as a client for filesystem or other agent backups. By following the below steps, it is possible to register virtual machines with different options
Open /usr/local/avamar/var/mc/server_data/prefs/mcserver.xml change the allow_duplicate_client_names to true
<entry key="allow_duplicate_client_names" value="true" />Restart the MCS service
# dpnctl stop mcs # dpnctl start mcs # dpnctl start sched
Disable Certificate Authentication
If you do not want to add vCenter authentication certificates to the Avamar MCS keystore, you must disable certificate authentication for all vCenter-to-Avamar MCS communications. Stop MCS:
# dpnctl stop mcs
Open /usr/local/avamar/var/mc/server_data/prefs/mcserver.xml and change Change the ignore_vc_cert setting to true.
<entry key="ignore_vc_cert" value="true" />
Start the MCS and scheduler
# dpnctl start mcs # dpnctl start sched
Create a Dedicated vCenter user Account
Using a separate vCenter user account ensures maximum clarity if it becomes necessary to examine vCenter logs. Use of a generic user account such as “Administrator” might hamper future troubleshooting efforts because it might not be clear which actions are actually interfacing or communicating with the Avamar server.
| Privilege type | Required privilege |
|---|---|
| Alarms | ● Create alarm ● Modify alarm |
| Datastore | ● Allocate space ● Browse datastore ● Configure datastore ● Low levefile operations ● Move datastore ● Remove datastore ● Remove File ● Rename datastore |
| Extension | ● Register extension ● Unregister extension ● Update extension |
| Folder | ● Create folder |
| Global | ● Cancel task ● Disable methods ● Enable methods ● Licenses ● Log event ● Manage custom attributes ● Set custom attribute ● Settings |
| Host | ● Configuration > Storage partition configuration |
| Network | ● Assign network ● Configure |
| Resource | ● Assign virtual machine to resource pool |
| Sessions | ● Validate session |
| Tasks | ● Create task ● Update task |
| Virtual Machine-Configuration | ● Add existing disk ● Add new disk ● Add or remove device ● Advanced ● Change CPU count ● Change resource ● Configure managed by ● Disk change tracking ● Disk Lease ● Extend virtuadisk ● Host USB device ● Memory ● Modify device settings ● Raw device ● Reload from path ● Remove disk ● Rename ● Reset guest information ● Set annotation ● Settings ● Swapfile placement ● Upgrade virtual machine Compatibility |
| Virtual Machine-Guest Operations | ● Guest Operation Modifications ● Guest Operation Program Execution ● Guest Operation Queries |
| Virtual Machine-Interaction | ● Console interaction ● DeviceConnection ● Guest operating system management by VIX API ● Power off ● Power on ● Reset ● VMware Tools install |
| Virtual Machine-Inventory | ● Create from existing ● Create new ● Register ● Remove ● Unregister |
| VirtuaMachine-Provisioning | ● Allow disk access ● Allow read-only disk access ● Allow virtual machine download ● Clone virtual machine ● Mark as template |
| Virtual Machine-Snapshot Management | ● Create snapshot ● Remove snapshot ● Revert to snapshot |
| vApp | ● Export ● Import ● vApp application configuration |
CheckPoint and Integrity Check
You can force validate a checkpoint with the following command, first stop the scheduler
root@ave:~/# dpnctl stop maint root@ave:~/# avmaint hfscheck --ava --full
Check status of its progression with the following command:
root@ave:~/#: avmaint hfscheckstatus
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<hfscheckstatus
nodes-queried="1"
nodes-replied="1"
nodes-total="1"
checkpoint="cp.20211231163838"
status="hfscheck"
phase="indexsweep"
type="full"
checks="full"
elapsed-time="105"
start-time="1640968765"
end-time="0"
check-start-time="1640968834"
check-end-time="0"
generation-time="1640968870"
stripes-checking="561"
stripes-completed="22"
offline-stripes="0"
minutes-to-completion="6"
percent-complete="8.61">
<hfscheckerrors/>
</hfscheckstatus>
Start the scheduler
root@ave:~/# dpnctl start maint
Signed SSL Certificate
Open AUI page in browser with fqdn: https://fqdn_of_avamar/aui
In the AUI, navigate to Administration > System > Certificate tab > Private Key tab. A private certificate entry for the Web Server appears in the table.
Click the radial button next to the Web Server entry > Click +REPLACE tab. The Replace Private Entry wizard displays.
- In the Private Key field, click Browse to locate and select your certificate’s private key. In our case, it is server.key placed on desktop.
- In the Certificate field, click Browse to locate and select your certificate file. it should be avamar_server.crt.
- (Optional) If the private key is protected, provide the passphrase, otherwise leave it blank and click on Next.
Certificate validation is initiated. If the validation fails (for example, if you selected server.key for the private key and ca.crt for the certificate), a message displays indicating the private key and certificate do not match.
When validation completes successfully, click FINISH.
- Under the Certificate tab, select the Trust Certificate tab > click +IMPORT. The Import Certificate wizard displays.
- In Alias field, provide any alias names, example: trustedCA. In the File field, click BROWSE to locate and import the appropriate trusted certificate. In our case, it is ca.crt on desktop. Click NEXT.
Click FINISH. After the import completes, review the trusted certificate details under the Trust Certificate tab.
Click RESTART SERVICES to apply certificate, and then click YES to verify you want to restart these services.
Reset all Certificates
Use Case:
- When MC or Rest API service down due to certificate issue, and unable to update certificates from AUI
- Avamar Upgrade workflow fails to update Java security for FIPS compliance, all keystores become either unreadable or keystore provider not being updated from none-FIPS compliance mode SUN to FIPS compliance mode JsafeJCE
- Misconfigured Certificates, would like to reset all certs back to default self-signed certificates and start again
Stop Avamar services
# dpnctl stop mcs && dpnctl stop emt && dpnctl stop avi
Reset MC certificate (keystore: /usr/local/avamar/lib/avamar_keystore):
# mv /usr/local/avamar/lib/avamar_keystore /usr/local/avamar/lib/avamar_keystore.bkp # mcrootca all
Reset MCSDK certificate (keystore: /usr/local/avamar/lib/rmi_ssl_keystore):
# mv /usr/local/avamar/lib/rmi_ssl_keystore /usr/local/avamar/lib/rmi_ssl_keystore.bkp # keytool -genkeypair -v -alias mcssl -keyalg RSA -sigalg SHA512withRSA -keysize 3072 -keystore /usr/local/avamar/lib/rmi_ssl_keystore -validity 3650 -dname "CN=Administrator, OU=Avamar, O=DELL-EMC, L=Irvine, ST=California, C=US" -storepass `ask_pass -r keystore_passphrase` # keytool -genkeypair -v -alias mcjwt -keyalg RSA -sigalg SHA512withRSA -keysize 3072 -keystore /usr/local/avamar/lib/rmi_ssl_keystore -validity 3650 -dname "CN=Administrator, OU=Avamar, O=DELL-EMC, L=Irvine, ST=California, C=US" -storepass `ask_pass -r keystore_passphrase`
Reset Admin certificate (keystore: for 19.4 or earlier versions: /home/admin/.keystore, for 19.5 or later versions: /home/apache/.keystore):
For Avamar 19.4 or earlier version: # mv /home/admin/.keystore /home/admin/.keystore.bkp # keytool -genkeypair -v -alias tomcat -keyalg RSA -sigalg SHA256withRSA -keysize 3072 -keystore /home/admin/.keystore -validity 3650 -dname "EMAILADDRESS=root, CN=`hostname -f`, OU=Dell EMC, O=Dell Technologies, L=Irvine, ST=California, C=US" -storepass `ask_pass -r keystore_passphrase` # keytool -list -alias mcssl -keystore /usr/local/avamar/lib/rmi_ssl_keystore -storepass `ask_pass -r keystore_passphrase` -rfc | keytool -import -trustcacerts -alias mcssl -keystore /home/admin/.keystore -storepass `ask_pass -r keystore_passphrase` -noprompt For Avamar 19.5 or later versions: # mv /home/apache/.keystore /home/apache/.keystore.bkp # keytool -genkeypair -v -alias tomcat -keyalg RSA -sigalg SHA256withRSA -keysize 3072 -keystore /home/apache/.keystore -validity 3650 -dname "EMAILADDRESS=root, CN=`hostname -f`, OU=Dell EMC, O=Dell Technologies, L=Irvine, ST=California, C=US" -storepass `ask_pass -r keystore_passphrase` # keytool -list -alias mcssl -keystore /usr/local/avamar/lib/rmi_ssl_keystore -storepass `ask_pass -r keystore_passphrase` -rfc | keytool -import -trustcacerts -alias mcssl -keystore /home/apache/.keystore -storepass `ask_pass -r keystore_passphrase` -noprompt
Reset Avinstaller certificate (keystore: /usr/local/avamar/lib/avi/avi_keystore):
# mv /usr/local/avamar/lib/avi/avi_keystore /usr/local/avamar/lib/avi/avi_keystore.bkp # gen-ssl-cert --norestart --noupdateapache --updateavi --keystorepwd=`ask_pass -r keystore_passphrase` --verbose # keytool -list -alias mcssl -keystore /usr/local/avamar/lib/rmi_ssl_keystore -storepass `ask_pass -r keystore_passphrase` -rfc | keytool -import -trustcacerts -alias mcssl -keystore /usr/local/avamar/lib/avi/avi_keystore -storepass `ask_pass -r keystore_passphrase` -noprompt
Reset Apache certificate (Note: Apache service would be restarted automatically):
# gen-ssl-cert --updateapache --noupdateavi --keystorepwd=`ask_pass -r keystore_passphrase` --verbose
Reset gsan certificate:
# enable_secure_config.sh --certs
Restart Avamar services:
# dpnctl stop mcs && dpnctl start mcs && dpnctl start sched # dpnctl stop emt && dpnctl start emt # avinstaller.pl --stop && avinstaller.pl --start
Update all client certificates (Only needed when MC root certificate updated)
# mccli client re-register-all
Make sure all services are up
Check that all services are running with the following command
# dpnctl status
Identity added: /home/admin/.ssh/admin_key (/home/admin/.ssh/admin_key)
dpnctl: INFO: gsan status: up
dpnctl: INFO: MCS status: up.
dpnctl: INFO: emt status: up.
dpnctl: INFO: Backup scheduler status: down.
dpnctl: INFO: Maintenance windows scheduler status: suspended.
dpnctl: INFO: Unattended startup status: enabled.
dpnctl: INFO: avinstaller status: up.
dpnctl: INFO: ConnectEMC status: up.
dpnctl: INFO: ddrmaint-service status: up.
If the Backup scheduler is down, start it
# dpnctl start sched Identity added: /home/admin/.ssh/admin_key (/home/admin/.ssh/admin_key) dpnctl: INFO: Resuming backup scheduler... dpnctl: INFO: Backup scheduler resumed. dpnctl: INFO: No /usr/local/avamar/var/dpn_service_status exist.
Restart Safely Avamar
https://www.dell.com/support/kbdoc/en-uk/000058451/avamar-instructions-to-safely-shutdown-and-restart-a-system-for-maintenance-purposes
- avosshutdown help (print this help message and exit)
- avosshutdown precheck (check if a clean Shutdown or Reboot can be performed)
- avosshutdown shutdown (System shutdown for all data nodes and utility node)
- avosshutdown reboot (Reboot all data nodes and utility node)
- avosshutdown reboot autorestart (On a multinode, automatically restart services after Shutdown or Reboot)
# avosshutdown precheck # avosshutdown reboot autorestart