Custom Certificates Are Overwritten During Mammoth Actions Resulting in SSL errors and CM Services Failing to Start
(Doc ID 2392571.1)
Last updated on JANUARY 28, 2020
Applies to:Big Data Appliance Integrated Software - Version 4.8.0 and later
Mammoth actions like upgrade, cluster expansion, reprovision can cause custom certificates to be overwritten. Both documents, Prerequisites Required Prior to Server Resiliency Actions on the BDA (Doc ID 2240618.1) and Upgrading Oracle Big Data Appliance(BDA) CDH Cluster to V4.11.0 from V4.7, V4.8, V4.9, V4.10 Releases using Mammoth Software Upgrade Bundle for OL6 (Doc ID 2346478.1), have a pre-server resilency and upgrade prerequisite section respectively for backing up the custom files which might get overwritten.
This can happen after reboot, during Step 5, and then between the end of Step 5 and beginning of Step 7, and during server resiliency actions like node migration in between the steps to InstallHadoop and the next step to StartHadoopServices.
The symptoms of this are SSL errors in Cloudera Manager (CM), CM services failing to start, and agents no longer heart-beating into the cluster. The impact of overwriting the custom setting depends on what is overwritten. Cloudera Manager SCM agents can stop heartbeating into the cluster if the config.ini is reset to the default. CM services will not come up healthy if CM parameters are reset to the default.
Any custom certificates can be overwritten, including internally signed certificates.
1 The Cloudera Manager certificates overwritten are:
a) Cloudera Manager TLS/SSL Certificate Trust Store File.
Navigate: Administration > Settings > Search: Cloudera Manager TLS/SSL Certificate Trust Store File
b) TLS/SSL Client Truststore File Location.
Navigate: Administration > Settings > Search: TLS/SSL Client Truststore File Location
2 Hue configurations overwritten are:
a) Hue TLS/SSL Server Certificate File (PEM Format)
Navigate: hue > Configuration > Search: Hue TLS/SSL Server Certificate File (PEM Format)
b) HUE TLS/SSL Server Private Key File (PEM Format)
Navigate: hue > Configuration > Search: HUE TLS/SSL Server Private Key File (PEM Format)
c) Hue TLS/SSL Server CA Certificate (PEM Format)
Navigate: hue > Configuration > Search: Hue TLS/SSL Server CA Certificate (PEM Format)
3. /etc/cloudera-scm-agent/config.ini can be overwritten on all nodes such that the custom internal certificate setup in that file is entirely replaced by a default config.ini.
On upgrade until Step 7 is completed you can check the md5sum of the "good" copy of /etc/cloudera-scm-agent/config.ini and the current one to see if it is overwritten.
4. Other customer certificates under /opt/cloudera/security can be overwritten as well.
From the 4.11 upgrade document, Upgrading Oracle Big Data Appliance(BDA) CDH Cluster to V4.11.0 from V4.7, V4.8, V4.9, V4.10 Releases using Mammoth Software Upgrade Bundle for OL6 (Doc ID 2346478.1), see the prerequisite section "Prerequisites for $JAVA_HOME/jre/lib/security/jssecacerts, custom $JAVA_HOME/jre/lib/security/cacerts, and custom certificates" which has the backup steps so that the custom certificates can be restored after being overwritten.
To view full details, sign in with your My Oracle Support account.
Don't have a My Oracle Support account? Click to get started!
In this Document
|Proactive Workaround Before Custom Certificates are Overwritten|
|Reactive Workaround After Custom Certificates are Overwritten|