PCA 2.3.x/2.4.x Upgrade not allowed if Certificates have been regenerated using ovmkeytool.sh. (Doc ID 2597439.1)

Last updated on SEPTEMBER 24, 2020

Applies to:

Symptoms

PCA upgrade will fail when SSL certificates are pointing to shared storage /nfs/shared_storage/wls1.
The following errors can be found in the pca_upgrader.log:

Starting ovmm
Starting Oracle VM Manager[OK]

Checking to see whether ovmcli is running
Attempting to start ovmcli
Starting ovmcli
Starting Oracle VM Manager CLI [OK]

No handlers could be found for logger "ovca.ovm"
Traceback (most recent call last):
File "/nfs/shared_storage/pca_upgrader/scripts/1.0-97.el6/run_ovm_upgrade", line 236, in <module>
main()
File "/nfs/shared_storage/pca_upgrader/scripts/1.0-97.el6/run_ovm_upgrade", line 214, in main
start_ovm_services()
File "/nfs/shared_storage/pca_upgrader/scripts/1.0-97.el6/run_ovm_upgrade", line 117, in start_ovm_services
if not wait_for_ovm():
File "/usr/lib/python2.6/site-packages/ovca/ovm.py", line 179, in wait_for_ovm
raise Exception('OVMM core is not ready to accept connections after 10 minutes')
Exception: OVMM core is not ready to accept connections after 10 minutes
[root@ovcamn05r1:~]
[2019-07-05 06:14:59 598784] ERROR (mn_upgrade_utils:735) Oracle VM upgrade of ovcamn05r1 management node failed. Please contact support
[2019-07-05 06:15:00 598784] ERROR (mn_upgrade_steps:426) Failure during Oracle VM upgrade on ovcamn05r1

AdminServer.log from the active Management Node shows the following messages:

####<2019-07-05T09:51:15.605+0000> <Error> <WebLogicServer> <ovcamn06r1> <AdminServer> <[ACTIVE] ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuning)'> <> <> <ee0a31dc-74c7-4bed-86d5-e7332f8f0a97-00000006> <1562320275605> <BEA-000297> <Inconsistent security configuration, weblogic.management.configuration.ConfigurationException: Cannot find identity keystore file /nfs/shared_storage/wls1/security/ovmm.jks on server AdminServer>
####<2019-07-05T09:51:15.605+0000> <Emergency> <Security> <ovcamn06r1> <AdminServer> <[ACTIVE] ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuning)'> <> <> <ee0a31dc-74c7-4bed-86d5-e7332f8f0a97-00000006> <1562320275605> <BEA-090034> <Not listening for SSL, java.io.IOException: Cannot find identity keystore file /nfs/shared_storage/wls1/security/ovmm.jks on server AdminServer.>

<..>

####<2019-07-05T09:55:06.610+0000> <Warning> <com.oracle.appfw.ovm.tasks.Synchronizer> <ovcamn06r1> <AdminServer> <Thread-76> <> <> <ee0a31dc-74c7-4bed-86d5-e7332f8f0a97-00000004> <1562320506610> <BEA-000000> <Unable to connect to core. Retrying in 30 seconds.>
####<2019-07-05T09:55:36.751+0000> <Warning> <com.oracle.appfw.ovm.tasks.Synchronizer> <ovcamn06r1> <AdminServer> <Thread-76> <> <> <ee0a31dc-74c7-4bed-86d5-e7332f8f0a97-00000004> <1562320536751> <BEA-000000> <Unable to connect to core. Retrying in 30 seconds.>

The following command on the active Management Node will show that the certificates are pointing to shared storage /nfs/shared_storage/wls1.
Look for the entries 'CA Keystore File', 'SSL Keystore File' and 'SSL Trust Keystore File' in the ouput:

# export MW_HOME=/u01/app/oracle/Middleware
# /u01/app/oracle/ovm-manager-3/ovm_upgrade/bin/ovmkeytool.sh show
Jan 28, 2019 6:22:13 AM oracle.security.jps.JpsStartup start
INFO: Jps initializing.
Jan 28, 2019 6:22:15 AM oracle.security.jps.JpsStartup start
INFO: Jps started.
CA Keystore File: /nfs/shared_storage/wls1/security/ovmca.jks
CA Key Alias: ca
Certificate details:
Algorithm: SHA256withRSA

<..>
-----BEGIN CERTIFICATE-----

<..>

-----END CERTIFICATE-----

SSL Keystore File: /nfs/shared_storage/wls1/security/ovmssl.jks
SSL Key Alias: ovmcore

<..>

-----BEGIN CERTIFICATE-----

<..>

-----END CERTIFICATE-----

SSL Trust Keystore File: /nfs/shared_storage/wls1/security/ovmtrust.jks
Trusted certificates: 

Changes

The SSL Certificates of the Oracle VM Manager have been re-generated using the ovmkeytool.sh tool with the 'setup' option.
This could happen when i.e. installing a custom SSL Certificate.

Cause

My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.