Agile PLM 9.3.3 Managed servers fail to start After Applying HF320

(Doc ID 2317156.1)

Last updated on OCTOBER 12, 2017

Applies to:

Oracle Agile PLM Framework - Version 9.3.3.0 and later
Information in this document applies to any platform.

Goal

Completely reinstalled the application from scratch. The app is a cluster setup ( 1 admin + 4 managed servers )

App01 server - 1 admin node, 2 ms
App02 server - only 2 ms

When apply HF320 again and run the encryption process, the encryption process now works, but then run into a new issue where  could not start any managed server nodes on app02 at all, they failed with some errors.

=======================================================================================================================================

CLASSPATH=/data/agile/Oracle/Middleware/patch_wls1211/profiles/default/sys_manifest_classpath/weblogic_patch.jar:/usr/java/jdk1.7.0_80/lib/tools.jar:/data/agile/Oracle/Middleware/wlserver_12.1/server/lib/weblogic_sp.jar:/data/agile/Oracle/Middleware/wlserver_12.1/server/lib/weblogic.jar:/data/agile/Oracle/Middleware/modules/features/weblogic.server.modules_12.1.1.0.jar:/data/agile/Oracle/Middleware/wlserver_12.1/server/lib/webservices.jar:/data/agile/Oracle/Middleware/modules/org.apache.ant_1.7.1/lib/ant-all.jar:/data/agile/Oracle/Middleware/modules/net.sf.antcontrib_1.1.0.0_1-0b2/lib/ant-contrib.jar:

PATH=/data/agile/Oracle/Middleware/wlserver_12.1/server/bin:/data/agile/Oracle/Middleware/modules/org.apache.ant_1.7.1/bin:/usr/java/jdk1.7.0_80/jre/bin:/usr/java/jdk1.7.0_80/bin:/usr/lib/qt-3.3/bin:/usr/local/bin:/bin:/usr/bin:/data/svc-agile/bin:/data/svc-agile/Agile:/data/agile/agile933/jdk:/data/svc-agile/bin

Your environment has been set.
AgileAuthenticationProviderImpl.initialize
log4j:WARN No appenders could be found for logger (com.agile.util.sql.AgileUtil).
log4j:WARN Please initialize the log4j system properly.
java.util.NoSuchElementException: Hashtable Enumerator
at java.util.Hashtable$Enumerator.nextElement(Hashtable.java:1157)
at com.agile.util.crypto.ContainerCryptoUtil.loadAliases(ContainerCryptoUtil.java:151)
at com.agile.util.crypto.ContainerCryptoUtil.<clinit>(ContainerCryptoUtil.java:78)
at com.agile.admin.security.weblogic.WLSLoginModule.login(WLSLoginModule.java:193)
at com.bea.common.security.internal.service.LoginModuleWrapper$1.run(LoginModuleWrapper.java:110)
at java.security.AccessController.doPrivileged(Native Method)
at com.bea.common.security.internal.service.LoginModuleWrapper.login(LoginModuleWrapper.java:106)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at javax.security.auth.login.LoginContext.invoke(LoginContext.java:762)
at javax.security.auth.login.LoginContext.access$000(LoginContext.java:203)
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:690)
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:688)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:687)
at javax.security.auth.login.LoginContext.login(LoginContext.java:595)
at com.bea.common.security.internal.service.JAASLoginServiceImpl.login(JAASLoginServiceImpl.java:113)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.bea.common.security.internal.utils.Delegator$ProxyInvocationHandler.invoke(Delegator.java:64)
at com.sun.proxy.$Proxy13.login(Unknown Source)
at weblogic.security.service.internal.WLSJAASLoginServiceImpl$ServiceImpl.login(WLSJAASLoginServiceImpl.java:89)
at com.bea.common.security.internal.service.JAASAuthenticationServiceImpl.authenticate(JAASAuthenticationServiceImpl.java:82)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.bea.common.security.internal.utils.Delegator$ProxyInvocationHandler.invoke(Delegator.java:64)
at com.sun.proxy.$Proxy34.authenticate(Unknown Source)
at weblogic.security.service.WLSJAASAuthenticationServiceWrapper.authenticate(WLSJAASAuthenticationServiceWrapper.java:40)
at weblogic.security.service.PrincipalAuthenticator.authenticate(PrincipalAuthenticator.java:338)
at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.doBootAuthorization(CommonSecurityServiceManagerDelegateImpl.java:930)
at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.initialize(CommonSecurityServiceManagerDelegateImpl.java:1054)
at weblogic.security.service.SecurityServiceManager.initialize(SecurityServiceManager.java:888)
at weblogic.security.SecurityService.start(SecurityService.java:148)
at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:256)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)
*** Can not initialize key store from agileks.jks. Encryption service will fail.
Error: Wrong Keystore password
<Oct 9, 2017 6:00:10 PM MDT> <Critical> <Security> <BEA-090402> <Authentication denied: Boot identity not valid; The user name and/or password from the boot identity file (boot.properties) is not valid. The boot identity may have been changed since the boot identity file was created. Please edit and update the boot identity file with the proper values of username and password. The first time the updated boot identity file is used to start the server, these new values are encrypted.>
<Oct 9, 2017 6:00:10 PM MDT> <Critical> <WebLogicServer> <BEA-000386> <Server subsystem failed. Reason: weblogic.security.SecurityInitializationException: Authentication denied: Boot identity not valid; The user name and/or password from the boot identity file (boot.properties) is not valid. The boot identity may have been changed since the boot identity file was created. Please edit and update the boot identity file with the proper values of username and password. The first time the updated boot identity file is used to start the server, these new values are encrypted.
weblogic.security.SecurityInitializationException: Authentication denied: Boot identity not valid; The user name and/or password from the boot identity file (boot.properties) is not valid. The boot identity may have been changed since the boot identity file was created. Please edit and update the boot identity file with the proper values of username and password. The first time the updated boot identity file is used to start the server, these new values are encrypted.
at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.doBootAuthorization(CommonSecurityServiceManagerDelegateImpl.java:960)
at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.initialize(CommonSecurityServiceManagerDelegateImpl.java:1054)
at weblogic.security.service.SecurityServiceManager.initialize(SecurityServiceManager.java:888)
at weblogic.security.SecurityService.start(SecurityService.java:148)
at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)
Truncated. see log file for complete stacktrace
Caused By: javax.security.auth.login.LoginException: Login Failure: all modules ignored
at javax.security.auth.login.LoginContext.invoke(LoginContext.java:913)
at javax.security.auth.login.LoginContext.access$000(LoginContext.java:203)
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:690)
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:688)
at java.security.AccessController.doPrivileged(Native Method)
Truncated. see log file for complete stacktrace
>
<Oct 9, 2017 6:00:10 PM MDT> <Error> <WebLogicServer> <BEA-000383> <A critical service failed. The server will shut itself down.>

 

Solution

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms