OPSS - Unable Start Admin Server Observing JPS-10000: There was an internal error in the policy store Error
(Doc ID 2566107.1)
Last updated on MARCH 01, 2023
Applies to:
Oracle Platform Security for Java - Version 12.2.1.3.0 and laterInformation in this document applies to any platform.
Symptoms
On : 12.2.1.3.0 version, Java Platform Security
Unable Start Admin Server Observing the next error:
Jul 02, 2019 2:19:26 PM oracle.security.jps.internal.config.xml.XmlConfigurationFactory validateFileLocation
INFO: JPS CONFIG:/MW_HOME/domains/domain/config/fmwconfig/jps-config.xml
INFO: Found persistence provider "org.eclipse.persistence.jpa.PersistenceProvider". OpenJPA will not be used.
INFO: Found persistence provider "org.eclipse.persistence.jpa.PersistenceProvider". OpenJPA will not be used.
Jul 02, 2019 2:19:34 PM oracle.security.jps.internal.credstore.ldap.LdapCredentialStore init
WARNING: Could not create credential store instance. Reason oracle.security.jps.service.policystore.PolicyStoreConnectivityException: JPS-10000: There was an internal error in the policy store.
<Jul 2, 2019 2:19:34 PM UTC> <Error> <Security> <BEA-090892> <The loading of OPSS java security policy provider failed due to exception, see the exception stack trace or the server log file for root cause. If still see no obvious cause, enable the debug flag -Djava.security.debug=jpspolicy to get more information. Error message: JPS-01055: Could not create credential store instance. Reason oracle.security.jps.service.policystore.PolicyStoreConnectivityException: JPS-10000: There was an internal error in the policy store.>
<Jul 2, 2019 2:19:34 PM UTC> <Critical> <WebLogicServer> <BEA-000386> <Server subsystem failed. Reason: weblogic.security.SecurityInitializationException: The loading of OPSS java security policy provider failed due to exception, see the exception stack trace or the server log file for root cause. If still see no obvious cause, enable the debug flag -Djava.security.debug=jpspolicy to get more information. Error message: JPS-01055: Could not create credential store instance. Reason oracle.security.jps.service.policystore.PolicyStoreConnectivityException: JPS-10000: There was an internal error in the policy store.
weblogic.security.SecurityInitializationException: The loading of OPSS java security policy provider failed due to exception, see the exception stack trace or the server log file for root cause. If still see no obvious cause, enable the debug flag -Djava.security.debug=jpspolicy to get more information. Error message: JPS-01055: Could not create credential store instance. Reason oracle.security.jps.service.policystore.PolicyStoreConnectivityException: JPS-10000: There was an internal error in the policy store.
at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.loadOPSSPolicy(CommonSecurityServiceManagerDelegateImpl.java:1402)
at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.initialize(CommonSecurityServiceManagerDelegateImpl.java:1022)
at weblogic.security.service.SecurityServiceManager.initialize(SecurityServiceManager.java:888)
at weblogic.security.SecurityService.start(SecurityService.java:141)
at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)
Truncated. see log file for complete stacktrace
Caused By: oracle.security.jps.JpsRuntimeException: JPS-01055: Could not create credential store instance. Reason oracle.security.jps.service.policystore.PolicyStoreConnectivityException: JPS-10000: There was an internal error in the policy store.
at oracle.security.jps.internal.policystore.PolicyDelegationController.<init>(PolicyDelegationController.java:173)
at oracle.security.jps.internal.policystore.JavaPolicyProvider.<init>(JavaPolicyProvider.java:411)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
Truncated. see log file for complete stacktrace
Caused By: oracle.security.jps.JpsException: JPS-01055: Could not create credential store instance. Reason oracle.security.jps.service.policystore.PolicyStoreConnectivityException: JPS-10000: There was an internal error in the policy store.
at oracle.security.jps.internal.policystore.PolicyUtil.getDefaultPDPService(PolicyUtil.java:3021)
at oracle.security.jps.internal.policystore.PolicyUtil.getPDPService(PolicyUtil.java:3308)
at oracle.security.jps.internal.policystore.PolicyDelegationController.<init>(PolicyDelegationController.java:170)
at oracle.security.jps.internal.policystore.JavaPolicyProvider.<init>(JavaPolicyProvider.java:411)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
Truncated. see log file for complete stacktrace
Caused By: oracle.security.jps.service.credstore.CredStoreException: JPS-01055: Could not create credential store instance. Reason oracle.security.jps.service.policystore.PolicyStoreConnectivityException: JPS-10000: There was an internal error in the policy store.
at oracle.security.jps.internal.credstore.ldap.LdapCredentialStore.init(LdapCredentialStore.java:129)
at oracle.security.jps.internal.credstore.ldap.LdapCredentialStore.<init>(LdapCredentialStore.java:120)
at oracle.security.jps.internal.credstore.ldap.LdapCredentialStoreProvider.getInstance(LdapCredentialStoreProvider.java:135)
at oracle.security.jps.internal.credstore.rdbms.DbmsCredentialStoreProvider.getInstance(DbmsCredentialStoreProvider.java:68)
at oracle.security.jps.internal.credstore.rdbms.DbmsCredentialStoreProvider.getInstance(DbmsCredentialStoreProvider.java:46)
Truncated. see log file for complete stacktrace
>
<Jul 2, 2019 2:19:34 PM UTC> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to FAILED>
INFO: JPS CONFIG:/MW_HOME/domains/domain/config/fmwconfig/jps-config.xml
INFO: Found persistence provider "org.eclipse.persistence.jpa.PersistenceProvider". OpenJPA will not be used.
INFO: Found persistence provider "org.eclipse.persistence.jpa.PersistenceProvider". OpenJPA will not be used.
Jul 02, 2019 2:19:34 PM oracle.security.jps.internal.credstore.ldap.LdapCredentialStore init
WARNING: Could not create credential store instance. Reason oracle.security.jps.service.policystore.PolicyStoreConnectivityException: JPS-10000: There was an internal error in the policy store.
<Jul 2, 2019 2:19:34 PM UTC> <Error> <Security> <BEA-090892> <The loading of OPSS java security policy provider failed due to exception, see the exception stack trace or the server log file for root cause. If still see no obvious cause, enable the debug flag -Djava.security.debug=jpspolicy to get more information. Error message: JPS-01055: Could not create credential store instance. Reason oracle.security.jps.service.policystore.PolicyStoreConnectivityException: JPS-10000: There was an internal error in the policy store.>
<Jul 2, 2019 2:19:34 PM UTC> <Critical> <WebLogicServer> <BEA-000386> <Server subsystem failed. Reason: weblogic.security.SecurityInitializationException: The loading of OPSS java security policy provider failed due to exception, see the exception stack trace or the server log file for root cause. If still see no obvious cause, enable the debug flag -Djava.security.debug=jpspolicy to get more information. Error message: JPS-01055: Could not create credential store instance. Reason oracle.security.jps.service.policystore.PolicyStoreConnectivityException: JPS-10000: There was an internal error in the policy store.
weblogic.security.SecurityInitializationException: The loading of OPSS java security policy provider failed due to exception, see the exception stack trace or the server log file for root cause. If still see no obvious cause, enable the debug flag -Djava.security.debug=jpspolicy to get more information. Error message: JPS-01055: Could not create credential store instance. Reason oracle.security.jps.service.policystore.PolicyStoreConnectivityException: JPS-10000: There was an internal error in the policy store.
at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.loadOPSSPolicy(CommonSecurityServiceManagerDelegateImpl.java:1402)
at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.initialize(CommonSecurityServiceManagerDelegateImpl.java:1022)
at weblogic.security.service.SecurityServiceManager.initialize(SecurityServiceManager.java:888)
at weblogic.security.SecurityService.start(SecurityService.java:141)
at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)
Truncated. see log file for complete stacktrace
Caused By: oracle.security.jps.JpsRuntimeException: JPS-01055: Could not create credential store instance. Reason oracle.security.jps.service.policystore.PolicyStoreConnectivityException: JPS-10000: There was an internal error in the policy store.
at oracle.security.jps.internal.policystore.PolicyDelegationController.<init>(PolicyDelegationController.java:173)
at oracle.security.jps.internal.policystore.JavaPolicyProvider.<init>(JavaPolicyProvider.java:411)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
Truncated. see log file for complete stacktrace
Caused By: oracle.security.jps.JpsException: JPS-01055: Could not create credential store instance. Reason oracle.security.jps.service.policystore.PolicyStoreConnectivityException: JPS-10000: There was an internal error in the policy store.
at oracle.security.jps.internal.policystore.PolicyUtil.getDefaultPDPService(PolicyUtil.java:3021)
at oracle.security.jps.internal.policystore.PolicyUtil.getPDPService(PolicyUtil.java:3308)
at oracle.security.jps.internal.policystore.PolicyDelegationController.<init>(PolicyDelegationController.java:170)
at oracle.security.jps.internal.policystore.JavaPolicyProvider.<init>(JavaPolicyProvider.java:411)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
Truncated. see log file for complete stacktrace
Caused By: oracle.security.jps.service.credstore.CredStoreException: JPS-01055: Could not create credential store instance. Reason oracle.security.jps.service.policystore.PolicyStoreConnectivityException: JPS-10000: There was an internal error in the policy store.
at oracle.security.jps.internal.credstore.ldap.LdapCredentialStore.init(LdapCredentialStore.java:129)
at oracle.security.jps.internal.credstore.ldap.LdapCredentialStore.<init>(LdapCredentialStore.java:120)
at oracle.security.jps.internal.credstore.ldap.LdapCredentialStoreProvider.getInstance(LdapCredentialStoreProvider.java:135)
at oracle.security.jps.internal.credstore.rdbms.DbmsCredentialStoreProvider.getInstance(DbmsCredentialStoreProvider.java:68)
at oracle.security.jps.internal.credstore.rdbms.DbmsCredentialStoreProvider.getInstance(DbmsCredentialStoreProvider.java:46)
Truncated. see log file for complete stacktrace
>
<Jul 2, 2019 2:19:34 PM UTC> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to FAILED>
Reviewed the DB connection properties, the connectivity between Domain and DB, and the ability to connect with DB trough SqlPlus. All of that correct.
As the error involved OPSS DB connection, Enabled DB debug trace, following:
How to Set the (Eclipselink.logging.level) to the FINER Logging? (Doc ID 2296072.1)
Seeing connection working fine, visible in the nohup file:
[EL Config]: 2019-07-03 12:41:56.515--ServerSession(1050829295)--Connection(655780620)--Thread(Thread[[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)',5,Pooled Threads])--connecting(DatabaseLogin(
platform=><PLATFORM>
user name=> "<DB_USER>"
datasource URL=> "jdbc:oracle:thin:@//<HOST>:<PORT>/<SERVICE>"
))
[EL Config]: 2019-07-03 12:41:56.607--ServerSession(1050829295)--Connection(1426699338)--Thread(Thread[[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)',5,Pooled Threads])--Connected: jdbc:oracle:thin:@//<HOST>:<PORT>/<SERVICE>
User: <DB_USER>
Database: Oracle Version: Oracle Database 11g Release 11.2.0.4.0 - 64bit Production
Driver: Oracle JDBC driver Version: 11.2.0.3.0
[EL Info]: 2019-07-03 12:41:57.294--ServerSession(1050829295)--Thread(Thread[[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)',5,Pooled Threads])--file:/u01/app/oracle/middleware/oracle_common/modules/oracle.jps_11.1.1/jps-internal.jar_JpsDBDataManagerV3_url=jdbc:oracle:thin:@//<HOST>:<PORT>/<SERVICE>_user=<DB_USER> login successful
platform=><PLATFORM>
user name=> "<DB_USER>"
datasource URL=> "jdbc:oracle:thin:@//<HOST>:<PORT>/<SERVICE>"
))
[EL Config]: 2019-07-03 12:41:56.607--ServerSession(1050829295)--Connection(1426699338)--Thread(Thread[[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)',5,Pooled Threads])--Connected: jdbc:oracle:thin:@//<HOST>:<PORT>/<SERVICE>
User: <DB_USER>
Database: Oracle Version: Oracle Database 11g Release 11.2.0.4.0 - 64bit Production
Driver: Oracle JDBC driver Version: 11.2.0.3.0
[EL Info]: 2019-07-03 12:41:57.294--ServerSession(1050829295)--Thread(Thread[[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)',5,Pooled Threads])--file:/u01/app/oracle/middleware/oracle_common/modules/oracle.jps_11.1.1/jps-internal.jar_JpsDBDataManagerV3_url=jdbc:oracle:thin:@//<HOST>:<PORT>/<SERVICE>_user=<DB_USER> login successful
At some point of the trace nohup file, seeing a kill message, in the middle of the OPSS DB processes:
[EL Finer]: 2019-07-03 12:42:00.447--ClientSession(1095138782)--Thread(Thread[[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)',5,Pooled Threads])--acquire unit of work: 2062654284
[EL Fine]: 2019-07-03 12:42:00.447--ServerSession(1050829295)--Connection(544443086)--Thread(Thread[[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)',5,Pooled Threads])--SELECT t0.JPS_ATTRS_ID, t0.ATTRNAME, t0.ATTRVAL, t0.JPS_DN_ENTRYID FROM JPS_ATTRS t0, JPS_DN t1 WHERE ((t0.JPS_DN_ENTRYID = t1.ENTRYID) AND (((t1.RDN = ?) AND (t1.PARENTDN = ?)) OR t1.PARENTDN LIKE ? ESCAPE ?))
bind => [cn=systemcomponent, cn=jpsroot,cn=jpscontext,cn=javaprod_domain,cn=auditstore,cn=xlf,, cn=jpsroot,cn=jpscontext,cn=javaprod\_domain,cn=auditstore,cn=xlf,cn=systemcomponent,%, \]
./startWebLogic.sh: line 180: 2609 Killed ${JAVA_HOME}/bin/java ${JAVA_VM} ${MEM_ARGS} -Dweblogic.Name=${SERVER_NAME} -Djava.security.policy=${WL_HOME}/server/lib/weblogic.policy ${JAVA_OPTIONS} ${PROXY_SETTINGS} ${SERVER_CLASS}
[EL Fine]: 2019-07-03 12:42:00.447--ServerSession(1050829295)--Connection(544443086)--Thread(Thread[[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)',5,Pooled Threads])--SELECT t0.JPS_ATTRS_ID, t0.ATTRNAME, t0.ATTRVAL, t0.JPS_DN_ENTRYID FROM JPS_ATTRS t0, JPS_DN t1 WHERE ((t0.JPS_DN_ENTRYID = t1.ENTRYID) AND (((t1.RDN = ?) AND (t1.PARENTDN = ?)) OR t1.PARENTDN LIKE ? ESCAPE ?))
bind => [cn=systemcomponent, cn=jpsroot,cn=jpscontext,cn=javaprod_domain,cn=auditstore,cn=xlf,, cn=jpsroot,cn=jpscontext,cn=javaprod\_domain,cn=auditstore,cn=xlf,cn=systemcomponent,%, \]
./startWebLogic.sh: line 180: 2609 Killed ${JAVA_HOME}/bin/java ${JAVA_VM} ${MEM_ARGS} -Dweblogic.Name=${SERVER_NAME} -Djava.security.policy=${WL_HOME}/server/lib/weblogic.policy ${JAVA_OPTIONS} ${PROXY_SETTINGS} ${SERVER_CLASS}
Changes
No changes, the problem appeared after restart the Server
Cause
To view full details, sign in with your My Oracle Support account. |
|
Don't have a My Oracle Support account? Click to get started! |
In this Document
| Symptoms |
| Changes |
| Cause |
| Solution |
| References |