EM 12c: emctl start agent Fails or Hangs or Timeout or Connection Refused Due to PKCS11 'security.pkcs11.P11SecureRandom.implNextBytes' Reported in Thread Dump 'Hang detected' (Doc ID 1427773.1)

Last updated on SEPTEMBER 08, 2016

Applies to:

Enterprise Manager Base Platform - Version 12.1.0.1.0 to 12.1.0.2.0 [Release 12.1]
Oracle Solaris on SPARC (64-bit)

Symptoms

$AGENT_INST/bin/emctl start agent

fails to start or gets hung

Oracle Enterprise Manager Cloud Control 12c Release 2
Copyright (c) 1996, 2012 Oracle Corporation.  All rights reserved.
Starting agent ...........................................................................................

OR

$AGENT_INST/bin/emctl start agent

fails with following error

Oracle Enterprise Manager 12c Cloud Control 12.1.0.1.0
Copyright (c) 1996, 2012 Oracle Corporation. All rights reserved.
Agent status could not be determined. Check the agent process
Consult the log files in: /oracle/product/Agent_home/agent_inst/sysman/log

$AGENT_INST/bin/emctl status agent

Oracle Enterprise Manager 12c Cloud Control 12.1.0.1.0
Copyright (c) 1996, 2012 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
Status agent Failure: Timed out after 300 seconds

 

/AGENT_INST/sysman/log/emagent.nohup reports th following message

---- Mon Jan 23 07:53:48 2012::8748::EMAgent exited at Mon Jan 23 07:53:48
2012 with signal 9 -----
----- Mon Jan 23 07:53:48 2012::8748::EMAgent either hung or in abnormal state.
----- Mon Jan 23 07:53:48 2012::8748::EMAgent will be restarted/thrashed.
----- Mon Jan 23 07:53:48 2012::8748::Restarting EMAgent.
..
----- Tue Jan 24 13:11:10 2012::27420::EMAgent exited at Tue Jan 24 13:11:10 2012 with signal 9 -----
----- Tue Jan 24 13:11:10 2012::27420::EMAgent has been forcibly killed.

OR

/AGENT_INST/sysman/log/emagent.nohup reports th following message

----- Wed Mar 25 06:13:00 2015::24757::Hang detected for EMAgent : 18731 -----
----- Wed Mar 25 06:13:00 2015::24757::Debugging component EMAgent -----
----- Wed Mar 25 06:13:00 2015::generate first thread dump file for diagnosis -----
----- Wed Mar 25 06:30:43 2015::generate second thread dump file for diagnosis -----
----- Wed Mar 25 06:30:59 2015::generate Threads.18731lsof.1 for diagnosis -----
----- Attempting to kill EMAgent : 18731 -----

 

When agent is unresponsive or hung, thread dump (kill -3 <agent process id>  or  jstack) shows the PKCS11 thread remain in RUNNABLE state

"GC.DPExecutor2.9
 (CDProps:oracle_database.emrepos.hscdb4.hsc.usf.edu:decideIncludeDB)" prio=3
 tid=0x0000000102875000 nid=0x4f runnable [0xffffffff6a4fd000]
 java.lang.Thread.State: RUNNABLE
 at sun.security.pkcs11.wrapper.PKCS11.C_GenerateRandom(Native Method)
 at
 sun.security.pkcs11.P11SecureRandom.implNextBytes(P11SecureRandom.java:166)
 at
 sun.security.pkcs11.P11SecureRandom.engineNextBytes(P11SecureRandom.java:118)
 - locked <0xffffffff44056740> (a [B)
 at java.security.SecureRandom.nextBytes(SecureRandom.java:433)
 - locked <0xffffffff453860a0> (a java.security.SecureRandom)

Multiple JAVA processes gets started from <AGENT_HOME>/core/{version}/jdk/bin location

$ps -ef | grep agent
oracle 23991 23910   0 09:24:18 pts/1       0:00 grep agent
oracle 17235     1   0 15:50:44 ?           0:00 /u01/app/oracle/product/agent12c/core/12.1.0.2.0/jdk/bin/sparcv9/java -Xmx128M
oracle 17173     1   0 15:50:41 ?           0:00 /u01/app/oracle/product/agent12c/core/12.1.0.2.0/jdk/bin/sparcv9/java -Xmx128M
oracle 17025     1   0 15:50:33 ?           0:00 /u01/app/oracle/product/agent12c/core/12.1.0.2.0/jdk/bin/sparcv9/java -Xmx128M
oracle 17181     1   0 15:50:42 ?           0:00 /u01/app/oracle/product/agent12c/core/12.1.0.2.0/jdk/bin/sparcv9/java -Xmx128M

 

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms