My Oracle Support Banner

EM 13.5 : Jobs Submitted with SSH Key Credentials Hang in Running Status Forever or Fails with "Job system internal error" (Doc ID 2798510.1)

Last updated on SEPTEMBER 16, 2021

Applies to:

Enterprise Manager Base Platform - Version 13.5.1.0.0 and later
Information in this document applies to any platform.

Symptoms

In EM 13.5 environment, when attempting to run Long running jobs, using SSH Key Host credentials, the job hangs and never completes.
Intermittently jobs may fail with "Job system internal error".

Same jobs work fine using direct user credentials instead of SSH Key Credentials.

$AGENT_INST_HOME/sysman/gcagent.log reports following errors:

java.lang.NullPointerException
at oracle.sysman.gcagent.sshclient.SSHProcess.exitValue(SSHProcess.java:158)
at oracle.sysman.gcagent.util.auth.os.ssh.SSHAuthProcess.destroy(SSHAuthProcess.java:303)
at oracle.sysman.gcagent.util.auth.os.ssh.SSHAuthProcess.destroy(SSHAuthProcess.java:257)
at oracle.sysman.gcagent.util.channel.OSSecureChannel.terminate(OSSecureChannel.java:447)
at oracle.sysman.gcagent.target.interaction.execution.JobExecutionMgr$CmdJobConnection.terminate(JobExecutionMgr.java:492)
at oracle.sysman.gcagent.target.interaction.execution.JobExecutionMgr$JobTask.terminate(JobExecutionMgr.java:931)
at oracle.sysman.gcagent.jobs.JobRequestDriver$JobReaper.startReaping(JobRequestDriver.java:1132)
at oracle.sysman.gcagent.jobs.JobRequestDriver$JobReaper.run(JobRequestDriver.java:735)
at oracle.sysman.gcagent.util.system.GCAThread$RunnableWrapper.run(GCAThread.java:198)
at java.lang.Thread.run(Thread.java:748)

OR

2021-08-09 04:21:12,154 [33:301163D6:GC.SysExecutor.0 (SendJobTaskStatistics-1660428)] WARN - SendJobTaskStatistics-1660428
oracle.sysman.gcagent.jobs.AsynchronousJobTaskAttachment$StreamClosedException: JobReceiverOutputStream[stepId:1660428] is CLOSED
at oracle.sysman.gcagent.jobs.AsynchronousJobTaskAttachment.sendNotification(AsynchronousJobTaskAttachment.java:1255)
at oracle.sysman.gcagent.jobs.AsynchronousJobTaskAttachment.access$500(AsynchronousJobTaskAttachment.java:97)
at oracle.sysman.gcagent.jobs.AsynchronousJobTaskAttachment$Statistics$SendStatisticsRequestTask.call(AsynchronousJobTaskAttachment.java:843)
at oracle.sysman.gcagent.jobs.AsynchronousJobTaskAttachment$Statistics$SendStatisticsRequestTask.call(AsynchronousJobTaskAttachment.java:828)
at oracle.sysman.gcagent.task.scheduler.DispatchingTaskScheduler$ReschedulingHelper$ReschedulingTask.call(DispatchingTaskScheduler.java:470)
at oracle.sysman.gcagent.task.scheduler.DispatchingTaskScheduler$ReschedulingHelper$ReschedulingTask.call(DispatchingTaskScheduler.java:432)
at oracle.sysman.gcagent.task.executor.DiagWrappedTask.call(DiagWrappedTask.java:60)

Debug logs may show the following messages as well in gcagent.log

2021-07-30 19:28:03,048 [6069:BF41B227] DEBUG - java.lang.IllegalThreadStateException: Process is still running: caught in JobReaper
java.lang.IllegalThreadStateException: Process is still running
at oracle.sysman.gcagent.sshclient.SSHProcess.exitValue(SSHProcess.java:163)
at oracle.sysman.gcagent.sshclient.SSHProcess.waitFor(SSHProcess.java:197)
at oracle.sysman.gcagent.util.auth.os.AuthProcess.waitFor(AuthProcess.java:355)
at oracle.sysman.gcagent.util.auth.os.ssh.SSHAuthProcess.waitFor(SSHAuthProcess.java:239)
at oracle.sysman.gcagent.util.channel.OSStreamingChannel.waitFor(OSStreamingChannel.java:119)

 

 

Cause

To view full details, sign in with your My Oracle Support account.

Don't have a My Oracle Support account? Click to get started!


In this Document
Symptoms
Cause
Solution
References


My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.