AGENT UNABLE TO START ORA.ASM DUE TO PROXYAGENT ERROR
(Doc ID 2425931.1)
Last updated on JUNE 18, 2023
Applies to:
Oracle Database - Enterprise Edition - Version 12.1.0.2 to 12.2.0.1 [Release 12.1 to 12.2]Information in this document applies to any platform.
Symptoms
ora.asm is failed to start and status shows offline or intermediate.
$crsctl stat res -t -init
--------------------------------------------------------------------------------
Name Target State Server State details
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.asm 1 ONLINE INTERMEDIATE
While starting the ora.asm getting below errors.
$ srvctl start asm
CRS-2674: Start of 'ora.asm' on 'node6' failed
CRS-5804: Communication error with agent process
CRS-2674: Start of 'ora.asm' on 'node6' failed
CRS-2528: Unable to place an instance of 'ora.asm' as all possible servers are occupied by the resource
Alert.log:
2016-09-07 14:13:30.012 [ORAAGENT(498)]CRS-5011: Check of resource "ora.asm" failed: details at "(:CLSN00006:)" in /scratch/oracle/base/diag/crs/node6/crs/trace/ohasd_oraagent_grid.trc"
2016-09-07 14:13:30.162 [ORAAGENT(498)]CRS-5011: Check of resource "ora.asm" failed: details at "(:CLSN00006:)" in "/scratch/oracle/base/diag/crs/node6/crs/trace/ohasd_oraagent_grid.trc"
2016-09-07 14:13:30.166 [ORAAGENT(498)]CRS-5011: Check of resource "ora.asm" failed: details at "(:CLSN00006:)" in "/scratch/oracle/base/diag/crs/node6/crs/trace/ohasd_oraagent_grid.trc"
ohasd_oraagent_grid_2.trc:
2016-09-07 14:13:29.990 : AGFW:3877193472: {1:23503:371} Agent received the message: CLSAGFW_RESOURCE_ACTION[ora.asm 1 1] ID 4102:1636
2016-09-07 14:13:29.991 :CLSDYNAM:3487557376: [ ora.asm]{1:23503:371} [action] (:CLSN00114:) clsn_agent::action {
2016-09-07 14:13:29.991 :CLSDYNAM:3487557376: [ ora.asm]{1:23503:371} [action] Agent::getResAgent resname:ora.asm version:12.1.0.1.0
2016-09-07 14:13:29.991 :CLSDYNAM:3487557376: [ ora.asm]{1:23503:371} [action] Agent::getAgent __IS_HASD_AGENT = TRUE
2016-09-07 14:13:29.993 :CLSDYNAM:3487557376: [ ora.asm]{1:23503:371} [action] Agent::Agent pAgent:0x7f01bc0ea8a0 &m_resName:0x7f01bc0ea920 resname:ora.asm
2016-09-07 14:13:29.993 :CLSDYNAM:3487557376: [ ora.asm]{1:23503:371} [action] Agent::Agent pAgent:0x7f01bc0c5680 &m_resName:0x7f01bc0c5700 resname:ora.dummyOcrDg.dg
2016-09-07 14:13:29.993 :CLSDYNAM:3487557376: [ ora.asm]{1:23503:371} [action] DgpAgent::DgpAgent: enter { this:0x7f01bc0c5680
2016-09-07 14:13:29.993 :CLSDYNAM:3487557376: [ ora.asm]{1:23503:371} [action] DgpAgent::DgpAgent: resname ora.dummyOcrDg.dg compId dummyOcrDg asmCommonAgent 0x7f01bc0eaab0
2016-09-07 14:13:29.993 :CLSDYNAM:3487557376: [ ora.asm]{1:23503:371} [action] DgpAgent::DgpAgent: exit }
2016-09-07 14:13:29.993 :CLSDYNAM:3487557376: [ ora.asm]{1:23503:371} [action] AsmCommonAgent::AsmCommonAgent m_pDgpAgent:bc0c5680
2016-09-07 14:13:29.994 : USRTHRD:3487557376: {1:23503:371} Thread:InstStartThread: thread constructor exit this:bc0ea2f0 m_pThnd:0 m_thndMX:bc0ea310, m_stopMX:bc0ea380 m_tintMX:bc0ea360 m_postMX:bc0ea340
2016-09-07 14:13:29.994 :CLSDYNAM:3487557376: [ ora.asm]{1:23503:371} [action] InstAgent constructor
2016-09-07 14:13:29.994 : USRTHRD:3487557376: {1:23503:371} Thread:ASM DedicatedThread thread constructor exit this:bc0ea510 m_pThnd:0 m_thndMX:bc0ea530, m_stopMX:bc0ea5a0 m_tintMX:bc0ea580 m_postMX:bc0ea560
2016-09-07 14:13:29.994 :CLSDYNAM:3487557376: [ ora.asm]{1:23503:371} [action] AsmAgent::AsmAgent Constructor resName ora.asm compId +ASM
2016-09-07 14:13:29.994 :CLSDYNAM:3487557376: [ ora.asm]{1:23503:371} [action] AsmAgent::refresh
2016-09-07 14:13:29.994 :CLSDYNAM:3487557376: [ ora.asm]{1:23503:371} [action] AsmAgent::refresh ORACLE_HOME = /scratch/oracle/crshome
2016-09-07 14:13:29.994 :CLSDYNAM:3487557376: [ ora.asm]{1:23503:371} [action] AsmAgent:getOracleSidAttrib 3 getResAttrib GEN_USR_ORA_INST_NAME oracle_sid:+ASM1
2016-09-07 14:13:29.994 :CLSDYNAM:3487557376: [ ora.asm]{1:23503:371} [action] AsmAgent:getOracleSidAttrib oracleSid:+ASM1
2016-09-07 14:13:29.994 :CLSDYNAM:3487557376: [ ora.asm]{1:23503:371} [action] AsmAgent::refresh ORACLE_SID = +ASM1
2016-09-07 14:13:29.994 :CLSDYNAM:3487557376: [ ora.asm]{1:23503:371} [action] InstAgent::init oracleSid +ASM1
2016-09-07 14:13:29.994 :CLSDYNAM:3487557376: [ ora.asm]{1:23503:371} [action] InstAgent::init resVersion 12.1.0.1.0
2016-09-07 14:13:29.994 :CLSDYNAM:3487557376: [ ora.asm]{1:23503:371} [action] ConnectionPool::ConnectionPool 2 constructor this:bc119020 m_oracleHome:/scratch/oracle/crshome, m_oracleSid:+ASM1, m_usrOraEnv: m_instanceType:2 m_instanceVersion:12.1.0.1.0
2016-09-07 14:13:29.994 :CLSDYNAM:3487557376: [ ora.asm]{1:23503:371} [action] ConnectionPool::ConnectionPool 2 constructor m_pResState:0x7f01bc119120
2016-09-07 14:13:29.994 : USRTHRD:3487557376: {1:23503:371} Thread:CssOsmGroupThread thread constructor exit this:bc0ea778 m_pThnd:0 m_thndMX:bc0ea798, m_stopMX:bc0ea808 m_tintMX:bc0ea7e8 m_postMX:bc0ea7c8
2016-09-07 14:13:29.994 :CLSDYNAM:3487557376: [ ora.asm]{1:23503:371} [action] AsmOhasdAgent constructor{
2016-09-07 14:13:29.994 :CLSDYNAM:3487557376: [ ora.asm]{1:23503:371} [action] AsmOhasdAgent constructor}
2016-09-07 14:13:29.994 :CLSDYNAM:3487557376: [ ora.asm]{1:23503:371} [action] Agent::setResType setResType type:ora.asm.type
2016-09-07 14:13:29.994 :CLSDYNAM:3487557376: [ ora.asm]{1:23503:371} [action] AsmOhasdAgent Action {
2016-09-07 14:13:29.994 :CLSDYNAM:3487557376: [ ora.asm]{1:23503:371} [action] InstAgent::check entry { m_connectionLock:0x7f01bc0ea4f0
2016-09-07 14:13:29.994 :CLSDYNAM:3487557376: [ ora.asm]{1:23503:371} [action] InstAgent::check m_comment:
2016-09-07 14:13:29.994 :CLSDYNAM:3487557376: [ ora.asm]{1:23503:371} [action] InstAgent::check call GEN_RESTART
2016-09-07 14:13:29.995 :CLSDYNAM:3487557376: [ ora.asm]{1:23503:371} [action] InstAgent::getGenRestart entry logLevel:1
2016-09-07 14:13:29.995 :CLSDYNAM:3487557376: [ ora.asm]{1:23503:371} [action] InstAgent::getGenRestart getting attributeName:GEN_RESTART
2016-09-07 14:13:29.995 :CLSDYNAM:3487557376: [ ora.asm]{1:23503:371} [action] InstAgent::getGenRestart exit GEN_RESTART:StartOption[1] }
2016-09-07 14:13:29.995 :CLSDYNAM:3487557376: [ ora.asm]{1:23503:371} [action] InstAgent::check GEN_RESTART:StartOption[1]
2016-09-07 14:13:29.995 :CLSDYNAM:3487557376: [ ora.asm]{1:23503:371} [action] InstAgent::checkstate: 000 entry s_SvcDbXLock:0x138ed28
2016-09-07 14:13:29.995 :CLSDYNAM:3487557376: [ ora.asm]{1:23503:371} [action] InstAgent::checkState 010 type:ffff reason THA:0 oracleSid:+ASM1
2016-09-07 14:13:29.995 :CLSDYNAM:3487557376: [ ora.asm]{1:23503:371} [action] InstAgent::checkState 030 new gimh oracleSid:+ASM1
2016-09-07 14:13:29.995 :CLSDYNAM:3487557376: [ ora.asm]{1:23503:371} [action] Gimh::constructor ohome:/scratch/oracle/crshome sid:+ASM1
2016-09-07 14:13:29.995 :CLSDYNAM:3487557376: [ ora.asm]{1:23503:371} [action] InstAgent::checkState 040 getStatusOfConnection
2016-09-07 14:13:29.995 :CLSDYNAM:3487557376: [ ora.asm]{1:23503:371} [action] InstAgent::checkState 050 m_prevState 3 poolState 0
2016-09-07 14:13:29.995 :CLSDYNAM:3487557376: [ ora.asm]{1:23503:371} [action] InstAgent::checkState 060 call checkCbk
2016-09-07 14:13:29.995 :CLSDYNAM:3487557376: [ ora.asm]{1:23503:371} [action] Gimh::check condition changes to (GIMH_NEXT_NUM) 0(Abnormal Termination) exists
2016-09-07 14:13:29.995 :CLSDYNAM:3487557376: [ ora.asm]{1:23503:371} [action] CLS_DRUID_REF(CLSN00006) AsmAgent::gimhChecks 100 failed gimh state 0
crsd_oraagent_grid.trc:
2016-09-07 14:13:29.943 : AGFW:1368995584: {1:23501:371} Agent received the message: RESOURCE_START[ora.asm 2 1] ID 4098:1416
2016-09-07 14:13:29.943 : AGFW:1368995584: {1:23501:371} Preparing START command for: ora.asm 2 1
2016-09-07 14:13:29.943 : AGFW:1368995584: {1:23501:371} ora.asm 2 1 state changed from: PLANNED_OFFLINE to: STARTING
2016-09-07 14:13:29.945 :CLSDYNAM:1361217280: [ ora.asm]{1:23501:371} [start] (:CLSN00107:) clsn_agent::start {
2016-09-07 14:13:29.945 :CLSDYNAM:1361217280: [ ora.asm]{1:23501:371} [start] AsmClusterAgent::start {
2016-09-07 14:13:29.945 :CLSDYNAM:1361217280: [ ora.asm]{1:23501:371} [start] AsmProxyAgent:start entry {
2016-09-07 14:13:29.945 : CLSCEVT:1361217280: (:CLSCE0028:)clsce_unsubscribe 0x7f7f40080430 successfully unsubscribed : 9
2016-09-07 14:13:29.945 : USRTHRD:1361217280: {1:23501:371} WorkerThread::removeWorker total workers: 4
2016-09-07 14:13:29.945 : USRTHRD:1361217280: {1:23501:371} Worker::schedule adding worker 'ReconnSub-AsmAgentSub+ASM' to thread 'ReconnectingSubscribers'
2016-09-07 14:13:29.946 : USRTHRD:1361217280: {1:23501:371} WorkerThread::addWorker total workers: 5
2016-09-07 14:13:29.946 : CLSCEVT:1365419776: clsce_subscribe 0x7f7f1846df00 filter='(^CRS_RESOURCE_PROFILE_CHANGE.*RESOURCE_CLASS='(scan_listener|listener|scan_vip|vip)')|(^CRS_RESOURCE_PROFILE_CHANGE.*TYPE='ora\.network\.type')|(^CRS_RESOURCE_STATE_CHANGE.*RESOURCE_CLASS='gns')', flags=1, handler=0xc69a38, arg=0x7f7f180bd040
2016-09-07 14:13:29.946 : USRTHRD:1361217280: {1:23501:371} Thread:ASM DedicatedThread stop { m_pThnd:(nil)
2016-09-07 14:13:29.946 : USRTHRD:1361217280: {1:23501:371} Thread:ASM DedicatedThread Thread stop:thread acquire m_pThnd:(nil) m_thndMX:0x7f7f18312a90
2016-09-07 14:13:29.946 : USRTHRD:1361217280: {1:23501:371} Thread:ASM DedicatedThread stop release m_thndMX:0x7f7f18312a90 }
2016-09-07 14:13:29.946 :CLSDYNAM:1361217280: [ ora.asm]{1:23501:371} [start] ProxyAgent::clearEnv
2016-09-07 14:13:29.946 :CLSDYNAM:1361217280: [ ora.asm]{1:23501:371} [start] ProxyAgent::setEnv
2016-09-07 14:13:29.946 :CLSDYNAM:1361217280: [ ora.asm]{1:23501:371} [start] Utils::getResAttrib entry attribName:USR_ORA_OPI required:0 loglevel:1
2016-09-07 14:13:29.946 :CLSDYNAM:1361217280: [ ora.asm]{1:23501:371} [start] Utils::getResAttrib: attribname USR_ORA_OPI value false len 5
2016-09-07 14:13:29.946 :CLSDYNAM:1361217280: [ ora.asm]{1:23501:371} [start] Utils::getResAttrib attribname:USR_ORA_OPI value:false exit
2016-09-07 14:13:29.946 :CLSDYNAM:1361217280: [ ora.asm]{1:23501:371} [start] Agent::valueOfAttribIs attrib: REASON compare value: user attribute value: user
2016-09-07 14:13:29.946 :CLSDYNAM:1361217280: [ ora.asm]{1:23501:371} [start] Agent::valueOfAttribIs returns 1
2016-09-07 14:13:29.946 :CLSDYNAM:1361217280: [ ora.asm]{1:23501:371} [start] proxyAgent start with flag 64
2016-09-07 14:13:29.949 : CLSCEVT:1365419776: (:CLSCE0022:)clsce_subscribe 0x7f7f1846df00 successfully subscribed : 12
2016-09-07 14:13:29.969 :CLSDYNAM:1361217280: [ ora.asm]{1:23501:371} [start] clsnUtils::cmdIdIsStart CmdId:257
2016-09-07 14:13:29.969 :CLSDYNAM:1361217280: [ ora.asm]{1:23501:371} [start] AsmProxyAgent::setGenUsrOraInst updating GEN_USR_ORA_INST_NAME@SERVERNAME(node6) to +ASM1
2016-09-07 14:13:29.969 :CLSDYNAM:1361217280: [ ora.asm]{1:23501:371} [start] clsnUtils::setResAttrib nonPerX current value GEN_USR_ORA_INST_NAME value +ASM1
2016-09-07 14:13:29.974 :CLSDYNAM:1361217280: [ ora.asm]{1:23501:371} [start] setResAttrib: attr GEN_USR_ORA_INST_NAME@SERVERNAME(node6) clsagfw attribVal +ASM1 clscrs tmpAttrValue +ASM1
2016-09-07 14:13:29.974 :CLSDYNAM:1361217280: [ ora.asm]{1:23501:371} [start] Utils::setResAttrib clsagfw_modify_attribute attr GEN_USR_ORA_INST_NAME@SERVERNAME(node6) value +ASM1 retCode 0
2016-09-07 14:13:29.974 :CLSDYNAM:1361217280: [ ora.asm]{1:23501:371} [start] ConnectionPool::stopConnection
2016-09-07 14:13:29.974 :CLSDYNAM:1361217280: [ ora.asm]{1:23501:371} [start] ConnectionPool::broadcastEvent 000 entry { OHSid:/scratch/oracle/crshome+ASM1 s_ohSidEventMapLock:0x139d830 action:2
2016-09-07 14:13:29.974 :CLSDYNAM:1361217280: [ ora.asm]{1:23501:371} [start] ConnectionPool::removeConnection connection count 0
2016-09-07 14:13:29.974 :CLSDYNAM:1361217280: [ ora.asm]{1:23501:371} [start] ConnectionPool::removeConnection freed 0
2016-09-07 14:13:29.974 :CLSDYNAM:1361217280: [ ora.asm]{1:23501:371} [start] ConnectionPool::stopConnection sid +ASM1 status 1
2016-09-07 14:13:29.974 : USRTHRD:1361217280: {1:23501:371} ConnectionPool::~ConnectionPool destructor this:400822a0 m_oracleHome:/scratch/oracle/crshome, m_oracleSid:+ASM1, m_usrOraEnv: m_pResState:0x7f7f401cdbe0
2016-09-07 14:13:29.974 :CLSDYNAM:1361217280: [ ora.asm]{1:23501:371} [start] ConnectionPool::ConnectionPool 2 constructor this:18321160 m_oracleHome:/scratch/oracle/crshome, m_oracleSid:+ASM1, m_usrOraEnv: m_instanceType:2 m_instanceVersion:12.2.0.2.0
2016-09-07 14:13:29.974 :CLSDYNAM:1361217280: [ ora.asm]{1:23501:371} [start] ConnectionPool::ConnectionPool 2 constructor m_pResState:0x7f7f180bd490
2016-09-07 14:13:29.974 :CLSDYNAM:1361217280: [ ora.asm]{1:23501:371} [start] AsmProxyAgent:start new ConnectionPool 18321160
2016-09-07 14:13:29.974 :CLSDYNAM:1361217280: [ ora.asm]{1:23501:371} [start] ConnectionPool::startConnection s_statusOfConnectionMap:0x139d920
2016-09-07 14:13:29.974 :CLSDYNAM:1361217280: [ ora.asm]{1:23501:371} [start] ConnectionPool::startConnection sid +ASM1 status 2
2016-09-07 14:13:29.974 :CLSDYNAM:1361217280: [ ora.asm]{1:23501:371} [start] AsmProxyAgent:start exit }
2016-09-07 14:13:29.974 :CLSDYNAM:1361217280: [ ora.asm]{1:23501:371} [start] AsmClusterAgent::start ProxyAgent::action
2016-09-07 14:13:30.157 : AGFW:1368995584: {1:23505:371} Agent received the message: RESOURCE_PROBE[ora.asm 2 1] ID 4097:1435
2016-09-07 14:13:30.157 : AGFW:1368995584: {1:23505:371} Preparing CHECK command for: ora.asm 2 1
2016-09-07 14:13:34.381 :CLSDYNAM:1361217280: [ ora.asm]{1:23501:371} [start] ProxyAgent::ProxyAgentClscrsCB Propogate Error
2016-09-07 14:13:34.381 :CLSDYNAM:1361217280: [ ora.asm]{1:23501:371} [start] clsnUtils::error Exception type=2 string=
CRS-5804: Communication error with agent process
2016-09-07 14:13:34.382 : AGFW:1368995584: {1:23501:371} Agent sending reply for: RESOURCE_START[ora.asm 2 1] ID 4098:1416
2016-09-07 14:13:34.383 :CLSDYNAM:1361217280: [ ora.asm]{1:23501:371} [start] checkCrsStat 2 CLSCRS_STAT ret: 200
2016-09-07 14:13:34.383 :CLSDYNAM:1361217280: [ ora.asm]{1:23501:371} [start] clsnUtils::error Exception type=2 string=
2016-09-07 14:13:34.383 :CLSDYNAM:1361217280: [ ora.asm]{1:23501:371} [start] checkCrsStat 2 clscrs_res_get_op_status CLSCRS_STAT status 200 err_msg CRS-0200: CRS general failure. No reason is provided.
crsd.trc:
2016-09-07 14:13:29.936 :UiServer:2157872896: {1:23501:371} Container [ Name: UI_START
API_HDR_VER:
TextMessage[3]
CLIENT:
TextMessage[]
CLIENT_NAME:
TextMessage[/usr/bin/java]
CLIENT_PID:
TextMessage[12841]
CLIENT_PRIMARY_GROUP:
TextMessage[oinstall]
FORCE_TAG:
TextMessage[1]
LOCALE:
TextMessage[AMERICAN_AMERICA.US7ASCII]
QUEUE_TAG:
TextMessage[1]
RESOURCE:
TextMessage[ora.asm]
]
2016-09-07 14:13:29.936 :UiServer:2157872896: {1:23501:371} Sending to PE. ctx= 0x7fede407eb00, ClientPID=12841 set Properties (grid,38257)
2016-09-07 14:13:29.942 : AGFW:2174682880: {1:23501:371} Agfw Proxy Server received the message: RESOURCE_START[ora.asm 2 1] ID 4098:4817
2016-09-07 14:13:29.942 :UiServer:2157872896: {1:23501:371} Response: c4|5!ORDERk7|MESSAGEt52|CRS-2672: Attempting to start 'ora.asm' on 'node6'k7|MSGTYPEt1|3k5|OBJIDt11|ora.asm 2 1k4|WAITt1|0
2016-09-07 14:13:29.942 : AGFW:2174682880: {1:23501:371} Agfw Proxy Server forwarding the message: RESOURCE_START[ora.asm 2 1] ID 4098:4817 to the agent /scratch/oracle/crshome/bin/oraagent_grid
2016-09-07 14:13:29.970 :UiServer:2157872896: {1:23503:371} Sending to PE. ctx= 0x7fede407fe80, ClientPID=28072 set Properties (grid,21858), tint: {1:23501:371}
2016-09-07 14:13:29.973 :UiServer:2157872896: {1:23503:371} Done for ctx=0x7fede407fe80
2016-09-07 14:13:30.029 :UiServer:2157872896: {1:23501:374} Container [ Name: UI_STOP
API_HDR_VER:
TextMessage[3]
ASYNC_TAG:
TextMessage[1]
CLIENT:
TextMessage[]
CLIENT_NAME:
TextMessage[/scratch/oracle/crshome/bin/oraagent.bin]
CLIENT_PID:
TextMessage[498]
CLIENT_PRIMARY_GROUP:
TextMessage[oinstall]
FILTER:
TextMessage[(((((STATE==ONLINE)||(STATE==INTERMEDIATE))||(STATE==UNKNOWN))&&((NAME==ora.asm)&&(LAST_SERVER==node6)))&&(STATE_DETAILScoUnder THA))]
FILTER_TAG:
TextMessage[1]
LOCALE:
TextMessage[AMERICAN_AMERICA.AL32UTF8]
TINT:
TextMessage[{1:23503:371}]
T_HA_FINISH_TAG:
TextMessage[1]
]
2016-09-07 14:13:30.029 :UiServer:2157872896: {1:23505:371} Sending to PE. ctx= 0x7fede407ff40, ClientPID=498 set Properties (grid,38386), tint: {1:23503:371}
2016-09-07 14:13:30.081 :UiServer:2157872896: {1:23505:371} Sending to PE. ctx= 0x7fede40809a0, ClientPID=498 set Properties (grid,38707), tint: {1:23503:371}
2016-09-07 14:13:30.085 :UiServer:2157872896: {1:23505:371} Done for ctx=0x7fede40809a0
2016-09-07 14:13:30.133 :UiServer:2157872896: {1:23505:371} Sending to PE. ctx= 0x7fede4080a40, ClientPID=498 set Properties (grid,38782), tint: {1:23503:371}
2016-09-07 14:13:30.136 :UiServer:2157872896: {1:23505:371} Done for ctx=0x7fede4080a40
2016-09-07 14:13:30.153 :UiServer:2157872896: {1:23501:379} Container [ Name: UI_CHECK
API_HDR_VER:
TextMessage[3]
CLIENT:
TextMessage[]
CLIENT_NAME:
TextMessage[/scratch/oracle/crshome/bin/oraagent.bin]
CLIENT_PID:
TextMessage[498]
DELAY_SECS:
TextMessage[0]
HOST:
TextMessage[rwsbf06]
HOST_TAG:
TextMessage[1]
LOCALE:
TextMessage[AMERICAN_AMERICA.AL32UTF8]
RESOURCE:
TextMessage[ora.asm]
TINT:
TextMessage[{1:23503:371}]
]
2016-09-07 14:13:30.153 :UiServer:2157872896: {1:23505:371} Sending to PE. ctx= 0x7fede4080710, ClientPID=498 set Properties (grid,38707), tint: {1:23503:371}
2016-09-07 14:13:30.156 : AGFW:2174682880: {1:23505:371} Agfw Proxy Server received the message: RESOURCE_PROBE[ora.asm 2 1] ID 4097:4823
2016-09-07 14:13:30.156 : AGFW:2174682880: {1:23505:371} Agfw Proxy Server forwarding the message: RESOURCE_PROBE[ora.asm 2 1] ID 4097:4823 to the agent /scratch/oracle/crshome/bin/oraagent_grid
2016-09-07 14:13:30.156 :UiServer:2157872896: {1:23505:371} Response: c1|7!UI_DATAk7|ora.asmt1|0
2016-09-07 14:13:30.156 :UiServer:2157872896: {1:23505:371} Done for ctx=0x7fede4080710
2016-09-07 14:13:34.383 : AGFW:2174682880: {1:23501:371} Received the reply to the message: RESOURCE_START[ora.asm 2 1] ID 4098:1416 from the agent /scratch/oracle/crshome/bin/oraagent_grid
2016-09-07 14:13:34.383 : AGFW:2174682880: {1:23501:371} Agfw Proxy Server sending the reply to PE for message:RESOURCE_START[ora.asm 2 1] ID 4098:4817
2016-09-07 14:13:34.385 : AGFW:2174682880: {1:23501:371} Received the reply to the message: RESOURCE_START[ora.asm 2 1] ID 4098:1416 from the agent /scratch/oracle/crshome/bin/oraagent_grid
2016-09-07 14:13:34.385 : AGFW:2174682880: {1:23501:371} Agfw Proxy Server sending the reply to PE for message:RESOURCE_START[ora.asm 2 1] ID 4098:4817
2016-09-07 14:13:34.386 :UiServer:2157872896: {1:23501:371} Response: c4|5!ORDERk7|MESSAGEt48|CRS-5804: Communication error with agent processk7|MSGTYPEt1|1k5|OBJIDt11|ora.asm 2 1k4|WAITt1|0
2016-09-07 14:13:34.409 : AGFW:2174682880: {1:23501:371} Received the reply to the message: RESOURCE_START[ora.asm 2 1] ID 4098:1416 from the agent /scratch/oracle/crshome/bin/oraagent_grid
2016-09-07 14:13:34.409 : AGFW:2174682880: {1:23501:371} Agfw Proxy Server sending the last reply to PE for message:RESOURCE_START[ora.asm 2 1] ID 4098:4817
2016-09-07 14:13:34.409 :UiServer:2157872896: {1:23503:371} Sending to PE. ctx= 0x7fede40808f0, ClientPID=28072 set Properties (grid,17423), tint: {1:23501:371}
2016-09-07 14:13:34.412 :UiServer:2157872896: {1:23503:371} Done for ctx=0x7fede40808f0
2016-09-07 14:13:34.412 :UiServer:2157872896: {1:23501:371} Response: c4|5!ORDERk7|MESSAGEt48|CRS-2674: Start of 'ora.asm' on 'node6' failedk7|MSGTYPEt1|1k5|OBJIDt11|ora.asm 2 1k4|WAITt1|0
2016-09-07 14:13:34.413 : AGFW:2174682880: {1:23501:371} Agfw Proxy Server received the message: RESOURCE_CLEAN[ora.asm 2 1] ID 4100:4854
2016-09-07 14:13:34.413 : AGFW:2174682880: {1:23501:371} Agfw Proxy Server forwarding the message: RESOURCE_CLEAN[ora.asm 2 1] ID 4100:4854 to the agent /scratch/oracle/crshome/bin/oraagent_grid
2016-09-07 14:13:34.413 :UiServer:2157872896: {1:23501:371} Response: c4|5!ORDERk7|MESSAGEt52|CRS-2679: Attempting to clean 'ora.asm' on 'node6'k7|MSGTYPEt1|3k5|OBJIDt11|ora.asm 2 1k4|WAITt1|0
2016-09-07 14:13:34.428 : AGFW:2174682880: {0:7:9} Agfw Proxy Server received the message: RESOURCE_STATUS[Proxy] ID 20481:475
2016-09-07 14:13:34.428 : AGFW:2174682880: {0:7:9} Verifying msg rid = ora.asm 2 1
2016-09-07 14:13:34.428 : AGFW:2174682880: {0:7:9} Agfw Proxy Server replying to the message: RESOURCE_STATUS[Proxy] ID 20481:475
2016-09-07 14:13:34.428 : AGFW:2174682880: {1:23505:371} Received the reply to the message: RESOURCE_PROBE[ora.asm 2 1] ID 4097:1435 from the agent /scratch/oracle/crshome/bin/oraagent_grid
2016-09-07 14:13:34.428 : AGFW:2174682880: {1:23505:371} ora.asm 2 1 received state from probe request. Old state = OFFLINE, New state = OFFLINE
2016-09-07 14:13:34.429 : AGFW:2174682880: {1:23505:371} Agfw Proxy Server sending the last reply to PE for message:RESOURCE_PROBE[ora.asm 2 1] ID 4097:4823
2016-09-07 14:13:34.450 :UiServer:2157872896: {1:23503:371} Sending to PE. ctx= 0x7fede40808f0, ClientPID=28072 set Properties (grid,17423), tint: {1:23501:371}
2016-09-07 14:13:34.453 :UiServer:2157872896: {1:23503:371} Done for ctx=0x7fede40808f0
Stack trace:
gdb /scratch/app/12.2.0/grid/bin/oraagent.bin /scratch/app/oracle/crsdata/rws1270138/core/core.oraagent.bin.13783
warning: no loadable sections found in added symbol-file system-supplied DSO at 0x7fffba106000
Core was generated by `/scratch/app/12.2.0/grid/bin/oraagent.bin'. Program terminated with signal 11, Segmentation fault.
#0 0x00007f2db06fda36 in SlfFopen () from
/scratch/app/12.2.0/grid/lib/libclntshcore.so.12.1 Missing separate debuginfos, use: debuginfo-install
glibc-2.12-1.166.el6_7.7.x86_64 libaio-0.3.107-10.el6.x86_64
libattr-2.4.44-7.el6.x86_64 libcap-2.16-5.5.el6.x86_64
libgcc-4.4.7-4.el6.x86_64 libstdc++-4.4.7-4.el6.x86_64
numactl-devel-2.0.7-8.el6.x86_64 (gdb) bt
....
#4 0x00007f2db2444a4e in sskgmsmr_op () from /scratch/app/12.2.0/grid/lib/libclntsh.so.12.1
#5 0x0000000000d61513 in gimh_init_inst_ctx ()
#6 0x00000000005354df in clsn_agent::Gimh::check (this=0x7f2d986c8080,
actx=0x7f2d88085e80, gimh_states=0x7f2dab485172) at clsnGimh.cpp:159
#7 0x000000000048a72e in clsn_agent::AsmAgent::gimhChecks (this=0x7f2d840c9610, actx=0x7f2d88085e80, pGimh=0x7f2d986c8080, druid=
"", agfwStates=..., stateStrs=..., instanceRestarted=0x0) at clsnAsmAgent.cpp:511
#8 0x000000000048b6fc in clsn_agent::AsmAgent::checkCbk
(this=0x7f2d840c9610, actx=0x7f2d88085e80, pGimh=0x7f2d986c8080, druid="") at clsnAsmAgent.cpp:782
#9 0x00000000005c88fc in clsn_agent::InstAgent::checkState
(this=0x7f2d840c9610, actx=0x7f2d88085e80) at clsnInstAgent.cpp:1617
#10 0x00000000005bfa2a in clsn_agent::InstAgent::check (this=0x7f2d840c9610,
actx=0x7f2d88085e80) at clsnInstAgent.cpp:1449
#11 0x00000000007324b4 in clsn_agent::AsmOhasdAgent::stop
(this=0x7f2d840c9610, actx=0x7f2d88085e80) at clsnAsmOhasdAgent.cpp:197
#12 0x00000000005bfd83 in clsn_agent::InstAgent::check (this=0x7f2d840c9610,
actx=0x7f2d88085e80) at clsnInstAgent.cpp:1473
#13 0x00000000007324b4 in clsn_agent::AsmOhasdAgent::stop
(this=0x7f2d840c9610, actx=0x7f2d88085e80) at clsnAsmOhasdAgent.cpp:197
Changes
Cause
To view full details, sign in with your My Oracle Support account. |
|
Don't have a My Oracle Support account? Click to get started! |
In this Document
| Symptoms |
| Changes |
| Cause |
| Solution |
| References |