Command "crsctl stat res -t" hangs on 11.2.0.4 RDBMS in GI 12.1.0.2

(Doc ID 2242873.1)

Last updated on MAY 11, 2017

Applies to:

Oracle Database - Enterprise Edition - Version 11.2.0.3 to 11.2.0.4 [Release 11.2]
Information in this document applies to any platform.

Symptoms

When running "crsctl stat res -t" command from the GI 12.1.0.2 home, not getting any output and hangs forever.

Observed below in the "crsd.trc" file

crsd.trc

2017-01-31 12:40:54.314516 :UiServer:239011584: {1:60389:2638} Sending to PE. ctx= 0x7f4ccc736600, ClientPID=17792, tint: {1:60387:2638}
2017-01-31 12:41:19.029847 :GIPCHTHR:1530824448: gipchaWorkerWork: workerThread heart beat, time interval since last heartBeat 30030loopCount 286
2017-01-31 12:41:20.031105 :GIPCHTHR:1528723200: gipchaDaemonWork: DaemonThread heart beat, time interval since last heartBeat 30030loopCount 38
2017-01-31 12:41:37.420834 :UiServer:239011584: {1:60389:2} Sending to PE. ctx= 0x7f4ccc736f10, ClientPID=17792, tint: {1:60387:2}
2017-01-31 12:41:37.427313 :UiServer:239011584: {1:60389:2} Done for ctx=0x7f4ccc736f10
2017-01-31 12:41:49.063817 :GIPCHTHR:1530824448: gipchaWorkerWork: workerThread heart beat, time interval since last heartBeat 30030loopCount 90
2017-01-31 12:41:50.065186 :GIPCHTHR:1528723200: gipchaDaemonWork: DaemonThread heart beat, time interval since last heartBeat 30030loopCount 41
2017-01-31 12:41:54.303103 :UiServer:239011584: {1:60389:2638} Sending to PE. ctx= 0x7f4ccc736ea0, ClientPID=17792, tint: {1:60387:2638}
2017-01-31 12:41:54.307346 :UiServer:239011584: {1:60389:2638} Done for ctx=0x7f4ccc736ea0
2017-01-31 12:41:54.323734 :UiServer:239011584: {1:60387:21883} Container [ Name: UI_REGISTER
....

2017-01-31 12:41:54.323870 :UiServer:239011584: {1:60389:2638} Sending to PE. ctx= 0x7f4ccc736ce0, ClientPID=17792, tint: {1:60387:2638}.
2017-01-31 12:42:19.096516 :GIPCHTHR:1530824448: gipchaWorkerWork: workerThread heart beat, time interval since last heartBeat 30030loopCount 292
2017-01-31 12:42:20.098903 :GIPCHTHR:1528723200: gipchaDaemonWork: DaemonThread heart beat, time interval since last heartBeat 30030loopCount 40
>>>>
2017-01-31 12:43:19.158982 :GIPCHTHR:1530824448: gipchaWorkerWork: workerThread heart beat, time interval since last heartBeat 30030loopCount 268
2017-01-31 12:43:20.161826 :GIPCHTHR:1528723200: gipchaDaemonWork: DaemonThread heart beat, time interval since last heartBeat 30030loopCount 41
>>>>
2017-01-31 12:45:19.295579 :GIPCHTHR:1530824448: gipchaWorkerWork: workerThread heart beat, time interval since last heartBeat 30040loopCount 56
2017-01-31 12:45:20.297935 :GIPCHTHR:1528723200: gipchaDaemonWork: DaemonThread heart beat, time interval since last heartBeat 30040loopCount 40

"pstack" output of crsd.bin process

Thread 1 (Thread 0x7f5e43b1e660 (LWP 28406)):
#0 0x000000391c4f80ce in __lll_lock_wait_private () from /lib64/libc.so.6                                
#1 0x000000391c47d349 in _L_lock_10151 () from /lib64/libc.so.6
#2 0x000000391c47ac33 in malloc () from /lib64/libc.so.6                                  > >>>>> trying to perform a malloc inside a signal handler
#3 0x00000039250bd0bd in operator new(unsigned long) () from /usr/lib64/libstdc++.so.6
#4 0x000000392509c3c9 in std::basic_string<char, std::char_traits<char>, std::allocator<char> >::_Rep::_S_create(unsigned long, unsigned long, std::allocator<char> const&) () from /usr/lib64/libstdc++.so.6
#5 0x000000392509d19b in std::basic_string<char, std::char_traits<char>, std::allocator<char> >::_Rep::_M_clone(std::allocator<char> const&, unsigned long) () from /usr/lib64/libstdc++.so.6
#6 0x000000392509d5ec in std::basic_string<char, std::char_traits<char>, std::allocator<char> >::reserve(unsigned long) () from /usr/lib64/libstdc++.so.6
#7 0x00000039250978f1 in std::basic_stringbuf<char, std::char_traits<char>, std::allocator<char> >::overflow(int) () from /usr/lib64/libstdc++.so.6
#8 0x0000003925092f30 in std::basic_ostream<char, std::char_traits<char> >::put(char) () from /usr/lib64/libstdc++.so.6
#9 0x000000000159552d in cls::TimerModule::Personality::handleMsgSubmit(cls::Message*) ()
#10 0x00000000015c32b3 in cls::ThreadModel::submitNewMsg(cls::Message*) ()
#11 0x0000000001633a4a in cls::M2M::processRequestMsg(cls::Message*, cls::ProcessAddress const&) ()
#12 0x000000000162fa7f in cls::M2M::processTasks() ()
#13 0x000000000163de36 in cls::M2M::doWork() ()
#14 0x000000000159e10b in cls::FrameworkManager::doWork() () 

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms