hp-ux: Node Crash Due To Large Amount Of Racgimon Threads or CRS_STAT/SRVCTL COMMAND HANG OS bug ( QX:QXCR1000940361 ) (Doc ID 883801.1)

Last updated on OCTOBER 24, 2016

Applies to:

Oracle Server - Enterprise Edition - Version: 10.2.0.3 to 11.1.0.7 - Release: 10.2 to 11.1
HP-UX Itanium
HP-UX PA-RISC (64-bit)

Symptoms

syslog.16:Aug 16 16:55:46 s51uf33 vmunix: kthread: table is full
syslog.16:Aug 16 16:56:09 s51uf33 vmunix: kthread: table is full
syslog.16:Aug 16 16:58:46 s51uf33 vmunix: kkthread: table is full
syslog.16:Aug 16 16:58:46 s51uf33 vmunix: kthread: tablket hirse afdu:l lt
syslog.16:Aug 16 16:58:46 s51uf33 vmunix: kthread: table is full
syslog.16:Aug 16 16:58:46 s51uf33 vmunix: kthread: table is full
syslog.16:Aug 16 16:58:46 s51uf33 vmunix: kkthread: table is full
syslog.16:Aug 16 16:58:46 s51uf33 vmunix: kthread: tablekthread: table is full
syslog.16:Aug 16 16:58:47 s51uf33 vmunix: kthread:k ttharbelaed :i st afbulle is full
syslog.16:Aug 16 16:58:48 s51uf33 vmunix: kthread: table is fkthread: table is full
....


crsd.log

2009-11-11 10:48:58.085: [ CRSEVT][53546] CAAMonitorHandler :: 0:Could not join /oracle/product/10.2/bin/racgwrap(check)
category: 1234, operation: scls_process_join, loc: childcrash, OS error: 0, other: Abnormal termination of the child


There are a lot of ragimon threads.

# Threads -B ; pview | grep racgimon | wc -l

# Loading all inuse kthreads ...
Loaded 4740 kthread_t entries in 'DefaultView'
3360

--> 33360 racgimon threads running

The following commands hang:

srvctl status service -d orcl
crs_stat -p
crs_stat -u

Also pstack for crs_stat shows it is hanging:

$ pstack 27576
27576: /opt/crs/oracle/product/10.2.0/crs/bin/crs_stat.bin

-------------------------------- lwpid : 2554775 ----------------------------

0: c000000000439190 : _poll_sys() + 0x30 (/usr/lib/hpux64/libc.so.1)
1: c00000000044d7a0 : poll() + 0xe0 (/usr/lib/hpux64/libc.so.1)
2: c00000000972bad0 : sntrecvhdl() + 0x1b0
(/opt/crs/oracle/product/10.2.0/crs/lib/libttsh10.so)
3: c000000009726c80 : ntevque() + 0x260
(/opt/crs/oracle/product/10.2.0/crs/lib/libttsh10.so)
4: c000000009687bf0 : nsevwait() + 0x9e0
(/opt/crs/oracle/product/10.2.0/crs/lib/libttsh10.so)
5: c000000007347d80 : clsc_cvtimewait() + 0x1730
(/opt/crs/oracle/product/10.2.0/crs/lib/libhasgen10.so)
6: c0000000073406e0 : clsc_select_ext() + 0x2e0
(/opt/crs/oracle/product/10.2.0/crs/lib/libhasgen10.so)
7: c00000000733a4d0 : clscsendstatus() + 0x1ac0
(/opt/crs/oracle/product/10.2.0/crs/lib/libhasgen10.so)
8: c0000000073300b0 : clscreceive() + 0x1030
(/opt/crs/oracle/product/10.2.0/crs/lib/libhasgen10.so)
9: c000000007343a60 : clsc_event_hndlr() + 0x1f40
(/opt/crs/oracle/product/10.2.0/crs/lib/libhasgen10.so)
10: c000000007353a90 : clsaauthmsg() + 0x270
(/opt/crs/oracle/product/10.2.0/crs/lib/libhasgen10.so)
11: c000000007354d50 : clsavalidate() + 0x490
(/opt/crs/oracle/product/10.2.0/crs/lib/libhasgen10.so)
12: c000000007329510 : clscanswer() + 0x3e00
(/opt/crs/oracle/product/10.2.0/crs/lib/libhasgen10.so)
13: c00000000732be10 : clscconnect() + 0x18f0
(/opt/crs/oracle/product/10.2.0/crs/lib/libhasgen10.so)
14: c000000007188df0 : proac_init() + 0x710
(/opt/crs/oracle/product/10.2.0/crs/lib/libocr10.so)
15: c000000007195ef0 : proa_init() + 0x2dd0
(/opt/crs/oracle/product/10.2.0/crs/lib/libocr10.so)
16: c0000000071adb80 : procr_init_ext2() + 0x260
(/opt/crs/oracle/product/10.2.0/crs/lib/libocr10.so)
17: c0000000071ad880 : procr_init_ext() + 0x70
(/opt/crs/oracle/product/10.2.0/crs/lib/libocr10.so)
18: c000000007355760 : clse_init() + 0x120
(/opt/crs/oracle/product/10.2.0/crs/lib/libhasgen10.so)
Memory fault(coredump)

pstack for crsd.bin shows a sleep in pthread_cond_wait:

$ pstack 27576
(/opt/crs/oracle/prop)

  0: c000000000435e10 : __ksleep() + 0x30 (/usr/lib/hpux64/libc.so.1)
  1: c0000000001a7ca0 : __sleep_1x1() + 0x260
  2: c0000000001a43d0 : __mxn_sleep() + 0x2250
  3: c0000000000ddf30 : pthread_cond_wait() + 0x1230 (/usr/lib/hpux64
  4: c0000000000dcd70 : pthread_cond_wait() + 0x70 (/usr/lib/hpux64
  5: c00000000b91e7d0 : sltspcwait() + 0x50 (/opt/mvs/crs/lib/libttsh11.so)
  6: c0000000075fe190 : clsc_cvwait() + 0x40 (/opt/mvs/crs/lib/libhasgen11.so)
  7: c000000007604a80 : clsc_thrd_wait_work() + 0x560 (/opt/mvs/crs/lib
  8: c00000000762f6b0 : clsc_disc_msg() + 0x580 (/opt/mvs/crs/lib
  9: c0000000075f54f0 : clscidisc() + 0x870 (/opt/mvs/crs/lib/libhasgen11.so)
 10: c0000000075dd3b0 : clsc_disc() + 0xb20 (/opt/mvs/crs/lib/libhasgen11.so)
 11: c0000000075f3380 : clscdisc() + 0x360 (/opt/mvs/crs/lib/libhasgen11.so)
 12: 4000000000568350 : crs_qstat_process() + 0x5e0



Changes

Specific to HP-UX Itanium operating system.

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms