AIX: RAC Node Listener Crashes with * service_died * LsnrAgt * 12537 When SEOS is in Use

(Doc ID 2336010.1)

Last updated on DECEMBER 23, 2017

Applies to:

Oracle Database - Enterprise Edition - Version 11.2.0.4 and later
Information in this document applies to any platform.

Symptoms

LsnrAgt reporting "service_died" error message (listener.log inside <oracle_base>/diag/tnslsnr/<host name>/listener/trace):

02-OCT-2017 02:46:42 * (CONNECT_DATA=(SID=<instance_name>)(CID=(PROGRAM=)(HOST=__jdbc__)(USER=oracle))) * (ADDRESS=(PROTOCOL=tcp)(HOST=<ip_address>)(PORT=30682)) * establish * <instance_name> * 0
02-OCT-2017 02:46:42 * service_update * +ASM1 * 0
02-OCT-2017 02:46:42 * service_update * <instance_name> * 0
No longer listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=<ip_address>)(PORT=1527)))
02-OCT-2017 02:46:42 * service_died * LsnrAgt * 12537
02-OCT-2017 02:46:46 * <unknown connect data> * 12537
Mon Oct 02 02:46:47 2017
TNS-12537: TNS:connection closed
TNS-12560: TNS:protocol adapter error
TNS-00507: Connection closed
IBM/AIX RISC System/6000 Error: 32: Broken pipe
02-OCT-2017 02:46:46 * service_update * +ASM1 * 0

There is no issue with IP addresses and just restarting the listener gets all back again, there is no setup or configuration problem observed.

Along with listener and other resources went UNKNOWN status due to OS calls timing out during the nearby time.

Clusterware alert log shows (alert<hostname>.log inside <grid home>/log/<host name>): 

2017-10-02 02:44:45.899:
[<GI_HOME>/bin/oraagent.bin(13959352)]CRS-5818:Aborted command 'check' for resource 'ora.DATA.dg'. Details at (:CRSAGF00113:) {0:3:4} in <GI_HOME>/log/<node>/agent/crsd/oraagent_grid//oraagent_grid.log.
2017-10-02 02:46:45.915:
[<GI_HOME>/bin/oraagent.bin(4522380)]CRS-5016:Process "<GI_HOME>/bin/lsnrctl" spawned by agent "<GI_HOME>/bin/oraagent.bin" for action "start" failed: details at "(:CLSN00010:)" in "<GI_HOME>/log/<node>/agent/crsd/oraagent_grid//oraagent_grid.log"
2017-10-02 02:46:47.753:
[<GI_HOME>/bin/oraagent.bin(4522380)]CRS-5016:Process "<GI_HOME>/bin/lsnrctl" spawned by agent "<GI_HOME>/bin/oraagent.bin" for action "check" failed: details at "(:CLSN00010:)" in "<GI_HOME>/log/<node>/agent/crsd/oraagent_grid//oraagent_grid.log"
2017-10-02 02:46:47.758:
[crsd(12648640)]CRS-2765:Resource 'ora.LISTENER.lsnr' has failed on server '<node>'.

 

 

Changes

SeOS enabled at OS level. SeOS stands for Security for Open Systems. SeOS became eTrust Access Control, which then became CA Access Control. 

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms