CRSCTL Status Showing Node Listener Offline as "lsnrctl" Executable is 0-byte in Size

(Doc ID 1908907.1)

Last updated on AUGUST 24, 2017

Applies to:

Oracle Database - Enterprise Edition - Version 11.2.0.3 and later
Information in this document applies to any platform.

Symptoms

 - crsctl  status resource -t shows node listener is status offline

ora.LISTENER.lsnr
               ONLINE  OFFLINE      Node01                                      
               ONLINE  ONLINE       Node02

 - ps -ef |grep tnslsnr showing listener process is running

Node01 /home/grid> ps -ef |grep tnslsnr
grid      2588  2062  0 16:15 pts/0    00:00:00 grep tnslsnr
grid     10173     1  0 May29 ?        08:02:20 /u01/app/11.2.0/grid/bin/tnslsnr LISTENER -inherit

 - And the listener log shows listener is accepting connections

17-JUL-2014 12:37:28 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=__jdbc__)(USER=))(SERVICE_NAME=orcl)(CID=(PROGRAM=)(HOST=__jdbc__)(USER=))) * (ADDRESS=(PROTOCOL=tcp)(HOST=XX.XXX.XX.XXX)(PORT=59225)) * establish * orcl * 0
17-JUL-2014 12:37:29 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=__jdbc__)(USER=))(SERVICE_NAME=orcl)(CID=(PROGRAM=)(HOST=__jdbc__)(USER=))) * (ADDRESS=(PROTOCOL=tcp)(HOST=XX.XXX.XX.XXX)(PORT=46862)) * establish * orcl * 0
17-JUL-2014 12:37:30 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=__jdbc__)(USER=))(SERVICE_NAME=orcl)) * (ADDRESS=(PROTOCOL=tcp)(HOST=XX.XXX.X.XXX)(PORT=52007)) * establish * orcl * 0
17-JUL-2014 12:37:30 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=__jdbc__)(USER=))(SERVICE_NAME=orcl)) * (ADDRESS=(PROTOCOL=tcp)(HOST=XX.XXX.X.XXX)(PORT=52008)) * establish * orcl * 0

- CRS alert log reports listener check failed

2014-07-16 21:23:21.372
[/u01/app/11.2.0/grid/bin/oraagent.bin(10049)]CRS-5016:Process "/u01/app/11.2.0/grid/bin/lsnrctl" spawned by agent "/u01/app/11.2.0/grid/bin/oraagent.bin" for action "start" failed: details at "(:CLSN00010:)" in "/u01/app/11.2.0/grid/log/Node01/agent/crsd/oraagent_grid/oraagent_grid.log"
2014-07-16 21:23:21.587
[/u01/app/11.2.0/grid/bin/oraagent.bin(10049)]CRS-5016:Process "/u01/app/11.2.0/grid/bin/lsnrctl" spawned by agent "/u01/app/11.2.0/grid/bin/oraagent.bin" for action "start" failed: details at "(:CLSN00010:)" in "/u01/app/11.2.0/grid/log/Node101/agent/crsd/oraagent_grid/oraagent_grid.log"
2014-07-16 21:23:21.692
[/u01/app/11.2.0/grid/bin/oraagent.bin(10049)]CRS-5016:Process "/u01/app/11.2.0/grid/bin/lsnrctl" spawned by agent "/u01/app/11.2.0/grid/bin/oraagent.bin" for action "check" failed: details at "(:CLSN00010:)" in "/u01/app/11.2.0/grid/log/Node01/agent/crsd/oraagent_grid/oraagent_grid.log"
2014-07-16 21:25:09.262
[/u01/app/11.2.0/grid/bin/oraagent.bin(10049)]CRS-5016:Process "/u01/app/11.2.0/grid/bin/lsnrctl" spawned by agent "/u01/app/11.2.0/grid/bin/oraagent.bin" for action "start" failed: details at "(:CLSN00010:)" in "/u01/app/11.2.0/grid/log/Node01/agent/crsd/oraagent_grid/oraagent_grid.log"
2014-07-16 21:25:09.465
[/u01/app/11.2.0/grid/bin/oraagent.bin(10049)]CRS-5016:Process "/u01/app/11.2.0/grid/bin/lsnrctl" spawned by agent "/u01/app/11.2.0/grid/bin/oraagent.bin" for action "start" failed: details at "(:CLSN00010:)" in "/u01/app/11.2.0/grid/log/Node01/agent/crsd/oraagent_grid/oraagent_grid.log"

agent/crsd/oraagent_grid/oraagent_grid.log file shows

2014-07-16 21:23:21.587: [ora.LISTENER.lsnr][939521792] {1:64277:64053} [check] LsnrAgent::check {
2014-07-16 21:23:21.587: [ora.LISTENER.lsnr][939521792] {1:64277:64053} [check] lsnrctl status LISTENER

2014-07-16 21:23:21.587: [ora.LISTENER.lsnr][939521792] {1:64277:64053} [check] getOracleHomeAttrib: oracle_home = /u01/app/11.2.0/grid
2014-07-16 21:23:21.588: [    AGFW][3422549760] {1:64277:64053} Agent sending reply for: RESOURCE_START[ora.LISTENER.lsnr node02 1] ID 4098:2270227
2014-07-16 21:23:21.588: [ora.LISTENER.lsnr][939521792] {1:64277:64053} [check] getOracleHomeAttrib: oracle_home = /u01/app/11.2.0/grid
2014-07-16 21:23:21.588: [ora.LISTENER.lsnr][939521792] {1:64277:64053} [check] Utils::getCrsHome crsHome /u01/app/11.2.0/grid
2014-07-16 21:23:21.588: [ora.LISTENER.lsnr][939521792] {1:64277:64053} [check] Utils::execCmd 1 USR_ORA_ENV:ORACLE_BASE=/u01/app/grid oracleHome:/u01/app/11.2.0/grid CrsHome:/u01/app/11.2.0/grid
2014-07-16 21:23:21.588: [ora.LISTENER.lsnr][939521792] {1:64277:64053} [check] Utils::getCrsHome crsHome /u01/app/11.2.0/grid
2014-07-16 21:23:21.588: [ora.LISTENER.lsnr][939521792] {1:64277:64053} [check] Adding Environment Variables ORACLE_HOME=/u01/app/11.2.0/grid
2014-07-16 21:23:21.588: [ora.LISTENER.lsnr][939521792] {1:64277:64053} [check] Adding Environment Variables TNS_ADMIN=/u01/app/11.2.0/grid/network/admin/
2014-07-16 21:23:21.588: [ora.LISTENER.lsnr][939521792] {1:64277:64053} [check] Adding Environment variable from USR_ORA_ENV ORACLE_BASE=/u01/app/grid
2014-07-16 21:23:21.588: [ora.LISTENER.lsnr][939521792] {1:64277:64053} [check] Utils:execCmd action = 3 flags = 38 ohome = (null) cmdname = lsnrctl.
2014-07-16 21:23:21.588: [ora.LISTENER.lsnr][939521792] {1:64277:64053} [check] getOracleHomeAttrib: oracle_home = /u01/app/11.2.0/grid
2014-07-16 21:23:21.692: [ora.LISTENER.lsnr][939521792] {1:64277:64053} [check] (:CLSN00010:)Utils:execCmd scls_process_join() uret 209

2014-07-16 21:23:21.692: [ora.LISTENER.lsnr][939521792] {1:64277:64053} [check] execCmd ret = 209
2014-07-16 21:23:21.692: [ora.LISTENER.lsnr][939521792] {1:64277:64053} [check] (:CLSN00010:)
2014-07-16 21:23:21.692: [ora.LISTENER.lsnr][939521792] {1:64277:64053} [check] return val (clsagfwret) = 1
2014-07-16 21:23:21.692: [ora.LISTENER.lsnr][939521792] {1:64277:64053} [check] LsnrAgent::check }

 
- Manually running Listener control (lsnrctl) to check the node listener process (tnslsnr) from OS prompt, it returns to the prompt with no errors reported. This is for all commands, ie lsnrctl status, lsnrctl services, etc.

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms