CLSRSC-180: 'srvctl stop listener -l LISTENER -n node -f' (error code null) in 12C GI upgrade due to wrong environment variable setting (Doc ID 1596816.1)

Last updated on NOVEMBER 20, 2013

Applies to:

Oracle Database - Enterprise Edition - Version 12.1.0.1 and later
Information in this document applies to any platform.

Symptoms

 Upgrading Grid Infrastructure with 3 nodes from 11.2.0.3 to 12.1.0.1; rootupgrade is successful on the first 2 nodes, but fails on the last node with below output:

rootcrs_<node>_<date>.log
-----------------
2013-10-31 12:54:16: Executing /apps/grid/product/12.1.0.1/grid/bin/srvctl upgrade model  -s 11.2.0.3.0 -d 12.1.0.1.0 -p last
2013-10-31 12:54:16: Executing cmd: /apps/grid/product/12.1.0.1/grid/bin/srvctl upgrade model  -s 11.2.0.3.0 -d 12.1.0.1.0 -p last
2013-10-31 12:54:24: srvctl upgrade model -last  ... passed
2013-10-31 12:54:24: Running as user grid: /apps/grid/product/12.1.0.1/grid/bin/srvctl add mgmtlsnr
2013-10-31 12:54:24: s_run_as_user2: Running /bin/su grid -c ' /apps/grid/product/12.1.0.1/grid/bin/srvctl add mgmtlsnr '
2013-10-31 12:54:58: The property LISTENER for checkpoint ROOTCRS_NODECONFIG does not exist
2013-10-31 12:54:58: leftVersion=11.2.0.3.0; rightVersion=11.2.0.0.0
2013-10-31 12:54:58: [11.2.0.3.0] is higher than [11.2.0.0.0]
2013-10-31 12:54:58: Stopping node listener 'LISTENER' ...
2013-10-31 12:54:58: leftVersion=11.2.0.3.0; rightVersion=11.2.0.0.0
2013-10-31 12:54:58: [11.2.0.3.0] is higher than [11.2.0.0.0]
2013-10-31 12:54:58: Invoking "/apps/oracle/product/11.2.0.2/grid/bin/srvctl stop listener -l LISTENER -n rac1 -f"
2013-10-31 12:54:58: trace file=/apps/grid/product/12.1.0.1/grid/cfgtoollogs/crsconfig/srvmcfg0.log
2013-10-31 12:54:58: Executing /apps/oracle/product/11.2.0.2/grid/bin/srvctl stop listener -l LISTENER -n rac1 -f
2013-10-31 12:54:58: Executing cmd: /apps/oracle/product/11.2.0.2/grid/bin/srvctl stop listener -l LISTENER -n rac1 -f
2013-10-31 12:55:06: Command output:
>  PRCR-1014 : Failed to stop resource ora.LISTENER.lsnr
>  PRCR-1065 : Failed to stop resource ora.LISTENER.lsnr
>  CRS-5016: Process "/apps/oracle/product/11.2.0.3/grid/bin/lsnrctl" spawned by agent "/apps/grid/product/12.1.0.1/grid/bin/oraagent.bin" for action "stop" failed: details at "(:CLSN00010:)" in "/apps/grid/product/12.1.0.1/grid/log/rac1/agent/crsd/oraagent_grid/oraagent_grid.log"
CRS-2675: Stop of 'ora.LISTENER.lsnr' on 'rac1' failed
>   
>End Command output
2013-10-31 12:55:06:   "/apps/oracle/product/11.2.0.2/grid/bin/srvctl stop listener -l LISTENER -n rac1 -f" failed with status 1.
2013-10-31 12:55:06: Executing cmd: /apps/grid/product/12.1.0.1/grid/bin/clsecho -p has -f clsrsc -m 180 "srvctl stop listener -l LISTENER -n rac1 -f" "null"
2013-10-31 12:55:07: Command output:
>  CLSRSC-180: An error occurred while executing the command 'srvctl stop listener -l LISTENER -n rac1 -f' (error code null)
>End Command output
2013-10-31 12:55:07: Executing cmd: /apps/grid/product/12.1.0.1/grid/bin/clsecho -p has -f clsrsc -m 180 "srvctl stop listener -l LISTENER -n rac1 -f" "null"
2013-10-31 12:55:07: Command output:
>  CLSRSC-180: An error occurred while executing the command 'srvctl stop listener -l LISTENER -n rac1 -f' (error code null)
>End Command output
2013-10-31 12:55:07: CLSRSC-180: An error occurred while executing the command 'srvctl stop listener -l LISTENER -n rac1 -f' (error code null)
2013-10-31 12:55:07: ###### Begin DIE Stack Trace ######
2013-10-31 12:55:07:     Package         File                 Line Calling   
2013-10-31 12:55:07:     --------------- -------------------- ---- ----------
2013-10-31 12:55:07:  1: main            rootcrs.pl            211 crsutils::dietrap
2013-10-31 12:55:07:  2: crsutils        crsutils.pm          9054 main::__ANON__
2013-10-31 12:55:07:  3: crsupgrade      crsupgrade.pm        3400 crsutils::stopNetLsnr
2013-10-31 12:55:07:  4: crsupgrade      crsupgrade.pm        3361 crsupgrade::upgradeNodeListener
2013-10-31 12:55:07:  5: crsupgrade      crsupgrade.pm         239 crsupgrade::upgrade_node
2013-10-31 12:55:07:  6: crsupgrade      crsupgrade.pm         190 crsupgrade::CRSUpgrade
2013-10-31 12:55:07:  7: main            rootcrs.pl            220 crsupgrade::new
2013-10-31 12:55:07: ####### End DIE Stack Trace #######

 

From oraagent log(CRS_HOME/log/nodename/agent/crsd/oraagent_grid/oraagent_grid.log)

=======================================================================


2013-10-31 12:55:07.944: [ora.LISTENER.lsnr][1681413888] {2:15925:2} [check] Adding Environment Variables ORACLE_HOME=/apps/oracle/product/11.2.0.3/grid
2013-10-31 12:55:07.944: [ora.LISTENER.lsnr][1681413888] {2:15925:2} [check] Adding Environment Variables TNS_ADMIN=/apps/oracle/product/11.2.0.3/grid/network/admin/
2013-10-31 12:55:07.944: [ora.LISTENER.lsnr][1681413888] {2:15925:2} [check] Adding Environment variable from USR_ORA_ENV ORACLE_BASE=/apps/oracle/grid
2013-10-31 12:55:07.944: [ora.LISTENER.lsnr][1681413888] {2:15925:2} [check] Adding Environment variable from USR_ORA_ENV LD_LIBRARY_PATH=
2013-10-31 12:55:07.944: [ora.LISTENER.lsnr][1681413888] {2:15925:2} [check] getOracleHomeAttrib: oracle_home = /apps/oracle/product/11.2.0.3/grid
2013-10-31 12:55:07.944: [ora.ons][1708955392] {2:15925:2} [check] getOracleHomeAttrib: oracle_home = /apps/grid/product/12.1.0.1/grid
2013-10-31 12:55:07.944: [ora.ons][1708955392] {2:15925:2} [check] Utils:execCmd action = 3 flags = 6 ohome = /apps/grid/product/12.1.0.1/grid/opmn/ cmdname = onsctli.
2013-10-31 12:55:07.944: [ora.LISTENER.lsnr][1681413888] {2:15925:2} [check] return val (clsagfwret) = 0
2013-10-31 12:55:07.944: [ora.LISTENER.lsnr][1681413888] {2:15925:2} [check] LsnrAgent::check }
2013-10-31 12:55:07.944: [ora.LISTENER_SCAN3.lsnr][1702651648] {2:15925:2} [check] LsnrAgent::check {
2013-10-31 12:55:07.944: [ora.LISTENER_SCAN3.lsnr][1702651648] {2:15925:2} [check] lsnrctl status LISTENER_SCAN3

 

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms