E-AS: / E-TUX: Unable to Shutdown / Start the Application Server Domain (Doc ID 3015614.1)

Last updated on APRIL 11, 2024

Applies to:

Symptoms

Unable to shutdown / restart the application server domain. The following errors are observed in the Tuxedo log file (<PS_CFG_HOME>/appserv//logs/TUXLOG.MMDDYY).

When attempting to shutdown, using the psadmin utility:
LIBTUX_CAT:577: ERROR: Unable to register because the slot is already owned by another process
FATAL: internal error: CMDTUX_CAT:764: ERROR: can't attach to BB

When attempting to start:
ERROR: msgrcv err(LIBTUX_CAT:666: ERROR: Message operation failed because the queue was removed): errno=43,qid=65540,buf=39457816,bytes=4712,type=-1073741824,flag=0
ERROR: msgrcv err(LIBTUX_CAT:669: ERROR: Message operation failed because of the invalid message queue identifier): errno=22,qid=65540,buf=39457816,bytes=4712,type=-1073741824,flag=0
LIBTUX_CAT:271: ERROR: System lock semop failure, key = 193698 (errno = 22)
LIBTUX_CAT:268: ERROR: Failed to stop serving

Or
LIBTUX_CAT:577: ERROR: Unable to register because the slot is already owned by another process

After cleaning IPC Resources, setting TraceSQL=31, and attempting to restart the domain, using the psadmin utility:
- In the Tuxedo log file (<PS_CFG_HOME>/appserv/<domain>/logs/TUXLOG.MMDDYY)

    120748.<hostname>!PSADMIN.41486: Begin boot attempt on domain <DomainName>
    (...)
    120827.<hostname>!PSMONITORSRV.41952.3010589184.-2: 04-03-2024: Tuxedo Version 12.2.2.0.0, 64-bit
    120827.<hostname>!PSMONITORSRV.41952.3010589184.-2: LIBTUX_CAT:262: INFO: Standard main starting
    121026.<hostname>!tmboot.41686.1598947840.-2: CMDTUX_CAT:1855: INFO: TM_BOOTPRESUMEDFAIL option is selected
    121026.<hostname>!tmboot.41686.1598947840.-2: CMDTUX_CAT:1859: ERROR: Server process ID 41952 failed to initialize within 120 seconds
    121026.<hostname>!tmboot.41686.1598947840.-2: tmboot: CMDTUX_CAT:827: ERROR: Fatal error encountered; initiating user error handler
    131857.<hostname>!PSMONITORSRV.41952.3010589184.-2: LIBTUX_CAT:252: WARN: Cannot synchronize with tmboot...continuing
    
    131903.<hostname>!TMUSREVT.41693.2916385280.0: CMDTUX_CAT:3031: INFO: TMUSREVT is exiting
    131906.<hostname>!BBL.41674.3124396544.0: CMDTUX_CAT:26: INFO: The BBL is exiting system
    131909.<hostname>!PSADMIN.41486End boot attempt on domain <DomainName>
    
    
- In the trace file <OPRID>_PSAPPSRV.tracesql (<PS_CFG_HOME>/appserv/<domain>/logs/<OPRID>_PSAPPSRV.tracesql):

    (...)
    PSAPPSRV.41778 (0)      1-990    12.08.09    0.000033 Cur#1.41778.<DomainName> RC=0 Dur=0.000021 COM Stmt=SELECT USEMON_SW, PM_UM_BUFFERSIZE, PTUMMON_STATE FROM PSLCMOPTIONS
    PSAPPSRV.41778 (0)      1-991    12.08.09    0.000893 Cur#1.41778.<DomainName> RC=0 Dur=0.000887 EXE
    PSAPPSRV.41778 (0)      1-992    12.08.09    0.000006 Cur#1.41778.<DomainName> RC=1 Dur=0.000001 Fetch
    PSAPPSRV.41778 (0)      1-993    12.08.09    0.000006 Cur#1.41778.<DomainName> RC=0 Dur=0.000001 Commit
    PSAPPSRV.41778 (0)      1-994    12.08.09    0.000006 Cur#2.41778.notSamTran RC=0 Dur=0.000000 Open Cursor Handle=0000000001EEE0B0
    PSAPPSRV.41778 (0)      1-995    12.08.09    0.000003 Cur#2.41778.<DomainName> RC=0 Dur=0.000001 Commit
    PSAPPSRV.41778 (0)      1-996    12.08.09    0.000006 Cur#2.41778.<DomainName> RC=0 Dur=0.000003 Disconnect
    PSAPPSRV.41778 (0)      1-997    12.08.09    0.000005 Cur#1.41778.<DomainName> RC=0 Dur=0.000002 Disconnect
    PSAPPSRV.41778 (0)      1-998    13.19.01 4251.862098 Cur#0.41778.notSamTran RC=0 Dur=0.001260 Destroy Connection Handle=0000000001EEE0B0
    PSAPPSRV.41778 (0)      1-999    13.19.01    0.005701 Cur#0.41778.notSamTran RC=0 Dur=0.005691 DON
    PSAPPSRV.41704 (0)      1-998    13.19.01 4255.020430 Cur#0.41704.notSamTran RC=0 Dur=0.001070 Destroy Connection Handle=0000000002DACFE0
    PSAPPSRV.41704 (0)      1-999    13.19.01    0.004157 Cur#0.41704.notSamTran RC=0 Dur=0.004138 DON
    => Attention to gap between 12.08.09 ~ 13.19.01. After 12.08.09, PSAPPSRV server processes waited for a long time.
    
    
- In the trace file <OPRID>_PSSAMSRV.tracesql (<PS_CFG_HOME>/appserv/<domain>/logs/<OPRID>_PSSAMSRV.tracesql): Same gap as in <OPRID>_PSAPPSRV.tracesql is observed.

    Last lines in the trace:
    PSSAMSRV.41847 (0)      1-1157   12.08.12    0.000027 Cur#1.41847.<DomainName> RC=0 Dur=0.000016 COM Stmt=SELECT COUNT(*) FROM PS_PT_URL_PROPS WHERE URL_ID = :1
    PSSAMSRV.41847 (0)      1-1158   12.08.12    0.000004 Cur#1.41847.<DomainName> RC=0 Dur=0.000000 Bind-1 type=2 length=11 value=PPM_MONITOR
    PSSAMSRV.41847 (0)      1-1159   12.08.12    0.000795 Cur#1.41847.<DomainName> RC=0 Dur=0.000791 EXE
    PSSAMSRV.41847 (0)      1-1160   12.08.12    0.000004 Cur#1.41847.<DomainName> RC=0 Dur=0.000000 Fetch
    PSSAMSRV.41847 (0)      1-1161   12.08.12    0.000014 Cur#1.41847.<DomainName> RC=0 Dur=0.000000 Commit
    PSSAMSRV.41847 (0)      1-1162   12.08.12    0.000004 Cur#1.41847.<DomainName> RC=0 Dur=0.000002 Disconnect
    PSSAMSRV.41847 (0)      1-1163   13.19.01 4249.529754 Cur#0.41847.notSamTran RC=0 Dur=0.001442 Destroy Connection Handle=00000000016F2A10
    PSSAMSRV.41847 (0)      1-1164   13.19.01    0.004509 Cur#0.41847.notSamTran RC=0 Dur=0.004499 DON
    => After 12.08.12, the PSSAMSRV server process waited for a long time.
    
    
- In the trace file <OPRID>_PSPUBHND_dflt.tracesql (<PS_CFG_HOME>/appserv/<domain>/logs/<OPRID>_PSPUBHND_dflt.tracesql) => 1 rollback. Same gap as in <OPRID>_PSAPPSRV.tracesql is observed.

    Last lines in the trace:
    (...)
    PSPUBHND_dflt.41883 (0)      1-1171   12.08.19    0.000005 Cur#2.41883.<DomainName> RC=0 Dur=0.000000 Commit
    PSPUBHND_dflt.41883 (0)      1-1172   13.19.00 4241.472272 Cur#5.41883.<DomainName> RC=0 Dur=0.000001 Rollback
    PSPUBHND_dflt.41883 (0)      1-1173   13.19.00    0.000014 Cur#5.41883.<DomainName> RC=0 Dur=0.000009 Disconnect
    PSPUBHND_dflt.41883 (0)      1-1174   13.19.00    0.000006 Cur#4.41883.<DomainName> RC=0 Dur=0.000003 Disconnect
    PSPUBHND_dflt.41883 (0)      1-1175   13.19.00    0.000007 Cur#3.41883.<DomainName> RC=0 Dur=0.000004 Disconnect
    PSPUBHND_dflt.41883 (0)      1-1176   13.19.00    0.000023 Cur#2.41883.<DomainName> RC=0 Dur=0.000020 Disconnect
    PSPUBHND_dflt.41883 (0)      1-1177   13.19.00    0.000006 Cur#1.41883.<DomainName> RC=0 Dur=0.000001 Commit
    PSPUBHND_dflt.41883 (0)      1-1178   13.19.00    0.000005 Cur#1.41883.<DomainName> RC=0 Dur=0.000002 Disconnect
    PSPUBHND_dflt.41883 (0)      1-1179   13.19.00    0.004251 Cur#0.41883.notSamTran RC=0 Dur=0.001007 Destroy Connection Handle=00000000025E87F0
    PSPUBHND_dflt.41883 (0)      1-1180   13.19.00    0.003538 Cur#0.41883.notSamTran RC=0 Dur=0.003529 DON
    => After 12.08.19, the PSPUBHND_dflt server process waited for a long time before rolling back.
    
    
- In the trace file <OPRID>_PSMONITORSRV.tracesql (<PS_CFG_HOME>/appserv/<domain>/logs/<OPRID>_PSMONITORSRV.tracesql):

    Last lines in the trace:
    PSMONITORSRV.41952 (0)      1-1196   12.09.20    0.000078 Cur#1.41952.<DomainName> RC=0 Dur=0.000032 COM Stmt=INSERT INTO PSPMAGENT (PM_AGENTID, PM_AGENT_TYPE, PM_DOMAIN_NAME, PM_DOMAIN_TYPE, PM_DOMAIN_DIR, PM_DOMAIN_MONITOR, PM_HOST_PORT, PM_INSTANCE, PM_SYSTEMID, PM_AGENT_INACTIVE, PM_JMX_RMI_PORT) VALUES(:1, :2, :3, :4, :5, :6, :7, :8, :9, 'N', :10)
    PSMONITORSRV.41952 (0)      1-1197   12.09.20    0.000005 Cur#1.41952.<DomainName> RC=0 Dur=0.000000 Bind-1 type=8 length=4 value=1
    PSMONITORSRV.41952 (0)      1-1198   12.09.20    0.000002 Cur#1.41952.<DomainName> RC=0 Dur=0.000000 Bind-2 type=2 length=12 value=PSMONITORSRV
    PSMONITORSRV.41952 (0)      1-1199   12.09.20    0.000002 Cur#1.41952.<DomainName> RC=0 Dur=0.000000 Bind-3 type=2 length=5 value=<DomainName>
    PSMONITORSRV.41952 (0)      1-1200   12.09.20    0.000002 Cur#1.41952.<DomainName> RC=0 Dur=0.000000 Bind-4 type=2 length=2 value=01
    PSMONITORSRV.41952 (0)      1-1201   12.09.20    0.000003 Cur#1.41952.<DomainName> RC=0 Dur=0.000000 Bind-5 type=2 length=32 value=/psoft/<DomainName>/ptcfg/appserv/<DomainName>
    PSMONITORSRV.41952 (0)      1-1202   12.09.20    0.000002 Cur#1.41952.<DomainName> RC=0 Dur=0.000000 Bind-6 type=1 length=1 value=Y
    PSMONITORSRV.41952 (0)      1-1203   12.09.20    0.000002 Cur#1.41952.<DomainName> RC=0 Dur=0.000000 Bind-7 type=2 length=16 value=<hostname>:<port>
    PSMONITORSRV.41952 (0)      1-1204   12.09.20    0.000002 Cur#1.41952.<DomainName> RC=0 Dur=0.000000 Bind-8 type=2 length=1 value=1
    PSMONITORSRV.41952 (0)      1-1205   12.09.20    0.000003 Cur#1.41952.<DomainName> RC=0 Dur=0.000000 Bind-9 type=8 length=4 value=1
    PSMONITORSRV.41952 (0)      1-1206   12.09.20    0.000002 Cur#1.41952.<DomainName> RC=0 Dur=0.000000 Bind-10 type=8 length=4 value=10100
    ===> PSMONITORSRV.41952 (0)      1-1207   13.18.56 4175.492053 Cur#1.41952.<DomainName> RC=0 Dur= 4175.492011 EXE
    PSMONITORSRV.41952 (0)      1-1208   13.18.56    0.000121 Cur#1.41952.<DomainName> RC=0 Dur=0.000055 COM Stmt=SELECT 1 FROM PSPMAGENTFILTER WHERE PM_SYSTEMID=:1 AND PM_AGENT_TYPE=:2
    PSMONITORSRV.41952 (0)      1-1209   13.18.56    0.000011 Cur#1.41952.<DomainName> RC=0 Dur=0.000000 Bind-1 type=8 length=4 value=1
    PSMONITORSRV.41952 (0)      1-1210   13.18.56    0.000003 Cur#1.41952.<DomainName> RC=0 Dur=0.000000 Bind-2 type=2 length=12 value=PSMONITORSRV
    (...)
    PSMONITORSRV.41952 (0)      1-1222   13.18.57    0.000102 Cur#1.41952.<DomainName> RC=0 Dur=0.000049 COM Stmt=DELETE FROM PSPTPNEVTCLT WHERE HOST = :1 AND PORT_NUM = :2
    PSMONITORSRV.41952 (0)      1-1223   13.18.57    0.000006 Cur#1.41952.<DomainName> RC=0 Dur=0.000001 Bind-1 type=2 length=11 value=<hostname>
    PSMONITORSRV.41952 (0)      1-1224   13.18.57    0.000003 Cur#1.41952.<DomainName> RC=0 Dur=0.000000 Bind-2 type=8 length=4 value=<port>
    PSMONITORSRV.41952 (0)      1-1225   13.18.58    0.329030 Cur#1.41952.<DomainName> RC=0 Dur=0.329017 EXE
    PSMONITORSRV.41952 (0)      1-1226   13.18.58    0.000822 Cur#1.41952.<DomainName> RC=0 Dur=0.000794 Commit
    PSMONITORSRV.41952 (0)      1-1227   13.18.58    0.000009 Cur#1.41952.<DomainName> RC=0 Dur=0.000006 Disconnect
    PSMONITORSRV.41952 (0)      1-1228   13.18.58    0.001475 Cur#0.41952.notSamTran RC=0 Dur=0.000809 Destroy Connection Handle=0000000000B30130
    PSMONITORSRV.41952 (0)      1-1229   13.18.58    0.012848 Cur#0.41952.notSamTran RC=0 Dur=0.012839 DON

Cause

My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.