Can not Start Instance with error: GIM-00092: OS failure occurred at: sskgmsmr_7 or at: sskgmsmr_18 (Doc ID 1224665.1)

Last updated on DECEMBER 01, 2013

Applies to:

Oracle Database - Enterprise Edition - Version 11.2.0.1 and later
Information in this document applies to any platform.

Symptoms

Case I.
Just upgrade the RAC database from 10.2.0.3 to 11.2.0.1 manually. After upgrade, srvctl start database can only start instance on node1, it fails on node2, but sqlplus can start instance 2.

[oracle@node02:/home/oracle]$ srvctl start database -d PRD
PRCR-1079 : Failed to start resource ora.prd.db
CRS-5011: Check of resource "PRD" failed: details at "(:CLSN00007:)" in "/u01/app/11.2/grid/log/node02/agent/crsd/oraagent_oracle/oraagent_oracle.log"
CRS-5011: Check of resource "PRD" failed: details at "(:CLSN00007:)" in "/u01/app/11.2/grid/log/node02/agent/crsd/oraagent_oracle/oraagent_oracle.log"
CRS-2674: Start of 'ora.prd.db' on 'node02' failed
CRS-2528: Unable to place an instance of 'ora.prd.db' as all possible servers are occupied by the resource already

[oracle@node01:/home/oracle]$ crsctl stat res -t
--------------------------------------------------------------------------------
NAME  TARGET   STATE     SERVER    STATE_DETAILS
--------------------------------------------------------------------------------
ora.prd.db
    1 ONLINE   ONLINE    node01    Open
    2 ONLINE   OFFLINE

[oracle@node02:/u01/app/11.2/db_1/dbs]$ sqlplus " / as sysdba"

SQL*Plus: Release 11.2.0.1.0 Production on Wed Sep 29 20:37:34 2010

Copyright (c) 1982, 2009, Oracle. All rights reserved.

Connected to an idle instance.

SQL> startup
ORACLE instance started.

Total System Global Area 534462464 bytes
Fixed Size 2215064 bytes
Variable Size 226493288 bytes
Database Buffers 297795584 bytes
Redo Buffers 7958528 bytes
Database mounted.
Database opened.


agent log shows:

oraagent_oracle.log
------------
2010-09-30 19:32:43.763: [ora.prd.db][1159723328] [start] DbAgent::startCbk: }
2010-09-30 19:32:43.764: [ora.prd.db][1159723328] [start] InstAgent::start: }
2010-09-30 19:32:43.764: [ora.prd.db][1159723328] [start] clsn_agent::start }
2010-09-30 19:32:43.764: [ AGFW][1159723328] Command: start for resource: ora.prd.db 2 1 completed with status: SUCCESS
2010-09-30 19:32:43.764: [ AGFW][1159723328] Executing command: check for resource: ora.prd.db 2 1
2010-09-30 19:32:43.765: [ora.prd.db][1159723328] [check] Gimh::check OH /u01/app/11.2/db_1 SID PRD2
2010-09-30 19:32:43.765: [ AGFW][1170213184] Agent sending reply for: RESOURCE_START[ora.prd.db 2 1] ID 4098:28482
2010-09-30 19:32:43.767: [ora.prd.db][1159723328] [check] GIMH: GIM-00104: Health check failed to connect to instance.
GIM-00090: OS-dependent operation: SMR file size check failed with status: 0
GIM-00091: OS failure message: Error 0
GIM-00092: OS failure occurred at: sskgmsmr_18


2010-09-30 19:32:43.767: [ora.prd.db][1159723328] [check] (:CLSN00007:)DbAgent::check failed gimh state 0
2010-09-30 19:32:43.767: [ora.prd.db][1159723328] [check] Exception type=2 string=CRS-5011: Check of resource "PRD" failed: details at "(:CLSN00007:)" in "/u01/app/11.2/grid/log/node02/agent/crsd/oraagent_oracle/oraagent_oracle.log"
2010-09-30 19:32:43.768: [ AGFW][1170213184] Agent sending reply for: RESOURCE_START[ora.prd.db 2 1] ID 4098:28482
2010-09-30 19:32:43.769: [ AGFW][1159723328] check for resource: ora.prd.db 2 1 completed with status: FAILED


Case II.
Both srvctl and sqlplus start instance fails with:

$ sqlplus / as sysdba

SQL*Plus: Release 11.2.0.2.0 Production on Tue Jan 24 08:08:38 2012

Copyright (c) 1982, 2010, Oracle.  All rights reserved.


Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.2.0 - Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Data Mining and Real Application Testing options

SQL> startup
ORA-27103: internal error
Linux Error: 2: No such file or directory
Additional information: 1441798
Additional information: 2


oraagent_oracle.log shows:

2012-01-24 08:08:56.742: [ AGFW][2833648528] {0:9:194} Command: start for resource: ora.a1.db 1 1 completed with status: FAIL
2012-01-24 08:08:56.743: [CLSFRAME][3039389392] TM [MultiThread] is changing desired thread # to 3. Current # is 2
2012-01-24 08:08:56.744: [ AGFW][2766531472] {0:9:194} Agent sending reply for: RESOURCE_START[ora.a1.db 1 1] ID 4098:169637
2012-01-24 08:08:56.751: [ora.a1.db][2900765584] {0:9:194} [check] Gimh::check OH /u01/app/oradb1/product/11.2.0/dbhome_1 SID A11
2012-01-24 08:08:56.751: [ora.a1.db][2900765584] {0:9:194} [check] GIMH: GIM-00104: Health check failed to connect to instance.
GIM-00090: OS-dependent operation:open failed with status: 2
GIM-00091: OS failure message: No such file or directory
GIM-00092: OS failure occurred at: sskgmsmr_7



Case III.
Both srvctl and sqlplus start instance fails with:

$ srvctl start instance -d A1 -i A11
PRCR-1013 : Failed to start resource ora.a1.db
PRCR-1064 : Failed to start resource ora.a1.db on node auw2k3
CRS-5017: The resource action "ora.a1.db start" encountered the following error:
ORA-03113: end-of-file on communication channel
Process ID: 23368
Session ID: 1 Serial number: 3

CRS-2674: Start of 'ora.a1.db' on 'auw2k3' failed

$ sqlplus / as sysdba

SQL*Plus: Release 11.2.0.2.0 Production on Tue Jan 24 08:09:31 2012

Copyright (c) 1982, 2010, Oracle.  All rights reserved.

Connected to an idle instance.

SQL> startup
ORACLE instance started.

Total System Global Area  790941696 bytes
Fixed Size                  1347084 bytes
Variable Size             591397364 bytes
Database Buffers          192937984 bytes
Redo Buffers                5259264 bytes
Database mounted.
ORA-03113: end-of-file on communication channel
Process ID: 22375
Session ID: 1 Serial number: 5


oraagent_oracle.log shows:

2012-01-24 08:15:36.593: [ora.a1.db][2800765840] {1:59886:13801} [start] clsnUtils::error Exception type=2 string=
CRS-5017: The resource action "ora.a1.db start" encountered the following error:
ORA-03113: end-of-file on communication channel
Process ID: 23368
Session ID: 1 Serial number: 3


2012-01-24 08:15:36.593: [ora.a1.db][2800765840] {1:59886:13801} [start] clsn_agent::start }
2012-01-24 08:15:36.593: [    AGFW][2800765840] {1:59886:13801} Command: start for resource: ora.a1.db 1 1 completed with status: FAIL
2012-01-24 08:15:36.594: [ora.a1.db][2800765840] {1:59886:13801} [check] Gimh::check OH /u01/app/oradb1/product/11.2.0/dbhome_1 SID A11
2012-01-24 08:15:36.594: [ora.a1.db][2800765840] {1:59886:13801} [check] GIMH: GIM-00104: Health check failed to connect to instance.
GIM-00090: OS-dependent operation: SMR file size check failed with status: 0
GIM-00091: OS failure message: Error 0
GIM-00092: OS failure occurred at: sskgmsmr_18



Case IV.

Sqlplus starts database isfine, but srvctl fails with:

  

 

 

Changes

Case I. Just upgrade the RAC database from 10.2.0.3 to 11.2.0.1

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms