SLES 12: Database Startup Error with ORA-27300 ORA-27301 ORA-27303 While Starting using Srvctl

(Doc ID 2340986.1)

Last updated on DECEMBER 25, 2017

Applies to:

Oracle Database - Enterprise Edition - Version 12.1.0.2 and later
Linux x86-64

Symptoms

12.1 RAC database error while starting via srvctl, alert log showing database startup and further goes down

alert_ORCL2.log inside <oracle base>/diag/rdbms/<db name>/<SID name>/trace:

Fri Nov 03 15:06:25 2017
Adjusting the default value of parameter parallel_max_servers
from 960 to 486 due to the value of parameter processes (600)
Starting ORACLE instance (normal) (OS id: 19684)
.
.
Fri Nov 03 15:10:45 2017
Process startup failed, error stack:
Fri Nov 03 15:10:45 2017
Errors in file /u01/app/oracle/diag/rdbms/ORCL/ORCL2/trace/ORCL2_psp0_19706.trc:
ORA-27300: OS system dependent operation:fork failed with status: 11
ORA-27301: OS failure message: Resource temporarily unavailable
ORA-27302: failure occurred at: skgpspawn3
Fri Nov 03 15:10:46 2017
Shutting down instance (abort)
License high water mark = 2
Fri Nov 03 15:10:46 2017
USER (ospid: 22067): terminating the instance
Fri Nov 03 15:10:47 2017
Instance terminated by USER, pid = 22067
Fri Nov 03 15:10:47 2017
Instance shutdown complete
Errors in file //u01/app/oracle/diag/rdbms/ORCL/ORCL2/trace/ORCL2_mmon_19853.trc (incident=14681):
ORA-00600: internal error code, arguments: [KSLGES_3], [], [], [], [], [], [], [], [], [], [], []
ORA-27300: OS system dependent operation:semop failed with status: 43
ORA-27301: OS failure message: Identifier removed
ORA-27302: failure occurred at: sskgpwwait1
Incident details in: /u01/app/oracle/diag/rdbms/ORCL/ORCL2/incident/incdir_14681/ORCL2_mmon_19853_i14681.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.

 

ORCL2_psp0_19706.trc: 

.
*** 2017-11-03 15:10:44.989
Process startup failed, error stack:
ORA-27300: OS system dependent operation:fork failed with status: 11
ORA-27301: OS failure message: Resource temporarily unavailable
ORA-27302: failure occurred at: skgpspawn3

*** 2017-11-03 15:10:45.993
Process startup failed, error stack:
ORA-27300: OS system dependent operation:fork failed with status: 11
ORA-27301: OS failure message: Resource temporarily unavailable
ORA-27302: failure occurred at: skgpspawn3

 

 ORCL2_mmon_19853.trc:

.
.
*** 2017-11-03 15:09:57.908
***KELR Apply Log: unable to schedule MMON Slave, error 3

*** 2017-11-03 15:10:47.847
Incident 14681 created, dump file: /u01/app/oracle/diag/rdbms/ORCL/ORCL2/incident/incdir_14681/ORCL2_mmon_19853_i14681.trc
ORA-00600: internal error code, arguments: [KSLGES_3], [], [], [], [], [], [], [], [], [], [], []
ORA-27300: OS system dependent operation:semop failed with status: 43
ORA-27301: OS failure message: Identifier removed
ORA-27302: failure occurred at: sskgpwwait1

KEBM: MMON action policy violation. 'PQ: Adjust Slave Pool' viol=0; err=600
error 0 detected in background process
kgxgnsdr: clssgsshdereg: warning: return status 26 (-558242808 )
OPIRIP: Uncaught error 447. Error stack:
ORA-00447: fatal error in background process
ORA-00600: internal error code, arguments: [KSLGES_3], [], [], [], [], [], [], [], [], [], [], []
ORA-27300: OS system dependent operation:semop failed with status: 43
ORA-27301: OS failure message: Identifier removed
ORA-27302: failure occurred at: sskgpwwait1
kgxgnsdr: clssgsshdereg: warning: return status 26 (-813323584 )

 

 

Following error reported in OS logs

messages: 

2017-11-03T15:10:11.690393+01:00 rachost1 Oracle Audit[7888]: LENGTH : '200' ACTION :[52] 'ALTER DATABASE MOUNT /* db agent *//* {2:39656:2} */' DATABASE USER:[1] '/' PRIVILEGE :[6] 'SYSDBA' CLIENT USER:[6] 'oracle' CLIENT TERMINAL:[0] '' STATUS:[1] '0' DBID:[10] '2949004148'
2017-11-03T15:10:47.700094+01:00 rachost1 kernel: [ 173.788859] cgroup: fork rejected by pids controller in /system.slice/ohasd.service <<<<<<<<<<<<<<<<<<<<<

Database able to startup with sqlplus. 

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms