ORA-00600: internal error code, arguments: [ksprcvsp2], [1596993584] (Doc ID 2310632.1)

Errors in file <ASM Trace Directory>/+ASM2_ora_22020242.trc (incident=263361):
ORA-00600: internal error code, arguments: [ksprcvsp2], [1596993584], [], [], [], [], [], [], [], [], [], []
Incident details in: <ASM Trace Directory>/incident/incdir_263361/+ASM2_ora_22020242_i263361.trc
Thu Sep 21 19:26:46 2017
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
ORA-15018: diskgroup cannot be created
ORA-00600: internal error code, arguments: [ksprcvsp2], [1596993584], [], [], [], [], [], [], [], [], [], []
ERROR: CREATE DISKGROUP OCRDG NORMAL REDUNDANCY FAILGROUP fg1 DISK '/dev/rhdisk93' FAILGROUP fg2 DISK '/dev/rhdisk94' FAILGROUP fg3 DISK '/dev/rhdisk95' ATTRIBUTE 'compatible.asm' = '11.2.0.4.0'
Thu Sep 21 19:26:46 2017
Dumping diagnostic data in directory=[cdmp_20170921192646], requested by (instance=2, osid=22020242), summary=[incident=263361].

 

*** 2017-09-21 19:26:42.910
*** SESSION ID:(826.1503) 2017-09-21 19:26:42.910
*** CLIENT ID:() 2017-09-21 19:26:42.910
*** SERVICE NAME:() 2017-09-21 19:26:42.910
*** MODULE NAME:(sqlplus@test1 (TNS V1-V3)) 2017-09-21 19:26:42.910
*** ACTION NAME:() 2017-09-21 19:26:42.910

Dump continued from file: <ASM Trace Directory>/+ASM2_ora_22020242.trc
ORA-00600: internal error code, arguments: [ksprcvsp2], [1596993584], [], [], [], [], [], [], [], [], [], []

========= Dump for incident 263361 (ORA 600 [ksprcvsp2]) ========

*** 2017-09-21 19:26:42.913
dbkedDefDump(): Starting incident default dumps (flags=0x2, level=3, mask=0x0)
----- Current SQL Statement for this session (sql_id=86w2rqzmu2w2p) -----
CREATE DISKGROUP OCRDG NORMAL REDUNDANCY FAILGROUP fg1 DISK '/dev/rhdisk93' FAILGROUP fg2 DISK '/dev/rhdisk94' FAILGROUP fg3                                            DISK '/dev/rhdisk95' ATTRIBUTE 'compatible.asm' = '11.2.0.4.0'

----- Call Stack Trace -----

ksprcvsp <- ksp_spfile_write <- ksp_spfile_open <- ksp_set_update_spfile <- kspsetv0 <- kfgDgParmAdd <- kfxdrvCreate <- kfxdrvEntry

..

ksprcvsp()+560 call ksesin() FFFFFFFFFFF2660      and ksp_spfile_open()+1 call ksp_spfile_write() FFFFFFFFFFF2660

..

Argument/Register addr=0xfffffffffff2660.
Dump of memory from 0xfffffffffff2620 to 0xfffffffffff2760                                 
FFFFFFFFFFF2620 00000000 00000000 00000002 00000002 [................]
FFFFFFFFFFF2630 00000000 00000000 0147AE14 0147AE14 [.........G...G..]
FFFFFFFFFFF2640 00000000 00000000 07000100 40028E28 [............@..(]
FFFFFFFFFFF2650 06FFFFFF 00000001 00000000 00000000 [................]
FFFFFFFFFFF2660 07000100 3C003778 00000002 00820101 [....<.7x........]               <== "FFFFFFFFFFF2660" Header information invalidity is detected in processing to                                                                                                                                            write to SPFILE

...

Dump of memory from 0x108e7cadc to 0x108e7cc1c
108E7CAD0 0A004942 [..IB]
108E7CAE0 6B737064 6D706862 6C6B3A20 70687962 [kspdmphblk: phyb]
108E7CAF0 6C6B7369 7A65203D 2025642C 20686472 [lksize = %d, hdr]
108E7CB00 626C6B73 697A6520 3D202564 2C20666F [blksize = %d, fo]
108E7CB10 726D6174 203D2025 640A0049 6B737064 [rmat = %d..Ikspd]
108E7CB20 6D706862 6C6B3A20 68647262 6C6B7520 [mphblk: hdrblku ]
108E7CB30 3D202578 2C206864 72626C6B 203D2025 [= %x, hdrblk = %]
108E7CB40 780A0049 6B737072 63767370 30004942 [x..Iksprcvsp0.IB]
108E7CB50 6B737072 63767370 31004942 6B737072 [ksprcvsp1.IBkspr]
108E7CB60 63767370 32004942 64617461 626C6B20 [cvsp2.IBdatablk ]
108E7CB70 3A206B73 70726376 73700049 6B737072 [: ksprcvsp.Ikspr]
108E7CB80 63767370 3A6B7366 64726561 645F7265 [cvsp:ksfdread_re]
108E7CB90 73696C76 65720049 6B73702E 63403130 [silver.Iksp.c@10]
108E7CBA0 35363100 6B737072 63767370 3A626164 [561.ksprcvsp:bad]
108E7CBB0 5F666F72 6D617400 6B737072 63767370 [_format.ksprcvsp]
108E7CBC0 3A6B7366 64777269 74650049 6B73702E [:ksfdwrite.Iksp.]
108E7CBD0 63403130 36303000 6B73702E 63403130 [c@10600.ksp.c@10]
108E7CBE0 36353800 6B73702E 63403130 36363300 [658.ksp.c@10663.]
108E7CBF0 6B67652E 683A4B47 45454E44 4652414D [kge.h:KGEENDFRAM]
108E7CC00 45206572 726F7220 6E6F7420 68616E64 [E error not hand]
108E7CC10 6C656400 6B737072 737A7370 [led.ksprszsp]

..

2017-09-21 19:26:39.885776*:BBF52335:KGF:kgflk.c@226:kgflkLockGet(): kgflkLockGet: name=+ASM-SPFILE-(null) mode=2        <========

...

[0004]: spfile=/dev/asm_spfile

 

Node#1                                                                                                                                                                                  

SQL> select INSTANCE_NAME,INSTANCE_NUMBER,HOST_NAME,to_char(startup_time,'DD-MON-YYYY HH24:MI:SS'),STATUS,DATABASE_STATUS,INSTANCE_ROLE from gv$instance;

INSTANCE_NAME INSTANCE_NUMBER HOST_NAME TO_CHAR(STARTUP_TIME STATUS DATABASE_STATUS INSTANCE_ROLE
---------------- --------------- ------------------------------ -------------------- ------------ ----------------- ------------------
+ASM1 1 test1 12-AUG-2017 21:45:40 STARTED ACTIVE UNKNOWN
+ASM2 2 test2 12-AUG-2017 21:46:52 STARTED ACTIVE UNKNOWN

SQL> show parameter spfile;

NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
spfile string /dev/asm_spfile
SQL> create pfile='/tmp/pfile.ora' from spfile;

File created.

SQL> !ls -lrt /tmp/pfile.ora
-rw-r--r-- 1 oracle oinstall 0 22 Sep 16:26 /tmp/pfile.ora               <==========   Pfile has 0 zero bytes

 

Node#2

SQL> select INSTANCE_NAME,INSTANCE_NUMBER,HOST_NAME,to_char(startup_time,'DD-MON-YYYY HH24:MI:SS'),STATUS,DATABASE_STATUS,INSTANCE_ROLE from gv$instance;

INSTANCE_NAME INSTANCE_NUMBER HOST_NAME TO_CHAR(STARTUP_TIME STATUS DATABASE_STATUS INSTANCE_ROLE
---------------- --------------- ------------------------------ -------------------- ------------ ----------------- ------------------
+ASM2 2 test2 12-AUG-2017 21:46:52 STARTED ACTIVE UNKNOWN
+ASM1 1 test1 12-AUG-2017 21:45:40 STARTED ACTIVE UNKNOWN

SQL> show parameter spfile;

NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
spfile string /dev/asm_spfile
SQL> create pfile='/tmp/pfile.ora' from spfile;

File created.

SQL> !ls -lrt /tmp/pfile.ora
-rw-r--r-- 1 oracle oinstall 0 22 Sep 16:30 /tmp/pfile.ora                                                         <========== Pfile has 0 zero bytes

SQL> exit;
Disconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Real Application Clusters and Automatic Storage Management options
test2*+ASM2-/u01/app/oracle
>lquerypv -h /tmp/pfile.ora
test2*+ASM2-/u01/app/oracle

SQL> select inst_id,NUM,NAME,TYPE,VALUE,DISPLAY_VALUE from gv$parameter where ISDEFAULT='FALSE';

INST_ID NUM NAME TYPE VALUE DISPLAY_VALUE
---------- ---------- -------------------- ---------- ------------------------------ ------------------------------
1 2733 asm_diskgroups 2 DATA, BACKUP, FBACK DATA, BACKUP, FBACK
2 2733 asm_diskgroups 2 DATA, BACKUP, FBACK DATA, BACKUP, FBACK

 

..

*** However we are able get the ASM parameters details from ASM Memory;

SQL> create pfile='/tmp/init+ASM2_bkp.ora' from memory;

File created.

*** When we do "strings" of pfile ,it's stuck.

>strings /dev/asm_spfile
Ý8îü<                                              <=========== Invalid details
+ASM1.__large_pool_size=8388608
+ASM2.__large_pool_size=8388608
+ASM1.asm_diskgroups='BACKUP','DATA','FBACK','DISKCHECK','OCRDG'#Manual Mount
+ASM2.asm_diskgroups='FBACK','DATA','BACKUP','DISKCHECK','OCRDG'#Manual Mount

<<<<<<<<< Hang Here <<<<<<<<<<<<<<<<<

 

*** gpnp tool returning the I/O error

>gpnptool get -o- | xmllint --format - | grep SPFile
ksh: xmllint: not found
Error: I/O error while trying to write out file.
Failed to save profile - error 7 (CLSGPNP_IO).
Requested profile size is 1983 bytes

Error CLSGPNP_IO getting profile.

 

*** gpnp tool says, the correct path of spfile

>gpnptool get
Warning: some command line parameters were defaulted. Resulting command line:
/app/oracle_cluster/oracle/product/11.2.0.4/grid/bin/gpnptool.bin get -o-

SPFile="/dev/asm_spfile"/>

 

*** SQL> SELECT * FROM V$ASM_DISKGROUP;

GROUP_NUMBER NAME SECTOR_SIZE BLOCK_SIZE ALLOCATION_UNIT_SIZE STATE TYPE TOTAL_MB FREE_MB HOT_USED_MB COLD_USED_MB REQUIRED_MIRROR_FREE_MB USABLE_FILE_MB OFFLINE_DISKS COMPATIBILITY DATABASE_COMPATIBILITY V
1 BACKUP 512 4096 1048576 MOUNTED EXTERN 2048000 563078 0 1484922 0 563078 0 11.2.0.4.0 11.2.0.4.0 N
2 DATA 512 4096 1048576 MOUNTED EXTERN 14336000 3344207 0 10991793 0 3344207 0 11.2.0.4.0 11.2.0.4.0 N
3 FBACK 512 4096 1048576 MOUNTED EXTERN 2048002 1791540 0 256462 0 1791540 0 11.2.0.4.0 11.2.0.4.0 N

*** SQL> SELECT * FROM V$ASM_ATTRIBUTE;

NAME VALUE GROUP_NUMBER ATTRIBUTE_INDEX ATTRIBUTE_INCARNATION READ_ON SYSTEM_

au_size 1048576 1 5 1 Y Y
sector_size 512 1 6 1 Y Y
compatible.asm 11.2.0.4.0 1 20 1 N Y
compatible.rdbms 11.2.0.4 1 21 1 N Y
compatible.advm 11.2.0.4 1 22 1 N Y

au_size 1048576 2 8 1 Y Y
sector_size 512 2 9 1 Y Y
compatible.asm 11.2.0.4.0 2 80 1 N Y
compatible.rdbms 11.2.0.4 2 81 1 N Y
compatible.advm 11.2.0.4 2 82 1 N Y

au_size 1048576 3 8 1 Y Y
sector_size 512 3 9 1 Y Y
compatible.asm 11.2.0.4.0 3 80 1 N Y
compatible.rdbms 11.2.0.4 3 81 1 N Y
compatible.advm 11.2.0.4 3 82 1 N Y