PQ Slave Initiating Offline of a Disk May Result in ORA-15078 or ORA-15130 ASM Diskgroup Forecibly Dismounted
(Doc ID 2636990.1)
Last updated on SEPTEMBER 28, 2021
Applies to:
Oracle Database - Enterprise Edition - Version 12.2.0.1 to 19.6.0.0.0 [Release 12.2 to 19]Information in this document applies to any platform.
Symptoms
DB instance Alert log will show PQ slaves initiating offline of a disk:
10-12 10:30:52.859464 10606 PDB_PRODNCM(3):NOTE: process
_p0la_cdbrac12 (302818) initiating offline of disk 120.2322156497
(DATAC1_FD_00_NZEPC01F03CE034XM) with mask 0x7e in group 1 (DATAC1) with
client assisting
This may happen with several disks until the diskgroup is forcibly taken offline. From the ASM alert log:
2019-10-13T12:45:32.691005-07:00
ERROR: too many offline disks in PST (grp 1)
2019-10-13T12:45:32.691874-07:00
NOTE: cache dismounting (not clean) group 1/0xB2A07837 (DATAC1)
DB instance alert log may report ORA-15178:
2019-10-12T10:45:28.595537-07:00
Errors in file /u01/app/oracle/diag/rdbms/cdbrac1/CDBRAC13/trace/CDBRAC13_ora_324683.trc:
ORA-15078: ASM diskgroup was forcibly dismounted
Errors in file /u01/app/oracle/diag/rdbms/cdbrac1/CDBRAC13/trace/CDBRAC13_ora_324683.trc:
ORA-15078: ASM diskgroup was forcibly dismounted
And other trace files may report ORA-15130:
*** 2019-10-13T12:45:33.395506-07:00
NOTE: handling error ORA-15130 while draining I/OsERROR: ORA-15130 thrown in
ARB0 for group number 1
ORA-15130: diskgroup "DATAC1" is being dismounted
NOTE: handling error ORA-15130 while draining I/OsERROR: ORA-15130 thrown in
ARB0 for group number 1
ORA-15130: diskgroup "DATAC1" is being dismounted
The PQ slave trace file from the PQ slave that initiated the disk offline will have reconnects in it like :
Reconnect to box 0x172eccb0 with timeout 0 failed with error 66 (Connect is
still in progress)
ossdebug_dump_stack()+185<-ossnet_vrpc_completion()+2663<-ossnet_process_netms
g_completion()+157<-ossnet_wait_all()+377<-ossnet_wait()+356<-oss_wait_ext()+3
27<-oss_wait()+108<-kfk_reap_oss_async_io()+519<-kfk_reap_ios_from_subsys()+21
0<-kfk_reap_ios()+432<-kfk_io1()+1461
...
*** 2019-10-13T12:45:31.095291-07:00 (PDB_PRODNCM(3))
Device Reopen async completion request 0x175db300, device handle 0x1734be50,
error code 0
Reconnect to box 0x173107d0 with timeout 1072 failed with error 38 (SKGXP
timeout)
still in progress)
ossdebug_dump_stack()+185<-ossnet_vrpc_completion()+2663<-ossnet_process_netms
g_completion()+157<-ossnet_wait_all()+377<-ossnet_wait()+356<-oss_wait_ext()+3
27<-oss_wait()+108<-kfk_reap_oss_async_io()+519<-kfk_reap_ios_from_subsys()+21
0<-kfk_reap_ios()+432<-kfk_io1()+1461
...
*** 2019-10-13T12:45:31.095291-07:00 (PDB_PRODNCM(3))
Device Reopen async completion request 0x175db300, device handle 0x1734be50,
error code 0
Reconnect to box 0x173107d0 with timeout 1072 failed with error 38 (SKGXP
timeout)
And may show a 15081 "failed xlate" error:
WARNING: failed xlate 3 kfiorq=0x7f3142e95fc0 fib=0x6e67af2d8 isread=1 sync=0
err=15081
NOTE: process _p07n_cdbexp11 (358769) initiating offline of disk
117.4043091467 (DATAC1_FD_05_NZEPC01F03CE033XM) with mask 0x7e in group 1
(DATAC1) with client assisting
err=15081
NOTE: process _p07n_cdbexp11 (358769) initiating offline of disk
117.4043091467 (DATAC1_FD_05_NZEPC01F03CE033XM) with mask 0x7e in group 1
(DATAC1) with client assisting
Changes
Possibly recently upgraded to 12.2+ or added more PQ slaves.
Cause
To view full details, sign in with your My Oracle Support account. |
|
Don't have a My Oracle Support account? Click to get started! |
In this Document
| Symptoms |
| Changes |
| Cause |
| Solution |
| References |