My Oracle Support Banner

ORA-07445 [SKGXPRQHCOMPLETE()+24] When Upgrade From 11204 TO 192 (Doc ID 2617196.1)

Last updated on JULY 28, 2020

Applies to:

Oracle Database - Enterprise Edition - Version 11.2.0.4 and later
Information in this document applies to any platform.

Symptoms

The rootupgrade.sh is failed on the first node while upgrading the GI from 11204 to 19.2, ora.asm failed to start 

<ORACLE BASE>/diag/crs/<HOSTNAME>/crs/trace/alert log:

2018-10-30 06:50:56.428 [OHASD(18419)]CRS-8500: Oracle Clusterware OHASD process is starting with operating system process ID 18419
2018-10-30 06:50:56.659 [OHASD(18419)]CRS-0714: Oracle Clusterware Release 19.0.0.0.0.
2018-10-30 06:50:56.696 [OHASD(18419)]CRS-2112: The OLR service started on node slc06eun.
2018-10-30 06:50:57.669 [OHASD(18419)]CRS-1301: Oracle High Availability Service started on node slc06eun

...

2018-10-30 06:52:55.114 [ORAAGENT(18647)]CRS-5017: The resource action "ora.asm start" encountered the following error:
2018-10-30 06:52:55.114+ORA-03113: end-of-file on communication channel
Process ID: 0
Session ID: 0 Serial number: 0
. For details refer to "(:CLSN00107:)" in "<oracle_base>/crs/<nodename>/crs/trace/ohasd_oraagent_crsusr.trc".
2018-10-30 08:01:34.657 [CRSCTL(1783)]CRS-1013: The OCR location in an ASM disk group is inaccessible. Details in "<oracle_base>/crs/<nodename>//crs/trace/crsctl_1783.trc.

 <ORACLE BASE>/crsdata/<nodename>/crsconfig/rootcrs_<nodename>-10-30_06-30-22AM.log:

 

2018-10-30 06:52:00: Executing cmd: /u01/app/19.2/grid/bin/crsctl start resource ora.asm -init
2018-10-30 06:52:57: Command output:
> CRS-2672: Attempting to start 'ora.asm' on 'node1'
> CRS-5017: The resource action "ora.asm start" encountered the following error:
> ORA-03113: end-of-file on communication channel
> Process ID: 0
> Session ID: 0 Serial number: 0
> . For details refer to "(:CLSN00107:)" in "<oracle_base>/diag/crs/slc06eun/crs/trace/ohasd_oraagent_crsusr.trc".
> CRS-2674: Start of 'ora.asm' on 'slc06eun' failed
> CRS-2679: Attempting to clean 'ora.asm' on 'node1'
> CRS-2681: Clean of 'ora.asm' on 'slc06eun' succeeded
> CRS-4000: Command Start failed, or completed with errors.
.....
2018-10-30 06:53:49: Executing cmd: /u01/app/19.2/grid/bin/clsecho -p has -f clsrsc -m 247
2018-10-30 06:53:49: Executing cmd: /u01/app/19.2/grid/bin/clsecho -p has -f clsrsc -m 247
2018-10-30 06:53:50: Command output:
> CLSRSC-247: Failed to start ASM
>End Command output
2018-10-30 06:53:50: CLSRSC-247: Failed to start ASM
2018-10-30 06:53:50: ###### Begin DIE Stack Trace ######
2018-10-30 06:53:50: Package File Line Calling
2018-10-30 06:53:50: --------------- -------------------- ---- ----------
2018-10-30 06:53:50: 1: main rootcrs.pl 355 crsutils::dietrap
2018-10-30 06:53:50: 2: crsupgrade crsupgrade.pm 1634 main::__ANON__
2018-10-30 06:53:50: 3: crsupgrade crsupgrade.pm 1582 crsupgrade::start_asm_resource
2018-10-30 06:53:50: 4: crsupgrade crsupgrade.pm 1490 crsupgrade::startStackFor112Upg
2018-10-30 06:53:50: 5: crsupgrade crsupgrade.pm 996 crsupgrade::upgrade_cluster
2018-10-30 06:53:50: 6: crsupgrade crsupgrade.pm 579 crsupgrade::CRSUpgrade
2018-10-30 06:53:50: 7: main rootcrs.pl 364 crsupgrade::new
2018-10-30 06:53:50: ####### End DIE Stack Trace #######

 <ORACLE BASE>/diag/crs/<HOSTNAME>/crs/trace/ohasd_oraagent_crsusr.trc:

2018-10-30 06:52:55.103 :CLSDYNAM:11: [ ora.asm]{0:0:152} [start] ORA-03113: end-of-file on communication channel
Process ID: 0
Session ID: 0 Serial number: 0

2018-10-30 06:52:55.106 :CLSDYNAM:11: [ ora.asm]{0:0:152} [start] InstAgent::startInstance 050 errcode 3113
2018-10-30 06:52:55.107 :CLSDYNAM:11: [ ora.asm]{0:0:152} [start] InstAgent::startInstance 250 untrapped error
2018-10-30 06:52:55.108 :CLSDYNAM:11: [ ora.asm]{0:0:152} [start] InstAgent::startInstance 350 errcode 3113
2018-10-30 06:52:55.108 : USRTHRD:11: [ INFO] {0:0:152} InstConnection:~InstConnection destructor this 106c2a010
2018-10-30 06:52:55.109 :CLSDYNAM:11: [ ora.asm]{0:0:152} [start] InstAgent::startInstance 380 throw excp what:ORA-03113: end-of-file on communication channel
Process ID: 0
Session ID: 0 Serial number: 0

2018-10-30 06:52:55.116 : AGFW:12: [ INFO] {0:0:152} Agent sending reply for: RESOURCE_START[ora.asm 1 1] ID 4098:586
2018-10-30 06:52:55.120 :CLSDYNAM:11: [ ora.asm]{0:0:152} [start] (:CLSN00107:) clsn_agent::start }
2018-10-30 06:52:55.121 : AGFW:11: [ INFO] {0:0:152} Command: start for resource: ora.asm 1 1 completed with status: FAIL

 <ORACLE BASE>/diag/asm/+asm/+ASM1/trace/alert_+ASM1.log:


2018-10-30T06:52:25.622471+00:00
NOTE: Cluster is in Rolling Migration from 11.2.0.4.0 to 19.0.0.0.0
2018-10-30T06:52:30.382177+00:00
Exception [type: SIGSEGV, Address not mapped to object] [ADDR:0xFFFFFFFF7A1F0000] [PC:0xFFFFFFFF7D3817AC, memcpy%sun4v-hwcap3()+184] [flags: 0x0, count: 1]
ERROR: Unable to normalize symbol name for the following short stack (at offset 247):
dbgexPopulateIncCtx()+476<-dbgexProcessError()+204<-dbgePostErrorKGE()+2180<-dbkePostKGE_kgsf()+36<-kgeade()+980<-kgerelv()+140<-kgerec6()+76<-ssexhd()+4460<-sslsshandler()+528<-__sighndlr()+12<-call_user_handler()+868<-sigacthandler()+92<-memcpy
Errors in file /u01/app/192base/diag/asm/+asm/+ASM1/trace/+ASM1_lmon_20412.trc (incident=113):
ORA-07445: exception encountered: core dump [memcpy%sun4v-hwcap3()+184] [SIGSEGV] [ADDR:0xFFFFFFFF7A1F0000] [PC:0xFFFFFFFF7D3817AC] [Address not mapped to object] []
Incident details in: <oracle_base>/diag/asm/+asm/+ASM1/incident/incdir_113/+ASM1_lmon_20412_i113.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
2018-10-30T06:52:40.096568+00:00
Exception [type: SIGBUS, Invalid address alignment] [ADDR:0x30000000B] [PC:0xFFFFFFFF1ECA8AD8, skgxprqhcomplete()+24] [flags: 0x0, count: 2]
Errors in file <oracle_base>/diag/asm/+asm/+ASM1/trace/+ASM1_lmon_20412.trc (incident=114):
ORA-07445: exception encountered: core dump [skgxprqhcomplete()+24] [SIGBUS] [ADDR:0x30000000B] [PC:0xFFFFFFFF1ECA8AD8] [Invalid address alignment] []
ORA-07445: exception encountered: core dump [memcpy%sun4v-hwcap3()+184] [SIGSEGV] [ADDR:0xFFFFFFFF7A1F0000] [PC:0xFFFFFFFF7D3817AC] [Address not mapped to object] []
Incident details in: <oracle_base>/diag/asm/+asm/+ASM1/incident/incdir_114/+ASM1_lmon_20412_i114.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
...
2018-10-30T06:52:56.083956+00:00
Instance terminated by PMON, pid = 20384
Instance terminated by USER, pid = 20954

<ORACLE BASE>/diag/asm/+asm/+ASM1/trace/+ASM1_lmon_20412.trc:

*** 2018-10-30T06:52:30.381096+00:00
Exception [type: SIGSEGV, Address not mapped to object] [ADDR:0xFFFFFFFF7A1F0000] [PC:0xFFFFFFFF7D3817AC, memcpy%sun4v-hwcap3()+184] [flags: 0x0, count: 1]
2018-10-30T06:52:30.430936+00:00
Incident 113 created, dump file: <oracle_base>/asm/+asm/+ASM1/incident/incdir_113/+ASM1_lmon_20412_i113.trc
ORA-07445: exception encountered: core dump [memcpy%sun4v-hwcap3()+184] [SIGSEGV] [ADDR:0xFFFFFFFF7A1F0000] [PC:0xFFFFFFFF7D3817AC] [Address not mapped to object] []

ssexhd: crashing the process...

*** 2018-10-30T06:52:40.095770+00:00
Exception [type: SIGBUS, Invalid address alignment] [ADDR:0x30000000B] [PC:0xFFFFFFFF1ECA8AD8, skgxprqhcomplete()+24] [flags: 0x0, count: 2]
2018-10-30T06:52:40.127905+00:00
Incident 114 created, dump file: <oracle_base>/asm/+asm/+ASM1/incident/incdir_114/+ASM1_lmon_20412_i114.trc
ORA-07445: exception encountered: core dump [skgxprqhcomplete()+24] [SIGBUS] [ADDR:0x30000000B] [PC:0xFFFFFFFF1ECA8AD8] [Invalid address alignment] []
ORA-07445: exception encountered: core dump [memcpy%sun4v-hwcap3()+184] [SIGSEGV] [ADDR:0xFFFFFFFF7A1F0000] [PC:0xFFFFFFFF7D3817AC] [Address not mapped to object] []

<ORACLE BASE>/diag/asm/+asm/+ASM1/incident/+ASM1_lmon_20412_i113.trc:

----- Incident Context Dump -----
Address: 0xffffffff7c600968
Incident ID: 113
Problem Key: ORA 7445 [memcpy%sun4v-hwcap3]
Error: ORA-7445 [memcpy%sun4v-hwcap3] [SIGSEGV] [ADDR:0xFFFFFFFF7A1F0000]
[PC:0xFFFFFFFF7D3817AC] [Address not mapped to object] [] [] [] [] [] [] []
[00]: dbgexPopulateIncCtx [diag_dde]
[01]: dbgexExplicitEndInc [diag_dde]
[02]: dbgeEndDDEInvocationImpl [diag_dde]
[03]: ssexhd []
[04]: sslsshandler []
[05]: __sighndlr []
[06]: call_user_handler []
[07]: sigacthandler []
[08]: memcpy%sun4v-hwcap3 []<-- Signaling
[09]: kjxgm_getIPcssdata [CGS]
[10]: kjxgm_pngreq_prepare [CGS]
[11]: kjxgm_pngreq_SubmitMap [CGS]
[12]: kjxgm_osnetcheck [CGS]
[13]: kjxgmpoll [CGS]
[14]: kjxggpoll [CGS]
[15]: kjfmact [LMHB]
[16]: kjfcln []
[17]: ksbrdp [background_proc]

 

<ORACLE BASE>/diag/asm/+asm/+ASM1/incident/+ASM1_lmon_20412_i114.trc:

----- Incident Context Dump -----
Address: 0xffffffff7c600968
Incident ID: 114
Problem Key: ORA 7445 [skgxprqhcomplete]
Error: ORA-7445 [skgxprqhcomplete] [SIGBUS] [ADDR:0x30000000B]
[PC:0xFFFFFFFF1ECA8AD8] [Invalid address alignment] [] [] [] [] [] [] []
[00]: dbgexPopulateIncCtx [diag_dde]
[01]: dbgexExplicitEndInc [diag_dde]
[02]: dbgeEndDDEInvocationImpl [diag_dde]
[03]: ssexhd []
[04]: sslsshandler []
[05]: __sighndlr []
[06]: call_user_handler []
[07]: sigacthandler []
[08]: skgxprqhcomplete []<-- Signaling
[09]: skgxpipdel []
[10]: skgxpicdel []
[11]: skgxpcdel []
[12]: ksxp_crash_reset [VOS]
[13]: ksdbgcra [VOS]
[14]: ssexhd []
[15]: sslsshandler []
[16]: __sighndlr []
[17]: call_user_handler []
[18]: sigacthandler []
[19]: memcpy%sun4v-hwcap3 []
[20]: kjxgm_getIPcssdata [CGS]
[21]: kjxgm_pngreq_prepare [CGS]
[22]: kjxgm_pngreq_SubmitMap [CGS]
[23]: kjxgm_osnetcheck [CGS]
[24]: kjxgmpoll [CGS]
[25]: kjxggpoll [CGS]
[26]: kjfmact [LMHB]
[27]: kjfcln []
[28]: ksbrdp [background_proc]

 

Cause

To view full details, sign in with your My Oracle Support account.

Don't have a My Oracle Support account? Click to get started!


In this Document
Symptoms
Cause
Solution
References

My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.