从11.2.0.4升级到19.2出现报错ORA-07445 [SKGXPRQHCOMPLETE()+24]

刘泽宇

SYMPTOMS

The rootupgrade.sh is failed on the first node while upgrading the GI from 11204 to 19.2, ora.asm failed to start

<ORACLE BASE>/diag/crs/<HOSTNAME>/crs/trace/alert log:

2018-10-30 06:50:56.428 [OHASD(18419)]CRS-8500: Oracle Clusterware OHASD process is starting with operating system process ID 18419
2018-10-30 06:50:56.659 [OHASD(18419)]CRS-0714: Oracle Clusterware Release 19.0.0.0.0.
2018-10-30 06:50:56.696 [OHASD(18419)]CRS-2112: The OLR service started on node slc06eun.
2018-10-30 06:50:57.669 [OHASD(18419)]CRS-1301: Oracle High Availability Service started on node slc06eun
...
2018-10-30 06:52:55.114 [ORAAGENT(18647)]CRS-5017: The resource action "ora.asm start" encountered the following error:
2018-10-30 06:52:55.114+ORA-03113: end-of-file on communication channel
Process ID: 0
Session ID: 0 Serial number: 0
. For details refer to "(:CLSN00107:)" in "<oracle_base>/crs/<nodename>/crs/trace/ohasd_oraagent_crsusr.trc".
2018-10-30 08:01:34.657 [CRSCTL(1783)]CRS-1013: The OCR location in an ASM disk group is inaccessible. Details in "<oracle_base>/crs/<nodename>//crs/trace/crsctl_1783.trc.

<ORACLE BASE>/crsdata/<nodename>/crsconfig/rootcrs_<nodename>-10-30_06-30-22AM.log:

2018-10-30 06:52:00: Executing cmd: /u01/app/19.2/grid/bin/crsctl start resource ora.asm -init
2018-10-30 06:52:57: Command output:
> CRS-2672: Attempting to start 'ora.asm' on 'node1'
> CRS-5017: The resource action "ora.asm start" encountered the following error:
> ORA-03113: end-of-file on communication channel
> Process ID: 0
> Session ID: 0 Serial number: 0
> . For details refer to "(:CLSN00107:)" in "<oracle_base>/diag/crs/slc06eun/crs/trace/ohasd_oraagent_crsusr.trc".
> CRS-2674: Start of 'ora.asm' on 'slc06eun' failed
> CRS-2679: Attempting to clean 'ora.asm' on 'node1'
> CRS-2681: Clean of 'ora.asm' on 'slc06eun' succeeded
> CRS-4000: Command Start failed, or completed with errors.
.....
2018-10-30 06:53:49: Executing cmd: /u01/app/19.2/grid/bin/clsecho -p has -f clsrsc -m 247
2018-10-30 06:53:49: Executing cmd: /u01/app/19.2/grid/bin/clsecho -p has -f clsrsc -m 247
2018-10-30 06:53:50: Command output:
> CLSRSC-247: Failed to start ASM
>End Command output
2018-10-30 06:53:50: CLSRSC-247: Failed to start ASM
2018-10-30 06:53:50: ###### Begin DIE Stack Trace ######
2018-10-30 06:53:50: Package File Line Calling
2018-10-30 06:53:50: --------------- -------------------- ---- ----------
2018-10-30 06:53:50: 1: main rootcrs.pl 355 crsutils::dietrap
2018-10-30 06:53:50: 2: crsupgrade crsupgrade.pm 1634 main::__ANON__
2018-10-30 06:53:50: 3: crsupgrade crsupgrade.pm 1582 crsupgrade::start_asm_resource
2018-10-30 06:53:50: 4: crsupgrade crsupgrade.pm 1490 crsupgrade::startStackFor112Upg
2018-10-30 06:53:50: 5: crsupgrade crsupgrade.pm 996 crsupgrade::upgrade_cluster
2018-10-30 06:53:50: 6: crsupgrade crsupgrade.pm 579 crsupgrade::CRSUpgrade
2018-10-30 06:53:50: 7: main rootcrs.pl 364 crsupgrade::new
2018-10-30 06:53:50: ####### End DIE Stack Trace #######

<ORACLE BASE>/diag/crs/<HOSTNAME>/crs/trace/ohasd_oraagent_crsusr.trc:

2018-10-30 06:52:55.103 :CLSDYNAM:11: [ ora.asm]{0:0:152} [start] ORA-03113: end-of-file on communication channel
Process ID: 0
Session ID: 0 Serial number: 0
2018-10-30 06:52:55.106 :CLSDYNAM:11: [ ora.asm]{0:0:152} [start] InstAgent::startInstance 050 errcode 3113
2018-10-30 06:52:55.107 :CLSDYNAM:11: [ ora.asm]{0:0:152} [start] InstAgent::startInstance 250 untrapped error
2018-10-30 06:52:55.108 :CLSDYNAM:11: [ ora.asm]{0:0:152} [start] InstAgent::startInstance 350 errcode 3113
2018-10-30 06:52:55.108 : USRTHRD:11: [ INFO] {0:0:152} InstConnection:~InstConnection destructor this 106c2a010
2018-10-30 06:52:55.109 :CLSDYNAM:11: [ ora.asm]{0:0:152} [start] InstAgent::startInstance 380 throw excp what:ORA-03113: end-of-file on communication channel
Process ID: 0
Session ID: 0 Serial number: 0
2018-10-30 06:52:55.116 : AGFW:12: [ INFO] {0:0:152} Agent sending reply for: RESOURCE_START[ora.asm 1 1] ID 4098:586
2018-10-30 06:52:55.120 :CLSDYNAM:11: [ ora.asm]{0:0:152} [start] (:CLSN00107:) clsn_agent::start }
2018-10-30 06:52:55.121 : AGFW:11: [ INFO] {0:0:152} Command: start for resource: ora.asm 1 1 completed with status: FAIL

<ORACLE BASE>/diag/asm/+asm/+ASM1/trace/alert_+ASM1.log:

2018-10-30T06:52:25.622471+00:00
NOTE: Cluster is in Rolling Migration from 11.2.0.4.0 to 19.0.0.0.0
2018-10-30T06:52:30.382177+00:00
Exception [type: SIGSEGV, Address not mapped to object] [ADDR:0xFFFFFFFF7A1F0000] [PC:0xFFFFFFFF7D3817AC, memcpy%sun4v-hwcap3()+184] [flags: 0x0, count: 1]
ERROR: Unable to normalize symbol name for the following short stack (at offset 247):
dbgexPopulateIncCtx()+476<-dbgexProcessError()+204<-dbgePostErrorKGE()+2180<-dbkePostKGE_kgsf()+36<-kgeade()+980<-kgerelv()+140<-kgerec6()+76<-ssexhd()+4460<-sslsshandler()+528<-__sighndlr()+12<-call_user_handler()+868<-sigacthandler()+92<-memcpy
Errors in file /u01/app/192base/diag/asm/+asm/+ASM1/trace/+ASM1_lmon_20412.trc (incident=113):
ORA-07445: exception encountered: core dump [memcpy%sun4v-hwcap3()+184] [SIGSEGV] [ADDR:0xFFFFFFFF7A1F0000] [PC:0xFFFFFFFF7D3817AC] [Address not mapped to object] []
Incident details in: <oracle_base>/diag/asm/+asm/+ASM1/incident/incdir_113/+ASM1_lmon_20412_i113.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
2018-10-30T06:52:40.096568+00:00
Exception [type: SIGBUS, Invalid address alignment] [ADDR:0x30000000B] [PC:0xFFFFFFFF1ECA8AD8, skgxprqhcomplete()+24] [flags: 0x0, count: 2]
Errors in file <oracle_base>/diag/asm/+asm/+ASM1/trace/+ASM1_lmon_20412.trc (incident=114):
ORA-07445: exception encountered: core dump [skgxprqhcomplete()+24] [SIGBUS] [ADDR:0x30000000B] [PC:0xFFFFFFFF1ECA8AD8] [Invalid address alignment] []
ORA-07445: exception encountered: core dump [memcpy%sun4v-hwcap3()+184] [SIGSEGV] [ADDR:0xFFFFFFFF7A1F0000] [PC:0xFFFFFFFF7D3817AC] [Address not mapped to object] []
Incident details in: <oracle_base>/diag/asm/+asm/+ASM1/incident/incdir_114/+ASM1_lmon_20412_i114.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
...
2018-10-30T06:52:56.083956+00:00
Instance terminated by PMON, pid = 20384
Instance terminated by USER, pid = 20954

<ORACLE BASE>/diag/asm/+asm/+ASM1/trace/+ASM1_lmon_20412.trc:

*** 2018-10-30T06:52:30.381096+00:00
Exception [type: SIGSEGV, Address not mapped to object] [ADDR:0xFFFFFFFF7A1F0000] [PC:0xFFFFFFFF7D3817AC, memcpy%sun4v-hwcap3()+184] [flags: 0x0, count: 1]
2018-10-30T06:52:30.430936+00:00
Incident 113 created, dump file: <oracle_base>/asm/+asm/+ASM1/incident/incdir_113/+ASM1_lmon_20412_i113.trc
ORA-07445: exception encountered: core dump [memcpy%sun4v-hwcap3()+184] [SIGSEGV] [ADDR:0xFFFFFFFF7A1F0000] [PC:0xFFFFFFFF7D3817AC] [Address not mapped to object] []
ssexhd: crashing the process...
*** 2018-10-30T06:52:40.095770+00:00
Exception [type: SIGBUS, Invalid address alignment] [ADDR:0x30000000B] [PC:0xFFFFFFFF1ECA8AD8, skgxprqhcomplete()+24] [flags: 0x0, count: 2]
2018-10-30T06:52:40.127905+00:00
Incident 114 created, dump file: <oracle_base>/asm/+asm/+ASM1/incident/incdir_114/+ASM1_lmon_20412_i114.trc
ORA-07445: exception encountered: core dump [skgxprqhcomplete()+24] [SIGBUS] [ADDR:0x30000000B] [PC:0xFFFFFFFF1ECA8AD8] [Invalid address alignment] []
ORA-07445: exception encountered: core dump [memcpy%sun4v-hwcap3()+184] [SIGSEGV] [ADDR:0xFFFFFFFF7A1F0000] [PC:0xFFFFFFFF7D3817AC] [Address not mapped to object] []

<ORACLE BASE>/diag/asm/+asm/+ASM1/incident/+ASM1_lmon_20412_i113.trc:

----- Incident Context Dump -----
Address: 0xffffffff7c600968
Incident ID: 113
Problem Key: ORA 7445 [memcpy%sun4v-hwcap3]
Error: ORA-7445 [memcpy%sun4v-hwcap3] [SIGSEGV] [ADDR:0xFFFFFFFF7A1F0000]
[PC:0xFFFFFFFF7D3817AC] [Address not mapped to object] [] [] [] [] [] [] []
[00]: dbgexPopulateIncCtx [diag_dde]
[01]: dbgexExplicitEndInc [diag_dde]
[02]: dbgeEndDDEInvocationImpl [diag_dde]
[03]: ssexhd []
[04]: sslsshandler []
[05]: __sighndlr []
[06]: call_user_handler []
[07]: sigacthandler []
[08]: memcpy%sun4v-hwcap3 []<-- Signaling
[09]: kjxgm_getIPcssdata [CGS]
[10]: kjxgm_pngreq_prepare [CGS]
[11]: kjxgm_pngreq_SubmitMap [CGS]
[12]: kjxgm_osnetcheck [CGS]
[13]: kjxgmpoll [CGS]
[14]: kjxggpoll [CGS]
[15]: kjfmact [LMHB]
[16]: kjfcln []
[17]: ksbrdp [background_proc]

<ORACLE BASE>/diag/asm/+asm/+ASM1/incident/+ASM1_lmon_20412_i114.trc:

----- Incident Context Dump -----
Address: 0xffffffff7c600968
Incident ID: 114
Problem Key: ORA 7445 [skgxprqhcomplete]
Error: ORA-7445 [skgxprqhcomplete] [SIGBUS] [ADDR:0x30000000B]
[PC:0xFFFFFFFF1ECA8AD8] [Invalid address alignment] [] [] [] [] [] [] []
[00]: dbgexPopulateIncCtx [diag_dde]
[01]: dbgexExplicitEndInc [diag_dde]
[02]: dbgeEndDDEInvocationImpl [diag_dde]
[03]: ssexhd []
[04]: sslsshandler []
[05]: __sighndlr []
[06]: call_user_handler []
[07]: sigacthandler []
[08]: skgxprqhcomplete []<-- Signaling
[09]: skgxpipdel []
[10]: skgxpicdel []
[11]: skgxpcdel []
[12]: ksxp_crash_reset [VOS]
[13]: ksdbgcra [VOS]
[14]: ssexhd []
[15]: sslsshandler []
[16]: __sighndlr []
[17]: call_user_handler []
[18]: sigacthandler []
[19]: memcpy%sun4v-hwcap3 []
[20]: kjxgm_getIPcssdata [CGS]
[21]: kjxgm_pngreq_prepare [CGS]
[22]: kjxgm_pngreq_SubmitMap [CGS]
[23]: kjxgm_osnetcheck [CGS]
[24]: kjxgmpoll [CGS]
[25]: kjxggpoll [CGS]
[26]: kjfmact [LMHB]
[27]: kjfcln []
[28]: ksbrdp [background_proc]

CAUSE

Due to Bug 28858720

SOLUTION

Bug 28858720 fixed in 20.1, request/apply patch 28858720 for your platform/version