Database Crashes With ORA-07445 [oss_desparsify] Errors While Trying An Upgra...

刘泽宇

SYMPTOMS
19.1.0.0.0 Oracle Home.
Oracle Solaris on SPARC (64-bit) ~ Super Cluster rack.

Trying to upgrade snapshot clone database from 12c to 19c and when it reaches OPEN MIGRATE mode, immediately database is hitting below ORA 7445 ::

NOTE: Startup the database from original 12c Oracle Home has no issue.

.....
2020-05-11T16:11:31.359797-05:00
ALTER DATABASE OPEN MIGRATE            <<<<<<<<<<<<<<<<<<<<<<<<<
TMI: adbdrv open database BEGIN 2020-05-11 16:11:31.359846
2020-05-11T16:11:31.509060-05:00
NOTE: ASMB mounting group 5 (SPARSE)
NOTE: Assigned CGID 0x10032 for group 5
NOTE: ASMB process initiating disk discovery for grp 5 (reqid:0)
2020-05-11T16:11:32.246029-05:00
NOTE: Assigning number (5,0) to disk (o/192.168.79.50;192.168.79.51/SPARSE_CD_07_bs01auceladm12)
..
..
Starting background process ARC4
2020-05-11T16:11:33.824656-05:00
Exception [type: SIGSEGV, Address not mapped to object] [ADDR:0x10] [PC:0xFFFFFFFEFABEEDB8, oss_desparsify()+88] [flags: 0x0, count: 1]
Errors in file /app/oracle/diag/rdbms/testdb/testdb1/trace/testdb1_ora_57189.trc (incident=36681):
ORA-07445: exception encountered: core dump [oss_desparsify()+88] [SIGSEGV] [ADDR:0x10] [PC:0xFFFFFFFEFABEEDB8] [Address not mapped to object] []       <<<<<<<<<<<<<<<<<<<<<<<<<
2020-05-11T16:11:33.838863-05:00
ARC4 started with pid=97, OS id=58369
Starting background process ARC5
Incident details in: /app/oracle/diag/rdbms/testdb/testdb1/incident/incdir_36681/testdb1_ora_57189_i36681.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
2020-05-11T16:11:33.896472-05:00
ARC5 started with pid=98, OS id=58399
Starting background process ARC6
..
..
..
Instance terminated by PMON, pid = 51987




Other findings ::

1. ----- Current SQL Statement for this session (sql_id=aps3qh1nhzkjx) -----
select line#, sql_text from bootstrap$ where obj# not in (:1, :2)

2. Main Stack:

  oss_desparsify ksfdss_checkdata ksfdss_read ksfdread kcfrbd1 kcbz_table_scan_read kcbzibmlt kcbzib kcbgtcr ktrget2 kdst_fetch0

3. Snapshot clone databases are resided within SPARSE ASM disk groups.


CHANGES
Snapshot clone database upgrade from 12c to 19c release.

CAUSE
INTERNAL BUG 31386394 - DATABASE UPGRADE ON SPARSE DISKGROUP FAILS WITH ORA-7445 [OSS_DESPARSIFY]


SOLUTION
WORKAROUND ::

1. Update the database parameter file (SPfile) to be attempted for upgrade to 19c release which has the ORA 7445 oss_desparsify error, and set the following and try starting the database again into upgrade mode.

  event="libcell.cellclnt_sparse_mode sparse_mode=2"


2. If the above does not work, then we need more tracing with below event, attempt the upgrade to 19c again.

  event="10298 trace name context forever, level 5"


PERMANENT SOLUTION (RECOMMENDED) ::

Download and apply the patch 31386394 on top of the new Oracle Home 19c.