DG环境恢复同步遇到报错ORA-00353ORA-00334以及ORA-00600[2619], [47745]

mahan

问题说明

   客户环境主库4节点RAC11.2.0.4，单实例DG环境，DG由于空间不足，导致同步中断，由于DG备库未应用的归档主库都再，本次恢复的方式，是开启dg mrp进程，自动同步追上主库。

  以下遇到的问题，都在此过程中，遇到的报错。

一 dg环境遇到

ORA-00353 ORA-00334  如果主库还存在归档日志，则可以mv ,rm删除备库有问题的归档，让dg产生gap,主库会再次像备库传输归档日志
另一种方法，就是如果主库无法自动像备库传输日志，可以从主库scp归档至备库dg，注册归档后，解决问题。
其次，本次通过主库，dg alert排除存在异常现象，也就是说再本次问题前，alert日志中并未出现日志块损坏现象！
ORA-354 ORA-353 Corrupt Redo Log Block Header against online redolog using EMC/Cisco SANTap Service (Doc ID 779711.1)
ORA-354 ORA-353 and ORA-312 on Redo Log Group members (Doc ID 332672.1)
复制代码
hu Nov 21 22:13:04 2019
Media Recovery Log /oradata/newora901dg/archivelog/NEWORA901DG/archivelog/2019_11_21/o1_mf_4_31416_gxf6v6p0_.arc
Incomplete read from log member '/oradata/newora901dg/archivelog/NEWORA901DG/archivelog/2019_11_21/o1_mf_1_46650_gxc023sj_.arc'.
Trying next member.
Errors in file /u01/app/oracle/diag/rdbms/newora901dg/ora901/trace/ora901_pr00_9153.trc  (incident=192300):
ORA-00353: log corruption near block 40 change 33381243833 time 11/19/2019 13:34:10
ORA-00334: archived log: '/oradata/newora901dg/archivelog/NEWORA901DG/archivelog/2019_11_21/o1_mf_1_46650_gxc023sj_.arc'
Incident details in: /u01/app/oracle/diag/rdbms/newora901dg/ora901/incident/incdir_192300/ora901_pr00_9153_i192300.trc
Errors with log /oradata/newora901dg/archivelog/NEWORA901DG/archivelog/2019_11_21/o1_mf_4_31416_gxf6v6p0_.arc
MRP0: Background Media Recovery terminated with error 354
Errors in file /u01/app/oracle/diag/rdbms/newora901dg/ora901/trace/ora901_pr00_9153.trc:
ORA-00354: corrupt redo log block header
ORA-00353: log corruption near block 40 change 33381243833 time 11/19/2019 13:34:10
ORA-00334: archived log: '/oradata/newora901dg/archivelog/NEWORA901DG/archivelog/2019_11_21/o1_mf_1_46650_gxc023sj_.arc'
Thu Nov 21 22:13:05 2019
Dumping diagnostic data in directory=[cdmp_20191121221305], requested by (instance=1, osid=9153 (PR00)), summary=[incident=192300].
Recovery interrupted!
Recovered data files to a consistent state at change 33381242204
Thu Nov 21 22:13:06 2019
MRP0: Background Media Recovery process shutdown (ora901)
Thu Nov 21 22:13:08 2019
Sweep [inc][192300]: completed
Sweep [inc][192299]: completed
Sweep [inc2][192300]: completed
Sweep [inc2][192299]: completed
MRP0: Background Media Recovery process shutdown (ora901)

操作：
dg环境
mv  有问题的归档 xxx.bak 后，dg备库启动Mrp进程
SQL> recover managed standby database disconnect from session;


Thu Nov 21 22:21:38 2019
ALTER DATABASE RECOVER  managed standby database disconnect from session  
Attempt to start background Managed Standby Recovery process (ora901)
Thu Nov 21 22:21:38 2019
MRP0 started with pid=37, OS id=9721
MRP0: Background Managed Standby Recovery process started (ora901)
started logmerger process
Thu Nov 21 22:21:43 2019
Managed Standby Recovery not using Real Time Apply
Parallel Media Recovery started with 64 slaves
Waiting for all non-current ORLs to be archived...
All non-current ORLs have been archived.
Media Recovery Log /oradata/newora901dg/archivelog/NEWORA901DG/archivelog/2019_11_21/o1_mf_2_47491_gxf6t3v7_.arc
Media Recovery Log /oradata/newora901dg/archivelog/NEWORA901DG/archivelog/2019_11_21/o1_mf_1_46650_gxc023sj_.arc
Error opening /oradata/newora901dg/archivelog/NEWORA901DG/archivelog/2019_11_21/o1_mf_1_46650_gxc023sj_.arc
Attempting refetch
Media Recovery Waiting for thread 1 sequence 46650
Fetching gap sequence in thread 1, gap sequence 46650-46650
Thu Nov 21 22:21:46 2019
RFS[23]: Allowing overwrite of partial archivelog for thread 1 sequence 46650
RFS[23]: Opened log for thread 1 sequence 46650 dbid -1930502573 branch 988229931
Archived Log entry 160947 added for thread 1 sequence 46650 rlc 988229931 ID 0x948808ac dest 2:
Completed: ALTER DATABASE RECOVER  managed standby database disconnect from session  
Thu Nov 21 22:21:56 2019
Media Recovery Log /oradata/newora901dg/archivelog/NEWORA901DG/archivelog/2019_11_21/o1_mf_1_46650_gxf7ctdd_.arc
Media Recovery Log /oradata/newora901dg/archivelog/NEWORA901DG/archivelog/2019_11_21/o1_mf_3_31565_gxbzzfxk_.arc
Media Recovery Log /oradata/newora901dg/archivelog/NEWORA901DG/archivelog/2019_11_21/o1_mf_4_31416_gxf6v6p0_.arc
Media Recovery Log /oradata/newora901dg/archivelog/NEWORA901DG/archivelog/2019_11_21/o1_mf_4_31417_gxbzvr69_.arc
Media Recovery Log /oradata/newora901dg/archivelog/NEWORA901DG/archivelog/2019_11_21/o1_mf_1_46651_gxc023sz_.arc

#备选方案，如果是10G环境，备库产生gap,但是主库很可能并不会再次将归档传输到备库，因此需要手工scp归档日志传输至备库。
SQL> select name,thread# from v$archived_log where sequence#=46650;
NAME
------------------------------------------------------------------------
+FRA/newora901/archivelog/2019_11_19/thread_1_seq_46650.4973.1024753747   
ASMCMD> cp thread_1_seq_46650.4973.1024753747 /home/grid/thread_1_seq_46650.arc
copying +FRA/newora901/archivelog/2019_11_19/thread_1_seq_46650.4973.1024753747 -> /home/grid/thread_1_seq_46650.arc
scp 传输归档到备库
SQL> ALTER DATABASE REGISTER LOGFILE '/zz/xxx.arc';

复制代码
二 DG同步遇到如下ORA 600报错，MOS并未找到相关文档

本次通过将报错的上一个归档日志，进行mv 移除后，再次开启mrp进程，dg产生gap,主库重传报错归档解决问题。

复制代码
Waiting for all non-current ORLs to be archived...
All non-current ORLs have been archived.
Media Recovery Log /oradata/newora901dg/archivelog/NEWORA901DG/archivelog/2019_11_21/o1_mf_2_47745_gxc193wj_.arc
Errors in file /u01/app/oracle/diag/rdbms/newora901dg/ora901/trace/ora901_pr00_17446.trc  (incident=208045):
ORA-00600: internal error code, arguments: [2619], [47745], [], [], [], [], [], [], [], [], [], []
Incident details in: /u01/app/oracle/diag/rdbms/newora901dg/ora901/incident/incdir_208045/ora901_pr00_17446_i208045.trc
Completed: ALTER DATABASE RECOVER  managed standby database disconnect from session  
Thu Nov 21 23:14:30 2019
Dumping diagnostic data in directory=[cdmp_20191121231430], requested by (instance=1, osid=17446 (PR00)),
summary=[incident=208045].
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Errors with log /oradata/newora901dg/archivelog/NEWORA901DG/archivelog/2019_11_21/o1_mf_2_47745_gxc193wj_.arc
MRP0: Background Media Recovery terminated with error 600
Errors in file /u01/app/oracle/diag/rdbms/newora901dg/ora901/trace/ora901_pr00_17446.trc:
ORA-00600: internal error code, arguments: [2619], [47745], [], [], [], [], [], [], [], [], [], []
Recovery interrupted!
Errors in file /u01/app/oracle/diag/rdbms/newora901dg/ora901/trace/ora901_pr00_17446.trc:
ORA-00600: internal error code, arguments: [2619], [47745], [], [], [], [], [], [], [], [], [], []
MRP0: Background Media Recovery process shutdown (ora901)
SQL> host mv /oradata/newora901dg/archivelog/NEWORA901DG/archivelog/2019_11_21/o1_mf_2_47745_gxc193wj_.arc
/oradata/newora901dg/archivelog/NEWORA901DG/archivelog/2019_11_21/o1_mf_2_47745_gxc193wj_.arc_20191121.bak