ORA-01115: IO error reading block from file 70 (block # 2037319)

mahan

检查数据库备库运行情况，发现备库日志没有同步应用，开启同步也失败，报错如下
Wed Feb 08 12:57:59 2023
alter database recover managed standby database disconnect from session
Attempt to start background Managed Standby Recovery process (orcldg)
Wed Feb 08 12:57:59 2023
MRP0 started with pid=22, OS id=2769
MRP0: Background Managed Standby Recovery process started (orcldg)
started logmerger process
Wed Feb 08 12:58:04 2023
Managed Standby Recovery not using Real Time Apply
Parallel Media Recovery started with 8 slaves
Waiting for all non-current ORLs to be archived...
All non-current ORLs have been archived.
Media Recovery Log +DATA/orcldg/archivelog/2023_01_28/thread_1_seq_92469.2002.1127301011
Completed: alter database recover managed standby database disconnect from session
Media Recovery Log +DATA/orcldg/archivelog/2023_01_28/thread_2_seq_92015.2643.1127301011
Wed Feb 08 12:58:06 2023
WARNING: Read Failed. group:1 disk:1 AU:618887 offset:581632 size:8192
WARNING: failed to read mirror side 1 of virtual extent 15916 logical extent 0 of file 4007 in group [1.4808517] from disk DATA02  allocation unit 618887 reason error; if possible, will try another mirror side
Slave exiting with ORA-1115 exception
Errors in file /u01/app/oracle/diag/rdbms/orcldg/orcldg/trace/orcldg_pr02_2777.trc:
ORA-01115: IO error reading block from file 70 (block # 2037319)
ORA-01110: data file 70: '+DATA/orcldg/datafile/zl9baseitem.4007.1077799315'
ORA-15081: failed to submit an I/O operation to a disk
ORA-15081: failed to submit an I/O operation to a disk
Recovery Slave PR02 previously exited with exception 1115
Errors with log +DATA/orcldg/archivelog/2023_01_28/thread_2_seq_92015.2643.1127301011
MRP0: Background Media Recovery terminated with error 448
Errors in file /u01/app/oracle/diag/rdbms/orcldg/orcldg/trace/orcldg_pr00_2771.trc:
ORA-00448: normal completion of background process
Recovery interrupted!
Recovered data files to a consistent state at change 14466462117210
MRP0: Background Media Recovery process shutdown (orcldg)



发现IO错误，就去检查是否磁盘问题

[root@hisdg ~]# ls -ltr /dev/oracleasm/disks/
total 0
brw-rw---- 1 grid asmadmin 8, 33 Feb  8 13:36 DATA02
brw-rw---- 1 grid asmadmin 8, 17 Feb  8 13:36 DATA01
[root@hisdg ~]# oracleasm querydisk -d -p data01
Disk "DATA01" is a valid ASM disk on device [8,17]
/dev/sdb1: LABEL="DATA01" TYPE="oracleasm"
[root@hisdg ~]# oracleasm querydisk -d -p data02
Disk "DATA02" is a valid ASM disk on device [8,33]
/dev/sdc1: LABEL="DATA02" TYPE="oracleasm"



发现磁盘还在，就去检查系统日志文件

[root@hisdg ~]# journalctl -p err
-- Logs begin at Wed 2023-02-08 12:47:32 CST, end at Wed 2023-02-08 13:33:16 CST. --
Feb 08 12:47:32 localhost.localdomain kernel: tsc: Fast TSC calibration failed
Feb 08 12:47:32 localhost.localdomain kernel: [Firmware Bug]: the BIOS has corrupted hw-PMU resources (MSR 186 is 43003c)
Feb 08 12:47:32 localhost.localdomain kernel: ERST: Failed to get Error Log Address Range.
Feb 08 12:47:32 localhost.localdomain kernel: qla2xxx [0000:22:00.0]-0034:2: MSI-X: Unsupported ISP 2432 SSVID/SSDID (0x103C,0x7040).
Feb 08 12:47:37 hisdg kernel: ip_local_port_range: prefer different parity for start/end values.
Feb 08 12:47:39 hisdg kernel: kvm: disabled by bios
Feb 08 12:47:39 hisdg kernel: kvm: disabled by bios
Feb 08 12:47:39 hisdg kernel: kvm: disabled by bios
Feb 08 12:47:39 hisdg kernel: kvm: disabled by bios
Feb 08 12:47:42 hisdg systemd[1]: Failed to listen on RPCbind Server Activation Socket.
Feb 08 12:47:49 hisdg logger[1269]: Oracle HA daemon is enabled for autostart.
Feb 08 12:47:50 hisdg postfix/sendmail[1427]: fatal: parameter inet_interfaces: no local interface found for ::1
Feb 08 12:47:50 hisdg postfix[1432]: fatal: parameter inet_interfaces: no local interface found for ::1
Feb 08 12:47:50 hisdg logger[1500]: exec /u01/app/grid/product/11.2.0/grid/perl/bin/perl -I/u01/app/grid/product/11.2.0/grid/perl/lib /u01/app/grid/product/11.2.0/grid/bin/crswrapexece.pl /u01/app/grid/pr
Feb 08 12:47:51 hisdg systemd[1]: Failed to start Postfix Mail Transport Agent.
Feb 08 12:47:52 hisdg spice-vdagent[1637]: Cannot access vdagent virtio channel /dev/virtio-ports/com.redhat.spice.0
Feb 08 12:50:42 hisdg kernel: blk_update_request: critical medium error, dev sdc, sector 1267481775
Feb 08 12:53:35 hisdg kernel: blk_update_request: critical medium error, dev sdc, sector 1267481775
Feb 08 12:58:06 hisdg kernel: blk_update_request: critical medium error, dev sdc, sector 1267481775


最终确定备库的磁盘坏道问题，导致无法正常写