OPATCHAUTO-68061 编制引擎失败

denglj

数据库：19.3
系统：CentOS 7.9
数据库升级过程中遭遇OPATCHAUTO-68061报错，如下：
[root@hisdb2 OPatch]# ./opatchauto resume

OPatchauto session is initiated at Sat Mar 19 23:29:20 2022
Session log file is /u01/app/19.3.0/grid/cfgtoollogs/opatchauto/opatchauto2022-03-19_11-29-20PM.log
Resuming existing session with id 4D22

Start applying binary patch on home /u01/app/oracle/product/19.3.0/db
Binary patch applied successfully on home /u01/app/oracle/product/19.3.0/db

Performing postpatch operation on home /u01/app/oracle/product/19.3.0/db
Postpatch operation completed successfully on home /u01/app/oracle/product/19.3.0/db

Start applying binary patch on home /u01/app/19.3.0/grid
Failed while applying binary patches on home /u01/app/19.3.0/grid

Execution of [OPatchAutoBinaryAction] patch action failed, check log for more details. Failures:
Patch Target : hisdb2->/u01/app/19.3.0/grid Type[crs]
Details: [
---------------------------Patching Failed---------------------------------
Command execution failed during patching in home: /u01/app/19.3.0/grid, host: hisdb2.
Command failed:  /u01/app/19.3.0/grid/OPatch/opatchauto  apply /soft/31750108 -oh /u01/app/19.3.0/grid -target_type cluster -binary -invPtrLoc /u01/app/19.3.0/grid/oraInst.loc -jre /u01/app/19.3.0/grid/OPatch/jre -persistresult /u01/app/19.3.0/grid/opatchautocfg/db/sessioninfo/sessionresult_hisdb2_crs_1.ser -analyzedresult /u01/app/19.3.0/grid/opatchautocfg/db/sessioninfo/sessionresult_analyze_hisdb2_crs_1.ser
Command failure output:
==Following patches FAILED in apply:

Patch: /soft/31750108/31771877
Log: /u01/app/19.3.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2022-03-19_23-34-30PM_1.log
Reason: Failed during Patching: oracle.opatch.opatchsdk.OPatchException: ApplySession failed in system modification phase... 'ApplySession::apply failed: java.io.IOException: oracle.sysman.oui.patch.PatchException: java.io.FileNotFoundException: /u01/app/oraInventory/ContentsXML/oui-patch.xml (Permission denied)'

After fixing the cause of failure Run opatchauto resume
]
OPATCHAUTO-68061: The orchestration engine failed.
OPATCHAUTO-68061: The orchestration engine failed with return code 1
OPATCHAUTO-68061: Check the log for more details.
OPatchAuto failed.

OPatchauto session completed at Sat Mar 19 23:39:15 2022
Time taken to complete the session 9 minutes, 56 seconds

opatchauto failed with error code 42

故障解决solution：
观察节点1与节点2 /oracle/oraInventory/ContentsXML/oui-patch.xml权限差异，并将节点二修改成与节点1一致。
[root@hisdb1 OPatch]# ll /u01/app/oraInventory/ContentsXML/oui-patch.xml
-rw-rw---- 1 grid oinstall 174 Mar 19 22:49 /u01/app/oraInventory/ContentsXML/oui-patch.xml
[root@hisdb2 OPatch]# ll /u01/app/oraInventory/ContentsXML/oui-patch.xml
-rw-r--r-- 1 oracle oinstall 174 Mar 19 23:34 /u01/app/oraInventory/ContentsXML/oui-patch.xml
[root@hisdb2 OPatch]# chown -R grid:oinstall /u01/app/oraInventory/ContentsXML/oui-patch.xml
[root@hisdb2 OPatch]# chmod g+w /u01/app/oraInventory/ContentsXML/oui-patch.xml
[root@hisdb2 OPatch]# ll /u01/app/oraInventory/ContentsXML/
total 20
-rw-rw---- 1 grid oinstall 300 Mar 19 23:34 comps.xml
-rw-rw---- 1 grid oinstall 555 Mar 14 23:14 inventory.xml
-rw-rw---- 1 grid oinstall 292 Mar 19 23:34 libs.xml
-rw-rw-r-- 1 grid oinstall 174 Mar 19 23:34 oui-patch.xml
-rw-r--r-- 1 grid oinstall 174 Mar 19 23:34 oui-patch.xml.back


./opatchauto resume 继续执行
[root@hisdb2 OPatch]# ./opatchauto resume                              

OPatchauto session is initiated at Sun Mar 20 08:45:24 2022
Session log file is /u01/app/19.3.0/grid/cfgtoollogs/opatchauto/opatchauto2022-03-20_08-45-24AM.log
Resuming existing session with id 4D22

Start applying binary patch on home /u01/app/19.3.0/grid
Failed while applying binary patches on home /u01/app/19.3.0/grid

Execution of [OPatchAutoBinaryAction] patch action failed, check log for more details. Failures:
Patch Target : hisdb2->/u01/app/19.3.0/grid Type[crs]
Details: [
---------------------------Patching Failed---------------------------------
Command execution failed during patching in home: /u01/app/19.3.0/grid, host: hisdb2.
Command failed:  /u01/app/19.3.0/grid/OPatch/opatchauto  apply /soft/31750108 -oh /u01/app/19.3.0/grid -target_type cluster -binary -invPtrLoc /u01/app/19.3.0/grid/oraInst.loc -jre /u01/app/19.3.0/grid/OPatch/jre -persistresult /u01/app/19.3.0/grid/opatchautocfg/db/sessioninfo/sessionresult_hisdb2_crs_1.ser -analyzedresult /u01/app/19.3.0/grid/opatchautocfg/db/sessioninfo/sessionresult_analyze_hisdb2_crs_1.ser
Command failure output:
==Following patches FAILED in apply:

Patch: /soft/31750108/31771877
Log: /u01/app/19.3.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2022-03-20_08-45-52AM_1.log
Reason: Failed during Analysis: CheckNApplyReport Failed, [ Prerequisite Status: FAILED, Prerequisite output:
The details are:
Unable to create patchObject
Possible causes are:
   ORACLE_HOME/inventory/oneoffs/31771877 is corrupted. PatchObject constructor: Input file "/u01/app/19.3.0/grid/inventory/oneoffs/31771877/etc/config/actions" or "/u01/app/19.3.0/grid/inventory/oneoffs/31771877/etc/config/inventory" does not exist.
]
After fixing the cause of failure Run opatchauto resume
]
OPATCHAUTO-68061: The orchestration engine failed.
OPATCHAUTO-68061: The orchestration engine failed with return code 1
OPATCHAUTO-68061: Check the log for more details.
OPatchAuto failed.

OPatchauto session completed at Sun Mar 20 08:45:57 2022
Time taken to complete the session 0 minute, 33 seconds

opatchauto failed with error code 42

依然报错
查看补丁信息：(也出现报错)
[grid@hisdb2:/home/grid]$ opatch lsinventory
Oracle Interim Patch Installer version 12.2.0.1.28
Copyright (c) 2022, Oracle Corporation.  All rights reserved.

Oracle Home       : /u01/app/19.3.0/grid
Central Inventory : /u01/app/oraInventory
   from           : /u01/app/19.3.0/grid/oraInst.loc
OPatch version    : 12.2.0.1.28
OUI version       : 12.2.0.7.0
Log file location : /u01/app/19.3.0/grid/cfgtoollogs/opatch/opatch2022-03-20_08-47-30AM_1.log

Lsinventory Output file location : /u01/app/19.3.0/grid/cfgtoollogs/opatch/lsinv/lsinventory2022-03-20_08-47-30AM.txt
--------------------------------------------------------------------------------

Inventory load failed... OPatch cannot load inventory for the given Oracle Home.

LsInventorySession failed: Unable to create patchObject
Possible causes are:
   ORACLE_HOME/inventory/oneoffs/31771877 is corrupted. PatchObject constructor: Input file "/u01/app/19.3.0/grid/inventory/oneoffs/31771877/etc/config/actions" or "/u01/app/19.3.0/grid/inventory/oneoffs/31771877/etc/config/inventory" does not exist.

OPatch failed with error code 73

故障解决，根据MOS2286025.1 solution：
从节点1拷贝
[grid@hisdb1:/u01/app/19.3.0/grid/inventory/oneoffs]$ scp -r 31771877/ hisdb2:/u01/app/19.3.0/grid/inventory/oneoffs/
inventory.xml                                                                                                                                                100%  319KB  46.9MB/s   00:00   
actions.xml                                                                                                                                                  100% 2331KB  70.4MB/s   00:00   
在节点2再次resume
[root@hisdb2 OPatch]# ./opatchauto resume

OPatchauto session is initiated at Sun Mar 20 08:51:40 2022
Session log file is /u01/app/19.3.0/grid/cfgtoollogs/opatchauto/opatchauto2022-03-20_08-51-41AM.log
Resuming existing session with id 4D22

Start applying binary patch on home /u01/app/19.3.0/grid
Binary patch applied successfully on home /u01/app/19.3.0/grid

Checking shared status of home.....

Performing postpatch operations on CRS - starting CRS service on home /u01/app/19.3.0/grid
Postpatch operation log file location: /u01/app/grid/crsdata/hisdb2/crsconfig/crs_postpatch_apply_inplace_hisdb2_2022-03-20_08-55-32AM.log
CRS service started successfully on home /u01/app/19.3.0/grid

Preparing home /u01/app/oracle/product/19.3.0/db after database service restarted
No step execution required.........

Trying to apply SQL patch on home /u01/app/oracle/product/19.3.0/db
SQL patch applied successfully on home /u01/app/oracle/product/19.3.0/db

OPatchAuto successful.
--------------------------------Summary--------------------------------
Patching is completed successfully. Please find the summary as follows:

Host:hisdb2
RAC Home:/u01/app/oracle/product/19.3.0/db
Version:19.0.0.0.0
Summary:

==Following patches were SKIPPED:

Patch: /soft/31750108/31773437
Reason: This patch is not applicable to this specified target type - "rac_database"

Patch: /soft/31750108/31780966
Reason: This patch is not applicable to this specified target type - "rac_database"

==Following patches were SUCCESSFULLY applied:

Patch: /soft/31750108/31771877
Log: /u01/app/oracle/product/19.3.0/db/cfgtoollogs/opatchauto/core/opatch/opatch2022-03-19_23-29-38PM_1.log

Patch: /soft/31750108/31772784
Log: /u01/app/oracle/product/19.3.0/db/cfgtoollogs/opatchauto/core/opatch/opatch2022-03-19_23-29-38PM_1.log

Host:hisdb2
CRS Home:/u01/app/19.3.0/grid
Version:19.0.0.0.0
Summary:

==Following patches were SKIPPED:

Patch: /soft/31750108/31771877
Reason: This patch is not applicable to this specified target type - "cluster"

==Following patches were SUCCESSFULLY applied:

Patch: /soft/31750108/31772784
Log: /u01/app/19.3.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2022-03-20_08-52-08AM_1.log

Patch: /soft/31750108/31773437
Log: /u01/app/19.3.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2022-03-20_08-52-08AM_1.log

Patch: /soft/31750108/31780966
Log: /u01/app/19.3.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2022-03-20_08-52-08AM_1.log

OPatchauto session completed at Sun Mar 20 09:07:08 2022
Time taken to complete the session 15 minutes, 28 seconds
执行成功