12c开始的flexasm 加白名单未加私网IP导致CRS不自动启动

郑全

问题：
     19C，RAC，加了白名单，重启机器后，发现CRS无法自动启动，
     检查集群报警文件：

...

2020-11-23 17:29:34.313 [OCTSSD(7726)]CRS-2401: The Cluster Time Synchronization Service started on host cquoda1.

2020-11-23 17:29:34.302 [OCSSD(7292)]CRS-1720: Cluster Synchronization Services daemon (CSSD) is ready for operation.

2020-11-23 17:29:35.767 [ORAROOTAGENT(5423)]CRS-5019: All OCR locations are on ASM disk groups [DATA], and none of these disk groups are mounted. Details are at "(:CLSN00140:)" in "/u01/app/grid/diag/crs/cquoda1/crs/trace/ohasd_orarootagent_root.trc".   

   

ohasd_orarootagent_root.trc:

2020-11-23 17:33:37.518 : USRTHRD:3502307072: [ INFO] {0:5:3} Thread:kgfoCheckMountExt Tasklet::doTask m_cv.timewait returned

2020-11-23 17:33:38.318 : USRTHRD:3502307072: [ INFO] {0:5:3} Thread:kgfoCheckMountExt Tasklet::doTask m_cv.timewait returned

2020-11-23 17:33:38.826 : CLSNS:2205349632: clsns_SetTraceLevel:trace level set to 1.

2020-11-23 17:33:38.847 : USRTHRD:2205349632: [ INFO] {0:5:3} [ora.storage] 9538 Error 4 querying length of attr ASM_DISCOVERY_ADDRESS

2020-11-23 17:33:38.850 : USRTHRD:2205349632: [ INFO] {0:5:3} [ora.storage] 9538 Error 4 querying length of attr ASM_STATIC_DISCOVERY_ADDRESS

2020-11-23 17:33:38.877 : USRTHRD:2205349632: [ INFO] {0:5:3} [ora.storage] 9256 Error 4 opening dom root in 0x7f2078085ca0

2020-11-23 17:33:39.118 : USRTHRD:3502307072: [ INFO] {0:5:3} Thread:kgfoCheckMountExt Tasklet::doTask m_cv.timewait returned

2020-11-23 17:33:39.915 : USRTHRD:2205349632: [ INFO] {0:5:3} [ora.storage] -- trace dump on error exit --

2020-11-23 17:33:39.915 : USRTHRD:2205349632: [ INFO] {0:5:3} [ora.storage] Error [kgfoAl06] in [kgfokge] at kgfo.c:3169

2020-11-23 17:33:39.915 : USRTHRD:2205349632: [ INFO] {0:5:3} [ora.storage] ORA-12547: TNS:lost contact

      手工去启动ASM，可以正常启动
      检查MOS，发现一篇文章：FLEXASM: Unable to Start CRS After Reboot ORA-12537 ora.storage ORA-12547 (Doc ID 2294385.1)
      现象和我这里一致，原因是没有把私网IP加入白名单，想象前两天做了IP限制，确实没有把私网 IP加入 ，加入后，重启集群，问题解决。

附MOS原文：
  FLEXASM: Unable to Start CRS After Reboot ORA-12537 ora.storage ORA-12547 (Doc ID 2294385.1)  
In this Document
Symptoms
Changes
Cause
Solution
APPLIES TO:
Oracle Database - Enterprise Edition - Version 12.1.0.2 and later
Oracle Database Cloud Schema Service - Version N/A and later
Oracle Database Exadata Cloud Machine - Version N/A and later
Oracle Database Exadata Express Cloud Service - Version N/A and later
Oracle Cloud Infrastructure - Database Service - Version N/A and later
Information in this document applies to any platform.
SYMPTOMS
This Document is applicable only for FLEX ASM configuration.
Good node ( Node 1)
CRS is up and running fine

SQL> select username from v$pwfile_users;
USERNAME
--------------------------------------------------------------------------------
SYS
CRSUSER__ASM_001
ASMSNMP
Bad node ( Node 2)
While starting CRS on node 2, CRSD and EVMD failed to start.
grid@node2 grid_12.2.0.1 +ASM2 $ crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4535: Cannot communicate with Cluster Ready Services <<< CRSD fails to start >>>
CRS-4529: Cluster Synchronization Services is online
CRS-4534: Cannot communicate with Event Manager  <<< EVMD fails to start >>>
ora.storage fails to start with the below error.
ohasd_orarootagent_root.trc
- - - - - - - - - -
2019-08-13 19:48:27.006 : USRTHRD:893126400: {0:5:3} 8154 Error 4 querying length of attr ASM_DISCOVERY_ADDRESS
2019-08-13 19:48:27.015 : USRTHRD:893126400: {0:5:3} 8154 Error 4 querying length of attr ASM_STATIC_DISCOVERY_ADDRESS
2019-08-13 19:48:27.084 : CLSCRED:893126400: (:CLSCRED1079:)clsCredOcrKeyExists: Obj dom : SYSTEM.credentials.domains.root.ASM.Self.303a14fb7cd04f7affd9f5ed7ebdcd83.root not found
2019-08-13 19:48:27.084 : USRTHRD:893126400: {0:5:3} 7872 Error 4 opening dom root in 0x7fd00820c3d0
2019-08-13 19:48:30.691 : USRTHRD:893126400: {0:5:3} ORA-12547: TNS:lost contact
ORA-12547: TNS:lost contact
ORA-15077: could not locate ASM instance serving a required diskgroup

CHANGES
Newly Implemented VNC (Valid Node checking) with all the required hosts in sqlnet.ora.
hostname restrictions in the sqlnet.ora
TCP.VALIDNODE_CHECKING = YES
TCP.INVITED_NODES=(all the required hosts for Valid node checking)
CAUSE
ASMNET1LSNR_ASM runs on top of Private IP.
[root@node1 ~]# oifcfg getif
eth0 192.168.56.0 global public
eth1 172.16.100.0 global cluster_interconnect,asm
[root@node1 ~]# lsnrctl status ASMNET1LSNR_ASM
Listening Endpoints Summary...
(DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=ASMNET1LSNR_ASM)))
(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=172.16.100.56)(PORT=1526)))   <<<< Private IP >>> Which is not included in sqlnet.ora (VNC)

Starting from FLEXASM crsd.bin will connect remotely using the ASMNET1LSNR_ASM.
1 SELECT s.inst_id, s.osuser, s.username, s.program, i.network_service_banner
2 FROM gv$session s, gv$session_connect_info i
3 WHERE s.sid = i.sid
4 AND s.inst_id = i.inst_id
5 AND s.program like 'crsd.bin%'
6 AND i.network_service_banner like '%Protocol%'
7* ORDER BY 1
INST_ID OSUSER USERNAME PROGRAM NETWORK_SERVICE_BANNER
---------- -------------------- ---------- ---------------------------------------- --------------------------------------------------------------------------------
1 root SYS crsd.bin@node1.localhost (TNS V1-V3) Oracle Bequeath NT Protocol Adapter for Linux: Version 12.1.0.2.0 - Production        <<<<<    Bequeath connection which is local >>>>              
1 root SYS crsd.bin@node2.localhost (TNS V1-V3) TCP/IP NT Protocol Adapter for Linux: Version 12.1.0.2.0 - Production                       <<<<<   TCP connection >>>>
The connections are blocked at the host level because the private IP's are not included in sqlnet.ora
Related errors are seen in crsd.trc
2017-07-26 12:49:26.311*:kgfo.c@989: kgfo_kge2slos error stack at kgfoAl06: ORA-12537: TNS:connection closed   <<< CRSD.BIN Connection using ASMNET1LSNR_ASM are getting closed due to VNC>>>
ORA-12537: TNS:connection closed
ORA-27300: OS system dependent operation:sslssunreghdlr failed with status: 0
ORA-27301: OS failure message: Error 0
ORA-27302: failure occurred at: sskgpreset1
ORA-15077: could not locate ASM instance serving a required diskgroup


SOLUTION
If the FLEX ASM is enabled along with VNC (Valid Node checking) in sqlnet.ora then the private IP's should be included in sqlnet.ora to have a proper functioning of CRS.
1) Modify the file sqlnet.ora
vi $GRID_HOME/network/admin/sqlnet.ora
2) Add the IP's which are used for Private interconnect.
e.g.
TCP.VALIDNODE_CHECKING = YES
TCP.INVITED_NODES=(node1.localhost, node2.localhost,node1-priv.localhost,node2-priv.localhost, node1-vip.localhost, node2-vip.localhost, application server VIPS)
3) Restart the CRS in issue node.
NOTE: This solution is only applicable if using VNC (Valid Node checking) in sqlnet.ora.