11gRAC报错CRS-4535, CRS-4000解决

jiawang

报错现象：

[grid@rac02~]$ crsctl status res -t

CRS-4535:Cannot communicate with Cluster Ready Services

CRS-4000:Command Status failed, or completed with errors.

此时数据库是可以被正常访问的。如下：

[oracle@rac02~]$ sqlplus / as sysdba

SQL*Plus:Release 11.2.0.4.0 Production on Wed Sep 5 10:42:48 2018

Copyright(c) 1982, 2013, Oracle.  All rightsreserved.

Connectedto:

OracleDatabase 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production

Withthe Partitioning, Real Application Clusters, Automatic Storage Management,OLAP,

DataMining and Real Application Testing options

SQL>

解决办法强制关闭crs

[root@rac01 /]# cd /u01/app/11.2.0/grid/bin

[root@rac01 bin]# crsctl stop crs -f

bash: crsctl: 未找到命令...

[root@rac01 bin]# ./crsctl stop crs -f

CRS-2791: Starting shutdown of Oracle HighAvailability Services-managed resources on 'rac01'

CRS-2673: Attempting to stop 'ora.crsd' on'rac01'

CRS-2790: Starting shutdown of ClusterReady Services-managed resources on 'rac01'

CRS-2673: Attempting to stop'ora.rac01.vip' on 'rac01'

CRS-2673: Attempting to stop 'ora.orcl.db'on 'rac01'

CRS-2673: Attempting to stop'ora.LISTENER.lsnr' on 'rac01'

CRS-2673: Attempting to stop'ora.LISTENER_SCAN1.lsnr' on 'rac01'

CRS-2677: Stop of 'ora.LISTENER.lsnr' on'rac01' succeeded

CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr'on 'rac01' succeeded

CRS-2673: Attempting to stop'ora.scan1.vip' on 'rac01'

CRS-2677: Stop of 'ora.rac01.vip' on'rac01' succeeded

CRS-2673: Attempting to stop'ora.rac02.vip' on 'rac01'

CRS-2677: Stop of 'ora.orcl.db' on 'rac01'succeeded

CRS-2673: Attempting to stop 'ora.DATA.dg'on 'rac01'

CRS-2677: Stop of 'ora.scan1.vip' on'rac01' succeeded

CRS-2677: Stop of 'ora.rac02.vip' on 'rac01'succeeded

CRS-2677: Stop of 'ora.DATA.dg' on 'rac01'succeeded

CRS-2673: Attempting to stop 'ora.asm' on'rac01'

CRS-2677: Stop of 'ora.asm' on 'rac01'succeeded

CRS-2673: Attempting to stop 'ora.ons' on'rac01'

CRS-2677: Stop of 'ora.ons' on 'rac01' succeeded

CRS-2673: Attempting to stop'ora.net1.network' on 'rac01'

CRS-2677: Stop of 'ora.net1.network' on'rac01' succeeded

CRS-2792: Shutdown of Cluster ReadyServices-managed resources on 'rac01' has completed

CRS-2677: Stop of 'ora.crsd' on 'rac01'succeeded

CRS-2673: Attempting to stop 'ora.ctssd' on'rac01'

CRS-2673: Attempting to stop 'ora.evmd' on'rac01'

CRS-2673: Attempting to stop 'ora.asm' on'rac01'

CRS-2673: Attempting to stop 'ora.mdnsd' on'rac01'

CRS-2677: Stop of 'ora.evmd' on 'rac01'succeeded

CRS-2677: Stop of 'ora.mdnsd' on 'rac01'succeeded

CRS-2677: Stop of 'ora.ctssd' on 'rac01'succeeded

CRS-2677: Stop of 'ora.asm' on 'rac01'succeeded

CRS-2673: Attempting to stop'ora.cluster_interconnect.haip' on 'rac01'

CRS-2677: Stop of'ora.cluster_interconnect.haip' on 'rac01' succeeded

CRS-2673: Attempting to stop 'ora.cssd' on'rac01'

CRS-2677: Stop of 'ora.cssd' on 'rac01'succeeded

CRS-2673: Attempting to stop 'ora.crf' on'rac01'

CRS-2677: Stop of 'ora.crf' on 'rac01'succeeded

CRS-2673: Attempting to stop 'ora.gipcd' on'rac01'

CRS-2677: Stop of 'ora.gipcd' on 'rac01'succeeded

CRS-2673: Attempting to stop 'ora.gpnpd' on'rac01'

CRS-2677: Stop of 'ora.gpnpd' on 'rac01'succeeded

CRS-2793: Shutdown of Oracle HighAvailability Services-managed resources on 'rac01' has completed

CRS-4133: Oracle High Availability Serviceshas been stopped.

[root@rac01 bin]#

查看crsctl资源状态，此时肯定没有

[root@rac01 bin]# ./crsctl stat res -t

CRS-4535: Cannot communicate with ClusterReady Services

CRS-4000: Command Status failed, orcompleted with errors.

查看相关进程也不存在

[root@rac01 bin]# ps -ef|grep crsd.bin

root      6792   3156  0 11:36 pts/1    00:00:00 grep --color=auto crsd.bin

[root@rac01 bin]# ps -ef|grep cssd.bin

root      6805   3156  0 11:36 pts/1    00:00:00 grep --color=auto cssd.bin

[root@rac01 bin]# ps -ef|grep evmd.bin

root      6810   3156  0 11:36 pts/1    00:00:00 grep --color=auto evmd.bin

[root@rac01 bin]#

查看pmon进程

[root@rac01 bin]# ps -ef| grep pmon

root      7902   3156  0 11:38 pts/1    00:00:00 grep --color=auto pmon

尝试启动crs,成功

[root@rac01 bin]# ./crsctl start crs

CRS-4123: Oracle High Availability Serviceshas been started.

验证GI的核心进程

[root@rac01 bin]# ps -ef|grep cssd.bin|grep-v grep

grid      8182      1 0 11:39 ?        00:00:00/u01/app/11.2.0/grid/bin/ocssd.bin

[root@rac01 bin]# ps -ef|grep evmd.bin

grid      8353      1  0 11:43 ?        00:00:00/u01/app/11.2.0/grid/bin/evmd.bin

root      8437   3156  0 11:43 pts/1    00:00:00 grep --color=auto evmd.bin

[root@rac01 bin]# ps -ef|grep crsd.bin

root      8427      1  7 11:43 ?        00:00:00/u01/app/11.2.0/grid/bin/crsd.bin reboot

root      8513   3156  0 11:43 pts/1    00:00:00 grep --color=auto crsd.bin

[root@rac01 bin]# ps -ef|grep pmon|grep -vgrep

grid      8369      1  0 11:43 ?        00:00:00 asm_pmon_+ASM1

[root@rac01 bin]#

验证集群状态

[root@rac01 bin]# ./crsctl stat res -t

--------------------------------------------------------------------------------

NAME           TARGET  STATE       SERVER                   STATE_DETAILS      

--------------------------------------------------------------------------------

Local Resources

--------------------------------------------------------------------------------

ora.DATA.dg

               ONLINE  ONLINE      rac01                                       

ora.LISTENER.lsnr

               ONLINE  ONLINE      rac01                                       

ora.asm

               ONLINE  ONLINE      rac01                   Started            

ora.gsd

               OFFLINE OFFLINE      rac01                                       

ora.net1.network

               ONLINE  ONLINE      rac01                                       

ora.ons

               ONLINE  ONLINE      rac01                                       

--------------------------------------------------------------------------------

Cluster Resources

--------------------------------------------------------------------------------

ora.LISTENER_SCAN1.lsnr

     1        ONLINE  ONLINE      rac01                                       

ora.cvu

     1        OFFLINE OFFLINE                                                  

ora.oc4j

     1        OFFLINE OFFLINE                                                  

ora.orcl.db

     1        ONLINE  ONLINE      rac01                    Open               

     2        ONLINE  OFFLINE                                                  

ora.rac01.vip

     1        ONLINE  ONLINE      rac01                                       

ora.rac02.vip

      1       ONLINE  INTERMEDIATE rac01                    FAILED OVER         

ora.scan1.vip

     1        ONLINE  ONLINE      rac01                                       

[root@rac01 bin]#