Troubleshooting Oracle Clusterware common startup failures

郑全 · 发表于 2017-6-16 12:00:44

本帖最后由郑全于 2017-6-16 12:13 编辑

As an Oracle DBA in a non-cluster environment, your responsibilities limited to manage, troubleshoot and diagnose problems that are pertaining to the database technologies. In contrast, you will have an additional responsibility of managing Clusterware and troubleshooting its problems in a cluster environment. The purpose of this article is to help you understanding the basics about Clusterware startup sequence and troubleshoot most common Clusterware startup failures. Additionally, this article also wills focus on some of the useful tools, utilities that are handy identifying the root cause of Clusterware related problems.
In my perspective and personal experience, the following is some of the challenges most DBAs in their cluster environment confront:

Node eviction
Cluster becoming unhealthy
Unable to start cluster and some of the Clusterware components

Clusterware startup sequenceIt’s worthwhile understanding how things get started or stopped while managing and troubleshooting a system. In this segment, we will closely look at how an Oracle Clusterware stack components are get started, and in which sequence they come up on a node reboot or manual cluster startup. This understanding will greatly help addressing most cluster stack common start-up failures and gives you a glance where to start the investigation in case any cluster component doesn’t start.
The diagram below depicts Oracle Cluster stack (components) startup sequence at various levels:

登录/注册后可看大图

chengci.png (106.94 KB, 下载次数: 468)

下载附件

chengci

2017-6-16 12:12 上传

Source: Expert Oracle RAC 12c

The entire Oracle Cluster stack and the services registered on the cluster automatically comes up when a node reboots or if the cluster stack manually starts. The startup process is segregated in five (05) levels, at each level, different processes are got started in a sequence.
On node reboot/crash, the init process on the OS spawns init.ohasd (as mentioned in the /etc/inittab file), which in turn commence Oracle High Availability Service Daemon (ohasd). The ohasd daemon is then responsible of starting off the other critical cluster daemon processes.
The new oraagent and oraclerootagent layers then brings up Cluster Synchronization Daemon (cssd), Cluster Ready Services (crsd), Event Manager Daemon (evmd) and other rest of Cluster stack in addition to ASM, RDBMS instances and other resources on the cluster at various levels.
Cluster-wide cluster commandsWith Oracle 11gR2, you can now start, stop and verify Cluster status of all nodes from a single node. Pre Oracle 11gR2, you must login to individual nodes to start, stop and verify cluster health status. Below are some of the cluster-wide cluster commands:

$ ./crsctl check cluster –all [verify cluster status on all nodes]
$ ./crsctl stop cluster –all [stop cluster on all nodes]
$ ./crsctl start cluster –all [start cluster on all nodes]
$ ./crsctl check cluster –n <nodename> [verify the cluster status on a particular remote node]

Troubleshooting common cluster startup problemsOnce you understand the startup sequence and how things get started in Oracle Cluster environment, it will easy for anyone to troubleshoot and solve most common start-up failure problems. In this segment, we will explain how to diagnose some of the common Cluster start-up problems.
Imagine your ‘crsctl check cluster/crs’ command and its gives the following errors:

$GRID_HOME/bin/crsctl check cluster

CRS-4639: Could not contact Oracle High Availability Services

CRS-4124: Oracle High Availability Services startup failed

CRS-4000: Command Check failed, or completed with errors

OR

CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4530: Communications failure contacting Cluster Synchronization Services daemon
CRS-4534: Cannot communicate with Event Manager

We will start taking the issues of the components startup failures in the same sequence they usually start. Let’s talk about ohasd startup failures which would result in ‘CRS-4639/4124/4000’ errors. The following are the main causes:

Verify if the /etc/inittab file contains the entry to start the ohasd process automatically.
Ensure Cluster auto startup is configured using the ‘crsctl config crs’ command. For any reasons it is not auto start configured, enable the auto start and start the cluster manually.
Refer alert.log and ohasd.log files under $ORA_GRID_HOME/log/hosname location.
Ensure the node has no issues accessing the OLR and OCR/Voting Disks. This can be verified in the alert or ohasd log files.

In the event of Cluster Synchronization Service Daemon process (cssd) start-up failures or reported unhealthy status, follow the below guide lines to resolve the problem:

Verify if the cssd.bin is active on the OS or not: ps –ef |grep cssd.bin
Review the alert log and ocssd.log files
Ensure there is no issues with regards to Voting Disks accessibility
Also, check the network heart beat between the nodes
Run crsctl stat res –t –init, and if the crsd process is unhealthy, start it up manually

$ ./crsctl start res ora.crsd –init

--------------------------------------------------------------------------------
NAME          TARGET STATE    SERVER                STATE_DETAILS
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.asm
   1    ONLINE ONLINE    rac2                Started
ora.cluster_interconnect.haip
   1    ONLINE ONLINE    rac2
ora.crsd
   1    ONLINE OFFLINE    rac2
ora.cssd
   1    ONLINE ONLINE    rac2
ora.cssdmonitor
   1    ONLINE ONLINE    rac2
ora.ctssd
   1    ONLINE ONLINE    rac2                OBSERVER
ora.diskmon
   1    ONLINE ONLINE    rac2
ora.evmd

If you encounter Cluster Ready Services Daemon (CRSD) start-up failures or reported unhealthy, you can perform the following check list:

Verify if the crsd.bin process is up on the OS: ps –ef |grep crsd.bin
Refer alert log and crsd logs to get useful information about the nature of the problem
Check the status of the resources using the above command, and start it manually if the status/target is OFFLINE
Ensure OCR disks are accessible to the node and there no OCR corruption

If you couldn’t resolve the problem reviewing the above guide lines, you might need to contact Oracle support and provide all required information to investigate further on the problem.
CRS logs and directory hierarchyEach component of Grid Infrastructure (Clusterware) maintains an individual log file and writes important events to the log file under typical circumstances. The information written to the log will help DBAs to understand the current state of the component, also assist troubleshooting cluster critical problems. DBA should comprehend the importance of these log files and able to understand the text to solve the problems. Each node in the cluster maintains an individual log directory under $GRID_HOME/log/<hostname> location for every cluster component, as shows in the following screen shot:

登录/注册后可看大图

file_list.png (140.44 KB, 下载次数: 462)

下载附件

2017-6-16 12:13 上传

Source: Expert Oracle RAC 12c

The following are some of the most referred logs as part of cluster maintenance and troubleshooting various Cluster related problems, such as, Node eviction, Cluster stack heath, OCR/Voting disk related problems etc:

alert_hostname.log
crsd.log
ocssd.log
ohasd.log

alert_hostname.log Each cluster node has its own alert log file and write important and useful information about cluster startup, node eviction, any cluster component start-up problems, OCR/Voting disk related information. It is therefore recommended to refer the log file frequently to know the cluster status, in the event of other node eviction, or wants to keep an eye on OCR/VD developments.
ocssd.log Yet another very critical and important log file which needs your attention. Whenever cluster encounters any serious snags with regards to Cluster Synchronization Service Daemon (CSSD) process, this is the file that needs to refer to understand the nature of the problem and to resolve the problem too. This is one of the busiest log files that is writing continuously and maintained by the cluster automatically. Manual maintenance on the file is not recommended, and once the file size reaches to 50MB, Oracle automatically archives the files and create a new log file. In the context, total 10 archive log files are maintenance at any given point in time.
crsd.log Cluster Ready Service Daemon (CRSD) process writes all important events to the file, such as, cluster resources startup, stop, failure and CRSD health status. If you have issues starting/stopping any cluster and non-cluster resources on the node, refer this log file to diagnose the issue. This files also maintained by the Oracle and remove the file is not recommended. Once the size of the file reaches to 10MB, it will be archived automatically and a new log file will be created. There will be 10 archived copies of the file will be kept under any point of time.
ohasd.log The log file is accessed and managed by the new Oracle High Availability Service Daemon (ohasd) process which was first introduced in Oracle 11gR2. If you encounter any issues running root.sh or rootupgrade.sh scripts, refer to this log file to understand troubleshooting the problem. If you face issues starting up the process, and if Oracle Local Registry (OLR) has any corruption or inaccessibility issues, also refer to this file. Like crsd and ocssd log files, this file also is maintained automatically by Oracle and archives the log file upon reaching 10MB size. Total of 10 archived log files are maintained at any given point in time.
Debugging and Tracing CRS componentsAs we have learned that CRS maintains a good amount of log files for various Cluster components which can be referred anytime to diagnose critical cluster startup issues, node eviction and others events. If the default debugging or logging information doesn’t provide sufficient feedback to resolve any particular issues, you can go on and increase the debug/trace level of a particular component or cluster resource to generate additional debugging information to address the problem.
Fortunately, Oracle let you adjust the default trace/debug levels of any cluster component or resource dynamically. To list the default trace/debug settings of a component or sub-component, login as root user and execute the following command from the GRID_HOME:

$ ./crsctl get log css/crs/evm/all

To adjust/change the default trace level, use the following example:

$ ./crsctl set log crs crsmain=4
$ ./crsctl set log crs all-=3 [all components/sub-components of CRS set to level 3]

You will have to seek Oracle support advice before you set or adjust any default settings. To disable the trace/debug level, set the level to value 0. You can set the levels from 1 to 5. The more the value, the more information will be generated and you must closely watch the log file growth and space on the filesystem to avoid any space related issues.
You can also turn on trace through OS settings; use the following OS specific command:

$ export SRVM_TRACE=true

Once the above is set on any UNIX OS system, trace/log files are generated under $GRID_HOME/cv/log destination. Once you exit from the terminal, tracing will end.
Oracle Clusterware Troubleshooting – tools & utilitiesOne of the prime responsibilities of an Oracle DBA is managing and troubleshooting the cluster system. In the context, the DBA must aware of all the internal and external tools and utilities provided by Oracle to maintain and diagnose cluster issues. The understanding and weighing the pros and cons of each individual tool/utility is essential. You must have a great knowledge and should choose the right tool/utility at the right moment; else, you will not only waste the time to resolve the issue, instead, will have a prolonged service interruption.
Let me zip through and explain you the benefits of some of the very important and mostly used tools and utilities here.

Cluster Verification Utility (CVU) – is used to collect pre and post cluster installation configuration details at various levels and various components. With 11gR2, it also provides the ability to verify the cluster health. Look at some of the useful commands below:

$ ./cluvfy comp healthcheck –collect cluster –bestpractice –html
$ ./cluvfy comp healthcheck –collect cluster|database

Real Time RAC DB monitoring (oratop) – is an external Oracle utility, currently available on Linux platform, which provides OS specific top alike output where you can monitor RAC databases/single instance databases in real time. The window provides statistics real-time, such as: DB Top event, top Oracle processes, blocking session information etc. You must download the oratop.zip from support.oracle.com and configure it.

RAC configuration audit tool (RACcheck) – yet another Oracle provided external tool developed by the RAC support team to perform audit on various cluster configuration. You must download the tool (raccheck.zip) from the support.oracle.com and configure it on one of the nodes of cluster. The tool performs cluster-wide configuration auditing at CRS,ASM, RDMS and generic database parameters settings. This tool also can be used to assess the readiness of the system for the upgrade. However, you need to keep upgrading the tool to get the latest recommendations.

Cluster Diagnostic Collection Tool (diagcollection.sh)– Since cluster manages so many log files, sometime it will be time consuming and cumbersome to visit/refer all the logs to understand the nature of the problem, or diagnose the issue. The diagcollection.sh tool refers various cluster log files and gathers required information to diagnose critical cluster problems. With this tool, you can gather the stats/information at various levels: Cluster, RDBMS, Core analysis, database etc. The tool encapsulates all file in a zip file and removes the individual files. The following .zip files are collected as part of the diagcollection run:
- ocrData_hostname_date.tar.gz è contains ocrdump, ocrcheck etc
- coreData_hostname_date.tar.gz à contains CRS core files
- osData_hostname_date.tar.gz à OS logs
- ocrData_hostname_date.tar.gz à OCR details

Example:

$ ./GRID_HOME/bin/diagcollection.sh –collect –crs $GRID_HOME
$ ./diagcollection.sh –help
--collect

[--crs] For collecting crs diag information

[--adr] For collecting diag information for ADR; specify ADR location

[--chmos] For collecting Cluster Health Monitor (OS) data

[--all] Default.For collecting all diag information. <<<>>>

[--core] Unix only. Package core files with CRS data

[--afterdate] Unix only. Collects archives from the specified date.

[--aftertime] Supported with -adr option. Collects archives after the specified

[--beforetime] Supported with -adr option. Collects archives before the specified

[--crshome] Argument that specifies the CRS Home location

[--incidenttime] Collects Cluster Health Monitor (OS) data from the specified

[--incidentduration] Collects Cluster Health Monitor (OS) data for the duration

NOTE:

            1. You can also do the following
            ./diagcollection.pl --collect --crs --crshome <CRS Home>
            --clean cleans up the diagnosability
                  information gathered by this script
Above all, there is many other important and useful tools: Cluster Health Monitoring (CHM) to diagnose node eviction issues, DB Hanganalysis, OSWatcher, The Light on Monitor, ProWatcher etc are available for your use under different circumstances.
In a nutshell, this paper make you understand the Cluster startup sequence, how things get started on node reboot and provides guide lines to solve most commonly faced Cluster startup issues by analyzing the cluster logs, using appropriate tools/utilities discussed.

帐号		自动登录	找回密码
密码			注册

[转载] Troubleshooting Oracle Clusterware common startup failures