Grid Infrastructure Startup During Patching, Install or Upgrade May Fail

郑全

Grid Infrastructure Startup During Patching, Install or Upgrade May Fail Due to Multicasting Requirement (文档 ID 1212703.1)        转到底部转到底部       

In this Document

Symptoms

Changes

Cause

Solution

        Scalability RAC Community

References

APPLIES TO:

Oracle Database - Enterprise Edition - Version 11.2.0.2 and later

Oracle Database Cloud Schema Service - Version N/A and later

Oracle Database Exadata Cloud Machine - Version N/A and later

Oracle Cloud Infrastructure - Database Service - Version N/A and later

Oracle Database Cloud Exadata Service - Version N/A and later

Information in this document applies to any platform.

This issue impacts environments that do not have multicast enabled for the private network in the following situations:

New installations of Oracle Grid Infrastructure 11.2.0.2 where multicast is not enabled on 230.0.1.0

Upgrades to Oracle Grid Infrastructure 11.2.0.2 from a pre-11.2.0.2 release where multicast is not enabled on 230.0.1.0 or 224.0.0.251

Installation of GI PSU 11.2.0.3.5, 11.2.0.3.6, 11.2.0.3.7 where multicast is not enabled on 230.0.1.0 or 224.0.0.251

Installation or upgrade to 12.1.0.1.0 where multicast is not enabled on 230.0.1.0 or 224.0.0.251

SYMPTOMS

If multicast based communication is not enabled as required either on the nodes of the cluster or on the network switches used for the private interconnect, the root.sh, which is called as part of a fresh installation of Oracle Grid Infrastructure 11.2.0.2, or the rootupgrade.sh (called as part of an upgrade to Oracle Grid Infrastructure 11.2.0.2) will only succeed on the first node of the cluster, but will fail on subsequent nodes with the symptoms shown below:

CRS-4402: The CSS daemon was started in exclusive mode but found an active CSS daemon on node <node1>, number 1, and is terminating

An active cluster was found during exclusive startup, restarting to join the cluster

Failed to start Oracle Clusterware stack

Failed to start Cluster Synchorinisation Service in clustered mode at /u01/app/crs/11.2.0.2/crs/install/crsconfig_lib.pm line 1016.

/u01/app/crs/11.2.0.2/perl/bin/perl -I/u01/app/crs/11.2.0.2/perl/lib -I/u01/app/crs/11.2.0.2/crs/install /u01/app/crs/11.2.0.2/crs/install/rootcrs.pl execution failed

Note:  The symptoms will be the same whether an upgrade or a fresh installation of Oracle Grid Infrastructure 11.2.0.2 is performed; so will be the required diagnostics. This issue is also documented in the Oracle Database Readme 11g Release 2 Section 2.39 - "Open Bugs" under BUG: 9974223.

Note: This issue also impacts the following 11.2.0.3 PSUs: 11.2.0.3.5, 11.2.0.3.6, 11.2.0.3.7 as well as 12.1.0.1 installations if multicast is not enabled on the 230.0.1.0 or 224.0.0.251 multicast addresses (one of the 2 must be enabled/functional). With 11.2.0.3 GI was enhanced to utilize broadcast or multicast to bootstrap. However the 11.2.0.3.5 GI PSU introduced a new issue that effectively disables the broadcast functionality (Bug 16547309).

Symptom verification

To verify that Oracle CSS daemon fails to start in clustered mode due to a multicasting issue on the network, the ocssd.log file (located under $GI_HOME/log/<nodename>/cssd/ocssd.log) must be reviewed. In case, joining the cluster fails because of such an issue, the following can be observed:

1. When CSS starts in clustered mode to join an existing cluster, we will see an entry in the CSSD log indicating that CSS will attempt to establish communication with a peer in the cluster. For this analysis, we see in the CSSD log for <node2> that communication is attempted with <node1>, which looks similar to:  

2010-09-16 23:13:14.862: [GIPCHGEN][1107937600] gipchaNodeCreate: adding new node 0x2aaab408d4a0 { host '<node1>', haName 'CSS_ttoprf10cluster', srcLuid 54d7bb0e-ef4a0c7e, dstLuid 00000000-00000000 numInf 0, contigSeq 0, lastAck 0, lastValidAck 0, sendSeq [0 : 0], createTime 9563084, flags 0x0 }

2.  Shortly after the above log entry we will see an attempt to establish communication to <node1> from <node2> via multicast address 230.0.1.0, port 42424 on the private interconnect (here: 192.168.x.x):

2010-09-16 23:13:14.862: [GIPCHTHR][1106360640] gipchaWorkerUpdateInterface: created remote interface for node '<node1>', haName 'CSS_mycluster', inf 'mcast://230.0.1.0:42424/192.168.x.x'

3.  If the communication can be established successfully, we will see a log entry on node2 containing "gipchaLowerProcessAcks: ESTABLISH finished" for the peer node (<node1>). If the communication cannot be established, we will not see this log entry. Instead, we will see an entry indicating that the network communication cannot be established. This entry will look similar to the one shown below:

2010-09-16 23:13:15.839: [ CSSD][1087465792]clssnmvDHBValidateNCopy: node 1, <node1>, has a disk HB, but no network HB, DHB has rcfg 180134562, wrtcnt, 8627, LATS 9564064, lastSeqNo 8624, uniqueness 1284701023, timestamp 1284703995/10564774

The above log entry indicates that CSSD is unable to establish network communication on the interface used for the private interconnect. In this particular case, the issue was that multicast communication on the 230.0.1.0 IP was blocked on the network used as the private interconnect.

CHANGES

New installations of Oracle Grid Infrastructure 11.2.0.2

Upgrades of a previous release to Oracle Grid Infrastructure 11.2.0.2

Installation of the 11.2.0.3.5, 11.2.0.3.6, 11.2.0.3.7 GI PSUs where multicast is not enabled on the 230.0.1.0 or 224.0.0.251 multicast addresses

New installations of Oracle Grid Infrastructure 12.1.0.1 where multicast is not enabled on the 230.0.1.0 or 224.0.0.251 multicast addresses

Upgrades of a previous release to Oracle Grid Infrastructure 12.1.0.1 where multicast is not enabled on the 230.0.1.0 or 224.0.0.251 multicast addresses

Note: 11.2.0.4 Grid Infrastructure is not impacted by this issue.

CAUSE

Assuming that Cluster Verify (cluvfy) has succeeded regarding the network checks on all nodes of the cluster or the symptoms described above are observed as part of an upgrade to Oracle Grid Infrastructure 11.2.0.2 (which means that the current release does not encounter such communication issues), this issue is probably caused by multicast not being enabled on the network used as the private interconnect.

Background information for 11.2.0.2

Oracle Grid Infrastructure 11.2.0.2 introduces a new feature called "Redundant Interconnect Usage", which provides an Oracle internal mechanism to make use of physically redundant network interfaces for the Oracle (private) interconnect. As part of this new feature, multicast based communication on the private interconnect is utilized to establish communication with peers in the cluster on each startup of the stack on a node. Once the connection with the peers in the cluster has been established, the communication is switched back to unicast. Per default, the 230.0.1.0 address (port 42424) on the private interconnect network is used for multicasting. Another IP can be enabled using the patch mentioned below, if it is determined that using the 230.0.1.0 IP causes the multicast communication to fail. Multicasting on either of these IPs and the respective port must, however, be enabled and functioning across the network and on each node meant to be part of the cluster. If multicasting is not enabled as required, nodes will fail to join the cluster with the symptoms discussed.

Background information for 11.2.0.3.5, 11.2.0.3.6, 11.2.0.3.7 GI PSUs and 12.1.0.1

With 11.2.0.3 GI was enhanced to utilize broadcast or multicast (on 230.0.1.0 or 224.0.0.251 addresses) to bootstrap. However the 11.2.0.3.5 GI PSU introcuces a new issue with effectivly disables the broadcast functionality (Bug 16547309).  Do note that most networks do support multicast on the 224.0.0.251 multicast address without any special configuration, therefore the odds of this being an issue for 11.2.0.3.5 - 11.2.0.3.7 and 12.1.0.1 are greatly reduced.

Note:  The Oracle CSS daemon may fail to establish network communication with peer nodes for other reasons than multicast not working as required on the private interconnect, which is discussed in this note. Therefore, refer to Note: 1054902.1 for general network communication troubleshooting, if you determine that multicasting is not the root cause for such issues on your system.

Configuring various network switches for multicast

While the most common reason for multicast based communication to fail with Oracle Grid Infrastructure 11.2.0.2 is probably the 230.0.1.0 multicast address being used by default, multicast being disabled or misconfigured on the network switches is an equally common cause. The mcasttest.pl-tool provided as part of this note will give you an indication whether or not multicasting has been disabled on the network switches, if it fails to use multicast on either the 230.0.1.0 or 224.0.0.251 multicast address for example. In general, work with your network administrator, if you suspect multicasting has been disabled on the network switches, affecting the private interconnect communication accordingly.

Cisco Nexus Switches - general

The Cisco Nexus switches can be complicated to configure with multicasting. If multicasting cannot be properly configured, then these switches have the ability to turn off IGMP Snooping on a per-VLAN basis. This has the effect of having the switch handle multicasts like it would a broadcast. Since the network used for the interconnect is supposed to be private (where only the members of the cluster have access to this network), broadcasting every multicast packet to every NIC on the VLAN should pose no problem. The command to turn off IGMP Snooping on a VLAN is:

no ip igmp snooping

(Consult the Cisco Nexus manual for the exact syntax of this command.)

Cisco Catalyst

Turning off IGMP Snooping on a Cisco Catalyst switch is a more complex process, which is not covered in this note. For the Catalyst family of switches, IGMP Snooping can only be turned off globally and not on a per-VLAN basis. Consult the Cisco Catalyst manuals for how to properly configure multicasting for the switches over-all usage. Note that turning off IGMP Snooping on the Catalyst may cause other systems or programs to fail.

Cisco Nexus 7000 Switch

A feature of the Cisco Nexus 7000 switch is a filter that effects large UDP messages. This command is designed to filter IP packets with the "do not fragment" bit set along with a non-zero "Fragment offset". Oracle RAC will send UDP messages that exceed the carrying capacity of a standard UDP packet. The IP layer sees this and sends the left-over data in IP Fragments. These IP Fragments will then have the "do not fragment" bit set along with a non-zero "Fragment offset". The switch will not deliver these packets as they are filtered out by this rule.

To validate this issue, perform a tcpdump (with the -s 0 flag to capture the entire packet) on each node during the Oracle Clusterware installation at the time the root.sh script is run. The root.sh command will start the Oracle cluster services on the first node. When it is run on the second node, large UDP packets can occur, which will generate the IP Fragments that will be dropped. When one reads the two tcpdump files, on will see an IP Fragment packet leave the second node, yet it will not arrive on the first node. The first node will send out ICMP "Fragment reassembly time exceeded" packets as a result. To turn this rule off issue the following command:

no hardware ip verify fragment

(Consult the Cisco Nexus manual for the exact syntax of this command.)

These settings apply only to the Cisco Nexus 7000 switch. This command may be available on other models of the Nexus switch, but not all of the Nexus models support it. Check the Cisco manual for the model of concern.

SOLUTION

For 11.2.0.2 Installations

It has been found that using a 230.0.1.0, port 42424, network address for multicasting can be problematic with some network configurations. Therefore, Oracle has released Patch: 9974223 on top of Oracle Grid Infrastructure 11.2.0.2. This patch enables multicasting on the 224.0.0.251 multicast address (port 42424) in addition to the 230.0.1.0 (port 42424) address used by default. Multicast must generally be enabled on one of these two addresses to allow Oracle Grid Infrastructure to successfully start on all nodes configured to join a particular cluster.

For 11.2.0.3.5, 11.2.0.3.6 and 11.2.0.3.7 GI PSU  and 12.1.0.1 Installations

11.2.0.3.5 - 11.2.0.3.7 GI PSU and 12.1.0.1 installations will ONLY be impacted if those installations were relying on the broadcast functionality that was initially introduced in 11.2.0.3 base.  For these installations the most simplistic solution is to enable multicast on either the 230.0.1.0 or 224.0.0.251 address (only 1 of the 2 addresses needs to be functional).  For those installations that are unable to enable multicast, Patch 16547309 is available for some platforms to re-enable broadcast functionality.  If you require this patch (due to not being able to enable multicast) and it is not available for your version/platofrm please raise an SR with support to request the fix.  The fix for Bug 16547309 will be included in the 11.2.0.3.8 and 12.1.0.1.2 GI PSUs.

For 12.2.0.1 Installations

"Multicasting is required on the private interconnect. For this reason, at a minimum, you must enable multicasting for the cluster for the following..."

Reference:  http://docs.oracle.com/database/ ... 2-88C0-CBE8E4893DF7

Problem validation

Before applying Patch: 9974223, Patch 16547309 or any subsequent Oracle Grid Infrastructure (GI) Bundle Patch including which include the mentioned fixes, understand that the issue described in this note does not apply under the following circumstances:

Note: The issue described in this note does not apply

on Windows

The Redundant Interconnect Usage feature does not exist on Windows.

on Solaris when using Solaris Cluster

When using Sun Cluster, Oracle Clusterware 11.2.0.2 will detect the presence of clprivnet0 and not enable any HAIPs. The Redundant Interconnect Usage feature is thereby disabled. clprivnet0 will provide the required availability for redundant network interfaces used for the interconnect.

on HP-UX or AIX when using Infiniband network cards

If on HP-UX or AIX using Infiniband network interfaces the installation of Oracle Grid Infrastructure will fail due to OS and network hardware related issues, unrelated to multicast.

These are known BUGs and Oracle is working with the respective OS vendors for a fix.

Contact Oracle Support if you plan on upgrading to Oracle Grid Infrastructure 11.2.0.2 on either of those platforms using Infiniband network cards.

using Primecluster

The Redundant Interconnect Usage feature is currently disabled when using Primecluster.

If you have already run the root.sh (rootupgrade.sh) on any node of the cluster:

Note: The README for Patch: 9974223 and Patch 16547309 contain instructions for those installations in which root.sh (or rootupgrade.sh) has failed due to this issue.  A complete reinstall of Grid Infrastructure should  not be required.

Multicast can be vaildated using the mcasttest.pl tool attached to this note to validate whether or not multicast on the 230.0.1.0 or 224.0.0.251 address can be used on the private interconnect interface(s).  CLUFFY in 11.2.0.3 and above does contain similar checks but will report pass if the broadcast check passes, with mentioned issue which breaks broadcast in 11.2.0.3.5 - 11.2.0.3.7 GI PSU and 12.1.0.1 this pass is a false positive.  For this reason it is recommended to use mcasttest.pl program to validate multicast functionality before patching/upgrading/installing the mentioned releases.

Note:  Multicast based communication only needs to be successful on either the 230.0.1.0 address or

the 224.0.0.251 address. A successful multicast communication on both addresses is not required.

How to use the mcasttest-tool

Download the mcasttest.pl perl program from this note and extract the contents of the package to node 1 of your cluster:

# gunzip mcasttest.tgz

# tar xvf mcasttest.tar

The mcasttest.pl program can be found in the mcasttest directory extracted in the previous step.

The mcasttest.pl program requires two arguments:

The node list (specified with -n)

The list of interfaces to be used for the private interconnect (specified with -i).

The mcasttest is only to be executed on a single node in the cluster. It will use password-less SSH based access (also required for the Grid Infrastructure software installation) to distribute

and execute the program on the remote nodes. The command string to execute mcasttest is:

# perl mcasttest.pl -n <node1>,<node2>,<node_n...> -i <interface1>,<interface2><interface_n...>

The example below tests multicast for a two node cluster (<node1> and <node2>), in

which two network interfaces (eth1 and eth2) are to be used for the private interconnect:

# perl mcasttest.pl -n <node1>,<node2> -i eth1,eth2

########### Setup for node <node1>##########

Checking node access '<node1>'

Checking node login '<node1>'

Checking/Creating Directory /tmp/mcasttest for binary on node '<node1>'

Distributing mcast2 binary to node '<node1>'

########### Setup for node <node2> ##########

Checking node access '<node2>'

Checking node login '<node2>'

Checking/Creating Directory /tmp/mcasttest for binary on node '<node2>'

Distributing mcast2 binary to node '<node2>'

########### testing Multicast on all nodes ##########

Test for Multicast address 230.0.1.0

Nov 8 09:05:33 | Multicast Failed for eth1 using address 230.0.1.0:42000

Nov 8 09:05:34 | Multicast Failed for eth2 using address 230.0.1.0:42001

Test for Multicast address 224.0.0.251

Nov 8 09:05:35 | Multicast Succeeded for eth1 using address 224.0.0.251:42002

Nov 8 09:05:36 | Multicast Succeeded for eth2 using address 224.0.0.251:42003

As shown in the example above, the test has failed for the 230.0.1.0 address, but succeeded for the 224.0.0.251 multicast address. In this case, Patch: 9974223 must be applied to enable Oracle Grid Infrastructure to use the 224.0.0.251 multicast address.  

Interpreting the outcome of the mcasttest-tool correctly:

Should the mcasttest.pl test-tool have failed for both, the 230.0.1.0 and the 224.0.0.251 address

you must work with your System and / or Network Administrator to enable multicast on one of the addresses.

Should the mcasttest.pl test-tool have failed for the 230.0.1.0 address only

Apply Patch: 9974223 or any subsequent GI PSU (fix provided GI PSU 1 and above)

Should the mcasttest.pl test-tool have returned "success" for both, the 230.0.1.0 and the 224.0.0.251 address

No specific patch application is required and you can proceed with the installation.  

When and how to patch for 11.2.0.2

Ideally, Patch: 9974223 or the latest 11.2.0.2 GI PSU  (fix included in GI PSU 1 and above) is applied before any root.sh or rootupgrade.sh has ever been run on any of the nodes meant to form a cluster. It is highly recommended that the latest 11.2.0.2 GI PSU is installed to obtain the fix for Bug: 9974223 (instead of Patch: 9974223 standalone). In order to apply Patch: 9974223 either directly or as part of a Bundle Patch, perform the following steps:

Perform an Oracle Grid Infrastructure 11.2.0.2 installation or upgrade

Right before the first root.sh (or rootupgrade.sh) is supposed to be run, leave the current installation behind.

Do not close the installer or abort an operation in progress.

Leave the current installation as-is and open a new terminal.

Download and prepare the application of Patch: 9974223 either directly or as part of a GI Bundle.

Unlike described in the patch readme (of the GI Bundle Patch),

do not use "opatch auto"

Instead, use: "opatch napply -local"

IF this is a fresh installation, you do not need to run rootcrs.pl -unlock and rootcrs.pl -patch, because root.sh is not run yet, so all the files are unlocked.

Note: as Opatch is used with the "-local" flag here, you need to perform this operation on every node.

After you have patched every node in the cluster, return to the original installation

Proceed to run the root.sh (rootupgrade.sh) on all nodes and follow the instructions on the screen.

When and how to patch for 11.2.0.3.5, 11.2.0.3.6, 11.2.0.3.7

As previously mentioned, the most simplistic solution is to enable multicast on either the 230.0.1.0 or 224.0.0.251 address to address.  If this is not possible within your environment, the recommended solution is to install the fix for Bug 16547309 prior to executing "rootcrs.pl -patch".

When and how to patch for 12.1.0.1

Again, the most simplistic solution is to enable multicast on either the 230.0.1.0 or 224.0.0.251 address to address this issue.  If this is not possible within your environment, the recommended solution is to install the fix for Bug 16547309 or the latest GI PSU containing the fix (12.1.0.1.2) prior to running root.sh or rootupgrade.sh.  In order to apply Patch 16547309 prior to the execution of root.sh or rootupgrade.sh, perform the following steps:

Perform an Oracle Grid Infrastructure 12.1.0.1 installation or upgrade

Right before the first root.sh (or rootupgrade.sh) is supposed to be run, leave the current installation behind.

Do not close the installer or abort an operation in progress.

Leave the current installation as-is and open a new terminal.

Download and prepare the application of Patch 16547309 either directly or as part of a GI Bundle.

Unlike described in the patch readme (of the GI Bundle Patch),

do not use "opatch auto"

Instead, use: "opatch napply -local"

IF this is a fresh installation, you do not need to run rootcrs.pl -unlock and rootcrs.pl -patch, because root.sh is not run yet, so all the files are unlocked.

Disclaimer and summary: Due to differences in network hardware and overall network topologies that may be used for the interconnect network, it is not possible for Oracle to provide a single solution for enabling multicast within your network environment. That being said, it is important that you work with your System and / or  Network Administrators to enable multicast functionality for the 230.0.1.0 or the 224.0.0.251 multicast address (using Patch: 9974223) . The mcasttest.pl program attached to this note should be used to ensure that multicast communication is functioning properly to support Oracle Grid Infrastructure.

Scalability RAC Community

To discuss this topic further with Oracle experts and industry peers, we encourage you to review, join or start a discussion in the My Oracle Support Scalability RAC Community.