Intermittently VIP Failback does not work after the Network Connection is Restored (文档 ID 1992370.1)
In this Document
Symptoms
Cause
Solution
References
APPLIES TO:
Oracle Database - Enterprise Edition - Version 11.2.0.4 and later
Oracle Net Services - Version 12.1.0.2 to 12.1.0.2 [Release 12.1]
Oracle Database Cloud Schema Service - Version N/A and later
Oracle Database Exadata Express Cloud Service - Version N/A and later
Oracle Database Exadata Cloud Machine - Version N/A and later
Information in this document applies to any platform.
SYMPTOMS
In a 2-node RAC cluster, perform a VIP failover test. After pulling the public network cable on node 2, VIP of node 2 successfully fails over to node 1 as expected. Reconnect the network cable on node 2. Intermittently VIP of node 2 does not fail back to node 2, it becomes OFFLINE. Manually using srvctl to start the VIP of node 2 works fine. Sometimes it fails back to node 2 successfully without intervention.
1. Pulling a cable to the public NIC of Node#2
Jun 9 21:41:29 EVENT START: network_down <hostname> net_ether_01
Jun 9 21:41:29 EVENT COMPLETED: network_down <hostname> net_ether_01 0
Jun 9 21:41:37 EVENT START: network_down_complete <hostname> net_ether_01
Jun 9 21:41:37 EVENT COMPLETED: network_down_complete <hostname> net_ether_01
2. VIP of node 2 fails over to node 1 as expected:
Mon Jun 9 21:41:03 KST 2014
------------------------------------------------------------------------------
NAME TARGET STATE SERVER STATE_DETAILS
------------------------------------------------------------------------------
ora.<host>.vip
1 ONLINE INTERMEDIATE <host> FAILED OVER
3. Reconnect cable to the NIC of Node#2
Jun 9 21:44:15 EVENT START: network_up <hostname> net_ether_01
Jun 9 21:44:15 EVENT COMPLETED: network_up <hostname> net_ether_01 0
Jun 9 21:44:15 EVENT START: network_up_complete <hostname> net_ether_01
Jun 9 21:44:15 EVENT COMPLETED: network_up_complete <hostname> net_ether_01 0
4. The Vip of node 2 does not failback, instead it becomes OFFLINE
Mon Jun 9 21:56:23 KST 2014
------------------------------------------------------------------------------
NAME TARGET STATE SERVER STATE_DETAILS
------------------------------------------------------------------------------
ora.<host>.vip
1 OFFLINE OFFLINE
5. Using srvctl start nodeapps, the VIP is started fine:
Mon Jun 9 21:56:38 KST 2014
------------------------------------------------------------------------------
NAME TARGET STATE SERVER STATE_DETAILS
------------------------------------------------------------------------------
ora.<host>.vip
1 ONLINE ONLINE racnode2
<GRID_HOME>/log/<node>/agent/crsd/orarootagent_root/orarootagent_root.log shows:
Oracle Database 11g Clusterware Release 11.2.0.4.0 - Production Copyright 1996, 2011 Oracle. All rights reserved.
2014-06-09 21:42:56.882: [ora.net1.network][4507]{0:1:17379} [check] ioctl Error
2014-06-09 21:42:56.883: [ AGFW][2057]{0:1:17379} Agent sending last reply for: RESOURCE_PROBE[ora.net1.network <hostname> 1] ID 4097:790184
2014-06-09 21:42:56.885: [ AGFW][2057]{0:1:17379} Agent received the message: RESOURCE_PROBE[ora.net1.network <hostname> 1] ID 4097:790187
2014-06-09 21:42:56.885: [ AGFW][2057]{0:1:17379} Preparing CHECK command for: ora.net1.network <hostname> 1
2014-06-09 21:42:56.886: [ora.net1.network][1166]{0:1:17379} [check] ioctl Error
2014-06-09 21:42:56.887: [ora.net1.network][1166]{0:1:17379} [check] (null) category: -1, operation: failed system call, loc: ioctl, OS error: 6, other:
2014-06-09 21:42:56.887: [ora.net1.network][1166]{0:1:17379} [check] clsn_agent::check: Exception IoctlException
2014-06-09 21:42:56.887: [ora.net1.network][1166]{0:1:17379} [check] ioctl Error
......
When there is no VIP failback, ora.net1.network status on node 2 shows INTERMEDIATE instead of ONLINE.
CAUSE
This is reported in BUG 19126172 - FAILBACK FROM NODE 1 TO NODE 2 NOT HAPPENING
Closed as unpublished BUG 17059927 - ORA.NET1.NETWORK KEEP INTERMEDIATE AFTER DOWN PUBLIC NIC
This is code defect, basically when ioctl fails we just re-throw the error this led to the network resource status keeps INTERMEDIATE and prevents vip from failback.
SOLUTION
The Bug 17059927 has been fixed in 11.0.2.0.4.5 PSU and Windows 11.2.0.4.12 Bundle and 12.1.0.2.
Please apply the PSU or bundle patch where the fix is included.