ASM Alert Logs Show Continuously: Attempting Voting File Relocation

郑全

巡检数据库时，发现ASM报警日志文件非常大，而且还在不停的增大，经检查发现，报警文件中，不停显示的是：

NOTE: Attempting voting file refresh on diskgroup GRID
NOTE: Voting file relocation is required in diskgroup GRID
NOTE: Attempting voting file relocation on diskgroup GRID
NOTE: Attempting voting file refresh on diskgroup GRID
NOTE: Voting file relocation is required in diskgroup GRID

NOTE: Attempting voting file relocation on diskgroup GRID

NOTE: Attempting voting file refresh on diskgroup GRID

经分析，该问题是一个已知的BUG,BUG号为13609187

解决办法，就是打补丁,但生产系统不能随意停，临时解决办法为：

alter diskgroup GRID drop disk <disk_name> rebalance power 0;

alter diskgroup GRID undrop disks;

经验证，通过以上临时办法，解决，暂时不再有新的日志产生。

附 ：ORACLE官方文档：

ASM Alert Logs Show Continuously: Attempting Voting File Relocation (Doc ID 1457886.1)
To Bottom

In this Document
Symptoms
Changes
Cause
Solution
References

Applies to:
Oracle Database Exadata Cloud Machine - Version N/A and later
Oracle Cloud Infrastructure - Database Service - Version N/A and later
Oracle Database Exadata Express Cloud Service - Version N/A and later
Oracle Database Cloud Exadata Service - Version N/A and later
Oracle Database Cloud Schema Service - Version N/A and later
Information in this document applies to any platform.
.
Symptoms
1) Following messages are observed in ASM alert log:
NOTE: Attempting voting file refresh on diskgroup GRID
NOTE: Voting file relocation is required in diskgroup GRID
NOTE: Attempting voting file relocation on diskgroup GRID
NOTE: Attempting voting file refresh on diskgroup GRID
NOTE: Voting file relocation is required in diskgroup GRID
NOTE: Attempting voting file relocation on diskgroup GRID
NOTE: Attempting voting file refresh on diskgroup GRID

2) Also an excessive logging is seen in cssd logs (eg  9 hours of logs are available in all 10 cssd logs under $GRID_HOME/log/<hostname>/cssd)
Most of these logs are related to CLSS_CONFIG_LOCK which is associated with configuration change requests- the most  common use of this is by ASM
2013-11-20 11:01:56.973: [    CSSD][38]clssgmHandleGrockRcfgUpdate: grock(#CLSS_CONFIG_LOCK), updateseq(26543197), status(0), sendresp(1)
2013-11-20 11:01:56.974: [    CSSD][38]clssgmDispatchCMXMSG: received a timed out RPC id(224) from node(2). Discarding.
2013-11-20 11:01:56.979: [    CSSD][5]clssgmExecuteClientRequest: Received group property update request from client (10285fcd0), type 3
........
2013-11-20 20:01:01.011: [    CSSD][38]clssgmFreeRPCIndex: freeing rpc 516
2013-11-20 20:01:01.011: [    CSSD][38]clssgmHandleGrockRcfgUpdate: grock(#CLSS_CONFIG_LOCK), updateseq(27161406), status(0), sendresp(1)
2013-11-20 20:01:01.011: [    CSSD][5]clssgmDiscEndpcl: gipcDestroy 359e018

3) We can see memory leak by RBAL process as below
zzz ***Tue Apr 11 15:00:01 BEIST 2017
USER PID PPID PRI %CPU %MEM VSZ RSS WCHAN S ELAPSED TIME COMMAND
grid 6553650 1 65 0.1 31.0 7633820 7660992 - A 183-05:16:42 4-19:59:13 asm_rbal_+ASM2 <<<<
grid 16842828 1 60 0.0 0.0 3584 32508 - A 183-05:16:44 00:09:26 asm_psp0_+ASM2
# svmon -P 6553650 -O segment=category
Unit: page
-------------------------------------------------------------------------------
Pid Command Inuse Pin Pgsp Virtual
6553650 oracle 2065376 27712 1827215 2071364
...............................................................................
SYSTEM segments Inuse Pin Pgsp Virtual
15923 10608 0 15923
...............................................................................
EXCLUSIVE segments Inuse Pin Pgsp Virtual <<<<
1916224 48 1810942 1916674
...............................................................................
SHARED segments Inuse Pin Pgsp Virtual
133229 17056 16273 138767
Rbal trace file records below
2017-04-12 12:01:26.204: [ CSSCLNT]clsssVoteDiskFormat: call clsscfgfmtbegin with leasedata 0, size 0
2017-04-12 12:01:26.204: [ CSSCLNT]clsssVoteDiskFormat: succ-ly format the Voting Disk
src:1107ffec8:7000100afb82a80:3:
/dev/rhdisk8:3f37656050374f2dbffd751d26675dbc:
2017-04-12 12:01:26.213: [ CSSCLNT]clsssVoteDiskFormat: call clsscfgfmtbegin with leasedata 0, size 0
2017-04-12 12:01:26.213: [ CSSCLNT]clsssVoteDiskFormat: succ-ly format the Voting Disk
src:1107fff30:7000100afb82e18:3:
/dev/rhdisk9:0fb332d276284f9abf5a653d4c66d627:
PST-old:1107ffe60:7000100afb82700:47:
/dev/rhdisk7:aa8985a6f6654f91bf876785474fd6dc:

Changes
Recent upgrade to 11.2.0.3
Cause
Because of bug 13609187, voting file was not getting relocated and fill up the ASM alertlog with the above messages (and of course the FS space)
Similar issue discussed also on bug 13904435.
Solution
The below operation forced the PST to move to all the disks (OCR/Vote DG) and the voting file relocation operation to complete:

alter diskgroup GRID drop disk <disk_name> rebalance power 0;
alter diskgroup GRID undrop disks;


Or check if any patch available for Bug: 13904435 (Fixed by 12.1) and apply to the GI home.
References
BUG:13609187 - NOTE: VOTING FILE RELOCATION IS REQUIRED IN DISKGROUP GRID

BUG:13906539 - TOO MANY OCCURRENCES OF NOTE: ATTEMPTING VOTING FILE REFRESH ON DISKGROUP
NOTE:13448206.8 - Bug 13448206 - Many alert messages "NOTE: attempting voting file .." in RAC with ASM