[公告]ORA-15041 Diskgroup Space Exhausted [ID 1367078.1]

郑全

 

ORA-15041 Diskgroup Space Exhausted [ID 1367078.1]

 

In this Document

   Purpose

   Last Review Date

   Instructions for the Reader

   Troubleshooting Details

      Cause:

      Solution

   References

 

--------------------------------------------------------------------------------

 

Applies to:

Oracle Server - Enterprise Edition - Version: 10.1.0.2 to 11.2.0.3 - Release: 10.1 to 11.2

Information in this document applies to any platform.

 

Purpose

 

You are encountering the following error:

 

ORA-15041: diskgroup space exhausted

 

 The error text is as below:

 

$ oerr ora 15041

15041, 00000, "diskgroup \"%s\" space exhausted"

// *Cause: The diskgroup ran out of space.

// *Action: Add more disks to the diskgroup, or delete some existing files.

 

This error can be seen at the following places:

 

?Console

?ASM alert.log and/or trace files

?During rebalance operation ( ARBx background process trace file will log this error )

?On Database ( RDBMS ) side, in alert.log and trace file

?This error might be accompanied with datafile/logfile/archivefile allocation/extention and RMAN restore etc ( accompanying errors like ORA-01237, ORA-17505 )

 

Last Review Date

 October 14, 2011

Instructions for the Reader

 

A Troubleshooting Guide is provided to assist in debugging a specific issue. When possible, diagnostic tools are included in the document to assist in troubleshooting.

 

Troubleshooting Details

 

Cause:

?This error is seen when allocation units are allocated/moved in a diskgroup. While doing this, ASM will consider ALL the disks in the diskgroup, the size of the disks and the ASM file size getting those allocations. If any one disk not able to satisfy the allocation will result in the error.

?There is ASM Metadata discrepancy

?There is ongoing rebalance operation and/or disk(s) not in CACHED/MEMBER status. There are offline disks.

 

Solution

 

o The first thing you should check if there is any ongoing rebalance. Query gv$asm_operation. If this is going on then allow this to complete. Check the ASM alert.log file(s) as well. If the rebalance encountered error for any reason, the following points should help completing this.

 

o Query v$asm_disk.total_mb and free_mb for each disk in the diskgroup. If any one disk is short of free_mb, then the error might be seen, even if there is sufficient free space in the whole diskgroup.

 

o This is a must that you have all the disks of same size, in a diskgroup. Starting 10.2, the total size of the disk is taken into consideration for allocations. So there will be imbalanced IO to disks. A future task would be to add/drop disks to have all the disks of same size.

 

1] Check if there is real shortage of space in the diskgroup. Check all the disks have similar v$asm_disk,total_mb and free_mb values. Then we need to add disk(s) to the diskgroup. You can use MOS Doc ID 351117.1 to get theh detailed space utilization, both from ASM side and RDBMS side.

 

2] Even after a successful completion of manual rebalance, and with sufficient v$asm_disk.free_mb in each disk, if ORA-15041 is seen then, On the ASM instance:

 

alter diskgroup check all norepair;

 

This can be run safely on active system. Once this command completes, check the ASM alert.log if it reports any "mismatch" of AT and DD. Something like

 

NOTE: disk DATA_00000, used AU total mismatch: DD=32169, AT=32172

 

If yes, then run:

 

alter diskgroup check all repair;

 

This can be run safely on active system and is meant to fix the AT and DD discrepancy. An AT and DD ASM Metadata discrepancy might manifest because of previous failed file allocation in the diskgroup. After the check all repair command, run a manual rebalance

 

alter diskgroup rebalance power n;

 

Allow the rebalance to complete, monitor ASM alert.log and gv$asm_operation. Once this is done, query v$asm_disk.free_mb for each disk.

 

NOTE: Checking the free_mb for each disk and periodically running check all norepair on the diskgroup to check AT and DD mismatch can be included in the system monitoring scripts.

 

3] If any one or more disk in the diskgroup has very less space ( say < 300 MB ), then manual rebalance/add of new disks will not help. Because rebalance operation needs some amount of free space in each disk. Also add of a new disk will invoke an automatic rebalance that will also fail with ORA-15041. In this case, you should look to remove/move some files from the diskgroup. Check for obsolete RMAN/dump files, temp files etc. Once we remove/move some files, from the diskgroup, query v$asm_disk.free_mb for each disk. If the free_mb is say > 300 MB, run a manual rebalance ( or add disk )

 

alter diskgroup rebalance power ;

 

4] If you are on 11gR1 ASM, after considering the above points, refer to MOS Doc ID 813013.1

 

Additional Resources

 

Community Discussions: Storage Management

 

Still have questions? Use the above community to search for similar discussions or start a new discussion on this subject.

 

References

 NOTE:351117.1 - Information to gather when diagnosing ASM space issues

 NOTE:813013.1 - FREE_MB in Same Size Disks Differ Even After Successful Rebalance