ALERT: Disable Transparent HugePages on SLES11, RHEL6, RHEL7, OL6, OL7, and U...

郑全

In this Document
In this Document

Description

Occurrence
Symptoms
Workaround
Patches
History
References

---

Applies to:   Oracle Database - Enterprise Edition
Oracle Database - Standard Edition
Linux x86
Linux x86-64

DescriptionStarting with RedHat6, RedHat7, OL6, OL7 SLES11 and UEK2 kernels, Transparent HugePages are implemented and enabled (default) in an attempt to improve the memory management.  Transparent HugePages are similar to the HugePages that have been available in previous Linux releases.  The main difference is that the Transparent HugePages are set up dynamically at run time by the khugepaged thread in kernel while the regular HugePages had to be preallocated at the boot up time.


Because Transparent HugePages are known to cause unexpected node reboots and performance problems with RAC, Oracle strongly advises to disable the use of Transparent HugePages. In addition, Transparent Hugepages may cause problems even in a single-instance database environment with unexpected performance problems or delays. As such, Oracle recommends disabling Transparent HugePages on all Database servers running Oracle.

Note:  This ONLY applies to the new feature Transparent HugePages, Oracle highly recommends the use of standard HugePages that were recommended for previous releases of Linux.  See Document 361323.1 for additional information on HugePages.

Note: on UEK2 and above, check the existence of the /sys/kernel/mm/transparent_hugepage/ directory.  If this directory does not exist, then the transparent hugepage is removed from the kernel, so there is no need to disable the transparent hugepage

Occurrence Starting with SLES11, RHEL6, RHEL7, OL6, OL7 and UEK2 Kernels, Transparent HugePages are introduced and enabled by default, and this can cause node reboots and performance problems.

On UEK2 and above, the transparent hugepages is already removed if /sys/kernel/mm/transparent_hugepage/ directory directory does not exist.  In that case, there is no further need to disable the transparent hugepages.
SymptomsNode reboots.
The ocssd.log may show some of the threads are blocked (but this does not show all the time):
2013-05-01 14:30:45.255: [    CSSD][224204544]clssscMonitorThreads clssnmvKillBlockThread not scheduled for 7500 msecs
2013-05-01 14:30:46.945: [    CSSD][224204544]clssscMonitorThreads clssnmvWorkerThread not scheduled for 8030 msecs

WorkaroundThe information provided here for Redhat and SLES needs to be confirmed by Redhat and SLES support because Redhat and SLES support may provide a different ways to disable the transparent hugepages.
Please contact Redhat and SLES support on how to disable the transparent hugepages on their respective platform.

The workaround provided here may not work for newer versions like Redhat 7 or SLES12.  If the following workaround does not work, please contact Redhat or SLES (SUSE) for actual instruction on how to disable the transparent hugepages.
For example, Redhat published how to disable transparent hugepates in Redhat 7 because the instruction is now different.  Please refer to
https://access.redhat.com/solutions/1320153

To check if the Transparent HugePages are enabled in your server execute the following:

Default/Enabled setting is  [always]:

# cat /sys/kernel/mm/transparent_hugepage/enabled
[always] never

Note 1: For RHEL kernel, the file path is different from above:
# cat /sys/kernel/mm/redhat_transparent_hugepage/enabled
[always] never
  
Please modify this file accordingly.

Note 2: For UEK2 kernel, as of 2.6.39-400.116.0 Transparent Hugepages has been removed from the kernel.  
If it is not compiled into the kernel then /sys/kernel/mm/transparent_hugepage will not exist.

Disabled setting is [never]:
# cat /sys/kernel/mm/transparent_hugepage/enabled
always [never]
If "enabled" is NOT set to "[never]", the Transparent HugePages are being used.
You can also issue:
# grep AnonHugePages /proc/meminfo
If the output contains a line like "AnonHugepages: xxxx kB", with a value > 0kB, the kernel is using Transparent HugePages.
Because the kernel currently uses Transparent HugePages only for the anonymous memory blocks like stack and heap, the value of AnonHugepages in /proc/meminfo is the current amount of Transparent HugePages that the kernel is using.

To disable Transparent HugePages boot time either one of the following 2 methods may be used:

Add the following to the kernel boot line in /etc/grub.conf (a symlink to /boot/grub/grub.conf) and reboot the server (this is the preferred method):
transparent_hugepage=never
Once modified the line will read similar to the following example:
title Oracle Linux Server (2.6.32-300.25.1.el6uek.x86_64)
        root (hd0,0)
       kernel /vmlinuz-2.6.32-300.25.1.el6uek.x86_64 ro root=LABEL=/ transparent_hugepage=never
        initrd /initramfs-2.6.32-300.25.1.el6uek.x86_64.img


OR

Add the following lines in /etc/rc.local and reboot the server (this still can be done on Redhat 7 although rc.local is being deprecated):

if test -f /sys/kernel/mm/transparent_hugepage/enabled; then   
  echo never > /sys/kernel/mm/transparent_hugepage/enabled
  fi

if test -f /sys/kernel/mm/transparent_hugepage/defrag; then   
  echo never > /sys/kernel/mm/transparent_hugepage/defrag
fi


Please change the file path for RHEL kernel to /sys/kernel/mm/redhat_transparent_hugepage/ accordingly.

===========================================================================
The following comment was received about SLES11 SP3.  

Please note that this is not tested, so this is provided only as a talking point to your sys admin who knows SLES11 SP3 and who should also contact SLES for correct information.


in SLES11 SP3 with YAST/GRUB boot loader /etc/grub.conf overrided usind YAST. parameter transparent_hugepage=never must be set in YAST-bootloader- Edit settings - in line: optional kernel parameter [before showopts].


Patches
none

History

06/05/2013 the alert is written
06/05/2013 the alert is published
06/06/2013 removed "cat /sys/kernel/mm/transparent_hugepage/enabled" from the section where we check the transparent hugepages are disabled.
06/06/2013 changed "Starting SLES11..." to 'Starting with SLES11"
06/07/2013 changed title to include "ALERT", modified to cover SIDB, corrected typo in 2nd solution, expanded the solution for disable via kernel boot line and validation via AnonHugePages.
11/26/2013 Include file path /sys/kernel/mm/redhat_transparent_hugepage/enabled for RHEL kernel and changes for UEK2 kernel

郑全

在rhel 6,7,如果要使用大页内存,就要关闭这个透明大页内存,在oel 6,7,已经去掉了这个,所以oel不存在这个问题.

rhel7以上版本要处理,步骤可以参考下面连接:
http://bbs.cqsztech.com/forum.ph ... id=17293&extra=