Oracle RAC Assessment Report

System Health Score is derived using following formula.

Every check has 10 points
Critical will deduct 10 points
Failure will deduct 10 points
Warning will deduct 5 points
Info will deduct 3 points
Skip will deduct 3 points

Total checks	231
Passed checks	161
Failed(fail/warn/info/skip) checks	70

..Hide

Report Feature

Show Failed checks only

Show checks with the following status:

Critical Fail Warning Info Pass

Show details of the following regions:

Maximum Availability Architecture (MAA) Scorecard
Patch Recommendation
Component Elapsed Times
Top 10 Time Consuming Checks

Show details of the checks:

Expand AllCollapse All

Show Check Ids

Remove finding from report

Check Id	Status	Type	Message	Status On	Details
F9519FD7525B4F01E04313C0E50AC339	FAIL	OS Check	One or more network card used by clusterware do not have HOTPLUG=NO	All Database Servers	View Hide

CB95A1BF5B1160ACE0431EC0E50A12EE	FAIL	SQL Check	Table AUD$[FGA_LOG$] should use Automatic Segment Space Management	All Databases	View Hide

CCF6F44765861F7AE0431EC0E50A72AD	FAIL	OS Check	Operating system hugepages count does not satisfy total SGA requirements	All Database Servers	View Hide

5C413530675BC12CE053D498EB0A3C84	FAIL	OS Check	The RMAN snapshot controlfile location is NOT shared on all database nodes in the cluster	All Database Servers	View Hide

E629958349EF3B7CE04313C0E50AC79D	FAIL	ASM Check	ASM_DISKSTRING parameter is either null or set to /dev/*	All ASM Instances	View Hide

429C554ED92E4F54E0530D98EB0AE367	FAIL	OS Check	Database parameter CLUSTER_INTERCONNECTS is not set to the recommended value	All Database Servers	View Hide

03EB4F8789A15B34E05312C0E50A7AF2	FAIL	OS Check	Bash is vulnerable to code injection (CVE-2014-6271)	All Database Servers	View Hide

F7394F6EBAD73B84E04313C0E50A08FA	WARNING	SQL Check	Consider investigating the frequency of SGA resize operations and take corrective action	All Databases	View Hide

F723B279BA9218A5E04313C0E50A1B01	WARNING	SQL Check	Consider investigating changes to the schema objects such as DDLs or new object creation	All Databases	View Hide

DC4495442D7A0CEBE04313C0E50A76E8	WARNING	OS Check	Package unixODBC-devel-2.2.14-11.el6-x86_64 is recommended but NOT installed	All Database Servers	View Hide

C1D1B240993425B8E0431EC0E50AFEF5	WARNING	OS Check	Package unixODBC-devel-2.2.14-11.el6-i686 is recommended but NOT installed	All Database Servers	View Hide

C1D0BD14BF4A3BCEE0431EC0E50A9DB5	WARNING	OS Check	Package unixODBC-2.2.14-11.el6-i686 is recommended but NOT installed	All Database Servers	View Hide

ADA4E48033A57D05E040E50A1EC0320D	WARNING	OS Check	CSS log file size has not been increased from default (50 MB) .	All Database Servers	View Hide

E1343A1CE2383400E04313C0E50A0F82	WARNING	ASM Check	All disk groups should have compatible.asm attribute set to recommended values	All ASM Instances	View Hide

53BAB95BC9B1D1D4E0530B98EB0AD1BA	WARNING	ASM Check	All disk groups should have compatible.advm attribute set to recommended values	All ASM Instances	View Hide

3B36CD93FCE82634E0530C98EB0A543F	WARNING	ASM Check	All disk groups should have compatible.rdbms attribute set to recommended values	All ASM Instances	View Hide

951C025701C65CC5E040E50A1EC0371F	WARNING	OS Check	OSWatcher is not running as is recommended.	All Database Servers	View Hide

8C9D63D9441C1F52E040E50A1EC0211F	WARNING	OS Check	NIC bonding is NOT configured for public network (VIP)	All Database Servers	View Hide

5EA8F4C6C6BDF8F0E0401490CACF067F	WARNING	OS Check	NIC bonding is not configured for interconnect	All Database Servers	View Hide

CB94D8434AA02210E0431EC0E50A7C40	WARNING	SQL Parameter Check	Database Parameter memory_target is not set to the recommended value	All Instances	View Hide

856A9B77AF14DD9FE040E50A1EC00285	WARNING	OS Check	SELinux is being Enforced.	All Database Servers	View Hide

833F68D88AE57B7CE040E50A1EC02BE7	WARNING	SQL Check	One or more redo log groups are not multiplexed	All Databases	View Hide

833F12C25516ACAFE040E50A1EC020F7	WARNING	SQL Check	Controlfile is NOT multiplexed	All Databases	View Hide

7EDDA570A1827FBAE040E50A1EC02EB1	WARNING	OS Check	$ORACLE_HOME/bin/oradism setuid bit is NOT set	All Database Servers	View Hide

65414495B2047F26E0401490CACF0FED	WARNING	OS Check	OCR is not being backed up daily	All Database Servers	View Hide

4826A4AB0A937E20E0530E98EB0ACBF3	WARNING	Patch Check	Oracle patch 19855835 is not applied on RDBMS_HOME	All Homes	View Hide

470D87B05DCBAFE5E0530E98EB0A11FD	WARNING	Patch Check	Oracle patch 20907061 is not applied on RDBMS_HOME	All Homes	View Hide

46E9E21E3FDB771EE0530D98EB0A0202	WARNING	Patch Check	Oracle patch 20476175 is not applied on RDBMS_HOME	All Homes	View Hide

46E9C37D792A0073E0530A98EB0A4E6F	WARNING	Patch Check	Oracle patch 20879889 is not applied on RDBMS_HOME	All Homes	View Hide

46E0AFAA5830BAFCE0530E98EB0A7EC1	WARNING	Patch Check	Oracle patch 19174639 is not applied on RDBMS_HOME	All Homes	View Hide

46D3EDCB2B3FCC87E0530D98EB0A46EF	WARNING	Patch Check	Oracle patch 24739928 is not applied on RDBMS_HOME	All Homes	View Hide

4670602AF0A5AA5BE0530D98EB0AE0EB	WARNING	Patch Check	Oracle patch 18498878 is not applied on RDBMS_HOME	All Homes	View Hide

467047CD7B6AE4F5E0530A98EB0AED0D	WARNING	Patch Check	Oracle patch 22113854 is not applied on RDBMS_HOME	All Homes	View Hide

45E6969F6C7CE26DE0530C98EB0A8166	WARNING	Patch Check	Oracle patch 23665623 is not applied on RDBMS_HOME	All Homes	View Hide

07A76669BB985B6CE05313C0E50ADA40	WARNING	SQL Check	There are some application objects with STALE statistics.	All Databases	View Hide

018D62D990EE516DE05313C0E50A49DB	WARNING	OS Check	Cluster Health Monitor (CHM) repository does not provide recommended level of retention	All Database Servers	View Hide

E850552FBF2B1871E04313C0E50AA887	INFO	OS Check	Consider increasing the COREDUMPSIZE size	All Database Servers	View Hide

DC3D819F5D2A50FEE04312C0E50AFF9F	INFO	OS Check	Parallel Execution Health-Checks and Diagnostics Reports	All Database Servers	View Hide

BBB4357BF09B79D6E0431EC0E50AFB57	INFO	OS Check	Information about hanganalyze and systemstate dump	All Database Servers	View Hide

F1C3BD95DE041056E04313C0E50A84B8	INFO	Patch Check	IMPORTANT: Oracle Database Patch 17478514 PSU is NOT applied to RDBMS Home	All Homes	View Hide

5E4956EE574FB034E0401490CACF2F84	INFO	OS Check	Jumbo frames (MTU >= 9000) are not configured for interconnect	All Database Servers	View Hide

8914F5D0A9AB85BAE040E50A1EC04A31	INFO	OS Check	Number of SCAN listeners is not equal to the recommended number of 3.	All Database Servers	View Hide

8347749B14C41005E040E50A1EC03F2E	INFO	ASM Check	oracleasm (asmlib) module is not loaded	All ASM Instances	View Hide

831B9FABDB6CFCB4E040E50A1EC034C0	INFO	OS Check	audit_file_dest has audit files older than 30 days	All Database Servers	View Hide

6890329C1FFFCEDDE040E50A1EC02FED	INFO	OS Check	At some times checkpoints are not being completed	All Database Servers	View Hide

66EBC49E368387CAE040E50A1EC03B98	INFO	OS Check	background_dump_dest has files older than 30 days	All Database Servers	View Hide

66E6B013BAE3EFBEE040E50A1EC01F87	INFO	OS Check	user_dump_dest has trace files older than 30 days	All Database Servers	View Hide

6556EAA74E28214FE0401490CACF6C89	INFO	OS Check	$CRS_HOME/log/hostname/client directory has too many older log files.	All Database Servers	View Hide

12D2EC59B5677B0BE05312C0E50AAF78	INFO	OS Check	Most recent ADR incidents	All Database Servers	View Hide

0585FC662BC556DCE05312C0E50AB518	INFO	OS Check	Important Storage Minimum Requirements for Grid & Database Homes	All Database Servers	View Hide

02DE6186B8040599E05312C0E50A76F6	INFO	ASM Check	Oracle Data Pump Best practices.	All ASM Instances	View Hide

0257EE4166C43F08E05312C0E50A7D8E	INFO	OS Check	Important Automatic Storage Management (ASM) Notes and Technical White Papers	All Database Servers	View Hide

E47ECDCFE09A122CE04313C0E50A35EC	PASS	OS Check	There are no duplicate parameter entries in the database init.ora(spfile) file	All Database Servers	View Hide

2FE891F7E8F6F03AE0530D98EB0A1F67	PASS	OS Check	Database parameter service_names is not defined in spfile	All Database Servers	View Hide

F8BFFF523E81644EE04312C0E50A53BA	PASS	OS Check	kernel.panic_on_oops parameter is configured according to recommendation	All Database Servers	View Hide

F733E8274650079DE04312C0E50AEF50	PASS	SQL Check	No waiter sessions found currently for application users	All Databases	View Hide

F723F251083F31B5E04312C0E50A6BFB	PASS	SQL Check	The database parameter session_cached_cursors is set to the recommended value	All Databases	View Hide

F02293F7261D1BCAE04312C0E50A4118	PASS	ASM Check	ASM disk permissions are set as recommended	All ASM Instances	View Hide

F01E3FEDBD2B243EE04312C0E50A4DC5	PASS	ASM Check	ASM disks have enough free space for rebalance	All ASM Instances	View Hide

ED58D80CE0FA4B48E04313C0E50A91B9	PASS	SQL Check	All optimizer related parameters are set to the default value (Parameter Set 2 of 3)	All Databases	View Hide

ED57C07F184E4B79E04312C0E50AB237	PASS	SQL Check	All optimizer related parameters are set to the default value (Parameter Set 3 of 3)	All Databases	View Hide

ED4F493804CE75EBE04313C0E50ABCF8	PASS	SQL Check	No obsolete initialization parameters are set	All Databases	View Hide

EC1D5CE0B4371BB6E04312C0E50A5C3A	PASS	SQL Check	Database parameter OPTIMIZER_FEATURES_ENABLE is set to the current database version	All Databases	View Hide

EC1BDA4542E327FDE04312C0E50A5BB6	PASS	SQL Check	Database parameter NLS_SORT is set to BINARY	All Databases	View Hide

E96A3ED23AC91994E04312C0E50ABD03	PASS	OS Check	Online (hot) patches are not applied to CRS_HOME.	All Database Servers	View Hide

E960DB20CA5A634FE04312C0E50A62E0	PASS	SQL Check	Table containing SecureFiles LOB storage belongs to a tablespace with extent allocation type that is SYSTEM managed (AUTOALLOCATE)	All Databases	View Hide

E94AC6ACDA502F3BE04312C0E50A290A	PASS	ASM Check	ASM memory_target is set to recommended value	All ASM Instances	View Hide

E94589BC1AC24CFBE04312C0E50A3849	PASS	OS Check	Online (hot) patches are not applied to ORACLE_HOME	All Database Servers	View Hide

E84C19843A7E7CA2E04313C0E50A3ED0	PASS	SQL Check	All optimizer related parameters are set to the default value (Parameter Set 1 of 3)	All Databases	View Hide

E7E213A6D9151EEBE04312C0E50A85C3	PASS	OS Check	Berkeley Database location points to correct GI_HOME	All Database Servers	View Hide

E1DF2A6140395D42E04312C0E50A0A6C	PASS	ASM Check	All diskgroups from v$asm_diskgroups are registered in clusterware registry	All ASM Instances	View Hide

E18D7F9837B7754EE04313C0E50AD4AA	PASS	OS Check	Package cvuqdisk-1.0.9-1-x86_64 meets or exceeds recommendation	All Database Servers	View Hide

E12A91DC10F31AD7E04312C0E50A6361	PASS	OS Check	pam_limits configured properly for shell limits	All Database Servers	View Hide

E10E99868C34569BE04313C0E50A44C1	PASS	OS Check	vm.min_free_kbytes is set as recommended.	All Database Servers	View Hide

DCB4C2CB907F4C76E04312C0E50A7667	PASS	OS Check	Linux transparent huge pages are disabled	All Database Servers	View Hide

D348A289DD032396E0431EC0E50A26D5	PASS	OS Check	OCR and Voting disks are stored in ASM	All Database Servers	View Hide

D0C2640EBA071F73E0431EC0E50AA159	PASS	OS Check	System clock is synchronized to hardware clock at system shutdown	All Database Servers	View Hide

DC28F07D94FD1B10E04313C0E50A9FD8	PASS	OS Check	TFA Collector is installed and running	All Database Servers	View Hide

DBC2C9218542349FE04312C0E50AC1E9	PASS	OS Check	No clusterware resource are in unknown state	All Database Servers	View Hide

D9A5C0E2DE430A85E04312C0E50AC8B0	PASS	ASM Check	No corrupt ASM header blocks indicated in ASM alert log (ORA-15196 errors)	All ASM Instances	View Hide

D957C871B811597AE04312C0E50A91BF	PASS	ASM Check	No disks found which are not part of any disk group	All ASM Instances	View Hide

D692927211732642E0431EC0E50A342B	PASS	OS Check	loopback address is configured as recommended in /etc/hosts	All Database Servers	View Hide

D35CE19AE68165F3E0431EC0E50A4C09	PASS	OS Check	Redo log write time is less than 500 milliseconds	All Database Servers	View Hide

CB5BD768E88F7F71E0431EC0E50A346F	PASS	OS Check	Package libgcc-4.4.4-13.el6-x86_64 meets or exceeds recommendation	All Database Servers	View Hide

6B515A724AB85906E040E50A1EC039F6	PASS	SQL Check	No read/write errors found for ASM disks	All Databases	View Hide

C1D39B834AA46E44E0431EC0E50A5366	PASS	OS Check	Package sysstat-9.0.4-11.el6-x86_64 meets or exceeds recommendation	All Database Servers	View Hide

C1D39B834AA36E44E0431EC0E50A5366	PASS	OS Check	Package libgcc-4.4.4-13.el6-i686 meets or exceeds recommendation	All Database Servers	View Hide

C1D34D17A4F45402E0431EC0E50A5DD9	PASS	OS Check	Package binutils-2.20.51.0.2-5.11.el6-x86_64 meets or exceeds recommendation	All Database Servers	View Hide

C1D348AB978E3873E0431EC0E50A19F0	PASS	OS Check	Package glibc-2.12-1.7.el6-x86_64 meets or exceeds recommendation	All Database Servers	View Hide

C1D30E313A4C0B0BE0431EC0E50A1931	PASS	OS Check	Package libstdc++-4.4.4-13.el6-x86_64 meets or exceeds recommendation	All Database Servers	View Hide

C1D2A95C2BF31FE4E0431EC0E50AB101	PASS	OS Check	Package libstdc++-4.4.4-13.el6-i686 meets or exceeds recommendation	All Database Servers	View Hide

C1D29B4860DA19C2E0431EC0E50AFB36	PASS	OS Check	Package glibc-2.12-1.7.el6-i686 meets or exceeds recommendation	All Database Servers	View Hide

C1D29B4860D919C2E0431EC0E50AFB36	PASS	OS Check	Package gcc-4.4.4-13.el6-x86_64 meets or exceeds recommendation	All Database Servers	View Hide

C1D1CC4D830F3B90E0431EC0E50A559F	PASS	OS Check	Package make-3.81-19.el6 meets or exceeds recommendation	All Database Servers	View Hide

C1D1CC4D830E3B90E0431EC0E50A559F	PASS	OS Check	Package libstdc++-devel-4.4.4-13.el6-i686 meets or exceeds recommendation	All Database Servers	View Hide

C1D1BA6C1CD213F9E0431EC0E50A8B9C	PASS	OS Check	Package libaio-devel-0.3.107-10.el6-x86_64 meets or exceeds recommendation	All Database Servers	View Hide

C1D1BA6C1CD013F9E0431EC0E50A8B9C	PASS	OS Check	Package libaio-0.3.107-10.el6-x86_64 meets or exceeds recommendation	All Database Servers	View Hide

C1D1B240991A25B8E0431EC0E50AFEF5	PASS	OS Check	Package compat-libstdc++-33-3.2.3-69.el6-i686 meets or exceeds recommendation	All Database Servers	View Hide

C1D1973D1B4C0EA1E0431EC0E50A9108	PASS	OS Check	Package glibc-devel-2.12-1.7.el6-x86_64 meets or exceeds recommendation	All Database Servers	View Hide

C1D15659D96376CBE0431EC0E50A74F5	PASS	OS Check	Package glibc-devel-2.12-1.7.el6-i686 meets or exceeds recommendation	All Database Servers	View Hide

C1D15659D96276CBE0431EC0E50A74F5	PASS	OS Check	Package compat-libcap1-1.10-1-x86_64 meets or exceeds recommendation	All Database Servers	View Hide

C1D0EE98B4BC4083E0431EC0E50ADCB2	PASS	OS Check	Package ksh-20100621-12.el6-x86_64 meets or exceeds recommendation	All Database Servers	View Hide

C1D0BD14BF493BCEE0431EC0E50A9DB5	PASS	OS Check	Package libaio-0.3.107-10.el6-i686 meets or exceeds recommendation	All Database Servers	View Hide

C1D0BD14BF483BCEE0431EC0E50A9DB5	PASS	OS Check	Package libstdc++-devel-4.4.4-13.el6-x86_64 meets or exceeds recommendation	All Database Servers	View Hide

C1CF431B59054969E0431EC0E50A9B88	PASS	OS Check	Package gcc-c++-4.4.4-13.el6-x86_64 meets or exceeds recommendation	All Database Servers	View Hide

C1CF431B59034969E0431EC0E50A9B88	PASS	OS Check	Package compat-libstdc++-33-3.2.3-69.el6-x86_64 meets or exceeds recommendation	All Database Servers	View Hide

C1CEC9D9E9432BDFE0431EC0E50AF329	PASS	OS Check	Package libaio-devel-0.3.107-10.el6-i686 meets or exceeds recommendation	All Database Servers	View Hide

89130F49748E6CC7E040E50A1EC07A44	PASS	OS Check	Remote listener is set to SCAN name	All Database Servers	View Hide

65F8FA5F9B838079E040E50A1EC059DC	PASS	OS Check	Value of remote_listener parameter is able to tnsping	All Database Servers	View Hide

D6972E101386682AE0431EC0E50A9FD9	PASS	OS Check	No server-side TNSNAMES.ora aliases resolve using SCAN EZ syntax	All Database Servers	View Hide

D6972E101384682AE0431EC0E50A9FD9	PASS	OS Check	ezconnect is configured in sqlnet.ora	All Database Servers	View Hide

BD7E4DDEE2B06BABE0431EC0E50AED9B	PASS	SQL Check	No Duplicate objects were found in the SYS and SYSTEM schemas	All Databases	View Hide

BD53C41D0F1102DDE0431EC0E50A5B4C	PASS	SQL Check	No Invalid Registry Components were found	All Databases	View Hide

B167E5248D476B74E0431EC0E50A3E27	PASS	SQL Check	All bigfile tablespaces have non-default maxbytes values set	All Databases	View Hide

AD6481CF9BDD6058E040E50A1EC021EC	PASS	OS Check	umask for RDBMS owner is set to 0022	All Database Servers	View Hide

A18401E2DDFF63D2E040E50A1EC077C4	PASS	OS Check	ORA_CRS_HOME environment variable is not set for Grid infrastructure owner	All Database Servers	View Hide

9EB883858764342DE040E50A1EC0359A	PASS	OS Check	Oracle database software owner hard stack shell limit is configured according to recommendation	All Database Servers	View Hide

9EB848CDD040ACADE040E50A1EC01840	PASS	OS Check	Oracle database software owner soft nofile shell limits is configured according to recommendation	All Database Servers	View Hide

9EB84272626BFF6EE040E50A1EC02AF3	PASS	OS Check	Oracle database software owner hard nproc shell limits is configured according to recommendation	All Database Servers	View Hide

64DC3E59CB88B984E0401490CACF1104	PASS	SQL Parameter Check	asm_power_limit is set to recommended value of 1	All Instances	View Hide

90DCECE833790E9DE040E50A1EC0750A	PASS	OS Check	CSS reboot time is set to the default value of 3	All Database Servers	View Hide

90DCB860F9380638E040E50A1EC07248	PASS	OS Check	CSS disktimeout is set to the default value of 200	All Database Servers	View Hide

8E1B5EE973BAA8C6E040E50A1EC0622E	PASS	OS Check	ohasd Log Ownership is Correct (root root)	All Database Servers	View Hide

8E1A46CB0BDA0608E040E50A1EC022CD	PASS	OS Check	ohasd/orarootagent_root Log Ownership is Correct (root root)	All Database Servers	View Hide

8E197A76D887BAC4E040E50A1EC07E0B	PASS	OS Check	crsd/orarootagent_root Log Ownership is Correct (root root)	All Database Servers	View Hide

8E19457488167806E040E50A1EC00310	PASS	OS Check	crsd Log Ownership is Correct (root root)	All Database Servers	View Hide

898E1DF96754C57FE040E50A1EC03224	PASS	ASM Check	CRS version is higher or equal to ASM version.	All ASM Instances	View Hide

8915B823FCEBC259E040E50A1EC04AD6	PASS	OS Check	Local listener init parameter is set to local node VIP	All Database Servers	View Hide

87604C73D768DF7AE040E50A1EC0566B	PASS	OS Check	All voting disks are online	All Database Servers	View Hide

90E150135F6859C4E040E50A1EC01FF5	PASS	OS Check	CSS misscount is set to the default value of 30	All Database Servers	View Hide

85F282CFD5DADCB4E040E50A1EC01BC9	PASS	SQL Check	All redo log files are of same size	All Databases	View Hide

8529D3798EA039F3E040E50A1EC07218	PASS	OS Check	Public interface is configured and exists in OCR	All Database Servers	View Hide

84C193C69EE36512E040E50A1EC06466	PASS	OS Check	ip_local_port_range is configured according to recommendation	All Database Servers	View Hide

84BE8B9C4817090DE040E50A1EC07DB8	PASS	OS Check	kernel.shmmax parameter is configured according to recommendation	All Database Servers	View Hide

84BE4DE1F00AD833E040E50A1EC07771	PASS	OS Check	Kernel Parameter fs.file-max is configuration meets or exceeds recommendation	All Database Servers	View Hide

841FD604C3C8F2B1E040E50A1EC0122F	PASS	OS Check	Free space in /tmp directory meets or exceeds recommendation of minimum 1GB	All Database Servers	View Hide

841F0977B92F0185E040E50A1EC070BB	PASS	OS Check	Oracle database software owner soft nproc shell limit is configured according to recommendation	All Database Servers	View Hide

841A3A9F4A74AC6AE040E50A1EC03FC0	PASS	OS Check	Oracle database software owner hard nofile shell limit is configured according to recommendation	All Database Servers	View Hide

83C301ACFF203C9BE040E50A1EC067EB	PASS	OS Check	Linux swap configuration meets or exceeds recommendation	All Database Servers	View Hide

834835A4EC032658E040E50A1EC056F6	PASS	OS Check	/tmp is on a dedicated filesystem	All Database Servers	View Hide

8343C0D6A9D8702BE040E50A1EC045C8	PASS	SQL Check	All data and temporary files are autoextensible	All Databases	View Hide

7EDE9EBEC9429FBAE040E50A1EC03AED	PASS	OS Check	$ORACLE_HOME/bin/oradism ownership is root	All Database Servers	View Hide

77029A014E159389E040E50A1EC02060	PASS	SQL Check	Avg message sent queue time on ksxp is <= recommended	All Databases	View Hide

770244572FC70393E040E50A1EC01299	PASS	SQL Check	Avg message sent queue time is <= recommended	All Databases	View Hide

7701CFDB2F6EF98EE040E50A1EC00573	PASS	SQL Check	Avg message received queue time is <= recommended	All Databases	View Hide

7674FEDB08C2FDA2E040E50A1EC0156F	PASS	SQL Check	No Global Cache lost blocks detected	All Databases	View Hide

7674C09669C5BCE6E040E50A1EC011E5	PASS	SQL Check	Failover method (SELECT) and failover mode (BASIC) are configured properly	All Databases	View Hide

670FE09A93E12317E040E50A1EC018E9	PASS	SQL Check	Avg GC CURRENT Block Receive Time Within Acceptable Range	All Databases	View Hide

670FE09A93E02317E040E50A1EC018E9	PASS	SQL Check	Avg GC CR Block Receive Time Within Acceptable Range	All Databases	View Hide

66FEB2848B21DB24E040E50A1EC00A0C	PASS	SQL Check	Tablespace allocation type is SYSTEM for all appropriate tablespaces	All Databases	View Hide

66EABE4A113A3B1EE040E50A1EC006B2	PASS	OS Check	Alert log is not too big	All Database Servers	View Hide

66EAB3BB6CF79C54E040E50A1EC06084	PASS	OS Check	No ORA-07445 errors found in alert log	All Database Servers	View Hide

66E70B43167837ABE040E50A1EC02FEA	PASS	OS Check	No ORA-00600 errors found in alert log	All Database Servers	View Hide

66E59E657BFC85F4E040E50A1EC0501D	PASS	OS Check	core_dump_dest does not have too many older core dump files	All Database Servers	View Hide

669862F59599CA2AE040E50A1EC018FD	PASS	OS Check	Kernel Parameter SEMMNS OK	All Database Servers	View Hide

66985D930D2DF070E040E50A1EC019EB	PASS	OS Check	Kernel Parameter kernel.shmmni OK	All Database Servers	View Hide

6697946779AC8AD3E040E50A1EC03C0E	PASS	OS Check	Kernel Parameter SEMMSL OK	All Database Servers	View Hide

6696C7B368784A66E040E50A1EC01B92	PASS	OS Check	Kernel Parameter SEMMNI OK	All Database Servers	View Hide

66959FC16B423896E040E50A1EC07CDC	PASS	OS Check	Kernel Parameter SEMOPM OK	All Database Servers	View Hide

6694F204EE47A92DE040E50A1EC07145	PASS	OS Check	Kernel Parameter kernel.shmall OK	All Database Servers	View Hide

6580DCAAE8A28F5BE0401490CACF6186	PASS	OS Check	The number of async IO descriptors is sufficient (/proc/sys/fs/aio-max-nr)	All Database Servers	View Hide

64C6C337366EA439E0401490CACF4D07	PASS	SQL Check	There are no Invalid Java Objects in the database.	All Databases	View Hide

6050196F644254BDE0401490CACF203D	PASS	OS Check	net.core.rmem_max is Configured Properly	All Database Servers	View Hide

60500BAFB377E3ADE0401490CACF2245	PASS	SQL Parameter Check	Instance is using spfile	All Instances	View Hide

5F01FA5CCC69964DE053D298EB0A992B	PASS	ASM Check	ASM Audit file destination file count <= 100,000	All ASM Instances	View Hide

5E5B7EEA0010DC6BE0401490CACF3B82	PASS	OS Check	Interconnect is configured on non-routable network addresses	All Database Servers	View Hide

5DC831A764D8711BE0401490CACF7D52	PASS	OS Check	OCR is mirrored using Oracle redundancy	All Database Servers	View Hide

5DC7EBCB6B72E046E0401490CACF321A	PASS	OS Check	None of the hostnames contains an underscore character	All Database Servers	View Hide

5ADE14B5205111D1E0401490CACF673B	PASS	OS Check	net.core.rmem_default Is Configured Properly	All Database Servers	View Hide

5ADD88EC8E0AFF2EE0401490CACF0C10	PASS	OS Check	net.core.wmem_max Is Configured Properly	All Database Servers	View Hide

5ADCECF64757E914E0401490CACF4BBD	PASS	OS Check	net.core.wmem_default Is Configured Properly	All Database Servers	View Hide

597839F837D44230E053D198EB0A7E73	PASS	SQL Check	cluster_database_instances database parameter is set to default value	All Databases	View Hide

596164523FF4C849E0401490CACF7EE3	PASS	SQL Check	ARCHIVELOG mode is enabled	All Databases	View Hide

4D0E759B219F24C3E0530A98EB0A6C08	PASS	OS Check	None of the hostnames contains an underscore character	All Database Servers	View Hide

4B8B98A9C9644FADE0401490CACF6528	PASS	SQL Check	SYS.AUDSES$ sequence cache size >= 10,000	All Databases	View Hide

4B881724781BB7BEE0401490CACF59FD	PASS	SQL Check	SYS.IDGEN1$ sequence cache size >= 1,000	All Databases	View Hide

4AD656FDB01EAAE1E0530D98EB0A84CF	PASS	OS Check	OCR and OCR backup locations are not in the same path	All Database Servers	View Hide

494DAA11615B67D6E0530D98EB0A34D3	PASS	OS Check	Private and public network subnet configuration is correct in OCR	All Database Servers	View Hide

39CE33CDAC4B81EDE0530C98EB0ACBAA	PASS	OS Check	Cluster nodes appear to be correctly numbered	All Database Servers	View Hide

2F1499DF66C0C395E0530D98EB0AF121	PASS	SQL Check	AWR Space consumption is less than 50% of total SYSAUX space	All Databases	View Hide

2EBB3345399525E2E0530A98EB0AA27A	PASS	SQL Check	Failover method (SELECT) and failover mode (BASIC) are configured properly	All Databases	View Hide

218A2D0660AD1EAAE0530C98EB0A1191	PASS	OS Check	/boot has enough free space	All Database Servers	View Hide

20FC8CAC7445DF10E0530E98EB0A09AF	PASS	OS Check	Oracle database software owner soft stack shell limit is configured according to recommendation	All Database Servers	View Hide

1847B3947928D848E0530B98EB0ABBDA	PASS	SQL Check	None of the tablespaces have recyclebin objects consuming space more than 100 KB.	All Databases	View Hide

1832702CA36E9A8BE0530B98EB0AB5A9	PASS	SQL Check	None of the database users have more than 10 recyclebin objects in it.	All Databases	View Hide

EC1C37C834BD3475E04312C0E50A7FEB	PASS	SQL Check	Database parameter DB_FILE_MULTIBLOCK_READ_COUNT is unset as recommended	All Databases	View Hide

135F449E978C3B44E05313C0E50ABE53	PASS	OS Check	PGA allocation for all databases is not more than total memory available on this system	All Database Servers	View Hide

0EC55FAA1A0125FFE05312C0E50AE9E4	PASS	OS Check	Hidden database initialization parameters are not set per best practice recommendations	All Database Servers	View Hide

0EC203BA8AB807D4E05312C0E50A0FA8	PASS	ASM Check	Hidden ASM initialization parameters are not set per best practice recommendations	All ASM Instances	View Hide

0D13FFDA6C7F5E01E05313C0E50A4C97	PASS	OS Check	Free space in /etc directory meets or exceeds recommendation of minimum 1GB	All Database Servers	View Hide

0D12A9722FBA54F4E05313C0E50ACFDF	PASS	OS Check	Free space in /var directory meets or exceeds recommendation of minimum 1GB	All Database Servers	View Hide

07A3E6A60EB103C7E05313C0E50A24DA	PASS	SQL Check	No non-default AWR settings found.	All Databases	View Hide

078C3A95E8BF672CE05312C0E50A0501	PASS	SQL Check	Non-AWR Space consumption is less than 50% of total SYSAUX space	All Databases	View Hide

0582BF80C7D10AB5E05313C0E50A1584	PASS	OS Check	free physical memory is at sufficient level on system	All Database Servers	View Hide

0582515965486615E05313C0E50AA8F8	PASS	OS Check	system is started with runlevel 3 or 5	All Database Servers	View Hide

05810A7A2BB35F1AE05313C0E50AB854	PASS	OS Check	/boot is mounted	All Database Servers	View Hide

0580BF75E8844B73E05312C0E50A2B65	PASS	OS Check	/dev/shm is mounted as temporary file system	All Database Servers	View Hide

03CAC195BC875E34E05313C0E50AC351	PASS	SQL Check	TIMED_STATISTICS is set to recommended value.	All Databases	View Hide

03C336799EEF3444E05313C0E50A0D3C	PASS	SQL Check	STATISTICS_LEVEL is set to recommended value.	All Databases	View Hide

026DF829123D2136E05313C0E50AFE7B	PASS	SQL Check	AWR_FLUSH_EMERGENCY_COUNT value is equal to ZERO	All Databases	View Hide

Maximum Availability Architecture (MAA) Scorecard

Outage Type

Status

Type

Message

Status On

Details

DATABASE FAILURE PREVENTION BEST PRACTICES

PASS

 [Updated: 03/09/17]

Oracle database can be configured with best practices that are applicable to all Oracle databases, including single-instance, Oracle RAC databases, Oracle RAC One Node databases, and the primary and standby databases in Oracle Data Guard or Oracle GoldenGate configurations.

Key HA Benefits:

(1) Improved recoverability
(2) Improved stability

CCAECE9BB07579D8E0431EC0E50A9209	PASS	SQL Check	All tablespaces are locally managed tablespace	All Databases	View Hide

CCADBFBF2F077638E0431EC0E50A9E9D	PASS	SQL Check	All tablespaces are using Automatic segment storage management	All Databases	View Hide

CCAD995A4067776EE0431EC0E50AD00E	PASS	SQL Check	Default temporary tablespace is set	All Databases	View Hide

CCAD9945F7797A3FE0431EC0E50A856B	PASS	SQL Check	Database Archivelog Mode is set to ARCHIVELOG	All Databases	View Hide

C31B975819621D1CE0431EC0E50AC731	PASS	OS Check	Redo log files are appropriately sized	All Database Servers	View Hide

666BBB359DF17AA6E040E50A1EC02674	PASS	SQL Check	The SYS and SYSTEM user ids have a default tablespace of SYSTEM	All Databases	View Hide

2285D66ED4BBD4F7E0530D98EB0A0166	PASS	OS Check	Local archive destination has alternate destination configured	All Database Servers	View Hide

COMPUTER FAILURE PREVENTION BEST PRACTICES

INFO

 [Updated: 03/09/17]

Oracle RAC and Oracle Clusterware allow Oracle Database to run any packaged or custom application across a set of clustered servers. This capability provides server side high availability and scalability. If a clustered server fails, then Oracle Database continues running on the surviving servers. When more processing power is needed, you can add another server without interrupting access to data.


Oracle RAC and Oracle Clusterware supports rolling upgrade for most hardware and software changes excluding Oracle RDBMS patch sets and new database releases.


 To achieve zero or near zero application downtime, integrated client failover best practices need to be incorporated which is covered in client failover operational best practices section of the MAA scorecard.


Key HA Benefits:

Zero database downtime for node and instance failures.  Application brownout can be zero or seconds compared to minutes and an hour with third party cold cluster failover solutions.
Zero database and potentially zero application downtime for most planned maintenance activities involving operating system, database or Grid Infrastructure software changes.

B16C9847624E1B5AE0431EC0E50AE353	WARNING	SQL Parameter Check	fast_start_mttr_target should be greater than or equal to 300.	All Instances	View Hide

DATA CORRUPTION PREVENTION BEST PRACTICES

PASS

 [Updated: 03/09/17]

To achieve the most comprehensive data corruption prevention and detection , use Oracle Active Data Guard and configure DB_BLOCK_CHECKSUM,DB_LOST_WRITE_PROTECT and DB_BLOCK_CHECKING database initialization parameters on the primary database and all standby databases in a Data Guard environment. Workload specific testing is required to assess whether the performance overhead with especially DB_BLOCK_CHECKING is acceptable.

Using ASM, RMAN, Exadata Storage and Zero Data Loss Recovery Appliance (Recovery Appliance) provide additional data protection checks and repair for Oracle databases and backups.    

 Key HA Benefits

Application downtime due to data corruptions can be reduced from hours and days to seconds to no downtime.
 Prevention, quick detection and fast repair of data block corruptions. 
 With Active Data Guard, physical data block corruptions can be repaired automatically using current blocks from primary or standby databases.
With ASM, Exadata Storage and Recovery Appliance, data corruptions can be detected in the background and in some cases repaired automatically.
With DB_BLOCK_CHECKING=MED or FULL, logical data block corruptions can be detected and prevented on the primary or standby database.

A97A01C7F17307CFE040E50A1EC06C63	FAIL	SQL Parameter Check	Database parameter DB_LOST_WRITE_PROTECT is not set to recommended value	All Instances	View Hide

9AC756552BAE5EA4E040E50A1EC0475E	FAIL	SQL Parameter Check	Database parameter DB_BLOCK_CHECKSUM is not set to recommended value	All Instances	View Hide

AE071F58E61C18F0E040E50A1EC01EA8	WARNING	OS Check	Database parameter DB_BLOCK_CHECKING on PRIMARY is NOT set to the recommended value.	All Database Servers	View Hide

DC223BE507001B0AE04313C0E50A359D	PASS	SQL Check	The data files are all recoverable	All Databases	View Hide

C33B66906DA21665E0431EC0E50A0759	PASS	SQL Check	No reported block corruptions in V$DATABASE_BLOCK_CORRUPTIONS	All Databases	View Hide

LOGICAL CORRUPTION PREVENTION BEST PRACTICES

PASS

 [Updated: 03/09/17]

Oracle Flashback Technology enables fast logical failure repair. Oracle recommends that you use automatic undo management with sufficient space to attain your desired undo retention guarantee, enable Oracle Flashback Database, and allocate sufficient space and I/O bandwidth in the fast recovery area.  Application monitoring is required for early detection.  Effective and fast repair comes from leveraging and rehearsing the most common application specific logical failures and using the different flashback features effectively (e.g flashback query, flashback version query, flashback transaction query, flashback transaction, flashback drop, flashback table, and flashback database, and 12.2 flashback pluggable database (PDB)).

Key HA Benefits:

With application monitoring and rehearsed repair actions with flashback technologies, application downtime can reduce from hours and days to the time to detect the logical inconsistency.
Fast repair for logical failures caused by malicious or accidental DML or DDL operations.
Effect fast point-in-time repair at the appropriate level of granularity: transaction, table, pluggable database, or database.
Questions that need to be addressed by your application and operations team:

Can your application or monitoring infrastructure detect logical inconsistencies?
Is your operations team prepared to use various flashback technologies to repair quickly and efficiently?
Is security practices enforced to prevent unauthorized privileges that can result logical inconsistencies?

B09D194B9AEF0F10E0431EC0E50ADE87	FAIL	SQL Check	Flashback on PRIMARY is not configured	All Databases	View Hide

B9BCC54F3A901F18E0431EC0E50AC1A0	PASS	SQL Parameter Check	RECYCLEBIN on PRIMARY is set to the recommended value	All Instances	View Hide

B16B9310FB504695E0431EC0E50A880B	PASS	SQL Parameter Check	Database parameter UNDO_RETENTION on PRIMARY is not null	All Instances	View Hide

DATABASE/CLUSTER/SITE FAILURE PREVENTION BEST PRACTICES

INFO

[Updated: 03/09/17]

Active Data Guard is the real-time data protection and availability solution that eliminates single point of failure by maintaining one or more synchronized physical replicas of the production database. If an unplanned outage of any kind impacts the production database, applications and users can quickly failover to a synchronized standby, minimizing downtime and preventing data loss. An Active Data Guard standby can be used to offload read-only applications, ad-hoc queries, and backups from the primary database or be dual-purposed as a test system at the same time it provides disaster protection. An Active Data Guard standby can also be used to minimize downtime for planned maintenance when upgrading to new Oracle Database patch sets and major database releases and for select migrations.

For zero data loss protection and fastest recovery time, deploy a local Data Guard standby database with Data Guard Fast-Start Failover and integrated client failover. For protection against outages impacting the entire data center, or a broad geography, deploy a Data Guard standby database at a remote location. Oracle 12c Far Sync Standby can achieve zero data loss protection across Wide Area Network or across regions.

Key HA Benefits:

Active Data Guard provides optimal data protection by using physical replication and comprehensive Oracle validation to maintain an exact byte-for-byte copy of the primary database that can be open read-only to offload reporting, ad-hoc queries and backups.
With Oracle 11g release 2 and higher Active Data Guard and real time apply, data block corruptions can be repaired automatically and downtime can be reduced from hours and days of application impact to zero downtime with zero data loss.
With MAA best practices, Data Guard Fast-Start Failover and integrated client failover, downtime from database, cluster and site failures can be reduced from hours to days to seconds.
With remote standby database (Disaster Recovery Site), you have protection from complete site failures.
In all cases, the Active Data Guard instances can be active and used for other activities.
Data Guard can reduce risks and downtime for planned maintenance activities by using Database rolling upgrade with transient logical standby or DBMS rolling, standby-first patch apply and database migrations.
For other advanced replication requirements where read-write access to a replica database is required while it is being synchronized with the primary database see Oracle GoldenGate logical replication. Oracle GoldenGate can be used to support heterogeneous database platforms and database releases, an effective read-write full or subset logical replica and to reduce or eliminate downtime for application, database or system changes. Oracle GoldenGate flexible logical replication solution's main trade-off is the additional administration for application developer and database administrators.

CD4D51B78F692408E0431EC0E50A3077	FAIL	SQL Check	Primary database is not protected with Data Guard (standby database) for real-time data protection and availability	All Databases	View Hide

CLIENT FAILOVER OPERATIONAL BEST PRACTICES

PASS

 [Updated: 03/09/17]

A highly available architecture requires the ability of the application tier to transparently fail over to a surviving instance or database advertising the required service. This ensures that applications are generally available or minimally impacted in the event of node failure, instance failure, or database failures.

227A6B4780B24E32E0530B98EB0AC9FD	WARNING	OS Check	Non-default database Services are not configured	All Database Servers	View Hide

B0B3017BC94455B9E0431EC0E50AAA2A	PASS	OS Check	Clusterware is running	All Database Servers	View Hide

ORACLE RECOVERY MANAGER(RMAN) BEST PRACTICES

PASS

[Updated: 03/09/17]

Oracle Recovery Manager (RMAN) is an Oracle Database utility to manage database backup and, more importantly, the recovery of the database. RMAN eliminates operational complexity while providing superior performance and availability of the database.

RMAN determines the most efficient method of executing the requested backup, restoration, or recovery operation and then submits these operations to the Oracle Database server for processing. RMAN and the server automatically identify modifications to the structure of the database and dynamically adjust the required operation to adapt to the changes.

RMAN has many unique HA capabilities that can be challenging or impossible for third party backup and restore utilities to deliver such as

In-depth Oracle data block checks during every backup or restore operation
Efficient block media recovery
Automatic recovery through complex database state changes such as resetlogs or past Data Guard role transitions
Fast incremental backup and restore operations
Integrated retention policies and backup file management with Oracle’s fast recovery area
Online backups without the need to put the database or data file in hot backup mode.

RMAN backups are strategic to MAA so a damaged database (complete database or subset of the database such as a data file or tablespace, log file, or controlfile) can be recovered but for the fastest recovery, use Data Guard or GoldenGate. RMAN operations are also important for detecting any corrupted blocks from data files that are not frequently accessed.

Oracle also now has the Zero Data Loss Recovery Appliance (Recovery Appliance) which provides the following key benefits:

Eliminate or Minimize Data Loss.
Minimal Impact Backups by offloading reduplication, compression, recovery+merge, and validation to Recovery Appliance.
Database Level Recoverability and Validation.
Cloud-scale and Data Center wide Data Protection for all backups.

E34EB3F020B35A64E04313C0E50A1DC7	WARNING	SQL Check	RMAN controlfile autobackup should be set to ON	All Databases	View Hide

E3508B6664085B53E04313C0E50AC2CA	PASS	OS Check	control_file_record_keep_time is within recommended range [1-9]	All Database Servers	View Hide

ORACLE GOLDENGATE FAILURE PREVENTION BEST PRACTICES

INFO

 [Updated: 03/09/17]

Oracle GoldenGate is Oracle's strategic logical replication solution for data distribution and data integration. Unlike replication solutions from other vendors, Oracle GoldenGate is more closely integrated with Oracle Database while also providing an open, modular architecture ideal for replication across heterogeneous database management systems. This combination of attributes eliminates compromise, making Oracle GoldenGate the preferred replication solution for addressing requirements that span Oracle Database and non-Oracle Database environments.

Key HA Benefits:

Potential zero or near zero application downtime for platform migration, database or application upgrades.
Active/Active environment with the ability to update both databases. Conflict resolution techniques will need to be considered.

To achieve the highest levels of availability resulting in zero or near-zero downtime for both unplanned outages and all planned maintenance activities, customers use the combination of Oracle Active Data Guard and Oracle GoldenGate.


Oracle Active Data Guard and Oracle GoldenGate are essential components of the Gold and Platinum MAA tiers.

OPERATIONAL BEST PRACTICES

INFO

 [Updated: 03/09/17]

Operational best practices are an essential prerequisite to high availability.

DATABASE CONSOLIDATION BEST PRACTICES

INFO

 [Updated: 03/09/17]

Database consolidation requires additional planning and management to ensure HA requirements are met.

Top

Patch Recommendation

Database server "rac1"

1 Recommended CRS patches for 112040 from /oracle/app/11.2.0/grid

Patch	CRS	ASM	RDBMS	RDBMS_HOME	Patch-Description
22502505	not-applied	n/a	not-applied	/oracle/app/oracle/product/11.2.0/db	ACFS PSU 11.2.0.4.160419
26392168	not-applied	n/a	not-applied	/oracle/app/oracle/product/11.2.0/db	DB PSU 11.2.0.4.171017 (INCLUDES CPUOCT2017)
26609929	not-applied	n/a	not-applied	/oracle/app/oracle/product/11.2.0/db	OCW PSU 11.2.0.4.170814

Top

1 Recommended RDBMS patches for 112040 from /oracle/app/oracle/product/11.2.0/db

Patch RDBMS ASM Type Patch-Description

26392168 not-applied n/a merge DB PSU 11.2.0.4.171017 (INCLUDES CPUOCT2017)
26609929 not-applied n/a merge OCW PSU 11.2.0.4.170814
Top

Patch	RDBMS	ASM	Type	Patch-Description
26392168	not-applied	n/a	merge	DB PSU 11.2.0.4.171017 (INCLUDES CPUOCT2017)
26609929	not-applied	n/a	merge	OCW PSU 11.2.0.4.170814

Verify no duplicate parameter entries in database init.ora(spfile)

Recommendation

Needs attention on -

Passed on rac1

Status on rac1:
PASS => There are no duplicate parameter entries in the database init.ora(spfile) file
DATA FROM rac1 - orcl DATABASE - Verify no duplicate parameter entries in database init.ora(spfile) orcl2.__db_cache_size=230686720 orcl1.__db_cache_size=260046848 orcl2.__java_pool_size=4194304 orcl1.__java_pool_size=4194304 orcl2.__large_pool_size=16777216 orcl1.__large_pool_size=16777216 orcl1.__oracle_base='/oracle/app/oracle'#ORACLE_BASE set from environment orcl2.__pga_aggregate_target=314572800 orcl1.__pga_aggregate_target=306184192 orcl2.__sga_target=465567744 orcl1.__sga_target=473956352 orcl2.__shared_io_pool_size=0 orcl1.__shared_io_pool_size=0 orcl2.__shared_pool_size=201326592 orcl1.__shared_pool_size=180355072 orcl2.__streams_pool_size=0
orcl1.__streams_pool_size=0 *.audit_file_dest='/oracle/app/oracle/admin/orcl/adump' *.audit_trail='db' *.cluster_database=true *.compatible='11.2.0.4.0' *.control_files='+DATA/orcl/controlfile/current.260.897348927' *.db_block_size=8192 *.db_create_file_dest='+DATA' *.db_domain='' *.db_name='orcl' *.diagnostic_dest='/oracle/app/oracle' *.dispatchers='(PROTOCOL=TCP) (SERVICE=orclXDB)' *.fal_client='pridb' *.fal_server='stydb' orcl1.instance_number=1 orcl2.instance_number=2 *.log_archive_config='dg_config=(orcl,standby)' *.log_archive_dest_1='location=+data valid_for=(all_logfiles,all_roles) db_unique_name=orcl' *.log_archive_dest_2='service=stydb valid_for=(online_logfiles,primary_role) db_unique_name=standby' *.memory_target=780140544 *.open_cursors=300 *.processes=150 *.remote_listener='scan-ip:1521' *.remote_login_passwordfile='exclusive' orcl2.thread=2 orcl1.thread=1 orcl2.undo_tablespace='UNDOTBS2' orcl1.undo_tablespace='UNDOTBS1'
Click for more data

Hide

Verify service_names parameter is not defined in spfile

Recommendation
Benefit / Impact:

Having the service_names parameter set within the spfile will cause the service to start (and register with the listener) with database startup.
This issue will be apparent with role-based services (primary role services being start on the standby database). In a RAC environment Clusterware manages these services based on policies you define when creating the service.

Action / Repair:

Reset the service_names parameter to its default value (which is null)

SQL> alter system reset service_names sid='*'

Needs attention on -

Passed on rac1

Status on rac1:
PASS => Database parameter service_names is not defined in spfile
DATA FROM rac1 - orcl DATABASE - Verify service_names parameter is not defined in spfile orcl2.__db_cache_size=230686720 orcl1.__db_cache_size=260046848 orcl2.__java_pool_size=4194304 orcl1.__java_pool_size=4194304 orcl2.__large_pool_size=16777216 orcl1.__large_pool_size=16777216 orcl1.__oracle_base='/oracle/app/oracle'#ORACLE_BASE set from environment orcl2.__pga_aggregate_target=314572800 orcl1.__pga_aggregate_target=306184192 orcl2.__sga_target=465567744 orcl1.__sga_target=473956352 orcl2.__shared_io_pool_size=0 orcl1.__shared_io_pool_size=0 orcl2.__shared_pool_size=201326592 orcl1.__shared_pool_size=180355072 orcl2.__streams_pool_size=0
orcl1.__streams_pool_size=0 *.audit_file_dest='/oracle/app/oracle/admin/orcl/adump' *.audit_trail='db' *.cluster_database=true *.compatible='11.2.0.4.0' *.control_files='+DATA/orcl/controlfile/current.260.897348927' *.db_block_size=8192 *.db_create_file_dest='+DATA' *.db_domain='' *.db_name='orcl' *.diagnostic_dest='/oracle/app/oracle' *.dispatchers='(PROTOCOL=TCP) (SERVICE=orclXDB)' *.fal_client='pridb' *.fal_server='stydb' orcl1.instance_number=1 orcl2.instance_number=2 *.log_archive_config='dg_config=(orcl,standby)' *.log_archive_dest_1='location=+data valid_for=(all_logfiles,all_roles) db_unique_name=orcl' *.log_archive_dest_2='service=stydb valid_for=(online_logfiles,primary_role) db_unique_name=standby' *.memory_target=780140544 *.open_cursors=300 *.processes=150 *.remote_listener='scan-ip:1521' *.remote_login_passwordfile='exclusive' orcl2.thread=2 orcl1.thread=1 orcl2.undo_tablespace='UNDOTBS2' orcl1.undo_tablespace='UNDOTBS1'
Click for more data

Hide

Verify hotplug attribute on network cards used by Clusterware

Recommendation

Risk:

listeners and services will become unavailable or moved to different node in cluster

Action / Repair:

Set hotplug=no for all network cards used by clusterware. To find all network cards used by clusterware run following command.

$CRS_HOME/bin/oifcfg get

NOTE: This configuration only applies to Linux 6 per the referenced Red Hat net.hotplug knowledgebase article.

Links
1 Red Hat Enterprise Linux 7 no longer calls the net.hotplug
2 Note: 1569028.1 - Network interface going down when dynamically adding disks to storage using udev in RHEL 6

Needs attention on rac1

Passed on -

Status on rac1:
FAIL => One or more network card used by clusterware do not have HOTPLUG=NO
DATA FROM RAC1 - VERIFY HOTPLUG ATTRIBUTE ON NETWORK CARDS USED BY CLUSTERWARE Contents of /etc/sysconfig/network-scripts/ifcfg-eth0 DEVICE="eth0" BOOTPROTO=none IPV6INIT="yes" NM_CONTROLLED="yes" ONBOOT="yes" TYPE="Ethernet" UUID="97ca30ff-3e7a-40b3-928d-0462fa2d35e0" HWADDR=00:0C:29:DB:C4:40 IPADDR=192.168.133.120 PREFIX=24 GATEWAY=192.168.133.1 DEFROUTE=yes IPV4_FAILURE_FATAL=yes IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes IPV6_PEERDNS=yes IPV6_PEERROUTES=yes IPV6_FAILURE_FATAL=no NAME="System eth0" LAST_CONNECT=1448958130 Contents of /etc/sysconfig/network-scripts/ifcfg-eth1
Click for more data

Hide

kernel.panic_on_oops

Recommendation
Benefit / Impact:

Setting this kernel parameter to 1 causes the kernel to panic if a kernel oops or bug is encountered.

Risk:

Kernel panic or oops represents potential problems with the server. If it's set to 0, the kernel will try to keep running (with whatever awful consequences). If it's set to 1, the kernel will enter the panic state and halt.

Action / Repair:

As root user, execute the following command from the shell prompt

echo 1 > /proc/sys/kernel/panic_on_oops

To set this kernel parameter permanently, edit /etc/sysctl.conf and make the following entry

kernel.panic_on_oops = 1

Needs attention on -

Passed on rac1

Status on rac1:
PASS => kernel.panic_on_oops parameter is configured according to recommendation
DATA FROM RAC1 - KERNEL.PANIC_ON_OOPS 1

Hide

Monitoring SGA resize operations

Recommendation
Benefit / Impact:

When the size of SGA is increased with the proper minimum value set for shared pool and buffer cache, the number of resize operations should be reduced.

Risk:

An undersized SGA can lead to frequent resizing operations of the SGA which could potentially cause slow performance or database hang situations.

Action / Repair:

Perform the following 3 actions:

1. Increase the SGA_TARGET or MEMORY_TARGET initialization parameters. If frequent resizing is occurring V$SGA_TARGET_ADVICE can be used to ensure that SGA_TARGET is optimally set. Refer to “Tuning SGA_TARGET Using V$SGA_TARGET_ADVICE (Doc ID 1323708.1)” for more assistance.

2. Set the minimum value for shared pool or buffer cache to the high watermark size of past resize assuming that is the peak level .

Ex:
SQL> SELECT MAX(final_size) FROM V$SGA_RESIZE_OPS WHERE component='shared pool';
SQL> ALTER SYSTEM SET SHARED_POOL_SIZE=<newvalue> scope=both;

3. Increase the time interval between resizes. To do this, the following parameter can be set dynamically:

SQL> alter system set "_memory_broker_stat_interval"=999;

This will increase the time between resize to at least 999 seconds and thereby reducing the number of resize operations. The parameter "_memory_broker_stat_interval" is expressed in seconds with a default setting of 30 seconds. If you are running on RAC you will need to do this on all nodes.

Links
1 Note: 742599.1 - High 'cursor: pin S wait on X' and/or 'library cache lock' Waits. Cause: Shared Pool/Buffer Cache Resize Activity
2 Note: 1344228.1 - Severe Performance Problems Due to Sudden drop/resize/shrink of buffer cache
3 Note: 1323708.1 - Tuning SGA_TARGET Using V$SGA_TARGET_ADVICE

Needs attention on orcl

Passed on -

Status on orcl:
WARNING => Consider investigating the frequency of SGA resize operations and take corrective action
DATA FOR ORCL FOR MONITORING SGA RESIZE OPERATIONS COMPONENT OPER_TYPE COUNT(1) ------------------------- ------------- ---------- DEFAULT buffer cache GROW 1

Hide

Monitoring waiter sessions

Recommendation
Benefit / Impact:

Identifying the database sessions that are currently waiting for the resources (such as: enqueues, latches, IO calls, CPU, Memory) and identifying the blocker session for the waiters can avoid database performance issues or database hang situations.

Risk:

Slow database performance or a database hang situation may happen when a high number of waiter sessions exist and a high number of blockers are found.

Action / Repair:

Please execute the following SQL to identify the sessions waiting.

set pages 100;
set lines 150;
col sid format 999999;
col event format a40;
col state format a15;
col waiting head "Waiting(in seconds)" format 9,999.99;
col waited head "Waited" format 9,999.99;
select sid, state, event,
seconds_in_wait waiting,
wait_time/100 waited, p1,p2,p3, BLOCKING_SESSION "Blocker"
from v$session
where event not in
(
'SQL*Net message from client',
'SQL*Net message to client',
'rdbms ipc message'
)
and state = 'WAITING'
and username not in ('SYS','SYSTEM','SYSMAN','DBSNMP');

Please refer the following interactive workflow to troubleshoot the database performance issues based on the wait events observed.

Troubleshooting Assistant: Troubleshooting Performance Issues (Doc ID 1543445.2)

This would be very helpful in narrowing down the performance problems without contacting Oracle support.

Links
1 Note: 1477599.1 - Best Practices: Proactive Data Collection for Performance Issues
2 Note: 452358.1 - How to Collect Diagnostics for Database Hanging Issues

Needs attention on -

Passed on orcl

Status on orcl:
PASS => No waiter sessions found currently for application users
DATA FOR ORCL FOR MONITORING WAITER SESSIONS Query returned no rows which is expected when the SQL check passes.

Hide

session_cached_cursors parameter

Recommendation
Benefit / Impact:

When the session_cached_cursors database parameter is set to a high value then the number of soft parses will be reduced. This will reduce the contention in the library cache for sharing the cursors during repeated executions of the SQL statements.

Risk:

The amount of soft parses could increase if the value for the session_cached_cursors parameter is not increased. This may lead to slow performance due to excessive parse calls.

Action / Repair:

The default value for session_cached_cursors is 50.
The parameter value can be modified based on the USAGE from the report.
If the current value is set to less than or equal to 300 and the USAGE is greater than or equal to 100% then the value can be doubled.

Set the parameter value in the parameter file (init.ora or spfile) and restart the database.

SQL> alter system set session_cached_cursors=<value> scope=spfile sid='*';

Links
1 Note: 208857.1 - SCRIPT - to Set the 'SESSION_CACHED_CURSORS' and 'OPEN_CURSORS' Parameters Based on Usage
2 Note: 62143.1 - Troubleshooting: Tuning the Shared Pool and Tuning Library Cache Latch Contention
3 Note: 30804.1 - Init.ora Parameter "SESSION_CACHED_CURSORS" Reference Note

Needs attention on -

Passed on orcl

Status on orcl:
PASS => The database parameter session_cached_cursors is set to the recommended value
DATA FOR ORCL FOR SESSION_CACHED_CURSORS PARAMETER SESSION_CACHED_CURSORS: Current Value = 50 Usage = 11 Percentage Usage = 22%

Hide

Monitoring changes to schema objects

Recommendation
Benefit / Impact:

Changes to schema objects could lead to bad execution plan for the SQL statements involving the objects that changed. Investigating the changes may help in identifying the root cause behind the sudden change in the execution plan for the SQL statements or slow application performance.

Risk:

The generation of bad execution plan due to the changes to schema objects might lead to application performance issues.

Action / Repair:

When performance issues are noticed on SQL statements, check with the DBA or Application Developers about the changes to the schema objects and take corrective action to revert it back if possible.

The report only shows top 20 objects created/modified during the last 7 days.
Please run the following SQL to get the complete list of objects.

set linesize 200;
set pagesize 100;
set heading on;
COLUMN OWNER FORMAT A15
COLUMN OBJECT_NAME FORMAT A25
COLUMN OBJECT_TYPE FORMAT A10
SELECT OWNER, OBJECT_NAME, OBJECT_TYPE, CREATED, LAST_DDL_TIME, TIMESTAMP,STATUS FROM DBA_OBJECTS WHERE (CREATED>=SYSDATE-7 or LAST_DDL_TIME>=SYSDATE-7) AND OWNER NOT IN ('SYS','SYSTEM','SYSMAN','DBSNMP') ORDER BY LAST_DDL_TIME DESC;

Links
1 Note: 1662591.1 - How To Track The Changes That May Affect Database Performance

Needs attention on orcl

Passed on -

Status on orcl:
WARNING => Consider investigating changes to the schema objects such as DDLs or new object creation
DATA FOR ORCL FOR MONITORING CHANGES TO SCHEMA OBJECTS OWNER OBJECT_NAME OBJECT_TYP CREATED LAST_DDL_ TIMESTAMP STATUS --------------- ------------------------- ---------- --------- --------- ------------------- ------- ORACLE_OCM MGMT_CONFIG_JOB JOB 24-AUG-13 24-JAN-18 2018-01-24:12:26:02 VALID EXFSYS RLM$SCHDNEGACTION JOB 24-AUG-13 24-JAN-18 2018-01-24:12:26:02 VALID EXFSYS RLM$EVTCLEANUP JOB 24-AUG-13 24-JAN-18 2018-01-24:12:26:01 VALID RYL TEST TABLE 23-JAN-18 23-JAN-18 2018-01-23:15:12:24 VALID ORACLE_OCM MGMT_STATS_CONFIG_JOB JOB 24-AUG-13 22-JAN-18 2018-01-22:10:16:59 VALID

Hide

Verify ASM disk permissions

Recommendation

Needs attention on -

Passed on rac1

Status on rac1:
PASS => ASM disk permissions are set as recommended
DATA FROM RAC1 - VERIFY ASM DISK PERMISSIONS crw-rw----. 1 grid asmadmin 162, 3 Jan 25 16:44 /dev/raw/raw3 crw-rw----. 1 grid asmadmin 162, 1 Jan 25 16:44 /dev/raw/raw1 crw-rw----. 1 grid asmadmin 162, 2 Jan 25 16:43 /dev/raw/raw2

Hide

Verify free disk space for rebalance

Recommendation
Benefit / Impact:

If any disk on a diskgroup irrespective of redundancy,have less than 50Mb of free space rebalance operation fails to complete with ORA-15041

Risk:

As rebalance does not complete successfully ,allocation unit distribution will not be in symmetrical nature. Results in future allocation unit failure . This causes imbalance within the diskgroup.

Action / Repair:

release space by removing some redundant files from this diskgroup.

Links
1 Note: 552082.1 - Script to report the list of files stored in ASM and CURRENTLY NOT OPENED

Needs attention on -

Passed on rac1

Status on rac1:
PASS => ASM disks have enough free space for rebalance
DATA FROM RAC1 - VERIFY FREE DISK SPACE FOR REBALANCE DATA_0001|/dev/raw/raw3|776 DATA_0000|/dev/raw/raw2|773 OCR_0000|/dev/raw/raw1|628

Hide

Optimizer Parameters with Non-Default Settings (Check 2)

Recommendation
Benefit / Impact:

One or more optimizer related parameters are set to a non-default value.

Risk:

The settings for these parameters should be recommended by the application vendor or by Oracle Support. If any optimizer related parameters are set, the application should be certified and tested to work well with the parameter settings. Ensure that the performance is as expected.

Non default settings of optimizer related parameters may result in performance issues. The settings of such optimizer parameters can have an impact on the entire application performance.

Please Note: If the database instance is running Siebel/Peoplesoft Applications then you may not need to change any settings, please check your product documentation.

Action / Repair:

To modify the parameter value, use the alter session or alter system commands. Refer to the Initialization Parameter chapter of the Oracle Database Reference Documentation for more details on setting these parameters.

Needs attention on -

Passed on orcl

Status on orcl:
PASS => All optimizer related parameters are set to the default value (Parameter Set 2 of 3)
DATA FOR ORCL FOR OPTIMIZER PARAMETERS WITH NON-DEFAULT SETTINGS (CHECK 2) Query returned no rows which is expected when the SQL check passes.

Hide

Optimizer Parameters with Non-Default Settings (Check 3)

Recommendation
Benefit / Impact:

One or more optimizer related parameters are set to a non-default value.

Risk:

The settings for these parameters should be recommended by the application vendor or by Oracle Support. If any optimizer related parameters are set, the application should be certified and tested to work well with the parameter settings. Ensure that the performance is as expected.

Non default settings of optimizer related parameters may result in performance issues. The settings of such optimizer parameters can have an impact on the entire application performance.

Please Note: If the database instance is running Siebel/Peoplesoft Applications then you may not need to change any settings, please check your product documentation.

Action / Repair:

To modify the parameter value, use the alter session or alter system commands. Refer to the Initialization Parameter chapter of the Oracle Database Reference Documentation for more details on setting these parameters.

Needs attention on -

Passed on orcl

Status on orcl:
PASS => All optimizer related parameters are set to the default value (Parameter Set 3 of 3)
DATA FOR ORCL FOR OPTIMIZER PARAMETERS WITH NON-DEFAULT SETTINGS (CHECK 3) Query returned no rows which is expected when the SQL check passes.

Hide

Obsolete Database Parameters for 11.2

Recommendation
Benefit / Impact:

An attempt to start a database using one or more obsolete initialization parameters will succeed, but a warning is returned and recorded in the alert log.

Risk:

In most cases setting an obsolete initialization parameter will have minimal to no impact on the database or application. However, there are some cases where this can lead to unexpected behavior or performance issues. Obsolete parameters could also affect the upgrade.

Action / Repair:

To modify the parameter value, use the alter session or alter system commands. Refer to the Initialization Parameter chapter of the Oracle Database Reference Documentation for more details on setting these parameters.

Obsolete Parameters:

dblink_encrypt_login
hash_join_enabled
log_parallelism
max_rollback_segments
mts_circuits
mts_dispatchers
mts_listener_address
mts_max_dispatchers
mts_max_servers
mts_multiple_listeners
mts_servers
mts_service
mts_sessions
optimizer_max_permutations
oracle_trace_collection_name
oracle_trace_collection_path
oracle_trace_collection_size
oracle_trace_enable
oracle_trace_facility_name
oracle_trace_facility_path
partition_view_enabled
plsql_native_c_compiler
plsql_native_linker
plsql_native_make_file_name
plsql_native_make_utility
row_locking
serializable
transaction_auditing
undo_suppress_errors
enqueue_resources
ddl_wait_for_locks
logmnr_max_persistent_sessions
plsql_compiler_flags
drs_start
gc_files_to_locks
max_commit_propagation_delay
plsql_native_library_dir
plsql_native_library_subdir_count
sql_version

Links
1 Note: 957543.1 - Health Check Alert: Consider removing obsolete initialization parameters from the PFILE/SPFILE

Needs attention on -

Passed on orcl

Status on orcl:
PASS => No obsolete initialization parameters are set
DATA FOR ORCL FOR OBSOLETE DATABASE PARAMETERS FOR 11.2 Query returned no rows which is expected when the SQL check passes.

Hide

optimizer_features_enable parameter

Recommendation
Benefit / Impact:

New optimizer features and fixes are enabled when the OPTIMIZER_FEATURES_ENABLE parameter is set to the current database version. Setting the OPTIMIZER_FEATURES_ENABLE parameter to a value other than the current database version should only be used as a short term measure at the suggestion of oracle support.

Risk:

By reducing the OPTIMIZER_FEATURES_ENABLE level, new optimizer features and fixes are disabled. This can potentially have a serious negative impact on performance because it eliminates the possibility of choosing better plans which are only available with the new features and fixes.

Action / Repair:

If Oracle Support has not recommended that you modify this parameter, then you should set the OPTIMIZER_FEATURES_ENABLE parameter to the current database version to ensure that new optimizer features are enabled. This parameter can be set with the following command:

SQL> alter system set optimizer_features_enable='X.X.X.X' scope=spfile;

Refer to the following MOS document for more details.

Links
1 Note: 1362332.1 - Use Caution if Changing the OPTIMIZER_FEATURES_ENABLE Parameter After an Upgrade

Needs attention on -

Passed on orcl

Status on orcl:
PASS => Database parameter OPTIMIZER_FEATURES_ENABLE is set to the current database version
DATA FOR ORCL FOR OPTIMIZER_FEATURES_ENABLE PARAMETER Database Version = 11.2.0.4.0 optimizer_features_enable parameter = 11.2.0.4

Hide

nls_sort parameter

Recommendation
Benefit / Impact:

Setting NLS_SORT to BINARY will result in the ability for your SQL to use standard indexes and queries will be more efficient.

Risk:

Setting NLS_SORT to a value other than BINARY will make regular indexes unusable. This will negatively affect the performance of your SQL unless you have specific indexes to support the new sorting.

Action / Repair:

Set NLS_SORT to BINARY. Change the value in the initialization parameter file and then restart the instance. If you do need to have a different sorting in place, then make sure you have the necessary indexes created using a different sorting (NLSSORT function).

Refer to the following MOS document for more details.

Links
1 Note: 227335.1 - Linguistic Sorting - Frequently Asked Questions

Needs attention on -

Passed on orcl

Status on orcl:
PASS => Database parameter NLS_SORT is set to BINARY
DATA FOR ORCL FOR NLS_SORT PARAMETER nls_sort parameter value from nls_database_parameters = BINARY nls_sort parameter value from nls_instance_parameters = nls_sort parameter value from nls_session_parameters = BINARY nls_sort parameter value from v$parameter =

Hide

Verify online (hot) patches are not applied on CRS_HOME

Recommendation
(2) Online patches consume additional memory and if kept permanently the memory consumption increases as the number of processes in the system increases.
(3) It is strongly recommended to rollback all Online patches and replace them with regular (offline) patches on next instance shutdown or the earliest maintenance window.

Links
1 Note: 761111.1 - RDBMS Online Patching Aka Hot Patching
2 Online patching

Needs attention on -

Passed on rac1

Status on rac1:
PASS => Online (hot) patches are not applied to CRS_HOME.
DATA FROM RAC1 - VERIFY ONLINE (HOT) PATCHES ARE NOT APPLIED ON CRS_HOME ls: cannot access /oracle/app/11.2.0/grid/hpatch/: No such file or directory

Hide

Tablespace extent management Allocation Type for SecureFiles LOB storage

Recommendation

Links
1 Note: 847951.1 - How to Change Extent Management of a Tablespace from Uniform Size to Autoallocate

Needs attention on -

Passed on orcl

Status on orcl:
PASS => Table containing SecureFiles LOB storage belongs to a tablespace with extent allocation type that is SYSTEM managed (AUTOALLOCATE)
DATA FOR ORCL FOR TABLESPACE EXTENT MANAGEMENT ALLOCATION TYPE FOR SECUREFILES LOB STORAGE Query returned no rows which is expected when the SQL check passes.

Hide

ASM MEMORY_TARGET Size

Recommendation
Benefit / Impact:

Enhanced database stability when using Oracle ASM for database storage

Risk:

In some cases ASM default memory allocation for some customers has proven to be too small leading to ORA-4031 errors which can in turn lead to database instability.

Action / Repair:

It is a best practice and recommended to use Automatic Memory Management (AMM) for the ASM instance in version 11.2 and above regardless of platform. AMM is configured in part through the use of the memory_target initialization parameter.

As a result, the default setting for memory_target in 11.2.0.4 and 12.1 versions has been increased to 1024M from the previous default of 256M. Oracle also advises increasing memory_target to 1024M as a minimum in version 11.2 as a proactive measure to match the new defaults. No patch is required in order to increase the value of memory_target.

Needs attention on -

Passed on rac1

Status on rac1:
PASS => ASM memory_target is set to recommended value
DATA FROM RAC1 - ASM MEMORY_TARGET SIZE MEMORY_TARGET -------------------------------------------------------------- ASM memory_target = 1076M

Hide

Verify online (hot) patches are not applied on ORACLE_HOME

Recommendation
(2) Online patches consume additional memory and if kept permanently the memory consumption increases as the number of processes in the system increases.
(3) It is strongly recommended to rollback all Online patches and replace them with regular (offline) patches on next instance shutdown or the earliest maintenance window.

Links
1 Note: 761111.1 - RDBMS Online Patching Aka Hot Patching
2 Online patching

Needs attention on -

Passed on rac1

Status on rac1:
PASS => Online (hot) patches are not applied to ORACLE_HOME
DATA FROM rac1 - orcl DATABASE - Verify online (hot) patches are not applied on ORACLE_HOME Patch File Name State ================ ========= No patches currently installed

Hide

System Core File Limit

Recommendation
Benefit / Impact:

Core dumps are sometimes needed for debugging purposes

Risk:

Improper setting of COREDUMPSIZE could prevent or prolong troubleshooting if the maximum core file size is too small.

Action / Repair:

The system may not have an adequate COREDUMPSISZE. It is recommended to set the OS core file size to 2GB or greater to aid in troubleshooting issues that generate core dumps (i.e. ORA-7445 errors).

Links
1 Note: 567524.1 - Setting Shell Limits For The User "oracle" In Case Of 10gR2 And 11gR1 RDBMS Software

Needs attention on rac1

Passed on -

Status on rac1:
INFO => Consider increasing the COREDUMPSIZE size
DATA FROM RAC1 - SYSTEM CORE FILE LIMIT

Hide

Optimizer Parameters with Non-Default Settings (Check 1)

Recommendation
Benefit / Impact:

One or more optimizer related parameters are set to a non-default value.

Risk:

The settings for these parameters should be recommended by the application vendor or by Oracle Support. If any optimizer related parameters are set, the application should be certified and tested to work well with the parameter settings. Ensure that the performance is as expected.

Non default settings of optimizer related parameters may result in performance issues. The settings of such optimizer parameters can have an impact on the entire application performance.

Please Note: If the database instance is running Siebel/Peoplesoft Applications then you may not need to change any settings, please check your product documentation.

Action / Repair:

To modify the parameter value, use the alter session or alter system commands. Refer to the Initialization Parameter chapter of the Oracle Database Reference Documentation for more details on setting these parameters.

Needs attention on -

Passed on orcl

Status on orcl:
PASS => All optimizer related parameters are set to the default value (Parameter Set 1 of 3)
DATA FOR ORCL FOR OPTIMIZER PARAMETERS WITH NON-DEFAULT SETTINGS (CHECK 1) Query returned no rows which is expected when the SQL check passes.

Hide

Verify Berkeley Database location for Cloned GI homes

Recommendation
Benefit / Impact:

After cloning a Grid Home the Berkeley Database configuration file ($GI_HOME/crf/admin/crf<node>.ora) in the new home should not be pointing to the previous GI home where it is cloned from. During previous patch set updates Berkeley Database configuration files were found still pointing to the 'before (previously cloned from) home'. It was due an invalid cloning procedure the Berkeley Database location of the 'new home' was not updated during the out of place bundle patching procedure.

Risk:

Berkeley Database configurations still pointing to the old GI home, will fail GI Upgrades to 11.2.0.3. Error messages in $GRID_HOME/log/crflogd/crflogdOUT.log logfile

Action / Repair:

Detect:
cat $GI_HOME/crf/admin/crf`hostname -s`.ora | grep CRFHOME | grep $GI_HOME | wc -l

cat $GI_HOME/crf/admin/crf`hostname -s`.ora | grep BDBLOC | egrep "default|$GI_HOME | wc -l

For each of the above commands, when no '1' is returned, the CRFHOME or BDBLOC as mentioned the crf.ora file has the correct reference to the GI_HOME in it.

To solve this, manually edit $GI_HOME/crf/admin/crf<node>.ora in the cloned Grid Infrastructure Home and change the values for BDBLOC and CRFHOME and make sure none of them point to the previous GI_HOME but to their current home. The same change needs to be done on all nodes in the cluster. It is recommended to set BDBLOC to "default". This needs to be done prior the upgrade.

Links
1 Note: 1485970.1 - Upgrade 11.2.0.1/11.2.0.2 to 11.2.0.3 Failure Of Rootupgrade Script on previously cloned GI

Needs attention on -

Passed on rac1

Status on rac1:
PASS => Berkeley Database location points to correct GI_HOME
DATA FROM RAC1 - VERIFY BERKELEY DATABASE LOCATION FOR CLONED GI HOMES BDBLOC=default MYNAME=rac1 CLUSTERNAME=rac-cluster USERNAME=grid CRFHOME=/oracle/app/11.2.0/grid rac1 5=127.0.0.1 0 rac1 1=127.0.0.1 0 rac1 0=10.25.133.120 61020 rac1 2=10.25.133.120 61021 BDBSIZE=61511 rac2 5=127.0.0.1 0 rac2 1=127.0.0.1 0 rac2 0=10.25.133.121 61020 HOSTS=rac1,rac2 rac2 2=10.25.133.121 61021 STATE=mutated
MASTERPUB=192.168.133.120 MASTER=rac1 DEAD= REPLICA=rac2 ACTIVE=rac1,rac2
Click for more data

Hide

Verify control_file_record_keep_time value is in recommended range

Recommendation
Benefit / Impact:

When a Recovery Manager catalog is not used, the initialization parameter "control_file_record_keep_time" controls the period of time for which circular reuse records are maintained within the database control file. RMAN repository records are kept in circular reuse records. The optimal setting is the maximum number of days in the past that is required to restore and recover a specific database without the use of a RMAN recovery catalog. Setting this parameter within a recommended range (1 to 9 days) has been shown to address most recovery scenarios by ensuring archive logs and backup records are not prematurely aged out making database recovery much more challenging.

Risk:

If the control_file_record_keep_time is set to 0, no RMAN repository records are retained in the controlfile, which causes a much more challenging database recovery operation if RMAN recovery catalog is not available.

If the control_file_record_keep_time is set too high, problems can arise with space management within the control file, expansion of the control file, and control file contention issues.

The impact of verifying that the initialization parameter control_file_record_keep_time value is in the recommended range is minimal. Increasing this value will increase the size of the controlfile and possible query time for backup meta data and archive data.

Action / Repair:

To verify that the FRA space management function is not blocked, as the owner userid of the oracle home with the environment properly set for the target database, execute the following command set:

CF_RECORD_KEEP_TIME="";
CF_RECORD_KEEP_TIME=$(echo -e "set heading off feedback off\n select value from V\$PARAMETER where name = 'control_file_record_keep_time';" | $ORACLE_HOME/bin/sqlplus -s "/ as sysdba");
if [[ $CF_RECORD_KEEP_TIME -ge "1" && $CF_RECORD_KEEP_TIME -le "9" ]]
then echo -e "\nPASS: control_file_record_keep_time is within recommended range [1-9]:" $CF_RECORD_KEEP_TIME;
elif [ $CF_RECORD_KEEP_TIME -eq "0" ]
then echo -e "\nFAIL: control_file_record_keep_time is set to zero:" $CF_RECORD_KEEP_TIME;
else echo -e "\nWARNING: control_file_record_keep_time is not within recommended range [1-9]:" $CF_RECORD_KEEP_TIME;
fi;

The expected output should be:

PASS: control_file_record_keep_time is within recommended range [1-9]: 7

If the output is not as expected, investigate and correct the condition(s).

NOTE: The use of an RMAN recovery catalog is recommended as the best way to avoid the loss of RMAN metadata because of overwritten control file records.

Links
1 Oracle® Database Backup and Recovery User's Guide 11g Release 2 (11.2)
2 Initialization parameter CONTROL_FILE_RECORD_KEEP_TIME

Needs attention on -

Passed on rac1

Status on rac1:
PASS => control_file_record_keep_time is within recommended range [1-9]
DATA FROM rac1 - orcl DATABASE - Verify control_file_record_keep_time value is in recommended range control_file_record_keep_time = 7

Hide

Verify rman controlfile autobackup is set to ON

Recommendation
Benefit / Impact:

The control file is a binary file that records the physical structure of the database and contains important meta data required to recover the database. The database cannot startup or stay up unless all control files are valid. When a Recovery Manager catalog is not used, the control file is needed for database recovery because it contains all backup and recovery meta data.

The impact of verifying and setting "CONTROLFILE AUTOBACKUP" to "ON" is minimal.

Risk:

When a Recovery Manager catalog is not used, loss of the controlfile results in loss of all backup and recovery meta data, which causes a much more challenging database recovery operation

Action / Repair:

To verify that RMAN "CONTROLFILE AUTOBACKUP" is set to "ON", as the owner userid of the oracle home with the environment properly set for the target database, execute the following command set:

RMAN_AUTOBACKUP_STATUS="";
RMAN_AUTOBACKUP_STATUS=$(echo -e "set heading off feedback off\n select value from V\$RMAN_CONFIGURATION where name = 'CONTROLFILE AUTOBACKUP';" | $ORACLE_HOME/bin/sqlplus -s "/ as sysdba");
if [ -n "$RMAN_AUTOBACKUP_STATUS" ] && [ "$RMAN_AUTOBACKUP_STATUS" = "ON" ]
then echo -e "\nPASS: RMAN "CONTROLFILE AUTOBACKUP" is set to \"ON\":" $RMAN_AUTOBACKUP_STATUS;
else
echo -e "\nFAIL: RMAN "CONTROLFILE AUTOBACKUP" should be set to \"ON\":" $RMAN_AUTOBACKUP_STATUS;
fi;

The expected output should be:

PASS: RMAN CONTROLFILE AUTOBACKUP is set to "ON": ON

If the output is not as expected, investigate and correct the condition(s).

For additional information, review information on CONFIGURE syntax in OracleÂ® Database Backup and Recovery Reference 11g Release 2 (11.2).

RMAN> CONFIGURE CONTROLFILE AUTOBACKUP ON;

NOTE: Oracle MAA also recommends periodically backing up the controlfile to trace as additional backup.

SQL> ALTER DATABASE BACKUP CONTROLFILE TO TRACE;

Needs attention on orcl

Passed on -

Status on orcl:
WARNING => RMAN controlfile autobackup should be set to ON
DATA FOR ORCL FOR VERIFY RMAN CONTROLFILE AUTOBACKUP IS SET TO ON

Hide

Registered diskgroups in clusterware registry

Recommendation

Needs attention on -

Passed on rac1

Status on rac1:
PASS => All diskgroups from v$asm_diskgroups are registered in clusterware registry
DATA FROM RAC1 - REGISTERED DISKGROUPS IN CLUSTERWARE REGISTRY Diskgroups from v$asm_diskgroups:- DATA OCR Diskgroups from Clusterware resources:- DATA OCR

Hide

Check for parameter cvuqdisk|1.0.9|1|x86_64

Recommendation

Needs attention on -

Passed on rac1

Status on rac1:
PASS => Package cvuqdisk-1.0.9-1-x86_64 meets or exceeds recommendation

Hide

pam_limits check

Recommendation

Add the following line to the /etc/pam.d/login file, if it does not already exist:

session required pam_limits.so

Links
1 Setting Shell Limits for the oracle User

Needs attention on -

Passed on rac1

Status on rac1:
PASS => pam_limits configured properly for shell limits
DATA FROM RAC1 - PAM_LIMITS CHECK #%PAM-1.0 auth [user_unknown=ignore success=ok ignore=ignore default=bad] pam_securetty.so auth include system-auth account required pam_nologin.so account include system-auth password include system-auth # pam_selinux.so close should be the first session rule session required pam_selinux.so close session required pam_loginuid.so session optional pam_console.so # pam_selinux.so open should only be followed by sessions to be executed in the user context session required pam_selinux.so open session required pam_namespace.so session optional pam_keyinit.so force revoke session include system-auth -session optional pam_ck_connector.so
session required pam_limits.so
Click for more data

Hide

Verify vm.min_free_kbytes

Recommendation
Benefit / Impact:

Maintaining vm.min_free_kbytes >= 524288 (512MB) && <= 1048576 (1GB) helps a Linux system to reclaim memory faster and avoid LowMem pressure issues which can lead to node eviction or other outage or performance issues.

The impact of verifying vm.min_free_kbytes >= 524288 && <= 1048576 is minimal. The impact of adjusting the parameter should include editing the /etc/sysctl.conf file and rebooting the system. It is possible, but not recommended, especially for a system already under LowMem pressure, to modify the setting interactively. However, a reboot should still be performed to make sure the interactive setting is retained through a reboot.

Risk:

Exposure to unexpected node eviction and reboot.

Action / Repair:

To verify that vm.min_free_kbytes is properly set to >= 524288 && <= 1048576 execute the following command

/sbin/sysctl -n vm.min_free_kbytes

cat /proc/sys/vm/min_free_kbytes

If the output is not as expected, investigate and correct the condition. For example if the value is incorrect in /etc/sysctl.conf but current memory matches the incorrect value, simply edit the /etc/sysctl.conf file to include the line "vm.min_free_kbytes >= 524288 && <= 1048576" and reboot the node.

Links
1 Note: 1367153.1 - Top 5 Issues That Cause Node Reboots or Evictions or Unexpected Recycle of CRS

Needs attention on -

Passed on rac1

Status on rac1:
PASS => vm.min_free_kbytes is set as recommended.
DATA FROM RAC1 - VERIFY VM.MIN_FREE_KBYTES Value in sysctl = 524288 Value in active memory (from /proc/sys/vm/min_free_kbytes) = 524288

Hide

Verify transparent hugepages are disabled

Recommendation
Benefit / Impact:

Linux transparent huge pages are enabled by default in OEL 6 and SuSE 11 which might cause soft lockup of CPUs and make system unresponsive which can cause node eviction.

if AnonHugePages are found and value is greater than 0 in /proc/meminfo, means transparent huge pages are enabled.

Risk:

Because Transparent HugePages are known to cause unexpected node reboots and performance problems with RAC, Oracle strongly advises to disable the use of Transparent HugePages. In addition, Transparent Hugepages may cause problems even in a single-instance database environment with unexpected performance problems or delays. As such, Oracle recommends disabling Transparent HugePages on all Database servers running Oracle

Action / Repair:

To turn this feature off,put this in /etc/rc.local:-

echo never > /sys/kernel/mm/transparent_hugepage/enabled
echo never > /sys/kernel/mm/transparent_hugepage/defrag

Links
1 Note: 1557478.1 - RAC - Disable Transparent HugePages on SLES11, RHEL6 and OEL6
2 Disabling transparent hugepages (THP) on Red Hat Enterprise Linux 6 is not taking effect

Needs attention on -

Passed on rac1

Status on rac1:
PASS => Linux transparent huge pages are disabled
DATA FROM RAC1 - VERIFY TRANSPARENT HUGEPAGES ARE DISABLED Note:- Look for AnonHugePages: MemTotal: 3908984 kB MemFree: 769444 kB Buffers: 36420 kB Cached: 1598928 kB SwapCached: 0 kB Active: 1763956 kB Inactive: 831372 kB Active(anon): 1035768 kB Inactive(anon): 399276 kB Active(file): 728188 kB Inactive(file): 432096 kB Unevictable: 229156 kB
Mlocked: 229156 kB SwapTotal: 4095992 kB SwapFree: 4095992 kB Dirty: 3080 kB Writeback: 0 kB AnonPages: 1189204 kB Mapped: 310200 kB Shmem: 411324 kB Slab: 168372 kB SReclaimable: 84492 kB SUnreclaim: 83880 kB KernelStack: 5736 kB PageTables: 58652 kB NFS_Unstable: 0 kB Bounce: 0 kB WritebackTmp: 0 kB CommitLimit: 6050484 kB Committed_AS: 2800012 kB VmallocTotal: 34359738367 kB VmallocUsed: 184592 kB VmallocChunk: 34359509584 kB HardwareCorrupted: 0 kB AnonHugePages: 0 kB HugePages_Total: 0 HugePages_Free: 0 HugePages_Rsvd: 0 HugePages_Surp: 0 Hugepagesize: 2048 kB DirectMap4k: 10240 kB DirectMap2M: 4184064 kB
Click for more data

Hide

Verify data files are recoverable

Recommendation
Benefit / Impact:

When you perform a DML or DDL operation using the NOLOGGING or UNRECOVERABLE clause, database backups made prior to the unrecoverable operation are invalidated and new backups are required. You can specify the SQL ALTER DATABASE or SQL ALTER TABLESPACE statement with the FORCE LOGGING clause to override the NOLOGGING setting; however, this statement will not repair a database that is already invalid.

Risk:

Changes under NOLOGGING will not be available after executing database recovery from a backup made prior to the unrecoverable change.

Action / Repair:

To verify that the data files are recoverable, execute the following Sqlplus command as the userid that owns the oracle home for the database:
select file#, unrecoverable_time, unrecoverable_change# from v$datafile where unrecoverable_time is not null;
If there are any unrecoverable actions, the output will be similar to:
FILE# UNRECOVER UNRECOVERABLE_CHANGE#
---------- --------- ---------------------
11 14-JAN-13 8530544
If nologging changes have occurred and the data must be recoverable then a backup of those datafiles that have nologging operations within should be done immediately. Please consult the following sections of the Backup and Recovery User guide for specific steps to resolve files that have unrecoverable changes

The standard best practice is to enable FORCE LOGGING at the database level (ALTER DATABASE FORCE LOGGING;) to ensure that all transactions are recoverable. However, placing the a database in force logging mode for ETL operations can lead to unnecessary database overhead. MAA best practices call for isolating data that does not need to be recoverable. Such data would include:

Data resulting from temporary loads
Data resulting from transient transformations
Any non critical data

To reduce unnecessary redo generation, do the following:

Specifiy FORCE LOGGING for all tablespaces that you explicitly wish to protect (ALTERTABLESPACE FORCE LOGGING;)
Specify NO FORCE LOGGING for those tablespaces that do not need protection (ALTERTABLESPACE NO FORCE LOGGING;).
Disable force logging at the database level (ALTER DATABASE NO FORCE LOGGING;) otherwise the database level settings will override the tablespace settings.

Once the above is complete, redo logging will function as follows:

Explicit no logging operations on objects in the no logging tablespace will not generate the normal redo (a small amount of redo is always generated for no logging operations to signal that a no logging operation was performed).

All other operations on objects in the no logging tablespace will generate the normal redo.
Operations performed on objects in the force logging tablespaces always generate normal redo.

Note:-Please seek oracle support assistance to mitigate this problem. Upon their guidance, the following commands could help validate, identify corrupted blocks.

oracle> dbv file=<data_file_returned_by_above_command> userid=sys/******
RMAN> validate check logical database;
SQL> select COUNT(*) from v$database_block_corruption;

Links
1 Backing Up Database Files with RMAN
2 Recovering NOLOGGING Tables and Indexes

Needs attention on -

Passed on orcl

Status on orcl:
PASS => The data files are all recoverable
DATA FOR ORCL FOR VERIFY DATA FILES ARE RECOVERABLE Query returned no rows which is expected when the SQL check passes.

Hide

Check for parameter unixODBC-devel|2.2.14|11.el6|x86_64

Recommendation

Needs attention on rac1

Passed on -

Status on rac1:
WARNING => Package unixODBC-devel-2.2.14-11.el6-x86_64 is recommended but NOT installed

Hide

OCR and Voting file location

Recommendation

Links
1 Adding, Replacing, Repairing, and Removing Oracle Cluster Registry Locations
2 Adding, Deleting, or Migrating Voting Disks

Needs attention on -

Passed on rac1

Status on rac1:
PASS => OCR and Voting disks are stored in ASM
DATA FROM RAC1 - OCR AND VOTING FILE LOCATION Status of Oracle Cluster Registry is as follows : Version : 3 Total space (kbytes) : 262120 Used space (kbytes) : 3084 Available space (kbytes) : 259036 ID : 164243238 Device/File Name : +OCR Device/File integrity check succeeded Device/File not configured Device/File not configured Device/File not configured Device/File not configured
Cluster registry integrity check succeeded Logical corruption check bypassed due to non-privileged user
Click for more data

Hide

Parallel Execution Health-Checks and Diagnostics Reports

Recommendation
For every database, there will be a zip file of format <pxhcdr_DBNAME_HOSTNAME_DBVERSION_DATE_TIMESTAMP.zip> in ORAchk output directory.

Needs attention on rac1

Passed on -

Status on rac1:
INFO => Parallel Execution Health-Checks and Diagnostics Reports
DATA FROM rac1 - orcl DATABASE - Parallel Execution Health-Checks and Diagnostics Reports left empty intentionally

Hide

Hardware clock synchronization

Recommendation

it should have commands like

[ -x /sbin/hwclock ] && action $"Syncing hardware clock to system time" /sbin/hwclock $CLOCKFLAGS

Needs attention on -

Passed on rac1

Status on rac1:
PASS => System clock is synchronized to hardware clock at system shutdown
DATA FROM RAC1 - HARDWARE CLOCK SYNCHRONIZATION [ -x /sbin/hwclock -a -e /dev/rtc ] && action $"Syncing hardware clock to system time" /sbin/hwclock --systohc

Hide

TFA Collector status

Recommendation
TFA provides the following key benefits:-
- Encapsulates diagnostic data collection for all CRS/GI and RAC components on all cluster nodes into a single command executed from a single node
- Ability to "trim" diagnostic files during data collection to reduce data upload size
- Ability to isolate diagnostic data collection to a given time period
- Ability to centralize collected diagnostic output to a single server in the cluster
- Ability to isolate diagnostic collection to a particular product component, e.g. ASM, RDBMS, Clusterware
- Optional Real Time Scan Alert Logs for conditions indicating a problem (DB
- Alert Logs, ASM Alert Logs, Clusterware Alert Logs, etc)
- Optional Automatic Data Collection based off of Real Time Scan findings
- Optional On Demand Scan (user initialted) of all log and trace files for conditions indicating a problem
- Optional Automatic Data Collection based off of On Demand Scan findings

Links
1 Note: 1513912.1 - TFA Collector- The Preferred Tool for Automatic or ADHOC Diagnostic Gathering Across All Cluster Nodes
2 Trace File Analyzer Collector (TFA)

Needs attention on -

Passed on rac1

Status on rac1:
PASS => TFA Collector is installed and running
DATA FROM RAC1 - TFA COLLECTOR STATUS -rwxr-xr-x. 1 root root 10437 Dec 1 2015 /etc/init.d/init.tfa -rw-r--r--. 1 root root 5 Jan 25 15:32 /oracle/app/11.2.0/grid/tfa/rac1/tfa_home/internal/.pidfile

Hide

Clusterware resource status

Recommendation

Links
1 Note: 845709.1 - Resource in UNKNOWN state and srvctl Cannot Start/Stop the Resource

Needs attention on -

Passed on rac1

Status on rac1:
PASS => No clusterware resource are in unknown state
DATA FROM RAC1 - CLUSTERWARE RESOURCE STATUS Oracle Clusterware active version on the cluster is [11.2.0.4.0] Oracle Clusterware version on node [rac1] is [11.2.0.4.0] CRS-4638: Oracle High Availability Services is online CRS-4537: Cluster Ready Services is online CRS-4529: Cluster Synchronization Services is online CRS-4533: Event Manager is online -------------------------------------------------------------------------------- NAME TARGET STATE SERVER STATE_DETAILS -------------------------------------------------------------------------------- Local Resources -------------------------------------------------------------------------------- ora.DATA.dg ONLINE ONLINE rac1 ONLINE ONLINE rac2 ora.LISTENER.lsnr ONLINE ONLINE rac1
ONLINE ONLINE rac2 ora.OCR.dg ONLINE ONLINE rac1 ONLINE ONLINE rac2 ora.asm ONLINE ONLINE rac1 Started ONLINE ONLINE rac2 Started ora.gsd OFFLINE OFFLINE rac1 OFFLINE OFFLINE rac2 ora.net1.network ONLINE ONLINE rac1 ONLINE ONLINE rac2 ora.ons ONLINE ONLINE rac1 ONLINE ONLINE rac2 ora.registry.acfs ONLINE ONLINE rac1 ONLINE ONLINE rac2 -------------------------------------------------------------------------------- Cluster Resources -------------------------------------------------------------------------------- ora.LISTENER_SCAN1.lsnr 1 ONLINE ONLINE rac2 ora.cvu 1 ONLINE ONLINE rac2 ora.oc4j 1 ONLINE ONLINE rac1 ora.orcl.db 1 ONLINE ONLINE rac1 Open 2 OFFLINE OFFLINE Instance Shutdown ora.rac1.vip 1 ONLINE ONLINE rac1 ora.rac2.vip 1 ONLINE ONLINE rac2 ora.scan1.vip 1 ONLINE ONLINE rac2 NAME=ora.DATA.dg TYPE=ora.diskgroup.type ACL=owner:grid:rwx,pgrp:oinstall:rwx,other::r-- ACTION_FAILURE_TEMPLATE= ACTION_SCRIPT= AGENT_FILENAME=%CRS_HOME%/bin/oraagent%CRS_EXE_SUFFIX% ALIAS_NAME= AUTO_START=always CHECK_INTERVAL=300 CHECK_TIMEOUT=30 DEFAULT_TEMPLATE= DEGREE=1 DESCRIPTION=CRS resource type definition for ASM disk group resource ENABLED=1 LOAD=1 LOGGING_LEVEL=1 NLS_LANG= NOT_RESTARTING_TEMPLATE= OFFLINE_CHECK_INTERVAL=0 PROFILE_CHANGE_TEMPLATE= RESTART_ATTEMPTS=5 SCRIPT_TIMEOUT=60 START_DEPENDENCIES=hard(ora.asm) pullup(ora.asm) START_TIMEOUT=900 STATE_CHANGE_TEMPLATE= STOP_DEPENDENCIES=hard(intermediate:ora.asm) STOP_TIMEOUT=180 TYPE_VERSION=1.2 UPTIME_THRESHOLD=1d USR_ORA_ENV= USR_ORA_OPI=false USR_ORA_STOP_MODE= VERSION=11.2.0.4.0 NAME=ora.LISTENER.lsnr TYPE=ora.listener.type ACL=owner:grid:rwx,pgrp:oinstall:rwx,other::r-- ACTION_FAILURE_TEMPLATE= ACTION_SCRIPT=%CRS_HOME%/bin/racgwrap%CRS_SCRIPT_SUFFIX% AGENT_FILENAME=%CRS_HOME%/bin/oraagent%CRS_EXE_SUFFIX% ALIAS_NAME=ora.%CRS_CSS_NODENAME_LOWER_CASE%.LISTENER_%CRS_CSS_NODENAME_UPPER_CASE%.lsnr AUTO_START=restore CHECK_INTERVAL=60 CHECK_TIMEOUT=120 DEFAULT_TEMPLATE=PROPERTY(RESOURCE_CLASS=listener) PROPERTY(LISTENER_NAME=PARSE(%NAME%, ., 2)) DEGREE=1 DESCRIPTION=Oracle Listener resource ENABLED=1 ENDPOINTS=TCP:1521 LOAD=1 LOGGING_LEVEL=1 NLS_LANG= NOT_RESTARTING_TEMPLATE= OFFLINE_CHECK_INTERVAL=0 ORACLE_HOME=%CRS_HOME% PORT=1521 PROFILE_CHANGE_TEMPLATE= RESTART_ATTEMPTS=5 SCRIPT_TIMEOUT=60 START_DEPENDENCIES=hard(type:ora.cluster_vip_net1.type) pullup(type:ora.cluster_vip_net1.type) START_TIMEOUT=180 STATE_CHANGE_TEMPLATE= STOP_DEPENDENCIES=hard(intermediate:type:ora.cluster_vip_net1.type) STOP_TIMEOUT=0 TYPE_VERSION=1.2 UPTIME_THRESHOLD=1d USR_ORA_ENV=ORACLE_BASE=/oracle/app/grid USR_ORA_OPI=false VERSION=11.2.0.4.0 NAME=ora.LISTENER_SCAN1.lsnr TYPE=ora.scan_listener.type ACL=owner:grid:rwx,pgrp:oinstall:r-x,other::r-- ACTION_FAILURE_TEMPLATE= ACTION_SCRIPT= ACTIVE_PLACEMENT=1 AGENT_FILENAME=%CRS_HOME%/bin/oraagent%CRS_EXE_SUFFIX% AUTO_START=restore CARDINALITY=1 CHECK_INTERVAL=60 CHECK_TIMEOUT=120 DEFAULT_TEMPLATE=PROPERTY(RESOURCE_CLASS=scan_listener) PROPERTY(LISTENER_NAME=PARSE(%NAME%, ., 2)) DEGREE=1 DESCRIPTION=Oracle SCAN listener resource ENABLED=1 ENDPOINTS=TCP:1521 FAILOVER_DELAY=0 FAILURE_INTERVAL=0 FAILURE_THRESHOLD=0 HOSTING_MEMBERS= LOAD=1 LOGGING_LEVEL=1 NLS_LANG= NOT_RESTARTING_TEMPLATE= OFFLINE_CHECK_INTERVAL=0 PLACEMENT=balanced PORT=1521 PROFILE_CHANGE_TEMPLATE= REGISTRATION_INVITED_NODES= REGISTRATION_INVITED_SUBNETS= RESTART_ATTEMPTS=5 SCRIPT_TIMEOUT=60 SERVER_POOLS=* START_DEPENDENCIES=hard(ora.scan1.vip) dispersion:active(type:ora.scan_listener.type) pullup(ora.scan1.vip) START_TIMEOUT=180 STATE_CHANGE_TEMPLATE= STOP_DEPENDENCIES=hard(intermediate:ora.scan1.vip) STOP_TIMEOUT=0 TYPE_VERSION=2.2 UPTIME_THRESHOLD=1d USR_ORA_ENV= USR_ORA_OPI=false VERSION=11.2.0.4.0 NAME=ora.OCR.dg TYPE=ora.diskgroup.type ACL=owner:grid:rwx,pgrp:oinstall:rwx,other::r-- ACTION_FAILURE_TEMPLATE= ACTION_SCRIPT= AGENT_FILENAME=%CRS_HOME%/bin/oraagent%CRS_EXE_SUFFIX% ALIAS_NAME= AUTO_START=never CHECK_INTERVAL=300 CHECK_TIMEOUT=30 DEFAULT_TEMPLATE= DEGREE=1 DESCRIPTION=CRS resource type definition for ASM disk group resource ENABLED=1 LOAD=1 LOGGING_LEVEL=1 NLS_LANG= NOT_RESTARTING_TEMPLATE= OFFLINE_CHECK_INTERVAL=0 PROFILE_CHANGE_TEMPLATE= RESTART_ATTEMPTS=5 SCRIPT_TIMEOUT=60 START_DEPENDENCIES=hard(ora.asm) pullup(ora.asm) START_TIMEOUT=900 STATE_CHANGE_TEMPLATE= STOP_DEPENDENCIES=hard(intermediate:ora.asm) STOP_TIMEOUT=180 TYPE_VERSION=1.2 UPTIME_THRESHOLD=1d USR_ORA_ENV= USR_ORA_OPI=false USR_ORA_STOP_MODE= VERSION=11.2.0.4.0 NAME=ora.asm TYPE=ora.asm.type ACL=owner:grid:rwx,pgrp:oinstall:rwx,other::r-- ACTION_FAILURE_TEMPLATE= ACTION_SCRIPT= AGENT_FILENAME=%CRS_HOME%/bin/oraagent%CRS_EXE_SUFFIX% ALIAS_NAME=ora.%CRS_CSS_NODENAME%.ASM%CRS_CSS_NODENUMBER%.asm AUTO_START=never CHECK_INTERVAL=60 CHECK_TIMEOUT=30 DEFAULT_TEMPLATE=PROPERTY(RESOURCE_CLASS=asm) ELEMENT(INSTANCE_NAME= %GEN_USR_ORA_INST_NAME%) DEGREE=1 DESCRIPTION=Oracle ASM resource ENABLED=1 GEN_USR_ORA_INST_NAME= GEN_USR_ORA_INST_NAME@SERVERNAME(rac1)=+ASM1 GEN_USR_ORA_INST_NAME@SERVERNAME(rac2)=+ASM2 LOAD=1 LOGGING_LEVEL=1 NLS_LANG= NOT_RESTARTING_TEMPLATE= OFFLINE_CHECK_INTERVAL=0 PROFILE_CHANGE_TEMPLATE= RESTART_ATTEMPTS=5 SCRIPT_TIMEOUT=60 START_DEPENDENCIES=weak(ora.LISTENER.lsnr) START_TIMEOUT=900 STATE_CHANGE_TEMPLATE= STOP_DEPENDENCIES= STOP_TIMEOUT=600 TYPE_VERSION=1.2 UPTIME_THRESHOLD=1d USR_ORA_ENV= USR_ORA_INST_NAME=+ASM%CRS_CSS_NODENUMBER% USR_ORA_OPEN_MODE=mount USR_ORA_OPI=false USR_ORA_STOP_MODE=immediate VERSION=11.2.0.4.0 NAME=ora.cvu TYPE=ora.cvu.type ACL=owner:grid:rwx,pgrp:oinstall:rwx,other::r-- ACTION_FAILURE_TEMPLATE= ACTION_SCRIPT=%CRS_HOME%/bin/cvures%CRS_SCRIPT_SUFFIX% ACTIVE_PLACEMENT=1 AGENT_FILENAME=%CRS_HOME%/bin/scriptagent AUTO_START=restore CARDINALITY=1 CHECK_INTERVAL=21600 CHECK_RESULTS= CHECK_TIMEOUT=600 DEFAULT_TEMPLATE= DEGREE=1 DESCRIPTION=Oracle CVU resource ENABLED=1 FAILOVER_DELAY=0 FAILURE_INTERVAL=0 FAILURE_THRESHOLD=0 HOSTING_MEMBERS= LOAD=1 LOGGING_LEVEL=1 NLS_LANG= NOT_RESTARTING_TEMPLATE= OFFLINE_CHECK_INTERVAL=0 PLACEMENT=balanced PROFILE_CHANGE_TEMPLATE= RESTART_ATTEMPTS=5 SCRIPT_TIMEOUT=30 SERVER_POOLS=* START_DEPENDENCIES=hard(ora.net1.network) pullup(ora.net1.network) START_TIMEOUT=0 STATE_CHANGE_TEMPLATE= STOP_DEPENDENCIES=hard(ora.net1.network) STOP_TIMEOUT=0 TYPE_VERSION=1.1 UPTIME_THRESHOLD=1h USR_ORA_ENV= VERSION=11.2.0.4.0 NAME=ora.gsd TYPE=ora.gsd.type ACL=owner:grid:rwx,pgrp:oinstall:r-x,other::r-- ACTION_FAILURE_TEMPLATE= ACTION_SCRIPT= AGENT_FILENAME=%CRS_HOME%/bin/oraagent%CRS_EXE_SUFFIX% ALIAS_NAME=ora.%CRS_CSS_NODENAME%.gsd AUTO_START=always CHECK_INTERVAL=60 DEFAULT_TEMPLATE= DEGREE=1 DESCRIPTION=Oracle GSD resource ENABLED=0 LOAD=1 LOGGING_LEVEL=1 NLS_LANG= NOT_RESTARTING_TEMPLATE= OFFLINE_CHECK_INTERVAL=0 PROFILE_CHANGE_TEMPLATE= RESTART_ATTEMPTS=5 SCRIPT_TIMEOUT=60 START_DEPENDENCIES= START_TIMEOUT=0 STATE_CHANGE_TEMPLATE= STOP_DEPENDENCIES= STOP_TIMEOUT=0 TYPE_VERSION=1.1 UPTIME_THRESHOLD=1d USR_ORA_ENV= VERSION=11.2.0.4.0 NAME=ora.net1.network TYPE=ora.network.type ACL=owner:root:rwx,pgrp:root:r-x,other::r--,group:oinstall:r-x,user:grid:r-x ACTION_FAILURE_TEMPLATE= ACTION_SCRIPT= AGENT_FILENAME=%CRS_HOME%/bin/orarootagent%CRS_EXE_SUFFIX% ALIAS_NAME= AUTO_START=restore CHECK_INTERVAL=1 DEFAULT_TEMPLATE= DEGREE=1 DESCRIPTION=Oracle Network resource ENABLED=1 LOAD=1 LOGGING_LEVEL=1 NLS_LANG= NOT_RESTARTING_TEMPLATE= OFFLINE_CHECK_INTERVAL=60 PROFILE_CHANGE_TEMPLATE= RESTART_ATTEMPTS=5 SCRIPT_TIMEOUT=60 START_DEPENDENCIES= START_TIMEOUT=0 STATE_CHANGE_TEMPLATE= STOP_DEPENDENCIES= STOP_TIMEOUT=0 TYPE_VERSION=2.2 UPTIME_THRESHOLD=1d USR_ORA_AUTO=static USR_ORA_ENV= USR_ORA_IF=eth0 USR_ORA_NETMASK=255.255.255.0 USR_ORA_SUBNET=192.168.133.0 VERSION=11.2.0.4.0 NAME=ora.oc4j TYPE=ora.oc4j.type ACL=owner:grid:rwx,pgrp:oinstall:rwx,other::r-- ACTION_FAILURE_TEMPLATE= ACTION_SCRIPT=%CRS_HOME%/bin/oc4jctl%CRS_SCRIPT_SUFFIX% ACTIVE_PLACEMENT=1 AGENT_FILENAME=%CRS_HOME%/bin/scriptagent AUTO_START=restore CARDINALITY=1 CHECK_INTERVAL=60 DEFAULT_TEMPLATE= DEGREE=1 DESCRIPTION=Oracle OC4J resource ENABLED=1 FAILOVER_DELAY=0 FAILURE_INTERVAL=3600 FAILURE_THRESHOLD=2 HOSTING_MEMBERS= LOAD=1 LOGGING_LEVEL=1 NLS_LANG= NOT_RESTARTING_TEMPLATE= OFFLINE_CHECK_INTERVAL=0 PLACEMENT=balanced PORT=23792 PROFILE_CHANGE_TEMPLATE= RESTART_ATTEMPTS=1 SCRIPT_TIMEOUT=60 SERVER_POOLS=* START_DEPENDENCIES= START_TIMEOUT=300 STATE_CHANGE_TEMPLATE= STOP_DEPENDENCIES= STOP_TIMEOUT=120 TYPE_VERSION=1.1 UPTIME_THRESHOLD=1d USR_ORA_ENV= VERSION=11.2.0.4.0 NAME=ora.ons TYPE=ora.ons.type ACL=owner:grid:rwx,pgrp:oinstall:r-x,other::r-- ACTION_FAILURE_TEMPLATE= ACTION_SCRIPT= AGENT_FILENAME=%CRS_HOME%/bin/oraagent%CRS_EXE_SUFFIX% ALIAS_NAME=ora.%CRS_CSS_NODENAME%.ons AUTO_START=always CHECK_INTERVAL=60 DEFAULT_TEMPLATE= DEGREE=1 DESCRIPTION=Oracle ONS resource EM_PORT=2016 ENABLED=1 LOAD=1 LOCAL_PORT=6100 LOGGING_LEVEL=1 NLS_LANG= NOT_RESTARTING_TEMPLATE= OFFLINE_CHECK_INTERVAL=0 PROFILE_CHANGE_TEMPLATE= REMOTE_HOSTS= REMOTE_PORT=6200 RESTART_ATTEMPTS=3 SCRIPT_TIMEOUT=60 START_DEPENDENCIES=hard(ora.net1.network) pullup(ora.net1.network) START_TIMEOUT=0 STATE_CHANGE_TEMPLATE= STOP_DEPENDENCIES=hard(ora.net1.network) STOP_TIMEOUT=0 TYPE_VERSION=2.1 UPTIME_THRESHOLD=1d USE_EVM=true USR_ORA_ENV= VERSION=11.2.0.4.0 NAME=ora.orcl.db TYPE=ora.database.type ACL=owner:oracle:rwx,pgrp:oinstall:r--,other::r--,group:dba:r-x,user:grid:r-x ACTION_FAILURE_TEMPLATE= ACTION_SCRIPT= ACTIVE_PLACEMENT=1 AGENT_FILENAME=%CRS_HOME%/bin/oraagent%CRS_EXE_SUFFIX% AUTO_START=restore CARDINALITY=2 CHECK_INTERVAL=1 CHECK_TIMEOUT=30 CLUSTER_DATABASE=true DATABASE_TYPE=RAC DB_UNIQUE_NAME=orcl DEFAULT_TEMPLATE=PROPERTY(RESOURCE_CLASS=database) PROPERTY(DB_UNIQUE_NAME= CONCAT(PARSE(%NAME%, ., 2), %USR_ORA_DOMAIN%, .)) ELEMENT(INSTANCE_NAME= %GEN_USR_ORA_INST_NAME%) ELEMENT(DATABASE_TYPE= %DATABASE_TYPE%) DEGREE=1 DESCRIPTION=Oracle Database resource ENABLED=1 FAILOVER_DELAY=0 FAILURE_INTERVAL=60 FAILURE_THRESHOLD=1 GEN_AUDIT_FILE_DEST=/oracle/app/oracle/admin/orcl/adump GEN_START_OPTIONS= GEN_START_OPTIONS@SERVERNAME(rac1)=open GEN_START_OPTIONS@SERVERNAME(rac2)=open GEN_USR_ORA_INST_NAME= GEN_USR_ORA_INST_NAME@SERVERNAME(rac1)=orcl1 GEN_USR_ORA_INST_NAME@SERVERNAME(rac2)=orcl2 HOSTING_MEMBERS= INSTANCE_FAILOVER=0 LOAD=1 LOGGING_LEVEL=1 MANAGEMENT_POLICY=AUTOMATIC NLS_LANG= NOT_RESTARTING_TEMPLATE= OFFLINE_CHECK_INTERVAL=0 ONLINE_RELOCATION_TIMEOUT=0 ORACLE_HOME=/oracle/app/oracle/product/11.2.0/db ORACLE_HOME_OLD= PLACEMENT=restricted PROFILE_CHANGE_TEMPLATE= RESTART_ATTEMPTS=2 ROLE=PRIMARY SCRIPT_TIMEOUT=60 SERVER_POOLS=ora.orcl SPFILE=+DATA/orcl/spfileorcl.ora START_DEPENDENCIES=hard(ora.DATA.dg) weak(type:ora.listener.type,global:type:ora.scan_listener.type,uniform:ora.ons,global:ora.gns) pullup(ora.DATA.dg) START_TIMEOUT=600 STATE_CHANGE_TEMPLATE= STOP_DEPENDENCIES=hard(intermediate:ora.asm,shutdown:ora.DATA.dg) STOP_TIMEOUT=600 TYPE_VERSION=3.2 UPTIME_THRESHOLD=1h USR_ORA_DB_NAME=orcl USR_ORA_DOMAIN= USR_ORA_ENV= USR_ORA_FLAGS= USR_ORA_INST_NAME= USR_ORA_INST_NAME@SERVERNAME(rac1)=orcl1 USR_ORA_INST_NAME@SERVERNAME(rac2)=orcl2 USR_ORA_OPEN_MODE=open USR_ORA_OPI=false USR_ORA_STOP_MODE=immediate VERSION=11.2.0.4.0 NAME=ora.rac1.vip TYPE=ora.cluster_vip_net1.type ACL=owner:root:rwx,pgrp:root:r-x,other::r--,group:oinstall:r-x,user:grid:r-x ACTION_FAILURE_TEMPLATE= ACTION_SCRIPT= ACTIVE_PLACEMENT=1 AGENT_FILENAME=%CRS_HOME%/bin/orarootagent%CRS_EXE_SUFFIX% AUTO_START=restore CARDINALITY=1 CHECK_INTERVAL=1 CHECK_TIMEOUT=120 DEFAULT_TEMPLATE=PROPERTY(RESOURCE_CLASS=vip) ELEMENT(HOSTING_MEMBERS=%HOSTING_MEMBERS%) DEGREE=1 DESCRIPTION=Oracle VIP resource ENABLED=1 FAILOVER_DELAY=0 FAILURE_INTERVAL=0 FAILURE_THRESHOLD=0 GEN_USR_ORA_STATIC_VIP= GEN_USR_ORA_VIP= HOSTING_MEMBERS=rac1 LOAD=1 LOGGING_LEVEL=1 NLS_LANG= NOT_RESTARTING_TEMPLATE= OFFLINE_CHECK_INTERVAL=0 PLACEMENT=favored PROFILE_CHANGE_TEMPLATE= RESTART_ATTEMPTS=0 SCRIPT_TIMEOUT=60 SERVER_POOLS=* START_DEPENDENCIES=hard(ora.net1.network) pullup(ora.net1.network) START_TIMEOUT=120 STATE_CHANGE_TEMPLATE= STOP_DEPENDENCIES=hard(intermediate:ora.net1.network) STOP_TIMEOUT=0 TYPE_VERSION=2.2 UPTIME_THRESHOLD=1h USR_ORA_ENV= USR_ORA_VIP=rac1-vip VERSION=11.2.0.4.0 NAME=ora.rac2.vip TYPE=ora.cluster_vip_net1.type ACL=owner:root:rwx,pgrp:root:r-x,other::r--,group:oinstall:r-x,user:grid:r-x ACTION_FAILURE_TEMPLATE= ACTION_SCRIPT= ACTIVE_PLACEMENT=1 AGENT_FILENAME=%CRS_HOME%/bin/orarootagent%CRS_EXE_SUFFIX% AUTO_START=restore CARDINALITY=1 CHECK_INTERVAL=1 CHECK_TIMEOUT=120 DEFAULT_TEMPLATE=PROPERTY(RESOURCE_CLASS=vip) ELEMENT(HOSTING_MEMBERS=%HOSTING_MEMBERS%) DEGREE=1 DESCRIPTION=Oracle VIP resource ENABLED=1 FAILOVER_DELAY=0 FAILURE_INTERVAL=0 FAILURE_THRESHOLD=0 GEN_USR_ORA_STATIC_VIP= GEN_USR_ORA_VIP= HOSTING_MEMBERS=rac2 LOAD=1 LOGGING_LEVEL=1 NLS_LANG= NOT_RESTARTING_TEMPLATE= OFFLINE_CHECK_INTERVAL=0 PLACEMENT=favored PROFILE_CHANGE_TEMPLATE= RESTART_ATTEMPTS=0 SCRIPT_TIMEOUT=60 SERVER_POOLS=* START_DEPENDENCIES=hard(ora.net1.network) pullup(ora.net1.network) START_TIMEOUT=120 STATE_CHANGE_TEMPLATE= STOP_DEPENDENCIES=hard(intermediate:ora.net1.network) STOP_TIMEOUT=0 TYPE_VERSION=2.2 UPTIME_THRESHOLD=1h USR_ORA_ENV= USR_ORA_VIP=rac2-vip VERSION=11.2.0.4.0 NAME=ora.registry.acfs TYPE=ora.registry.acfs.type ACL=owner:root:rwx,pgrp:asmadmin:rwx,other::r-- ACTION_FAILURE_TEMPLATE= ACTION_SCRIPT= AGENT_FILENAME=%CRS_HOME%/bin/orarootagent%CRS_EXE_SUFFIX% ALIAS_NAME= AUTO_START=restore CHECK_INTERVAL=30 CHECK_TIMEOUT=600 DEFAULT_TEMPLATE= DEGREE=1 DESCRIPTION="ACFS Global Mount Cluster Registry Component" ENABLED=1 LOAD=1 LOGGING_LEVEL=1 NLS_LANG= NOT_RESTARTING_TEMPLATE= OFFLINE_CHECK_INTERVAL=0 PROFILE_CHANGE_TEMPLATE= RESTART_ATTEMPTS=5 SCRIPT_TIMEOUT=120 START_DEPENDENCIES=hard(ora.asm) pullup:always(ora.asm) START_TIMEOUT=180 STATE_CHANGE_TEMPLATE= STOP_DEPENDENCIES=hard(intermediate:ora.asm) STOP_TIMEOUT=900 TYPE_VERSION=1.1 UPTIME_THRESHOLD=1d USR_ORA_ENV= VERSION=11.2.0.4.0 NAME=ora.scan1.vip TYPE=ora.scan_vip.type ACL=owner:root:rwx,pgrp:root:r-x,other::r--,group:oinstall:r-x,user:grid:r-x ACTION_FAILURE_TEMPLATE= ACTION_SCRIPT= ACTIVE_PLACEMENT=1 AGENT_FILENAME=%CRS_HOME%/bin/orarootagent%CRS_EXE_SUFFIX% AUTO_START=restore CARDINALITY=1 CHECK_INTERVAL=1 CHECK_TIMEOUT=120 DEFAULT_TEMPLATE=PROPERTY(RESOURCE_CLASS=scan_vip) ELEMENT(HOSTING_MEMBERS=%HOSTING_MEMBERS%) DEGREE=1 DESCRIPTION=Oracle SCAN VIP resource ENABLED=1 FAILOVER_DELAY=0 FAILURE_INTERVAL=0 FAILURE_THRESHOLD=0 GEN_USR_ORA_STATIC_VIP= GEN_USR_ORA_VIP= HOSTING_MEMBERS= LOAD=1 LOGGING_LEVEL=1 NLS_LANG= NOT_RESTARTING_TEMPLATE= OFFLINE_CHECK_INTERVAL=0 PLACEMENT=balanced PROFILE_CHANGE_TEMPLATE= RESTART_ATTEMPTS=0 SCAN_NAME=scan-ip SCRIPT_TIMEOUT=60 SERVER_POOLS=* START_DEPENDENCIES=hard(ora.net1.network) dispersion:active(type:ora.scan_vip.type) pullup(global:ora.net1.network) START_TIMEOUT=120 STATE_CHANGE_TEMPLATE= STOP_DEPENDENCIES=hard(intermediate:ora.net1.network) STOP_TIMEOUT=0 TYPE_VERSION=1.1 UPTIME_THRESHOLD=1h USR_ORA_ENV= USR_ORA_VIP=192.168.133.140 VERSION=11.2.0.4.0
Click for more data

Hide

ORA-15196 errors in ASM alert log

Recommendation

Needs attention on -

Passed on rac1

Status on rac1:
PASS => No corrupt ASM header blocks indicated in ASM alert log (ORA-15196 errors)
DATA FROM RAC1 - ORA-15196 ERRORS IN ASM ALERT LOG

Hide

Disks without Disk Group

Recommendation

select name,path,HEADER_STATUS,GROUP_NUMBER from gv$asm_disk where group_number=0;

Needs attention on -

Passed on rac1

Status on rac1:
PASS => No disks found which are not part of any disk group
DATA FROM RAC1 - DISKS WITHOUT DISK GROUP no rows selected

Hide

localhost entry in /etc/hosts

Recommendation
Benefit / Impact:

Various Oracle processes use the loopback address for checking the status of a localhost.

Risk:

Incorrect loopback address configuration in /etc/hosts may result in failure of those processes.

Action / Repair:

IP address 127.0.0.1 should only map to localhost and/or localhost.localdomain, not anything else.

Correct configuration for the loopback address in /etc/hosts is as follows:

127.0.0.1 localhost.localdomain localhost
or
127.0.0.1 localhost localhost.localdomain

The following configuration would be considered incorrect:

127.0.0.1 localhost myhost.com localhost.localdomain
or
127.0.0.1 myhost.com localhost localhost.localdomain

Where myhost.com is the actual hostname or network identity of the host and which should map to the public IP address

In other words, the actual short hostname and/or the fully qualified hostname should not be configured at all for the loopback address.

See the below referenced documents for more details on this subject.

Links
1 Note: 315125.1 - What to Check If The Cluster Synchronization Service Daemon (OCSSD) Does not Start
2 Note: 315720.1 - TNS Listener Fails To Start With Errors: TNS-12537...Error: 29: Illegal seek
3 Note: 1054902.1 - How to Validate Network and Name Resolution Setup for the Clusterware and RAC

Needs attention on -

Passed on rac1

Status on rac1:
PASS => loopback address is configured as recommended in /etc/hosts
DATA FROM RAC1 - LOCALHOST ENTRY IN /ETC/HOSTS 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4

Hide

Redo log file write time latency

Recommendation

Warning: log write elapsed time 564ms, size 2KB

Even though this threshold is very high and latencies below this range could impact the application performance, it is still worth to capture and report it to customers for necessary action.The performance impact of LGWR latencies include commit delays,Broadcast-on-Commit delays etc.

Links
1 Note: 601316.1 - LGWR Is Generating Trace file with "Warning: Log Write Time 540ms, Size 5444kb" In 10.2.0.4 Database

Needs attention on -

Passed on rac1

Status on rac1:
PASS => Redo log write time is less than 500 milliseconds
DATA FROM rac1 - orcl DATABASE - Redo log file write time latency

Hide

Primary database protection with Data Guard

Recommendation

Active Data Guard is the real-time data protection and availability solution that eliminates single point of failure by maintaining one or more synchronized physical replicas of the production database. If an unplanned outage of any kind impacts the production database, applications and users can quickly failover to a synchronized standby, minimizing downtime and preventing data loss. An Active Data Guard standby can be used to offload read-only applications, ad-hoc queries, and backups from the primary database or be dual-purposed as a test system at the same time it provides disaster protection. An Active Data Guard standby can also be used to minimize downtime for planned maintenance when upgrading to new Oracle Database patch sets and major database releases and for select migrations.

For zero data loss protection and fastest recovery time, deploy a local Data Guard standby database with Data Guard Fast-Start Failover and integrated client failover. For protection against outages impacting the entire data center, or a broad geography, deploy a Data Guard standby database at a remote location. Oracle 12c Far Sync Standby can achieve zero data loss protection across Wide Area Network or across regions.

Key HA Benefits:
<ul>
<li>Active Data Guard provides optimal data protection by using physical replication and comprehensive Oracle validation to maintain an exact byte-for-byte copy of the primary database that can be open read-only to offload reporting, ad-hoc queries and backups.</li>
<li>With Oracle 11g release 2 and higher Active Data Guard and real time apply, data block corruptions can be repaired automatically and downtime can be reduced from hours and days of application impact to zero downtime with zero data loss.</li>
<li>With MAA best practices, Data Guard Fast-Start Failover and integrated client failover, downtime from database, cluster and site failures can be reduced from hours to days to seconds.</li>
<li>With remote standby database (Disaster Recovery Site), you have protection from complete site failures.</li>
<li>In all cases, the Active Data Guard instances can be active and used for other activities.</li>
<li>Data Guard can reduce risks and downtime for planned maintenance activities by using Database rolling upgrade with transient logical standby or DBMS rolling, standby-first patch apply and database migrations.</li>
<li>For other advanced replication requirements where read-write access to a replica database is required while it is being synchronized with the primary database see Oracle GoldenGate logical replication. Oracle GoldenGate can be used to support heterogeneous database platforms and database releases, an effective read-write full or subset logical replica and to reduce or eliminate downtime for application, database or system changes. Oracle GoldenGate flexible logical replication solution's main trade-off is the additional administration for application developer and database administrators.</li>
</ul>

Links
1 Oracle Data Guard: Disaster Recovery for Oracle Exadata Database Machine
2 Note: 1265700.1 - Oracle Patch Assurance - Data Guard Standby-First Patch Apply
3 Note: 1302539.1 - Best Practices for Corruption Detection, Prevention, and Automatic Repair - in a Data Guard Configuration
4 HABP: Chapter 9: Configuring Oracle Data Guard 11gR2
5 Goldengate resources

Needs attention on orcl

Passed on -

Status on orcl:
FAIL => Primary database is not protected with Data Guard (standby database) for real-time data protection and availability
DATA FOR ORCL FOR PRIMARY DATABASE PROTECTION WITH DATA GUARD

Hide

Locally managed tablespaces

Recommendation

Links
1 Use Locally Managed Tablespaces
2 Note: 105120.1 - Advantages of Using Locally Managed vs Dictionary Managed Tablespaces

Needs attention on -

Passed on orcl

Status on orcl:
PASS => All tablespaces are locally managed tablespace
DATA FOR ORCL FOR LOCALLY MANAGED TABLESPACES dictionary_managed_tablespace_count = 0

Hide

Automatic segment storage management

Recommendation

Links
1 Use Automatic Segment Space Management

Needs attention on -

Passed on orcl

Status on orcl:
PASS => All tablespaces are using Automatic segment storage management
DATA FOR ORCL FOR AUTOMATIC SEGMENT STORAGE MANAGEMENT Query returned no rows which is expected when the SQL check passes.

Hide

Default Temporary Tablespace

Recommendation

Links
1 Use Temporary Tablespaces and Specify a Default Temporary Tablespace

Needs attention on -

Passed on orcl

Status on orcl:
PASS => Default temporary tablespace is set
DATA FOR ORCL FOR DEFAULT TEMPORARY TABLESPACE DEFAULT_TEMP_TABLESPACE TEMP

Hide

Archivelog Mode

Recommendation

Links
1 Set the Database ARCHIVELOG Mode and FORCE LOGGING Mode

Needs attention on -

Passed on orcl

Status on orcl:
PASS => Database Archivelog Mode is set to ARCHIVELOG
DATA FOR ORCL FOR ARCHIVELOG MODE Archivelog Mode = ARCHIVELOG

Hide

Verify AUD$ and FGA_LOG$ tables use Automatic Segment Space Management

Recommendation
Benefit / Impact:

With AUDIT_TRAIL set for database (AUDIT_TRAIL=db), and the AUD$ and FGA_LOG$ tables located in a dictionary segment space managed SYSTEM tablespace, "gc" wait events are sometimes observed during heavy periods of database logon activity. Testing has shown that under such conditions, placing the AUD$ and FGA_LOG$ tables in the SYSAUX tablespace, which uses automatic segment space management, reduces the space related wait events.

The impact of verifying that the AUD$ and FGA_LOG$ tables are in the SYSAUX table space is low. Moving them if they are not located in the SYSAUX does not require an outage, but should be done during a scheduled maintenance period or slow audit record generation window.

Risk:

If AUD$ and FGA_LOG$ tables are not verifed to use automatic segment space management, there is a risk of a performance slowdown during periods of high database login activity.

Action / Repair:

To verify the segment space management policy currently in use by the AUD$ and FGA_LOG$ tables, use the following Sqlplus command:

select t.table_name,ts.segment_space_management from dba_tables t, dba_tablespaces ts where ts.tablespace_name = t.tablespace_name and t.table_name in ('AUD$','FGA_LOG$');

The output should be:
TABLE_NAME SEGMEN
------------------------------ ------
FGA_LOG$ AUTO
AUD$ AUTO
If one or both of the AUD$ or FGA_LOG$ tables return "MANUAL", use the DBMS_AUDIT_MGMT package to move them to the SYSAUX tablespace:

BEGIN
DBMS_AUDIT_MGMT.set_audit_trail_location(audit_trail_type => DBMS_AUDIT_MGMT.AUDIT_TRAIL_AUD_STD,--this moves table AUD$
audit_trail_location_value => 'SYSAUX');
END;
/

BEGIN
DBMS_AUDIT_MGMT.set_audit_trail_location(audit_trail_type => DBMS_AUDIT_MGMT.AUDIT_TRAIL_FGA_STD,--this moves table FGA_LOG$
audit_trail_location_value => 'SYSAUX');
END;
/

The output should be similar to:
PL/SQL procedure successfully completed.

If the output is not as above, investigate and correct the condition.
NOTE: This "DBMS_AUDIT_MGMT.set_audit_trail" command should be executed as part of the dbca template post processing scripts, but for existing databases, the command can be executed, but since it moves the AUD$ & FGA_LOG$ tables using "alter table ... move" command, it should be executed at a "quiet" time

Links
1 Use Automatic Segment Space Management

Needs attention on orcl

Passed on -

Status on orcl:
FAIL => Table AUD$[FGA_LOG$] should use Automatic Segment Space Management
DATA FOR ORCL FOR VERIFY AUD$ AND FGA_LOG$ TABLES USE AUTOMATIC SEGMENT SPACE MANAGEMENT AUD$ MANUAL FGA_LOG$ MANUAL

Hide

Check for parameter libgcc|4.4.4|13.el6|x86_64

Recommendation

Needs attention on -

Passed on rac1

Status on rac1:
PASS => Package libgcc-4.4.4-13.el6-x86_64 meets or exceeds recommendation

Hide

ASM disk read write error

Recommendation

If the write to the same location succeeds, then the underlying allocation unit (sector) is deemed healthy. This might be because the underlying disk did its own bad block reallocation.

If the write fails, ASM attempts to write the extent to a new allocation unit on the same disk. If this write succeeds, the original allocation unit is marked as unusable. If the write fails, the disk is taken offline.

One unique benefit on ASM-based mirroring is that the database instance is aware of the mirroring. For many types of logical corruptions such as a bad checksum or incorrect System Change Number (SCN), the database instance proceeds through the mirror side looking for valid content and proceeds without errors. If the process in the database that encountered the read is in a position to obtain the appropriate locks to ensure data consistency, it writes the correct data to all mirror sides.

When encountering a write error, a database instance sends the ASM instance a disk offline message.

If database can successfully complete a write to at least one extent copy and receive acknowledgment of the offline disk from ASM, the write is considered successful.

If the write to all mirror side fails, database takes the appropriate actions in response to a write error such as taking the tablespace offline.

When the ASM instance receives a write error message from an database instance or when an ASM instance encounters a write error itself, ASM instance attempts to take the disk offline. ASM consults the Partner Status Table (PST) to see whether any of the disk's partners are offline. If too many partners are already offline, ASM forces the dismounting of the disk group. Otherwise, ASM takes the disk offline.

The ASMCMD remap command was introduced to address situations where a range of bad sectors exists on a disk and must be corrected before ASM or database I/O

Needs attention on -

Passed on orcl

Status on orcl:
PASS => No read/write errors found for ASM disks
DATA FOR ORCL FOR ASM DISK READ WRITE ERROR 0 0

Hide

Block Corruptions

Recommendation

Needs attention on -

Passed on orcl

Status on orcl:
PASS => No reported block corruptions in V$DATABASE_BLOCK_CORRUPTIONS
DATA FOR ORCL FOR BLOCK CORRUPTIONS 0 block_corruptions found in v$database_block_corruptions

Hide

Redo Log File Size

Recommendation
Benefit / Impact:

For performance reasons redo logs should be sized so that they do not switch more than once every 10 minutes or 7 switches in an hour during peak throughput.

Risk:

Switching logs too often causes poor recovery performance on the standby as recovery only checkpoints at a log boundary which pauses recovery while the buffer cache is flushed.

Action / Repair:

To verify redo log size, as the owner userid of a given database, and with the environment properly set to access that database, execute the following command set:

function logswitches {
$ORACLE_HOME/bin/sqlplus -s '/ as sysdba' <<EOF
set feedback off newpage none termout on
alter session set nls_date_format='YYYY/MM/DD HH24:MI:SS';
select * from (
select thread#,sequence#,first_time "LOG START TIME",(blocks*block_size/1024/1024)/((next_time-first_time)*86400) "REDO RATE(MB/s)", \
(((blocks*block_size)/a.average)*100) pct_full
from v\$archived_log, (select avg(bytes) average from v\$log) a
where ((next_time-first_time)*86400<300)
and first_time > (sysdate-90)
and (((blocks*block_size)/a.average)*100)>80
and dest_id=1
order by 4 desc
)
where rownum<11;
exit
EOF
}

export SWITCHES=$(logswitches)

if [ $(echo "$SWITCHES"| wc -l) -le 1 ]
then
echo -e "SUCCESS: Redo logs are appropriately sized"
else
echo
echo -e "WARNING: Redo logs are potentially mis-sized. Below is a list of archived logs from"
echo -e "the previous 90 days which were active for less than 5 minutes and the redo rate seen"
echo -e "for the duration of that log. These indicate the peak redo rate. Resizing of the log"
echo -e "files to accomodate this rate may be required.\n"
echo "$SWITCHES"
fi

The expected output is:

SUCCESS: Redo logs are appropriately sized

Example of a "WARNING" result:

WARNING: Redo logs are potentially mis-sized. Below is a list of archived logs from
the previous 90 days which were active for less than 5 minutes and the redo rate seen
for the duration of that log. These indicate the peak redo rate. Resizing of the log
files to accomodate this rate may be required.

THREAD# SEQUENCE# LOG START TIME REDO RATE(MB/s) PCT_FULL
---------- ---------- ------------------- --------------- ----------
2 12374 2017/02/24 15:03:04 412.881913 95.7611859
2 12382 2017/02/24 15:15:10 408.75571 89.8144871
2 12296 2017/02/24 12:06:26 396.99937 92.0774907
2 12380 2017/02/24 15:09:31 371.651196 90.7351553
2 12385 2017/02/24 15:17:28 370.488538 90.4513031
2 12378 2017/02/24 15:06:18 362.405012 95.1136202
2 12389 2017/02/24 15:20:51 356.672996 95.7862049
2 12377 2017/02/24 15:05:32 354.660687 99.5751441
2 12350 2017/02/24 13:27:40 353.577854 97.113058
2 12354 2017/02/24 13:30:45 347.344079 97.5209206

If logs are frequently switching, the redo logs should be increased in size according to the following chart:

Peak redo rate according
to EM or AWR reports Recommended redo log group size
<1 MB/sec 1 GB
<=3 MB/sec 3 GB
<= 5 MB/sec 4 GB
<= 25 MB/sec 16 GB
<= 50 MB/sec 32 GB
> 50 MB/sec 64 GB

Needs attention on -

Passed on rac1

Status on rac1:
PASS => Redo log files are appropriately sized
DATA FROM rac1 - orcl DATABASE - Redo Log File Size SUCCESS: Redo logs are appropriately sized

Hide

Check for parameter sysstat|9.0.4|11.el6|x86_64

Recommendation

Needs attention on -

Passed on rac1

Status on rac1:
PASS => Package sysstat-9.0.4-11.el6-x86_64 meets or exceeds recommendation

Hide

Check for parameter libgcc|4.4.4|13.el6|i686

Recommendation

Needs attention on -

Passed on rac1

Status on rac1:
PASS => Package libgcc-4.4.4-13.el6-i686 meets or exceeds recommendation

Hide

Check for parameter binutils|2.20.51.0.2|5.11.el6|x86_64

Recommendation

Needs attention on -

Passed on rac1

Status on rac1:
PASS => Package binutils-2.20.51.0.2-5.11.el6-x86_64 meets or exceeds recommendation

Hide

Check for parameter glibc|2.12|1.7.el6|x86_64

Recommendation

Needs attention on -

Passed on rac1

Status on rac1:
PASS => Package glibc-2.12-1.7.el6-x86_64 meets or exceeds recommendation

Hide

Check for parameter libstdc++|4.4.4|13.el6|x86_64

Recommendation

Needs attention on -

Passed on rac1

Status on rac1:
PASS => Package libstdc++-4.4.4-13.el6-x86_64 meets or exceeds recommendation

Hide

Check for parameter libstdc++|4.4.4|13.el6|i686

Recommendation

Needs attention on -

Passed on rac1

Status on rac1:
PASS => Package libstdc++-4.4.4-13.el6-i686 meets or exceeds recommendation

Hide

Check for parameter glibc|2.12|1.7.el6|i686

Recommendation

Needs attention on -

Passed on rac1

Status on rac1:
PASS => Package glibc-2.12-1.7.el6-i686 meets or exceeds recommendation

Hide

Check for parameter gcc|4.4.4|13.el6|x86_64

Recommendation

Needs attention on -

Passed on rac1

Status on rac1:
PASS => Package gcc-4.4.4-13.el6-x86_64 meets or exceeds recommendation

Hide

Check for parameter make|3.81|19.el6|

Recommendation

Needs attention on -

Passed on rac1

Status on rac1:
PASS => Package make-3.81-19.el6 meets or exceeds recommendation

Hide

Check for parameter libstdc++-devel|4.4.4|13.el6|i686

Recommendation

Needs attention on -

Passed on rac1

Status on rac1:
PASS => Package libstdc++-devel-4.4.4-13.el6-i686 meets or exceeds recommendation

Hide

Check for parameter libaio-devel|0.3.107|10.el6|x86_64

Recommendation

Needs attention on -

Passed on rac1

Status on rac1:
PASS => Package libaio-devel-0.3.107-10.el6-x86_64 meets or exceeds recommendation

Hide

Check for parameter libaio|0.3.107|10.el6|x86_64

Recommendation

Needs attention on -

Passed on rac1

Status on rac1:
PASS => Package libaio-0.3.107-10.el6-x86_64 meets or exceeds recommendation

Hide

Check for parameter unixODBC-devel|2.2.14|11.el6|i686

Recommendation

Needs attention on rac1

Passed on -

Status on rac1:
WARNING => Package unixODBC-devel-2.2.14-11.el6-i686 is recommended but NOT installed

Hide

Check for parameter compat-libstdc++-33|3.2.3|69.el6|i686

Recommendation

Needs attention on -

Passed on rac1

Status on rac1:
PASS => Package compat-libstdc++-33-3.2.3-69.el6-i686 meets or exceeds recommendation

Hide

Check for parameter glibc-devel|2.12|1.7.el6|x86_64

Recommendation

Needs attention on -

Passed on rac1

Status on rac1:
PASS => Package glibc-devel-2.12-1.7.el6-x86_64 meets or exceeds recommendation

Hide

Check for parameter glibc-devel|2.12|1.7.el6|i686

Recommendation

Needs attention on -

Passed on rac1

Status on rac1:
PASS => Package glibc-devel-2.12-1.7.el6-i686 meets or exceeds recommendation

Hide

Check for parameter compat-libcap1|1.10|1|x86_64

Recommendation

Needs attention on -

Passed on rac1

Status on rac1:
PASS => Package compat-libcap1-1.10-1-x86_64 meets or exceeds recommendation

Hide

Check for parameter ksh|20100621|12.el6|x86_64

Recommendation

Needs attention on -

Passed on rac1

Status on rac1:
PASS => Package ksh-20100621-12.el6-x86_64 meets or exceeds recommendation

Hide

Check for parameter unixODBC|2.2.14|11.el6|i686

Recommendation

Needs attention on rac1

Passed on -

Status on rac1:
WARNING => Package unixODBC-2.2.14-11.el6-i686 is recommended but NOT installed

Hide

Check for parameter libaio|0.3.107|10.el6|i686

Recommendation

Needs attention on -

Passed on rac1

Status on rac1:
PASS => Package libaio-0.3.107-10.el6-i686 meets or exceeds recommendation

Hide

Check for parameter libstdc++-devel|4.4.4|13.el6|x86_64

Recommendation

Needs attention on -

Passed on rac1

Status on rac1:
PASS => Package libstdc++-devel-4.4.4-13.el6-x86_64 meets or exceeds recommendation

Hide

Check for parameter gcc-c++|4.4.4|13.el6|x86_64

Recommendation

Needs attention on -

Passed on rac1

Status on rac1:
PASS => Package gcc-c++-4.4.4-13.el6-x86_64 meets or exceeds recommendation

Hide

Check for parameter compat-libstdc++-33|3.2.3|69.el6|x86_64

Recommendation

Needs attention on -

Passed on rac1

Status on rac1:
PASS => Package compat-libstdc++-33-3.2.3-69.el6-x86_64 meets or exceeds recommendation

Hide

Check for parameter libaio-devel|0.3.107|10.el6|i686

Recommendation

Needs attention on -

Passed on rac1

Status on rac1:
PASS => Package libaio-devel-0.3.107-10.el6-i686 meets or exceeds recommendation

Hide

Remote listener set to scan name

Recommendation
instance.

Links
1 SINGLE CLIENT ACCESS NAME (SCAN)

Needs attention on -

Passed on rac1

Status on rac1:
PASS => Remote listener is set to SCAN name
DATA FROM rac1 - orcl DATABASE - Remote listener set to scan name remote listener name = scan-ip scan name = scan-ip

Hide

tnsping to remote listener parameter

Recommendation

Links
1 Note: 453293.1 - 10g & 11g :Configuration of TAF(Transparent Application Failover) and Load Balancing

Needs attention on -

Passed on rac1

Status on rac1:
PASS => Value of remote_listener parameter is able to tnsping
DATA FROM rac1 - orcl DATABASE - tnsping to remote listener parameter TNS Ping Utility for Linux: Version 11.2.0.4.0 - Production on 25-JAN-2018 16:42:54 Copyright (c) 1997, 2013, Oracle. All rights reserved. Used parameter files: Used HOSTNAME adapter to resolve the alias Attempting to contact (DESCRIPTION=(CONNECT_DATA=(SERVICE_NAME=))(ADDRESS=(PROTOCOL=TCP)(HOST=192.168.133.140)(PORT=1521))) OK (0 msec)

Hide

SCAN TNSNAMES.ora resolution

Recommendation
Benefit / Impact:

Database instance SCAN listener registration, connection failover and load balancing implications

Risk:

Potential problems with database instances registering with SCAN listeners, connection load balancing and fail over issues

Action / Repair:

No server-side TNSNAMES.ora should have an alias resolving using the SCAN EZ-syntax,e.g.,

myscan.oracle.com:1521= (DESCRIPTION= (ADDRESS_LIST=(ADDRESS=...

If any such syntax exists, for whatever reason such as third party application configuration, it should be removed.

Needs attention on -

Passed on rac1

Status on rac1:
PASS => No server-side TNSNAMES.ora aliases resolve using SCAN EZ syntax
DATA FROM rac1 - orcl DATABASE - SCAN TNSNAMES.ora resolution scan name = scan-ip /oracle/app/oracle/product/11.2.0/db/network/admin/tnsnames.ora file is ORCL = (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = scan-ip)(PORT = 1521)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = orcl) ) ) pridb =
(DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST =scan-ip)(PORT = 1521)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME =orcl) ) ) stydb = (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = hisdg)(PORT = 1521)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = orcl) ) )
Click for more data

Hide

ezconnect configuration in sqlnet.ora

Recommendation

Links
1 Note: 1070607.1 - 11.2 Scan and Node TNS Listener Setup Examples

Needs attention on -

Passed on rac1

Status on rac1:
PASS => ezconnect is configured in sqlnet.ora
DATA FROM RAC1 - EZCONNECT CONFIGURATION IN SQLNET.ORA NAMES.DIRECTORY_PATH= (TNSNAMES, EZCONNECT) ADR_BASE = /oracle/app/grid

Hide

Duplicate sys/system objects

Recommendation

Read the Exceptions carefully before taking action.

Links
1 Note: 1030426.6 - How to Clean Up Duplicate Objects Owned by SYS and SYSTEM Schema

Needs attention on -

Passed on orcl

Status on orcl:
PASS => No Duplicate objects were found in the SYS and SYSTEM schemas
DATA FOR ORCL FOR DUPLICATE SYS/SYSTEM OBJECTS Query returned no rows which is expected when the SQL check passes.

Hide

Invalid Registry Component Check

Recommendation

Links
1 Note: 472937.1 - Information On Installed Database Components and Schemas
2 Note: 753041.1 - How To Diagnose Components With NON VALID Status In DBA_REGISTRY After an Upgrade

Needs attention on -

Passed on orcl

Status on orcl:
PASS => No Invalid Registry Components were found
DATA FOR ORCL FOR INVALID REGISTRY COMPONENT CHECK AMD OLAP Catalog VALID 11.2.0.4.0 APEX Oracle Application Express VALID 3.2.1.00.12 APS OLAP Analytic Workspace VALID 11.2.0.4.0
CATALOG Oracle Database Catalog Views VALID 11.2.0.4.0 CATJAVA Oracle Database Java Packages VALID 11.2.0.4.0 CATPROC Oracle Database Packages and Types VALID 11.2.0.4.0 CONTEXT Oracle Text VALID 11.2.0.4.0 EM Oracle Enterprise Manager VALID 11.2.0.4.0 EXF Oracle Expression Filter VALID 11.2.0.4.0 JAVAVM JServer JAVA Virtual Machine VALID 11.2.0.4.0 ORDIM Oracle Multimedia VALID 11.2.0.4.0 OWB OWB VALID 11.2.0.4.0 OWM Oracle Workspace Manager VALID 11.2.0.4.0 RAC Oracle Real Application Clusters VALID 11.2.0.4.0 RUL Oracle Rules Manager VALID 11.2.0.4.0 SDO Spatial VALID 11.2.0.4.0 XDB Oracle XML Database VALID 11.2.0.4.0 XML Oracle XDK VALID 11.2.0.4.0 XOQ Oracle OLAP API VALID 11.2.0.4.0
Click for more data

Hide

Hang and Deadlock material

Recommendation

1. V$Wait_Chains - The DB (the dia0 BG process) samples local hanganalyze every 3 seconds and global hanganalyze every 10 seconds and stores it in memory. V$Wait_Chains is an interface to seeing this "hanganalyze cache". That means at any moment you can query v$wait_chains and see what hanganalyze knows about the current wait chains at any given time. In 11.2 with a live hang this is the first thing you can use to know who the blocker and final blockers are.For more info see following: NOTE:1428210.1 - Troubleshooting Database Contention With V$Wait_Chains

2. Procwatcher - In v11, this script samples v$wait_chains every 90 seconds and collects interesting info about the processes involved in wait chains (short stacks, current wait, current SQL, recent ASH data, locks held, locks waiting for, latches held, etc...). This script works in RAC and non-RAC and is a proactive way to trap hang data even if you can't predict when the problem will happen. Some very large customers are proactively either planning to, or using this script on hundreds of systems to catch session contention. For more info see followings: NOTE:459694.1 - Procwatcher: Script to Monitor and Examine Oracle DB and Clusterware Processes AND NOTE:1352623.1 - How To Troubleshoot Database Contention With Procwatcher.

3. Hanganalyze Levels - Hanganalyze format and output is completely different starting in version 11. In general we recommend getting hanganalyze dumps at level 3. Make sure you always get a global hanganalyze in RAC.

4. Systemstate Levels - With a large SGA and a large number of processes, systemstate dumps at level 266 or 267 can dump a HUGE amount of data and take even hours to dump on large systems. That situation should be avoided. One lightweight alternative is a systemstate dump at level 258. This is basically a level 2 systemstate plus short stacks and is much cheaper than level 266 or 267 and level 258 still has the most important info that support engineers typically look at like process info, latch info, wait events, short stacks, and more at a fraction of the cost.

Note that bugs 11800959 and 11827088 have significant impact on systemstate dumps. If not on 11.2.0.3+ or a version that has both fixes applied, systemstate dumps at levels 10, 11, 266, and 267 can be VERY expensive in RAC. In versions < 11.2.0.3 without these fixes applied, systemstate dumps at level 258 would typically be advised.

NOTE:1353073.1 - Exadata Diagnostic Collection Guide while for Exadata many of the concepts for hang detection and analysis are the same for normal RAC systems.

5.Hang Management and LMHB provide good proactive hang related data. For Hang Management see following note: 1270563.1 Hang Manager 11.2.0.2

Links
1 Note: 459694.1 - Procwatcher: Script to Monitor and Examine Oracle DB and Clusterware Processes
2 Note: 1352623.1 - How To Troubleshoot Database Contention With Procwatcher
3 Note: 1428210.1 - Troubleshooting Database Contention With V$Wait_Chains
4 Note: 1353073.1 - Exadata Diagnostic Collection Guide

Needs attention on rac1

Passed on -

Status on rac1:
INFO => Information about hanganalyze and systemstate dump
DATA FROM RAC1 - HANG AND DEADLOCK MATERIAL left blank intentionally

Hide

Check for parameter recyclebin

Recommendation
Benefit / Impact:

Experience and testing has shown that certain database initialization parameters should be set at specific values. These are the best practice values set at deployment time. By setting these database initialization parameters as recommended, known problems may be avoided and performance maximized.
The parameters are common to all database instances. The impact of setting these parameters is minimal. The performance related settings provide guidance to maintain highest stability without sacrificing performance. Changing the default performance settings can be done after careful performance evaluation and clear understanding of the performance impact.

Risk:

If the database initialization parameters are not set as recommended, a variety of issues may be encountered, depending upon which initialization parameter is not set as recommended, and the actual set value.

Action / Repair:

"RECYCLEBIN = ON" provides higher availability by enabling the Flashback Drop feature. "ON" is the default value and should not be changed.

Needs attention on -

Passed on orcl1

Status on orcl1:
PASS => RECYCLEBIN on PRIMARY is set to the recommended value
orcl1.recyclebin = on

Hide

Check ORACLE_PATCH 17478514 for RDBMS home

Recommendation

Apply the latest recommended CPU or PSU patches based on your maintenance schedule.

For further information review the referenced MOS document.

Links
1 Note: 1393363.1 - Installing, Executing and Interpreting output from the "scnhealthcheck.sql" script

Needs attention on rac1:/oracle/app/oracle/product/11.2.0/db

Passed on -

Status on rac1:/oracle/app/oracle/product/11.2.0/db:
INFO => IMPORTANT: Oracle Database Patch 17478514 PSU is NOT applied to RDBMS Home
Oracle Interim Patch Installer version 11.2.0.3.16 Copyright (c) 2018, Oracle Corporation. All rights reserved. Oracle Home : /oracle/app/oracle/product/11.2.0/db Central Inventory : /oracle/app/oraInventory from : /oracle/app/oracle/product/11.2.0/db/oraInst.loc OPatch version : 11.2.0.3.16 OUI version : 11.2.0.4.0 Log file location : /oracle/app/oracle/product/11.2.0/db/cfgtoollogs/opatch/opatch2018-01-25_16-37-33PM_1.log Lsinventory Output file location : /oracle/app/oracle/product/11.2.0/db/cfgtoollogs/opatch/lsinv/lsinventory2018-01-25_16-37-33PM.txt ------------------------------------------------------------------------------------------------------ Local Machine Information:: Hostname: rac1 ARU platform id: 226 ARU platform description:: Linux x86-64
Installed Top-level Products (1): Oracle Database 11g 11.2.0.4.0 There are 1 products installed in this Oracle Home. There are no Interim patches installed in this Oracle Home. ------------------------------------------------------------------------------------------------------ OPatch succeeded.
Click for more data

Hide

Check for parameter fast_start_mttr_target

Recommendation
Benefit / Impact:

To optimize run time performance for write/redo generation intensive workloads. Increasing fast_start_mttr_target from the default will reduce checkpoint writes from DBWR processes, making more room for LGWR IO.

Risk:

Performance implications if set too aggressively (lower setting = more aggressive), but a trade-off between performance and availability. This trade-off and the type of workload needs to be evaluated and a decision made whether the default is needed to meet RTO objectives. fast_start_mttr_target should be set to the desired RTO (Recovery Time Objective) while still maintaining performance SLAs. So this needs to be evaluated on a case by case basis.

Action / Repair:

Consider increasing fast_start_mttr_target to 300 (five minutes) from the default. The trade-off is that instance recovery will run longer, so if instance recovery is more important than performance, then keep fast_start_mttr_target at the default.

Keep in mind that an application with inadequately sized redo logs will likely not see an affect from this change due to frequent log switches so follow best practices for sizing redo logs.

Considerations for a direct writes in a data warehouse type of application: Even though direct operations aren't using the buffer cache, fast_start_mttr_target is very effective at controlling crash recovery time because it ensures adequate checkpointing for the few buffers that are resident (ex: undo segment headers).

Needs attention on orcl1

Passed on -

Status on orcl1:
WARNING => fast_start_mttr_target should be greater than or equal to 300.
orcl1.fast_start_mttr_target = 0

Hide

Check for parameter undo_retention

Recommendation

Oracle Flashback Technology enables fast logical failure repair. Oracle recommends that you use automatic undo management with sufficient space to attain your desired undo retention guarantee, enable Oracle Flashback Database, and allocate sufficient space and I/O bandwidth in the fast recovery area. Application monitoring is required for early detection. Effective and fast repair comes from leveraging and rehearsing the most common application specific logical failures and using the different flashback features effectively (e.g flashback query, flashback version query, flashback transaction query, flashback transaction, flashback drop, flashback table, and flashback database, and 12.2 flashback pluggable database (PDB)).

Key HA Benefits:
<ul>
<li>With application monitoring and rehearsed repair actions with flashback technologies, application downtime can reduce from hours and days to the time to detect the logical inconsistency.</li>
<li>Fast repair for logical failures caused by malicious or accidental DML or DDL operations.</li>
<li>Effect fast point-in-time repair at the appropriate level of granularity: transaction, table, pluggable database, or database.</li></ul>
Questions that need to be addressed by your application and operations team:
<ol>
<li>Can your application or monitoring infrastructure detect logical inconsistencies?</li>
<li>Is your operations team prepared to use various flashback technologies to repair quickly and efficiently?</li>
<li>Is security practices enforced to prevent unauthorized privileges that can result logical inconsistencies?</li>
</ol>

Needs attention on -

Passed on orcl1

Status on orcl1:
PASS => Database parameter UNDO_RETENTION on PRIMARY is not null
orcl1.undo_retention = 900

Hide

Verify all "BIGFILE" tablespaces have non-default "MAXBYTES" values set

Recommendation
Benefit / Impact:

"MAXBYTES" is the SQL attribute that expresses the "MAXSIZE" value that is used in the DDL command to set "AUTOEXTEND" to "ON". By default, for a bigfile tablespace, the value is "3.5184E+13", or "35184372064256". The benefit of having "MAXBYTES" set at a non-default value for "BIGFILE" tablespaces is that a runaway operation or heavy simultaneous use (e.g., temp tablespace) cannot take up all the space in a diskgroup.

The impact of verifying that "MAXBYTES" is set to a non-default value is minimal. The impact of setting the "MAXSIZE" attribute to a non-default value "varies depending upon if it is done during database creation, file addition to a tablespace, or added to an existing file.

Risk:

The risk of running out of space in a diskgroup varies by application and cannot be quantified here. A diskgroup running out of space may impact the entire database as well as ASM operations (e.g., rebalance operations).

Action / Repair:

To obtain a list of file numbers and bigfile tablespaces that have the "MAXBYTES" attribute at the default value, enter the following sqlplus command logged into the database as sysdba:
select file_id, a.tablespace_name, autoextensible, maxbytes
from (select file_id, tablespace_name, autoextensible, maxbytes from dba_data_files where autoextensible='YES' and maxbytes = 35184372064256) a, (select tablespace_name from dba_tablespaces where bigfile='YES') b
where a.tablespace_name = b.tablespace_name
union
select file_id,a.tablespace_name, autoextensible, maxbytes
from (select file_id, tablespace_name, autoextensible, maxbytes from dba_temp_files where autoextensible='YES' and maxbytes = 35184372064256) a, (select tablespace_name from dba_tablespaces where bigfile='YES') b
where a.tablespace_name = b.tablespace_name;

The output should be:no rows returned

If you see output similar to:

FILE_ID TABLESPACE_NAME AUT MAXBYTES
---------- ------------------------------ --- ----------
1 TEMP YES 3.5184E+13
3 UNDOTBS1 YES 3.5184E+13
4 UNDOTBS2 YES 3.5184E+13

Investigate and correct the condition.

Needs attention on -

Passed on orcl

Status on orcl:
PASS => All bigfile tablespaces have non-default maxbytes values set
DATA FOR ORCL FOR VERIFY ALL "BIGFILE" TABLESPACES HAVE NON-DEFAULT "MAXBYTES" VALUES SET Query returned no rows which is expected when the SQL check passes.

Hide

Clusterware status

Recommendation

Links
1 Client Failover Best Practices for Highly Available Oracle Databases: Oracle Database 11g Release 2

Needs attention on -

Passed on rac1

Status on rac1:
PASS => Clusterware is running
DATA FROM RAC1 - CLUSTERWARE STATUS -------------------------------------------------------------------------------- NAME TARGET STATE SERVER STATE_DETAILS -------------------------------------------------------------------------------- Local Resources -------------------------------------------------------------------------------- ora.DATA.dg ONLINE ONLINE rac1 ONLINE ONLINE rac2 ora.LISTENER.lsnr ONLINE ONLINE rac1 ONLINE ONLINE rac2 ora.OCR.dg ONLINE ONLINE rac1 ONLINE ONLINE rac2 ora.asm ONLINE ONLINE rac1 Started
ONLINE ONLINE rac2 Started ora.gsd OFFLINE OFFLINE rac1 OFFLINE OFFLINE rac2 ora.net1.network ONLINE ONLINE rac1 ONLINE ONLINE rac2 ora.ons ONLINE ONLINE rac1 ONLINE ONLINE rac2 ora.registry.acfs ONLINE ONLINE rac1 ONLINE ONLINE rac2 -------------------------------------------------------------------------------- Cluster Resources -------------------------------------------------------------------------------- ora.LISTENER_SCAN1.lsnr 1 ONLINE ONLINE rac2 ora.cvu 1 ONLINE ONLINE rac2 ora.oc4j 1 ONLINE ONLINE rac1 ora.orcl.db 1 ONLINE ONLINE rac1 Open 2 OFFLINE OFFLINE Instance Shutdown ora.rac1.vip 1 ONLINE ONLINE rac1 ora.rac2.vip 1 ONLINE ONLINE rac2 ora.scan1.vip 1 ONLINE ONLINE rac2
Click for more data

Hide

Flashback database on primary

Recommendation

Oracle Flashback Technology enables fast logical failure repair. Oracle recommends that you use automatic undo management with sufficient space to attain your desired undo retention guarantee, enable Oracle Flashback Database, and allocate sufficient space and I/O bandwidth in the fast recovery area. Application monitoring is required for early detection. Effective and fast repair comes from leveraging and rehearsing the most common application specific logical failures and using the different flashback features effectively (e.g flashback query, flashback version query, flashback transaction query, flashback transaction, flashback drop, flashback table, and flashback database, and 12.2 flashback pluggable database (PDB)).

Key HA Benefits:
<ul>
<li>With application monitoring and rehearsed repair actions with flashback technologies, application downtime can reduce from hours and days to the time to detect the logical inconsistency.</li>
<li>Fast repair for logical failures caused by malicious or accidental DML or DDL operations.</li>
<li>Effect fast point-in-time repair at the appropriate level of granularity: transaction, table, pluggable database, or database.</li></ul>
Questions that need to be addressed by your application and operations team:
<ol>
<li>Can your application or monitoring infrastructure detect logical inconsistencies?</li>
<li>Is your operations team prepared to use various flashback technologies to repair quickly and efficiently?</li>
<li>Is security practices enforced to prevent unauthorized privileges that can result logical inconsistencies?</li>
</ol>

Links
1 HABP: Chapter 5.1.4: Enable Flashback Database in 11gR2

Needs attention on orcl

Passed on -

Status on orcl:
FAIL => Flashback on PRIMARY is not configured
DATA FOR ORCL FOR FLASHBACK DATABASE ON PRIMARY Flashback status = NO

Hide

Database init parameter DB_BLOCK_CHECKING

Recommendation
Benefit / Impact:

Initially db_block_checking is set to off due to potential performance impact. Performance testing is particularly important given that overhead is incurred on every block change. Block checking typically causes 1% to 10% overhead, but for update and insert intensive applications (such as Redo Apply at a standby database) the overhead can be much higher. OLTP compressed tables also require additional checks that can result in higher overhead depending on the frequency of updates to those tables. Workload specific testing is required to assess whether the performance overhead is acceptable.

Risk:

If the database initialization parameters are not set as recommended, a variety of issues may be encountered, depending upon which initialization parameter is not set as recommended, and the actual set value.

Action / Repair:

Based on performance testing results set the primary or standby database to either medium or full depending on the impact. If performance concerns prevent setting DB_BLOCK_CHECKING to either FULL or MEDIUM at a primary database, then it becomes even more important to enable this at the standby database. This protects the standby database from logical corruption that would be undetected at the primary database.
For higher data corruption detection and prevention, enable this setting but performance impacts vary per workload.Evaluate performance impact.

Links
1 Note: 1302539.1 - Best Practices for Corruption Detection, Prevention, and Automatic Repair - in a Data Guard Configuration
2 Protect Against Data Corruption

Needs attention on rac1

Passed on -

Status on rac1:
WARNING => Database parameter DB_BLOCK_CHECKING on PRIMARY is NOT set to the recommended value.
DATA FROM rac1 - orcl DATABASE - Database init parameter DB_BLOCK_CHECKING DB_BLOCK_CHECKING = FALSE

Hide

CSS log file size

Recommendation
Benefit / Impact:

The default size for the CSS logs in $GRID_HOME/log/<nodename>/cssd directory is 50M. Increase the CSS log file size to 150M to save more log data needed in the event of a problem.

The overhead for this verification step is minimal.

The cost of this increase is about 1GB of filesystem space per node.

Risk:

For certain types of problems, this smaller size causes a wrap and important information is lost to debug problems.

Action / Repair:

To get current css log file size, run the following command:

# $GRID_HOME/bin/crsctl get css logfilesize

The output should be similar to:

CRS-4676: Successful get logfilesize 52428800 for Cluster Synchronization Services.

The command to increase this size is:

# $GRID_HOME/bin/crsctl set css logfilesize 157286400

and should be run on all nodes. The cost of this increase is about 1GB of filesystem space per node.

Needs attention on rac1

Passed on -

Status on rac1:
WARNING => CSS log file size has not been increased from default (50 MB) .
DATA FROM RAC1 - CSS LOG FILE SIZE CRS-4676: Successful get logfilesize 52428800 for Cluster Synchronization Services.

Hide

umask setting for RDBMS owner

Recommendation

Links
1 Note: 68304.1 - WHAT IS UMASK
2 Configuring the oracle User's Environment

Needs attention on -

Passed on rac1

Status on rac1:
PASS => umask for RDBMS owner is set to 0022
DATA FROM RAC1 - UMASK SETTING FOR RDBMS OWNER oracle user's umask = 0022

Hide

Check for parameter db_lost_write_protect

Recommendation
Benefit / Impact:

Experience and testing has shown that certain database initialization parameters should be set at specific values. These are the best practice values set at deployment time. By setting these database initialization parameters as recommended, known problems may be avoided and performance maximized.
The parameters are common to all database instances. The impact of setting these parameters is minimal.
The performance related settings provide guidance to maintain highest stability without sacrificing performance. Changing the default performance settings can be done after careful performance evaluation and clear understanding of the performance impact.

Risk:

If the database initialization parameters are not set as recommended, a variety of issues may be encountered, depending upon which initialization parameter is not set as recommended, and the actual set value.

Action / Repair:

This is important for data block lost write detection and repair. Enable for
primary and standby databases.

See referenced MOS notes for more info.

Links
1 Note: 1265884.1 - Resolving ORA-752 or ORA-600 [3020] During Standby Recovery
2 Note: 1302539.1 - Best Practices for Corruption Detection, Prevention, and Automatic Repair - in a Data Guard Configuration
3 Protect Against Data Corruption

Needs attention on orcl1

Passed on -

Status on orcl1:
FAIL => Database parameter DB_LOST_WRITE_PROTECT is not set to recommended value
orcl1.db_lost_write_protect = NONE

Hide

ORA_CRS_HOME env variable for Grid infrastructure owner

Recommendation
Benefit / Impact:

Unset Oracle related environment variables especially ORA_CRS_HOME before starting OUI or before executing any root script for both root and grid user.

This requirement is documented in Oracle® Grid Infrastructure Installation Guide 11g Release 2 (11.2) .

"Unset Oracle environment variables. If you have set ORA_CRS_HOME as an environment variable, then unset it before starting an installation or upgrade. You should never use ORA_CRS_HOME as an environment variable.

If you have had an existing installation on your system, and you are using the same user account to install this installation, then unset the following environment variables: ORA_CRS_HOME; ORACLE_HOME; ORA_NLS10; TNS_ADMIN"

Risk:

Setting this variable can cause problems for various Oracle components, and it is never necessary for CRS programs because they all have wrapper scripts.

Links
1 Unset Oracle Environment Variables
2 Note: 952925.1 - NETCA & ASMCA Fail during Upgrade of CRS/ASM to Grid Infrastructure 11gR2
3 Note: 1502996.1 - Environment Variable ORA_CRS_HOME MUST be UNSET in 11gR2

Needs attention on -

Passed on rac1

Status on rac1:
PASS => ORA_CRS_HOME environment variable is not set for Grid infrastructure owner
DATA FROM RAC1 - ORA_CRS_HOME ENV VARIABLE FOR GRID INFRASTRUCTURE OWNER ORA_CRS_HOME environment variable not set

Hide

Oracle database software owner hard stack shell limits

Recommendation
Benefit / Impact:

Documented value is the /etc/security/limits.conf file as documented in 11gR2 Grid Infrastructure Installation Guide, section 2.15.3 Setting Resource Limits for the Oracle Software Installation Users.

If the /etc/security/limits.conf file is not configured as described in the documentation then log in to the system as the database software owner (e.g., oracle) and check the hard stack configuration as described below.

Risk:

The hard stack shell limit for the Oracle DB software install owner as defined in /etc/security/limits.conf should be >= 10240. As long as the hard stack limit is 10240 or above then the configuration should be OK.

Action / Repair:

Change DB software install owner hard stack shell limit

$ ulimit -Hs
10240

Needs attention on -

Passed on rac1

Status on rac1:
PASS => Oracle database software owner hard stack shell limit is configured according to recommendation
DATA FROM RAC1 - ORACLE DATABASE SOFTWARE OWNER HARD STACK SHELL LIMITS oracle user's hard stack shell limit = unlimited

Hide

Oracle database software owner soft nofile shell limits

Recommendation
Benefit / Impact:

Documented value, cluster stability

The soft nofile shell limit for the Oracle DB software install owner as defined in /etc/security/limits.conf should be >= 1024.

Risk:

Resource starvation leading to node instability

Action / Repair:

Change DB software install owner soft nofile shell limit

Needs attention on -

Passed on rac1

Status on rac1:
PASS => Oracle database software owner soft nofile shell limits is configured according to recommendation
DATA FROM RAC1 - ORACLE DATABASE SOFTWARE OWNER SOFT NOFILE SHELL LIMITS oracle user'ssoft nofile shell limit = 65536

Hide

Oracle database software owner hard nproc shell limits

Recommendation
Benefit / Impact:

Documented value, cluster stability

The hard nproc shell limit for the Oracle DB software install owner as defined in /etc/security/limits.conf should be >= 16384.

Risk:

Resource starvation (processes) leading to node instability

Action / Repair:

Change DB software install owner hard nproc shell limit

Needs attention on -

Passed on rac1

Status on rac1:
PASS => Oracle database software owner hard nproc shell limits is configured according to recommendation
DATA FROM RAC1 - ORACLE DATABASE SOFTWARE OWNER HARD NPROC SHELL LIMITS oracle user's hard nproc shell limit = 16384

Hide

Check for parameter db_block_checksum

Recommendation
Benefit / Impact:

Experience and testing has shown that certain database initialization parameters should be set at specific values. These are the best practice values set at deployment time. By setting these database initialization parameters as recommended, known problems may be avoided and performance maximized.
The parameters are common to all database instances. The impact of setting these parameters is minimal.
The performance related settings provide guidance to maintain highest stability without sacrificing performance. Changing the default performance settings can be done after careful performance evaluation and clear understanding of the performance impact.

Risk:

If the database initialization parameters are not set as recommended, a variety of issues may be encountered, depending upon which initialization parameter is not set as recommended, and the actual set value.

Action / Repair:

DB_BLOCK_CHECKSUM = FULL aids in block corruption detection. Enable for primary and standby databases

Links
1 Note: 1302539.1 - Best Practices for Corruption Detection, Prevention, and Automatic Repair - in a Data Guard Configuration
2 Protect Against Data Corruption

Needs attention on orcl1

Passed on -

Status on orcl1:
FAIL => Database parameter DB_BLOCK_CHECKSUM is not set to recommended value
orcl1.db_block_checksum = TYPICAL

Hide

ASM disk group compatible.asm attribute

Recommendation
Benefit / Impact:

The components in the I/O stack are tightly integrated in Exadata. You must use the proper versions of software both on the storage servers and the database servers. Setting compatible attributes defines available functionality. Setting CELL.SMART_SCAN_CAPABLE enables the offloading of certain query work to the storage servers. Setting AU_SIZE maximizes available disk technology and throughput by reading 4MB of data before performing a disk seek to a new sector location.

There is minimal impact to verify and configure these settings.

Risk:

If these attributes are not set as directed, performance will be sub-optimal.

Action / Repair:

For the ASM disk group containing Oracle Exadata Storage Server grid disks, verify the attribute settings as follows:

* COMPATIBLE.ASM attribute is set to the Oracle ASM version in use.
* COMPATIBLE.RDBMS attribute is set to the minimum Oracle database software version in use.
* CELL.SMART_SCAN_CAPABLE attribute is TRUE.
* AU_SIZE attribute is 4M.

If compatible.rdbms is not set properly, then the rdbms software version and database must be upgraded or the diskgroup must be recreated.

Needs attention on rac1

Passed on -

Status on rac1:
WARNING => All disk groups should have compatible.asm attribute set to recommended values
DATA FROM RAC1 - ASM DISK GROUP COMPATIBLE.ASM ATTRIBUTE Oracle Clusterware active version on the cluster is [11.2.0.4.0] ASM DATA.compatible.asm = 11.2.0.0.0 ASM OCR.compatible.asm = 11.2.0.0.0

Hide

ASM disk group compatible.advm attribute (non-engineered systems)

Recommendation
Benefit / Impact:

Setting compatible attributes defines available functionality. There is minimal impact to verify and configure these settings. It is considered a best practice to set compatible.advm equal to the active version of the Grid Infrastructure

Risk:

Inability to take advantage of newer features

Action / Repair:

For the ASM disk groups verify the attribute settings as follows:
* COMPATIBLE.ASM attribute is set to the Oracle ASM version in use.
* COMPATIBLE.ADVM attribute is set to the Oracle ASM version in use.
* COMPATIBLE.RDBMS attribute is set to the minimum Oracle database software version in use.

When advancing disk group compatibility, you must advance the COMPATIBLE.ASM attribute before the COMPATIBLE.RDBMS or COMPATIBLE.ADVM attribute to ensure a valid combination of compatible attribute settings as shown in Table 4-2 of the referenced document. You can advance only one compatibility attribute in a single ALTER DISKGROUP statement.

The value of COMPATIBLE.ASM must always be greater than or equal to the value of COMPATIBLE.RDBMS and COMPATIBLE.ADVM.

Needs attention on rac1

Passed on -

Status on rac1:
WARNING => All disk groups should have compatible.advm attribute set to recommended values
DATA FROM RAC1 - ASM DISK GROUP COMPATIBLE.ADVM ATTRIBUTE (NON-ENGINEERED SYSTEMS) Oracle Clusterware active version on the cluster is [11.2.0.4.0] None of the diskgroups have compatible.advm attribute set

Hide

ASM disk group compatible.rdbms attribute

Recommendation
Benefit / Impact:

The components in the I/O stack of the database and ASM are tightly integrated. You must use the proper versions of software on the database servers. Setting compatible attributes defines available functionality. Setting AU_SIZE maximizes available disk technology and throughput by reading 4MB of data before performing a disk seek to a new sector location.

There is minimal impact to verify and configure these settings.

Risk:

If these attributes are not set as directed, performance can be sub-optimal.

Action / Repair:

For the ASM disk group verify the attribute settings as follows:

* COMPATIBLE.ASM attribute is set to the Oracle ASM software version in use.
* COMPATIBLE.RDBMS attribute is set to the minimum Oracle database software version in use.
* AU_SIZE attribute is 4M.

if these attributes are not set properly, correct the condition.

NOTE: If the compatible.rdbms diskgroup parameter is higher than any database instance version than that instance can not access those higher compatibility disk groups . If compatible.rdbms is not set properly, then the rdbms software version and database must be upgraded or the diskgroup must be recreated.

Needs attention on rac1

Passed on -

Status on rac1:
WARNING => All disk groups should have compatible.rdbms attribute set to recommended values
DATA FROM RAC1 - ASM DISK GROUP COMPATIBLE.RDBMS ATTRIBUTE lowest rdbms version = 112040 ASM DATA.compatible.rdbms = 10.1.0.0.0 ASM OCR.compatible.rdbms = 10.1.0.0.0

Hide

Check for parameter asm_power_limit

Recommendation

Syntax to specify power limit while adding or dropping disk is :- alter diskgroup <diskgroup_name> add disk '/dev/raw/raw37' rebalance power 10;

Needs attention on -

Passed on +ASM1

Status on +ASM1:
PASS => asm_power_limit is set to recommended value of 1
+ASM1.asm_power_limit = 1

Hide

Jumbo frames configuration for interconnect

Recommendation

To validate whether jumbo frames are configured correctly end to end (ie., NICs and switches), run the following commands as root. Invoking ping using a specific interface requires root.

export CRS_HOME= To your GI or clusterware home like export CRS_HOME=/u01/app/12.1.0/grid

/bin/ping -s 9000 -c 2 -M do -I `$CRS_HOME/bin/oifcfg getif |grep cluster_interconnect|tail -1|awk '{print $1}'` hostname

Substitute your frame size as required for 9000 in the above command. The actual frame size varies from one networking vendor to another.

If you get errors similar to the following then jumbo frames are not configured properly for your frame size.

From 192.168.122.186 icmp_seq=1 Frag needed and DF set (mtu = 1500)
From 192.168.122.186 icmp_seq=1 Frag needed and DF set (mtu = 1500)

--- hostname@company.com ping statistics ---
0 packets transmitted, 0 received, +2 errors

if jumbo frames are configured properly for your frame size you should obtain output similar to the following:

9000 bytes from hostname (10.208.111.43): icmp_seq=1 ttl=64 time=0.683 ms
9000 bytes from hostname(10.208.111.43): icmp_seq=2 ttl=64 time=0.243 ms

--- hostname ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1000ms
rtt min/avg/max/mdev = 0.243/0.463/0.683/0.220 ms

Links
1 CISCO GIANT/JUMBO sized frame support
2 JUMBO Frame Clean Network Gear
3 INTEL PC1000 Network info
4 Note: 563566.1 - GC Lost Blocks Diagnostics
5 Jumbo Frames for RAC Interconnect
6 Note: 341788.1 - Recommendation for the Real Application Cluster Interconnect and Jumbo Frames
7 Netgear - What is the Jumbo Frame Supported by Switches and Adapters?

Needs attention on rac1

Passed on -

Status on rac1:
INFO => Jumbo frames (MTU >= 9000) are not configured for interconnect
DATA FROM RAC1 - JUMBO FRAMES CONFIGURATION FOR INTERCONNECT CPU Model = eth1 Link encap:Ethernet HWaddr 00:0C:29:DB:C4:4A inet addr:10.25.133.120 Bcast:10.25.133.255 Mask:255.255.255.0 inet6 addr: fe80::20c:29ff:fedb:c44a/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:82581 errors:0 dropped:0 overruns:0 frame:0 TX packets:69252 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:52633805 (50.1 MiB) TX bytes:35467010 (33.8 MiB)

Hide

OSWatcher status

Recommendation
Benefit / Impact:

OSWatcher is used to collect and archive operating system and network metrics to aid support in diagnosing performance issues. Checking OSWatcher status helps to ensure critical data is available for analysis when system occurring issues.

There is no impact from verifying the OSWatcher status.

Risk:

When OSWatcher is not running, data is not gathered on a regular basis. System engineers may not be able to monitor system health, to analyze and resolve issues, which may result in much longer maintenance windows.

Action / Repair:
To check the current status of OSWatcher, run the command:

# ps -elf | grep OSWatcher | grep -v grep

To start, stop or reset the OSWatcher utility, please reference the latest OSWatcher User's Guide (Doc ID 1531223.1) :

Links
1 Note: 1531223.1 - OS Watcher User's Guide
2 Note: 301137.1 - OS Watcher User Guide
3 Note: 2044825.1 - How-To Update the OSC-Exawatcher Package on All Database Domains

Needs attention on rac1

Passed on -

Status on rac1:
WARNING => OSWatcher is not running as is recommended.
DATA FROM RAC1 - OSWATCHER STATUS

Hide

CSS reboot time

Recommendation

Links
1 Note: 294430.1 - CSS Timeout Computation in Oracle Clusterware
2 12.1 CRSCTL Utility Reference
3 CRSCTL Utility Reference

Needs attention on -

Passed on rac1

Status on rac1:
PASS => CSS reboot time is set to the default value of 3
DATA FROM RAC1 - CSS REBOOT TIME CRS-4678: Successful get reboottime 3 for Cluster Synchronization Services.

Hide

CSS disktimeout

Recommendation

Links
1 Note: 294430.1 - CSS Timeout Computation in Oracle Clusterware

Needs attention on -

Passed on rac1

Status on rac1:
PASS => CSS disktimeout is set to the default value of 200
DATA FROM RAC1 - CSS DISKTIMEOUT CRS-4678: Successful get disktimeout 200 for Cluster Synchronization Services.

Hide

ohasd Log File Ownership

Recommendation

$ls -l $GRID_HOME/log/`hostname`/crsd/*
$ls -l $GRID_HOME/log/`hostname`/ohasd/*
$ls -l $GRID_HOME/log/`hostname`/agent/crsd/orarootagent_root/*
$ls -l $GRID_HOME/log/`hostname`/agent/ohasd/orarootagent_root/*

If any of those files' ownership is NOT root:root then you should change the ownership of the files individually or as follows (as root):

# chown root:root $GRID_HOME/log/`hostname`/crsd/*
# chown root:root $GRID_HOME/log/`hostname`/ohasd/*
# chown root:root $GRID_HOME/log/`hostname`/agent/crsd/orarootagent_root/*
# chown root:root $GRID_HOME/log/`hostname`/agent/ohasd/orarootagent_root/*

Links
1 Oracle Bug # 9837321 - OWNERSHIP OF CRSD TRACES GOT CHANGE FROM ROOT TO ORACLE BY PATCHING SCRIPT
2 Note: 9837321.8 - Bug 9837321 - Ownership of crsd traces gets changed from root by patching script

Needs attention on -

Passed on rac1

Status on rac1:
PASS => ohasd Log Ownership is Correct (root root)
DATA FROM RAC1 - OHASD LOG FILE OWNERSHIP total 6056 -rw-r--r--. 1 root root 6191426 Jan 25 16:41 ohasd.log -rw-r--r--. 1 root root 2379 Jan 25 15:32 ohasdOUT.log

Hide

ohasd/orarootagent_root Log File Ownership

Recommendation

$ls -l $GRID_HOME/log/`hostname`/crsd/*
$ls -l $GRID_HOME/log/`hostname`/ohasd/*
$ls -l $GRID_HOME/log/`hostname`/agent/crsd/orarootagent_root/*
$ls -l $GRID_HOME/log/`hostname`/agent/ohasd/orarootagent_root/*

If any of those files' ownership is NOT root:root then you should change the ownership of the files individually or as follows (as root):

# chown root:root $GRID_HOME/log/`hostname`/crsd/*
# chown root:root $GRID_HOME/log/`hostname`/ohasd/*
# chown root:root $GRID_HOME/log/`hostname`/agent/crsd/orarootagent_root/*
# chown root:root $GRID_HOME/log/`hostname`/agent/ohasd/orarootagent_root/*

Links
1 Oracle Bug # 9837321 - OWNERSHIP OF CRSD TRACES GOT CHANGE FROM ROOT TO ORACLE BY PATCHING SCRIPT

Needs attention on -

Passed on rac1

Status on rac1:
PASS => ohasd/orarootagent_root Log Ownership is Correct (root root)
DATA FROM RAC1 - OHASD/ORAROOTAGENT_ROOT LOG FILE OWNERSHIP total 3856 -rw-r--r--. 1 root root 3938253 Jan 25 16:41 orarootagent_root.log -rw-r--r--. 1 root root 5 Jan 25 15:33 orarootagent_root.pid -rw-r--r--. 1 root root 0 Dec 1 2015 orarootagent_rootOUT.log

Hide

crsd/orarootagent_root Log File Ownership

Recommendation

$ls -l $GRID_HOME/log/`hostname`/crsd/*
$ls -l $GRID_HOME/log/`hostname`/ohasd/*
$ls -l $GRID_HOME/log/`hostname`/agent/crsd/orarootagent_root/*
$ls -l $GRID_HOME/log/`hostname`/agent/ohasd/orarootagent_root/*

If any of those files' ownership is NOT root:root then you should change the ownership of the files individually or as follows (as root):

# chown root:root $GRID_HOME/log/`hostname`/crsd/*
# chown root:root $GRID_HOME/log/`hostname`/ohasd/*
# chown root:root $GRID_HOME/log/`hostname`/agent/crsd/orarootagent_root/*
# chown root:root $GRID_HOME/log/`hostname`/agent/ohasd/orarootagent_root/*

Links
1 Oracle Bug # 9837321 - CSS Timeout Computation in Oracle Clusterware - OWNERSHIP OF CRSD TRACES GOT CHANGE FROM ROOT TO ORACLE BY PATCHING SCRIPT
2 Note: 9837321.8 - Bug 9837321 - Ownership of crsd traces gets changed from root by patching script

Needs attention on -

Passed on rac1

Status on rac1:
PASS => crsd/orarootagent_root Log Ownership is Correct (root root)
DATA FROM RAC1 - CRSD/ORAROOTAGENT_ROOT LOG FILE OWNERSHIP total 5728 -rw-r--r--. 1 root root 5855160 Jan 25 16:41 orarootagent_root.log -rw-r--r--. 1 root root 5 Jan 25 15:35 orarootagent_root.pid -rw-r--r--. 1 root root 0 Jan 22 09:40 orarootagent_rootOUT.log

Hide

crsd Log File Ownership

Recommendation

Links
1 Oracle Bug # 9837321 - Bug 9837321 - Ownership of crsd traces gets changed from root by patching script - OWNERSHIP OF CRSD TRACES GOT CHANGE FROM ROOT TO ORACLE BY PATCHING SCRIPT

Needs attention on -

Passed on rac1

Status on rac1:
PASS => crsd Log Ownership is Correct (root root)
DATA FROM RAC1 - CRSD LOG FILE OWNERSHIP total 12272 -rw-r--r--. 1 root root 10514819 Jan 25 15:38 crsd.l01 -rw-r--r--. 1 root root 2035830 Jan 25 16:41 crsd.log -rw-r--r--. 1 root root 1740 Jan 25 15:35 crsdOUT.log

Hide

VIP NIC bonding config.

Recommendation

Links
1 Note: 298891.1 - Configuring Linux for the Oracle 10g VIP or private interconnect using bonding driver

Needs attention on rac1

Passed on -

Status on rac1:
WARNING => NIC bonding is NOT configured for public network (VIP)
DATA FROM RAC1 - VIP NIC BONDING CONFIG. eth0 Link encap:Ethernet HWaddr 00:0C:29:DB:C4:40 inet addr:192.168.133.120 Bcast:192.168.133.255 Mask:255.255.255.0 inet6 addr: fe80::20c:29ff:fedb:c440/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:454713 errors:0 dropped:0 overruns:0 frame:0 TX packets:371690 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:560035305 (534.0 MiB) TX bytes:249682774 (238.1 MiB)

Hide

Interconnect NIC bonding config.

Recommendation

NOTE: If customer is on 11.2.0.2 or above and HAIP is in use with two or more interfaces then this finding can be ignored.

Links
1 Note: 298891.1 - Configuring Linux for the Oracle 10g VIP or private interconnect using bonding driver

Needs attention on rac1

Passed on -

Status on rac1:
WARNING => NIC bonding is not configured for interconnect
DATA FROM RAC1 - INTERCONNECT NIC BONDING CONFIG. eth1 10.25.133.0 global cluster_interconnect

Hide

Verify operating system hugepages count satisfies total SGA requirements

Recommendation
Benefit / Impact:

Properly configuring operating system hugepages on Linux and using the database initialization parameter "use_large_pages" to "only" results in more efficient use of memory and reduced paging.
The impact of validating that the total current hugepages are greater than or equal to estimated requirements for all currently active SGAs is minimal. The impact of corrective actions will vary depending on the specific configuration, and may require a reboot of the database server.

Risk:

The risk of not correctly configuring operating system hugepages in advance of setting the database initialization parameter "use_large_pages" to "only" is that if not enough huge pages are configured, some databases will not start after you have set the parameter.

Action / Repair:

Pre-requisite: All database instances that are supposed to run concurrently on a database server must be up and running for this check to be accurate.

NOTE: Please refer to below referenced My Oracle Support notes for additional details on configuring hugepages.

NOTE: If you have not reviewed the below referenced My Oracle Support notes and followed their guidance BEFORE using the database parameter "use_large_pages=only", this check will pass the environment but you will still not be able to start instances once the configured pool of operating system hugepages have been consumed by instance startups. If that should happen, you will need to change the "use_large_pages" initialization parameter to one of the other values, restart the instance, and follow the instructions in the below referenced My Oracle Support notes. The brute force alternative is to increase the huge page count until the newest instance will start, and then adjust the huge page count after you can see the estimated requirements for all currently active SGAs.

Links
1 Note: 401749.1 - Shell Script to Calculate Values Recommended Linux HugePages / HugeTLB Configuration
2 Note: 361323.1 - HugePages on Linux: What It Is... and What It Is Not...

Needs attention on rac1

Passed on -

Status on rac1:
FAIL => Operating system hugepages count does not satisfy total SGA requirements
DATA FROM RAC1 - VERIFY OPERATING SYSTEM HUGEPAGES COUNT SATISFIES TOTAL SGA REQUIREMENTS

Hide

Check for parameter memory_target

Recommendation
MEMORY_TARGET
MEMORY_MAX_TARGET

Needs attention on orcl1

Passed on -

Status on orcl1:
WARNING => Database Parameter memory_target is not set to the recommended value
orcl1.memory_target = 780140544

Hide

CRS and ASM version comparison

Recommendation
Benefit / Impact:

According to MOS Note 337737.1, CRS version must be higher or equal to ASM version to guarantee version compatibility.

Risk:

If CRS version is older than ASM version, ASM will not be supported

Action / Repair:

The script below compares the two version values.
echo $ver_asm $ver_crs | awk '{
split($1, a, ".");
split($2, b, ".");
for (i = 1; i <= 4; i++)
if (a[i] < b[i]) {
print "pass";
break;
} else if (a[i] > b[i]) {
print "fail";
break;
} else if (i == 4)
print "pass";
}'
.
The output should indicate pass if CRS version is higher than or equal to ASM version or fail vice versa.

Links
1 Note: 337737.1 - Oracle Clusterware (CRS/GI) - ASM - Database Version Compatibility

Needs attention on -

Passed on rac1

Status on rac1:
PASS => CRS version is higher or equal to ASM version.
DATA FROM RAC1 - CRS AND ASM VERSION COMPARISON CRS_ACTIVE_VERSION = 112040 ASM Version = 112040

Hide

Local listener set to node VIP

Recommendation

Links
1 SINGLE CLIENT ACCESS NAME (SCAN)

Needs attention on -

Passed on rac1

Status on rac1:
PASS => Local listener init parameter is set to local node VIP
DATA FROM rac1 - orcl DATABASE - Local listener set to node VIP Local Listener= (ADDRESS=(PROTOCOL=TCP)(HOST=192.168.133.130)(PORT=1521)) VIP Names=rac1-vip VIP IPs=192.168.133.130

Hide

Number of SCAN listeners

Recommendation
Benefit / Impact:

Application scalability and/or availability

Risk:

Potential reduced scalability and/or availability of applications

Action / Repair:

The recommended number of SCAN listeners is 3.... See the referenced document for more details.

Links
1 SINGLE CLIENT ACCESS NAME (SCAN)

Needs attention on rac1

Passed on -

Status on rac1:
INFO => Number of SCAN listeners is not equal to the recommended number of 3.
DATA FROM RAC1 - NUMBER OF SCAN LISTENERS SCAN Listener LISTENER_SCAN1 exists. Port: TCP:1521

Hide

Verify voting disks status

Recommendation
Benefit / Impact:

Stability, Availability

Risk:

Cluster instability

Action / Repair:

Voting disks that are not online would indicate a problem with the clusterware
and should be investigated as soon as possible. All voting disks are expected to be ONLINE.

Use the following command to list the status of the voting disks

$CRS_HOME/bin/crsctl query css votedisk|sed 's/^ //g'|grep ^[0-9]

The output should look similar to the following, one row were voting disk, all disks should indicate ONLINE

1. ONLINE 192c8f030e5a4fb3bf77e43ad3b8479a (o/192.168.10.102/DBFS_DG_CD_02_sclcgcel01) [DBFS_DG]
2. ONLINE 2612d8a72d194fa4bf3ddff928351c41 (o/192.168.10.104/DBFS_DG_CD_02_sclcgcel03) [DBFS_DG]
3. ONLINE 1d3cceb9daeb4f0bbf23ee0218209f4c (o/192.168.10.103/DBFS_DG_CD_02_sclcgcel02) [DBFS_DG]

Needs attention on -

Passed on rac1

Status on rac1:
PASS => All voting disks are online
DATA FROM RAC1 - VERIFY VOTING DISKS STATUS ## STATE File Universal Id File Name Disk group -- ----- ----------------- --------- --------- 1. ONLINE 06a3e4da34154f57bff7384023ec9f7d (/dev/raw/raw1) [OCR] Located 1 voting disk(s).

Hide

css misscount

Recommendation

Links
1 Note: 294430.1 - CSS Timeout Computation in Oracle Clusterware

Needs attention on -

Passed on rac1

Status on rac1:
PASS => CSS misscount is set to the default value of 30
DATA FROM RAC1 - CSS MISSCOUNT CRS-4678: Successful get misscount 30 for Cluster Synchronization Services.

Hide

Same size of redo log files

Recommendation
column member format a50
select f.member,l.bytes/1024/1024 as "Size in MB" from v$log l,v$logfile f where l.group#=f.group#;
Resizing redo logs to make it same size does not need database downtime.

Links
1 Note: 602066.1 - How To Resize and/or Add Redo Logs

Needs attention on -

Passed on orcl

Status on orcl:
PASS => All redo log files are of same size
DATA FOR ORCL FOR SAME SIZE OF REDO LOG FILES 1 .048828125 2 .048828125 3 .048828125 4 .048828125

Hide

SELinux status

Recommendation
installation gives the error:
'scriptlet failed, exit status 255'

The default selinux settings are used
# cat /etc/sysconfig/selinux
SELINUX=enforcing
SELINUXTYPE=targeted
e.g. on installing asm rpms:
# rpm -ivh *.rpm
Preparing... ###########################################
[100%]
1:oracleasm-support ########################################### [33%]
error: %post(oracleasm-support-2.0.2-1.x86_64) scriptlet failed, exit status 255
2:oracleasm-2.6.9-34.ELsm########################################### [67%]
error: %post(oracleasm-2.6.9-34.ELsmp-2.0.2-1.x86_64) scriptlet failed, exit status 255
3:oracleasmlib ########################################### [100%]

However, asm rpms gets installed
# rpm -qa | grep asm
oracleasm-support-2.0.2-1
oracleasmlib-2.0.2-1
oracleasm-2.6.9-34.ELsmp-2.0.2-1

There is no error during oracleasm configure, creadisks, Also, oracleasm is able to start on reboot and the tests done around rac/asm seems to be fine.

# rpm -q -a | grep -i selinux
selinux-policy-targeted-1.17.30-2.126
selinux-policy-targeted-sources-1.17.30-2.126
libselinux-1.19.1-7
libselinux-1.19.1-7

Solution
--
If the machine is installed with selinux --disabled, it is possible that the selinux related pre/post activities have not been performed during the installation and as a result extended attribute is not getting set for /bin/*sh

1. ensure that the kickstart config file does not have 'selinux --disabled'
Also, not specifying selinux in the config file will default to selinux --enforcing and the extended attribute will get set for /bin/*sh
OR
2. If the machine has been installed with selinux --disabled then perform the below step manually # setfattr -n security.selinux --value="system_u:object_r:shell_exec_t\000"
bin/sh

Links
1 Note: 457458.1 - How to Disable or set SELinux to Permissive mode
2 Note: 790470.1 - ASMCMD fails to start on module libnnz10.so, permission denied
3 Note: 454196.1 - ./sqlplus: error on libnnz11.so: cannot restore segment prot after reloc

Needs attention on rac1

Passed on -

Status on rac1:
WARNING => SELinux is being Enforced.
DATA FROM RAC1 - SELINUX STATUS Enforcing

Hide

Public interface existence

Recommendation

Needs attention on -

Passed on rac1

Status on rac1:
PASS => Public interface is configured and exists in OCR
DATA FROM RAC1 - PUBLIC INTERFACE EXISTENCE eth0 192.168.133.0 global public eth1 10.25.133.0 global cluster_interconnect

Hide

ip_local_port_range

Recommendation

Links
1 Configuring Kernel Parameters

Needs attention on -

Passed on rac1

Status on rac1:
PASS => ip_local_port_range is configured according to recommendation
DATA FROM RAC1 - IP_LOCAL_PORT_RANGE minimum port range = 9000 maximum port range = 65500

Hide

kernel.shmmax

Recommendation
Benefit / Impact:

Optimal system memory management.

Risk:

In an Oracle RDBMS application, setting kernel.shmmax too high is not needed and could enable configurations that may leave inadequate system memory for other necessary functions.

Action / Repair:

Oracle Support officially recommends a "minimum" for SHMMAX of 1/2 of physical RAM. However, many Oracle customers choose a higher fraction, at their discretion. Setting the kernel.shmmax as recommended only causes a few more shared memory segments to be used for whatever total SGA that you subsequently configure in Oracle.

Links
1 Configuring Kernel Parameters
2 Note: 567506.1 - Maximum SHMMAX values for Linux x86 and x86-64

Needs attention on -

Passed on rac1

Status on rac1:
PASS => kernel.shmmax parameter is configured according to recommendation
DATA FROM RAC1 - KERNEL.SHMMAX NOTE: All results reported in bytes kernel.shmmax actual = 68719476736 total system memory = 4002799616 1/2 total system memory = 2001399808

Hide

Check for parameter fs.file-max

Recommendation

- At a _MINIMUM_ we require 512*PROCESSES (init parameter) file descriptors per database instance + some for the OS and other non-oracle processes

- Since we cannot know at install time how many database instances the customer may run and how many PROCESSES they may configure for those instances and whether they will use automatic memory management or how many non-Oracle processes may be run and how many file descriptors they will require we recommend the file descriptor limit be set to a very high number (6815744) to minimize the potential for running out.

- Setting fs.file-max "too high" doesn't hurt anything because file descriptors are allocated dynamically as needed up to the limit of fs.file-max

- Oracle is not aware of any customers having problems from setting fs.file-max "too high" but we have had customers have problems from setting it too low. A problem from having too few file descriptors is preventable.

- As for a formula, given 512*PROCESSES (as a minimum) fs.file-max should be a sufficiently high number to minimize the chance that ANY customer would suffer an outage from having fs.file-max set too low. At a limit of 6815744 customers are likely to have other problems to worry about before they hit that limit.

- If an individual customer wants to deviate from fs.file-max = 6815744 then they are free to do so based on their knowledge of their environment and implementation as long as they make sure they have enough file descriptors to cover all their database instances, other non-oracle processes and the OS.

Links
1 11.2 Kernel Settings
2 12.1 kernel settings
3 Note: 1587357.1 - Oracle Database (RDBMS) on Unix AIX,HP-UX,Linux,Solaris and MS Windows Operating Systems Installation and Configuration
4 Configuring Kernel Parameters
5 Note: 169706.1 - Oracle Database OS Installation and Configuration Requirements Quick Reference

Needs attention on -

Passed on rac1

Status on rac1:
PASS => Kernel Parameter fs.file-max is configuration meets or exceeds recommendation

Hide

/tmp directory free space

Recommendation

Links
1 Disk Space Requirements

Needs attention on -

Passed on rac1

Status on rac1:
PASS => Free space in /tmp directory meets or exceeds recommendation of minimum 1GB
DATA FROM RAC1 - /TMP DIRECTORY FREE SPACE Filesystem Size Used Avail Use% Mounted on /dev/sda3 16G 13G 2.3G 86% /

Hide

Oracle database software owner soft nproc shell limit

Recommendation
Benefit / Impact:

The soft nproc shell limit for the Oracle DB software install owner as defined in /etc/security/limits.conf should be >= 2047.

Risk:

The soft nproc shell limit for the Oracle DB software install owner as defined in /etc/security/limits.conf should be >= 2047. As long as the soft nproc limit is 2047 or above then the configuration should be OK.

Action / Repair:

Change DB software install owner soft nproc shell limit

$ ulimit -Su
2047

Links
1 Note: 788480.1 - Resource Limits Using /etc/security/limits.conf and /etc/profile
2 Note: 169706.1 - Oracle Database OS Installation and Configuration Requirements Quick Reference

Needs attention on -

Passed on rac1

Status on rac1:
PASS => Oracle database software owner soft nproc shell limit is configured according to recommendation
DATA FROM RAC1 - ORACLE DATABASE SOFTWARE OWNER SOFT NPROC SHELL LIMIT oracle user's soft nproc shell limit = 16384

Hide

Oracle database software owner hard nofile shell limits

Recommendation
Benefit / Impact:

The hard nofile shell limit for the Oracle database software install owner as defined in /etc/security/limits.conf should be >= 65536.

Risk:

Resource starvation (file descriptors) leading to node instability

Links
1 Setting Resource Limits for the Oracle Software Installation Users

Needs attention on -

Passed on rac1

Status on rac1:
PASS => Oracle database software owner hard nofile shell limit is configured according to recommendation
DATA FROM RAC1 - ORACLE DATABASE SOFTWARE OWNER HARD NOFILE SHELL LIMITS oracle user's hard nofile shell limit = 65536

Hide

Linux Swap Size

Recommendation

Note:
On Linux, the Hugepages feature allocates non-swappable memory for large page tables using memory-mapped files. If you enable Hugepages, then you should deduct the memory allocated to Hugepages from the available RAM before calculating swap space.

RAM between 1 GB and 2 GB, Swap 1.5 times the size of RAM (minus memory allocated to Hugepages)

RAM between 2 GB and 16 GB, Swap equal to the size of RAM (minus memory allocated to Hugepages)

RAM (minus memory allocated to Hugepages)
more than 16 GB, Swap 16 GB

In other words the maximum swap size for Linux that Oracle would recommend would be 16GB

Links
1 Note: 1587357.1 - Oracle Database (RDBMS) on Unix AIX,HP-UX,Linux,Solaris and MS Windows Operating Systems Installation and Configuration
2 Linux Memory Requirements
3 Note: 169706.1 - Oracle Database OS Installation and Configuration Requirements Quick Reference

Needs attention on -

Passed on rac1

Status on rac1:
PASS => Linux swap configuration meets or exceeds recommendation
DATA FROM RAC1 - LINUX SWAP SIZE Total meory on system(Physical RAM - Huge Pagess Size) = Swap memory found on system = 4095992 Recommended Swap = 0

Hide

/tmp on dedicated filesystem

Recommendation
Benefit / Impact:

It is a best practice to locate the /tmp directory on a dedicated filesystem. For example, Oracle creates socket files in /var/tmp.

Risk:

Accidentally filling up /tmp could lead to filling up the root (/) filesystem as the result of other file management (logs, traces, etc.) and lead to availability problems

Action / Repair:

Make sure 1GB of free space is maintained in /tmp.

Needs attention on -

Passed on rac1

Status on rac1:
PASS => /tmp is on a dedicated filesystem
DATA FROM RAC1 - /TMP ON DEDICATED FILESYSTEM Filesystem 1K-blocks Used Available Use% Mounted on /dev/sda3 16307112 13169248 2309500 86% /

Hide

ASMLIB check

Recommendation

Links
1 Note: 359266.1 - FAQ ASMLIB CONFIGURE,VERIFY, TROUBLESHOOT

Needs attention on rac1

Passed on -

Status on rac1:
INFO => oracleasm (asmlib) module is not loaded
DATA FROM RAC1 - ASMLIB CHECK Look for oracleasm module Module Size Used by iptable_filter 2793 0 ip_tables 17831 1 iptable_filter bridge 79078 0 oracleacfs 1990406 0 oracleadvm 250040 0 oracleoks 427672 2 oracleacfs,oracleadvm 8021q 25317 0 garp 7152 1 8021q stp 2218 2 bridge,garp llc 5546 3 bridge,garp,stp ip6t_REJECT 4628 2 nf_conntrack_ipv6 8748 2 nf_defrag_ipv6 11182 1 nf_conntrack_ipv6
xt_state 1492 2 nf_conntrack 79357 2 nf_conntrack_ipv6,xt_state ip6table_filter 2889 1 ip6_tables 19458 1 ip6table_filter ipv6 321422 390 ip6t_REJECT,nf_conntrack_ipv6,nf_defrag_ipv6 uinput 7992 0 e1000 170646 0 snd_ens1371 21587 2 snd_rawmidi 22825 1 snd_ens1371 snd_ac97_codec 124967 1 snd_ens1371 ac97_bus 1452 1 snd_ac97_codec snd_seq 55759 0 snd_seq_device 6500 2 snd_rawmidi,snd_seq snd_pcm 86635 2 snd_ens1371,snd_ac97_codec snd_timer 22411 2 snd_seq,snd_pcm snd 69907 11 snd_ens1371,snd_rawmidi,snd_ac97_codec,snd_seq,snd_seq_device,snd_pcm,snd_timer soundcore 7958 1 snd snd_page_alloc 8470 1 snd_pcm sg 29350 0 microcode 112653 0 vmware_balloon 7199 0 i2c_piix4 12608 0 i2c_core 31084 1 i2c_piix4 shpchp 32778 0 ext4 363408 2 mbcache 8193 1 ext4 jbd2 90230 1 ext4 sr_mod 15177 0 cdrom 39085 1 sr_mod ahci 41127 0 sd_mod 38976 7 crc_t10dif 1541 1 sd_mod mptspi 16603 44 mptscsih 36700 1 mptspi mptbase 93615 2 mptspi,mptscsih scsi_transport_spi 25863 1 mptspi pata_acpi 3701 0 ata_generic 3837 0 ata_piix 24121 0 dm_mirror 14133 0 dm_region_hash 12085 1 dm_mirror dm_log 9930 2 dm_mirror,dm_region_hash dm_mod 82839 2 dm_mirror,dm_log
Click for more data

Hide

Non-autoextensible data and temp files

Recommendation
Benefit / Impact:

The benefit of having "AUTOEXTEND" on is that applications may avoid out of space errors.
The impact of verifying that the "AUTOEXTEND" attribute is "ON" is minimal. The impact of setting "AUTOEXTEND" to "ON" varies depending upon if it is done during database creation, file addition to a tablespace, or added to an existing file.

Risk:

The risk of running out of space in either the tablespace or diskgroup varies by application and cannot be quantified here. A tablespace that runs out of space will interfere with an application, and a diskgroup running out of space could impact the entire database as well as ASM operations (e.g., rebalance operations).

Action / Repair:

To obtain a list of tablespaces that are not set to "AUTOEXTEND", enter the following sqlplus command logged into the database as sysdba:

select file_id, file_name, tablespace_name from dba_data_files where autoextensible <>'YES'
union
select file_id, file_name, tablespace_name from dba_temp_files where autoextensible <> 'YES';
The output should be:
no rows selected
If any rows are returned, investigate and correct the condition.

NOTE: Configuring "AUTOEXTEND" to "ON" requires comparing space utilization growth projections at the tablespace level to space available in the diskgroups to permit the expected projected growth while retaining sufficient storage space in reserve to account for ASM rebalance operations that occur either as a result of planned operations or component failure. The resulting growth targets are implemented with the "MAXSIZE" attribute that should always be used in conjunction with the "AUTOEXTEND" attribute. The "MAXSIZE" settings should allow for projected growth while minimizing the prospect of depleting a disk group. The "MAXSIZE" settings will vary by customer and a blanket recommendation cannot be given here.

NOTE: When configuring a file for "AUTOEXTEND" to "ON", the size specified for the "NEXT" attribute should cover all disks in the diskgroup to optimize balance. For example, with a 4MB AU size and 168 disks, the size of the "NEXT" attribute should be a multiple of 672M (4*168).

Needs attention on -

Passed on orcl

Status on orcl:
PASS => All data and temporary files are autoextensible
DATA FOR ORCL FOR NON-AUTOEXTENSIBLE DATA AND TEMP FILES Query returned no rows which is expected when the SQL check passes.

Hide

Non-multiplexed redo logs

Recommendation

Needs attention on orcl

Passed on -

Status on orcl:
WARNING => One or more redo log groups are not multiplexed
DATA FOR ORCL FOR NON-MULTIPLEXED REDO LOGS 1 1 2 1 4 1 3 1

Hide

Multiplexed controlfiles

Recommendation

Needs attention on orcl

Passed on -

Status on orcl:
WARNING => Controlfile is NOT multiplexed
DATA FOR ORCL FOR MULTIPLEXED CONTROLFILES +DATA/orcl/controlfile/current.260.897348927

Hide

Check audit_file_dest

Recommendation

Needs attention on rac1

Passed on -

Status on rac1:
INFO => audit_file_dest has audit files older than 30 days
DATA FROM rac1 - orcl DATABASE - Check audit_file_dest Number of audit files last modified over 30 days ago at /oracle/app/oracle/admin/orcl/adump = 39

Hide

oradism executable ownership

Recommendation
Benefit / Impact:

The oradism executable is invoked after database startup to change the scheduling priority of LMS and other database background processes to the realtime scheduling class in order to maximize the ability of these key processes to be scheduled on the CPU in a timely way at times of high CPU utilization.

Risk:

If oradism ownership is not root and the owner s-bit is not set then something must have gone wrong in the installation process or the ownership or the permission was otherwise changed.

Action / Repair:

The oradism executable should be owned by root and the owner s-bit should be set, e.g., -rwsr-x---, where the s is the setuid bit (s-bit) for root in this case. If the LMS process is not running at the proper scheduling priority it can lead to instance evictions due to IPC send timeouts or ORA-29740 errors. oradism must be owned by root and it's s-bit set in order to be able to change the scheduling priority.

Please check with Oracle Support to determine the best course to take for your platform to correct the problem.

Needs attention on -

Passed on rac1

Status on rac1:
PASS => $ORACLE_HOME/bin/oradism ownership is root
DATA FROM rac1 - /oracle/app/oracle/product/11.2.0/db DATABASE_HOME - oradism executable ownership -rwsr-x---. 1 root oinstall 71790 Aug 24 2013 /oracle/app/oracle/product/11.2.0/db/bin/oradism

Hide

oradism executable permission

Recommendation
Benefit / Impact:

The oradism executable is invoked after database startup to change the scheduling priority of LMS and other database background processes to the realtime scheduling class in order to maximize the ability of these key processes to be scheduled on the CPU in a timely way at times of high CPU utilization.

Risk:

If the LMS process is not running at the proper scheduling priority it can lead to instance evictions due to IPC send timeouts or ORA-29740 errors.

Action / Repair:

The oradism executable should be owned by root and the owner s-bit should be set, e.g., -rwsr-x---, where the s is the setuid bit (s-bit) for root in this case. oradism must be owned by root and it's s-bit set in order to be able to change the scheduling priority. If oradism ownership is not root and the owner s-bit is not set then something must have gone wrong in the installation process or the ownership or the permission was otherwise changed.

Please check with Oracle Support to determine the best course to take for your platform to correct the problem.

Needs attention on rac1

Passed on -

Status on rac1:
WARNING => $ORACLE_HOME/bin/oradism setuid bit is NOT set
DATA FROM rac1 - /oracle/app/oracle/product/11.2.0/db DATABASE_HOME - oradism executable permission -rwsr-x---. 1 root oinstall 71790 Aug 24 2013 /oracle/app/oracle/product/11.2.0/db/bin/oradism

Hide

Avg message sent queue time on ksxp

Recommendation

Interconnect performance should be investigated further by analysis using AWR and ASH reports and other network diagnostic tools.

Needs attention on -

Passed on orcl

Status on orcl:
PASS => Avg message sent queue time on ksxp is <= recommended
DATA FOR ORCL FOR AVG MESSAGE SENT QUEUE TIME ON KSXP Query returned no rows which is expected when the SQL check passes.

Hide

Avg message sent queue time (ms)

Recommendation

Interconnect performance should be investigated further by analysis using AWR and ASH reports and other network diagnostic tools.

Needs attention on -

Passed on orcl

Status on orcl:
PASS => Avg message sent queue time is <= recommended
DATA FOR ORCL FOR AVG MESSAGE SENT QUEUE TIME (MS) Query returned no rows which is expected when the SQL check passes.

Hide

Avg message received queue time

Recommendation

Interconnect performance should be investigated further by analysis using AWR and ASH reports and other network diagnostic tools.

Needs attention on -

Passed on orcl

Status on orcl:
PASS => Avg message received queue time is <= recommended
DATA FOR ORCL FOR AVG MESSAGE RECEIVED QUEUE TIME Query returned no rows which is expected when the SQL check passes.

Hide

GC block lost

Recommendation
Benefit / Impact:

If the RDBMS reports global cache lost blocks statistics ("gc cr block lost" and/or "gc current block lost") it could indicate a negative impact on interconnect performance and global cache processing.

Risk:

The vast majority of escalations attributed to RDBMS global cache lost blocks can be directly related to faulty or misconfigured interconnects.

Action / Repair:

For more details please see the referenced MOS notes.

Links
1 Note: 417983.1 - Enterprise Linux: Common Network Configuration / Services Issues
2 Note: 400959.1 - POOR RAC-INTERCONNECT PERFORMANCE AFTER UPGRADE FROM RHEL3 TO RHEL4/OEL4
3 Note: 563566.1 - gc lost blocks diagnostics

Needs attention on -

Passed on orcl

Status on orcl:
PASS => No Global Cache lost blocks detected
DATA FOR ORCL FOR GC BLOCK LOST Query returned no rows which is expected when the SQL check passes.

Hide

Session Failover configuration

Recommendation
Benefit / Impact:

Higher application availability

Risk:

Application availability problems in case of failed nodes or database instances

Action / Repair:

Application connection failover and load balancing is highly recommended for OLTP environments but may not apply for DSS workloads. DSS application customers may want to ignore this warning.

The following query will identify the application user sessions that do not have basic connection failover configured:

select username, sid, serial#,process,failover_type,failover_method FROM gv$session where upper(failover_method) != 'BASIC' and upper(failover_type) !='SELECT' and upper(username) not in ('SYS','SYSTEM','SYSMAN','DBSNMP');

Links
1 Note: 453293.1 - 10g & 11g :Configuration of TAF(Transparent Application Failover) and Load Balancing

Needs attention on -

Passed on orcl

Status on orcl:
PASS => Failover method (SELECT) and failover mode (BASIC) are configured properly
DATA FOR ORCL FOR SESSION FAILOVER CONFIGURATION Query returned no rows which is expected when the SQL check passes.

Hide

Redo log Checkpoint not complete

Recommendation

Links
1 Note: 265831.1 - Automatic Checkpoint Tuning in Oracle 10g
2 Note: 147468.1 - Checkpoint Tuning and Troubleshooting Guide

Needs attention on rac1

Passed on -

Status on rac1:
INFO => At some times checkpoints are not being completed
DATA FROM rac1 - orcl DATABASE - Redo log Checkpoint not complete Checkpoint not complete messages in /oracle/app/oracle/diag/rdbms/orcl/orcl1/trace/alert_orcl1.log found 11 times

Hide

Avg GC Current Block Receive Time

Recommendation
Benefit / Impact:

The average gc current block receive time should typically be less than 15 milliseconds depending on your system configuration and volume. This is the average latency of a current request round-trip from the requesting instance to the holding instance and back to the requesting instance.

Risk:

Potential global cache performance issues.

Action / Repair:

Use the following query to determine the average gc current block receive time for each instance.

set numwidth 20
column "AVG CURRENT BLOCK RECEIVE TIME (ms)" format 9999999.9
select b1.inst_id, ((b1.value / decode(b2.value,0,1)) * 10) "AVG CURRENT BLOCK RECEIVE TIME (ms)"
from gv$sysstat b1, gv$sysstat b2
where b1.name = 'gc current block receive time' and
b2.name = 'gc current blocks received' and b1.inst_id = b2.inst_id ;

Needs attention on -

Passed on orcl

Status on orcl:
PASS => Avg GC CURRENT Block Receive Time Within Acceptable Range
DATA FOR ORCL FOR AVG GC CURRENT BLOCK RECEIVE TIME avg_gc_current_block_receive_time_15ms_exceeded = 10

Hide

Avg GC CR Block Receive Time

Recommendation
Benefit / Impact:

The average gc cr block receive time should typically be less than 15 milliseconds depending on your system configuration and volume. This is the average latency of a consistent-read request round-trip from the requesting instance to the holding instance and back to the requesting instance.

Risk:

Potential global cache performance issues

Action / Repair:

Use the following query to determine the average gc cr block receive time for each instance.

set numwidth 20
column "AVG CR BLOCK RECEIVE TIME (ms)" format 9999999.9
select b1.inst_id, ((b1.value / decode(b2.value,0,1)) * 10) "AVG CR BLOCK RECEIVE TIME (ms)"
from gv$sysstat b1, gv$sysstat b2
where b1.name = 'gc cr block receive time' and
b2.name = 'gc cr blocks received' and b1.inst_id = b2.inst_id ;

Needs attention on -

Passed on orcl

Status on orcl:
PASS => Avg GC CR Block Receive Time Within Acceptable Range
DATA FOR ORCL FOR AVG GC CR BLOCK RECEIVE TIME avg_gc_cr_block_receive_time_15ms_exceeded = 10

Hide

Tablespace allocation type

Recommendation

Links
1 Note: 847951.1 - How to Change Extent Management of a Tablespace from Uniform Size to Autoallocate

Needs attention on -

Passed on orcl

Status on orcl:
PASS => Tablespace allocation type is SYSTEM for all appropriate tablespaces
DATA FOR ORCL FOR TABLESPACE ALLOCATION TYPE Query returned no rows which is expected when the SQL check passes.

Hide

Old trace files in background dump destination

Recommendation

Needs attention on rac1

Passed on -

Status on rac1:
INFO => background_dump_dest has files older than 30 days
DATA FROM rac1 - orcl DATABASE - Old trace files in background dump destination files at background dump destination(/oracle/app/oracle/diag/rdbms/orcl/orcl1/trace) older than 30 days = 192

Hide

Alert log file size

Recommendation

Needs attention on -

Passed on rac1

Status on rac1:
PASS => Alert log is not too big
DATA FROM rac1 - orcl DATABASE - Alert log file size -rw-r-----. 1 oracle asmadmin 335957 Jan 25 15:44 /oracle/app/oracle/diag/rdbms/orcl/orcl1/trace/alert_orcl1.log

Hide

Check ORA-07445 errors

Recommendation

Links
1 Note: 1521910.1 - ORA-7445 Troubleshooting Tool

Needs attention on -

Passed on rac1

Status on rac1:
PASS => No ORA-07445 errors found in alert log
DATA FROM rac1 - orcl DATABASE - Check ORA-07445 errors

Hide

Check ORA-00600 errors

Recommendation

Links
1 Note: 1521912.1 - ORA-600 Troubleshooting Tool

Needs attention on -

Passed on rac1

Status on rac1:
PASS => No ORA-00600 errors found in alert log
DATA FROM rac1 - orcl DATABASE - Check ORA-00600 errors

Hide

Check user_dump_destination

Recommendation

Needs attention on rac1

Passed on -

Status on rac1:
INFO => user_dump_dest has trace files older than 30 days
DATA FROM rac1 - orcl DATABASE - Check user_dump_destination 96 files found at /oracle/app/oracle/diag/rdbms/orcl/orcl1/trace which are older than 30 days

Hide

Check core_dump_destination

Recommendation

For this check to pass, you should not have core files older than 30 days in core_dump_destination.

Needs attention on -

Passed on rac1

Status on rac1:
PASS => core_dump_dest does not have too many older core dump files
DATA FROM rac1 - orcl DATABASE - Check core_dump_destination 0 files found at /oracle/app/oracle/diag/rdbms/orcl/orcl1/cdump which are older than 30 days

Hide

Check for parameter semopm

Recommendation
Benefit / Impact:

SEMOPM should be set >= 100

Risk:

Kernel Parameter SEMOPM Is Lower Than The Recommended Value

Action / Repair:

SEMOPM should be set >= 100

Links
1 Note: 1587357.1 - Oracle Database (RDBMS) on Unix AIX,HP-UX,Linux,Solaris and MS Windows Operating Systems Installation and Configuration
2 Configuring Kernel Parameters
3 Note: 169706.1 - Oracle Database OS Installation and Configuration Requirements Quick Reference

Needs attention on -

Passed on rac1

Status on rac1:
PASS => Kernel Parameter SEMOPM OK

Hide

Verify sys and system users default tablespace is system

Recommendation
Benefit / Impact:

It's recommended to Keep the Default Tablespace for SYS and SYSTEM Schema mapped to the Default SYSTEM. All Standard Dictionary objects as well as all the added option will be located in the same place with no risk to record Dictionary data in other Datafiles.

Risk:

If Default tablespace for SYS and SYSTEM is not set to SYSTEM, Data dictionary Object can be created in other locations and cannot be controlled during maintenance activities of the database. Due to this, there's a potential risk to run into severe Data Dictionary Corruption that may implicate time consuming Recovery Steps.

Action / Repair:

If SYS or SYSTEM schema have a Default Tablespace different than SYSTEM, it's recommended to follow instruction given into Note ID : 1111111.2

SQL> SELECT username, default_tablespace
FROM dba_users
WHERE username in ('SYS','SYSTEM');

If DEFAULT_TABLESPACE is anything other than SYSTEM tablespace, modify the default tablespace to SYSTEM by using the below command.

Needs attention on -

Passed on orcl

Status on orcl:
PASS => The SYS and SYSTEM user ids have a default tablespace of SYSTEM
DATA FOR ORCL FOR VERIFY SYS AND SYSTEM USERS DEFAULT TABLESPACE IS SYSTEM SYSTEM SYSTEM

Hide

Maximum parallel asynch io

Recommendation

Warning: OS async I/O limit 128 is lower than recovery batch 1024

Links
1 Note: 471846.1 - Warning: OS async I/O limit 128 is lower than recovery batch 1024 in Alert log

Needs attention on -

Passed on rac1

Status on rac1:
PASS => The number of async IO descriptors is sufficient (/proc/sys/fs/aio-max-nr)
DATA FROM RAC1 - MAXIMUM PARALLEL ASYNCH IO aio-max-nr = 1048576

Hide

Old log files in client directory in crs_home

Recommendation
Benefit / Impact:

Oracle recommends to clean up unwanted log files or archive them to reclaim disk place. There is no impact from checking the old log files in client directory in CRS_HOME

Risk:

Having many old log files in $CRS_HOME/log/hostname/client directory can cause CRS performance issue. So please delete log files older than 15 days.

Action / Repair:

To check the number of log files in the $CRS_HOME/log/hostname/client directory, run the command:
# ls $CRS_HOME/log/hostname/client | wc -l
If the result is very large, please delete the oldest log files from the directory.

Needs attention on rac1

Passed on -

Status on rac1:
INFO => $CRS_HOME/log/hostname/client directory has too many older log files.
DATA FROM RAC1 - OLD LOG FILES IN CLIENT DIRECTORY IN CRS_HOME 21 files in /oracle/app/11.2.0/grid/log/rac1/client directory are older than 15 days

Hide

OCR backup

Recommendation
Benefit / Impact:

Oracle Clusterware automatically creates OCR backups every four hours. At any one time, Oracle Database always retains the last three backup copies of the OCR. The CRSD process that creates the backups also creates and retains an OCR backup for each full day and at the end of each week.

Risk:

Missing backup causes severe issues for OCR recovery operations

Action / Repair:

To check the backup configuration of OCR, run the following command:

# $CRS_HOME/bin/ocrconfig -showbackup auto

The result should list 5 backup files. Three of them are the most recent backups created every four hours within the latest 12 hours, the rest two are for the previous day and the previous week respectively.

Needs attention on rac1

Passed on -

Status on rac1:
WARNING => OCR is not being backed up daily
DATA FROM RAC1 - OCR BACKUP PROT-24: Auto backups for the Oracle Cluster Registry are not available

Hide

Invalid Java Object Check

Recommendation

Links
1 Note: 397770.1 - How to Check if the Oracle JVM is Installed Correctly in the Database

Needs attention on -

Passed on orcl

Status on orcl:
PASS => There are no Invalid Java Objects in the database.
DATA FOR ORCL FOR INVALID JAVA OBJECT CHECK Query returned no rows which is expected when the SQL check passes.

Hide

Check for parameter net.core.rmem_max

Recommendation

net.core.rmem_default =262144
net.core.rmem_max = 2097152 (10g)
net.core.rmem_max = 4194304 (11g and above)

net.core.wmem_default =262144
net.core.wmem_max =1048576

See referenced documents for more details

Links
1 Configuring Kernel Parameters

Needs attention on -

Passed on rac1

Status on rac1:
PASS => net.core.rmem_max is Configured Properly

Hide

Check for parameter spfile

Recommendation

Links
1 Note: 136327.1 - The Server Parameter File in a Real Application Clusters Environment

Needs attention on -

Passed on orcl1

Status on orcl1:
PASS => Instance is using spfile
orcl1.spfile = +DATA/orcl/spfileorcl.ora

Hide

Manage ASM Audit File Directory Growth with cron

Recommendation
Benefit / Impact:

The audit file destination directories for an ASM instance can grow to contain a very large number of files if they are not regularly maintained. Use the Linux cron(8) utility and the find(1) command to manage the number of files in the audit file destination directories.

The impact of using cron(8) and find(1) to manage the number of files in the audit file destination directories is minimal.

Risk:

Having a very large number of files can cause the file system to run out of free disk space or inodes, or can cause Oracle to run very slowly due to file system directory scaling limits, which can have the appearance that the ASM instance is hanging on startup.

Action / Repair:

Refer to the referenced MOS Note

Links
1 Note: 1298957.1 - Manage Audit File Directory Growth with cron

Needs attention on -

Passed on rac1

Status on rac1:
PASS => ASM Audit file destination file count <= 100,000
DATA FROM RAC1 - MANAGE ASM AUDIT FILE DIRECTORY GROWTH WITH CRON Number of audit files at /oracle/app/11.2.0/grid/rdbms/audit = 653

Hide

Non-routable network for interconnect

Recommendation
Benefit / Impact:

Secure and efficient interconnect performance.

Risk:

Latency and security issues.

Action / Repair:

The cluster interconnect should be a completely private/isolated (layer 2 packet processing), non-routable network (the only nodes connected to it are the cluster members themselves).

The 10.x.x.x, 172.x.x.x and 192.x.x.x networks are defined as non-routable networks by standard. Customers who use other networks for the interconnect should ensure that they are not being routed, in which case this finding can be ignored.

Links
1 RFC 1918

Needs attention on -

Passed on rac1

Status on rac1:
PASS => Interconnect is configured on non-routable network addresses
DATA FROM RAC1 - NON-ROUTABLE NETWORK FOR INTERCONNECT eth0 192.168.133.0 global public eth1 10.25.133.0 global cluster_interconnect

Hide

OCR mirroring

Recommendation
Benefit / Impact:

Oracle Clusterware provides the creation of a mirrored Oracle Cluster Registry (OCR) file, enhancing cluster reliability.

There is minimal impact to verify OCR redundancy is being maintained.

Risk:

OCR redundancy optimises availability, performance and reliability of the cluster.

Action / Repair:

Oracle recommends that you maintain a redundant copy of the Oracle Cluster Registry (OCR) either using multiple copies of the OCR file (Oracle redundancy) or by using external redundancy as is provided by a storage array configured for fault tolerance (eg, disk mirroring).If OCR is in ASM and diskgroup is created with external redundancy then you can ignore this warning.

In this case it appears that Oracle's redundancy scheme is not being used. Please verify that the OCR file is located on a fault tolerant disk array or create a second OCR file in a separate physical location using the ocrconfig utility provided by Oracle which is located in the $ORACLE_HOME/bin directory.

$ ocrconfig -help

To check if OCR redundancy is being used, execute the following command:
# $CRS_HOME/bin/ocrcheck
The output should list all OCR mirror files and devices. For example:
Device/File Name : +DATASEPN1
Device/File integrity check succeeded

For version 11.2 or onward, up to five OCR devices/files are allowed. If all
five Device/File entries shows:

Device/File not configured

redundancy is not configured. It is recommended to add OCR redundancy file or
device to current instance. To add/remove/replace/move OCR mirror, please
refer to MOS Note 428681.1.

Links
1 Note: 428681.1 - OCR / Vote disk Maintenance Operations: (ADD/REMOVE/REPLACE/MOVE)

Needs attention on -

Passed on rac1

Status on rac1:
PASS => OCR is mirrored using Oracle redundancy
DATA FROM RAC1 - OCR MIRRORING ocrconfig_loc=+OCR local_only=FALSE

Hide

Hostname formatting

Recommendation
Benefit / Impact:

Per standards for protocols RFC 952 , host names cannot contain the underscore character (“_”)

Risk:

Potential failed upgrades or installations of 12.1 and above

Action / Repair:

Change the hostnames

Links
1 Note: 1957895.1 - One or more node names “host_name” contains "_" character
2 Note: 1603775.1 - Hostname Restriction/Limitations For Database Server

Needs attention on -

Passed on rac1

Status on rac1:
PASS => None of the hostnames contains an underscore character
DATA FROM RAC1 - HOSTNAME FORMATTING rac1 rac2

Hide

Verify RMAN snapshot control file location is properly shared

Recommendation
Benefit / Impact:

For effective backup and recovery operations, it is mandatory that the RMAN snapshot controlfile be shared on all database nodes in a cluster.

Risk:

RMAN backup/recovery operations may be negatively impacted.

Action / Repair:

Please refer to the following examples and Oracle Database Documentation for steps to locate RMAN snapshot controlfile onto a shared storage.

Examples:
Shared storage location on ASM:
$ rman target /
RMAN> SHOW SNAPSHOT CONTROLFILE NAME;
RMAN> CONFIGURE SNAPSHOT CONTROLFILE NAME TO '+RECODG/CDB88/snapcf_CDB88.ctl';
RMAN> exit;

Shared storage location on ACFS (cluster filesystem):
$ rman target /
RMAN> SHOW SNAPSHOT CONTROLFILE NAME;
RMAN> CONFIGURE SNAPSHOT CONTROLFILE NAME TO '/acfs01/oracle/oradata/fast_recovery_area/cdb88/CDB88/snapcf_CDB88.ctl';
RMAN> exit;

Links
1 6.4.2 Setting the Location of the Snapshot Control File

Needs attention on rac1

Passed on -

Status on rac1:
FAIL => The RMAN snapshot controlfile location is NOT shared on all database nodes in the cluster
DATA FROM rac1 - orcl DATABASE - Verify RMAN snapshot control file location is properly shared Snapshot Controlfile is not created because: select distinct value from v$rman_configuration where name='SNAPSHOT CONTROLFILE NAME' did not return any value.

Hide

Check for parameter net.core.rmem_default

Recommendation

net.core.rmem_default =262144
net.core.rmem_max = 2097152 (10g)
net.core.rmem_max = 4194304 (11g and above)

net.core.wmem_default =262144
net.core.wmem_max =1048576

See referenced documents for more details

Links
1 Configuring Kernel Parameters

Needs attention on -

Passed on rac1

Status on rac1:
PASS => net.core.rmem_default Is Configured Properly

Hide

Check for parameter net.core.wmem_max

Recommendation

net.core.rmem_default =262144
net.core.rmem_max = 2097152 (10g)
net.core.rmem_max = 4194304 (11g and above)

net.core.wmem_default =262144
net.core.wmem_max =1048576

See referenced documents for more details

Links
1 Configuring Kernel Parameters

Needs attention on -

Passed on rac1

Status on rac1:
PASS => net.core.wmem_max Is Configured Properly

Hide

Check for parameter net.core.wmem_default

Recommendation

net.core.rmem_default =262144
net.core.rmem_max = 2097152 (10g)
net.core.rmem_max = 4194304 (11g and above)

net.core.wmem_default =262144
net.core.wmem_max =1048576

See referenced documents for more details

Links
1 Configuring Kernel Parameters

Needs attention on -

Passed on rac1

Status on rac1:
PASS => net.core.wmem_default Is Configured Properly

Hide

Verify cluster_database_instances database parameter

Recommendation
Benefit / Impact:

If undersized internal algorithms will fail. The default value is sufficient as long as customers don’t run multiple instances for the same database on the same server. Even if customer has this value explicitly set to the Oracle default value, it should be un-set because the default value may change in the future.

Risk:

Algorithms will fail

Action / Repair:

Unset cluster_database_instances database parameter using alter system reset cluster_database_instances scope=spfile sid='*';

Needs attention on -

Passed on orcl

Status on orcl:
PASS => cluster_database_instances database parameter is set to default value
DATA FOR ORCL FOR VERIFY CLUSTER_DATABASE_INSTANCES DATABASE PARAMETER 1 cluster_database_instances TRUE

Hide

Archive log mode

Recommendation

Risk:

Without ARCHIVELOG mode the database cannot be recovered from an online backup and Data Guard cannot be used.

Needs attention on -

Passed on orcl

Status on orcl:
PASS => ARCHIVELOG mode is enabled
DATA FOR ORCL FOR ARCHIVE LOG MODE archive_log_mode_check = ARCHIVELOG

Hide

ASM_DISKSTRING

Recommendation
Benefit / Impact:

Enhanced availability for normal clusterware operation and avoid problems during upgrades

Risk:

CSSD may fail to start correctly

Action / Repair:

Set ASM_DISKSTRING parameter to appropriate discovery string.
Verbose discovery strings (e.g., /dev/*) can lead to failure of the GI stack to successfully start.

The default value of the ASM_DISKSTRING parameter is a NULL string. A NULL value causes Oracle ASM to search a default path for all disks.

Prior to 12.1, the default discovery string/path is to scan all raw and/or block devices (platform dependent string) which on some systems may be too verbose.

Note: For 12.1 onwards on Linux RAW devices are not supported and the default discovery string/path is /dev/sd* and block devices are supported as default.

Links
1 Note: 1917543.1 - FAILS TO START ORA.CSSD WHEN UPGRADING GRID 12C
2 Note: 1916106.1 - Wrong DiscoveryString /dev/*: rootupgrade.sh/root.sh hangs: Check OCR key using ocrdump
3 Note: 1579762.1 - Things to Consider Before Upgrading to 12.1.0.1 Grid Infrastructure/ASM
4 Note: 1581553.1 - rootupgrade.sh to 12.1.0.1 GI fails with "CLSRSC-115: Start of resource 'ora.cssd' failed" if asm_diskstring is null

Needs attention on rac1

Passed on -

Status on rac1:
FAIL => ASM_DISKSTRING parameter is either null or set to /dev/*
DATA FROM RAC1 - ASM_DISKSTRING ASM_DISKSTRING ------------------------------

Hide

AUDSES$ sequence cache size

Recommendation
There are problems reported with Audses$ and ora_tq_base$ which are both internal sequences . Also particularly if the order of the application sequence is not important or this is used during the login process and hence can be involved in a login storm then this needs to be taken care of. Some sequences need to be presented in a particular order and hence caching those is not a good idea but in the interest of performance if order does not matter then this could be cached and presented. This also manifests itself as waits in "rowcache" for "dc_sequences" which is a rowcache type for sequences.

For Applications this can cause significant issues especially with Transactional Sequences.
Please see note attached.

Oracle General Ledger - Version: 11.5.0 to 11.5.10
Oracle Payables - Version: 11.5.0 to 11.5.10
Oracle Receivables - Version: 11.5.10.2
Information in this document applies to any platform.
ARXTWAI,ARXRWMAI

Increase IDGEN1$ to a value of 1000, see notes below. This is the default as of 11.2.0.1.

Links
1 Note: 561414.1 - Transactional Sequences in Applications in a RAC environment
2 Note: 432508.1 - High SQ Enqueue Contention with LOBs or Advanced Replication

Needs attention on -

Passed on orcl

Status on orcl:
PASS => SYS.AUDSES$ sequence cache size >= 10,000
DATA FOR ORCL FOR AUDSES$ SEQUENCE CACHE SIZE audses$.cache_size = 10000

Hide

IDGEN$ sequence cache size

Recommendation

Links
1 Note: 561414.1 - Transactional Sequences in Applications in a RAC environment
2 Note: 432508.1 - High SQ Enqueue Contention with LOBs or Advanced Replication
3 Oracle Bug # 7694580 - High SQ Enqueue Contention with LOBs or Advanced Replication - HIGH ENQ:SQ CONTENTION WITH CONCURRENT SECUREFILE INSERTS/LOADS
4 Note: 268476.1 - LOB Performance Guideline

Needs attention on -

Passed on orcl

Status on orcl:
PASS => SYS.IDGEN1$ sequence cache size >= 1,000
DATA FOR ORCL FOR IDGEN$ SEQUENCE CACHE SIZE idgen1$.cache_size = 1000

Hide

OCR and OCR backup location

Recommendation
Benefit / Impact:

For reasons of avaiability the OCR and OCR backups ought to be in different locations, eg., different ASM diskgroups or different local filesystems, etc.

Risk:

Potential loss of Oracle Cluster Registry and downtime.

Action / Repair:

Store the OCR backups in different ASM diskgroups or filesystems than where the OCR is stored.

Needs attention on -

Passed on rac1

Status on rac1:
PASS => OCR and OCR backup locations are not in the same path
DATA FROM RAC1 - OCR AND OCR BACKUP LOCATION ocr_loc = +OCR ocrbackup_loc = .

Hide

Verify private and public network subnet configuration in Oracle Clusterware registry

Recommendation
Benefit / Impact:

The subnets for Private and Public networks are stored in the OCR (Oracle Cluster Registry) when the Clusterware is installed. These subnets are the one's which are specified during Installation time.

It is possible that the actual subnet associated with an Network interface may get changed due to some Network configuration change (eg., netmask is changed AFTER clusterware installation or improperly specified at install time) but the subnet configured in the OCR remains unchanged.

This causes a mismatch between the subnets in the OCR and actual subnets. In order to verify you can compare the output of the following commands:

1. $CRS_HOME/bin/oifcfg getif <==returns subnet configuration stored in OCR
2. $CRS_HOME/bin/oifcfg iflist <==returns actual subnet configuration

Compare the subnets for each network interface and they should match. If they do not then that means there is a mis-configuration. Here is an example of a mis-configuration:

$./oifcfg getif
eth6 192.168.3.0 global public
eth7 192.168.1.0 global cluster_interconnect <<<subnet stored in OCR

$./oifcfg iflist
eth7 192.168.1.8 <<<actual subnet due to net mask being changed after clusterware install
eth6 192.168.3.0

Risk:

Some of the most common issues faced (but not limited to) due to
wrong/miscalculated Network subnets are:
1) HAIP startup issues.
2) RDBMS/ASM Instance startup issues
3) VIP/SCAN startup issues.
4) GIPCD not using certain Interfaces for HAIP

Action / Repair:

1)To modify the subnet associated with Private Network (cluster Interconnect) we need to issue the following commands:

oifcfg delif -global <interface>/<wrong_subnet>
oifcfg setif -global <interface>/<correct_subnet>:cluster_interconnect

2) To modify the subnet associated with Public Network we need to issue the following commands:

oifcfg delif -global <interface>/<wrong_subnet>
oifcfg setif -global <interface>/<correct_subnet>:public

For changes associated with private network/cluster interconnect or the public network, please refer to the referenced MOS notes.

Links
1 Note: 1386709.1 - The Basics of IPv4 Subnet and Oracle Clusterware
2 Note: 1059759.1 - How to calculate the correct subnet for an interface
3 Note: 283684.1 - How to Modify Private Network Information in Oracle Clusterware
4 Note: 276434.1 - How to Modify Public Network Information including VIP in Oracle Clusterware

Needs attention on -

Passed on rac1

Status on rac1:
PASS => Private and public network subnet configuration is correct in OCR
DATA FROM RAC1 - VERIFY PRIVATE AND PUBLIC NETWORK SUBNET CONFIGURATION IN ORACLE CLUSTERWARE REGISTRY oifcfg iflist output eth0 192.168.133.0 eth1 10.25.133.0 oifcfg getif output eth0 192.168.133.0 global public eth1 10.25.133.0 global cluster_interconnect

Hide

Check ORACLE_PATCH 19855835 for RDBMS home

Recommendation
Benefit / Impact:

This is one of the top popular issues reported in 12.1.0.2.
Applying this patch proactively would help avoiding issues related to slow upgrade if there is a large amount of data in the stats history tables.

Risk:

During the upgrade to 11.2.0.4 or higher, a large amount of time is spent creating indexes on WRI$_OPTSTAT_HISTHEAD_HISTORY or WRI$_OPTSTAT_HISTGRM_HISTORY.

Action / Repair:

It is highly recommended to download & apply the Patch 19855835.
This fix is for the upgrade script and should be applied to the new ORACLE_HOME upgrading the database/s.

Links
1 Note: 19855835.8 - Bug 19855835 - Upgrade slow when reorganizing large stats history tables

Needs attention on rac1:/oracle/app/oracle/product/11.2.0/db

Passed on -

Status on rac1:/oracle/app/oracle/product/11.2.0/db:
WARNING => Oracle patch 19855835 is not applied on RDBMS_HOME
Oracle Interim Patch Installer version 11.2.0.3.16 Copyright (c) 2018, Oracle Corporation. All rights reserved. Oracle Home : /oracle/app/oracle/product/11.2.0/db Central Inventory : /oracle/app/oraInventory from : /oracle/app/oracle/product/11.2.0/db/oraInst.loc OPatch version : 11.2.0.3.16 OUI version : 11.2.0.4.0 Log file location : /oracle/app/oracle/product/11.2.0/db/cfgtoollogs/opatch/opatch2018-01-25_16-37-33PM_1.log Lsinventory Output file location : /oracle/app/oracle/product/11.2.0/db/cfgtoollogs/opatch/lsinv/lsinventory2018-01-25_16-37-33PM.txt ------------------------------------------------------------------------------------------------------ Local Machine Information:: Hostname: rac1 ARU platform id: 226 ARU platform description:: Linux x86-64
Installed Top-level Products (1): Oracle Database 11g 11.2.0.4.0 There are 1 products installed in this Oracle Home. There are no Interim patches installed in this Oracle Home. ------------------------------------------------------------------------------------------------------ OPatch succeeded.
Click for more data

Hide

Check ORACLE_PATCH 20907061 for RDBMS home

Recommendation
Benefit / Impact:

This is one of the top popular issues while upgrading to 11.2.0.4 / 12.1.0.2.
Applying this patch proactively would help avoiding the issues regarding high no.of executions / parse calls on col$.

Risk:

There would be high no.of executions on recursive SQL: 47r1y8yn34jmj.
This might lead to heavy shared pool contention due to lot of parse calls on the SQL.

Action / Repair:

It is highly recommended to download & apply the Patch 20907061.

Links
1 Note: 20907061.8 - Bug 20907061 - high number of executions for recursive call on col$

Needs attention on rac1:/oracle/app/oracle/product/11.2.0/db

Passed on -

Status on rac1:/oracle/app/oracle/product/11.2.0/db:
WARNING => Oracle patch 20907061 is not applied on RDBMS_HOME
Oracle Interim Patch Installer version 11.2.0.3.16 Copyright (c) 2018, Oracle Corporation. All rights reserved. Oracle Home : /oracle/app/oracle/product/11.2.0/db Central Inventory : /oracle/app/oraInventory from : /oracle/app/oracle/product/11.2.0/db/oraInst.loc OPatch version : 11.2.0.3.16 OUI version : 11.2.0.4.0 Log file location : /oracle/app/oracle/product/11.2.0/db/cfgtoollogs/opatch/opatch2018-01-25_16-37-33PM_1.log Lsinventory Output file location : /oracle/app/oracle/product/11.2.0/db/cfgtoollogs/opatch/lsinv/lsinventory2018-01-25_16-37-33PM.txt ------------------------------------------------------------------------------------------------------ Local Machine Information:: Hostname: rac1 ARU platform id: 226 ARU platform description:: Linux x86-64
Installed Top-level Products (1): Oracle Database 11g 11.2.0.4.0 There are 1 products installed in this Oracle Home. There are no Interim patches installed in this Oracle Home. ------------------------------------------------------------------------------------------------------ OPatch succeeded.
Click for more data

Hide

Check ORACLE_PATCH 20476175 for RDBMS home

Recommendation
Benefit / Impact:

This is one of the top popular issues while upgrading to 11.2.0.4 / 12.1.0.2.
Applying this patch proactively would help avoiding the issues regarding high version count due to Optimizer mismatch.

Risk:

Cursors are not shared across same SQL when it has an OPT_PARAM hint with _fix_control.
You will see high version_count in V$SQLAREA due to Optimizer Mismatch.

Action / Repair:

It is highly recommended to download & apply the Patch 20476175.

Links
1 Note: 20476175.8 - Bug 20476175 - High VERSION_COUNT (in V$SQLAREA) for query with OPT_PARAM('_fix_control') hint

Needs attention on rac1:/oracle/app/oracle/product/11.2.0/db

Passed on -

Status on rac1:/oracle/app/oracle/product/11.2.0/db:
WARNING => Oracle patch 20476175 is not applied on RDBMS_HOME
Oracle Interim Patch Installer version 11.2.0.3.16 Copyright (c) 2018, Oracle Corporation. All rights reserved. Oracle Home : /oracle/app/oracle/product/11.2.0/db Central Inventory : /oracle/app/oraInventory from : /oracle/app/oracle/product/11.2.0/db/oraInst.loc OPatch version : 11.2.0.3.16 OUI version : 11.2.0.4.0 Log file location : /oracle/app/oracle/product/11.2.0/db/cfgtoollogs/opatch/opatch2018-01-25_16-37-33PM_1.log Lsinventory Output file location : /oracle/app/oracle/product/11.2.0/db/cfgtoollogs/opatch/lsinv/lsinventory2018-01-25_16-37-33PM.txt ------------------------------------------------------------------------------------------------------ Local Machine Information:: Hostname: rac1 ARU platform id: 226 ARU platform description:: Linux x86-64
Installed Top-level Products (1): Oracle Database 11g 11.2.0.4.0 There are 1 products installed in this Oracle Home. There are no Interim patches installed in this Oracle Home. ------------------------------------------------------------------------------------------------------ OPatch succeeded.
Click for more data

Hide

Check ORACLE_PATCH 20879889 for RDBMS home

Recommendation
Benefit / Impact:

This is one of the top popular issues reported in 12.1.0.2 / 11.2.0.4.
Applying this patch proactively would help avoiding the issues related to cursor leak for sessions performing DML against tables with a materialized view log.

Risk:

Sessions performing DML against tables with a materialized view log on them can leak cursors
for the insert into the MV log.

If a session leaks cursors for "INSERT INTO MLOG$_<table_name>" statements and the fix for bug 16019728 is present then it is likely due to this bug.
If ORA-04025 is seen on "INSERT INTO MLOG$_<TABLE_NAME>" statements, then it is also likely you are hitting this bug.

Action / Repair:

It is highly recommended to download & apply the Patch 20879889.
Or, reduce the session_cached_cursors parameter on sessions running DML on tables that have an MV log as the workaround.

Links
1 Note: 20879889.8 - Bug 20879889 - Open cursor leak from DML on table with a materialized view log

Needs attention on rac1:/oracle/app/oracle/product/11.2.0/db

Passed on -

Status on rac1:/oracle/app/oracle/product/11.2.0/db:
WARNING => Oracle patch 20879889 is not applied on RDBMS_HOME
Oracle Interim Patch Installer version 11.2.0.3.16 Copyright (c) 2018, Oracle Corporation. All rights reserved. Oracle Home : /oracle/app/oracle/product/11.2.0/db Central Inventory : /oracle/app/oraInventory from : /oracle/app/oracle/product/11.2.0/db/oraInst.loc OPatch version : 11.2.0.3.16 OUI version : 11.2.0.4.0 Log file location : /oracle/app/oracle/product/11.2.0/db/cfgtoollogs/opatch/opatch2018-01-25_16-37-33PM_1.log Lsinventory Output file location : /oracle/app/oracle/product/11.2.0/db/cfgtoollogs/opatch/lsinv/lsinventory2018-01-25_16-37-33PM.txt ------------------------------------------------------------------------------------------------------ Local Machine Information:: Hostname: rac1 ARU platform id: 226 ARU platform description:: Linux x86-64
Installed Top-level Products (1): Oracle Database 11g 11.2.0.4.0 There are 1 products installed in this Oracle Home. There are no Interim patches installed in this Oracle Home. ------------------------------------------------------------------------------------------------------ OPatch succeeded.
Click for more data

Hide

Check ORACLE_PATCH 19174639 for RDBMS home

Recommendation
Benefit / Impact:

This is one of the top popular issues reported in 12.1.0.2 / 11.2.0.4.
Applying this patch proactively would help avoiding the issues related to OJPPD (old style push join predicate).

Risk:

Optimizer might not choose old style JPPD which could lead to suboptimal plan.

Action / Repair:

It is highly recommended to download & apply the Patch 19174639.

Links
1 Note: 19174639.8 - Bug 19174639 - OJPPD not occurring when expected with fix for bug 16736611

Needs attention on rac1:/oracle/app/oracle/product/11.2.0/db

Passed on -

Status on rac1:/oracle/app/oracle/product/11.2.0/db:
WARNING => Oracle patch 19174639 is not applied on RDBMS_HOME
Oracle Interim Patch Installer version 11.2.0.3.16 Copyright (c) 2018, Oracle Corporation. All rights reserved. Oracle Home : /oracle/app/oracle/product/11.2.0/db Central Inventory : /oracle/app/oraInventory from : /oracle/app/oracle/product/11.2.0/db/oraInst.loc OPatch version : 11.2.0.3.16 OUI version : 11.2.0.4.0 Log file location : /oracle/app/oracle/product/11.2.0/db/cfgtoollogs/opatch/opatch2018-01-25_16-37-33PM_1.log Lsinventory Output file location : /oracle/app/oracle/product/11.2.0/db/cfgtoollogs/opatch/lsinv/lsinventory2018-01-25_16-37-33PM.txt ------------------------------------------------------------------------------------------------------ Local Machine Information:: Hostname: rac1 ARU platform id: 226 ARU platform description:: Linux x86-64
Installed Top-level Products (1): Oracle Database 11g 11.2.0.4.0 There are 1 products installed in this Oracle Home. There are no Interim patches installed in this Oracle Home. ------------------------------------------------------------------------------------------------------ OPatch succeeded.
Click for more data

Hide

Check ORACLE_PATCH 24739928 for RDBMS home

Recommendation
Benefit / Impact:

This is one of the top popular issues reported in 12.1.0.2 / 11.2.0.4.
Applying this patch proactively would help avoiding the issues related to mutex waits like "library cache: mutex X", "cursor: pin S".

Risk:

ORA-7445 [kglMutexCleanupAll] or Mutex contention due to "library cache: mutex X" or "cursor: pin S" if the patch is not installed.

Action / Repair:

It is highly recommended to download & apply the Patch 24739928 to avoid mutex related issues in DB.

Links
1 Note: 24739928.8 - ORA-7445 [kglMutexCleanupAll] with fix for Bug 13542050
2 Note: 2051456.1 - Troubleshooting Databases Hang Due to Heavy Contention for 'library cache: mutex X' Waits (Oracle 11.2 and Later)

Needs attention on rac1:/oracle/app/oracle/product/11.2.0/db

Passed on -

Status on rac1:/oracle/app/oracle/product/11.2.0/db:
WARNING => Oracle patch 24739928 is not applied on RDBMS_HOME
Oracle Interim Patch Installer version 11.2.0.3.16 Copyright (c) 2018, Oracle Corporation. All rights reserved. Oracle Home : /oracle/app/oracle/product/11.2.0/db Central Inventory : /oracle/app/oraInventory from : /oracle/app/oracle/product/11.2.0/db/oraInst.loc OPatch version : 11.2.0.3.16 OUI version : 11.2.0.4.0 Log file location : /oracle/app/oracle/product/11.2.0/db/cfgtoollogs/opatch/opatch2018-01-25_16-37-33PM_1.log Lsinventory Output file location : /oracle/app/oracle/product/11.2.0/db/cfgtoollogs/opatch/lsinv/lsinventory2018-01-25_16-37-33PM.txt ------------------------------------------------------------------------------------------------------ Local Machine Information:: Hostname: rac1 ARU platform id: 226 ARU platform description:: Linux x86-64
Installed Top-level Products (1): Oracle Database 11g 11.2.0.4.0 There are 1 products installed in this Oracle Home. There are no Interim patches installed in this Oracle Home. ------------------------------------------------------------------------------------------------------ OPatch succeeded.
Click for more data

Hide

Check ORACLE_PATCH 18498878 for RDBMS home

Recommendation
Benefit / Impact:

This is one of the top popular issues reported in 12.1.0.2.
Applying this patch proactively would help avoiding the issues regarding unnecessary direct reads.

Risk:

Medium size tables (_small_table_treshold < table size < 10% db block buffers) may not be cached properly and consistently even after repeatedly full scan, causing unnecessary direct reads.

Action / Repair:

It is highly recommended to download & apply the Patch 18498878.

Links
1 Note: 18498878.8 - Bug 18498878 - medium size tables do not cached consistently

Needs attention on rac1:/oracle/app/oracle/product/11.2.0/db

Passed on -

Status on rac1:/oracle/app/oracle/product/11.2.0/db:
WARNING => Oracle patch 18498878 is not applied on RDBMS_HOME
Oracle Interim Patch Installer version 11.2.0.3.16 Copyright (c) 2018, Oracle Corporation. All rights reserved. Oracle Home : /oracle/app/oracle/product/11.2.0/db Central Inventory : /oracle/app/oraInventory from : /oracle/app/oracle/product/11.2.0/db/oraInst.loc OPatch version : 11.2.0.3.16 OUI version : 11.2.0.4.0 Log file location : /oracle/app/oracle/product/11.2.0/db/cfgtoollogs/opatch/opatch2018-01-25_16-37-33PM_1.log Lsinventory Output file location : /oracle/app/oracle/product/11.2.0/db/cfgtoollogs/opatch/lsinv/lsinventory2018-01-25_16-37-33PM.txt ------------------------------------------------------------------------------------------------------ Local Machine Information:: Hostname: rac1 ARU platform id: 226 ARU platform description:: Linux x86-64
Installed Top-level Products (1): Oracle Database 11g 11.2.0.4.0 There are 1 products installed in this Oracle Home. There are no Interim patches installed in this Oracle Home. ------------------------------------------------------------------------------------------------------ OPatch succeeded.
Click for more data

Hide

Check ORACLE_PATCH 22113854 for RDBMS home

Recommendation
Benefit / Impact:

This is one of the top popular issues reported in 12.1.0.2.
Applying this patch proactively would help avoiding the issues regarding ALL_SYNONYMS in 12c.

Risk:

There could be likelihood of issues related to slow query on ALL_SYNONYMS in 12c if the patch is not installed.

Action / Repair:

It is highly recommended to download & apply the Patch 22113854.

Links
1 Note: 22113854.8 - Bug 22113854 - In 12c slow query on ALL_SYNONYMS

Needs attention on rac1:/oracle/app/oracle/product/11.2.0/db

Passed on -

Status on rac1:/oracle/app/oracle/product/11.2.0/db:
WARNING => Oracle patch 22113854 is not applied on RDBMS_HOME
Oracle Interim Patch Installer version 11.2.0.3.16 Copyright (c) 2018, Oracle Corporation. All rights reserved. Oracle Home : /oracle/app/oracle/product/11.2.0/db Central Inventory : /oracle/app/oraInventory from : /oracle/app/oracle/product/11.2.0/db/oraInst.loc OPatch version : 11.2.0.3.16 OUI version : 11.2.0.4.0 Log file location : /oracle/app/oracle/product/11.2.0/db/cfgtoollogs/opatch/opatch2018-01-25_16-37-33PM_1.log Lsinventory Output file location : /oracle/app/oracle/product/11.2.0/db/cfgtoollogs/opatch/lsinv/lsinventory2018-01-25_16-37-33PM.txt ------------------------------------------------------------------------------------------------------ Local Machine Information:: Hostname: rac1 ARU platform id: 226 ARU platform description:: Linux x86-64
Installed Top-level Products (1): Oracle Database 11g 11.2.0.4.0 There are 1 products installed in this Oracle Home. There are no Interim patches installed in this Oracle Home. ------------------------------------------------------------------------------------------------------ OPatch succeeded.
Click for more data

Hide

Check ORACLE_PATCH 23665623 for RDBMS home

Recommendation
Benefit / Impact:

This is one of the top popular issues reported in 12.1.0.2.
Applying this patch proactively would help avoiding the issues like ORA-7445 [kkeics].

Risk:

ORA-7445 [kkeics] may be seen when fix for Bug 21091518 is installed.

Action / Repair:

It is highly recommended to download & apply the Patch 23665623.

Links
1 Note: 23665623.8 - Bug 23665623 - ORA-7445 [kkeics] with fix for bug 21091518

Needs attention on rac1:/oracle/app/oracle/product/11.2.0/db

Passed on -

Status on rac1:/oracle/app/oracle/product/11.2.0/db:
WARNING => Oracle patch 23665623 is not applied on RDBMS_HOME
Oracle Interim Patch Installer version 11.2.0.3.16 Copyright (c) 2018, Oracle Corporation. All rights reserved. Oracle Home : /oracle/app/oracle/product/11.2.0/db Central Inventory : /oracle/app/oraInventory from : /oracle/app/oracle/product/11.2.0/db/oraInst.loc OPatch version : 11.2.0.3.16 OUI version : 11.2.0.4.0 Log file location : /oracle/app/oracle/product/11.2.0/db/cfgtoollogs/opatch/opatch2018-01-25_16-37-33PM_1.log Lsinventory Output file location : /oracle/app/oracle/product/11.2.0/db/cfgtoollogs/opatch/lsinv/lsinventory2018-01-25_16-37-33PM.txt ------------------------------------------------------------------------------------------------------ Local Machine Information:: Hostname: rac1 ARU platform id: 226 ARU platform description:: Linux x86-64
Installed Top-level Products (1): Oracle Database 11g 11.2.0.4.0 There are 1 products installed in this Oracle Home. There are no Interim patches installed in this Oracle Home. ------------------------------------------------------------------------------------------------------ OPatch succeeded.
Click for more data

Hide

cluster_interconnects

Recommendation
Benefit / Impact:

Experience and testing has shown that certain database initialization parameters should be set at specific values. These are the best practice values set at deployment time. By setting these database initialization parameters as recommended, known problems may be avoided and performance maximized. The parameters are common to all database instances. The impact of setting these parameters is minimal. The performance related settings provide guidance to maintain highest stability without sacrificing performance. Changing the default performance settings can be done after careful performance evaluation and clear understanding of the performance impact.

Risk:

If the database initialization parameters are not set as recommended, a variety of issues may be encountered, depending upon which initialization parameter is not set as recommended, and the actual set value.

Action / Repair:

Database parameter CLUSTER_INTERCONNECTS should be a colon delimited string of the IP addresses returned from sbin/ifconfig for each cluster_interconnect interface returned by oifcfg.

This is used to avoid the Clusterware HAIP address as its use is not supported on Exadata (the only exception being with RAC One Node).

Needs attention on rac1

Passed on -

Status on rac1:
FAIL => Database parameter CLUSTER_INTERCONNECTS is not set to the recommended value
DATA FROM rac1 - orcl DATABASE - cluster_interconnects Cluster Interconnect Value from instance = Network card name and IP from oifcfg eth1 = 10.25.133.120

Hide

3rd Party Clusterware Node Numbering

Recommendation
Benefit / Impact:

Nodes in a RAC cluster should be numbered beginning with 1 (one) when the output of $CRS_HOME/bin olsnodes is displayed whether using 3rd party vendor clusterware or not.

Risk:

Potential problems during the upgrade process.

Action / Repair:

Correct the problem as described in the referenced MOS note.

Links
1 Note: 1917917.1 - 12c GI rootupgrade.sh Fails on First Node With ORA-01034 if Node Number Starts From 0

Needs attention on -

Passed on rac1

Status on rac1:
PASS => Cluster nodes appear to be correctly numbered
DATA FROM RAC1 - 3RD PARTY CLUSTERWARE NODE NUMBERING rac1 1 rac2 2

Hide

Check Id	Name	Type	Target	Execution Duration
5ACC89B1D10CAF91E0401490CACF41F5	Cluster interconnect (clusterware)	OS Collection	rac1:None	13 secs
078C3A95E8BF672CE05312C0E50A0501	Monitoring Non-AWR space usage in SYSAUX	SQL Check	rac1:orcl	12 secs
B5191156EF114416E0431EC0E50AFBE0	Patches for RDBMS Home	OS Collection	rac1:None	7 secs
DC3D819F5D2A50FEE04312C0E50AFF9F	Parallel Execution Health-Checks and Diagnostics Reports	OS Check	rac1:orcl	7 secs
5EA8F4C6C6BDF8F0E0401490CACF067F	Interconnect NIC bonding config.	OS Check	rac1	6 secs
81DB99F842EE95C9E040E50A1EC03CC3	NIC Bonding Mode for interconnect	OS Check	rac1	6 secs
8529D3798EA039F3E040E50A1EC07218	Public interface existence	OS Check	rac1	6 secs
494DAA11615B67D6E0530D98EB0A34D3	Verify private and public network subnet configuration in Oracle Clusterware registry	OS Check	rac1	6 secs
C75507CE93A92E2EE0431EC0E50A187F	Subnet mask for cluster_interconnect NICS branch	OS Check	rac1	6 secs
E1AB0C30BA3D0A78E04312C0E50AA0B6	More than one card for interconnect?	OS Check	rac1	6 secs

Oracle RAC Assessment Report failed_checks System Health Score is 86 out of 100 (detail)

Cluster Summary

Table of Contents

Report Feature

Database Server

Maximum Availability Architecture (MAA) Scorecard

Patch Recommendation

Database server "rac1"

Verify no duplicate parameter entries in database init.ora(spfile)

Verify service_names parameter is not defined in spfile

Verify hotplug attribute on network cards used by Clusterware

kernel.panic_on_oops

Monitoring SGA resize operations

Monitoring waiter sessions

session_cached_cursors parameter

Monitoring changes to schema objects

Verify ASM disk permissions

Verify free disk space for rebalance

Optimizer Parameters with Non-Default Settings (Check 2)

Optimizer Parameters with Non-Default Settings (Check 3)

Obsolete Database Parameters for 11.2

optimizer_features_enable parameter

nls_sort parameter

Verify online (hot) patches are not applied on CRS_HOME

Tablespace extent management Allocation Type for SecureFiles LOB storage

ASM MEMORY_TARGET Size

Verify online (hot) patches are not applied on ORACLE_HOME

System Core File Limit

Optimizer Parameters with Non-Default Settings (Check 1)

Verify Berkeley Database location for Cloned GI homes

Verify control_file_record_keep_time value is in recommended range

Verify rman controlfile autobackup is set to ON

Registered diskgroups in clusterware registry

Check for parameter cvuqdisk|1.0.9|1|x86_64

pam_limits check

Verify vm.min_free_kbytes

Verify transparent hugepages are disabled

Verify data files are recoverable

Check for parameter unixODBC-devel|2.2.14|11.el6|x86_64

OCR and Voting file location

Parallel Execution Health-Checks and Diagnostics Reports

Hardware clock synchronization

TFA Collector status

Clusterware resource status

ORA-15196 errors in ASM alert log

Disks without Disk Group

localhost entry in /etc/hosts

Redo log file write time latency

Primary database protection with Data Guard

Locally managed tablespaces

Automatic segment storage management

Default Temporary Tablespace

Archivelog Mode

Verify AUD$ and FGA_LOG$ tables use Automatic Segment Space Management

Check for parameter libgcc|4.4.4|13.el6|x86_64

ASM disk read write error

Block Corruptions

Redo Log File Size

Check for parameter sysstat|9.0.4|11.el6|x86_64

Check for parameter libgcc|4.4.4|13.el6|i686

Check for parameter binutils|2.20.51.0.2|5.11.el6|x86_64

Check for parameter glibc|2.12|1.7.el6|x86_64

Check for parameter libstdc++|4.4.4|13.el6|x86_64

Check for parameter libstdc++|4.4.4|13.el6|i686

Check for parameter glibc|2.12|1.7.el6|i686

Check for parameter gcc|4.4.4|13.el6|x86_64

Check for parameter make|3.81|19.el6|

Check for parameter libstdc++-devel|4.4.4|13.el6|i686

Check for parameter libaio-devel|0.3.107|10.el6|x86_64

Check for parameter libaio|0.3.107|10.el6|x86_64

Check for parameter unixODBC-devel|2.2.14|11.el6|i686

Check for parameter compat-libstdc++-33|3.2.3|69.el6|i686

Check for parameter glibc-devel|2.12|1.7.el6|x86_64

Check for parameter glibc-devel|2.12|1.7.el6|i686

Check for parameter compat-libcap1|1.10|1|x86_64

Check for parameter ksh|20100621|12.el6|x86_64

Check for parameter unixODBC|2.2.14|11.el6|i686

Check for parameter libaio|0.3.107|10.el6|i686

Check for parameter libstdc++-devel|4.4.4|13.el6|x86_64

Check for parameter gcc-c++|4.4.4|13.el6|x86_64

Oracle RAC Assessment Report

failed_checks
System Health Score is 86 out of 100 (detail)