In this Document
Applies to: Oracle Database - Enterprise Edition - Version 11.2.0.4 and later Information in this document applies to any platform.
Symptoms
One of the node CRS was not coming up properly.
ASM did not come up automatically - it failed to start with "ORA-01017: invalid username/password; logon denied" error in the /log//agent/ohasd/oraagent_grid/oraagent_grid.log
alert.log
~~~~~~~~~~~~~~~~~~~~
[client(3990)]CRS-10001:10-Jun-14 14:35 ACFS-9156: Detecting control device '/dev/ofsctl'.
[client(4002)]CRS-10001:10-Jun-14 14:35 ACFS-9322: completed
2014-06-10 14:36:18.128:
[ohasd(48895)]CRS-2807:Resource 'ora.asm' failed to start automatically.
2014-06-10 14:36:18.128:
[ohasd(48895)]CRS-2807:Resource 'ora.crsd' failed to start automatically.
oraagent_grid.log
~~~~~~~~~~~~~~~~~~
2014-06-10 14:35:15.585: [ora.asm][3219122496]{0:0:2} [start] makeConnectStr = (DESCRIPTION=(ADDRESS=(PROTOCOL=beq)(PROGRAM=/u01/app/11.2.0.4/grid/bin/oracle)(ARGV0=oracle+ASM1)(ENVS='ORACLE_HOME=/u01/app/11.2.0.4/grid,ORACLE_SID=+ASM1')(ARGS='(DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))'))(CONNECT_DATA=(SID=+ASM1)))
2014-06-10 14:35:15.587: [ora.asm][3219122496]{0:0:2} [start] Container:start oracle home /u01/app/11.2.0.4/grid
2014-06-10 14:35:15.587: [ora.asm][3219122496]{0:0:2} [start] InstConnection::connectInt: server not attached
2014-06-10 14:35:15.621: [ora.asm][3219122496]{0:0:2} [start] ORA-01017: invalid username/password; logon denied <<<<<<<<<
2014-06-10 14:36:12.960: [ A**][2622531904]{0:0:2} A** Proxy Server sending the reply to PE for message:RESOURCE_CLEAN[ora.asm 1 1] ID 4100:456
2014-06-10 14:36:12.960: [ CRSPE][2543831360]{0:0:2} Received reply to action [Clean] message ID: 456
2014-06-10 14:36:12.960: [ CRSPE][2543831360]{0:0:2} Got agent-specific msg: ORA-01017: invalid username/password; logon denied
ASM instance can be started manually using sqlplus "/as sysasm" but the CRSD fails to start with same error.
crsd.log
~~~~~~~~~
2014-06-05 11:05:28.617: [ CRSMAIN][4003182320] Initializing OCR
[ CLWAL][4003182320]clsw_Initialize: OLR initlevel [70000]
2014-06-05 11:05:29.938: [ OCRASM][4003182320]proprasmo: Error in open/create file in dg [DATA]
[ OCRASM][4003182320]SLOS : SLOS: cat=7, opn=kgfoAl06, dep=1017, loc=kgfokge
2014-06-05 11:05:29.938: [ OCRASM][4003182320]ASM Error Stack : ORA-01017: invalid username/password; logon denied <<<<<<<<
2014-06-05 11:05:30.985: [ OCRASM][4003182320]proprasmo: kgfoCheckMount returned [7]
2014-06-05 11:05:30.985: [ OCRASM][4003182320]proprasmo: The ASM instance is down
2014-06-05 11:05:30.985: [ OCRRAW][4003182320]proprioo: Failed to open [+DATA]. Returned proprasmo() with [26]. Marking location as UNAVAILABLE.
2014-06-05 11:05:30.985: [ OCRRAW][4003182320]proprioo: No OCR/OLR devices are usable
Cause
SQLNET.ORA has below entry
# sqlnet.ora.Network Configuration File: /u01/app/11.2.0.4/grid/network/admin/sqlnet.ora
# Generated by Oracle configuration tools.
NAMES.DIRECTORY_PATH= (TNSNAMES, EZCONNECT)
SQLNET.AUTHENTICATION_SERVICES=none <<<<<<<<<<<<<<<<<<<<
ADR_BASE = /u01/app/grid
CRS uses OS authentication method to connect ASM instance.
If SQLNET.AUTHENTICATION_SERVICES=none then a valid username and password is need to connect to ASM instance.
Solution
1) Remove "SQLNET.AUTHENTICATION_SERVICES=none" from SQLNET.ORA
2) Restart the CRS with force.
crsctl stop crs -f
crsctl satrt crs