ORACLE OMS启动失败之BEA-300040 I/O error while writing node manager status

橡皮草帽

1、OMS启动异常信息[oracle@orcl bin]$ ./emctl start oms
Oracle Enterprise Manager Cloud Control 12c Release 4
Copyright (c) 1996, 2014 Oracle Corporation.  All rights reserved.
Starting Oracle Management Server...
Starting WebTier...
WebTier Successfully Started
Oracle Management Server Could Not Be Started
Check EM Server log file for details: /u01/app/oracle/oms/domains/GCDomainrvers/ORCL_OMS1/logs/ORCL_OMS1.out
Oracle Management Server is Down
[oracle@ orcl bin]$


2、查看 ORCL_OMS1.out日志
Mar 19, 2020 5:55:44 PM oracle.security.jps.internal.audit.AuditServiceImpl registerInternal
WARNING: Cannot register to audit service for component "JPS".
Mar 19, 2020 5:55:44 PM oracle.security.jps.internal.policystore.ldap.LdapPolicyStore initial
INFO: Container is not ready at the point null
<Mar 19, 2020 5:55:53 PM CST> <Error> <NodeManager> <BEA-300040> <I/O error while writing node manager status change message.
java.io.IOException: No space left on device
        at java.io.UnixFileSystem.createFileExclusively(Native Method)
        at java.io.File.checkAndCreate(File.java:1705)
        at java.io.File.createTempFile0(File.java:1726)
        at java.io.File.createTempFile(File.java:1803)
        at weblogic.nodemanager.util.ConcurrentUnixFile.write(ConcurrentUnixFile.java:38)
        Truncated. see log file for complete stacktrace
>
<Mar 19, 2020 5:55:54 PM> <FINEST> <NodeManager> <Waiting for the process to die: 6906>
<Mar 19, 2020 5:55:54 PM> <INFO> <NodeManager> <Server failed during startup so will not be restarted>
<Mar 19, 2020 5:55:54 PM> <FINEST> <NodeManager> <runMonitor returned, setting finished=true and notifying waiters>


3、根据.out日志提示，跟之前asm和db启动异常报错一致，也是之前oracle数据库审计日志过多导致文件系统inode耗尽其他进程无法写入导致。

橡皮草帽

解决办法：
1、对于oms来说，单纯的重启oms无法解决问题，因为nodemanager进程未正常停止。
2、使用ps -ef|grep java，把oms相关的java进程杀掉kill -9 java_process_number。
3、再次尝试启动oms进程，成功启动。