EM 13c : emctl start oms Getting Hung 'ORA-27072 File I/O error ORA-19502 write error on file' Reported in alert.log

(Doc ID 2230409.1)

Last updated on FEBRUARY 25, 2017

Applies to:

Enterprise Manager Base Platform - Version 13.2.0.0.0 and later
Information in this document applies to any platform.

Symptoms

emctl start oms  is getting hung  and 'ORA-27072 File I/O error ORA-19502 write error on file' Reported in alert.log


$/oms/middleware13c/oms/bin/emctl start oms

Oracle Enterprise Manager Cloud Control 13c Release 2
Copyright (c) 1996, 2016 Oracle Corporation. All rights reserved.
Starting Oracle Management Server...


emctl.log shows admin server status is FAILED_NOT_RESTARTABLE.
Connection to repository database was getting hung .

From /gc_inst/em/EMGC_OMS1/sysman/log/emctl..log

2017-02-03 14:29:17,666 [main] INFO ctrl_extn.EmctlCtrlExtnLoader logp.251 - rsp is 0 message is JVMD Engine is Up
2017-02-03 14:29:17,666 [main] INFO commands.BaseCommand printMessage.426 - extensible_sample rsp is 0 message is JVMD Engine is Up
2017-02-03 14:29:17,666 [main] INFO commands.BaseCommand logAndPrint.653 - JVMD Engine is Up
2017-02-03 14:29:17,709 [main] INFO commands.BaseCommand logAndPrint.653 - BI Publisher Server is Up
2017-02-03 14:34:12,780 [main] INFO wls.OMSController main.220 - Executing emctl command : status
2017-02-03 14:34:13,108 [main] INFO commands.BaseCommand logAndPrint.653 - Oracle Management Server is Down
2017-02-03 14:34:13,159 [main] INFO commands.BaseCommand printMessage.413 - statusOMS finished with result: 8
.........
2017-02-03 15:45:13,745 [main] INFO wls.OMSController main.220 - Executing emctl command : stop
..
2017-02-03 15:45:53,866 [main] INFO wls.OMSController main.220 - Executing emctl command : start
..
2017-02-03 15:53:38,317 [main] INFO util.EmctlUtil logp.251 - Unable to get mbean conn over t3s :t3s://omshostname.oracle.com:7102 Destination x.x.x.x, 7102 unreachable; nested exception is:
java.net.ConnectException: Connection refused; No available router to destination
2017-02-03 15:53:38,317 [main] INFO util.EmctlUtil logp.251 - Unable to connect to AdminServer. Ignoring the exception: t3s://omshostname.oracle.com:7102 Destination x.x.x.x, 7102 unreachable; nested exception is:
java.net.ConnectException: Connection refused; No available router to destination
2017-02-03 15:54:16,063 [main] INFO wls.OMSController main.220 - Executing emctl command : status
...
2017-02-03 16:01:04,522 [main] INFO wls.OMSController main.220 - Executing emctl command : stop
..
2017-02-04 08:47:46,768 [main] INFO wls.OMSController main.220 - Executing emctl command : start
..
2017-02-04 08:48:03,779 [Thread-1] INFO commands.BaseCommand run.605 - current status of EMGC_ADMINSERVER:
2017-02-04 08:48:03,781 [Thread-1] INFO commands.BaseCommand run.605 -
2017-02-04 08:48:03,782 [Thread-1] INFO commands.BaseCommand run.605 - FAILED_NOT_RESTARTABLE
.....
2017-02-04 08:48:03,779 [Thread-1] INFO commands.BaseCommand run.605 - current status of EMGC_ADMINSERVER:
2017-02-04 08:48:03,781 [Thread-1] INFO commands.BaseCommand run.605 -
2017-02-04 08:48:03,782 [Thread-1] INFO commands.BaseCommand run.605 - FAILED_NOT_RESTARTABLE
2017-02-04 08:48:03,782 [Thread-1] INFO commands.BaseCommand run.605 -
2017-02-04 08:48:03,782 [Thread-1] INFO commands.BaseCommand run.605 - Starting server EMGC_ADMINSERVER
2017-02-04 08:48:03,782 [Thread-1] INFO commands.BaseCommand run.605 - Starting server EMGC_ADMINSERVER ...

From /gc_inst/userprojects/Domain/GCDomain/servers/EMGC_ADMINSERVER/logs/EMGC_ADMINSERVER.out

Feb 04, 2017 8:48:08 AM oracle.security.jps.JpsStartup start
INFO: Jps initializing.
Feb 04, 2017 8:48:11 AM org.hibernate.validator.util.Version
INFO: Hibernate Validator 12.1.3.0.0
Feb 04, 2017 8:48:11 AM org.hibernate.validator.engine.resolver.DefaultTraversableResolver detectJPA
INFO: Instantiated an instance of org.hibernate.validator.engine.resolver.JPATraversableResolver.

From repository Database alert.log

Sat Feb 04 12:45:06 2017
Errors in file /oracle/app/oracle/diag/rdbms/oem13ctest/oem13ctest/trace/oem13ctest_arc2_601.trc:
ORA-27072: File I/O error
Additional information: 4
Additional information: 835584
Additional information: 577536
ORA-19502: write error on file "/oraarch/1_39_933767009.dbf", block number 835584 (block size=512)
Sat Feb 04 12:53:05 2017
ARC0: Closing local archive destination LOG_ARCHIVE_DEST_1: '/oraarch/1_39_933767009.dbf' (error 19502) (oem13ctest)
Sat Feb 04 12:53:05 2017

 

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms