My Oracle Support Banner

Odacli update-dcscomponents command hungs (Doc ID 2775277.1)

Last updated on MAY 13, 2021

Applies to:

Oracle Database Appliance Software - Version 18.5.0.0.0 Bare Metal and later
Information in this document applies to any platform.

Symptoms

You will get a hung when you run the following command:

# /opt/oracle/dcs/bin/odacli update-dcscomponents -v 18.8.0.0.0

 

You should see output similar to the following in dcs-agent.log

2021-02-15 10:46:15,055 INFO [Thread-2] [] c.o.d.a.DCSApplication: DCS Server is shutting down...
2021-02-15 10:46:15,055 INFO [Thread-2] [] c.o.d.a.z.DCSZooKeeper: DCS node id is - node_0
2021-02-15 10:46:15,061 INFO [Thread-17] [] o.e.j.s.AbstractConnector: Stopped application@7a3c2341{SSL,[ssl, http/1.1]}{0.0.0.0:7070}
2021-02-15 10:46:15,061 INFO [Thread-17] [] o.e.j.s.AbstractConnector: Stopped admin@66d1cdea{HTTP/1.1,[http/1.1]}{0.0.0.0:7071}
2021-02-15 10:46:15,062 INFO [Thread-17] [] o.e.j.s.h.ContextHandler: Stopped i.d.j.MutableServletContextHandler@4c864fd0{/,null,UNAVAILABLE}
2021-02-15 10:46:15,065 INFO [Thread-2] [] c.o.d.a.DCSApplication: Persist new Agent state as Unavailable
2021-02-15 10:46:15,065 DEBUG [Thread-2] [] c.o.d.c.u.FileUtils: deleteAllResponseFiles
2021-02-15 10:46:15,065 INFO [Thread-2] [] c.o.d.a.DCSApplication: RPC Service is stopped
2021-02-15 10:46:15,065 INFO [Thread-2] [] c.o.d.a.z.DCSZooKeeper: DCS node id is - node_0
2021-02-15 10:46:15,068 INFO [Thread-2] [] c.o.d.a.z.DCSZooKeeper: Closing ZK Session: 31197062669008899 ...
2021-02-15 10:46:15,070 DEBUG [ZK_Initialize_Task_Executor_Thread-EventThread] [] c.o.d.a.p.s.SyncService: Agent connected to cluster, check Sync service.
2021-02-15 10:46:15,070 DEBUG [ZK_Initialize_Task_Executor_Thread-EventThread] [] c.o.d.a.p.s.SyncService: Check is there a need to sync
2021-02-15 10:46:15,074 DEBUG [DCSZJsonRequestHandler] [] c.o.d.c.n.MessageUtil: load locale as en_US
2021-02-15 10:46:15,074 DEBUG [ZK_Initialize_Task_Executor_Thread-EventThread] [] c.o.d.c.n.MessageUtil: load locale as en_US
2021-02-15 10:46:15,074 ERROR [DCSZJsonRequestHandler] [] c.o.d.a.r.DCSZJsonRequestHandler: Exception while getting znode from the InQueue:
! java.lang.InterruptedException: null
! at java.lang.Object.wait(Native Method)
! at java.lang.Object.wait(Object.java:502)
! at com.oracle.dcs.agent.zk.DCSZQueue.npeek(DCSZQueue.java:93)
! ... 2 common frames omitted
! Causing: com.oracle.dcs.commons.exception.DcsException: DCS-10001:Internal error encountered: null.
! at com.oracle.dcs.commons.exception.DcsException$Builder.build(DcsException.java:68)
! at com.oracle.dcs.agent.zk.DCSZQueue.npeek(DCSZQueue.java:98)
! at com.oracle.dcs.agent.rpc.DCSZJsonRequestHandler.run(DCSZJsonRequestHandler.java:61)
! at java.lang.Thread.run(Thread.java:748)
2021-02-15 10:46:15,074 DEBUG [DCSZJsonRequestHandler] [] c.o.d.a.r.DCSZJsonRequestHandler: Sleeping for 3000 milli seconds...30
2021-02-15 10:46:15,074 ERROR [ZK_Initialize_Task_Executor_Thread-EventThread] [] c.o.d.a.p.s.SyncService: Fail to sync : DCS-10001:Internal error encountered: KeeperErrorCode = ConnectionLoss for /elections.
2021-02-15 10:46:15,093 INFO [Thread-17] [] o.e.j.s.h.ContextHandler: Stopped i.d.j.MutableServletContextHandler@68cca23e{/,null,UNAVAILABLE}
2021-02-15 10:46:15,097 INFO [Thread-17] [] c.o.d.a.s.DcsSchedulerManager: Stop scheduler
2021-02-15 10:46:15,098 INFO [Thread-17] [] o.q.c.QuartzScheduler: Scheduler DefaultQuartzScheduler_$_NON_CLUSTERED shutting down.
2021-02-15 10:46:15,098 INFO [Thread-17] [] o.q.c.QuartzScheduler: Scheduler DefaultQuartzScheduler_$_NON_CLUSTERED paused.
2021-02-15 10:46:15,173 ERROR [ZK_Initialize_Task_Executor_Thread-EventThread] [] o.a.z.ClientCnxn: Error while calling watcher
! org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /elections
! at org.apache.zookeeper.KeeperException.create(KeeperException.java:134)
! at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
! at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:2589)
! at com.oracle.dcs.agent.persistence.sync.SyncService.resetNodeWatch(SyncService.java:50)
! ... 3 common frames omitted
! Causing: com.oracle.dcs.commons.exception.DcsException: DCS-10001:Internal error encountered: resetNodeWatch.
! at com.oracle.dcs.commons.exception.DcsException$Builder.build(DcsException.java:68)
! at com.oracle.dcs.agent.persistence.sync.SyncService.resetNodeWatch(SyncService.java:53)
! at com.oracle.dcs.agent.persistence.sync.SyncService.process(SyncService.java:93)
! at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535)
! at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510)
2021-02-15 10:46:15,173 INFO [ZK_Initialize_Task_Executor_Thread-EventThread] [] c.o.d.a.z.DCSZooKeeper: Received ZK Event: WatchedEvent state:Closed type:None path:null
2021-02-15 10:46:15,173 INFO [ZK_Initialize_Task_Executor_Thread-EventThread] [] c.o.d.a.DCSApplication: DCS Aapplication notified ZK event: Closed
2021-02-15 10:46:15,173 INFO [ZK_Initialize_Task_Executor_Thread-EventThread] [] c.o.d.a.DCSApplication: Unhandled ZK Event: WatchedEvent state:Closed type:None path:null
2021-02-15 10:46:15,174 INFO [Thread-2] [] o.a.z.ZooKeeper: Session: 0x6ed5902e6c0003 closed
2021-02-15 10:46:15,174 INFO [Thread-2] [] c.o.d.a.z.DCSZooKeeper: Zk Session: 31197062669008899 closed
2021-02-15 10:46:15,174 INFO [Thread-2] [] c.o.d.a.DCSApplication: ZK Service is stopped
2021-02-15 10:46:15,174 INFO [Thread-2] [] c.o.d.a.DCSApplication: Task Service is stopped
2021-02-15 10:46:15,174 INFO [Thread-2] [] c.o.d.a.DCSApplication: Stopping dcs scheduler manager
2021-02-15 10:46:15,174 INFO [Thread-2] [] c.o.d.a.s.DcsSchedulerManager: Stop scheduler
2021-02-15 10:46:15,174 DEBUG [Thread-2] [] c.o.d.c.p.PersistenceService: trying to close the derby session with retry number - 0
2021-02-15 10:46:15,174 INFO [Thread-2] [] o.h.o.c.pooling: HHH10001008: Cleaning up connection pool [jdbc:derby:/opt/oracle/dcs/repo/node_0;create=true]
2021-02-15 10:46:15,176 INFO [ZK_Initialize_Task_Executor_Thread-EventThread] [] o.a.z.ClientCnxn: EventThread shut down for session: 0x6ed5902e6c0003
2021-02-15 10:46:15,178 DEBUG [Thread-2] [] c.o.d.c.p.PersistenceService: trying to close the derby session with retry number - 1
2021-02-15 10:46:15,178 DEBUG [Thread-2] [] c.o.d.c.p.PersistenceService: derby session is successfully closed
2021-02-15 10:46:15,178 DEBUG [Thread-2] [] c.o.d.c.p.PersistenceService: shutting down the derby
2021-02-15 10:46:15,266 DEBUG [Thread-2] [] c.o.d.c.p.PersistenceService: shutting down the derby Completed
2021-02-15 10:46:15,270 INFO [Thread-2] [] c.o.d.a.DCSApplication: Persistence Service is stopped
2021-02-15 10:46:15,277 INFO [Thread-2] [] c.o.d.a.DCSApplication: Feature tracking collector scheduler is stopped.
2021-02-15 10:46:15,281 INFO [Thread-2] [] c.o.d.a.DCSApplication: DCS Server is down
2021-02-15 10:46:15,380 INFO [Thread-17] [] o.q.c.QuartzScheduler: Scheduler DefaultQuartzScheduler_$_NON_CLUSTERED shutdown complete.

Changes

 

Cause

To view full details, sign in with your My Oracle Support account.

Don't have a My Oracle Support account? Click to get started!


In this Document
Symptoms
Changes
Cause
Solution
References


My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.