Managed Servers Hang When Attempting To Create Secure Administration Channel (Doc ID 2196160.1)

Last updated on DECEMBER 02, 2016

Applies to:

Oracle WebLogic Server - Version 10.3.6 and later
Information in this document applies to any platform.

Symptoms

4 managed servers in a cluster, 2 on each machine (Super cluster Zones). After abruptly shutting down one of the zones, the managed servers on the other zone cannot be started. It hangs at the following stage:

Starting service weblogic.server.channels.EnableAdminListenersService, 79 of 132.

Last service weblogic.server.ServerLifeCycleService took 0ms.>


After this, several thread dumps were collected. Thread dumps shows, two threads that are involved:

11370 "DynamicJSSEListenThread[DefaultAdministration]" daemon prio=3 tid=0x0000000107811800 nid=0x31 runnable [0xffffffff628fe000]
11371 java.lang.Thread.State: RUNNABLE
11372 at java.net.PlainSocketImpl.socketAccept(Native Method)
11373 at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:439)
11374 - locked (a java.net.SocksSocketImpl)
11375 at java.net.ServerSocket.implAccept(ServerSocket.java:468)
11376 at java.net.ServerSocket.accept(ServerSocket.java:436)
11377 at weblogic.socket.WeblogicServerSocket.accept(WeblogicServerSocket.java:38)
11378 at weblogic.server.channels.DynamicListenThread$SocketAccepter.accept(DynamicListenThread.java:535)
11379 at weblogic.server.channels.DynamicListenThread$SocketAccepter.access$200(DynamicListenThread.java:417)
11380 at weblogic.server.channels.DynamicListenThread.run(DynamicListenThread.java:173)
11381 at java.lang.Thread.run(Thread.java:682)


11485 "[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=3 tid=0x00000001066ea000 nid=0x24 in Object.wait() [0xffffffff642fe000]
11486 java.lang.Thread.State: TIMED_WAITING (on object monitor)
11487 at java.lang.Object.wait(Native Method)
11488 - waiting on (a weblogic.server.channels.DynamicJSSEListenThread)
11489 at weblogic.server.channels.DynamicListenThread.start(DynamicListenThread.java:272)
11490 - locked (a weblogic.server.channels.DynamicJSSEListenThread)
11491 at weblogic.server.channels.AdminPortService.bindListeners(AdminPortService.java:82)
11492 - locked (a weblogic.server.channels.AdminPortService)
11493 at weblogic.server.channels.EnableAdminListenersService.start(EnableAdminListenersService.java:39)
11494 at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)
11495 at weblogic.work.ExecuteThread.execute(ExecuteThread.java:252)
11496 at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)



After about 18 minutes, saw the following:

<Channel "DefaultAdministration" is now listening on 10.104.30.37:23701 for protocols admin, CLUSTER-BROADCAST-SECURE, ldaps, https.>



However, after this message, the server still struggled to connect to AdminServer and we see following exceptions in server logs:

4050 14051 java.rmi.UnknownHostException: Could not discover administration URL for server 'tt150_fpscore_admin'
14052 at weblogic.protocol.URLManager.findAdministrationURL(URLManager.java:177)
14053 at weblogic.server.ServerLifeCycleRuntime.getLifeCycleOperationsRemote(ServerLifeCycleRuntime.java:1087)
14054 at weblogic.t3.srvr.ServerRuntime.sendStateToAdminServer(ServerRuntime.java:429)
14055 at weblogic.t3.srvr.ServerRuntime.updateRunState(ServerRuntime.java:415)
14056 at weblogic.t3.srvr.T3Srvr.setState(T3Srvr.java:206)
14057 at weblogic.t3.srvr.T3Srvr.initializeStandby(T3Srvr.java:886)
14058 at weblogic.t3.srvr.T3Srvr.startup(T3Srvr.java:568)
14059 at weblogic.t3.srvr.T3Srvr.run(T3Srvr.java:469)
14060 at weblogic.Server.main(Server.java:71)
14061 >
14062
14063
14064 [Loaded weblogic.scheduler.JobSchedulerBindingService from file:/fs01/app/sw/wls10.3.6/patchset12A/wlserver_10.3/server/lib/weblogic.jar]
14065 [Loaded weblogic.server.AdminServerListenerService from file:/fs01/app/sw/wls10.3.6/patchset12A/wlserver_10.3/server/lib/weblogic.jar]
14066 [Loaded weblogic.security.utils.AdminServerListener from file:/fs01/app/sw/wls10.3.6/patchset12A/modules/com.bea.core.weblogic.security.wls_1.0.0.0_6-2-0-0.jar]
14067 14068 java.rmi.UnknownHostException: Could not discover administration URL for server 'tt150_fpscore_admin'
14069 at weblogic.protocol.URLManager.findAdministrationURL(URLManager.java:177)
14070 at weblogic.server.ServerLifeCycleRuntime.getLifeCycleOperationsRemote(ServerLifeCycleRuntime.java:1087)
14071 at weblogic.security.utils.AdminServerListener.startDisconnectListener(AdminServerListener.java:113)
14072 at weblogic.security.utils.AdminServerListener.startListening(AdminServerListener.java:100)
14073 at weblogic.security.utils.AdminServerListener.start(AdminServerListener.java:73)
14074 at weblogic.server.AdminServerListenerService.initializeAdminServerListener(AdminServerListenerService.java:31)
14075 at weblogic.server.AdminServerListenerService.start(AdminServerListenerService.java:24)
14076 at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)
14077 at weblogic.work.ExecuteThread.execute(ExecuteThread.java:252)
14078 at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)

Changes

 

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms