OAM 11g: SSO Login Hangs / Produces Timeout Error With Stuck Thread / auth_cred_submit (Doc ID 1377415.1)

Last updated on MARCH 08, 2017

Applies to:

Oracle Access Manager - Version 11.1.1.3.0 and later
Information in this document applies to any platform.

Symptoms


Intermittent Oracle Access Manager (OAM) SSO login failure / WebLogic stuck thread issue in OAM 11g environment.

After OAM 11g has been running for some time (could be half an hour or 4 hours, no regularity, no pattern), after submitting credentials in the OAM login page the browser just works...... then eventually a timeout error is displayed when the WebLogic Server reaches Stuck Thread Max Time and gives up.

If 'kill -3 <pid>' is run on the OAM managed server Process ID (PID) then the OAMMgdServerName.log file shows that OAM Server is waiting for response from the LDAP Server.

Example log entries:

<Aug 11, 2011 8:25:16 PM SGT> <Error> <WebLogicServer> <BEA-000337> <[STUCK] ExecuteThread: '1' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "235" seconds working on the request "weblogic.servlet.internal.ServletRequestImpl@4f8e0de7[
POST /oam/server/auth_cred_submit HTTP/1.1
Accept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, application/x-shockwave-flash, application/vnd.ms-excel, application/vnd.ms-powerpoint, application/msword, application/x-ms-application, application/x-ms-xbap, application/vnd.ms-xpsdocument, application/xaml+xml, */*
Referer: https://login.oracle.com/pages/login.jsp?request_id=-8104348353696688194
Accept-Language: en-us
Content-Type: application/x-www-form-urlencoded
UA-CPU: x86
Accept-Encoding: gzip, deflate
User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; FDM)
Content-Length: 70
Cache-Control: no-cache
Cookie: oracle.uix=0^^GMT+8:00^p; OAM_ID=CP1CVhhKzork6Ez9cjvioMIDp1Ee+a3IMAH/tP3N5EZuZzEpCDWuuZqEjOq7zUEowywm09BNO3dkj6dwjSdWsLjyrneakyKiwrUZq+Vl+fRQxCZFpACZx2aKzUSkuTwYNGPjwGSgCkfnB1VALiu8kfk/6ZsPJ6KqmlalWjFL7Cxmx1EcDNW0SO/6Dp/KYNup6NKN5OGOGcup2Z2TeSWXqJGTPkUQO/VHxmHD110EdDpRT0N4oEy7e6081vRgXBxnR8qUInQrOw4RSnuz2mEgYHuA6/YcI+7ZITjiKn+K/cGTmEyYsRsIwJ0XNmXctO8RMKzMR59Cs01F9O94yeIhzTJMgjzw4+hJdgk+GpiMY5pCotE0MhC6pqiblc1PNZjPdPz6gVGIFemXGQLcH003Xw==; OAM_REQ=VERSION_4~lYRK66UySg7hwItOQusiivsd24Q6xeIJUfDvIMj5eZxfiip5WRThIWeOnM481gPUrHmO4ibRqwzLEghLSGCS6vtkBevkpDcVwsigyy02biDQbsTui1QchRVR6OarqFT+gN583EcnkQI7NWdHUppI6D4eoMMkH5pvMTLuNsnRz8tPEZLxvVVwsqQlbf1EvbLmdCroKIyTXx4wwvnDluSD6/tpExZmJWVl6Ap5dBrBtErU1HdCjt90lJtPEfCgL5xxZHWAqnuyh+xxMExS9Ni2MutDxEhONQIaiTKmYujsouAX+9cZ+5fWe9DFREYjwW91wtGcxnIMvfXQ4hV5R2Pt5OnMDsv4NRG99MPlFyEcd7GD7Xzwiz/qADZAv7t0T7ZgCJrVl8SKfEYvPeYLt1AY+yB5V709VVlRkuB/eF+UMlrH/hyva62jj6QLhWgnN+5sHB2v5gkE5feqaoGBE6rnnDLfVEWNMvDTVD8JICMihZjvqdqWXjhGtpat/QQ7T5apaSUoAXZ9vpaIBwmvClbWf5yf2QQ8NWvPfLR7DiOtUW+/qrpZSFrUZ+2UXozDM8htYj92sHK0Smu34ios26Tdn56WXcfbP1yxHjJy2mPQ+3VcFOli0Gh895fbVXISTgOh2hxr/poAvQM+b+IcBR0O6cM+4OPCN0YekqWp0XDqgGpDETKQji3GMkd6ZUXppbzlQaIFQ7irlhxSqkrL1Gb4PQx7httixux6Op/qwaWDP37l0ik6HeNmDBcCgmgVqL65IcXDUxuZ6QtYCJB9UxvO5JKPlYca1jspg+AAw4rIjdAvzdxPVdQwv1bGBxPWync5gVM1baaHyFoeSY+1UxTDSu9ry8UdGcf+VPadtiDA2aWjnho95IhxIcwQcV3ciLEBGLptRld1P7pLVM+REFNGDXa/3N7sTfhXeIXpYSP/me8w9xcStbchVA2W26kYFzptdYIvUNolCrBa6yMs28hTJmB3msRHR1F/2NLcmA0/25oyGDv79v4wrvx8glpAQWwtYKoo4PNwQpeyUphFZdHCfgGOrqyGC/VnkGMf5bJOD6vgdyVaETVDklFoUdrBj4JHwnPDR/7eXIMRU7BVZvXK84DPTKE7Td58Er7J9sUZLjP2BWS9dHNkc6dclmV+BUSfUItitv3G1EdLFJDnH46yG3YrzYznn68M0aqj/ywmYOMNxnEd1VjABaGDyET4RBeNvEdXC+Bo3T5k2d6JG7cYEcX/lj1tgTg06DlIPySrtauo6r5Zruhu+CJXVCLmrSlPgOgKAgv5izw2tekzyiAYJZPTt+PezK2yRwn3R4XvW56LRRQYE2N/Y4qmqVY+N57M81fmeHlU1JL2LyUfEsF8USYz9BfI1ez2HJ6vzk3F3nJj82o9Nq900DC5t1WLN0fLwxwuW9fhIbmWGvNMiwWN6vzPhQL9OYFJNFf7S85SBY72u40m1acdkHPrJXvEPTeR; ObSSOCookie=4ZhI%2BOPw8F0fjLV8eB2JQv8nsq0QbShiMP7Qk54FOYFop3b3VyUTGt%2BALvoVO0k73tykGEHkPZxMRT8Wua%2FG%2B%2FRIi%2B9ylpQiflPJqc22yHgwPFVU6AhBStCYyYsMUfTKivtXAALQYe8PsEzBq14yb10S3ZryXMh7RCoKrHn7HVWVMhaR%2FS1GWZX%2B4jZDLaFkcrksc0fRl6wYsNR%2FsNInbVbGkFEYEdWkeIOgmhHThEQgq4URIB1YKXvZLK6mnCJm
ECID-Context: 1.004eZbLcZKj1vXs5ojL6iY0000_P0000AD;kXjE1ZDLIPJKjFPOhVBS_IUT_IVBXLSTcVNOmLOPVJTTYTPQoG
Connection: Keep-Alive
X-WebLogic-KeepAliveSecs: 30
X-WebLogic-Request-ClusterInfo: true
x-weblogic-cluster-hash: Pz9a7H9UWYj0Y9fI+z4P5jVPZlM

]", which is more than the configured time (StuckThreadMaxTime) of "120" seconds. Stack trace:
java.lang.Object.wait(Native Method)
com.sun.jndi.ldap.Connection.readReply(Connection.java:434)
com.sun.jndi.ldap.LdapClient.getSearchReply(LdapClient.java:611)
com.sun.jndi.ldap.LdapClient.search(LdapClient.java:534)
com.sun.jndi.ldap.LdapCtx.doSearch(LdapCtx.java:1962)
com.sun.jndi.ldap.LdapCtx.searchAux(LdapCtx.java:1824)
com.sun.jndi.ldap.LdapCtx.c_search(LdapCtx.java:1749)
com.sun.jndi.toolkit.ctx.ComponentDirContext.p_search(ComponentDirContext.java:368)
com.sun.jndi.toolkit.ctx.PartialCompositeDirContext.search(PartialCompositeDirContext.java:338)
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:597)
oracle.security.am.common.jndi.ldap.proxy.DirContextProxy.invoke(DirContextProxy.java:186)
$Proxy138.search(Unknown Source)
javax.naming.directory.InitialDirContext.search(InitialDirContext.java:257)
oracle.security.idm.providers.stdldap.util.DirectSearchResponse.search(DirectSearchResponse.java:351)
oracle.security.idm.providers.stdldap.util.DirectSearchResponse.getNextPage(DirectSearchResponse.java:335)
oracle.security.idm.providers.stdldap.util.DirectSearchResponse.initSearch(DirectSearchResponse.java:132)
oracle.security.idm.providers.stdldap.util.NonPagedSearchResponse.<init>(NonPagedSearchResponse.java:52)
oracle.security.idm.providers.stdldap.util.LDAPRealm.searchUsers(LDAPRealm.java:428)
oracle.security.idm.providers.stdldap.LDIdentityStore.searchUser(LDIdentityStore.java:389)
oracle.security.idm.providers.stdldap.LDIdentityStore.searchUser(LDIdentityStore.java:438)
oracle.security.am.engines.common.identity.provider.impl.IdentityProviderImpl.authenticateUserByName(IdentityProviderImpl.java:1820)
oracle.security.am.engines.common.identity.provider.impl.IdentityProviderDelegate.authenticateUserByName(IdentityProviderDelegate.java:103)
oracle.security.am.engines.common.identity.provider.OracleUserIdentityProvider.authenticateUserByName(OracleUserIdentityProvider.java:81)
oracle.security.am.engine.authn.internal.executor.AuthenticationModuleExecutor.execute(AuthenticationModuleExecutor.java:129)
oracle.security.am.engine.authn.internal.executor.AuthenticationSchemeExecutor.execute(AuthenticationSchemeExecutor.java:94)
oracle.security.am.engine.authn.internal.controller.AuthenticationEngineControllerImpl.validateUser(AuthenticationEngineControllerImpl.java:271)
oracle.security.am.engines.enginecontroller.AuthnEngineController.processEvent(AuthnEngineController.java:301)
oracle.security.am.controller.MasterController.processEvent(MasterController.java:331)
oracle.security.am.controller.MasterController.processRequest(MasterController.java:491)
oracle.security.am.controller.MasterController.process(MasterController.java:433)
oracle.security.am.pbl.PBLFlowManager.delegateToMasterController(PBLFlowManager.java:205)
oracle.security.am.pbl.PBLFlowManager.handleBaseEvent(PBLFlowManager.java:145)
oracle.security.am.pbl.PBLFlowManager.processRequest(PBLFlowManager.java:105)
oracle.security.am.pbl.transport.http.AMServlet.doPost(AMServlet.java:133)
javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:227)
weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:125)
weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:300)
weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:26)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
oracle.dms.wls.DMSServletFilter.doFilter(DMSServletFilter.java:330)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.doIt(WebAppServletContext.java:3684)
weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3650)
weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
weblogic.security.service.SecurityManager.runAs(SecurityManager.java:121)
weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2268)
weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2174)
weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1446)
weblogic.work.ExecuteThread.execute(ExecuteThread.java:201)
weblogic.work.ExecuteThread.run(ExecuteThread.java:173)




At the server side the thread is stuck. Nothing is printed out in log file until the WebLogic Server StuckThreadMaxTime exceeds. Weblogic server self-tuning thread will terminate that stuck thread and print out its information.

After the stuck thread is terminated, the OAM Server starts serving OAM login requests again. Users can login again.

Steps to reproduce

1. Start OAM and WebGate webserver, everything works fine.

After some time e.g. 30 minutes, 4 hours, no regularity, no pattern, users start getting timeout errors when performing SSO login:

2. At the client side, user submits user id and password in the OAM SSO login page
3. The browser keeps loading and loading, eventually a timeout page is displayed. This behaviour continues for a while even if browser is closed and a new one opened.
4. When the WebLogic Server StuckThreadMaxTime exceeds, Weblogic server self-tuning thread terminates the stuck thread
5. After the stuck thread is terminated, the OAM Server starts serving OAM login requests again. Users can login again.

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms