Creating Pages Fails with Large Number of Existing Pages (Doc ID 1547498.1)

Last updated on OCTOBER 02, 2023

Applies to:

Symptoms

There are stuck threads reported in custom WebCenter Portal when trying to create a new page. The page creation fails.
The following is seen in the diagnostic.log:

####<Apr 4, 2013 4:54:19 AM CDT> <Notice> <Diagnostics> <HOSTNAME> <wc-portlet_s_1> <[STANDBY] ExecuteThread: '4' for queue: 'weblogic.kernel.Default (self-tuning)'> <> <> <> <1365069259739> <BEA-320068> <Watch 'StuckThread' with severity 'Notice' on server '<HOSTNAME>' has triggered at Apr 4, 2013 4:54:19 AM CDT. Notification details:
WatchRuleType: Log
WatchRule: (SEVERITY = 'Error') AND ((MSGID = '<MSGID>') OR (MSGID = '<MSGID>'))
WatchData: DATE = Apr 4, 2013 4:54:19 AM CDT SERVER = <SERVERNAME> MESSAGE = [STUCK] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "652" seconds working on the request "weblogic.servlet.internal.ServletRequestImpl@25efc568[
POST /<HOSTNAME>/faces/oracle/webcenter/portalwebapp/pages/admin.jspx?_adf.ctrl-state=ve92tdo2m_13 HTTP/1.1
User-Agent: Mozilla/5.0 (Windows NT 5.2; rv:7.0.1) Gecko/20100101 Firefox/7.0.1
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-us,en;q=0.5
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Content-Type: application/x-www-form-urlencoded; charset=UTF-8
Adf-Ads-Page-Id: 1
Adf-Rich-Message: true
Referer: http://<HOSTNAME>:<PORT>/mycompany/faces/oracle/webcenter/portalwebapp/pages/admin.jspx?_afrLoop=71013774861000&_afrWindowMode=0&_afrWindowId=ve92tdo2m_1
Content-Length: 590
Cookie: regionId=bae:rff; baeSessionId=<SESSION_ID>; ACE-COOKIE=R411675158; JSESSIONID=<JSESSION_ID>
Pragma: no-cache
Cache-Control: no-cache
ECID-Context: <ECID>
ClientIP: <IP ADD>
Chronos: aggregate
Oracle-WebCache-Hdr-Inv-Capab: on, version="1"
SSL-Https: off
Calypso-Control: H_Req,173949988, 7777, <ip>
Surrogate-Capability: orcl="webcache/1.0 Surrogate/1.0 ESI/1.0 ESI-Inline/1.0 ESI-INV/1.0 ORAESI/9.0.4 OWV/1.0 POST-Restore/1.0"
Connection: Keep-Alive
Proxy-Client-IP: <IP ADD>
X-Forwarded-For: <IP ADD>
X-WebLogic-KeepAliveSecs: 25
X-WebLogic-Request-ClusterInfo: true
x-weblogic-cluster-hash: OyPz999oIJPZG1ubbDSuZLK+ajg

]", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
Thread-73 "[STUCK] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'" <alive, in native, suspended, priority=1, DAEMON> {
    jrockit.net.SocketNativeIO.readBytesPinned(SocketNativeIO.java:???)
    jrockit.net.SocketNativeIO.socketRead(SocketNativeIO.java:24)
    java.net.SocketInputStream.socketRead0(SocketInputStream.java:???)
    java.net.SocketInputStream.read(SocketInputStream.java:107)
    oracle.net.nt.MetricsEnabledInputStream.read(TcpNTAdapter.java:707)
    oracle.net.ns.Packet.receive(Packet.java:243)
    oracle.net.ns.DataPacket.receive(DataPacket.java:106)
    oracle.net.ns.NetInputStream.getNextPacket(NetInputStream.java:309)
    oracle.net.ns.NetInputStream.read(NetInputStream.java:254)
    oracle.net.ns.NetInputStream.read(NetInputStream.java:187)
    oracle.net.ns.NetInputStream.read(NetInputStream.java:92)
    oracle.jdbc.driver.T4CSocketInputStreamWrapper.readNextPacket(T4CSocketInputStreamWrapper.java:122)
    oracle.jdbc.driver.T4CSocketInputStreamWrapper.read(T4CSocketInputStreamWrapper.java:77)
    oracle.jdbc.driver.T4CMAREngine.unmarshalUB1(T4CMAREngine.java:1172)
    oracle.jdbc.driver.T4CMAREngine.unmarshalSB1(T4CMAREngine.java:1153)
    oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:303)
    oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:181)
    oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:382)
    oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:70)
    oracle.jdbc.driver.T4CPreparedStatement.executeForDescribe(T4CPreparedStatement.java:911)
    oracle.jdbc.driver.OracleStatement.executeMaybeDescribe(OracleStatement.java:1226)
    oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1330)
    oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3722)
    oracle.jdbc.driver.OraclePreparedStatement.executeQuery(OraclePreparedStatement.java:3789)
    ^-- Holding lock: oracle.jdbc.driver.T4CConnection@210b49fb[thin lock]
    oracle.jdbc.driver.OraclePreparedStatementWrapper.executeQuery(OraclePreparedStatementWrapper.java:1667)
    weblogic.jdbc.wrapper.PreparedStatement.executeQuery(PreparedStatement.java:123)
    oracle.mds.internal.persistence.db.shredded.DBMetadataStoreQuery.execute(DBMetadataStoreQuery.java:596)
    oracle.mds.internal.persistence.MetadataStoreQuery.execute(MetadataStoreQuery.java:169)
    oracle.mds.internal.persistence.db.shredded.ShreddedDBMSConnection.executeQuery(ShreddedDBMSConnection.java:216)
    oracle.mds.persistence.DelegatingMetadataStoreConnection.executeQuery(DelegatingMetadataStoreConnection.java:139)
    oracle.mds.internal.query.QueryImpl.listResults(QueryImpl.java:964)
    oracle.mds.internal.query.QueryImpl.listElements(QueryImpl.java:810)
    oracle.mds.internal.query.ElementQueryImpl.execute(ElementQueryImpl.java:162)
    oracle.webcenter.page.internal.model.PageServiceImpl.searchChildPages(PageServiceImpl.java:1327)
    oracle.webcenter.pagemanager.view.PageManagerBean.create(PageManagerBean.java:513)
    sun.reflect.NativeMethodAccessorImpl.invoke0(NativeMethodAccessorImpl.java:???)
    sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:27)
...
    weblogic.work.ExecuteThread.execute(ExecuteThread.java:203)
    weblogic.work.ExecuteThread.run(ExecuteThread.java:170)
}

 SUBSYSTEM = WebLogicServer USERID = <WLS Kernel> SEVERITY = Error THREAD = [ACTIVE] ExecuteThread: '6' for queue: 'weblogic.kernel.Default (self-tuning)' MSGID = BEA-000337 MACHINE = <HOSTNAME> TXID =  CONTEXTID =  TIMESTAMP = 1365069259737
WatchAlarmType: AutomaticReset
WatchAlarmResetPeriod: 600000
>
####<Apr 4, 2013 4:54:19 AM CDT> <Alert> <Diagnostics> <HOSTNAME> <SERVERNAME> <oracle.dfw.impl.incident.DiagnosticsDataExtractorImpl - Incident Dump Executor (created: Thu Apr 04 04:54:19 CDT 2013)> <> <> <> <1365069259971> <BEA-320016> <Creating diagnostic image in /<DIRECTORY>/admin/dmp996_prod/mserver/dmp996_prod_stg/servers/wc-portlet_s_1/adr/diag/ofm/dmp996_prod_stg/wc-portlet_s_1/incident/incdir_1521 with a lockout minute period of 1.> 

In another environment where this problem occurs the following exception is logged to the diagnostic.log:

[2013-03-07T16:03:49.202-06:00] [wc-portlet_s_2] [NOTIFICATION] [] [oracle.webcenter.pagemanager.view.PageManagerBean] [tid: [ACTIVE].ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'] [userId: <USERID>] [ecid: <ECID>] [APP: BAE_SYSTEMS_app#1.1.6] [URI: /mycompany/faces/oracle/webcenter/portalwebapp/pages/admin.jspx] [[
oracle.webcenter.page.model.LockUnavailableException
        at oracle.webcenter.page.internal.model.PageServiceImpl.createHierarchyPage(PageServiceImpl.java:3385)
        at oracle.webcenter.pagemanager.view.PageManagerBean.create(PageManagerBean.java:595)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at com.sun.el.parser.AstValue.invoke(Unknown Source)
        at com.sun.el.MethodExpressionImpl.invoke(Unknown Source)
        at org.apache.myfaces.trinidad.component.UIXComponentBase.broadcastToMethodExpression(UIXComponentBase.java:1300)
        at oracle.adf.view.rich.component.UIXDialog.broadcast(UIXDialog.java:97)
        at oracle.adf.view.rich.component.fragment.UIXRegion.broadcast(UIXRegion.java:148)
        at oracle.adf.view.rich.component.fragment.UIXRegion.broadcast(UIXRegion.java:148)
        at oracle.adf.view.rich.component.fragment.ContextSwitchingComponent$1.run(ContextSwitchingComponent.java:93)
        at oracle.adf.view.rich.component.fragment.ContextSwitchingComponent._processPhase(ContextSwitchingComponent.java:361)
        at oracle.adf.view.rich.component.fragment.ContextSwitchingComponent.broadcast(ContextSwitchingComponent.java:96)
        at oracle.adf.view.rich.component.fragment.UIXInclude.broadcast(UIXInclude.java:102)
        at oracle.adf.view.rich.component.fragment.ContextSwitchingComponent$1.run(ContextSwitchingComponent.java:93)
        at oracle.adf.view.rich.component.fragment.ContextSwitchingComponent._processPhase(ContextSwitchingComponent.java:361)
        at oracle.adf.view.rich.component.fragment.ContextSwitchingComponent.broadcast(ContextSwitchingComponent.java:96)
        at oracle.adf.view.rich.component.fragment.UIXInclude.broadcast(UIXInclude.java:96)
        at oracle.adfinternal.view.faces.lifecycle.LifecycleImpl.broadcastEvents(LifecycleImpl.java:902)
        at oracle.adfinternal.view.faces.lifecycle.LifecycleImpl._executePhase(LifecycleImpl.java:313)
        at oracle.adfinternal.view.faces.lifecycle.LifecycleImpl.execute(LifecycleImpl.java:186)
        at com.mycompany.portal.intercept.BAEInterceptingLifecycle.execute(BAEInterceptingLifecycle.java:105)
        at javax.faces.webapp.FacesServlet.service(FacesServlet.java:265)
        at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:227)
        at weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:125)
        at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:300)
        at weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:27)
        at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:57)
        at oracle.portlet.client.adapter.adf.ADFPortletFilter.doFilter(ADFPortletFilter.java:34)
        at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:57)
        at oracle.webcenter.framework.events.dispatcher.EventDispatcherFilter.doFilter(EventDispatcherFilter.java:45)
        at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:57)
        at oracle.adf.model.servlet.ADFBindingFilter.doFilter(ADFBindingFilter.java:205)
        at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:57)
        at oracle.adfinternal.view.faces.webapp.rich.RegistrationFilter.doFilter(RegistrationFilter.java:106)
        at org.apache.myfaces.trinidadinternal.webapp.TrinidadFilterImpl$FilterListChain.doFilter(TrinidadFilterImpl.java:447)
        at oracle.adfinternal.view.faces.activedata.AdsFilter.doFilter(AdsFilter.java:60)
        at org.apache.myfaces.trinidadinternal.webapp.TrinidadFilterImpl$FilterListChain.doFilter(TrinidadFilterImpl.java:447)
        at org.apache.myfaces.trinidadinternal.webapp.TrinidadFilterImpl._doFilterImpl(TrinidadFilterImpl.java:271)
        at org.apache.myfaces.trinidadinternal.webapp.TrinidadFilterImpl.doFilter(TrinidadFilterImpl.java:177)
        at org.apache.myfaces.trinidad.webapp.TrinidadFilter.doFilter(TrinidadFilter.java:92)
        at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:57)
        at com.mycompany.libs.filter.SessionExpiredFilter.doFilter(SessionExpiredFilter.java:62)
        at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:57)
        at com.mycompany.libs.filter.RegionURLFilter.doFilter(RegionURLFilter.java:113)
        at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:57)
        at oracle.wcps.client.PersonalizationFilter.doFilter(PersonalizationFilter.java:75)
        at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:57)
        at oracle.webcenter.content.integration.servlets.ContentServletFilter.doFilter(ContentServletFilter.java:168)
        at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:57)
        at com.mycompany.libs.filter.LoopbackRemovalFilter.doFilter(LoopbackRemovalFilter.java:66)
        at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:57)
        at oracle.webcenter.lifecycle.filter.LifecycleLockFilter.doFilter(LifecycleLockFilter.java:151)
        at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:57)
        at oracle.adf.library.webapp.LibraryFilter.doFilter(LibraryFilter.java:176)
        at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:57)
        at oracle.security.jps.ee.http.JpsAbsFilter$1.run(JpsAbsFilter.java:111)
        at oracle.security.jps.util.JpsSubject.doAsPrivileged(JpsSubject.java:313)
     ........

In the same time the database process crashes as seen in the database alert.log:

Wed Apr 03 12:54:33 2013
Exception [type: SIGSEGV, Address not mapped to object] [ADDR:<ADDR>] [PC:0x4A05C52, qkerfex()+658] [flags: 0x0, count: 1]
Errors in file /<DIRECTORY>/diag/rdbms/fmwdb/FMWDB1/trace/<TRC_FILENAME>.trc  (incident=245653):
ORA-07445: exception encountered: core dump [qkerfex()+658] [SIGSEGV] [ADDR:<ADDR>] [PC:0x4A05C52] [Address not mapped to object] []
Incident details in: /<DIRECTORY>/diag/rdbms/fmwdb/FMWDB1/incident/incdir_245653/F<TRC_FILENAME>.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Wed Apr 03 12:54:35 2013
Dumping diagnostic data in directory=[cdmp_20130403125435], requested by (instance=1, osid=<OSID>), summary=[incident=245653].
Wed Apr 03 12:54:36 2013
Sweep [inc][245653]: completed
Sweep [inc2][245653]: completed

There are about 3000 pages already, all at the same hierarchy level, as there are no subpages at all.

Steps to Reproduce:

1. Make sure you have a lot of pages created at the same level without subpages (i.e. 3000)
2. Try to create a new page from the Admin Console
3. Notice that it fails after a long time.

Changes

Cause