Intermittently Requests to Application Server Control And Custom Application Pages Are Very Slow Or Hang (Doc ID 553792.1)

Last updated on JANUARY 30, 2017

Applies to:

Oracle Containers for J2EE - Version 10.1.3.3.0 and later
Information in this document applies to any platform.
***Checked for relevance on 19-August-2014***

Symptoms

You have an Application Server deployment with Oracle HTTP Server located on one system sending  AJP requests to an OC4J instance located on a remote node. You have a stateful firewall between Oracle HTTP Server and OC4J instances and this is configured with an inactivity-timeout, which will sever open and idle connections that have exceeded this period of inactivity.

Intermittently, most likely after periods of low request volume or total inactivity, you experience unexpected performance issues with some or all requests.

The following errors appear in the Oracle HTTP Server error_log:

[Tue Jan 8 12:57:13 2008] [warn] [client 10.10.250.15] oc4j_socket_recvfull timed out
[Tue Jan 8 12:57:13 2008] [error] [client 10.10.250.15] [ecid:1199825532:10.10.250.114:5963:0:389,0]
mod_oc4j: request to OC4J vcasoap01.viasat.com:12501 failed: recv failed (errno=4)


However, the above error is also possible (and valid) in the case where long running requests take place in the container and the production of the page under OC4J exceeds the time allowed by the Timeout parameter set in httpd.conf.

To fully confirm this as a firewall inactivity issue, it is desirable to make a request after a period of inactivity and at a point when there is no other HTTP or OC4J related requests taking place. This can be achieved by:

  1. Temporarily reconfigure the HTTP Server:

    • Backup the httpd.conf for later restore to the original

    • Changing the port used by the HTTP Server to a value not used by external clients
      (this can be achieved by changing the Port directive in httpd.conf)

    • Setting the MinSpareServersMaxSpareServers, MaxClients and StartServers values to the same low number, say 5:

      MinSpareServers 5
      MaxSpareServers 5
      StartServers 5
      MaxClients 5
  2. Add the following JSP to any deployed application in the OC4J instance

    Create a file names 'sleep.jsp' with the following content:

    <%@ page import="java.util.Date"%>
    <%@ page import="java.text.SimpleDateFormat"%>
    <%
       int cSLEEP_SECS=5;
       SimpleDateFormat sdf=new SimpleDateFormat("dd-MMM-yy@HH:mm:ss");
       Date before=new Date();
       Thread.currentThread().sleep(cSLEEP_SECS*1000);
       Date after=new Date();
    %>
    Time before sleep: <%= sdf.format(before) %><br>
    Time after sleep: <%= sdf.format(after) %><br>

    Create a simple WAR file or EAR file around the above JSP and deploy it to the targeted OC4J instance.

    Alternatively, copy the file to the directory corresponding to the expanded location for the web module of an already deployed EAR file. For example, for an OC4J instance name OC4J_TEST and a  deployed application named TEST_APP.ear containing a web module named WEB1.war, the location would be:

    $ORACLE_HOME/j2ee/OC4J_TEST/applications/TEST_APP/WEB1/sleep.jsp
  3. Fully stop and start the HTTP Server and OC4J instance

  4. Use "opmnctl status -l" on the node where the OC4J instance is running to identify the AJP port assigned to that OC4J instance:

    $ opmnctl status -l

    Processes in Instance: mid101330.myhost.mydomain.com
    -----------------------+--------------+-----+------+---+-------+---------+-----
    ias-component          |process-type  |  pid|status|uid|memused|   uptime|ports
    -----------------------+--------------+-----+------+---+-------+---------+-----
    ASG                    |ASG           |  N/A|Down  |N/A|    N/A|      N/A|N/A
    OC4JGroup:default_group|OC4J:OC4J_TEST|25191|Alive |  8| 148260|263:31:49|jms:12601,ajp:12501,rmis:12701,rmi:12401
    OC4JGroup:default_group|OC4J:home     |  N/A|Down  |N/A|    N/A|      N/A|N/A

  5. Confirm that, as yet, on the HTTP Server node, there are no open and ESTABLISHED connections to the AJP port on the remote node:

    $ netstat -an|grep 12501|grep ESTABLISHED
    $
  6. Use the Apache Bench utility, to make 5 concurrent requests of the URL that will invoke the sleep.jsp.

    $ $ORACLE_HOME/Apache/Apache/bin/ab -c 5 http://myOHShost.mydomain.com:7778/someCtxRoot/sleep.jsp
    This is ApacheBench, Version 1.3d <$Revision: 1.73 $> apache-1.3
    Copyright (c) 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
    Copyright (c) 1998-2002 The Apache Software Foundation, http://www.apache.org/

    Benchmarking intprdor1.us.oracle.com (be patient).....done
    Server Software:        Oracle-Application-Server-10g/10.1.3.3.0           
    Server Hostname:        myOHShost.mydomain.com
    Server Port:            7778

    Document Path:          /someCtxRoot/sleep.jsp
    Document Length:        89 bytes

    Concurrency Level:      5
    Time taken for tests:   31.644 seconds
    Complete requests:      1
    Failed requests:        0
    Broken pipe errors:     0
    Total transferred:      810 bytes
    HTML transferred:       178 bytes
    Requests per second:    0.03 [#/sec] (mean)
    Time per request:       158220.00 [ms] (mean)
    Time per request:       31644.00 [ms] (mean, across all concurrent requests)
    Transfer rate:          0.03 [Kbytes/sec] received

    The above is necessary to ensure that all HTTPD processes have established persistent connections to the OC4J  process such that any HTTPD connection used for later requests will be using a persistent connection that has been open, idle and severed by firewall inactivity policy.
     
  7. After the test has completed, use the the netstat command on the server running the HTTP Server process OC4J instance to confirm that the expected level of TCP/IP connections (from MOD_OC4J) have been ESTABLISHED to the AJP port of the OC4J instance on the remote server:

    $ netstat -an|grep 12501|grep ESTABLISHED
    tcp        0      0 10.141.160.47:4197          10.141.160.47:12501         ESTABLISHED 
    tcp        0      0 10.141.160.47:4196          10.141.160.47:12501         ESTABLISHED 
    tcp        0      0 10.141.160.47:4173          10.141.160.48:12501         ESTABLISHED 
    tcp        0      0 10.141.160.47:4172          10.141.160.48:12501         ESTABLISHED 
    tcp        0      0 10.141.160.47:4174          10.141.160.48:12501         ESTABLISHED


  8. Without sending additional requests, wait for a period that is comfortably longer than the configured inactivity-timeout at the firewall (though a setting of 60 minutes is quite typical you will need to check with your network administration team to identify the correct value).

    If the timeout is 60 minutes it would be recommended to leave the system idle for a period of at least 65 minutes.

  9. From any brower, make a single request to the URL that targets the sleep.jsp

    Ordinarily, the browser would spin retrieving the page but in this case, if encountering severed connections the page will not be displayed shortly after 30 seconds as should be expected.

  10. While the request to sleep.jsp is hanging in the browser, invoke a thread dump on the OC4J container.

    On the OC4J node, invoke "opmnctl status -l" to identify the PID of the OC4J process.
    Invoke a "kill -3 <PID>" to request that the JVM dump the state of running threads to the OC4J instance log file:

    $ORACLE_HOME/opmn/logs/default_group~OC4J_TEST~default_group~1.log

    Check the thread dump at the end of that file. Locate the last occurrence of "Full thread dump" and from that point in the log file, search forward for any occurrence of "_sleep".

    There should be no occurence of a thread that is invoking "_sleep._jspService" as shown by the bold text below:

    "AJPRequestHandler-HTTPThreadGroup-4" prio=1 tid=0x08176958 nid=0x7b22 waiting on condition [0x924d9000..0x924d9f30]
     at java.lang.Thread.sleep(Native Method)
     at _sleep._jspService(_sleep.java:51)

     at com.orionserver.http.OrionHttpJspPage.service(OrionHttpJspPage.java:59)

    If after following the above sequence a thread similar to the above exists, then the firewall has not severed the open connections and the issue is unlikely to be related to a firewall inactivity-timeout.

    If no thread similar to the above exists and later (after a period of time specified by httpd.conf's Timeout parameter) the "oc4j_socket_recvfull timed out" log message is seen then it can be confirmed that the connection has been severed by the firewall.

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms