OHS httpd.worker Processes Consume CPU and Slows Down System During CGI Application Processing (Doc ID 1560799.1)

Last updated on MARCH 08, 2017

Applies to:

Oracle HTTP Server - Version 11.1.1.2.0 and later
Information in this document applies to any platform.

Symptoms

Using Oracle HTTP Server 11g, there are two visible symptoms:


1) Upon a spike in heavier load, httpd.worker processes consume more CPU then they should until system reaches a CPU limit. Even if load subsides without hitting the limit, some httpd.worker processes still consume more CPU than others, but apparently not processing requests. Restarting OHS is the only way to reset.

2) If system is configured to automatically allocate more CPU resources, the system slows down every once in a while in a random fashion. It is during a heavier load time and appears as if tuning is required.

The two issues are related because over time the CPU gradually increases until it reaches a limit, either way. It does not completely correct itself until a restart. This will most likely not be seen in normal runtime, it must be with a higher load to force a specific condition:

The following is seen on the OHS error log, snippets from during a slowdown and then at a core dump time:

 

[2013-05-28T18:25:13.4431+00:00] [OHS] [WARNING:32] [OHS-9999] [mod_cgi.c] [host_id: <hostname removed>] [host_addr: <ip removed>] [tid: 9] [user: root] [ecid: 00007Doy0CUALQ46RVZBEP00042P00020a] [rid: 0] [VirtualHost: <hostname removed>:443] Timeout waiting for output from CGI script /web-sites/<hostname removed>/cgi-bin/display_report.sh

[2013-05-28T18:25:21.9536+00:00] [OHS] [WARNING:32] [OHS-9999] [mod_cgi.c] [host_id: <hostname removed>] [host_addr: <ip removed>] [tid: 45] [user: root] [ecid: 00007DpUNd_ALQ46RVZBEP00042P000251] [rid: 0] [VirtualHost: <hostname removed>:443] Timeout waiting for output from CGI script /web-sites/<hostname removed>/cgi-bin/display_report.sh, referer: https://<hostname removed>.<domain removed>/cgi-bin/display_report.sh?REPORTS=/usr/tools//reports/M2CEFMP1&TITLE=CUSTOMER+ORDER+ACCEPTANCE+PRINT+SCREEN&TWO_TASK=m2cefmp1&FILE=dd448_261.pdf

[2013-05-28T18:25:50.1548+00:00] [OHS] [WARNING:32] [OHS-9999] [mod_cgi.c] [host_id: <hostname removed>] [host_addr: <ip removed>] [tid: 47] [user: root] [ecid: 00007Dr9PKtALQ46RVZBEP00042P0002Fq] [rid: 0] [VirtualHost: <hostname removed>:443] Timeout waiting for output from CGI script /web-sites/<hostname removed>/cgi-bin/display_report.sh

[2013-05-28T18:25:55.7663+00:00] [OHS] [WARNING:32] [OHS-9999] [mod_cgi.c] [host_id: <hostname removed>] [host_addr: <ip removed>] [tid: 30] [user: root] [ecid: 00007DrVQ3YALQ46RVZBEP00042P0002I9] [rid: 0] [VirtualHost: <hostname removed>:443] Timeout waiting for output from CGI script /web-sites/<hostname removed>/cgi-bin/display_report.sh

[2013-05-28T18:26:02.8246+00:00] [OHS] [WARNING:32] [OHS-9999] [mod_cgi.c] [host_id: <hostname removed>] [host_addr: <ip removed>] [tid: 24] [user: root] [ecid: 00007DrtpnwALQ46RVZBEP00042P0002L0] [rid: 0] [VirtualHost: <hostname removed>:443] Timeout waiting for output from CGI script /web-sites/<hostname removed>/cgi-bin/display_report.sh

[2013-05-28T18:28:53.6918+00:00] [OHS] [NOTIFICATION:16] [OHS-9999] [core.c] [host_id: <hostname removed>] [host_addr: <ip removed>] [pid: 16537] [tid: 1] [user: root] [VirtualHost: main]  child pid 27202 exit signal Bus error (10), possible coredump in /orasoft/product/middleware/asinst_1/config/OHS/ohs1

  

Reviewing a stack trace from an httpd.worker core file, the last seemingly working thread ends at the following:

 

----------------- lwp# 24 / thread# 24 --------------------
ffffffff7721c078 apr_pool_cleanup_kill (1006129f0, 1006129f0,
ffffffff77215d40, 18, 118818, 0) + 58
ffffffff6f403a34 cgi_handler (1009c1c70, 1001f9ed0, 10056c208, 0,
ffffffff6f504af0, 100287e00) + 774
0000000100044500 ap_run_handler (1009c1c70, ffffffffffffffff, 13c00, 2, 100295e78, 100194d58) + 80
0000000100044e5c ap_invoke_handler (1009c1c70, 1002952e0, ffffffff6e002a00, 100287e00, 100180e90, 0) + fc
000000010005a398 ap_process_request (1009c1c70, 9c9f99ae67, 9000, 126b6c, 100180e90, 0) + 78
00000001000571f0 ap_process_http_connection (10056c208, 9000, 1, 100180e90, 1000, 5) + 90
0000000100051968 ap_run_process_connection (10056c208, 14400, ffffffffffffffff, 100180e90, 10056c200, 1002967e8) + 68
00000001000601c4 process_socket (10056c028, 10056c0b0, 4, 13, 10056e038, 10056c208) + 84
0000000100060c3c worker_thread (1001ea620, 1117e, 0, 4, 13, 100180e90) + 23c ffffffff75bd8afc _lwp_start (0, 0, 0, 0, 0, 0)

 

The other threads listed in the stack trace are waiting for malloc(). The following is a pstack of the httpd.worker at the time of increased CPU usage, which happens with or without the core dump:

 

** zombie (exited, not detached, not yet joined) **
----------------- lwp# 52 / thread# 52 --------------------
ffffffff7722b8c0 dummy_worker(), exit value = 0x0000000000000000
** zombie (exited, not detached, not yet joined) **
----------------- lwp# 53 / thread# 53 --------------------
ffffffff7722b8c0 dummy_worker(), exit value = 0x0000000000000000
** zombie (exited, not detached, not yet joined) **
----------------- lwp# 54 / thread# 54 --------------------
ffffffff7722b8c0 dummy_worker(), exit value = 0x0000000000000000
** zombie (exited, not detached, not yet joined) **

 

 

 

Changes

Because of 10g and 11g differences, this may be popular to see when using a CGI application with Oracle Application Server 10g, then migrating/upgrading to Oracle Fusion Middleware 11g. This issue shows a good reason to require additional 10g-to-11g migration/upgrade steps before going production with the new version.

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms