OPMN Continuously Restarts OHS or Web Cache 11g When the HTTP Listen Port is Disabled - NZ Library Error (Doc ID 1349498.1)

Last updated on JULY 19, 2016

Applies to:

Oracle Fusion Middleware - Version 11.1.1.2.0 to 11.1.2.2.0 [Release Oracle11g]
Oracle HTTP Server - Version 11.1.1.2.0 to 11.1.1.9.0 [Release Oracle11g]
Web Cache - Version 11.1.1.2.0 to 11.1.1.9.0 [Release Oracle11g]
Information in this document applies to any platform.
This has also been seen with Oracle HTTP Server and Oracle Web Cache. It can occur on a Web-Tier 11g Release 1 (11.1.1), Identity Management 11g Release 1 (11.1.1), or "Portal Forms, Reports & Discoverer" 11g Release 1 (11.1.1), and Oracle Forms & Reports Release 2 (11.1.2) installations, (where OPMN exists).

Symptoms

This document outlines an issue which may be otherwise general, making it difficult to troubleshoot an issue. The symptoms arise when client/server SSL connections are failing and the client closes the connection. This may have many different causes. In these cases, everything may actually seem to work, but the underlying processing is continually failing. Because of this, it may be difficult to notice a configuration or patching change caused an underlying ssl connection problem. You may only notice it with more advanced applications or at production time because of increased load.

The underlying problem starts when OPMN continually restarts Oracle HTTP Server due to failed OPMN ping requests. Requests will still work in between these restarts, so may not be initially recognizable. The following errors are seen in the ORACLE_INSTANCE/diagnostics/logs/OHS/ohs1/ohs1.log file:

[OHS] [INCIDENT_ERROR:32] [OHS-2079] [core.c] [host_id: <hostname>] [host_addr: <ipaddress>] [pid: 25208]
[tid: 1141205312] [user: oracle] [VirtualHost: <hostname>:4443] nzos handshake error, nzos_Handshake returned 29039(server <hostname>:4443, client 127.0.0.1)
[OHS] [INCIDENT_ERROR:32] [OHS-2171] [core.c] [host_id: <hostname>] [host_addr: <ipaddress>] [pid: 25208]
[tid: 1141205312] [user: oracle] [VirtualHost: <hostname>:4443] NZ Library Error: SSL negotiation error [Hint: too restrictive SSLCipherSuite]
[OHS] [INCIDENT_ERROR:32] [OHS-2079] [core.c] [host_id: <hostname>] [host_addr: <ipaddress>] [pid: 25307]
[tid: 1134385472] [user: oracle] [VirtualHost: <hostname>:4443] nzos handshake error, nzos_Handshake returned 29039(server <hostname>:4443, client 127.0.0.1)
[OHS] [INCIDENT_ERROR:32] [OHS-2171] [core.c] [host_id: <hostname>] [host_addr: <ipaddress>] [pid: 25307]
[tid: 1134385472] [user: oracle] [VirtualHost: <hostname>:4443] NZ Library Error: SSL negotiation error [Hint: too restrictive SSLCipherSuite]

 

You may see the following if CPU patches are applied, eliminating the SSLCipherSuite symptom but still producing an incompatible Wallet problem during the handshake:

[OHS] [ERROR:32] [OHS-2079] [core.c] [host_id: <hostname>] [host_addr: <ipaddress>] [pid: 4748] [tid: 1596] [user: oracle] [VirtualHost: <hostname>:4443]  nzos handshake error, nzos_Handshake returned 28862(server kv-t420.us.oracle.com:4443, client 127.0.0.1)
[OHS] [ERROR:32] [OHS-2171] [core.c] [host_id: <hostname>] [host_addr: <ipaddress>] [pid: 4748] [tid: 1596] [user: oracle] [VirtualHost: <hostname>:4443]  NZ Library Error: SSL IO error [Hint: the client stop the connection unexpectedly]
[OHS] [ERROR:32] [OHS-2079] [core.c] [host_id: <hostname>] [host_addr: <ipaddress>] [pid: 4748] [tid: 1596] [user: oracle] [VirtualHost: <hostname>:4443]  nzos handshake error, nzos_Handshake returned 28860(server <hostname>:4443, client 127.0.0.1)
[OHS] [ERROR:32] [OHS-2171] [core.c] [host_id: <hostname>] [host_addr: <ipaddress>] [pid: 4748] [tid: 1596] [user: oracle] [VirtualHost: <hostname>:4443]  NZ Library Error: SSL fatal alert

 

You may see the following write error in the wlproxy.log generated by the WLS Proxy Plugin (mod_wl_ohs) when the initial connection is made, but OHS restarted forcing a reconnection for the requests being made (and basically needs to start over):

Write to the client failed: calling URL::close at line 509 of BaseProxy.cpp
*******Exception type [WRITE_ERROR_TO_CLIENT] raised at line 510 of BaseProxy.cpp
got exception in sendResponse phase: WRITE_ERROR_TO_CLIENT [os error=0,  line 510 of BaseProxy.cpp]:  at line 566
*NOT* failing over after sendResponse() exception: WRITE_ERROR_TO_CLIENT
request [/app/application.someServlet] did NOT process successfully..................

 

If CPU NZ <Patch 17337741> is applied there are no SSL errors, OHS just simply restarts, which will cause performance issues and cascading issues with an application being processed during that time. Various issues and symptoms may appear. The common symptom to look for is OHS restarting when you have not explicitly initiated the shutdown. The problem here is with OPMN's requirement to be able to ping the managed component, upon failure, it will restart the component since it will assume it is already down.

Changes

This document is also synonymous with "How to Disable the Non-SSL Port in httpd.conf" because there are specific steps missed when doing this. OPMN was previously depending on non-SSL to obtain a status and out of the box is unable to with only SSL enabled.

Oracle HTTP Server 11g is using SSL exclusively and the Oracle HTTP Listen value in the httpd.conf has been disabled:

#Listen 7777

> opmnctl status -l

Processes in Instance: instance1
---------------------------------+--------------------+---------+----------+------------+----------+-----------+------
ias-component | process-type | pid | status | uid | memused | uptime | ports
---------------------------------+--------------------+---------+----------+------------+----------+-----------+------
webcache1 | WebCache-admin | 25174 | Alive | 886326979 | 84404 | 0:00:08 | http_admin:7786
webcache1 | WebCache | 25173 | Alive | 886326978 | 103828 | 0:00:08 |

http_stat:7787,http_invalidation:7788,https_listen:7789,http_listen:7785
ohs1 | OHS | 25172 | Alive | 886326977 | 440780 | 0:00:08 | https:9999,https:4443    <-- No http port 


If the Oracle HTTP Server listen port is made active again, OPMN checks this port for its ping test and all works fine.

The same issue occurs if HTTP listen port is disabled for Oracle Web Cache:

webcache1  | WebCache |   22441 | Alive  |  388 |  155044 | 0:35:49 | https_listen:443,http_stat:8092,http_invalidation:8093  <--No http_listen

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms