OVD 11g HA Instances Stop Responding and Solaris Capped Zones Hung. Must Reboot the Zones to Gain Access and Recover. (Doc ID 1952777.1)

Last updated on MARCH 08, 2017

Applies to:

Oracle Virtual Directory - Version 11.1.1.0 and later
Oracle Solaris on SPARC (64-bit)

Symptoms

Oracle Virtual Directory (OVD) 11g, e.g.  11.1.1.7.0 version.

Twelve (12) Solaris 11.2 capped zones: 4 CPUs with 8 GB of memory

Two OVD instances stop responding to LDAP requests at the same time. OVD hung and Solaris capped zone hung at 50%.  It is not possible to kill the OVD processes and also unable to manage any other processes. No traces can be obtained because the only means to access and recover is to reboot the zones.

Sequence of events:
• The unix LDAP authentication fails for all LDAP clients
• Verification if 2 OVD zones are up: OK
• Connection to zones and attempts of requests to 2 OVDs: No response.
• Verification of servers ovd: No response
• Attempts to stop normally the instances OVD: OVD processes cannot be stopped
• Kill and kill -9 of OVD process: No success
• Attempt to restart for OVD1: No success (blocked on shutdown, seems due to OVD process)
• Attempt to restart for OVD2: sane as above
• Halt of the two zones
• Restart of zones
• OVD starts and is back to work

OVD's are hanging every two days.

After setting OVD debug at trace:32, was able to obtain one set of logs.  The diagnostic.log shows:

...<snip>...
[2014-11-12T23:17:28.417+01:00] [octetstring] [TRACE] [] [com.octetstring.vde.DoSManager] [tid: 12] [ecid: 0000KaQrBkt1zWGpIw0Fyf1KOFo5000003,0] [SRC_CLASS: com.octetstring.vde.util.VDELogger] [SRC_METHOD: debug] Current operations per connection (/10.240.122.3, 0/0).
[2014-11-12T23:17:39.472+01:00] [octetstring] [NOTIFICATION] [OVD-60053] [com.octetstring.vde.ConnectionHandler] [tid: 207870] [ecid: 0000Kaa5M6h1zWGpIw0Fyf1KOFo501MxDF,0] Error accepting connection from server socket: Remote host closed connection during handshake.[[
javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:882)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1203)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1230)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1214)
at com.octetstring.vde.ConnectionHandler.initializeConnection(ConnectionHandler.java:174)
at com.octetstring.vde.ConnectionHandler.run(ConnectionHandler.java:218)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.EOFException: SSL peer shut down incorrectly
at com.sun.net.ssl.internal.ssl.InputRecord.read(InputRecord.java:333)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:863)
... 8 more

]]
[2014-11-12T23:18:03.417+01:00] [octetstring] [ERROR] [OVD-60055] [com.octetstring.vde.frontend.LDAP] [tid: 16] [ecid: 0000KaQrCfm1zWGpIw0Fyf1KOFo5000005,0] Out of memory error: Printing stack trace.[[
java.lang.OutOfMemoryError: GC overhead limit exceeded
at java.util.HashMap.addEntry(HashMap.java:753)
at java.util.HashMap.put(HashMap.java:385)
at java.util.ListResourceBundle.loadLookup(ListResourceBundle.java:180)
at java.util.ListResourceBundle.handleKeySet(ListResourceBundle.java:144)
at java.util.ResourceBundle.containsKey(ResourceBundle.java:1669)
at oracle.core.ojdl.logging.ODLFormatter.getString(ODLFormatter.java:925)
at oracle.core.ojdl.logging.ODLFormatter.toLogMessage(ODLFormatter.java:486)
at oracle.core.ojdl.logging.ODLFormatter.toLogMessage(ODLFormatter.java:229)
at oracle.core.ojdl.logging.ODLHandler.publish(ODLHandler.java:764)
at oracle.core.ojdl.logging.ODLHandler.publish(ODLHandler.java:738)
at oracle.core.ojdl.logging.ODLLogger.doLog(ODLLogger.java:878)
at oracle.core.ojdl.logging.ODLLogger.logRec(ODLLogger.java:914)
at oracle.core.ojdl.logging.ODLLogger.logRec(ODLLogger.java:901)
at oracle.core.ojdl.logging.ODLLogger.log(ODLLogger.java:666)
at com.octetstring.vde.util.VDELogger.info(VDELogger.java:219)
at com.octetstring.vde.frontend.LDAP.run(LDAP.java:319)
at java.lang.Thread.run(Thread.java:662)

]]
[2014-11-12T23:17:35.790+01:00] [octetstring] [TRACE] [] [com.octetstring.vde.Connection] [tid: 207868] [ecid: 0000Kaa5LCi1zWGpIw0Fyf1KOFo501MxDD,0] [SRC_CLASS: com.octetstring.vde.util.VDELogger] [SRC_METHOD: debug] ConnID:193,729 Added mesgID:1 to Q, QSize:1
[2014-11-12T23:18:54.117+01:00] [octetstring] [NOTIFICATION] [OVD-60053] [com.octetstring.vde.ConnectionHandler] [tid: 207870] [ecid: 0000Kaa5M6h1zWGpIw0Fyf1KOFo501MxDF,0] Error accepting connection from server socket: Remote host closed connection during handshake.[[
javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:882)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1203)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1230)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1214)
at com.octetstring.vde.ConnectionHandler.initializeConnection(ConnectionHandler.java:174)
at com.octetstring.vde.ConnectionHandler.run(ConnectionHandler.java:218)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.EOFException: SSL peer shut down incorrectly
at com.sun.net.ssl.internal.ssl.InputRecord.read(InputRecord.java:333)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:863)
... 8 more

]]
...<end>...



Reviewed:  OVD Diagnostic Logs Show SSL Handshake Errors and "Remote host closed connection during handshake" Document 1268605.1

And:
JVM tuning in OVD Dev's blog
OVD 11g Search Fails To Returns All Results / Search Halts And Throws Operations Error / OVD Log Shows java.lang.OutOfMemoryError Exception Document 1088258.1

However, all tuning has already been applied.


Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms