Errror: "UnmarshalException: Error Unmarshaling Return Header.." When Trying to Open a File from AutoVue that is installed in environment that contains McAfee HIPS (Doc ID 2197449.1)

Last updated on OCTOBER 26, 2016

Applies to:

Oracle AutoVue 2D Professional - Version 21.0.0 and later
Information in this document applies to any platform.

Symptoms

When trying to view a file in AutoVue Client Server 21.0.0, the following error shows up on the Applet :

java.rmi.UnmarshalException: Error unmarshaling return header; nested exception is:
java.net.SocketException: Connection reset
at sun.rmi.transport.StreamRemoteCall.executeCall(Unknown Source)
at sun.rmi.server.UnicastRef.invoke(Unknown Source)
at com.cimmetry.jvueserver.docserver.JVcetControlImpl_Stub.fileOpen(Unknown Source)
at com.cimmetry.jvueserver.Document.setFile(Unknown Source)
at com.cimmetry.jvueserver.DocumentManager.setFile(Unknown Source)
at com.cimmetry.jvueserver.SessionManager.setFile(Unknown Source)
at com.cimmetry.jvueserver.VCETConnection.fileOpen(Unknown Source)
at com.cimmetry.jvueserver.VCETConnection.fileOpen(Unknown Source)
at sun.reflect.GeneratedMethodAccessor108.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at com.cimmetry.jvueserver.Channel.invokeMethod(Unknown Source)
at com.cimmetry.jvueserver.Channel.exec(Unknown Source)
at com.cimmetry.jvueserver.Channel.exec(Unknown Source)
at com.cimmetry.jvueserver.Channel.access$800(Unknown Source)
at com.cimmetry.jvueserver.Channel$ChannelThread.run(Unknown Source)
Caused by: java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(Unknown Source)
at java.net.SocketInputStream.read(Unknown Source)
at java.io.BufferedInputStream.fill(Unknown Source)
at java.io.BufferedInputStream.read(Unknown Source)
at java.io.DataInputStream.readByte(Unknown Source)
... 16 more

AutoVue Logs shows that there is a crash of AutoVue DocServers and a hotspot log is created:

2016-09-09 08:29:24,849 INFO JVueServer0: Document.open,SessionID: -3953390239651183666,pss,DocID: 10, URL: upload://10.xxx.92.xxx/C:\usr\work\NEDL\P8\sample.pdf, DMS Info: null, DMS ID: null, Name: null Display Name: null, Type: 0, Parent: null
2016-09-09 08:30:24,226 INFO JVueServer0: DocServer 1 : #
2016-09-09 08:30:24,226 INFO JVueServer0: DocServer 1 : # A fatal error has been detected by the Java Runtime Environment:
2016-09-09 08:30:24,226 INFO JVueServer0: DocServer 1 : #
2016-09-09 08:30:24,226 INFO JVueServer0: DocServer 1 : # EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x7268bd2a, pid=6712, tid=6824
2016-09-09 08:30:24,226 INFO JVueServer0: DocServer 1 : #
2016-09-09 08:30:24,226 INFO JVueServer0: DocServer 1 : # JRE version: Java(TM) SE Runtime Environment (7.0_45-b18) (build 1.7.0_45-b18)
2016-09-09 08:30:24,226 INFO JVueServer0: DocServer 1 : # Java VM: Java HotSpot(TM) Client VM (24.45-b08 mixed mode windows-x86 )
2016-09-09 08:30:24,226 INFO JVueServer0: DocServer 1 : # Problematic frame:
2016-09-09 08:30:24,226 INFO JVueServer0: DocServer 1 : # V http://jvm.dll+0x13bd2a
2016-09-09 08:30:24,226 INFO JVueServer0: DocServer 1 : #
2016-09-09 08:30:24,226 INFO JVueServer0: DocServer 1 : # Core dump written. Default location: D:\Oracle\AutoVue\bin\hs_err_pid6712.mdmp
2016-09-09 08:30:24,226 INFO JVueServer0: DocServer 1 : #
2016-09-09 08:30:24,226 INFO JVueServer0: DocServer 1 : # An error report file with more information is saved as:
2016-09-09 08:30:24,226 INFO JVueServer0: DocServer 1 : # D:\Oracle\AutoVue\bin\hs_err_pid6712.log
2016-09-09 08:30:24,272 INFO JVueServer0: DocServer 1 : #
2016-09-09 08:30:24,272 INFO JVueServer0: DocServer 1 : # If you would like to submit a bug report, please visit:
2016-09-09 08:30:24,272 INFO JVueServer0: DocServer 1 : # http://bugreport.sun.com/bugreport/crash.jsp
2016-09-09 08:30:24,272 INFO JVueServer0: DocServer 1 : #
2016-09-09 08:30:24,476 ERROR JVueServer0: Failed to open file
com.cimmetry.core.VueRemoteException: Remote exception received from 6712@wadcp8vd1
nested exception is:
java.rmi.UnmarshalException: Error unmarshaling return header; nested exception is:
java.net.SocketException: Connection reset
at sun.rmi.transport.StreamRemoteCall.executeCall(Unknown Source)
at sun.rmi.server.UnicastRef.invoke(Unknown Source)
at com.cimmetry.jvueserver.docserver.JVcetControlImpl_Stub.fileOpen(Unknown Source)

====================================================================

McAfee HIPS is used in the machine where AutoVue is installed.

 

It appears that McAfee HIPS is causing the issue. The security operations group turned off HIPS on the AutoVue server (ServerName) and AutoVue worked fine now. So...what is the known issue with McAfee HIPS and AutoVue?

Here is the error logged in McAfee HIPS:

An attempt to execute code resulting from a buffer overflow was detected. The buffer overflow occurred in process D:\ORACLE\AUTOVUE\JRE\BIN\JAVA.EXE running with the privileges of user {Domain\userid} on the system with Agent ServerName.

 

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms