RP/TUX 6.5/TUX 8.1/TUX 9.1 - Invalid Fchg oc causes app server core (SIGSEGV) or WSH LIBTUX_CAT:1526 Memory Allocation fail (Doc ID 770057.1)

Goal

The client is a workstation client which passes an FML or FML32 type buffer to a server.  The server uses Fchg or
Fchg32 to add a new field to the buffer, and then return the buffer to the client.  The server does not initialize the
occurrence number before calling Fchg.  

Fchg (3FML):

Fchg, Fchg32-change field occurrence value 

Synopsis:
     #include <stdio.h> 
     #include "fml.h" 
int Fchg(FBFR *fbfr, FLDID fieldid, FLDOCC oc, char *value, FLDLEN len)
     #include "fml32.h" 
int Fchg32(FBFR32 *fbfr, FLDID32 fieldid, FLDOCC32 oc, char *value, FLDLEN32 len)

Description:
The maximum occurrence number allowed in an FML32 buffer is 33554431.  
Yet, an uninitialized  'long' type variable in an application server can yield a random value which exceeds the maximum for the occurrence number.  
Values of 715827880 and higher cause a segmentation fault when an application server calls Fchg32, or put Work Station Handler 
(WSH) in an infinite loop when the server returns the reply buffer to the client.  
When the WSH loops, it continuously writes the following messages into the userlog:

134239.TuxMachineName!WSH.xxxx: WSNAT_CAT:1029: ERROR: Sending of reply message to client failed
134239.TuxMachineName!WSH.xxxx: LIBTUX_CAT:1526: ERROR: Memory allocation failure BEMALLOC - malloc failed
134239.TuxMachineName!WSH.xxxx: WSNAT_CAT:1026: ERROR: Handler received unexpected message
134239.TuxMachineName!WSH.xxxx: LIBTUX_CAT:1526: ERROR: Memory allocation failure BEMALLOC - malloc failed
134239.TuxMachineName!WSH.xxxx: WSNAT_CAT:1026: ERROR: Handler received unexpected message                                   
                                        
.
.
.

To stop the WSH from writing these messages to the userlog, it is necessary to reboot the Work Station Listener (WSL), or kill the WSH.

Solution

My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.