E-IB: Subscription Contract Stuck in Status "Working" while PSSUBHND_* Process Keeps Running (Doc ID 622499.1)

Symptoms

A message is coming in from a 3rd party.  Doing a flat file conversion, and it has been working for about a year.  Subscriber is PS FDM on Tools 8.16.  The site ran purgeall, rebooted appserver, and published the messages again, but the first one always gets stuck in Working on the Sub Contract.

The site is using 2 messages.  Each one is custom, and each one is using it's own delivered Channel.  1 message is using the Vendor Channel, and the other message is using the Voucher Channel.  They are both getting processed through an AE program that grabs a flat file, creates a message, etc..., and the Vendor messages work, but the Voucher one is the one that gets stuck in Working.
                   
- The site has not applied any related maintenance patches, bundles, or updates.  
- The site has not modified any of the Subscription PeopleCode.
- The site has been trying different messages, it has not been the exact same message that has been attempting over and over.
- This same message works in another environment.
- Dev is working.  The site tried using the Dev app server with this prod db, but same problem

The appsrv.log with LogFence=5 shows that the PSSUBDSP never makes a call to the PSSUBHND, the hand off does not seem to take place

Cause