Custom Master Channel Program Not Running Enough Threads, Dequeues Slowly

Last updated on NOVEMBER 15, 2017

Applies to:

Symptoms

A custom channel master program, written by the customer to perform customer-specific action - using the MTA SDK (pdf) - when dequeuing from their application-specific channel, often does not run as many threads as expected.

If there are several messages in the channel queue when job_controller starts the process, it will start several threads and dequeue messages quickly - for a short time.

But when there are no messages in the queue and one arrives and job_controller starts the master program, if several more arrive while that first process is running, a new process is started.  The new process starts several threads and seems to behave as expected - for a short time.  The original process eventually reduces to only a single thread.  Eventually the additional processes started reduce to only a single thread.  So eventually, if there is a steady stream of messages enqueued to the channel, the maximum number of processes allowed for the channel are all running only a single thread - significantly fewer threads than required to keep ahead of the number of incoming messages.

If you restart job_controller while there are many messages in the queue for this channel, it will start multiple processes, all running multiple threads and they will dequeue quickly - for a short time - until they all reduce to only a single thread.

These details can be observed with the prstat or ps commands to see the number of threads. Also the imsimta qm sum -data output shows a large number of messages "Pending" but very few "Active".

Changes

The custom channel program is a new development.  This behavior was noticed as soon as they began testing it under load.

Cause