My Oracle Support Banner

Oracle Linux:Docker ps / containerd hangs; build-up of CLOSE_WAIT sockets when using UEK6 kernel (Doc ID 2984615.1)

Last updated on NOVEMBER 02, 2023

Applies to:

Linux OS - Version Oracle Linux 7.9 with Unbreakable Enterprise Kernel [5.4.17] to Oracle Linux 8.8 with Unbreakable Enterprise Kernel [5.4.17] [Release OL7U9 to OL8U8]
Information in this document applies to any platform.

Symptoms

Typical OCI running ODO symptoms would be: failure (intermittent or "slowness") to pull/stop/start applications through odo, including the failure ((unhealthy) in docker ps) of docker container ongoing
health checks. The following command has been used to determine if an compute instance odo containerd was indeed in a hung state:

time sudo /opt/odo/docker/bin/ctr -a /run/odo/containerd/containerd.sockversion

If the call takes more than 1 second, something is wrong. (typical when hung: 20 seconds context deadline timeout)

OCI Services running nginx would have symptoms to include a pile-up of tcp6 sockets in close_wait status, check with:

sudo netstat -anp | grep CLOSE_WAIT | wc -l

Technically all software using epoll and threads are susceptible to eventually get hung on affected UEK6 kernel versions. Most of OCI uses ODO, which uses Docker (golang) and Containerd (golang). Golang uses the efficient epoll and threads for accepting connections on Linux.

Other notable users of epoll are nginx, Jetty (from our typical Dropwizard apps) and Netty.

Changes

OCI internal systems (both SE and Overlay, not customers' host) were updated through "OS updater" from "5.4.17-2136.311.6.1" to "5.4.17-2136.312.3.4" during Nov-2022 patching cycle.
After that, we noticed that with 5.4.17-2136.312.3.4 kernel "odo docker" operation encounters hung(timeout) when accessing "unix:///run/odo/docker.sock"

Cause

To view full details, sign in with your My Oracle Support account.

Don't have a My Oracle Support account? Click to get started!


In this Document
Symptoms
Changes
Cause
Solution

My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.