NFS Mount Failed During Export - Export Job Aborted and Remained Hung With Repeated ORA 700 [ksuxdl: cleanup failures] (Doc ID 3074169.1)

Last updated on MARCH 21, 2025

Applies to:

Oracle Database - Enterprise Edition - Version 19.23.0.0.0 and later
Information in this document applies to any platform.

Symptoms

DataPump export was initiated for PDB Database on <NODE1> and it got hung after running for some duration due to NFS hang issue on node 0.
Both <NODE0> and <NODE1> share same NFS mount. (but only <NODE0> had NFS mount issue).

Operating system log - messages.

Jan 23 20:21:14 <NODE0> kernel: sdn: sdn1 sdn2
Jan 23 20:26:55 <NODE0> kernel: nfs: server <SERVER>.<DOMAINNAME> not responding, still trying
Jan 23 20:26:57 <NODE0> kernel: nfs: server <SERVER>.<DOMAINNAME> not responding, still trying
Jan 23 20:27:44 <NODE0> kernel: nfs: server <SERVER>.<DOMAINNAME> not responding, still trying
Jan 23 20:28:37 <NODE0> kernel: sdc: sdc1 sdc2

Jan 24 09:42:54 <NODE0> kernel: nfs: server <SERVER>.<DOMAINNAME> not responding, still trying
Jan 24 09:42:54 <NODE0> kernel: nfs: server <SERVER>.<DOMAINNAME> not responding, still trying
Jan 24 09:45:05 <NODE0> kernel: nfs: server <SERVER>.<DOMAINNAME> not responding, still trying
Jan 24 09:46:05 <NODE0> systemd[1]: Starting Cleanup of Temporary Directories...

Noticed the PDB DataPump export on <NODE1> seemed to be "hung", and the NFS mount on <NODE0> was also hung.
Attached to the DataPump job and killed it, and then unmounted/remounted the NFS mount.
After doing above actions, received an ORA-00600 [12240] followed by an ORA-00700 [ksuxdl: cleanup failures]

<NODE1> - Database alert log
======

025-01-24T07:11:52.710418-05:00
Dumping diagnostic data in directory=[cdmp_20250124071152], requested by (instance=2, osid=74560 (DM00)), summary=[incident=502777].
2025-01-24T07:15:28.149649-05:00
IPC Receiver dump detected. Sender instance 2 Receiver pnum 67 ospid 33229 [oracle@<NODE0> (RSMN)], pser 1
2025-01-24T07:15:28.150277-05:00
Errors in file /u01/app/odaorabase/oracle/diag/rdbms/<DB_UNIQUE_NAME>/<INSTANCE1>/trace/<INSTANCE1>_rsmn_33229.trc:
2025-01-24T07:27:27.744390-05:00

<Node2> - Database alert log
======
2025-01-24T07:06:50.484512-05:00
<PDBNAME>(5):Stopped service SYS.KUPC$S_2_20250124004502_0
<PDPNAME>(5):DM00 stopped with pid=447, OS id=74560, job SYSTEM.SYS_EXPORT_FULL_01
2025-01-24T07:09:03.019276-05:00
CLMN: clean deferred state objects - failed
2025-01-24T07:11:50.963473-05:00
Errors in file /u01/app/odaorabase/oracle/diag/rdbms/<DB_UNIQUE_NAME>/<INSTANCE2>/trace/<INSTANCE2>_dm00_74560.trc (incident=502777):
ORA-00600: internal error code, arguments: [12240], [], [], [], [], [], [], [], [], [], [], []
Incident details in: /u01/app/odaorabase/oracle/diag/rdbms/<DB_UNIQUE_NAME>/<INSTANCE2>/incident/incdir_502777/<INSTANCE2>_dm00_74560_i502777.trc
2025-01-24T07:11:52.642857-05:00
.
.
2025-01-24T09:53:59.570227-05:00
WARNING: CL01 has failed to cleanup a dead process for 1000 attempts
Errors in file /u01/app/odaorabase/oracle/diag/rdbms/<DB_UNIQUE_NAME>/<INSTANCE2>/trace/<INSTANCE2>_cl01_71550.trc (incident=501593) (PDBNAME=CDB$ROOT):
ORA-00700: soft internal error, arguments: [ksuxdl: cleanup failures], [0xBD03CCD80], [process], [1000], [0xB003B0D58], [ksv reaper], [0], [], [], []
Incident details in: /u01/app/odaorabase/oracle/diag/rdbms/<DB_UNIQUE_NAME>/<INSTANCE2>/incident/incdir_501593/<INSTANCE2>_cl01_71550_i501593.trc
2025-01-24T09:54:01.626748-05:00

Later, observed the following on each node's database alert log repeating over and over:

NODE0:

2025-01-24T11:34:32.573087-05:00
DIAG (ospid: 33062) has not called a wait for 812 secs.
2025-01-24T11:34:32.573256-05:00
DIAG (ospid: 33062) is hung in an acceptable location (trcdump 0x102.08).

NODE1:

2025-01-24T13:13:10.253401-05:00
IPC Send timeout detected. Sender: ospid 69264 [oracle@dfspoda1 (RSMN)]
Receiver: inst 1 binc 4 ospid 33229
2025-01-24T13:13:57.538229-05:00
WARNING: CL02 has failed to cleanup a dead process for 2200 attempts

Load average from top on <NODE0> reporting very high numbers, but <NODE1> was okay.

Cause

	To view full details, sign in with your My Oracle Support account.
	Don't have a My Oracle Support account? Click to get started!

In this Document

My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.

NFS Mount Failed During Export - Export Job Aborted and Remained Hung With Repeated ORA 700 [ksuxdl: cleanup failures] (Doc ID 3074169.1)

Applies to:

Symptoms

Cause

To view full details, sign in with your My Oracle Support account.

Don't have a My Oracle Support account? Click to get started!