ODA Upgrade to 12.1.2.11 or .12 Causes ORA-27301 ORA-27302: failure occurred at: sskgxpsnd2 "The OS has most likely run out of buffers (rval: 4)" with Multiple asmcmd daemon
(Doc ID 2484025.1)
Last updated on APRIL 17, 2023
Applies to:
Oracle Database Appliance Software - Version 12.1.2.11 to 12.2.1.3.0 [Release 12.1 to 12.2]Information in this document applies to any platform.
Symptoms
After an ODA upgrade to 12.1.2.11 (or higher) multiple asmcmd daemon processes are causing high loads on the server.
ASM Alert.log
===========
skgxpvfynet: mtype: 61 process 94183 failed because of a resource problem in the OS. The OS has most likely run out of buffers (rval: 4)
Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_m000_94183.trc (incident=109823):
ORA-00603: ORACLE server session terminated by fatal error
ORA-27504: IPC error creating OSD context
ORA-27300: OS system dependent operation:sendmsg failed with status: 105
ORA-27301: OS failure message: No buffer space available
ORA-27302: failure occurred at: sskgxpsnd2
Incident details in: /u01/app/grid/diag/asm/+asm/+ASM1/incident/incdir_109823/+ASM1_m000_94183_i109823.trc
===========
skgxpvfynet: mtype: 61 process 94183 failed because of a resource problem in the OS. The OS has most likely run out of buffers (rval: 4)
Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_m000_94183.trc (incident=109823):
ORA-00603: ORACLE server session terminated by fatal error
ORA-27504: IPC error creating OSD context
ORA-27300: OS system dependent operation:sendmsg failed with status: 105
ORA-27301: OS failure message: No buffer space available
ORA-27302: failure occurred at: sskgxpsnd2
Incident details in: /u01/app/grid/diag/asm/+asm/+ASM1/incident/incdir_109823/+ASM1_m000_94183_i109823.trc
From OS Watcher data we can see thousands of these ASMCMD Daemon processes.
Line 229708: 1 S grid 95617 1 18 80 0 - 163992 pipe_w 15:24 ? 00:49:36 asmcmd daemon
Line 229709: 1 S grid 95631 1 4 80 0 - 70493 pipe_w 18:04 ? 00:05:06 asmcmd daemon
Line 229710: 1 S grid 95791 1 4 80 0 - 74713 pipe_w 17:20 ? 00:07:43 asmcmd daemon
Line 229712: 1 S grid 96641 1 13 80 0 - 127698 pipe_w 15:49 ? 00:34:32 asmcmd daemon
Line 229714: 1 S grid 96969 1 15 80 0 - 139686 pipe_w 15:42 ? 00:39:20 asmcmd daemon
Line 229717: 1 S grid 97825 1 6 80 0 - 83501 pipe_w 16:47 ? 00:11:55 asmcmd daemon
Line 229718: 1 S grid 97928 1 3 80 0 - 61524 pipe_w 19:04 ? 00:01:46 asmcmd daemon
Line 229719: 1 S grid 98454 1 3 80 0 - 59919 pipe_w 18:51 ? 00:02:14 asmcmd daemon
Line 229723: 1 S grid 99378 1 16 80 0 - 150297 pipe_w 15:35 ? 00:44:03 asmcmd daemon
Line 229709: 1 S grid 95631 1 4 80 0 - 70493 pipe_w 18:04 ? 00:05:06 asmcmd daemon
Line 229710: 1 S grid 95791 1 4 80 0 - 74713 pipe_w 17:20 ? 00:07:43 asmcmd daemon
Line 229712: 1 S grid 96641 1 13 80 0 - 127698 pipe_w 15:49 ? 00:34:32 asmcmd daemon
Line 229714: 1 S grid 96969 1 15 80 0 - 139686 pipe_w 15:42 ? 00:39:20 asmcmd daemon
Line 229717: 1 S grid 97825 1 6 80 0 - 83501 pipe_w 16:47 ? 00:11:55 asmcmd daemon
Line 229718: 1 S grid 97928 1 3 80 0 - 61524 pipe_w 19:04 ? 00:01:46 asmcmd daemon
Line 229719: 1 S grid 98454 1 3 80 0 - 59919 pipe_w 18:51 ? 00:02:14 asmcmd daemon
Line 229723: 1 S grid 99378 1 16 80 0 - 150297 pipe_w 15:35 ? 00:44:03 asmcmd daemon
Changes
The key change was the OS upgrade to UEK4. UEK4 was first introduced as of ODA 12.1.2.11 or 12.1.2.12.
Cause
To view full details, sign in with your My Oracle Support account. |
|
Don't have a My Oracle Support account? Click to get started! |
In this Document
| Symptoms |
| Changes |
| Cause |
| Solution |