Database Cloud Service Terminated by Linux Out-of-Memory (OOM) Killer

(Doc ID 2382285.1)

Last updated on APRIL 07, 2018

Applies to:

Oracle Database as a Service - Version N/A to N/A [Release 1.0]
Information in this document applies to any platform.

Symptoms

User asked what caused their production instance to get terminated. We have verified the alert.log and OS messages.

From alert.log:

TNS-12537: TNS:connection closed
ns secondary err code: 12560
nt main err code: 0
nt secondary err code: 0
nt OS err code: 0
opiodr aborting process unknown ospid (9080) as a result of ORA-609
Tue Mar 20 10:41:37 2018
Warning: VKTM detected a time drift.
Time drifts can result in an unexpected behavior such as time-outs. Please check trace file for more details.
Tue Mar 20 10:41:43 2018
LGWR waiting for instance termination
Tue Mar 20 10:41:44 2018
PMON (ospid: 18540): terminating the instance due to error 471 <-------------- PMON initiating instance termination
Tue Mar 20 10:41:46 2018
System state dump requested by (instance=1, osid=18551 (DIAG)), summary=[abnormal instance termination].
System State dumped to trace file /u01/app/oracle/diag/rdbms/orcl/ORCL/trace/ORCL_diag_18551_20180320104146.trc
Termination issued to instance processes. Waiting for the processes to exit
Tue Mar 20 10:42:00 2018
Instance termination failed to kill one or more processes
Instance terminated by PMON, pid = 18540 <-------------- instance terminated
Tue Mar 20 10:44:22 2018
Starting ORACLE instance (normal)
************************ Large Pages Information *******************
Per process system memlock (soft) limit = 128 GB

Total Shared Global Region in Large Pages = 5826 MB (37%)

Large Pages used by this instance: 2913 (5826 MB)
Large Pages unused system wide = 4 (8192 KB)
Large Pages configured system wide = 2918 (5836 MB)
Large Page size = 2048 KB

RECOMMENDATION:
Total System Global Area size is 15 GB. For optimal performance,
prior to the next instance restart:
1. Increase the number of unused large pages by
at least 4764 (page size 2048 KB, total size 9528 MB) system wide to
get 100% of the System Global Area allocated with large pages
.........

From /var/log/messages:

Mar 20 10:41:37 FARMS kernel: [1163754.634368] [ 9242] 0 9242 8689 945 20 3 0 0 perl
Mar 20 10:41:37 FARMS kernel: [1163754.643316] [ 9243] 0 9243 8689 945 19 3 0 0 perl
Mar 20 10:41:37 FARMS kernel: [1163754.650929] Out of memory: Kill process 18559 (oracle) score 182 or sacrifice child <-------------- Out of memory
Mar 20 10:41:37 FARMS kernel: [1163754.658710] Killed process 18559 (oracle) total-vm:15981976kB, anon-rss:5432kB, file-rss:6026568kB <-------------- Process killed in OS level

 

Changes

 No changes reported recently.

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms