OCI DBCS VM Rebooted With Error 'ORA-27102: out of memory' Due to Configuration Issue With Parameter USE_LARGE_PAGES
(Doc ID 2507490.1)
Last updated on AUGUST 30, 2023
Applies to:
Oracle Cloud Infrastructure - Database Service - Version N/A to N/A [Release 1.0]Linux x86-64
Symptoms
VM Rebooted With Error 'ORA-27102: out of memory'. This happened multiple times.
During our analysis through OS Watcher, iwe observed that the "kswapd0" process was utilizing 100% cpu during issue time, following which the system is crashed.
Also lot of oracle processes in D state during that time. The assumption was kswpad was trying to move oracle db related pages to swap.
Looks like hugepages were enabled which should have addressed this issue but not used.
MemFree: 1277540 kB
MemAvailable: 710044 kB
SwapTotal: 2097148 kB
SwapFree: 0 kB
HugePages_Total: 32256
HugePages_Free: 32256
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
From alert.log:
Starting ORACLE instance (normal)
************************ Large Pages Information *******************
Parameter use_large_pages = false
Per process system memlock (soft) limit = 88 GB
Large Pages unused system wide = 32256 (63 GB)
Large Pages configured system wide = 32256 (63 GB)
Large Page size = 2048 KB
WARNING:
Large Pages are not used as parameter use_large_pages=false.
For optimal performance, it is recommended to use Large Pages.
********************************************************************
............
Starting Data Guard Broker (DMON)
..............
WARNING: ASM communication error: op 0 state 0x0 (15055)
ERROR: direct connection failure with ASM
Mon Dec 31 10:18:28 2018
Starting ORACLE instance (normal)
************************ Large Pages Information *******************
Parameter use_large_pages = false
Per process system memlock (soft) limit = 88 GB
Large Pages unused system wide = 32256 (63 GB)
Large Pages configured system wide = 32256 (63 GB)
Large Page size = 2048 KB
WARNING:
Large Pages are not used as parameter use_large_pages=false.
For optimal performance, it is recommended to use Large Pages.
...................
Thu Jan 03 02:57:23 2019
WARNING: ASM communication error: op 0 state 0x0 (15055)
ERROR: direct connection failure with ASM
..........
ORA-01034: ORACLE not available
ORA-27102: out of memory
Linux-x86_64 Error: 12: Cannot allocate memory
Additional information: 2731
Additional information: 131076
Additional information: 1623195648
WARNING: ASM communication error: op 18 state 0x40 (1034)
ERROR: slave communication error with ASM
NOTE: Deferred communication with ASM instance
...........
ORA-01034: ORACLE not available
ORA-27102: out of memory
Linux-x86_64 Error: 12: Cannot allocate memory
Additional information: 2731
Additional information: 131076
Additional information: 1623195648
Thu Jan 03 09:28:15 2019
Starting ORACLE instance (normal)
************************ Large Pages Information *******************
Parameter use_large_pages = false
Per process system memlock (soft) limit = 88 GB
Large Pages unused system wide = 32256 (63 GB)
Large Pages configured system wide = 32256 (63 GB)
Large Page size = 2048 KB
WARNING:
Large Pages are not used as parameter use_large_pages=false.
For optimal performance, it is recommended to use Large Pages.
...........
We didn't find much information on /var/log/messages.
Changes
As per customer, no changes were performed.
Cause
To view full details, sign in with your My Oracle Support account. |
|
Don't have a My Oracle Support account? Click to get started! |
In this Document
| Symptoms |
| Changes |
| Cause |
| Solution |