OCI DBCS Database Failed With Error "PMON (ospid: 93887): terminating the instance due to error 471"
(Doc ID 2650941.1)
Last updated on AUGUST 30, 2023
Applies to:
Oracle Database Cloud Service - Version N/A to N/A [Release 1.0]Linux x86-64
Symptoms
Customer observed their OCI DBCS database got terminated abruptly.
From /var/log/messages
Mar 6 01:57:58 c03 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Mar 6 01:57:58 c03 kernel: Node 0 hugepages_total=16128 hugepages_free=122 hugepages_surp=0 hugepages_size=2048kB
Mar 6 01:57:58 c03 kernel: 6171413 total pagecache pages
Mar 6 01:57:58 c03 kernel: 898 pages in swap cache
Mar 6 01:57:58 c03 kernel: Swap cache stats: add 59023830, delete 59022932, find 87936250/96982653
Mar 6 01:57:58 c03 kernel: Free swap = 0kB
Mar 6 01:57:58 c03 kernel: Total swap = 16777212kB
Mar 6 01:57:58 c03 kernel: 15727683 pages RAM
Mar 6 01:57:58 c03 kernel: 0 pages HighMem/MovableOnly
Mar 6 01:57:58 c03 kernel: 332527 pages reserved
...........
Mar 6 01:57:58 c03 kernel: Out of memory: Kill process 94028 (ora_dbw0_<db>) score 223 or sacrifice child
Mar 6 01:57:58 c03 kernel: Killed process 94028 (ora_dbw0_<db>) total-vm:79671804kB, anon-rss:50632kB, file-rss:17349448kB
Mar 6 01:58:07 c03 kernel: ora_imr0_<db> invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0
Mar 6 01:58:07 c03 kernel: ora_imr0_<db> cpuset=/ mems_allowed=0
Mar 6 01:57:58 c03 kernel: Node 0 hugepages_total=16128 hugepages_free=122 hugepages_surp=0 hugepages_size=2048kB
Mar 6 01:57:58 c03 kernel: 6171413 total pagecache pages
Mar 6 01:57:58 c03 kernel: 898 pages in swap cache
Mar 6 01:57:58 c03 kernel: Swap cache stats: add 59023830, delete 59022932, find 87936250/96982653
Mar 6 01:57:58 c03 kernel: Free swap = 0kB
Mar 6 01:57:58 c03 kernel: Total swap = 16777212kB
Mar 6 01:57:58 c03 kernel: 15727683 pages RAM
Mar 6 01:57:58 c03 kernel: 0 pages HighMem/MovableOnly
Mar 6 01:57:58 c03 kernel: 332527 pages reserved
...........
Mar 6 01:57:58 c03 kernel: Out of memory: Kill process 94028 (ora_dbw0_<db>) score 223 or sacrifice child
Mar 6 01:57:58 c03 kernel: Killed process 94028 (ora_dbw0_<db>) total-vm:79671804kB, anon-rss:50632kB, file-rss:17349448kB
Mar 6 01:58:07 c03 kernel: ora_imr0_<db> invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0
Mar 6 01:58:07 c03 kernel: ora_imr0_<db> cpuset=/ mems_allowed=0
From alert.log
2020-03-06T09:58:17.972468+08:00
Instance Critical Process (pid: 27, ospid: 94028, DBW0) died unexpectedly
2020-03-06T09:58:29.206329+08:00
System state dump requested by (instance=1, osid=93887 (PMON)), summary=[abnormal instance termination].
.........
2020-03-06T09:58:29.547017+08:00
PMON (ospid: 93887): terminating the instance due to error 471
2020-03-06T09:58:30.740760+08:00
..........
2020-03-06T09:58:32.160729+08:00
Instance terminated by PMON, pid = 93887
2020-03-06T10:20:53.963905+08:00
Instance Critical Process (pid: 27, ospid: 94028, DBW0) died unexpectedly
2020-03-06T09:58:29.206329+08:00
System state dump requested by (instance=1, osid=93887 (PMON)), summary=[abnormal instance termination].
.........
2020-03-06T09:58:29.547017+08:00
PMON (ospid: 93887): terminating the instance due to error 471
2020-03-06T09:58:30.740760+08:00
..........
2020-03-06T09:58:32.160729+08:00
Instance terminated by PMON, pid = 93887
2020-03-06T10:20:53.963905+08:00
From the OSWatcher logs, at the time of instance failure, we can see that the physical memory is available, and none of the processes are using high memory.
But swap memory is decreasing gradually and then came to a point where available swap is 0. Then the database instance terminated by PMON.
KiB Swap: 16777212 total, 6221632 free, 10555580 used. 646680 avail Mem
KiB Swap: 16777212 total, 6032064 free, 10745148 used. 558516 avail Mem
KiB Swap: 16777212 total, 5956544 free, 10820668 used. 671820 avail Mem
...............
KiB Swap: 16777212 total, 1450172 free, 15327040 used. 527004 avail Mem
KiB Swap: 16777212 total, 1227100 free, 15550112 used. 602736 avail Mem
KiB Swap: 16777212 total, 1049584 free, 15727628 used. 609832 avail Mem
KiB Swap: 16777212 total, 988584 free, 15788628 used. 500096 avail Mem
KiB Swap: 16777212 total, 910056 free, 15867156 used. 471208 avail Mem
...................
KiB Swap: 16777212 total, 342480 free, 16434732 used. 446148 avail Mem
KiB Swap: 16777212 total, 166164 free, 16611048 used. 516216 avail Mem
KiB Swap: 16777212 total, 151208 free, 16626004 used. 511936 avail Mem
KiB Swap: 16777212 total, 4744 free, 16772468 used. 497532 avail Mem
KiB Swap: 16777212 total, 4396 free, 16772816 used. 548976 avail Mem
KiB Swap: 16777212 total, 0 free, 16777212 used. 457164 avail Mem <----- Fri Mar 6 09:56:50 +08 2020
KiB Swap: 16777212 total, 0 free, 16777212 used. 341372 avail Mem
KiB Swap: 16777212 total, 0 free, 16777212 used. 243116 avail Mem
KiB Swap: 16777212 total, 0 free, 16777212 used. 195420 avail Mem <----- Fri Mar 6 09:58:24 +08 2020 - Instance terminated by PMON
KiB Swap: 16777212 total, 11994076 free, 4783136 used. 25454584 avail Mem
KiB Swap: 16777212 total, 11994320 free, 4782892 used. 25460600 avail Mem
KiB Swap: 16777212 total, 11994392 free, 4782820 used. 25453456 avail Mem
KiB Swap: 16777212 total, 6032064 free, 10745148 used. 558516 avail Mem
KiB Swap: 16777212 total, 5956544 free, 10820668 used. 671820 avail Mem
...............
KiB Swap: 16777212 total, 1450172 free, 15327040 used. 527004 avail Mem
KiB Swap: 16777212 total, 1227100 free, 15550112 used. 602736 avail Mem
KiB Swap: 16777212 total, 1049584 free, 15727628 used. 609832 avail Mem
KiB Swap: 16777212 total, 988584 free, 15788628 used. 500096 avail Mem
KiB Swap: 16777212 total, 910056 free, 15867156 used. 471208 avail Mem
...................
KiB Swap: 16777212 total, 342480 free, 16434732 used. 446148 avail Mem
KiB Swap: 16777212 total, 166164 free, 16611048 used. 516216 avail Mem
KiB Swap: 16777212 total, 151208 free, 16626004 used. 511936 avail Mem
KiB Swap: 16777212 total, 4744 free, 16772468 used. 497532 avail Mem
KiB Swap: 16777212 total, 4396 free, 16772816 used. 548976 avail Mem
KiB Swap: 16777212 total, 0 free, 16777212 used. 457164 avail Mem <----- Fri Mar 6 09:56:50 +08 2020
KiB Swap: 16777212 total, 0 free, 16777212 used. 341372 avail Mem
KiB Swap: 16777212 total, 0 free, 16777212 used. 243116 avail Mem
KiB Swap: 16777212 total, 0 free, 16777212 used. 195420 avail Mem <----- Fri Mar 6 09:58:24 +08 2020 - Instance terminated by PMON
KiB Swap: 16777212 total, 11994076 free, 4783136 used. 25454584 avail Mem
KiB Swap: 16777212 total, 11994320 free, 4782892 used. 25460600 avail Mem
KiB Swap: 16777212 total, 11994392 free, 4782820 used. 25453456 avail Mem
Changes
Cause
To view full details, sign in with your My Oracle Support account. |
|
Don't have a My Oracle Support account? Click to get started! |
In this Document
| Symptoms |
| Changes |
| Cause |
| Solution |