My Oracle Support Banner

One of Several Storage Servers is Not Updated During Pathing (Doc ID 2750634.1)

Last updated on JANUARY 26, 2022

Applies to:

Oracle Exadata Storage Server Software - Version 11.2.1.2.0 and later
Linux x86-64

Symptoms

One of 10 cell servers was not updated to Exadata 20.1.3 from 19.3.7, when the machine is rebooted it still showing the old image version.

The filesystem /dev/md5 is having filesystem errors:

1) The OS log file shows filesystem errors:


Feb 6 14:09:49 <MACHINE> kernel: [ 26.073570] EXT4-fs error (device md5): ext4_mb_generate_buddy:757: group 17, block bitmap and bg descriptor inconsistent: 20356 vs 20357 free clusters
Feb 6 14:14:41 <MACHINE> kernel: [ 332.720353] EXT4-fs (md5): error count since last fsck: 12
Feb 6 14:14:41 <MACHINE> kernel: [ 332.720358] EXT4-fs (md5): initial error at time 1609315580: ext4_ext_map_blocks:4329: inode 131077
Feb 6 14:14:41 <MACHINE> kernel: [ 332.720362] EXT4-fs (md5): last error at time 1612638574: ext4_mb_generate_buddy:757
Feb 6 17:06:01 <MACHINE> kernel: [ 14.217170] EXT4-fs (md5): warning: mounting fs with errors, running e2fsck is recommended




2) The same IO errors are causing:

Feb 6 14:17:16 <MACHINE> systemd-logind[1532]: Removed session 187.
Feb 6 14:17:21 <MACHINE> kernel: [ 492.448503] INFO: task sync:14441 blocked for more than 120 seconds.
Feb 6 14:17:21 <MACHINE> kernel: [ 492.455713] Not tainted 4.14.35-1902.301.1.el7uek.x86_64 #2
Feb 6 14:17:21 <MACHINE> kernel: [ 492.462651] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Feb 6 14:17:21 <MACHINE> kernel: [ 492.471521] sync D 0 14441 12091 0x00000080
Feb 6 14:17:21 <MACHINE> kernel: [ 492.471523] Call Trace:
Feb 6 14:17:21 <MACHINE> kernel: [ 492.471541] __schedule+0x2bc/0x89b
Feb 6 14:17:21 <MACHINE> kernel: [ 492.471550] ? SyS_tee+0x3b0/0x3a1
Feb 6 14:17:21 <MACHINE> kernel: [ 492.471554] schedule+0x36/0x7c
Feb 6 14:17:21 <MACHINE> kernel: [ 492.471568] wb_wait_for_completion+0x64/0x8e
Feb 6 14:17:21 <MACHINE> kernel: [ 492.471571] ? remove_wait_queue+0x60/0x60
Feb 6 14:17:21 <MACHINE> kernel: [ 492.471573] sync_inodes_sb+0xa6/0x2b5
Feb 6 14:17:21 <MACHINE> kernel: [ 492.471586] ? SyS_tee+0x3b0/0x3a1
Feb 6 14:17:21 <MACHINE> kernel: [ 492.471588] ? SyS_tee+0x3b0/0x3a1
Feb 6 14:17:21 <MACHINE> kernel: [ 492.471590] sync_inodes_one_sb+0x15/0x17
Feb 6 14:17:21 <MACHINE> kernel: [ 492.471594] iterate_supers+0xc7/0x129
Feb 6 14:17:21 <MACHINE> kernel: [ 492.471606] sys_sync+0x44/0xac
Feb 6 14:17:21 <MACHINE> kernel: [ 492.471624] do_syscall_64+0x79/0x1b9
Feb 6 14:17:21 <MACHINE> kernel: [ 492.471632] entry_SYSCALL_64_after_hwframe+0x169/0x0
Feb 6 14:17:21 <MACHINE> kernel: [ 492.471638] RIP: 0033:0x7fe554359b77
Feb 6 14:17:21 <MACHINE> kernel: [ 492.471643] RSP: 002b:00007fff02c2f2f8 EFLAGS: 00000202 ORIG_RAX: 00000000000000a2
Feb 6 14:17:21 <MACHINE> kernel: [ 492.471654] RAX: ffffffffffffffda RBX: 00007fff02c2f408 RCX: 00007fe554359b77
Feb 6 14:17:21 <MACHINE> kernel: [ 492.471657] RDX: 00007fe55462b280 RSI: 0000000000000001 RDI: 00007fe5543ecfb7
Feb 6 14:17:21 <MACHINE> kernel: [ 492.471658] RBP: 0000000000000001 R08: 0000000000000000 R09: 0000000000000000
Feb 6 14:17:21 <MACHINE> kernel: [ 492.471660] R10: 00007fff02c2ed60 R11: 0000000000000202 R12: 000000000040156d
Feb 6 14:17:21 <MACHINE> kernel: [ 492.471661] R13: 00007fff02c2f400 R14: 0000000000000000 R15: 0000000000000000


3) Note that /dev/md5 is the current system partition:

Kernel version: 4.14.35-1902.301.1.el7uek.x86_64 #2 SMP Tue Mar 31 16:50:32 PDT 2020 x86_64
Cell version: OSS_19.3.7.0.0_LINUX.X64_200428
Cell rpm version: cell-19.3.7.0.0_LINUX.X64_200428-1.x86_64

Active image version: 19.3.7.0.0.200428
Active image kernel version: 4.14.35-1902.301.1.el7uek
Active image created: 2020-04-29 02:41:36 -0700
Active image activated: 2021-02-06 14:20:07 -0500
Active image type: production
Active image status: success
Active image label: OSS_19.3.7.0.0_LINUX.X64_200428
Active node type: STORAGE
Install type: Bare Metal with InfiniBand
Active system partition on device: /dev/md5
Active software partition on device: /dev/md7

 

 

 

Changes

 

Cause

To view full details, sign in with your My Oracle Support account.

Don't have a My Oracle Support account? Click to get started!


In this Document
Symptoms
Changes
Cause
Solution
References


My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.