Post ExaRU/Image update, Polkit service is not activating on 1 (of 2) compute nodes. (Doc ID 2844001.1)

Last updated on FEBRUARY 15, 2022

Applies to:

Symptoms

Exadata X7 (On-Prem) with 2 compute nodes (node1 and node2)
Image : 20.1.15
Kernel : 4.14.35-1902.306.2.10

+ Polkit service, Authorization Manager, was not activating on compute node 1:

<<<
[root@node1 ~]# service polkit status
Redirecting to /bin/systemctl status polkit.service
● polkit.service - Authorization Manager
Loaded: loaded (/usr/lib/systemd/system/polkit.service; static; vendor preset: disabled)
Active: failed (Result: exit-code) since Sat 2022-01-29 14:38:42 EST; 2min 42s ago <<-----------
Docs: man:polkit(8)
Process: 113591 ExecStart=/usr/lib/polkit-1/polkitd --no-debug (code=exited, status=127)
Main PID: 113591 (code=exited, status=127)Jan 29 14:38:42 node1.com polkitd[113591]: Started polkitd version 0.112
Jan 29 14:38:42 node1.com systemd[1]: Failed to start Authorization Manager.
[root@node1 ~]#
>>>

+ However, service is loaded and active on good node (node2) though, which as well gone through similar changes as problem node (node1).

<<<
[root@node2 ~]# service polkit status
Redirecting to /bin/systemctl status polkit.service
● polkit.service - Authorization Manager
Loaded: loaded (/usr/lib/systemd/system/polkit.service; static; vendor preset: disabled)
Active: active (running) since Sat 2022-01-29 12:15:45 EST; 2h 22min ago <<-------
Docs: man:polkit(8)
Main PID: 187109 (polkitd)
Tasks: 7
Memory: 15.2M
CGroup: /system.slice/polkit.service
>>>

+ This particular Customer (CT) uses Centrify, a Third party software, to manage server access and authentication which in turn is dependent on Polkit service.
+ As Polkit is not running, Centrify installation is not working also.
+ Centrify was uninstalled Centrify on both nodes prior to patching to avoid issues.
+ Nothing substantial or concrete reported in messages log either.
+ Verified and compared polkit set-up across both nodes, especially permissions/ownership on crtitical files/dirs (/, /etc, /var etc). Everything looks same.
   Also reset the permissions and user/group ownership for all files provided by the polkit and polkit-pkla-compat packages.
   However, problem still persisted. So, ruled-out similar known issue discussed in DocID 2521614.1
+ Also "dbus" service is loaded/active on problem node.
+ Polkit rpm versions are same on both nodes.

<<<
[root@node1 log]# rpm -qa|grep -i polkit
polkit-0.112-26.0.1.el7.x86_64
polkit-pkla-compat-0.1-4.el7.x86_64
[root@node1 log]#

[root@node2 ~]# rpm -qa|grep -i polkit
polkit-pkla-compat-0.1-4.el7.x86_64
polkit-0.112-26.0.1.el7.x86_64
[root@node2 ~]#
>>>

+ Executed polkit in Debug mode which then provided some clues:

<<<
[root@node1 ~]# /usr/lib/polkit-1/polkitd
Successfully changed to user polkitd
/usr/lib/polkit-1/polkitd: symbol lookup error: /usr/lib64/gio/modules/libgiofam.so: undefined symbol: g_local_directory_monitor_get_type <<---------------
[root@node1 ~]#
>>>

+ Service is trying to refer library "libgiofam.so" which should not be there in first place. Especially same library, along with few others under "/usr/lib64/gio/modules" are NOT present on Node2.
+ Following above, renamed all libs under "/usr/lib64/gio/modules", and then attempted to restart.
+ With this change, it's no more getting connection time-out error when starting polkit.
+ It's getting loaded, but Activate state started reported as "inactive (dead)", previously it was "Failed".

Changes

+ Exadata Image was successfully updated from 19.2.19 to 20.1.15 (on both nodes - 1 and 2)
+ System is booting-up without any issues.

Cause

My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.