SRVCTL Stop Instance/DB Hang For 10 Minutes And Then CRSD Agent Terminate (Doc ID 2729264.1)

While stopping database with srvctl, command hang for 10 minutes and then instance terminate

alert_<sid>.log

===========

2020-08-13T10:35:40.612779+02:00
Shutting down ORACLE instance (immediate) (OS id: 21821)   >>>>>>>>>
2020-08-13T10:35:40.612882+02:00
Shutdown is initiated by oraagent.bin@<host> (TNS.
2020-08-13T10:35:43.022835+02:00
Stopping background process SMCO
2020-08-13T10:35:44.040150+02:00
Shutting down instance: further logons disabled
2020-08-13T10:35:44.055878+02:00
Stopping background process CJQ0
2020-08-13T10:35:44.087436+02:00
Process termination requested for pid 22216 [source = rdbms], [info = 2]
[request issued by pid: 21763, uid: 400]
Stopping background process MMNL

Reconfiguration started (old inc 3, new inc 5)
List of instances (total 1) :
 1
Dead instances (total 1) :
 2
My inst 1  
publish big name space -  dead or down/up instance detected, invalidate
domain 0
 Global Resource Directory frozen
* dead instance detected - domain 0 invalid = TRUE
 Communication channels reestablished

 Master broadcasted resource hash value bitmaps
 Non-local Process blocks cleaned out
2020-08-13T10:36:02.900579+02:00
 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived, skipped 0
 Set master node info
 Dwn-cvts replayed, VALBLKs dubious
 All grantable enqueues granted
 Post SMON to start 1st pass IR
Reconfiguration complete (total time 0.1 secs)
Decreasing priority of 1 RS
2020-08-13T10:36:03.982614+02:00
freeing rdom 0
freeing the fusion rht of pdb 0
2020-08-13T10:36:11.047328+02:00
Instance shutdown complete (OS id: 21821)   >>>>>>>>>>>>>>>>>>>>>>>>>>>
===========================================

CRS alert.log
=================
2020-08-13 09:13:51.462 [CLSECHO(5289)]ACFS-9294: updating file /etc/sysconfig/oracledrivers.conf
2020-08-13 09:17:27.320 [OLOGGERD(13475)]CRS-9621: CHM table space size is
2048 MB and free space is 55 MB.
2020-08-13 10:45:40.600 [ORAAGENT(16522)]CRS-5818: Aborted command 'stop' for
resource 'ora.<DB_NAME>.db'. Details at (:CRSAGF00113:) {1:15161:5464} in
<ORACLE_BASE>/diag/crs/<HOSTNAME>/crs/trace/crsd_oraagent_oracle.trc
2020-08-13 10:46:51.139 [ORAAGENT(16522)]CRS-5832: Agent
'/opt/oracle/gi/gi_home1/bin/oraagent_oracle' was unable to process commands.
Details at (:CRSAGF00128:) {2:119:24755} in
<ORACLE_BASE>/diag/crs/<HOSTNAME>/crs/trace/crsd_oraagent_oracle.trc.
2020-08-13 10:46:51.408 [ORAAGENT(1888)]CRS-8500: Oracle Clusterware ORAAGENT
process is starting with operating system process ID 1888
=====================

crsd_oraagent_oracle.trc
======================

2020-08-13 10:35:40.600 :    AGFW:3045549824: [     INFO] {1:15161:5464}
Agent received the message: RESOURCE_STOP[ora.<DB_NAME>.db 1 1] ID 4099:19861
2020-08-13 10:35:40.600 :    AGFW:3045549824: [     INFO] {1:15161:5464}
Preparing STOP command for: ora.<DB_NAME>.db 1 1
2020-08-13 10:35:40.600 :    AGFW:3045549824: [     INFO] {1:15161:5464}
ora.<DB_NAME>.db 1 1 state changed from: ONLINE to: STOPPING  
>>>>>>>>>>>>>>>>>>>>>>>>
2020-08-13 10:35:40.600 :CLSDYNAM:1841764096: [ora.<DB_NAME>.db]{1:15161:5464}
[stop] (:CLSN00108:) clsn_agent::stop {
2020-08-13 10:35:40.600 :CLSDYNAM:1841764096: [ora.<DB_NAME>.db]{1:15161:5464}
[stop] InstAgent::getStopModeAttrib stop mode immediate option 1  
>>>>>>>>>>>>>>>>>>
2020-08-13 10:35:40.600 :CLSDYNAM:1841764096: [ora.<DB_NAME>.db]{1:15161:5464}
[stop] InstAgent::stop 000 {
..
2020-08-13 10:35:57.582 : USRTHRD:1841764096: [     INFO] {1:15161:5464}
DbAgent::cleanup: reset listener flags
2020-08-13 10:35:57.582 :CLSDYNAM:1841764096: [ora.<DB_NAME>.db]{1:15161:5464}
[stop] DbAgent::stopCbk 999 exit }    
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2020-08-13 10:35:57.582 :CLSDYNAM:1841764096: [ora.<DB_NAME>.db]{1:15161:5464}
[stop] InstAgent::stop 660 shutdown
..
2020-08-13 10:45:44.409 : USRTHRD:1837561600: [     INFO] {1:15161:4387}
Thread:DedicatedThread DbAgent::Dedica
..
2020-08-13 10:46:51.139 :    AGFW:3045549824: [     INFO] {2:119:24755} Agent
received the message: AGENT_HB[Engine] ID 12293:20612
2020-08-13 10:46:51.139 :    AGFW:3045549824: [     INFO] {2:119:24755}
Command: stop for resource: ora.<DB_NAME>.db 1 1 has excedeed the maximum
allowed time limit!
2020-08-13 10:46:51.139 :    AGFW:3045549824: [     INFO] {2:119:24755}
numThreads = 6,maxThreads = 25
2020-08-13 10:46:51.139 :    AGFW:3045549824: [     INFO] {2:119:24755} Agent
will terminate as abort is stuck
2020-08-13 10:46:51.140 :   AGENT:3045549824: [     NONE] {2:119:24755}
{2:119:24755} Created alert : (:CRSAGF00128:) :  Command executor module is
unable to process the commands.
2020-08-13 10:46:51.140 :   AGENT:3045549824: [     INFO] {2:119:24755} Agfw
calling user exitCB, will exit on return
2020-08-13 10:46:51.140 :   AGENT:3045549824: [     INFO] {2:119:24755}
returned from user exitCB, exiting
2020-08-13 10:46:51.140 :   AGENT:3045549824: [     INFO] {2:119:24755} Agent
is exiting with exit code: 1
Trace file
<ORACLE_BASE>/diag/crs/<HOSTNAME>/crs/trace/crsd_oraagent_oracle.trc
Oracle Database 19c Clusterware Release 19.0.0.0.0 - Production
Version 19.8.0.0.0 Copyright 1996, 2020 Oracle. All rights reserved.