12.2 Oracle Restart: SRVCTL start database fails with error ORA-01017 (Doc ID 2313555.1)

[oradb@testorcl dbs]$ srvctl start database -d <dbname>

PRCR-1079 : Failed to start resource ora.<dbname>.db
CRS-5017: The resource action "ora.<dbname>.db start" encountered the following error:
ORA-01017: invalid username/password; logon denied
. For details refer to "(:CLSN00107:)" in "$TRACE/ohasd_oraagent_orcl.trc".

CRS-2674: Start of 'ora.<dbname>.db' on '<nodename1>' failed
ORA-01017: invalid username/password; logon denied

2017-09-28 13:42:04.269 :CLSDYNAM:3727267584: [ora.orcl.db]{0:0:902} [start] InstAgent::start 200 StartInstance with startoption:2 pfile:null
2017-09-28 13:42:04.269 :CLSDYNAM:3727267584: [ora.orcl.db]{0:0:902} [start] InstAgent::start 400 startInstance
2017-09-28 13:42:04.269 :CLSDYNAM:3727267584: [ora.orcl.db]{0:0:902} [start] InstAgent::startInstance 000 { startOption:2
2017-09-28 13:42:04.269 :CLSDYNAM:3727267584: [ora.orcl.db]{0:0:902} [start] DbAgent:getConnection oracleHome:$DB_HOME oracleSid:<instancename> usrOraEnv:ORA_SERVER_THREAD_ENABLED=FALSE resVersion:12.2.0.1.0
2017-09-28 13:42:04.269 :CLSDYNAM:3727267584: [ora.orcl.db]{0:0:902} [start] InstConnection:InstConnection: init:bc111a10 oracleHome:$DB_HOME oracleSid:<dbname> instanceType:1 instanceVersion:12.2.0.1.0
2017-09-28 13:42:04.269 :CLSDYNAM:3727267584: [ora.orcl.db]{0:0:902} [start] clsnInstConnection::makeConnectStr UsrOraEnv ,ORA_SERVER_THREAD_ENABLED=FALSE,ORACLE_BASE= m_oracleHome $DB_HOME Crshome $GRID_HOME
2017-09-28 13:42:04.269 :CLSDYNAM:3727267584: [ora.orcl.db]{0:0:902} [start] clsnInstConnection::makeConnectStr = (DESCRIPTION=(ADDRESS=(PROTOCOL=beq)(PROGRAM=$DB_HOME/bin/oracle)(ARGV0=oracleiso12c2)(ENVS='ORACLE_HOME=$DB_HOME,ORACLE_SID=orcl,LD_LIBRARY_PATH=,ORA_SERVER_THREAD_ENABLED=FALSE,ORACLE_BASE=')(ARGS='(DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))')(CONNECT_DATA=(SID=<instancename>))))
2017-09-28 13:42:04.270 :CLSDYNAM:3727267584: [ora.orcl.db]{0:0:902} [start] Container:start oracle home $DB_HOME
2017-09-28 13:42:04.271 :CLSDYNAM:3727267584: [ora.orcl.db]{0:0:902} [start] InstAgent::startInstance 020 connect logmode:a
2017-09-28 13:42:04.271 :CLSDYNAM:3727267584: [ora.orcl.db]{0:0:902} [start] InstConnection::connectInt 020 server not attached
2017-09-28 13:42:05.289 :CLSDYNAM:3727267584: [ora.orcl.db]{0:0:902} [start] ORA-01017: invalid username/password; logon denied

2017-09-28 13:42:05.337 :CLSDYNAM:3727267584: [ora.orcl.db]{0:0:902} [start] InstConnection::connectInt 160 Exception OCIException
2017-09-28 13:42:05.338 :CLSDYNAM:3727267584: [ora.orcl.db]{0:0:902} [start] InstConnection:connect 040 excp OCIException OCI error 1017 count 1
2017-09-28 13:42:05.338 :CLSDYNAM:3727267584: [ora.orcl.db]{0:0:902} [start] InstAgent::startInstance 050 errcode 1017
2017-09-28 13:42:05.338 :CLSDYNAM:3727267584: [ora.orcl.db]{0:0:902} [start] InstAgent::startInstance 250 untrapped error
2017-09-28 13:42:05.338 :CLSDYNAM:3727267584: [ora.orcl.db]{0:0:902} [start] InstAgent::startInstance 350 errcode 1017
2017-09-28 13:42:05.338 :CLSDYNAM:3727267584: [ora.orcl.db]{0:0:902} [start] InstAgent::startInstance 360 clean
2017-09-28 13:42:05.338 : USRTHRD:3727267584: {0:0:902} InstConnection::~InstConnection destructor this pConnxn:0x7efcbc111a10
2017-09-28 13:42:05.338 :CLSDYNAM:3727267584: [ora.orcl.db]{0:0:902} [start] InstAgent::startInstance 371 throw excp what:ORA-01017: invalid username/password; logon denied

2017-09-28 13:42:05.338 :CLSDYNAM:3727267584: [ora.orcl.db]{0:0:902} [start] InstAgent::start 500 exception OCIException
2017-09-28 13:42:05.339 :CLSDYNAM:3727267584: [ora.orcl.db]{0:0:902} [start] ORA-01017: invalid username/password; logon denied

2017-09-28 13:42:05.339 :CLSDYNAM:3727267584: [ora.orcl.db]{0:0:902} [start] Agent::commonStart Exception OCIException
2017-09-28 13:42:05.339 :CLSDYNAM:3727267584: [ora.orcl.db]{0:0:902} [start] clsnUtils::error Exception type=2 string=
CRS-5017: The resource action "ora.orcl.db start" encountered the following error:
ORA-01017: invalid username/password; logon denied
. For details refer to "(:CLSN00107:)" in "$TRACE/ohasd_oraagent_<username>.trc".

2017-09-28 13:42:05.339 : AGFW:3727267584: {0:0:902} sending status msg [CRS-5017: The resource action "ora.<dbname>.db start" encountered the following error:
ORA-01017: invalid username/password; logon denied
. For details refer to "(:CLSN00107:)" in "$TRACE/ohasd_oraagent_<username>.trc".
] for start for resource: ora.orcl.db 1 1
2017-09-28 13:42:05.340 :CLSDYNAM:3727267584: [ora.<dbname>.db]{0:0:902} [start] (:CLSN00107:) clsn_agent::start }
2017-09-28 13:42:05.340 : AGFW:3727267584: {0:0:902} Command: start for resource: ora.<dbname>.db 1 1 completed with status: FAIL

# id  <username1>

uid=612(<username1>) gid=601(oinstall) groups=601(oinstall),602(dbdba),603(asmadmin),604(asmdba),605(asmoper)                          <------ racdba group missing

# id <username2>

uid=611(<username2>) gid=601(oinstall) groups=601(oinstall),602(dbdba),603(asmadmin),604(asmdba),607(dgdba),608(backupdba),609(kmdba),611(racdba),610(oper)

 

cat /etc/group:

oinstall:x:601:
dbdba:x:602:<username2>,<username1>
asmadmin:x:603:<username2>,<username1>
asmdba:x:604:<username2>,<username1>
asmoper:x:605:<username1>
dgdba:x:607:<username2>
backupdba:x:608:<username2>
kmdba:x:609:<username2>
racdba:x:611:<username2>                               <==============<username1>user missing for the racdba group
oper:x:610:<username2>
postdrop:x:4105:postfix