My Oracle Support Banner

Bug 28834364 - OSYSMOND TAKES HIGH CPU USAGE AFTER GI UPGRADE FAILURE AT LAST NODE FROM 12102 (Doc ID 2720939.1)

Last updated on MAY 25, 2021

Applies to:

Oracle Database - Enterprise Edition - Version 12.1.0.2 and later
Information in this document applies to any platform.

Symptoms

 
GI upgrade attempt from 12.1.0.2 flex cluster and upgrade is failed on last node.

 #  top
top - 22:36:28 up 63 days,  2:47, 10 users,  load average: 1513.77, 1496.12,1469.59
Tasks: 2146 total,   3 running, 2134 sleeping,   0 stopped,   9 zombie
%Cpu(s): 58.1 us,  4.4 sy,  0.0 ni, 37.3 id,  0.1 wa,  0.0 hi,  0.1 si,  0.0st
KiB Mem : 31906304 total,  1316556 free, 10925648 used, 19664100 buff/cache
KiB Swap: 23225324 total, 22310688 free,   914636 used. 12163584 avail Mem
PID  USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND  
31071 root      rt   0 1730964 811580 107724 S 102.1  2.5   1168:18 osysmond.bin  >>>>>>High CPU       

osysmond.trc:
---------------
2020-01-24 22:22:01.120 : GIPCNET:3914790656:  gipcmodNetworkResolve: failed to create new address for osName '<hostname>:w', name 'tcp://hostname:w'
2020-01-24 22:22:01.120 : GIPCNET:3914790656:  gipcmodNetworkResolve: slos op:  sgipcnPopulateAddrInfo
2020-01-24 22:22:01.120 : GIPCNET:3914790656:  gipcmodNetworkResolve: slosdep :  Error 0 (0)
2020-01-24 22:22:01.120 : GIPCNET:3914790656:  gipcmodNetworkResolve: slosloc :  getaddrinfo(
2020-01-24 22:22:01.120 : GIPCNET:3914790656:  gipcmodNetworkResolve: slosinfo:  service not available on socket
2020-01-24 22:22:01.120 : GIPCGEN:3914790656:  gipcInternalResolve: failed toresolve ipv4, retrying ipv6
2020-01-24 22:22:01.123 : GIPCNET:3914790656:  gipcmodNetworkResolve: failedto create new address for osName 'hostname:w', name 'tcp://hostname:w'
2020-01-24 22:22:01.123 : GIPCNET:3914790656:  gipcmodNetworkResolve: slos op:  sgipcnPopulateAddrInfo
2020-01-24 22:22:01.123 : GIPCNET:3914790656:  gipcmodNetworkResolve: slosdep :  Error 0 (0)
2020-01-24 22:22:01.123 : GIPCNET:3914790656:  gipcmodNetworkResolve: slosloc :  getaddrinfo(
2020-01-24 22:22:01.123 : GIPCNET:3914790656:  gipcmodNetworkResolve: slosinfo:  service not available on socket
2020-01-24 22:22:01.123 : GIPCGEN:3914790656:  gipcInternalResolve: failed toresolve ipv6, retrying ipv4
2020-01-24 22:22:01.123 :GIPCXCPT:3914790656:  gipcInternalResolve: failed to
resolve addr 0x7f03c45583e0 [0000000000aa928a] { gipcAddress:name'tcp://hostname:w',objFlags 0x0, addrFlags 0xc},retgipcretAddressNotFound(21)
2020-01-24 22:22:01.123 :GIPCXCPT:3914790656:  gipcResolveF[gipcInternalConnect : gipcInternal.c : 735]: EXCEPTION[ ret
gipcretAddressNotFound (21) ]failed to resolve address 0x7f03c45583e0[0000000000aa928a]{ gipcAddress:name 'tcp://hostname:w',objFlags0x0,
addrFlags 0xc }, flags 0x4000
2020-01-24 22:22:01.123 :GIPCTRAC:3914790656:  gipcConnectF[crfm_connect_timeout : crfm.c : 3297]: EXCEPTION[ ret gipcretAddressNotFound
(21) ]  failed connect endp 0x7f03c4557ea0 [0000000000aa9286] { gipcEndpoint: localAddr '', remoteAddr 'tcp://hostname:w', numPend 0, numReady 0, numDone0, numDead 0, numTransfer 0, objFlags 0x0, pidPeer 0, readyRef (nil), ready0, wobj (nil), sendp (nil) status 13flags 0x20080000, flags-2 0x0, usrFlags0x20 }, addr 0x7f03c4515920 [0000000000aa928d] { gipcAddress : name
'tcp://hostname:w', objFlags 0x0, addrFlags 0x4 }, cookie (nil),flags 0x02020-01-24 22:22:01.123 :    CRFM:3914790656: crfm_connect_to: Cantconnecttcp://hostname:w(gipcret: 21)
2020-01-24 22:22:01.123 :    CRFM:3914790656: crfmctx dump follows
2020-01-24 22:22:01.123 :    CRFM:3914790656: ****************************
2020-01-24 22:22:01.123 :    CRFM:3914790656: crfm_dumpctx: connection localname:
2020-01-24 22:22:01.123 :    CRFM:3914790656: crfm_dumpctx: connection peername:  tcp://hostname:wU
2020-01-24 22:22:01.123 :    CRFM:3914790656: crfm_dumpctx: connaddr:  tcp://hostname:wU
2020-01-24 22:22:01.123 :    CRFM:3914790656: crfm_dumpctx: ctype:  0
2020-01-24 22:22:01.123 :    CRFM:3914790656: crfm_dumpctx: mytype:  0
2020-01-24 22:22:01.123 :    CRFM:3914790656: crfm_dumpctx: hostname  
2020-01-24 22:22:01.123 :    CRFM:3914790656: crfm_dumpctx: myport:  
2020-01-24 22:22:01.123 :    CRFM:3914790656: crfm_dumpctx: rhostname  hostname
2020-01-24 22:22:01.123 :    CRFM:3914790656: crfm_dumpctx: rport:  
2020-01-24 22:22:01.123 :    CRFM:3914790656: crfm_dumpctx: flags:  1
2020-01-24 22:22:01.123 :    CRFM:3914790656: ****************************
2020-01-24 22:22:01.123 : CRFMOND:3914790656: Loggerd Not Reachable,
consulting with hostname sysmond, total 4...2020-01-24 22:22:01.124 : GIPCNET:3915843328:  gipcmodNetworkDestroyAddress:
failed to destroy address 0x7f03c45583e0 [0000000000aa928a] { gipcAddress :name '(dying)', objFlags 0x2, addrFlags 0xc }
2020-01-24 22:22:01.124 : GIPCNET:3915843328:  gipcmodNetworkDestroyAddress:slos op  :  sgipcnDestroyAddress
2020-01-24 22:22:01.124 : GIPCNET:3915843328:  gipcmodNetworkDestroyAddress:slos dep :  Resource temporarily unavailable (11)
2020-01-24 22:22:01.124 : GIPCNET:3915843328:  gipcmodNetworkDestroyAddress:slos loc :  argument
2020-01-24 22:22:01.124 : GIPCNET:3915843328:  gipcmodNetworkDestroyAddress:slos info:  Invalid addr argument
2020-01-24 22:22:01.256 :    CRFM:3926378240: Persisting data forhost: and conntype:0 as host/ip:HAIP and port:<port>
2020-01-24 22:22:01.259 : CRFMOND:3914790656: Got SETM from hostname. master set to hostname.



This behavior has been seen especially when the node is left in partially upgraded state.

Changes

 

Cause

To view full details, sign in with your My Oracle Support account.

Don't have a My Oracle Support account? Click to get started!


In this Document
Symptoms
Changes
Cause
Solution
References

My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.