Bug 28834364 - OSYSMOND TAKES HIGH CPU USAGE AFTER GI UPGRADE FAILURE AT LAST NODE FROM 12102
(Doc ID 2720939.1)
Last updated on JULY 20, 2024
Applies to:
Oracle Database - Enterprise Edition - Version 12.1.0.2 and laterInformation in this document applies to any platform.
Symptoms
GI upgrade attempt from 12.1.0.2 flex cluster and upgrade is failed on last node.
# top
top - 22:36:28 up 63 days, 2:47, 10 users, load average: 1513.77, 1496.12,1469.59
Tasks: 2146 total, 3 running, 2134 sleeping, 0 stopped, 9 zombie
%Cpu(s): 58.1 us, 4.4 sy, 0.0 ni, 37.3 id, 0.1 wa, 0.0 hi, 0.1 si, 0.0st
KiB Mem : 31906304 total, 1316556 free, 10925648 used, 19664100 buff/cache
KiB Swap: 23225324 total, 22310688 free, 914636 used. 12163584 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
31071 root rt 0 1730964 811580 107724 S 102.1 2.5 1168:18 osysmond.bin >>>>>>High CPU
osysmond.trc:
---------------
2020-01-24 22:22:01.120 : GIPCNET:3914790656: gipcmodNetworkResolve: failed to create new address for osName '<hostname>:w', name 'tcp://hostname:w'
2020-01-24 22:22:01.120 : GIPCNET:3914790656: gipcmodNetworkResolve: slos op: sgipcnPopulateAddrInfo
2020-01-24 22:22:01.120 : GIPCNET:3914790656: gipcmodNetworkResolve: slosdep : Error 0 (0)
2020-01-24 22:22:01.120 : GIPCNET:3914790656: gipcmodNetworkResolve: slosloc : getaddrinfo(
2020-01-24 22:22:01.120 : GIPCNET:3914790656: gipcmodNetworkResolve: slosinfo: service not available on socket
2020-01-24 22:22:01.120 : GIPCGEN:3914790656: gipcInternalResolve: failed toresolve ipv4, retrying ipv6
2020-01-24 22:22:01.123 : GIPCNET:3914790656: gipcmodNetworkResolve: failedto create new address for osName 'hostname:w', name 'tcp://hostname:w'
2020-01-24 22:22:01.123 : GIPCNET:3914790656: gipcmodNetworkResolve: slos op: sgipcnPopulateAddrInfo
2020-01-24 22:22:01.123 : GIPCNET:3914790656: gipcmodNetworkResolve: slosdep : Error 0 (0)
2020-01-24 22:22:01.123 : GIPCNET:3914790656: gipcmodNetworkResolve: slosloc : getaddrinfo(
2020-01-24 22:22:01.123 : GIPCNET:3914790656: gipcmodNetworkResolve: slosinfo: service not available on socket
2020-01-24 22:22:01.123 : GIPCGEN:3914790656: gipcInternalResolve: failed toresolve ipv6, retrying ipv4
2020-01-24 22:22:01.123 :GIPCXCPT:3914790656: gipcInternalResolve: failed to
resolve addr 0x7f03c45583e0 [0000000000aa928a] { gipcAddress:name'tcp://hostname:w',objFlags 0x0, addrFlags 0xc},retgipcretAddressNotFound(21)
2020-01-24 22:22:01.123 :GIPCXCPT:3914790656: gipcResolveF[gipcInternalConnect : gipcInternal.c : 735]: EXCEPTION[ ret
gipcretAddressNotFound (21) ]failed to resolve address 0x7f03c45583e0[0000000000aa928a]{ gipcAddress:name 'tcp://hostname:w',objFlags0x0,
addrFlags 0xc }, flags 0x4000
2020-01-24 22:22:01.123 :GIPCTRAC:3914790656: gipcConnectF[crfm_connect_timeout : crfm.c : 3297]: EXCEPTION[ ret gipcretAddressNotFound
(21) ] failed connect endp 0x7f03c4557ea0 [0000000000aa9286] { gipcEndpoint: localAddr '', remoteAddr 'tcp://hostname:w', numPend 0, numReady 0, numDone0, numDead 0, numTransfer 0, objFlags 0x0, pidPeer 0, readyRef (nil), ready0, wobj (nil), sendp (nil) status 13flags 0x20080000, flags-2 0x0, usrFlags0x20 }, addr 0x7f03c4515920 [0000000000aa928d] { gipcAddress : name
'tcp://hostname:w', objFlags 0x0, addrFlags 0x4 }, cookie (nil),flags 0x02020-01-24 22:22:01.123 : CRFM:3914790656: crfm_connect_to: Cantconnecttcp://hostname:w(gipcret: 21)
2020-01-24 22:22:01.123 : CRFM:3914790656: crfmctx dump follows
2020-01-24 22:22:01.123 : CRFM:3914790656: ****************************
2020-01-24 22:22:01.123 : CRFM:3914790656: crfm_dumpctx: connection localname:
2020-01-24 22:22:01.123 : CRFM:3914790656: crfm_dumpctx: connection peername: tcp://hostname:wU
2020-01-24 22:22:01.123 : CRFM:3914790656: crfm_dumpctx: connaddr: tcp://hostname:wU
2020-01-24 22:22:01.123 : CRFM:3914790656: crfm_dumpctx: ctype: 0
2020-01-24 22:22:01.123 : CRFM:3914790656: crfm_dumpctx: mytype: 0
2020-01-24 22:22:01.123 : CRFM:3914790656: crfm_dumpctx: hostname
2020-01-24 22:22:01.123 : CRFM:3914790656: crfm_dumpctx: myport:
2020-01-24 22:22:01.123 : CRFM:3914790656: crfm_dumpctx: rhostname hostname
2020-01-24 22:22:01.123 : CRFM:3914790656: crfm_dumpctx: rport:
2020-01-24 22:22:01.123 : CRFM:3914790656: crfm_dumpctx: flags: 1
2020-01-24 22:22:01.123 : CRFM:3914790656: ****************************
2020-01-24 22:22:01.123 : CRFMOND:3914790656: Loggerd Not Reachable,
consulting with hostname sysmond, total 4...2020-01-24 22:22:01.124 : GIPCNET:3915843328: gipcmodNetworkDestroyAddress:
failed to destroy address 0x7f03c45583e0 [0000000000aa928a] { gipcAddress :name '(dying)', objFlags 0x2, addrFlags 0xc }
2020-01-24 22:22:01.124 : GIPCNET:3915843328: gipcmodNetworkDestroyAddress:slos op : sgipcnDestroyAddress
2020-01-24 22:22:01.124 : GIPCNET:3915843328: gipcmodNetworkDestroyAddress:slos dep : Resource temporarily unavailable (11)
2020-01-24 22:22:01.124 : GIPCNET:3915843328: gipcmodNetworkDestroyAddress:slos loc : argument
2020-01-24 22:22:01.124 : GIPCNET:3915843328: gipcmodNetworkDestroyAddress:slos info: Invalid addr argument
2020-01-24 22:22:01.256 : CRFM:3926378240: Persisting data forhost: and conntype:0 as host/ip:HAIP and port:<port>
2020-01-24 22:22:01.259 : CRFMOND:3914790656: Got SETM from hostname. master set to hostname.
This behavior has been seen especially when the node is left in partially upgraded state.
Changes
Cause
To view full details, sign in with your My Oracle Support account. |
|
Don't have a My Oracle Support account? Click to get started! |
In this Document
| Symptoms |
| Changes |
| Cause |
| Solution |
| References |