Processes calling system call memcntl() cause slowness in large system due to scalability of UFS (Doc ID 2030329.1)

Last updated on OCTOBER 27, 2016

Applies to:

Solaris SPARC Operating System - Version 10 3/05 HW2 to 11.2 [Release 10.0 to 11.0]
Information in this document applies to any platform.

Symptoms

Processes calling system call memcntl() cause slowness in large system due to scalability of UFS.

When memcntl() system call is issued, UFS blindly fetches all pages from the disk. When the system calls are issued multiple times for a single file, UFS fetches pages for each instance separately. This results in heavy I/O loading and potentially blocks the system for some time.

memcntl() is called by libc library when processes start.

One of the failing scenario is when Oracle database is running, there are many new connections within a short period (may be a few minutes) to the listener and listener is trying to create processes to handle such a surge in sessions. However due to this bug 15700705, the creation of processes fail to do so and listener queue is being filled up.

From the vmstat, we can see there seems to be blocking in I/O, but actually it is blocked due to resources in UFS.

kthr      memory            page            disk          faults      cpu
r b w   swap  free  re  mf pi po fr de sr m1 m1 m1 m2   in   sy   cs us sy id
0 10 0 526163016 351802752 28081 8099 117381 3266 3265 0 0 9 7 7 0 67617 181029 64512 4 3 93
7 30 0 485205296 322717496 34200 49741 102718 13 13 0 0 1 0 1 0 75681 647754 111979 12 8 81
0 28 0 484274568 322118400 25129 50903 53313 21 21 0 0 0 0 0 0 76479 602113 105742 11 8 81
1 22 0 483158008 321374592 31353 47423 54529 26 24 0 0 0 0 0 0 128629 636008 114659 11 8 82
0 24 0 482135952 320761248 31595 48291 83897 37 37 0 0 0 0 0 0 129927 629553 112079 11 8 81
4 25 0 481222160 320365760 32755 51041 60135 23 23 0 0 0 0 0 0 126548 622904 111071 10 8 82
1 21 0 480237152 319629216 22357 51317 71462 20 18 0 0 6 5 5 0 126837 671369 111978 11 8 81
1 14 0 479328864 319002504 15229 28909 37000 32304 32302 0 0 1 1 1 0 106947 436795 88973 8 10 82
0 55 0 478819584 318837440 23919 23822 60582 51906 51906 0 0 0 0 0 0 154325 507160 163795 11 10 79
0 23 0 477841768 318197064 14631 24154 73112 14 14 0 0 0 0 0 0 142696 413246 144051 9 8 82
0 23 0 477108616 317788336 17471 17914 59710 27 26 0 0 0 0 0 0 135794 390128 134991 8 8 84
1 47 0 476612464 317801448 28395 17171 55861 12 11 0 0 0 0 0 0 136386 393329 134461 9 8 83
2 33 0 476172744 317569032 33976 19013 70341 5 5 0 0 6 6 6 0 138891 360018 136798 9 8 83
131 19 0 475592808 317205992 25015 15843 45408 18 18 0 0 0 0 0 0 117618 338320 113453 7 23 70
1 37 0 474991408 316838360 25693 13017 61746 9 9 0 0 0 0 0 0 121306 337008 113685 8 8 85
2 60 0 474670768 316657072 20045 9977 52672 35 35 0 0 0 0 0 0 121062 354792 114434 8 7 85
1 17 0 474366552 316413136 18619 11739 56421 46 45 0 0 0 0 0 0 120058 434667 113272 8 7 85
0 18 0 474049656 316162400 20805 10593 52604 43 42 0 0 0 0 0 0 115914 432113 106780 8 7 85

Note the column "b" means blocking here but not actually I/O. The iostat shows no issue in I/O average access time.

From the thread list, many threads call memcntl() system call and blocks in UFS buffer cache

 PC: sema_p+0x138    CMD: oraclebvsbipr (LOCAL=NO)
 stack pointer for thread 3028fd8b4e0: 2a13128ea51
 [ 000002a13128ea51 sema_p+0x138() ]
   getblk_common+0x150()
   bread_common+0x84()
   bmap_read+0x1f8()
   ufs_getpage_ra+0x98()
   ufs_getpage+0x574()
   fop_getpage+0x44()
   segvn_faulta+0x120()
   as_faulta+0xec()
   memcntl+0x570()
   syscall_trap+0xac()

There could be a few thousands of this thread due to a surge in sessions.

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms