Exalogic ZFSA : Performance Impact as ZFS Adaptive Replacement Cache (ARC) Down to the Minimum Allowed Amount of 64MB in AK 2013.1.1.x releases
Last updated on MARCH 25, 2016
Applies to:Oracle Exalogic Elastic Cloud Software - Version 220.127.116.11.7 to 18.104.22.168.0
Oracle Solaris on x86-64 (64-bit)
Oracle Virtual Server ( x86-64)
Severe Performance drop is seen in the applications (e.g. JMS Transactions in FMW) deployed on the Exalogic Nodes/guest vServers.
The active ZFS Storage head is very slow and not responding to the commands.
In a shared shell, Oracle Support Engineer restarted the affected active node and generated system core dump using NMI ( please review Document ID : 1173064.1 - Oracle ZFS Storage Appliance: How to generate a system core dump in case of system hang (BUI and CLI fails to respond) using NMI when directed to do so by an Oracle Support Engineer).
From the core dump, it could be seen that the active storage head hitting a sever 'bug 18562374 missing call to arc_free_data_block in l2arc_read()' which is found in all 2013.1.1.x releases of the ZFS-SA code (i.e. affected Exalogic PSU releases : 22.214.171.124.7, 126.96.36.199.7, 188.8.131.52.2, 184.108.40.206.0, 220.127.116.11.1, 18.104.22.168.2 )
This bug introduces a huge performance impact as it will force the system to shrink ZFS Adaptive Replacement Cache (ARC) down to the minimum allowed amount of 64MB, thus it will trigger all IO to be served from disk instead of having a large amount of IO being served out of RAM.
The supported way to check the cache is in BUI/CLI, and is:
Sign In with your My Oracle Support account
Don't have a My Oracle Support account? Click to get started
My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms