ACFSK-0052: ACFS VOLUME MOUNT FAILED. VOLUME IS OUT OF SPACE (Doc ID 1389624.1)

Last updated on DECEMBER 27, 2011

Applies to:

Oracle Server - Enterprise Edition - Version: 11.2.0.2 and later   [Release: 11.2 and later ]
Information in this document applies to any platform.

Symptoms

On 11.2.0.2 and with no interim patches, unable to mount an ACFS volume for recently added nodes:

ACFSK-0052: Volume asm/var-301 (Major: 252, Minor: 154113) mount failed. Volume is out of space.

More details from the OS system logs:

...
/archive/syslog/system/2011-07-12/kern:2011-07-12T12:40:22.000+00:00
node4/<IP address> [kern.warning] kernel[]: ACFSK-0052: Volume asm/var-301
(Major: 252, Minor: 154113) mount failed. Volume is out of space.
...
/archive/syslog/system/2011-07-14/kern:2011-07-14T10:16:43.000+00:00
node5/<IP address> [kern.warning] kernel[]: ACFSK-0058: I/O failure
0xc0000185 with device minor 0x0 during a metadata synch
write.<4>ACFSK-0054: Volume loop0 (Major: 7, Minor: 0) mount failed with
internal status code (-1073741435).
/archive/syslog/system/2011-07-14/kern:2011-07-14T10:18:13.000+00:00
node4/<IP address> [kern.warning] kernel[]: ACFSK-0058: I/O failure
0xc0000185 with device minor 0x0 during a metadata synch write.[Oracle ACFS]
ASSERTION FAILURE: !(iflags & OFS_BF_INVAL_DIRTY) File:
/scratch/builds/aime/aime_usm_758305/usm/src/ofs/driver/./gen/ofsgenmeta.c
Line: 5283
/archive/syslog/system/2011-07-14/kern:2011-07-14T10:18:13.000+00:00
node4/<IP address> [kern.warning] kernel[]: KsDumpStack: Stack call
traceback
...
node4/<IP address> [kern.warning] kernel[]: Call Trace:
...
:oracleacfs:OfsDoAssertion+0x11e/0x159
...
:oracleacfs:ofs_meta_invalidate+0x50f/0xf3e
...
:oracleacfs:OfsDlmUpdateCommitSeqNum+0xaa/0xb1
...
:oracleacfs:OfsWriteBuffer+0xc4/0xd2
...
:oracleacfs:ProcessTransRecord+0x398/0x116a
...
:oracleacfs:ofs_FETA_create_meta_file_int+0x303/0x44f
...
:oracleacfs:ProcessTrans+0x3a/0x56
...
:oracleacfs:OfsAbortTrans+0x5f/0x11d
...
:oracleacfs:ofs_FETA_create_local_meta_files_int+0xc6a/0xcd2
...
:oracleacfs:ofs_FETA_load_meta_files+0x27f/0x37f
...
:oracleacfs:OfsSetupVolume+0x3fd/0x150d
...
/archive/syslog/system/2011-07-14/kern:2011-07-14T10:18:13.000+00:00
node4/<IP address> [kern.warning] kernel[]: ACFSK-0059: Fenum Unknown.
Starting offset: 3221226496 . Length of data transfer: 512 bytes.
/archive/syslog/system/2011-07-14/kern:2011-07-14T10:18:13.000+00:00
node4/<IP address> [kern.warning] kernel[]: ACFSK-0060: Impact: Cluster
Object: MetaData Operation Context: Write Snapshot?: ? AcfsObjectID: 0 .
Internal ACFS Location: 200 .
...

Snippet from orarootagent_root.log:

/app/grid/oh2/log/node5/agent/crsd/orarootagent_root/orarootagent_root.log:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
...
2011-07-12 12:49:35.528: [ora.mycelium_data.var.acfs][3120564544]
{5:29894:19082} [start] (:CLSN00010:)ACFS-9257: Mounting device
'/dev/asm/var-301' on mount point '/app/var'.
2011-07-12 12:49:35.528: [ora.mycelium_data.var.acfs][3120564544]
{5:29894:19082} [start] (:CLSN00010:)
2011-07-12 12:49:35.730: [ora.mycelium_data.var.acfs][3120564544]
{5:29894:19082} [start] (:CLSN00010:)mount: wrong fs type, bad option, bad
superblock on /dev/asm/var-301,
2011-07-12 12:49:35.730: [ora.mycelium_data.var.acfs][3120564544]
{5:29894:19082} [start] (:CLSN00010:) missing codepage or other error
2011-07-12 12:49:35.730: [ora.mycelium_data.var.acfs][3120564544]
{5:29894:19082} [start] (:CLSN00010:) In some cases useful info is
found in syslog - try
2011-07-12 12:49:35.730: [ora.mycelium_data.var.acfs][3120564544]
{5:29894:19082} [start] (:CLSN00010:) dmesg | tail or so
2011-07-12 12:49:35.730: [ora.mycelium_data.var.acfs][3120564544]
{5:29894:19082} [start] (:CLSN00010:)
2011-07-12 12:49:35.730: [ora.mycelium_data.var.acfs][3120564544]
{5:29894:19082} [start] (:CLSN00010:)
2011-07-12 12:49:35.831: [ora.mycelium_data.var.acfs][3120564544]
{5:29894:19082} [start] (:CLSN00010:)ACFS-9258: Failed to mount device
'/dev/asm/var-301' on mount point '/app/var'.

...

However, seems there is available space in the ACFS volume
* From df -h, shows it is 66% available (30GB-20GB=11GB)
* From 'acfsutil info fs /app/var' shows: size:32212254720 (30 GB), free:18913095680 (17.6 GB).

Changes

New changes: Added 2 new nodes to an existing 3 node cluster.

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms