Minor numbers are too big for 32-bit applications due to Solaris limitation (Doc ID 1011866.1)

Last updated on OCTOBER 05, 2022

Applies to:

Symptoms

Messages such as the following are issued by the Solaris kernel.

WARNING: sd33873:a minor 0x42288 too big for 32-bit applications

WARNING: sd35165:a minor 0x44ae8 too big for 32-bit applications

WARNING: sd33874:a minor 0x42290 too big for 32-bit applications

WARNING: sd33879:a minor 0x422b8 too big for 32-bit applications

Such warnings can only appear on systems running Solaris 7 or later.

The warning is somewhat misleading in that it would seem to only be relevant for 32-bit applications, but creation of the minor node actually failed, meaning the device is not visible in the system at all.

They are issued during device auto-configuration, or more specifically, during device instance attachment.  For each physical device controlled by a particular device driver, the kernel associates an instance number which is unique within the set of all instance numbers associated with devices controlled by the device driver.  A dev_info node is created for each device instance.  Instance numbers are nonnegative, but are stored in the dev_info node as a signed 32 bit integer, so in effect, an instance number is a 31 bit unsigned integer.

When the kernel attaches the dev_info node to the device tree it calls the attach(9E) entry point of the driver.  The device driver is responsible for managing the numbering of minor nodes.  In the case of the sd or ssd drivers, a total of 16 minor nodes (8 character, 8 block) are needed for each disk device instance, as each disk is allowed 8 partitions.  These drivers construct minor numbers by shifting the instance number left 3 bits to make room for these 16 minor nodes.

In a 32-bit kernel or a 32-bit application, both the major and minor numbers for a device must fit into a dev_t structure, which is 32 bits in size, so 14 bits are reserved for the major number and 18 bits are reserved for the minor number.  In the 64-bit kernel or a 64-bit application, the dev_t structure is 64 bits, so 32 bits are reserved for the major number and 32 bits are reserved for the minor number.  However, the 64-bit kernel supports 32-bit applications (e.g., ls) which may need to obtain minor numbers from the kernel.  So even in the 64-bit kernel, the size of minor numbers is limited to 18 bits. For the sd and ssd drivers, this limits device instance numbers to 15 bits.

Hence minor numbers must be less than 262144 (0x40000), and sd and ssd instance numbers must be less than 32768 (0x8000).  When multiple disk arrays are attached to a system, this limit can be reached.  For instance, a system with 8 controllers, 32 targets per controller, and 128 luns per target, will reach this limit (reference Bug 14995322 - SUNBT4259471).  Multipathing further exacerbates the problem as an instance is consumed for each path to each device.

There are also circumstances under which systems with fewer disk devices will encounter this problem.

The kernel will create sd device instances in accordance with the entries in both the /etc/path_to_inst and /kernel/drv/sd.conf file.  If entries for new luns are added to the sd.conf file, a reconfiguration reboot will create new instances and new lines in path_to_inst for those instances.  However, if lun entries are removed from the sd.conf file, a reconfiguration reboot will delete neither the corresponding instances nor the corresponding lines in path_to_inst.  These policies meet the purpose of the path_to_inst file, which is to ensure the persistence of instance numbers across reboots.

Now suppose a system has 9 controllers with 4096 sd devices attached to each of controllers 0-5.  If the sd devices on controllers 0-2 are moved to controllers 6-8 and a reconfiguration reboot is performed, the sd instance numbers will exceed  32768.

Changes

Cause