openssl speed -engine pkcs11 rsa fails with errors RSA sign failure. No RSA sign will be done. (Doc ID 1376708.1)

Last updated on JULY 29, 2016

Applies to:

Solaris Operating System - Version 10 3/05 to 10 1/13 U11 [Release 10.0]
Information in this document applies to any platform.

Symptoms

Running the following openssl command on a Solaris 10 LDOM would return several errors:

/usr/sfw/bin/openssl speed -engine pkcs11 rsa


RSA sign failure. No RSA sign will be done.
538:error:80088078:lib(128):PK11_SETUP_SESSION:C_OpenSession failed:/on10/build-nd/F10U10B17/usr/src/common/openssl/crypto/engine/hw_pk11_err.c:248:PK11 CK_RV=0X3
538:error:8107107C:lib(129):func(113):reason(124):/on10/build-nd/F10U10B17/usr/src/common/openssl/crypto/engine/hw_pk11_err.c:248:PK11 CK_RV=0X62
RSA verify failure. No RSA verify will be done.
538:error:04077077:rsa routines:RSA_verify:wrong signature length:/on10/build-nd/F10U10B17/usr/src/common/openssl/crypto/rsa/rsa_sign.c:154:

It was observed that this was done in the primary domain when the with the MAU enabled for the CPU core. When the MAU was disabled with the following command the openssl command would run but slowly.

ldm remove-mau -f 1 primary



Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms