pem_read_bio_key has additional exceptions in openssl1.1.1 than openssl3

[liuxingbaoyu]

This is an issue at nodejs/node#42793.

In openssl3, exceptions generated by pem_read_bio_key_decoder are cleaned up internally.

But in openssl1.1.1 it will be preserved.

Use ERR_peek_last_error in nodejs to determine whether PEM_read_bio_PrivateKey is successful.

Is this an openssl1.1.1 bug or the wrong way to use it?

openssl3:

openssl/crypto/pem/pem_pkey.c

Lines 232 to 243 in 5cffc49

	ERR_set_mark();
	ret = pem_read_bio_key_decoder(bp, x, ossl_pw_pem_password, &pwdata,
	libctx, propq, selection);
	if (ret == NULL
	&& (BIO_seek(bp, pos) < 0
	|| (ret = pem_read_bio_key_legacy(bp, x,
	ossl_pw_pem_password, &pwdata,
	libctx, propq,
	selection)) == NULL))
	ERR_clear_last_mark();
	else
	ERR_pop_to_mark();

c++ Example:

ERR_clear_error();
EVP_PKEY* key = PEM_read_bio_PrivateKey(bio, nullptr, PasswordCallback, passphrase);

unsigned long err = ERR_peek_last_error();
if (err != 0)
  key = NULL;

js Example:

const { privateKey, publicKey } = crypto.generateKeyPairSync('rsa', {
  modulusLength: 2048,
  publicKeyEncoding: {
    type: 'spki',
    format: 'pem'
  },
  privateKeyEncoding: {
    type: 'pkcs8',
    format: 'pem',
    cipher: 'aes-128-ecb',
    passphrase: 'abcdef'
  }
});

[paulidale]

It's more a limitation in the new world of 3.0 decoders.

[liuxingbaoyu]

Thanks for your answer!

What should I do to make openssl1.1.1 behave the same as openssl3? Without modifying the openssl source code.

[paulidale]

ERR_set_mark(); ... ERR_pop_to_mark(); might work. #ifdef would be the alternative.

[liuxingbaoyu]

pkey->reset(PEM_read_bio_PrivateKey(bio.get(),
                                    nullptr,
                                    PasswordCallback,
                                    &passphrase));
// OpenSSL can fail to parse the key but still return a non-null pointer.
unsigned long err = ERR_peek_error();  // NOLINT(runtime/int)
if (err != 0) 
  pkey->reset();

The original code of nodejs used ERR_peek_error to determine whether PEM_read_bio_PrivateKey was successful.

pem_read_bio_key_decoder is called inside PEM_read_bio_PrivateKey and I can't find a way to just ignore this type of exception.

[t8m]

The version 1.1.1 is not supported anymore. Closing. If this is still an issue with currently supported releases, please open a new issue.