Exceptions when using privateEncrypt/privateDecrypt with certain ciphers

[PANCHO7532B]

Version

v20.0.0-nightly2022110286088ab78e

Platform

Linux EURO01 5.4.0-131-generic #147-Ubuntu SMP Fri Oct 14 17:07:22 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

Subsystem

crypto

What steps will reproduce the bug?

(continuation of #40814)
I made an snippet that tests RSA key pair generations on every cipher made by crypto.getCiphers(), an fix suggested by #42793 only applies for ECB ciphers, but most cipher variants are still unusable by the same error or variants of it or segfaults

const crypto = require("crypto");
crypto.getCiphers().forEach((cipherString) => {
    console.log(`Testing: ${cipherString}`);
    if(cipherString == "des3-wrap" || cipherString == "aes128-wrap" || cipherString == "aes192-wrap" || cipherString == "aes256-wrap" || cipherString == "id-aes128-wrap" || cipherString == "id-aes128-wrap-pad" || cipherString == "id-aes192-wrap" || cipherString == "id-aes192-wrap-pad" || cipherString == "id-aes256-wrap" || cipherString == "id-aes256-wrap-pad") {
        console.log(`\x1b[33m[SKIPPED] ${cipherString} due to segmentation fault`);
        console.log("\x1b[37m"); // turn text back to white
        return;
    }
    try {
        let { privateKey, publicKey } = crypto.generateKeyPairSync("rsa", {
            modulusLength: 2048,
            publicKeyEncoding: {
                type: "spki",
                format: "pem"
            },
            privateKeyEncoding: {
                type: 'pkcs8',
                format: 'pem',
                cipher: cipherString,
                passphrase: "abcdef"
            }
        });
        const encryptedString = crypto.privateEncrypt({
            key: privateKey,
            passphrase: "abcdef"
        }, Buffer.from("The quick brown fox jumps over the lazy dog")).toString("base64");
        const decryptedString = crypto.publicDecrypt(publicKey, Buffer.from(encryptedString, "base64")).toString();
        console.log(`\x1b[32m[PASS]`);
        console.log(`Encrypted: ${encryptedString}`);
        console.log(`Decrypted: ${decryptedString}`);
    } catch(err) {
        console.log(`\x1b[31m[FAILED] ${err.stack}`);
    }
    console.log("\x1b[37m"); // turn text back to white
});

How often does it reproduce? Is there a required condition?

These are the broken ciphers:

aes-128-ccm - error:0300006B:digital envelope routines::unsupported cipher
aes-128-ctr - error:0680006C:asn1 encoding routines::cipher has no object identifier
aes-128-gcm - error:0300007A:digital envelope routines::cipher parameter error
aes-128-ocb - error:0680006C:asn1 encoding routines::cipher has no object identifier
aes-128-xts - error:0300006B:digital envelope routines::unsupported cipher
aes-192-ccm - error:0300006B:digital envelope routines::unsupported cipher
aes-192-ctr - error:0680006C:asn1 encoding routines::cipher has no object identifier
aes-192-gcm - error:0300007A:digital envelope routines::cipher parameter error
aes-192-ocb - error:0680006C:asn1 encoding routines::cipher has no object identifier
aes-256-ccm - error:0300006B:digital envelope routines::unsupported cipher
aes-256-ctr - error:0680006C:asn1 encoding routines::cipher has no object identifier
aes-256-gcm - error:0300007A:digital envelope routines::cipher parameter error
aes-256-ocb - error:0680006C:asn1 encoding routines::cipher has no object identifier
aes-256-xts - error:0300006B:digital envelope routines::unsupported cipher
aria-128-ccm - error:0300006B:digital envelope routines::unsupported cipher
aria-128-cfb1 - error:0680006C:asn1 encoding routines::cipher has no object identifier
aria-128-cfb8 - error:0680006C:asn1 encoding routines::cipher has no object identifier
aria-128-gcm - error:0300007A:digital envelope routines::cipher parameter error
aria-192-ccm - error:0300006B:digital envelope routines::unsupported cipher
aria-192-cfb1 - error:0680006C:asn1 encoding routines::cipher has no object identifier
aria-192-cfb8 - error:0680006C:asn1 encoding routines::cipher has no object identifier
aria-192-gcm - error:0300007A:digital envelope routines::cipher parameter error
aria-256-ccm - error:0300006B:digital envelope routines::unsupported cipher
aria-256-cfb1 - error:0680006C:asn1 encoding routines::cipher has no object identifier
aria-256-cfb8 - error:0680006C:asn1 encoding routines::cipher has no object identifier
aria-256-gcm - error:0300007A:digital envelope routines::cipher parameter error
camellia-128-cfb1 - error:0680006C:asn1 encoding routines::cipher has no object identifier
camellia-128-cfb8 - error:0680006C:asn1 encoding routines::cipher has no object identifier
camellia-192-cfb1 - error:0680006C:asn1 encoding routines::cipher has no object identifier
camellia-192-cfb8 - error:0680006C:asn1 encoding routines::cipher has no object identifier
camellia-256-cfb1 - error:0680006C:asn1 encoding routines::cipher has no object identifier
camellia-256-cfb8 - error:0680006C:asn1 encoding routines::cipher has no object identifier
chacha20 - error:0680006C:asn1 encoding routines::cipher has no object identifier
chacha20-poly1305 - error:0680006C:asn1 encoding routines::cipher has no object identifier
des-ede-cbc - error:0680006C:asn1 encoding routines::cipher has no object identifier
des-ede-cfb - error:0680006C:asn1 encoding routines::cipher has no object identifier
des-ede-ofb - error:0680006C:asn1 encoding routines::cipher has no object identifier
des-ede3 - error:0680006C:asn1 encoding routines::cipher has no object identifier
des-ede3-cfb - error:0308010C:digital envelope routines::unsupported
des-ede3-cfb1 - error:0308010C:digital envelope routines::unsupported
des-ede3-cfb8 - error:0308010C:digital envelope routines::unsupported
des-ede3-ecb - error:0680006C:asn1 encoding routines::cipher has no object identifier
des-ede3-ofb - error:0680006C:asn1 encoding routines::cipher has no object identifier
id-aes128-CCM - error:0300006B:digital envelope routines::unsupported cipher
id-aes128-GCM - error:0300007A:digital envelope routines::cipher parameter error
id-aes192-CCM - error:0300006B:digital envelope routines::unsupported cipher
id-aes192-GCM - error:0300007A:digital envelope routines::cipher parameter error
id-aes256-CCM - error:0300006B:digital envelope routines::unsupported cipher
id-aes256-GCM - error:0300007A:digital envelope routines::cipher parameter error
id-smime-alg-CMS3DESwrap - error:1C800066:Provider routines::cipher operation failed

Additionally, the following ciphers throw an Segmentation fault:

aes128-wrap
aes192-wrap
aes256-wrap
id-aes128-wrap
id-aes128-wrap-pad
id-aes192-wrap
id-aes192-wrap-pad
id-aes256-wrap
id-aes256-wrap-pad

With the exception of des3-wrap who fails with error:1C800066:Provider routines::cipher operation failed before throwing the following as I try a few more times to encrypt:

corrupted size vs. prev_size
Aborted (core dumped)

What is the expected behavior?

An successful encryption/decryption

What do you see instead?

An exception (it varies depending the algorithm used)
For example, when using aes-256-ccm:

Error: error:0300006B:digital envelope routines::unsupported cipher
    at Object.generateKeyPairSync (node:internal/crypto/keygen:108:63)
    at /home/pancho7532/Downloads/node-v20.0.0-nightly2022110286088ab78e-linux-x64/bin/example.js:10:48
    at Array.forEach (<anonymous>)
    at Object.<anonymous> (/home/pancho7532/Downloads/node-v20.0.0-nightly2022110286088ab78e-linux-x64/bin/example.js:2:21)
    at Module._compile (node:internal/modules/cjs/loader:1205:14)
    at Module._extensions..js (node:internal/modules/cjs/loader:1259:10)
    at Module.load (node:internal/modules/cjs/loader:1068:32)
    at Module._load (node:internal/modules/cjs/loader:909:12)
    at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:82:12)
    at node:internal/main/run_main_module:23:47

For aes-256-ctr:

Error: error:0680006C:asn1 encoding routines::cipher has no object identifier
    at Object.generateKeyPairSync (node:internal/crypto/keygen:108:63)
    at /home/pancho7532/Downloads/node-v20.0.0-nightly2022110286088ab78e-linux-x64/bin/example.js:10:48
    at Array.forEach (<anonymous>)
    at Object.<anonymous> (/home/pancho7532/Downloads/node-v20.0.0-nightly2022110286088ab78e-linux-x64/bin/example.js:2:21)
    at Module._compile (node:internal/modules/cjs/loader:1205:14)
    at Module._extensions..js (node:internal/modules/cjs/loader:1259:10)
    at Module.load (node:internal/modules/cjs/loader:1068:32)
    at Module._load (node:internal/modules/cjs/loader:909:12)
    at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:82:12)
    at node:internal/main/run_main_module:23:47

For aes-256-gcm:

Error: error:0300007A:digital envelope routines::cipher parameter error
    at Object.generateKeyPairSync (node:internal/crypto/keygen:108:63)
    at /home/pancho7532/Downloads/node-v20.0.0-nightly2022110286088ab78e-linux-x64/bin/example.js:10:48
    at Array.forEach (<anonymous>)
    at Object.<anonymous> (/home/pancho7532/Downloads/node-v20.0.0-nightly2022110286088ab78e-linux-x64/bin/example.js:2:21)
    at Module._compile (node:internal/modules/cjs/loader:1205:14)
    at Module._extensions..js (node:internal/modules/cjs/loader:1259:10)
    at Module.load (node:internal/modules/cjs/loader:1068:32)
    at Module._load (node:internal/modules/cjs/loader:909:12)
    at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:82:12)
    at node:internal/main/run_main_module:23:47

For id-smime-alg-CMS3DESwrap:

Error: error:1C800066:Provider routines::cipher operation failed
    at Object.generateKeyPairSync (node:internal/crypto/keygen:108:63)
    at /home/pancho7532/Downloads/node-v20.0.0-nightly2022110286088ab78e-linux-x64/bin/example.js:10:48
    at Array.forEach (<anonymous>)
    at Object.<anonymous> (/home/pancho7532/Downloads/node-v20.0.0-nightly2022110286088ab78e-linux-x64/bin/example.js:2:21)
    at Module._compile (node:internal/modules/cjs/loader:1205:14)
    at Module._extensions..js (node:internal/modules/cjs/loader:1259:10)
    at Module.load (node:internal/modules/cjs/loader:1068:32)
    at Module._load (node:internal/modules/cjs/loader:909:12)
    at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:82:12)
    at node:internal/main/run_main_module:23:47

Additional information

On my opinion it's still kinda valid? Since this should work by all ciphers detailed by getCiphers()
Sorry in advance if this might not be valid.

[bnoordhuis]

cc @nodejs/crypto - I can reproduce the segfaults with e.g. aes128-wrap¹ so at the very least we should do something about those.

¹ nullptr dereference in WritePrivateKey() -> PEM_write_bio_PKCS8PrivateKey() -> lots of openssl -> asn1_ex_i2c()

[prettydiff]

I am not sure if this is related: nodejs/help#4211

I am starting to run out of combinations to try for a resolution. I did notice my OpenSSL error number is not listed in this thread among the many other OpenSSL error numbers: 03000096