[Bug]: Get a certificate via SCEP fails with: scep post request failed: crypto/rsa: verification error

[birdie1]

Steps to Reproduce

We set up our step ca a few days ago. After setting it up it was possible to get certificates via scep. Today I tried it again and it is not possible anymore.

After I found this message at startup:2024/02/19 13:06:36 failed validating SCEP authority: SCEP provisioner "802.1x" does not have a decrypter certificate
I added a decrpyter certificate to my config (Thanks to this bug request #1560). This message is gone, but the scep error is still there.

I tried setting the enviroment variable STEPDEBUG=1, but the output of step-ca still only show the error without any debug information.

Your Environment

• OS - Debian 12
• step-ca Version - 0.25.2 and 0.25.3-rc5-36-gbb296c9d

Expected Behavior

Get a certificate via SCEP.

Actual Behavior

Error: scep post request failed: crypto/rsa: verification error

Additional Context

Config:

{
	"root": "/etc/step-ca/certs/root_ca.crt",
	"federatedRoots": null,
	"crt": "/etc/step-ca/certs/client_ca.crt",
        "key": "pkcs11:id=01;object=***",
	"kms": {
        	"type": "pkcs11",
        	"uri": "pkcs11:module-path=/usr/lib/x86_64-linux-gnu/pkcs11/opensc-pkcs11.so;model=PKCS%2315%20emulated;manufacturer=www.CardContact.de;serial=***;id=%10;object=***;type=public;pin-value=***"
	},
	"address": ":8443",
        "insecureAddress": ":8080",
	"crl": {
		"enabled": true,
		"cacheDuration": "24h",
		"generateOnRevoke": true,
                "idpURL": "http://***/crl"
	},
	"dnsNames": [
                "***"
	],
	"logger": {
		"format": "text"
	},
	"db": {
		"type": "postgresql",
		"dataSource": "postgresql://***@127.0.0.1:5432/",
		"database": "stepca"
	},
	"authority": {
                "claims": {
                        "minTLSCertDuration": "24h",
                        "maxTLSCertDuration": "87600h",
                        "defaultTLSCertDuration": "8760h",
                        "disableRenewal": false,
                        "allowRenewalAfterExpiry": true
                },
	        "policy": {
                	"x509": {
                        	"allow": {
                                	        "dns": ["***"]
                        	        }
                	}
        	},
		"provisioners": [
			{
				"type": "JWK",
				"name": "***",
				"key": {
					"use": "sig",
					"kty": "EC",
					"kid": "***",
					"crv": "P-256",
					"alg": "ES256",
					"x": "***",
					"y": "***"
				},
				"encryptedKey": "***"
			},
			{
				"type": "SCEP",
				"name": "802.1x",
				"challenge": "***",
				"minimumPublicKeyLength": 2048,
                                "decrypterCertificate": "***",
                                "decrypterKeyPEM": "***",
                                "decrypterKeyPassword": "***",
				"includeRoot": true,
				"forceCN": true,
				"encryptionAlgorithmIdentifier": 2,
				"options": {
					"x509": {"templateFile": "templates/x509/leaf-crl.tpl"},
					"ssh": {}
				},
				"claims": {
					"enableSSHCA": true,
					"disableRenewal": false,
					"allowRenewalAfterExpiry": true,
					"minTLSCertDuration": "24h",
					"maxTLSCertDuration": "87600h",
					"defaultTLSCertDuration": "43800h"
				}
			}
		],
		"template": {},
		"backdate": "1m0s"
	},
	"tls": {
		"cipherSuites": [
			"TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256",
			"TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256"
		],
		"minVersion": 1.2,
		"maxVersion": 1.3,
		"renegotiation": false
	},
	"commonName": "Step Online CA"
}

Log Output:

2024/02/19 14:07:53 Building new tls configuration using step-ca x509 Signer Interface
2024/02/19 14:07:54 Starting Smallstep CA/0.25.3-rc5-36-gbb296c9d (linux/amd64)
2024/02/19 14:07:54 Documentation: https://u.step.sm/docs/ca
2024/02/19 14:07:54 Community Discord: https://u.step.sm/discord
2024/02/19 14:07:54 Config file: /etc/step-ca/config/ca.json
2024/02/19 14:07:54 The primary server URL is https://***:8443
2024/02/19 14:07:54 Root certificates are available at https://***:8443/roots.pem
2024/02/19 14:07:54 Additional configured hostnames: ***
2024/02/19 14:07:54 X.509 Root Fingerprint: ***
2024/02/19 14:07:54 Serving HTTPS on :8443 ...
2024/02/19 14:07:54 Serving HTTP on :8080 ...
INFO[0009]                                               duration="196.663µs" duration-ns=196663 fields.time="2024-02-19T14:08:00+01:00" method=GET name=ca path="/scep/802.1x?operation=GetCACert" protocol=HTTP/1.0 referer= remote-address="::1" request-id=cn9l5c6mrdksekp7edm0 size=3679 status=200 user-agent=Go-http-client/1.1 user-id=
INFO[0009]                                               duration="42.702µs" duration-ns=42702 fields.time="2024-02-19T14:08:00+01:00" method=GET name=ca path="/scep/802.1x?operation=GetCACaps" protocol=HTTP/1.0 referer= remote-address=127.0.0.1 request-id=cn9l5c6mrdksekp7edmg size=66 status=200 user-agent=Go-http-client/1.1 user-id=
ERRO[0009]                                               duration="736.591µs" duration-ns=736591 error="scep post request failed: crypto/rsa: verification error" fields.time="2024-02-19T14:08:00+01:00" method=POST name=ca path="/scep/802.1x?operation=PKIOperation" protocol=HTTP/1.0 referer= remote-address="::1" request-id=cn9l5c6mrdksekp7edn0 size=57 status=500 user-agent=Go-http-client/1.1 user-id=

Contributing

Vote on this issue by adding a 👍 reaction.
To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

[hslatman]

@birdie1 what SCEP client are you using? I see it reports as Go-http-client/1.1, but I'm curious what the actual program is. And do you access the SCEP endpoint using HTTPS, or HTTP?

We generally advise to use "includeRoot": false, (you can also leave it out; false is the default), because we've seen some weird behavior with certain clients if it is included, especially on renewals.

Seeing "crt": "/etc/step-ca/certs/client_ca.crt", is a bit surprising, but if that's what you configured before, then I guess you configured your intermediate there? Is the intermediate an ECDSA or RSA certificate?

[birdie1]

Currently we are using the scepclient from https://github.com/micromdm/scep for testing purposes. We can access the SCEP endpoint via http and https, but both having the same error.

I removed "includeRoot": false, but it didn't change anything.

Yes, the whole step ca config is under /etc/step-ca.
From my understanding it is RSA, here is the opennsl output:

root@ca1 ~ # openssl x509 -in /etc/step-ca/certs/client_ca.crt -text -noout
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            ***
        Signature Algorithm: sha512WithRSAEncryption
        Issuer: ***
        Validity
            Not Before: Mar 27 12:26:16 2023 GMT
            Not After : Mar 24 12:26:16 2033 GMT
        Subject: ***
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    ***
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Subject Key Identifier: 
                ***
            X509v3 Authority Key Identifier: 
                ***
            X509v3 Basic Constraints: critical
                CA:TRUE, pathlen:0
            X509v3 Key Usage: critical
                Digital Signature, Certificate Sign, CRL Sign
    Signature Algorithm: sha512WithRSAEncryption
    Signature Value:
        ***

[hslatman]

I removed "includeRoot": false, but it didn't change anything.

👍

Yes, the whole step ca config is under /etc/step-ca. From my understanding it is RSA, here is the opennsl output:

root@ca1 ~ # openssl x509 -in /etc/step-ca/certs/client_ca.crt -text -noout
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            ***
        Signature Algorithm: sha512WithRSAEncryption
        Issuer: ***
        Validity
            Not Before: Mar 27 12:26:16 2023 GMT
            Not After : Mar 24 12:26:16 2033 GMT
        Subject: ***
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    ***
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Subject Key Identifier: 
                ***
            X509v3 Authority Key Identifier: 
                ***
            X509v3 Basic Constraints: critical
                CA:TRUE, pathlen:0
            X509v3 Key Usage: critical
                Digital Signature, Certificate Sign, CRL Sign
    Signature Algorithm: sha512WithRSAEncryption
    Signature Value:
        ***

Looks OK too.

Currently we are using the scepclient from https://github.com/micromdm/scep for testing purposes. We can access the SCEP endpoint via http and https, but both having the same error.

I would advise to use a different SCEP client for testing to reduce the number of moving parts. I've used the one you're using for testing too, and I've observed some non-compatible behavior with it (might be on the CA side; could also be client side), I think especially with renewals. Also see e.g. micromdm/scep#217, which is another case of something I observed while testing with step-ca. I've had good experiences with https://github.com/certnanny/sscep for simple testing purposes, but we've done extensive testing with iOS, macOS and Windows too.

Note that we've forked the MicroMDM repository to maintain the corescep package (in consultation with the awesome people on their side): https://github.com/smallstep/scep. It's possible to build your own client using that lower level client, so that you have more control over the actual behavior. We've done a bit of work to integrate a SCEP client in our client tooling, but that's not available in the CLI yet.

[birdie1]

I tried sscep and the first time I got a certificate (Like with the micromdm as well). After that I receive again an error 500, but with a different error message.
Step CA output:

2024-02-21T09:23:21.692495+01:00 ca1 step-ca[19965]: time="2024-02-21T09:23:21+01:00" level=error duration="395.362µs" duration-ns=395362 error="scep post request failed: error decrypting encrypted pkcs7 content: pkcs7: no enveloped recipient for provided certificate" fields.time="2024-02-21T09:23:21+01:00" method=POST name=ca path="/scep/802.1x?operation=PKIOperation" protocol=HTTP/1.0 referer= remote-address="::1" request-id=cnar5uemrdkoss8hq6kg size=123 status=500 user-agent= user-id=

sscep output:

sscep enroll -u http://***/scep/802.1x -k privkey.pem -r csr.pem -l client.crt -c ca.crt -v
sscep: starting sscep, version 0.10.0
sscep: new transaction
sscep: transaction id: D41D8CD98F00B204E9800998ECF8427E
sscep: hostname: ***
sscep: directory: scep/802.1x
sscep: port: 80
sscep: SCEP_OPERATION_GETCAPS
sscep: connecting to ***:80
sscep: server response status code: 200, MIME header: text/plain
Renewal
SHA-1
SHA-256
AES
DES3
SCEPStandard
POSTPKIOperation
sscep:  Read request with transaction id: 99E814C8EE0F47E0AE4F055AB3AE47A3
sscep: generating selfsigned certificate
sscep: requesting certificate with serial number 0 and issuer ***
sscep: SCEP_OPERATION_ENROLL
sscep: sending certificate request
sscep: request data dump 
-----BEGIN CERTIFICATE REQUEST-----
***
-----END CERTIFICATE REQUEST-----
sscep: data payload size: 724 bytes
sscep: successfully encrypted payload
sscep: envelope size: 1215 bytes
sscep: creating outer PKCS#7
sscep: PKCS#7 data written successfully
sscep: payload size: 2841 bytes
sscep: connecting to ***:80
sscep: server response status code: 500, MIME header: text/plain
sscep: wrong (or missing) MIME content type
sscep: error while sending message

Any ideas what is wrong there?
Is there any change to get step ca to produce more output?

[hslatman]

@birdie1 I believe I've seen that same behavior as well, although I don't find that specific error message in my notes. I think sscep tries to use an existing certificate in the second request or changes its behavior slightly in some other manner on the renewal. If you remove the privkey.pem, client.crt and csr.pem, and retry, does it work again then?

Based on the two different errors for the two SCEP clients, it seems it might be related to the SCEP client maybe using a wrong recipient public key.

What is the actual SCEP client you're using / integrating with? Currently our CA is in use mostly with iOS, macOS and Windows clients, and for those the current (default) configuration options work well. It has to be noted that these generally request a fully new certificate, not using SCEP renewal, so maybe your problem is related to that? I may be able to try testing it with a client that's closer to your environment to see what's off.

We'll be working on logging improvements in the coming weeks across our stack, so improvements in this area can be expected.

[birdie1]

@birdie1 I believe I've seen that same behavior as well, although I don't find that specific error message in my notes. I think sscep tries to use an existing certificate in the second request or changes its behavior slightly in some other manner on the renewal. If you remove the privkey.pem, client.crt and csr.pem, and retry, does it work again then?

After creating a completely new key and csr (with other CN as well) the error persist.

Based on the two different errors for the two SCEP clients, it seems it might be related to the SCEP client maybe using a wrong recipient public key.

What do you mean with this? What can I change to use a valid recipient public key?

What is the actual SCEP client you're using / integrating with? Currently our CA is in use mostly with iOS, macOS and Windows clients, and for those the current (default) configuration options work well. It has to be noted that these generally request a fully new certificate, not using SCEP renewal, so maybe your problem is related to that? I may be able to try testing it with a client that's closer to your environment to see what's off.

We want to use it for our certifcate based 802.1x network authentication. We use ansible for linux, baramundi for windows and Vmware Airwatch for our mobile devices. The SCEP we need for Vmware Airwatch. For the others we can use scripts to provide the certificate with step cli over ssh.

[birdie1]

I found at least one thing:
It is working if I use the decrypter certificate, instead of the ca certificate on the sscep cmd:

sscep enroll -u http://ca-clients.disy.net/scep/802.1x -k privkey.pem -r csr.pem -l client.crt -c decryptor.crt -v

[hslatman]

Sounds plausible. The decrypter certificate is the one used to encrypt the message against, and thus used as the recipient. Some clients may need explicit configuration of the recipient, or else might pick the wrong certificate to encrypt against. One of the reasons we added support for the decrypter configuration was to be able to (continue to) use EC keys for signing, but still support SCEP easily. Before we added support, one was required to configure an RSA intermediate (at least) or an RSA chain. In SCEP jargon the decrypter is also known as the registration authority, but we abstained from calling it like that in step-ca, as we already had this concept, and is used in a somewhat (but not exactly) similar way.