Cannot get cloud-init phone_home to use self-signed certificate added to registry

[dark2phoenix]

Describe the bug

Using photon OS with cloud-init and Aria Automation's phone home feature. AA adds the following syntax to the cloud-init file:

phone_home:
  url: https://vra8va-1.lab.mccannical.net/provisioning/mgmt/cloud-init-phone-home-callback/4ccfac06-af53-4048-94ae-9577858292c1?token=e52f0716-b903-4de9-8ed4-a04ef4a38fb6
  post:
  - instance_id
  tries: 0

Call home constantly fails with:

2023-12-13 15:26:20,630 - url_helper.py[DEBUG]: [0/1] open 'https://vra8va-1.lab.mccannical.net/provisioning/mgmt/cloud-init-phone-home-callback/4ccfac06-af53-4048-94ae-9577858292c1?token=e52f0716-b903-4de9-8ed4-a04ef4a38fb6' with {'url': 'https://vra8va-1.lab.mccannical.net/provisioning/mgmt/cloud-init-phone-home-callback/4ccfac06-af53-4048-94ae-9577858292c1?token=e52f0716-b903-4de9-8ed4-a04ef4a38fb6', 'stream': False, 'allow_redirects': True, 'method': 'POST', 'headers': {'User-Agent': 'Cloud-Init/23.4'}} configuration
2023-12-13 15:26:20,701 - util.py[WARNING]: Failed to post phone home data to https://vra8va-1.lab.mccannical.net/provisioning/mgmt/cloud-init-phone-home-callback/4ccfac06-af53-4048-94ae-9577858292c1?token=e52f0716-b903-4de9-8ed4-a04ef4a38fb6 in 0 tries
2023-12-13 15:26:20,702 - util.py[DEBUG]: Failed to post phone home data to https://vra8va-1.lab.mccannical.net/provisioning/mgmt/cloud-init-phone-home-callback/4ccfac06-af53-4048-94ae-9577858292c1?token=e52f0716-b903-4de9-8ed4-a04ef4a38fb6 in 0 tries
Traceback (most recent call last):
  File "/usr/lib/python3.10/site-packages/cloudinit/config/cc_phone_home.py", line 195, in handle
    url_helper.read_file_or_url(
  File "/usr/lib/python3.10/site-packages/cloudinit/url_helper.py", line 87, in read_file_or_url
    return readurl(url, **kwargs)
  File "/usr/lib/python3.10/site-packages/cloudinit/url_helper.py", line 370, in readurl
    raise excps[-1]
cloudinit.url_helper.UrlError: HTTPSConnectionPool(host='vra8va-1.lab.mccannical.net', port=443): Max retries exceeded with url: /provisioning/mgmt/cloud-init-phone-home-callback/4ccfac06-af53-4048-94ae-9577858292c1?token=e52f0716-b903-4de9-8ed4-a04ef4a38fb6 (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1007)')))

certificate chain added to /etc/ssl/certs/ and rehash done. Verified that curl can properly work using new certificates:

curl -vvv https://vra8va-1.lab.mccannical.net/
*   Trying 172.28.7.223:443...
* Connected to vra8va-1.lab.mccannical.net (172.28.7.223) port 443 (#0)
* ALPN: offers http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
*  CAfile: /etc/pki/tls/certs/ca-bundle.crt
*  CApath: none
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN: server accepted http/1.1
* Server certificate:
*  subject: C=US; ST=North Carolina; L=Cary; O=McCannically Inclined; OU=Lab; CN=vra8.lab.mccannical.net
*  start date: Jun 15 16:01:31 2022 GMT
*  expire date: Dec 31 02:21:15 2023 GMT
*  subjectAltName: host "vra8va-1.lab.mccannical.net" matched cert's "vra8va-1.lab.mccannical.net"
*  issuer: DC=net; DC=mccannical; DC=home; CN=McCannical Issuing CA
*  SSL certificate verify ok.
* using HTTP/1.1
> GET / HTTP/1.1
> Host: vra8va-1.lab.mccannical.net
> User-Agent: curl/8.1.2
> Accept: */*
> 
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
< HTTP/1.1 200 OK
< Cache-Control: no-store
< Content-Type: text/html
< Date: Wed, 13 Dec 2023 15:58:17 GMT
< Etag: W/"64f19e64-2a1"
< Last-Modified: Fri, 01 Sep 2023 08:18:44 GMT
< Server: nginx
< Strict-Transport-Security: max-age=31536000; includeSubDomains
< Vary: Accept-Encoding
< X-Content-Type-Options: nosniff
< X-Frame-Options: sameorigin
< X-Xss-Protection: 1; mode=block
< Transfer-Encoding: chunked
< 
<!doctype html>
<html lang="en">
<head>
  <meta http-equiv="X-UA-Compatible" content="IE=edge"/>
  <meta charset="utf-8">
  <title>VMware Cloud Services</title>
  <base href="/">

  <meta name="viewport" content="width=device-width, initial-scale=1">
  <link rel="icon" type="image/x-icon" href="favicon.ico">
<link rel="stylesheet" href="styles.3edba6a455e4d892.css"></head>
<body>
  <landing-app>
    <span class="spinner main-spinner"></span>
  </landing-app>
<script src="runtime.2edbdc7857cd77cf.js" type="module"></script><script src="polyfills.fa8dceb6cccf1f72.js" type="module"></script><script src="main.340ab36d83fe813f.js" type="module"></script></body>
</html>
* Connection #0 to host vra8va-1.lab.mccannical.net left intact

This problem occurs with both Photon 4.0r2 and Photon 5.

Reproduction steps

1. Add stanza cited above to cloud-init
2. Add certificate chain to /etc/ssl/certs
3. run rehash_ca_certificates.sh to add local certs to trusted list
4. execute cloud-init
   ...

Expected behavior

cloud-init should honor the certificates that are trusted in the /etc/ssl/certificates directory after rehash_ca_certificates.sh is executed.

Additional context

No response

[sshedi]

What is the output of:

openssl s_client -connect vra8va-1.lab.mccannical.net:443 -showcerts

[dark2phoenix]

CONNECTED(00000003)
depth=2 CN = McCannical Offline Root CA
verify return:1
depth=1 DC = net, DC = mccannical, DC = home, CN = McCannical Issuing CA
verify return:1
depth=0 C = US, ST = North Carolina, L = Cary, O = McCannically Inclined, OU = Lab, CN = vra8.lab.mccannical.net
verify return:1
---
Certificate chain
 0 s:C = US, ST = North Carolina, L = Cary, O = McCannically Inclined, OU = Lab, CN = vra8.lab.mccannical.net
   i:DC = net, DC = mccannical, DC = home, CN = McCannical Issuing CA
   a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
   v:NotBefore: Jun 15 16:01:31 2022 GMT; NotAfter: Dec 31 02:21:15 2023 GMT
-----BEGIN CERTIFICATE-----
MIIHIzCCBgugAwIBAgITewAAAGDd044lbE+Q0AAAAAAAYDANBgkqhkiG9w0BAQsF
ADBnMRMwEQYKCZImiZPyLGQBGRYDbmV0MRowGAYKCZImiZPyLGQBGRYKbWNjYW5u
aWNhbDEUMBIGCgmSJomT8ixkARkWBGhvbWUxHjAcBgNVBAMTFU1jQ2FubmljYWwg
SXNzdWluZyBDQTAeFw0yMjA2MTUxNjAxMzFaFw0yMzEyMzEwMjIxMTVaMIGFMQsw
CQYDVQQGEwJVUzEXMBUGA1UECBMOTm9ydGggQ2Fyb2xpbmExDTALBgNVBAcTBENh
cnkxHjAcBgNVBAoTFU1jQ2FubmljYWxseSBJbmNsaW5lZDEMMAoGA1UECxMDTGFi
MSAwHgYDVQQDExd2cmE4LmxhYi5tY2Nhbm5pY2FsLm5ldDCCASIwDQYJKoZIhvcN
AQEBBQADggEPADCCAQoCggEBAOzX0Sp1qX4V8cSMYlAO2i2tt2pkewVKCdpSMxBl
envgHPgiA7kxIoCI8ksVhN4GzEYKL5y62Z8a+QFptR+8HRX068YUAT0CFLupg13B
ejPfD43e1/PW8aopngDE4PFoizwA1sAMLsB8U/VWNRHhQraaV/DOX17SzdQPmZ+m
gIwkbJ+gVtobzEKLgykvg1GdixLLtBTWr3XFtHyC+ZOssEH5HI3OI7Y1Q5vCx2sx
8JWmP7+8NGQWNEljqt7cIIvxpYV8XrhJ8CriWKmeu+bS2uMn65dZMhc9uMVWCuix
/RQnNkRdTSBSZciYr8GkcMtXhOXsFI+ZhE6zh/X7CLtV77cCAwEAAaOCA6cwggOj
MA4GA1UdDwEB/wQEAwIF4DAdBgNVHSUEFjAUBggrBgEFBQcDAQYIKwYBBQUHAwIw
eQYDVR0RBHIwcIIbdnJhOHZhLTEubGFiLm1jY2FubmljYWwubmV0ght2cmE4dmEt
Mi5sYWIubWNjYW5uaWNhbC5uZXSCG3ZyYTh2YS0zLmxhYi5tY2Nhbm5pY2FsLm5l
dIIXdnJhOC5sYWIubWNjYW5uaWNhbC5uZXQwHQYDVR0OBBYEFAg65AuwM+d/xRW1
aa7xAtE8fDs5MB8GA1UdIwQYMBaAFCFd7seugULtmOnH1SzrmpaAULdPMIIBLAYD
VR0fBIIBIzCCAR8wggEboIIBF6CCAROGgcdsZGFwOi8vL0NOPU1jQ2FubmljYWwl
MjBJc3N1aW5nJTIwQ0EsQ049REMxLENOPUNEUCxDTj1QdWJsaWMlMjBLZXklMjBT
ZXJ2aWNlcyxDTj1TZXJ2aWNlcyxDTj1Db25maWd1cmF0aW9uLERDPWhvbWUsREM9
bWNjYW5uaWNhbCxEQz1uZXQ/Y2VydGlmaWNhdGVSZXZvY2F0aW9uTGlzdD9iYXNl
P29iamVjdENsYXNzPWNSTERpc3RyaWJ1dGlvblBvaW50hkdodHRwOi8vY3JsLmhv
bWUubWNjYW5uaWNhbC5uZXQvQ2VydEVucm9sbC9NY0Nhbm5pY2FsJTIwSXNzdWlu
ZyUyMENBLmNybDCCAUUGCCsGAQUFBwEBBIIBNzCCATMwgcMGCCsGAQUFBzAChoG2
bGRhcDovLy9DTj1NY0Nhbm5pY2FsJTIwSXNzdWluZyUyMENBLENOPUFJQSxDTj1Q
dWJsaWMlMjBLZXklMjBTZXJ2aWNlcyxDTj1TZXJ2aWNlcyxDTj1Db25maWd1cmF0
aW9uLERDPWhvbWUsREM9bWNjYW5uaWNhbCxEQz1uZXQ/Y0FDZXJ0aWZpY2F0ZT9i
YXNlP29iamVjdENsYXNzPWNlcnRpZmljYXRpb25BdXRob3JpdHkwawYIKwYBBQUH
MAKGX2h0dHA6Ly9jcmwuaG9tZS5tY2Nhbm5pY2FsLm5ldC9DZXJ0RW5yb2xsL0RD
MS5ob21lLm1jY2FubmljYWwubmV0X01jQ2FubmljYWwlMjBJc3N1aW5nJTIwQ0Eu
Y3J0MD4GCSsGAQQBgjcVBwQxMC8GJysGAQQBgjcVCIX+9CiB/6gphYGRBoKV/HOG
jolYgVSE1Y5rgaaRUwIBZAIBAzANBgkqhkiG9w0BAQsFAAOCAQEAhUqIqlSPR7dp
rPc/vkuyhF2njbHzdejTSksDLKvU3432EtS1s2NN6pMo97GKwxvvRpoX47CkUQTW
sS1ALhjjPJ1c0It4uPm1TOFaV451ybPx90SrpMhKrfKRY4V5W7BCgr3gb6BscTUm
ikv6yud6eOas6PyV2enxa+nHEyrlxcN9RzuOHOVvbmDh65A7vyxmoeQ6sLbLaT0A
BwXCmV599vR6HXSzMFr4Ngq3YZf66Z9+knR4l3jx5y+slxXgeKe3KkJI565hYyOn
77wf1XTWTMdaEvncTRtQtCBZJ63vdCnAlbd/roKfjciWvA6RHLpU3AJ2o7OkbKrG
8XHZNzE1Mw==
-----END CERTIFICATE-----
 1 s:DC = net, DC = mccannical, DC = home, CN = McCannical Issuing CA
   i:CN = McCannical Offline Root CA
   a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
   v:NotBefore: Dec 31 02:11:15 2018 GMT; NotAfter: Dec 31 02:21:15 2023 GMT
-----BEGIN CERTIFICATE-----
MIIEejCCA2KgAwIBAgITEQAAAAKeo/gZwsD0awAAAAAAAjANBgkqhkiG9w0BAQsF
ADAlMSMwIQYDVQQDExpNY0Nhbm5pY2FsIE9mZmxpbmUgUm9vdCBDQTAeFw0xODEy
MzEwMjExMTVaFw0yMzEyMzEwMjIxMTVaMGcxEzARBgoJkiaJk/IsZAEZFgNuZXQx
GjAYBgoJkiaJk/IsZAEZFgptY2Nhbm5pY2FsMRQwEgYKCZImiZPyLGQBGRYEaG9t
ZTEeMBwGA1UEAxMVTWNDYW5uaWNhbCBJc3N1aW5nIENBMIIBIjANBgkqhkiG9w0B
AQEFAAOCAQ8AMIIBCgKCAQEA3aNLOwtJbiHxAoyvghrGwrzNIJUBzKORVPs8ZC2U
+nd/PbC0nwINt3ys3PlP59OvnD5fJPmwQFy/OnCszglezfM2wGvYGhOMd53DtbTE
lSEX/F+mxTDOrgAoXUbxrRUJhDvaiBRCMZDDEK0LGTkGwT/YPvOOgphqc/yEJIT/
JLIYWdoWVSoYFuLbO54IcjoSAcUNQDV1xPGTAAt5mP6m92mSS7qlXm9uwJhYCZFX
AmrBareRNI6E+tfy9lnsTmRi7gT82cxppoXolV0SzCckjxJJZzXa0b03HopDyfpq
kMxCRj3aZnJM7ZiWj77/KBDQaQx21ad882oP0t3jeedGnQIDAQABo4IBXzCCAVsw
EAYJKwYBBAGCNxUBBAMCAQAwHQYDVR0OBBYEFCFd7seugULtmOnH1SzrmpaAULdP
MBkGCSsGAQQBgjcUAgQMHgoAUwB1AGIAQwBBMAsGA1UdDwQEAwIBhjAPBgNVHRMB
Af8EBTADAQH/MB8GA1UdIwQYMBaAFEN3DrVu/6i32TL4orzTyxKNwSR0MF8GA1Ud
HwRYMFYwVKBSoFCGTmh0dHA6Ly9jcmwuaG9tZS5tY2Nhbm5pY2FsLm5ldC9DZXJ0
RW5yb2xsL01jQ2FubmljYWwlMjBPZmZsaW5lJTIwUm9vdCUyMENBLmNybDBtBggr
BgEFBQcBAQRhMF8wXQYIKwYBBQUHMAKGUWh0dHA6Ly9jcmwuaG9tZS5tY2Nhbm5p
Y2FsLm5ldC9DZXJ0RW5yb2xsL2NhX01jQ2FubmljYWwlMjBPZmZsaW5lJTIwUm9v
dCUyMENBLmNydDANBgkqhkiG9w0BAQsFAAOCAQEASFTMDr20Ja2pqrbm1mU0PQBm
gpHJNVPEZADH5YXAgiXbh/B5XWv5d9Jv1qJjypcfT7D96kqs0YjX0kB8eXLofP+O
4h/i+A5BaDN0Xo02L3+kH//yrTzRxtuTNRysl0lHz0Lo/EnUqrINHe1QV0pFqXAf
nYpViO7+wEOhPWfUMH0qeBZVr4lhhpQmPPiPY5NnvtA8v71MAM+J7CDcZ3uoKWDM
+DcU4aDTlYdDvMCriwOuCsPWp/Jag9mA5aR2iCU32/u3y+QXhx07MzotrUhBHiHg
vahAIXBSjKn300P9TGLTcaL3DAi9/W0PX5v810wZ5HT+dgp2/0ZrB0QwOOZnsw==
-----END CERTIFICATE-----
 2 s:CN = McCannical Offline Root CA
   i:CN = McCannical Offline Root CA
   a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
   v:NotBefore: Dec 31 01:56:04 2018 GMT; NotAfter: Dec 31 02:06:02 2028 GMT
-----BEGIN CERTIFICATE-----
MIIDJTCCAg2gAwIBAgIQQdnZ6Pz1R4RK2OD5yCA56jANBgkqhkiG9w0BAQsFADAl
MSMwIQYDVQQDExpNY0Nhbm5pY2FsIE9mZmxpbmUgUm9vdCBDQTAeFw0xODEyMzEw
MTU2MDRaFw0yODEyMzEwMjA2MDJaMCUxIzAhBgNVBAMTGk1jQ2FubmljYWwgT2Zm
bGluZSBSb290IENBMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAk0k6
+4Y0zZQXBXi+MNk2mUrKkOsDRD2cTSvSErBsg7euZhp8eR6kirvHDtNgjSpAs34V
H80zmhV1cWPI5T04RxY0kFpzNPaJG4j4yFs6e5qk9xQPiOGbHRNlH266m9KiXcXa
h0LDs7c3tyX0OByvNR3auFE9glvVVHFilTqwnwKZhI4nai5ygJr9O4y/fVia2D4b
6ePo23M3tf6Cs+JIAACHVYwftabMaWoCohJMleFX8gV7TG5qOaae7Xi/YDPwapiB
FT2OlA6pVsr9bNvr9A7ibGTZvTc5AEtK1Yq7UV44PVnVZWp8/JpBkhnlxkOkrE6Q
bXFjkacPoJrLRQ+QewIDAQABo1EwTzALBgNVHQ8EBAMCAYYwDwYDVR0TAQH/BAUw
AwEB/zAdBgNVHQ4EFgQUQ3cOtW7/qLfZMviivNPLEo3BJHQwEAYJKwYBBAGCNxUB
BAMCAQAwDQYJKoZIhvcNAQELBQADggEBADhQNygzVzjSCo2++WEq+C+ToJda4OX6
3WqHs1RzgyfnZIzyGZys0Ety++Lc0raHQWnbS1ATAsYheGt8oLqsARR17fFTtvwK
2hwc6ySP8x4YQAQ1rI2/DmkBNM84qcwHNc5wzvuP8uNGMSUxf28G3UnAeoKmGpOk
RfaFrAzN0rrWtqrtiioP61dVad8WZjyD24+c+j9WsHDbSi1oDrFvdygtc4zXHDGQ
zgQvaY/56aV5N9oqfZn+BtLTgtdLvc+MW5nFnKw+Z0GkcdxpwDjmBc9Am+ByH7WJ
MSBqSZxT+sbkgiblYflECY6edv9iU5hEHkmmZSkzKQ/Bxnik5igoOZA=
-----END CERTIFICATE-----
---
Server certificate
subject=C = US, ST = North Carolina, L = Cary, O = McCannically Inclined, OU = Lab, CN = vra8.lab.mccannical.net
issuer=DC = net, DC = mccannical, DC = home, CN = McCannical Issuing CA
---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 4340 bytes and written 397 bytes
Verification: OK
---
New, TLSv1.3, Cipher is TLS_AES_128_GCM_SHA256
Server public key is 2048 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---
---
Post-Handshake New Session Ticket arrived:
SSL-Session:
    Protocol  : TLSv1.3
    Cipher    : TLS_AES_128_GCM_SHA256
    Session-ID: 42078E3735B6AA9F5C16BB7191BDF98E22F912DBA4C02550EA5BC1FDDE6E724E
    Session-ID-ctx: 
    Resumption PSK: C45BA7F9669737A4752096ADE741EF92001F8A6B16C2B199ED9CA34CDD5EE69A
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 604800 (seconds)
    TLS session ticket:
    0000 - ff c1 d7 2e 29 5c d2 c7-c2 15 7f d1 5a 65 04 ba   ....)\......Ze..
    0010 - 4b af 2d 1a 0d 78 39 24-f1 d8 0a 7c f1 23 8c 6b   K.-..x9$...|.#.k
    0020 - 47 56 5a 7b 23 20 42 6f-c2 47 ed e5 7f 78 01 51   GVZ{# Bo.G...x.Q
    0030 - 44 0f 85 a8 fc 3b 9e 19-13 90 86 06 f6 36 a2 e9   D....;.......6..
    0040 - a2 67 19 77 e7 31 0b 8d-2c 63 b0 13 f4 e9 d2 ff   .g.w.1..,c......
    0050 - 51 2e 3b 33 d8 65 67 14-b3 0b 7a 7a fb 3f 4a ff   Q.;3.eg...zz.?J.
    0060 - b3 07 b0 3a 83 91 b1 82-ea cd 0b 39 6a d5 6e 88   ...:.......9j.n.
    0070 - fc                                                .

    Start Time: 1702563120
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
    Extended master secret: no
    Max Early Data: 0
---
read R BLOCK

HTTP/1.1 400 Bad Request
Content-Type: text/plain; charset=utf-8
Connection: close

400 Bad Requestclosed```

[dark2phoenix]

Any updates or ideas on this?

[dark2phoenix]

I"ll try that tomorrow and report back.

[dark2phoenix]

Here is the version of cloud-init data from photon 4r2:
cloud-init.noarch 23.4-2.ph4 @System
root@localhost [ ~ ]# cloud-init --version /usr/bin/cloud-init 23.4

I don't think it's possible for me to add the extra lines to a cloud-init configuration. Aria Automation creates that stanza automatically and seems to discard anything I put in the general cloud-init config. I'll continue to play around and see if I can figure out if I can maybe hand manipulate it.

[dark2phoenix]

While I mess around trying to get those post parameters added, is there any finer debug we can enable in cloud-init that might enumerate what's going on?

[dark2phoenix]

OK, so I've looked through the docs you sent me:

1. Salt and Aria Auto. have certs from my authority properly installed and recognized
2. Looked over those, and can't change the stanza it is 100% controlled by Aria Auto
3. Very familiar with these, but not sure exactly how they are relevant in this scenario
4. Setup and working properly - cloud-init waits for custom vSphere config spec to run and reboot the box before executing
5. I don't think this applies in this case. I am using a vSphere template with cloud-init already installed and configured
6. This is also I think irrelevant unless you want me re-write the cloud-config before the phone_home module runs (which I guess I could do, but not sure exactly how that helps)

This problem, to me, is between the phone_home module and the photon OS. It's like the phone_home module is not using the same certs registry that the OS is (thus it can't find the 3rd party certs I added to the configuration of the photon VM). normal wget sees it fine, but phone_home does not. Is there perhaps an alternate/different registry that I should be using? Normally I would have expected to just put the cert into the cacerts stanza in cloud-init, but I see that is specifically ignored when it's photon.

[dark2phoenix]

Thank you, I'm aware that AA is a commercial product. However, I don't see how that matters. You can create this configuration yourself without AA on the other end (honestly doesn't matter what you set as the call home target as long as it is using a 3rd party root cert you add to Photon to trust), because it will never get beyond validating the certificate. This problem seems squarely, to me, in how/where cloud-init is looking for the default trusted certificate store. Is that something you can configure/customize in cloud-init? I see a photon.py but I can't see were it would be configured. I'm 99.5% positive if I go to the AA team (I work for VMware/Broadcom) they will say "this syntax works in rhel, ubuntu, etc... it must be something in Photon" and just send me back here.

Do you have any docs on cert store locations that cloud-init uses?

[dark2phoenix]

OK,

So after doing a lot more debugging/looking at files I finally figured out how to at least "get it to work". The default certificate authority seems to come from the cc_cacerts.py config module. In that module, I added a section for photon:

DISTRO_OVERRIDES = { "fedora": { "ca_cert_path": "/etc/pki/ca-trust/", "ca_cert_local_path": "/usr/share/pki/ca-trust-source/", "ca_cert_filename": "anchors/cloud-init-ca-cert-{cert_index}.crt", "ca_cert_config": None, "ca_cert_update_cmd": ["update-ca-trust"], }, "rhel": { "ca_cert_path": "/etc/pki/ca-trust/", "ca_cert_local_path": "/usr/share/pki/ca-trust-source/", "ca_cert_filename": "anchors/cloud-init-ca-cert-{cert_index}.crt", "ca_cert_config": None, "ca_cert_update_cmd": ["update-ca-trust"], }, "opensuse": { "ca_cert_path": "/etc/pki/trust/", "ca_cert_local_path": "/usr/share/pki/trust/", "ca_cert_filename": "anchors/cloud-init-ca-cert-{cert_index}.crt", "ca_cert_config": None, "ca_cert_update_cmd": ["update-ca-certificates"], }, "photon":{ "ca_cert_path": "/etc/ssl/certs/", "ca_cert_local_path": "/etc/ssl/certs/", "ca_cert_filename": "cloud-init-ca-cert-{cert_index}.crt", "ca_cert_config": None, "ca_cert_update_cmd": ["rehash_ca_certificates.sh"], }, }
That alone, unfortunately, doesn't work ( and I don't think the paths are correct really ), because you must also let photon be an acceptable distro:

distros = [ "alpine", "debian", "fedora", "rhel", "opensuse", "opensuse-microos", "opensuse-tumbleweed", "opensuse-leap", "sle_hpc", "sle-micro", "sles", "ubuntu", "photon", ]

Doing both of those things allows cloud-init to load in the default store so phone_home works. I believe this also allows the certificate to be added effectively enabling the entire cacerts stanza in cloud-init (something else I would would like to see).

As I'm not a proper cloud-init developer though, I don't have the proper test harness or know-how to really enable this. Is this something you can carry forward so we can get cloud-init to properly support 3rd party certs and cacerts @dcasota ?

[sshedi]

@dark2phoenix thanks for the analysis. Sorry, I got busy with other things.
Yeah, you are probably right, will check further on this.

[dark2phoenix]

I created the pull request at cloud-init: canonical/cloud-init#4763

[sshedi]

@prashant1221 @sshedi attached is a simple distro check source checker and a report of extracted files found in make-build-Ph5.0-x86_64-./stage/SOURCES/-sources containing "distro" and "rhel" but not "photon". If it's useful as feasibility indicator for future QA ambitions, feel free to use it to justify effort estimates. output.txt (will be updated - on progress ) distrochecksourcechecker.txt

This results in a lot of false positives. We might be having a patch file or some instruction in specs to enable Photon distro.
Also, we don't always need Photon distro in source, most of the projects are distro agnostic. cloud-init is a special case and ca_certs part was missed somehow.

[dark2phoenix]

FYI - they had some additional fixes back on my PR over at canonical. I incorporated their changes in. My hope is that if they accept this upstream we can just pull it into photon4r2 and photon5

[dark2phoenix]

Hiya! The work I did was accepted into the main cloud-init branch! canonical/cloud-init#4763

Possible to pull it into 4.0rev2 and 5.0 please?

[sshedi]

I will take it soon.

[dark2phoenix]

Let me know and I'll try it out.

[dark2phoenix]

Hello,

FINALLY got around to testing this. I see 2 issues that I'm not 100% sure are problems with cloud-init vs problems in Photon (and subsequently where to resolve them. I also don't understand why the unit testing didn't catch this - perhaps canonical doesn't use photon VM's in their test bed.

1. The certificate files that cloud-init adds are not in the proper folder according to rehash_ca_certificates.sh. the rehash script explicitly looks only in "/etc/ssl/certs" but cloud-init is configured to put them in "/etc/pki/tls/certs/" I originally had "/etc/ssl/certs/" in the pull request to Canonical but a VMware code review said to change it. It should be changed in cloud-init to match what rehash_ca_certificates.sh expects.

2. cloud-init writes ".crt" files, but Photon's rehash script expects".pem" files. I could go either way with this. We could change the reshash script to process both .crt and .pem files, or I can make the pull request above explicitly write out ".pem" files (cloud-init can specify the file extension). The advantage of changing the rehash script is that people can use either ".crt" or ".pem" files, but it means coordinating changes to both photon and cloud-init.

@sshedi - How do you think we should proceed to resolve this?

[dark2phoenix]

I decided to incorporate both fixes into a pull request for Canonical. It's available here:
canonical/cloud-init#5048

[dark2phoenix]

Hello @prashant1221 @sshedi,

So it turns out I made 5048 incorrectly, so they had me resubmit the changes to canonical/cloud-init#5077. That pull request was merged yesterday and the pull request was closed. Let me know when you accept it into Photon 4.0r2 and Photon 5.x so I can retest it with the final packages. We may finally be able to get this fixed once and for all!

[sshedi]

I will take it with cloud-init's next major release. In ph4 & above branches of Photon.

[dcasota]

Hi,

Two questions

1. Is there a situation in which a fallback with sysconfig would be preferable?
   For the sake of completeness, in https://cloudinit.readthedocs.io/en/latest/reference/network-config.html it is described:
   PhotonOS disables fallback networking configuration by default, leaving network unrendered when no other network config is provided. If fallback config is still desired on PhotonOS, it can be enabled by providing disable_fallback_netcfg: false in /etc/cloud/cloud.cfg:sys_config settings. fyi The file https://github.com/canonical/cloud-init/blob/main/cloudinit/net/sysconfig.py contains a check "KNOWN_DISTRO" in which Photon OS isn't included. Hence, it won't be proceeded.

2. Photon OS' network-config-manager with nmctl, are there plans to explicitly make it eligible in network-config as renderer? e.g.
   system_info:
network:
renderers: ['netplan', 'network-manager', 'eni', 'sysconfig', 'freebsd', 'netbsd', 'openbsd','nmctl']
activators: ['eni', 'netplan', 'network-manager', 'networkd']

[dark2phoenix]

Just saw the update above - I don't understand how how it is related to this bug @dcasota so I'm don't know how to answer that.

@sshedi - Has the upstream changes been integrated and released yet? I am not seeing the changes in a brand new photon deployment of either 40r2 or 50.

[dcasota]

@dark2phoenix unnecessary to answer. cloud-init works for any platforms with Photon OS' network-config-manager only. There is no fallback, no options strategy. That was the old comment content.

[dark2phoenix]

@sshedi I saw updates for cloud-init in Photon 4.0r2 and 5.0 but when I look at the files it doesn't look like it caught the upstream fixes in canonical/cloud-init#5077. I looked and that has been merged and is available in main for cloud-init.
Any ETA on when we can pull this in?

[dcasota]

Just throwing 2 cents in, that pretraining custom code generator models helps for accelerating

• writing bash, python code and spec files
• populating upstream fixes and tests automatically
• summarizing git diffs and git changes

https://github.com/vmware-private-ai/VMware-generative-ai-reference-architecture/blob/main/Starter-Packs/Code_Assistant/ is in a quite early stage. I don't think Fabric and DeepSpeed with StarCoder or DeepSeekCoder, etc. are already "over the top". There is space. If the team needs help, let's do this.

[sshedi]

@sshedi I saw updates for cloud-init in Photon 4.0r2 and 5.0 but when I look at the files it doesn't look like it caught the upstream fixes in canonical/cloud-init#5077. I looked and that has been merged and is available in main for cloud-init. Any ETA on when we can pull this in?

The rpms are already published. Do tdnf update cloud-init to get the latest release of cloud-init. The ISOs won't be re-released, releasing OS is not simple and straight forward as releasing an application. Don't get confused.