Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bump OpenSSH version to v9.5.0.0p1-Beta #388

Conversation

github-actions[bot]
Copy link
Contributor

@github-actions github-actions bot commented Mar 29, 2024

Bump OpenSSH version

Update the OpenSSH version for Windows Nanoserver

changed lines [98] of file "/tmp/updatecli/github/jenkinsci/docker-ssh-agent/windows/nanoserver/Dockerfile"

v9.5.0.0p1-Beta
Release published on the 2023-12-18 16:57:34 +0000 UTC at the url https://github.com/PowerShell/Win32-OpenSSH/releases/tag/v9.5.0.0p1-Beta

This is a beta-release (non-production ready)

This release includes:

- Upstream changes from [OpenSSH 9.5p1](https://www.openssh.com/txt/release-9.5) and a cherry-pick of the "strict KEX" protocol extension changes from [OpenSSH 9.6p1](https://www.openssh.com/txt/release-9.6).

- Breaking Changes:

    - Remove sha1-based MACs from default configuration options in https://github.com/PowerShell/openssh-portable/pull/706/files.

- Security Fixes: 
  - Service paths for SSHD and SSH-Agent in contrib\win32\openssh\install-sshd.ps1 are encapsulated in double-quotes.

- Non-Security Fixes:

  - [SCP/SFTP] Permit data upload that originates from a named pipe in https://github.com/PowerShell/openssh-portable/pull/704.  
  - Upgrade to LibreSSL 3.8.2. Please refer to https://ftp.openbsd.org/pub/OpenBSD/LibreSSL/libressl-3.8.2-relnotes.txt.
  - Upgrade to LibFido2 1.14.0. Please refer to https://developers.yubico.com/libfido2/Release_Notes.html.
Update the OpenSSH version for Windows Server Core

changed lines [86] of file "/tmp/updatecli/github/jenkinsci/docker-ssh-agent/windows/windowsservercore/Dockerfile"

v9.5.0.0p1-Beta
Release published on the 2023-12-18 16:57:34 +0000 UTC at the url https://github.com/PowerShell/Win32-OpenSSH/releases/tag/v9.5.0.0p1-Beta

This is a beta-release (non-production ready)

This release includes:

- Upstream changes from [OpenSSH 9.5p1](https://www.openssh.com/txt/release-9.5) and a cherry-pick of the "strict KEX" protocol extension changes from [OpenSSH 9.6p1](https://www.openssh.com/txt/release-9.6).

- Breaking Changes:

    - Remove sha1-based MACs from default configuration options in https://github.com/PowerShell/openssh-portable/pull/706/files.

- Security Fixes: 
  - Service paths for SSHD and SSH-Agent in contrib\win32\openssh\install-sshd.ps1 are encapsulated in double-quotes.

- Non-Security Fixes:

  - [SCP/SFTP] Permit data upload that originates from a named pipe in https://github.com/PowerShell/openssh-portable/pull/704.  
  - Upgrade to LibreSSL 3.8.2. Please refer to https://ftp.openbsd.org/pub/OpenBSD/LibreSSL/libressl-3.8.2-relnotes.txt.
  - Upgrade to LibFido2 1.14.0. Please refer to https://developers.yubico.com/libfido2/Release_Notes.html.
GitHub Action workflow link
Updatecli logo

Created automatically by Updatecli

Options:

Most of Updatecli configuration is done via its manifest(s).

  • If you close this pull request, Updatecli will automatically reopen it, the next time it runs.
  • If you close this pull request and delete the base branch, Updatecli will automatically recreate it, erasing all previous commits made.

Feel free to report any issues at github.com/updatecli/updatecli.
If you find this tool useful, do not hesitate to star our GitHub repository as a sign of appreciation, and/or to tell us directly on our chat!

Copy link
Contributor

@dduportal dduportal left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This updates looks good for Windows Server Core images (2019 and 2022).

However it makes the test fails for the 3 nanoserver images (1809, 2019 and 2022). Need to be investigated (I won't have time to spend on this analysis)

@lemeurherve
Copy link
Member

lemeurherve commented Apr 25, 2024

Tried with version v9.2.2.0p1-Beta like in #319 with 84c9755, all green ✅

Trying out with the next version before the v9.5.0.0p1-Beta, the https://github.com/PowerShell/Win32-OpenSSH/releases/tag/v9.4.0.0p1-Beta

@lemeurherve
Copy link
Member

lemeurherve commented Apr 25, 2024

Good news (ahem), the nanoserver image builds fail with version v9.4.0.0p1-Beta, less changes to take in account 😅

Breaking changes - see upstream release notes for more information:

  • the ssh-agent will now refuse requests to load PKCS#11 modules issued by remote clients by default. A flag has been added to restore the previous behavior "-Oallow-remote-pkcs11".

Security Fixes:

  • the sshd service will check the $env:ProgramData\ssh folder permissions upon startup to ensure only SYSTEM and Administrator accounts have write access to the folder; similar to the existing check upon install in contrib\win32\openssh\install-sshd.ps1.

https://ci.jenkins.io/job/Packaging/job/docker-ssh-agent/job/PR-388/4/

Will take a look later, we can meanwhile merge with v9.2.2.0p1-Beta.

WDYT @jenkinsci/team-docker-packaging?

@lemeurherve
Copy link
Member

Got more details about the failing tests starting with OpenSSH v9.4.0.0p1-beta, working on it.

Tests error logs:
Describing [jenkins/ssh-agent:nanoserver-ltsc2019-jdk21] create agent container with pubkey as argument 
cmd & params: docker.exe port pester-jenkins-ssh-agent-nanoserver-ltsc2019-jdk21 22
cmd & params: C:\Users\herve\dockerssh\docker-ssh-agent\tests\ssh.exe -v -i "C:\Users\herve\AppData\Local\Temp\tmpB395.tmp" -o LogLevel=quiet -o UserKnownHostsFile=NUL -o StrictHostKeyChecking=no -l jenkins localhost -p 50933 pwsh.exe -No
Logo -C "Write-Host 'f00'"
[err] stdout: 

[err] stderr:
OpenSSH_for_Windows_8.1p1, LibreSSL 2.9.2
debug1: Connecting to localhost [::1] port 50933.
debug1: connect to address ::1 port 50933: Connection refused
debug1: Connecting to localhost [127.0.0.1] port 50933.
debug1: connect to address 127.0.0.1 port 50933: Connection timed out
ssh: connect to host localhost port 50933: Connection timed out

[err] cmd:
C:\Users\herve\dockerssh\docker-ssh-agent\tests\ssh.exe
[err] params:

  [-] runs commands via ssh 22.21s (22.2s|7ms) 
   Expected 0, but got 255.
   at $exitCode | Should -Be 0, C:\Users\herve\dockerssh\docker-ssh-agent\tests\sshAgent.Tests.ps1:137
   at <ScriptBlock>, C:\Users\herve\dockerssh\docker-ssh-agent\tests\sshAgent.Tests.ps1:137
cmd & params: docker.exe inspect --format "{{.State.Running}}" pester-jenkins-ssh-agent-nanoserver-ltsc2019-jdk21 
 
Describing [jenkins/ssh-agent:nanoserver-ltsc2019-jdk21] create agent container with pubkey as envvar 
cmd & params: docker.exe port pester-jenkins-ssh-agent-nanoserver-ltsc2019-jdk21 22
cmd & params: C:\Users\herve\dockerssh\docker-ssh-agent\tests\ssh.exe -v -i "C:\Users\herve\AppData\Local\Temp\tmp2626.tmp" -o LogLevel=quiet -o UserKnownHostsFile=NUL -o StrictHostKeyChecking=no -l jenkins localhost -p 50941 pwsh.exe -No
Logo -C "Write-Host 'f00'"
[err] stdout: 

[err] stderr:
OpenSSH_for_Windows_8.1p1, LibreSSL 2.9.2
debug1: Connecting to localhost [::1] port 50941.
debug1: connect to address ::1 port 50941: Connection refused
debug1: Connecting to localhost [127.0.0.1] port 50941.
debug1: connect to address 127.0.0.1 port 50941: Connection timed out
ssh: connect to host localhost port 50941: Connection timed out

[err] cmd:
C:\Users\herve\dockerssh\docker-ssh-agent\tests\ssh.exe
[err] params:

  [-] runs commands via ssh 22.22s (22.21s|8ms) 
   Expected 0, but got 255.
   at $exitCode | Should -Be 0, C:\Users\herve\dockerssh\docker-ssh-agent\tests\sshAgent.Tests.ps1:154
   at <ScriptBlock>, C:\Users\herve\dockerssh\docker-ssh-agent\tests\sshAgent.Tests.ps1:154
cmd & params: docker.exe inspect --format "{{.State.Running}}" pester-jenkins-ssh-agent-nanoserver-ltsc2019-jdk21 

Describing [jenkins/ssh-agent:nanoserver-ltsc2019-jdk21] create agent container like docker-plugin with '/usr/sbin/sshd -D -p 22' as argument 
cmd & params: docker.exe port pester-jenkins-ssh-agent-nanoserver-ltsc2019-jdk21 22
cmd & params: C:\Users\herve\dockerssh\docker-ssh-agent\tests\ssh.exe -v -i "C:\Users\herve\AppData\Local\Temp\tmp98F6.tmp" -o LogLevel=quiet -o UserKnownHostsFile=NUL -o StrictHostKeyChecking=no -l jenkins localhost -p 50948 pwsh.exe -No
Logo -C "Write-Host 'f00'"
[err] stdout: 

[err] stderr:
OpenSSH_for_Windows_8.1p1, LibreSSL 2.9.2
debug1: Connecting to localhost [::1] port 50948.
debug1: connect to address ::1 port 50948: Connection refused
debug1: Connecting to localhost [127.0.0.1] port 50948.
debug1: connect to address 127.0.0.1 port 50948: Connection timed out
ssh: connect to host localhost port 50948: Connection timed out

[err] cmd:
C:\Users\herve\dockerssh\docker-ssh-agent\tests\ssh.exe
[err] params:

  [-] runs commands via ssh 22.19s (22.19s|7ms) 
   Expected 0, but got 255.
   at $exitCode | Should -Be 0, C:\Users\herve\dockerssh\docker-ssh-agent\tests\sshAgent.Tests.ps1:174
   at <ScriptBlock>, C:\Users\herve\dockerssh\docker-ssh-agent\tests\sshAgent.Tests.ps1:174

@lemeurherve
Copy link
Member

lemeurherve commented Apr 27, 2024

Almost certain the issue comes from the vendored ssh.exe 🤦

From a VM created with one of our packer image template:

PS C:\Users\herve\docker-ssh-agent\tests> ssh.exe -V
OpenSSH_9.2p1, OpenSSL 1.1.1t  7 Feb 2023

PS C:\Users\herve\docker-ssh-agent\tests> .\ssh.exe -V
OpenSSH_for_Windows_8.1p1, LibreSSL 2.9.2

EDIT: same error with ssh.exe OpenSSH_9.2p1 & 9.4p1, my assumption was wrong.

Describing [jenkins/ssh-agent:nanoserver-ltsc2019-jdk21] create agent container with pubkey as envvar 
cmd & params: docker.exe port pester-jenkins-ssh-agent-nanoserver-ltsc2019-jdk21 22
cmd & params: C:\Users\herve\dockerssh\docker-ssh-agent\tests\ssh.exe -v -i "C:\Users\herve\AppData\Local\Temp\tmp47F8.tmp" -o LogLevel=quiet -o UserKnownHostsFile=NUL -o Stri
ctHostKeyChecking=no -l jenkins localhost -p 52092 pwsh.exe -NoLogo -C "Write-Host 'f00'"
[err] stdout: 

[err] stderr:
OpenSSH_9.2p1, OpenSSL 1.1.1t  7 Feb 2023
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Connecting to localhost [::1] port 52092.
debug1: connect to address ::1 port 52092: Connection refused
debug1: Connecting to localhost [127.0.0.1] port 52092.
debug1: connect to address 127.0.0.1 port 52092: Connection timed out
ssh: connect to host localhost port 52092: Connection timed out

@lemeurherve
Copy link
Member

lemeurherve commented Apr 27, 2024

Trying to log in the docker image including OpenSSH v9.5.0.0p-beta1 with the same command as the beforeAll of the SSH tests docker run --detach --tty --name="pester-jenkins-ssh-agent-nanoserver-ltsc2019-jdk17" --publish-all "jenkins/ssh-agent:nanoserver -ltsc2019-jdk17" "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEAvnRN27LdPPQq2OH3GiFFGWX/SH5TCPVePLR21ngMFV8nAthXgYrFkRi/t+Wafe3ByTu2XYUDlXHKGIPIoAKo4gz5dIjUFfoac1ZuCDIbEiqPEjkk4tkfc2qr/ BnIZsOYQi4Mbu+Z40VZEsAQU7eBinnZaHE1qGMHjS1xfrRtp2rdeO1EBz92FJ8dfnkUnohTXo3qPVSFGIPbh7UKEoKcyCosRO1P41iWD1rVsH1SLLXYAh2t49L7IPiplg09Dep6H47LyQVbxU9eXY8yMtUrRuwEk9IUX/IqpxNhk5hn gHPP3JjsP0hyyrYSPkZlbs3izd9kk3y09Wn/ElHidiEk0Q==", I've got this error message:

Authorizing ssh pubkey found in params.
Start-Service: C:\ProgramData\Jenkins\setup-sshd.ps1:103
Line |
 103 |  Start-Service sshd
     |  ~~~~~~~~~~~~~~~~~~
     | Failed to start service 'OpenSSH SSH Server (sshd)'.

Windows IP Configuration


Ethernet adapter vEthernet (Ethernet):
  
   Connection-specific DNS Suffix  . : y0yhre0hl2eezoulv1kedzh0bc.cx.internal.cloudapp.net
   Link-local IPv6 Address . . . . . : fe80::7a1f:c67e:269b:6802%18
   IPv4 Address. . . . . . . . . . . : 172.26.63.61
   Subnet Mask . . . . . . . . . . . : 255.255.240.0
   Default Gateway . . . . . . . . . : 172.26.48.1

Active Connections

  Proto  Local Address          Foreign Address        State
  TCP    0.0.0.0:135            8eff802d9057:0         LISTENING
  TCP    0.0.0.0:49163          8eff802d9057:0         LISTENING
  TCP    0.0.0.0:49164          8eff802d9057:0         LISTENING
  TCP    0.0.0.0:49165          8eff802d9057:0         LISTENING
  TCP    172.26.53.111:49160    13.107.4.50:http       TIME_WAIT
  TCP    172.26.63.61:49167     52.239.160.36:https    TIME_WAIT
  TCP    [::]:135               8eff802d9057:0         LISTENING
  TCP    [::]:49163             8eff802d9057:0         LISTENING
  TCP    [::]:49164             8eff802d9057:0         LISTENING
  TCP    [::]:49165             8eff802d9057:0         LISTENING
  UDP    0.0.0.0:5353           *:*
  UDP    0.0.0.0:5355           *:*
  UDP    [::]:5353              *:*
  UDP    [::]:5355              *:*
Get-Content: C:\ProgramData\Jenkins\setup-sshd.ps1:110
Line |
 110 |  Get-Content -Path "C:\ProgramData\ssh\logs\sshd.log" -Wait
     |  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
     | Cannot find path 'C:\ProgramData\ssh\logs\sshd.log' because it does not exist.
PS C:\Users\jenkins\Work> cd \
PS C:\> dir '.\ProgramData\ssh'  

    Directory: C:\ProgramData\ssh

Mode                 LastWriteTime         Length Name
----                 -------------         ------ ----
-a---           4/27/2024  2:26 PM           2331 sshd_config

Continuing around there...

@lemeurherve lemeurherve marked this pull request as draft April 27, 2024 14:46
@lemeurherve
Copy link
Member

lemeurherve commented Apr 27, 2024

This updates looks good for Windows Server Core images (2019 and 2022).

This was a mirage unfortunately.

They're also failing on the SSH tests, it's just they were skipped since a previous PR to reduce the tests runtime deliver a new image cf notes of #289 .

Restored in 47a9e03

@lemeurherve
Copy link
Member

lemeurherve commented Apr 27, 2024

Trying to log in the docker image including OpenSSH v9.5.0.0p-beta1 with the same command as the beforeAll of the SSH tests <...>

I don't understand yet why but login in the image, re-running C:/Program Files/OpenSSH-Win64/install-sshd.ps1 then re-running the setup-sshd.ps1 entry script resolves the issue, the sshd service can be started properly 🤔🤷

Current workaround: 76140cd 😅

@lemeurherve
Copy link
Member

lemeurherve commented Apr 28, 2024

I don't understand yet why but login in the image, re-running C:/Program Files/OpenSSH-Win64/install-sshd.ps1 then re-running the setup-sshd.ps1 entry script resolves the issue, the sshd service can be started properly 🤔🤷

After scratching my head why SSH tests were working on Nanoserver with my dirty hack 8a8e0fc but not on Windows Server Core, opening #391 which made me dig in #292 again with working results when running test commands locally, I finally found the root cause: we have to wait a bit for the SSH server to be ready before trying to connect to it 😅
876590b

Opened #391 to restore and fix them.

@lemeurherve lemeurherve self-assigned this Apr 28, 2024
lemeurherve added a commit that referenced this pull request Apr 28, 2024
While working on #388 I remembered we disabled SSH specific tests for Windows Server Core image in order to deliver a new image cf notes of #289.

This PR removes this skip and fix them by waiting 10s before trying to connect with SSH (5s wasn't enough), as discovered in #388 (comment).
lemeurherve added a commit to lemeurherve/docker-ssh-agent that referenced this pull request Apr 28, 2024
@lemeurherve
Copy link
Member

Opened #396 will all changes unrelated to the bump of OpenSSH v9.5.0.0p-beta1.

@lemeurherve
Copy link
Member

Restored the branch state to only include my workaround in nanoserver Dockerfile and the bump of OpenSSH to v9.5.0.0p-beta1.

@lemeurherve lemeurherve marked this pull request as ready for review April 29, 2024 08:02
@lemeurherve lemeurherve requested review from dduportal and a team April 29, 2024 08:02
@lemeurherve lemeurherve marked this pull request as draft April 29, 2024 18:17
@lemeurherve lemeurherve marked this pull request as ready for review April 29, 2024 23:18
@lemeurherve lemeurherve marked this pull request as draft April 30, 2024 00:02
@lemeurherve
Copy link
Member

lemeurherve commented Apr 30, 2024

With the less dirty hack 1239b46, install-sshd.ps1 manages to fix permissions so sshd service can be started without the following error described in #388 (comment)

Unfortunately install-sshd.ps1 throws the following error (that I'm catching):

  [*] C:\Program Files\OpenSSH-Win64\moduli 
Inheritance is removed from 'C:\Program Files\OpenSSH-Win64\moduli'. 
'BUILTIN\Users' now has Read access to 'C:\Program Files\OpenSSH-Win64\moduli'.  
      Repaired permissions 

  [*] C:\ProgramData\ssh
Inheritance is removed from 'C:\ProgramData\ssh'. 
New-Object: Exception calling ".ctor" with "5" argument(s): "The value '268435456' is not 
valid for this usage of the type FileSystemRights. (Parameter
'fileSystemRights')"
MethodInvocationException: Exception calling "RemoveAccessRule" with "1" argument(s): "Value cannot be 
null. (Parameter 'rule')"
'BUILTIN\Users' has no more access to 'C:\ProgramData\ssh'.
'BUILTIN\Users' has no more access to 'C:\ProgramData\ssh'.
      Repaired permissions

  [*] C:\ProgramData\ssh\sshd_config 
Inheritance is removed from 'C:\ProgramData\ssh\sshd_config'. 
      looks good

[SC] SetServiceObjectSecurity SUCCESS 
[SC] ChangeServiceConfig2 SUCCESS 
[SC] ChangeServiceConfig2 SUCCESS 
sshd and ssh-agent services successfully installed
Updated Machine PATH to include OpenSSH directory, restart/re-login required to take effect globally 

It seems to come from the fact that the folder C:\ProgramData\ssh has in its ACL the following one, not supported by this OpenSSH script:

FileSystemRights  : 268435456
AccessControlType : Allow
IdentityReference : CREATOR OWNER
IsInherited       : True
InheritanceFlags  : ContainerInherit, ObjectInherit
PropagationFlags  : InheritOnly
pwsh -command "(Get-Acl 'C:\ProgramData\ssh').Access"
FileSystemRights  : FullControl 
AccessControlType : Allow
IdentityReference : NT AUTHORITY\SYSTEM
IsInherited       : True
InheritanceFlags  : ContainerInherit, ObjectInherit
PropagationFlags  : None

FileSystemRights  : FullControl
AccessControlType : Allow
IdentityReference : BUILTIN\Administrators
IsInherited       : True
InheritanceFlags  : ContainerInherit, ObjectInherit
PropagationFlags  : None

FileSystemRights  : FullControl
AccessControlType : Allow
IdentityReference : User Manager\ContainerAdministrator
IsInherited       : True
InheritanceFlags  : None
PropagationFlags  : None

FileSystemRights  : 268435456
AccessControlType : Allow
IdentityReference : CREATOR OWNER
IsInherited       : True
InheritanceFlags  : ContainerInherit, ObjectInherit
PropagationFlags  : InheritOnly

FileSystemRights  : ReadAndExecute, Synchronize
AccessControlType : Allow
IdentityReference : BUILTIN\Users
IsInherited       : True
InheritanceFlags  : ContainerInherit, ObjectInherit
PropagationFlags  : None

FileSystemRights  : Write
AccessControlType : Allow
IdentityReference : BUILTIN\Users 
IsInherited       : True
InheritanceFlags  : ContainerInherit
PropagationFlags  : None

I haven't found (yet?) how to properly fix this but the result is still that sshd is starting correctly.

@github-actions github-actions bot force-pushed the updatecli_master_97163665c8ff58584e464835b75cb49c3aeb70ccb5a0633ff95c11cf8076d4c3 branch from 1239b46 to 3690ade Compare May 6, 2024 11:50
@lemeurherve
Copy link
Member

Damn, it took its time but updatecli finally overridden my changes 🥲

Will open my own pull request, let's keep this one in draft meanwhile.

Copy link
Member

@lemeurherve lemeurherve left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nanoserver failing.

@github-actions github-actions bot force-pushed the updatecli_master_97163665c8ff58584e464835b75cb49c3aeb70ccb5a0633ff95c11cf8076d4c3 branch from e33cdd7 to 43823fa Compare May 10, 2024 23:41
@github-actions github-actions bot force-pushed the updatecli_master_97163665c8ff58584e464835b75cb49c3aeb70ccb5a0633ff95c11cf8076d4c3 branch from dad7a2d to 26922fd Compare May 13, 2024 12:07
@github-actions github-actions bot force-pushed the updatecli_master_97163665c8ff58584e464835b75cb49c3aeb70ccb5a0633ff95c11cf8076d4c3 branch 6 times, most recently from f0bb47d to 9e7802d Compare May 15, 2024 03:13
@github-actions github-actions bot closed this May 15, 2024
@github-actions github-actions bot force-pushed the updatecli_master_97163665c8ff58584e464835b75cb49c3aeb70ccb5a0633ff95c11cf8076d4c3 branch from ee927f5 to 110b752 Compare May 15, 2024 06:33
@lemeurherve lemeurherve deleted the updatecli_master_97163665c8ff58584e464835b75cb49c3aeb70ccb5a0633ff95c11cf8076d4c3 branch May 15, 2024 06:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

nanoserver build error: Failed to load XML document C:\"C:\Program Files\OpenSSH-Win64\openssh-events.man"
3 participants