Feature: system packages for Linux servers, CI and workstations

[alexellis]

Feature

Feature: system packages for Linux servers and workstations

There are so many times that I log into a server and install server software using a combination of curl and tar.

This provides the fastest way to install software, ensures the latest versions are available, something that package managers struggle to keep up with.

Generally, during scripting and automation, you'll find this kind of pattern. The popularity of arkade get has shown that people are comfortable installing software in this way, and it makes CI - trivial - https://github.com/hopefulramen/k3sup-tester/blob/master/.github/workflows/build.yml

For example:

arkade system install go@v1.18.1
arkade system install node@v1.16
arkade system install firecracker@v1.0.0
arkade system install containerd@v1.16.1
arkade system install cni-plugins

This is present in openfaas/faasd, and on every Linux workstation I use, I tend to need these tools.

When a tool is already present, installation will be skipped, unless --force is passed.

The version will be detected the same way as arkade get without using the GitHub API, but using a HTTP HEAD request to the releases page, where no version is given that is.

A Go template will be used to form the URL, which will have a unit test for ARM32, ARM64 and AMD64.

Sometimes there are additional steps to run like systemd enable, or making a directory in a known location.

This would be a new command aimed at ARM32, ARM64 and AMD/Intel workstations and servers, and would save a lot of time for CI and setting up workstations.

I'd suggest we create a basic fluid SDK when adding the first app for Golang, for instance:

export ARCH=$(uname -p)

export GOPATH=$HOME/go/
export PATH=$PATH:/usr/local/go/bin/

install_go() {
    DL_ARCH=$ARCH
    if [ "$ARCH" == "x86_64" ]; then
        DL_ARCH=amd64
    else 
        if [ "$ARCH" == "aarch64" ]; then
            DL_ARCH=arm64
        fi
    fi

    if ! [ -e /usr/local/go/bin/go ]; then

        curl -SLf https://go.dev/dl/go$GOVER.linux-$DL_ARCH.tar.gz > /tmp/go.tgz
        sudo rm -rf /usr/local/go/
        sudo mkdir -p /usr/local/go/
        sudo tar -xvf /tmp/go.tgz -C /usr/local/go/ --strip-components=1

        go version
        echo "export GOPATH=\$HOME/go/" | tee -a $HOME/.bashrc
        echo "export PATH=\$PATH:/usr/local/go/bin/" | tee -a $HOME/.bashrc
    fi
}

Would become:

system.
  WithTarget("/usr/local/go/).
  WithArch("x86_64").
  WithOverwrite(false).
  WithVersion("v1.18.1")
  WithMessageTemplate(`
# Add to your PATH variable or .bashrc:
export GOPATH=\$HOME/go/" | tee -a $HOME/.bashrc
echo "export PATH=\$PATH:/usr/local/go/bin/" | tee -a $HOME/.bashrc`)

Each command file would be added and populatet the Arch / Version etc in a similar way to how we do this for arkade get

Containerd is more involved:

export CONTAINERD_VERSION=1.6.1

install_containerd() {
    DL_ARCH=$ARCH
    if [ "$ARCH" == "x86_64" ]; then
        DL_ARCH=amd64
    else 
        if [ "$ARCH" == "aarch64" ]; then
            DL_ARCH=arm64
        fi
    fi

    mkdir -p /etc/containerd

    if ! [ -e /etc/containerd/config.toml ]; then
        cd $START_PWD
        cp ./config.toml /etc/containerd/config.toml
    fi

    if ! [ -e /usr/local/bin/containerd ]; then
        curl -sSL https://github.com/containerd/containerd/releases/download/v$CONTAINERD_VERSION/containerd-$CONTAINERD_VERSION-linux-$DL_ARCH.tar.gz > /tmp/containerd.tar.gz \
        && sudo tar -xvf /tmp/containerd.tar.gz -C /usr/local/bin/ --strip-components=1

        containerd -version

        curl -sLS https://raw.githubusercontent.com/containerd/containerd/v1.5.4/containerd.service > /tmp/containerd.service

        echo "[Manager]" | tee -a /tmp/containerd.service
        echo "DefaultTimeoutStartSec=3m" | tee -a /tmp/containerd.service

        sudo cp /tmp/containerd.service /lib/systemd/system/
        sudo systemctl enable containerd || echo "Failed to enable containerd, may be masked."

        sudo systemctl daemon-reload
        sudo systemctl restart containerd
    fi
}

Therefore we'd want some additional SDK methods, as per the method we use for faasd: https://github.com/openfaas/faasd/blob/2885bb0c514a403d317b93e6d8add1ad52239a13/pkg/systemd/systemd.go

.WithFetch("https://raw.githubusercontent.com/containerd/containerd/v1.5.4/containerd.service", "/tmp/containerd.service")
.WithCp("/tmp/containerd.service", "/lib/systemd/system")
.WithDaemonReload().
.WithRestart("containerd")

I'm looking for someone to volunteer to add the first app, it needn't use an SDK for the first pass, we can extract this as we add apps.

An initial list of apps:

• Prometheus
• Golang
• containerd
• firecracker
• nodejs
• CNI plugins

I do not see this list becoming much larger than the above, but am open to suggestions from the community.

[richardcase]

I could use this in a number of places. Using linux package managers is ok but not all the tools we use are available or they are in a different repo or an older version. Having something to download the binary would be great.

[iximiuz]

This would be definitely helpful! I was checking if arkade could install Go on my system a couple of days ago - having arkade already used to install a bunch of tools (kind, kubectl, etc.), it felt absolutely natural to employ arkade as a replacement of a package manager too.

My only concern is about the secureness of this procedure. Pulling random stuff from the Internet becomes more and more dangerous these days :) So, maybe having some trusted registry with package signatures is a good idea. But this might defeat the sole purpose of having a lightweight and up-to-date installation mechanism.

[alexellis]

@iximiuz "pulling stuffed from some random place on the internet"

Explain how you got that from the proposal?

This is designed to be no different to arkade get, or the way that many people already install Go, containerd, etc.

[iximiuz]

@alexellis Sure! When we do things like wget https://go. dev/dl/go1.18.linux-amd64.tar.gz, it does look like pulling random stuff from the Internet to me. Even though in this particular case the source is (more or less) trusted.

Validating the source of the artifact is an important step during an installation of a package, but having a checksum verification is probably an even more important step. But all these things must be transparent. When I use a package manager like apt, I kinda offload this trust to the maintainers of the system repositories at the expense of having (quite) outdated versions of software installed. But if I were to start installing packages with arkade, I would need a crystal clear understanding of:

• From where a package comes from
• What validations (e.g. checksums) arkade does before installing a package.

And yes, arkade get can also benefit from something similar to that.

[alexellis]

Whilst I agree that apt and pacman etc use GPG or other signing keys, I wouldn't have considered Google's download server to be untrusted or random. I checked, and the installation guide for Go follows the same pattern, they don't even supply a command to verify some checksum. https://go.dev/doc/install, I can't even find checksums on their download page: https://go.dev/dl/

Given that you use arkade get today, which downloads over HTTPS, it doesn't seem like a blocker? Is your main concern that you believe that TLS encryption could be broken and intercepted?

[Shikachuu]

When we talk about Prometheus in this case, are we talking about prometheus itself only or with "standard" exporters like node_exporter as well?

[Shikachuu]

I do not see this list becoming much larger than the above, but am open to suggestions from the community.

@alexellis may I suggest adding buildkit to the list?

[alexellis]

@Shikachuu prometheus = the download called prometheus

buildkit - potentially. I haven't had a need to add it stand-alone on a machine, and doesn't installing Docker add it?

Keep suggesting things that you personally install on Linux servers/workstations/CI hosts.

[Shikachuu]

@Shikachuu prometheus = the download called prometheus

buildkit - potentially. I haven't had a need to add it stand-alone on a machine, and doesn't installing Docker add it?

Keep suggesting things that you personally install on Linux servers/workstations/CI hosts.

buildkit itself doesn't require Docker, only runc, crun or containerd.

I acutally use buildkit with containerd and nerdctl to substitue docker on my workstation, my laptop and on my "build" server as well.
(Also if you download the nerdctl-full package from their github release, it contains this combination of tools too.)

[alexellis]

Completed:

• firecracker
• Golang
• prometheus
• containerd
• actions-runner
• node
• cni-plugins (alias cni)

[dirien]

Really good idea, but I may suggest to start directly with cosign signed packages and show that its signed (or not signed). Think that would help tremendously!

[alexellis]

These tools are not signed with Cosign, so I'm not sure what you're suggesting?

[dirien]

These tools are not signed with Cosign, so I'm not sure what you're suggesting?

Uhh just saw that you are right. Maybe for the future!

[alexellis]

@dirien thanks for the PR for Prometheus, could you or @Shikachuu please send in a PR where you make the "arch" value come from a flag if .Changed is true - default it to ""? See how we for it for the get command. We need this for all of the system apps and it helps with testing.

[alexellis]

@Shikachuu for the containerd app, please make sure you install the systemd service as per the code I DM'd you on Discord. Then put the systemd behaviour behind a flag like --systemd=true (default true)

See also: https://github.com/openfaas/faasd/blob/2885bb0c514a403d317b93e6d8add1ad52239a13/pkg/systemd/systemd.go

[dirien]

@dirien thanks for the PR for Prometheus, could you or @Shikachuu please send in a PR where you make the "arch" value come from a flag if .Changed is true - default it to ""? See how we for it for the get command. We need this for all of the system apps and it helps with testing.

@alexellis Like this #673?

[alexellis]

That is exactly what I had in mind, could you cover the other apps too?

[dirien]

Hi @alexellis, added the node support and I saw you added the the CNI. 👍

[alexellis]

We now have all the original apps requested along with actions-runner to get the latest GitHub Actions runner version.