tracking: testing

[cafkafk]

Status

Current testing is based on https://github.com/ogham/exa, which eza forks.

Here, vagrant, as well as https://github.com/ogham/specsheet is a huge pain.

• It's currently not working
• No active contributor knows (or wants to know) how it works
• Vagrant is NOT open source

Fixing something we don't like, instead of just finding a more permanent solution would be short sighted. So instead, I'm looking into various options.

Notice that generating releases will be done through nix at some point (via cross), as it allows for docker-tier build isolation, and could easily later be done on a build server, which would be very nice. We use cross now.

See: #137 (comment)

---

Progress

trycmd seems like it will be the option we go for.

Todo:

• tests: trycmd for integration testing #261
• cleanup "itests" (@cafkafk) #273
• sandbox trycmd with nix (@cafkafk) #274
• adding convenience functions to justfile #275
• Clean up unused files #276
• some logic for platform dependent testing (this was already done in tests: trycmd for integration testing #261)
• Added nix Flake check to flake.yml #323
• ci: deb package ci script #347
• Add more tests #288
• Improving the itest dir #279
• Windows specific testing #280
• documentation of how to use this (not assigned, shouldn't be done before we have more experience)

And also just writing a lot of tests when this gets merged.

---

If there is anything else we should do, comment and I'll add it.

[cafkafk]

I think we could get something similar to what specsheet currently provides from https://github.com/charmbracelet/vhs, which has the advantage of being:

• Maintained actively
• Having github actions already made
• Seeming to have a much nicer UX
• Less importantly, doubling up as a way for us to generate gifs for the readme

This is the current option I'm tentatively exploring.

[cafkafk]

Also, see this example of vhs being used on a PR to generate gifs of the output antlr/antlr4#4383

That's so so so so so so so cool. Specially if we add automation.

[PThorpe92]

They are so awesome and make some really cool stuff.

That is a use case I never considered.. With automated output? Great idea 👍

• No active contributor knows (or wants to know) how it works

Truth

OK so I never had any luck with that docker container from ogham/exa#1230
but it does seem like there have been a bunch of recent commits to that branch. Is that something we are still interested in checking out? I definitely agree that ditching Vagrant is a good long term move, although another solution will have to be found for the kinds of FS testing that it allows for.

What do we think about assigning different areas and writing a bunch of regular old unit tests? Maybe start by making a sub directory in the xtests folder and moving some of the tests out of the files themselves and expanding on them?

[cafkafk]

Is that something we are still interested in checking out?

I'm honestly not sure. We might pick it apart for inspiration if we go that route, but then I think the current solution I'm working on could trivially be put in a docker container, and we'd not be using most of it.

Probably something I'll explore more after #147 doesn't feel as experimental.

What do we think about assigning different areas and writing a bunch of regular old unit tests?

I'm not sure what you mean by assigning different areas, like saying someone takes care of testing some part of the codebase?

Maybe start by making a sub directory in the xtests folder and moving some of the tests out of the files themselves and expanding on them?

I think maybe this effort would be better spend in #147, but also, I'm not sure of the value of actually migrating this, because it is my goal to be able to generate something equivalent automatically (but also allowing manual edits).

---

Also, I opened an issue in VHS, I'm not sure if I'm just not aware of how to do something or we'd need them to change their codebase ever so slightly. If I have some time next week I'll probably send them a PR for what I have in mind and then we can build VHS from that PR's branch until/if they merge it.

edit: I've also already found a hacky workaround for this, which isn't ideal, but makes sure this is not blocking.

But currently, the prototype kinda works, but without more detailed instructions of how to use it it's probably not that approachable, will also get to that as soon as possible.

[PThorpe92]

I'm not sure what you mean by assigning different areas, like saying someone takes care of testing some part of the codebase?

Like opening a Unit Test issue with a checklist of areas in the codebase that more traditional tests should be written for, and we can submit PR's for each area and check them off, etc after removing the tests from the bottom of the files and rewriting some more complete tests in the tests directory.

I guess I'm just kinda lost as far as the format of all the existing ones.. My thoughts were to just start replacing them with regular unit tests one chunk at a time.

[ariasuni]

I’m was wondering about how to transition from the Vagrant tests to a new solution. My idea at the time was to continue using specsheet (I’m in favor of using something else that’s maintained), but using Docker instead so that it could also run on the CI. Right now, the tests need a specific setup of files, directories and locales to be able to compare the outputs and test every feature of eza, that could be reproduced in a Docker container.

I’m also asking because #162 should have been caught by the xtests (the only reason I caught it, is because I ran them) and I worry about other bugs being added without anyone noticing. I think it would be nice if tests were a bit more separated, because right now it’s a mess. If you want to add a testcase for e.g. links, other unrelated tests break and if you modify colors, almost every test is broken (which is why it’s very broken right now).

Anyway I’m looking forward contributing to tests when something’s in place. In the meantime, I’ll try to finish the MR I started instead of finding new/old bugs and fixing them. ;)

[cafkafk]

(I’m in favor of using something else that’s maintained), but using Docker instead so that it could also run on the CI. Right now, the tests need a specific setup of files, directories and locales to be able to compare the outputs and test every feature of eza, that could be reproduced in a Docker container.

Yea, my plan was essentially to use docker if I couldn't find something better. Also creating a subset of tests that could be run outside of docker, which would probably mean using NO_COLOR=true, --no-user, and maybe even --no-time (and any other flags that reduce impurity).

The main argument against docker is that it's kind of slow and heavy. I think I'd like to see if using the nix sandbox via nix would be a good solution, as it sets the time on files to the unix epoch, uses a consistent user name, and just generally seems like it's very fit for purpose.

[ariasuni]

Well, mainly I thought about using Docker/Podman, but I don’t have a preference, I just want it to be easy to use. I think running some tests by selectively removing information that depend on the host system is a good idea.

I’ve tried using nix with the eza setup, it was confusing to setup, not sure what it brings for us. But honestly I don’t really care as long as there are integration tests running on the CI so that new bugs don’t get introduced.

[polarathene]

The main argument against docker is that it's kind of slow and heavy.

It shouldn't be? Depends how you approach it I guess 🤷‍♂️

I maintain a project built around docker so I'm biased there. Our tests run via shell scripts with BATS (although I'd rather something else), and despite our large image the tests can run rather quickly. Bulk of the project is just bash scripts running in a container providing all the dependencies needed (which get configured and tested against).

There will be situations where it's notably faster to run on the host directly (eg: using jq vs pulling in Alpine or similar with jq to run it via the container), while there isn't as big of a difference when using testssl.sh to run a bunch of logic vs running the equivalent on the host.

Our CI with Github Actions builds the AMD64 image (over 500MB) very quickly due to caching earlier build layers from prior runs that would take much longer. The ARM64 image doesn't utilize the caching at present and takes approx 2 mins (relies on QEMU emulation). The test suite was originally taking over an hour to build the images and run all tests we have, but we've brought that down to around 5 minutes or less.

A smaller binary without the more complicated test environment that the docker project I work on requires, should do much better? I haven't looked over exa / eza test suite, so I'm not familiar with it to know if they'd be much benefit from containers.

Nix has it's perks, I like it's immutability benefits but I also found it was not as good as a Dockerfile for certain projects. Especially for most contributors it was quite foreign to them to gain familiarity with vs lower learning curve with Docker.

[9glenda]

#323 will add nix flake check tests. They ensure proper formatting and check rather the nix package builds on all targets it supports.

[MartinFillon]

#591 will start the transition to using the generated test_dir