Tests optimization

[dgarus]

Tests which use com.artipie.test.TestDeployment take too long time.
We should optimize these tests to reduces execution time, make test results more convenient for a bug fixing.
To achieve these aims, we could start Artipie's server on a local host, reduce restart server (restart only in the tests where it's really needing). The client part of TestDeployment often could be started only once in the BeforeAll method.

Building client images for integration tests takes a lot of time.
We could create according images in advance and use it in these tests and this approach makes tests running 2-3 time faster.

• can we push client images to the GitHub?
• script to auto-build and push client images
• docker client image
• maven client image
• debian client image

[ChGen]

Currently our tests are slow not because of Artipie docker image. artipie/artpie image is fine.
Main culprit is preparation of client or extra working containers. Those are based on clean linux image, e.g. ubuntu:22.04 and for every test class start by maven, container is prepared from the scratch and then dropped.
Basically: apt update ... -> apt install ... -> pip3 update ... -> pip3 install ... -> . So, we should cache such container under some tag on the first run, and than reuse it on subsequent runs. There are corner cases, like, we should track code changes and drop caches somehow. But solving this could give us great speedup.

[ChGen]

Actually, testcontainers may have some caching features, this is to be investigated too.

[ChGen]

Also, when actively debugging/running tests some may start to fail in package downloading, probably due to very frequent http requests to the same files in the repos. Yet another reason to refactor test containers usage.

[ChGen]

Currently, local docker client image caching is supported in: conda, PyPy, rpm and debian S3 integration test cases in artipie-main and adapters. And on conan-adapter for regular integration tests.

[ChGen]

Currently Integration Tests run about one hour.
Below are maven results from logs for -Pitcase for artipie:

2024-02-28T10:29:54.5017318Z [INFO] build-tools ........................................ SUCCESS [  0.538 s]
2024-02-28T10:29:54.5018394Z [INFO] artipie ............................................ SUCCESS [  1.457 s]
2024-02-28T10:29:54.5019362Z [INFO] asto ............................................... SUCCESS [  0.539 s]
2024-02-28T10:29:54.5020450Z [INFO] asto-core .......................................... SUCCESS [ 29.312 s]
2024-02-28T10:29:54.5021555Z [INFO] artipie-core ....................................... SUCCESS [  9.737 s]
2024-02-28T10:29:54.5022703Z [INFO] asto-etcd .......................................... SUCCESS [ 13.887 s]
2024-02-28T10:29:54.5023830Z [INFO] asto-redis ......................................... SUCCESS [ 16.404 s]
2024-02-28T10:29:54.5024941Z [INFO] asto-s3 ............................................ SUCCESS [ 24.219 s]
2024-02-28T10:29:54.5026042Z [INFO] asto-vertx-file .................................... SUCCESS [  2.300 s]
2024-02-28T10:29:54.5027071Z [INFO] The Vert.x server .................................. SUCCESS [  2.066 s]
2024-02-28T10:29:54.5028116Z [INFO] Artipie HTTP client ................................ SUCCESS [ 18.891 s]
2024-02-28T10:29:54.5029261Z [INFO] files-adapter ...................................... SUCCESS [  3.891 s]
2024-02-28T10:29:54.5030385Z [INFO] npm-adapter ........................................ SUCCESS [01:42 min]
2024-02-28T10:29:54.5031570Z [INFO] hexpm-adapter ...................................... SUCCESS [ 28.976 s]
2024-02-28T10:29:54.5032711Z [INFO] maven-adapter ...................................... SUCCESS [07:11 min]
2024-02-28T10:29:54.5033867Z [INFO] rpm-adapter ........................................ SUCCESS [03:10 min]
2024-02-28T10:29:54.5034999Z [INFO] gem-adapter ........................................ SUCCESS [ 46.752 s]
2024-02-28T10:29:54.5036145Z [INFO] composer-files ..................................... SUCCESS [ 19.223 s]
2024-02-28T10:29:54.5037107Z [INFO] goproxy ............................................ SUCCESS [ 18.460 s]
2024-02-28T10:29:54.5038198Z [INFO] nuget-adapter ...................................... SUCCESS [ 19.073 s]
2024-02-28T10:29:54.5039334Z [INFO] pypi-adapter ....................................... SUCCESS [02:19 min]
2024-02-28T10:29:54.5040658Z [INFO] helm-adapter ....................................... SUCCESS [  8.620 s]
2024-02-28T10:29:54.5042089Z [INFO] docker-adapter ..................................... SUCCESS [ 21.425 s]
2024-02-28T10:29:54.5043270Z [INFO] debian-adapter ..................................... SUCCESS [ 38.358 s]
2024-02-28T10:29:54.5044639Z [INFO] conda-adapter ...................................... SUCCESS [05:19 min]
2024-02-28T10:29:54.5045804Z [INFO] conan-adapter ...................................... SUCCESS [01:30 min]
2024-02-28T10:29:54.5046952Z [INFO] artipie-main ....................................... SUCCESS [30:21 min]
2024-02-28T10:29:54.5048034Z [INFO] helm-bench ......................................... SUCCESS [  2.779 s]
2024-02-28T10:29:54.5049118Z [INFO] debian-bench ....................................... SUCCESS [  3.670 s]
2024-02-28T10:29:54.5050393Z [INFO] conda-bench ........................................ SUCCESS [  3.207 s]
2024-02-28T10:29:54.5051491Z [INFO] rpm-bench .......................................... SUCCESS [  2.860 s]
2024-02-28T10:29:54.5052609Z [INFO] asto-artipie ....................................... SUCCESS [  1.381 s]

So, artipie-main is the biggest culprit, runs about 30 minutes. It followed by maven (7min) and conda (5 min) modules.

When run locally artipie-main IT tests, top slowest tests were:

All in artipie-main: 467 total, 1 error, 2 ignored, 464 passed - 34m18s
MavenITCase 4m45s
GemITCase 4m18s
RpmITCase 2m54s
CondaITCase 2m37s
MetadataMavenITCase 1m24s
DockerLocalAuthIT 1m20s
MetadataDockerITCase 59m27s
MavenMultiProxyIT 59m22s
CondaS3ITCase 54m37s

[ChGen]

My current prospect on tests optimization:

1. Use single integration test client image per test(adapter) type, whenever possible. I've noticed that minor differences there take actually lots of extra time and generates new/different layers for docker images. So it affects both ways, downloading prebuilt image and building new image locally. Also it will have smaller total cache footprint, which is nice for local builts and could be more important with GitHub builds.
2. Build tests-related docker images by script from the set of Dockerfiles. Push all images to artipie Docker account by the script. Use prebuilt images in IT tests code. Generating images in code seems easier, but in order to achieve proper level of optimization, extra refactoring still would be required. Also to push images and track versions extra steps would be required too.
3. Extra: could try storing test images on GitHub registry.