Speed up pipeline #340

deemp · 2024-05-14T21:24:35Z

close Cache results of pipeline tests without normalization #87
close Install optimized normalizer in the pipeline script #339
close Run tests and pipeline in parallel #346
close Fix pipeline failure on Windows #341
close Speed up pipeline #66

Additionally:

fix scripts to work on Windows and non-Windows

TODO

write about cygwin and cygpath on the pipeline page on the site
fix artifact name
in Setup.hs, run the command in bash
Cache generated Syntax files so that it's not necessary to rebuild the project only because of them
- Restore them before stack-action using the hash of Syntax.cf and stack.yaml.lock as a key
- Create an output flag that is true if all files that must be generated are present
- If this output if false, run full rebuild in stack-action

PR-Codex overview

This PR focuses on updating the pipeline scripts, adding Normalizer installation, and enhancing pipeline lock management.

Detailed summary

Added Normalizer installation and path setup
Enhanced pipeline lock management
Updated pipeline scripts and directory structure

The following files were skipped due to too many changes: scripts/pipeline.sh, .github/workflows/ghc.yml

✨ Ask PR-Codex anything about this PR by commenting with /codex {your question}

- move some functions to lib - use pipeline lock to decide whether to run tests without normalization - some directories - don't clean directories in "normalize"

…rkflow

- add EO version constant - add eo function

so that pipeline.lock can be compared with pipeline_new.lock even if core.autocrlf is true

deemp added 14 commits May 14, 2024 20:13

fix(pipeline): clean output directory at each step

6066113

fix(pipeline): shellcheck warnings

eb0d312

fix(pipeline): run normalizer executable instead of "stack run"

32c9d30

feat(pipeline): add pipeline.lock that tracks eo and pipeline config

91204af

feat(vscode): add shellcheck setting to allow script imports

746e92a

feat(scripts): add a library

acf00c3

feat(gitignore): unignore pipeline lock

fa16ee6

fix(pipeline):

fab23ad

- move some functions to lib - use pipeline lock to decide whether to run tests without normalization - some directories - don't clean directories in "normalize"

refactor(scripts): move update_pipeline_lock to lib to simplify GH wo…

fc41efa

…rkflow

fix(ci): let the pipeline script calculate EO version

e366e94

feat(scripts): add function to commit and push changed files

e1c9ea6

feat(ci): add steps to restore and cache pipeline files

4af9292

feat(ci): add step to commit updated pipeline lock

46b08ce

refactor(ci): use a lib function

b1f837f

deemp linked an issue May 14, 2024 that may be closed by this pull request

Cache results of pipeline tests without normalization #87

Closed

deemp added 4 commits May 15, 2024 00:39

refactor(scripts): use constants

652b79f

fix(scripts, ci): force pipeline_lock_changed if no cache hit

538ba83

fix(ci): use a ternary expression

b4acdcd

fix(ci): use a comparison expression

4045c68

deemp force-pushed the 87-cache-results-of-tests-without-normalization branch from fe3c911 to 4045c68 Compare May 14, 2024 22:58

fix(ci): use comparisons in both expressions

1d1cf5d

deemp force-pushed the 87-cache-results-of-tests-without-normalization branch from c7b1dbf to 1d1cf5d Compare May 14, 2024 23:50

fix(ci): set PATH on macOS

fc6c0f6

deemp force-pushed the 87-cache-results-of-tests-without-normalization branch from a25caf2 to fc6c0f6 Compare May 15, 2024 00:06

fix(ci): prefix pipeline cache

6e69307

deemp requested a review from fizruk May 15, 2024 00:29

deemp added 4 commits May 15, 2024 10:14

fix(scripts): default import_dir

05d702a

refactor(scripts): use lib

7ce29d5

fix(scripts): pass arguments to eoc

19e67fd

feat(scripts):

253f32d

- add EO version constant - add eo function

deemp added 4 commits May 16, 2024 15:19

fix(scripts, ci): pipeline lock code

c93cce5

fix(scripts, lib): set UTF-8 in lib

61eb977

refactor(scripts): calculate EO version when necessary to not require yq

cf79b67

refactor(ci): install yq and check pipeline lock when necessary

25ea6ce

deemp force-pushed the 87-cache-results-of-tests-without-normalization branch 3 times, most recently from abfd86c to 46e06ea Compare May 16, 2024 13:48

fix(git): line endings in pipeline.lock

1c3d707

so that pipeline.lock can be compared with pipeline_new.lock even if core.autocrlf is true

deemp force-pushed the 87-cache-results-of-tests-without-normalization branch from 46e06ea to 1c3d707 Compare May 16, 2024 13:53

fix(ci): commit files only in case of success

da39299

deemp force-pushed the 87-cache-results-of-tests-without-normalization branch from 0377fac to 0cb793c Compare May 16, 2024 14:21

fix(ci): run tests and pipeline in separate jobs

c33d373

deemp force-pushed the 87-cache-results-of-tests-without-normalization branch from 0cb793c to c33d373 Compare May 16, 2024 14:22

deemp changed the title ~~87 cache results of pipeline tests without normalization~~ Speed up pipeline May 16, 2024

deemp added 7 commits May 16, 2024 18:53

fix(scripts): cygpath name

3f40201

feat(site): write about cygpath on the site

0e434ec

fix(ci): artifact logic

4b3333f

fix(ci): set code page before stack-action

725d5e4

fix(normalizer): run a bash command in Setup.hs

2057e19

try(ci): cache syntax files

1a039d5

fix(ci): use non-empty strigs because they're truthy

a409bb9

deemp force-pushed the 87-cache-results-of-tests-without-normalization branch from 54eabc4 to 0daeaa8 Compare May 17, 2024 00:28

fix(ci): check that syntax files exist

c1888c9

deemp force-pushed the 87-cache-results-of-tests-without-normalization branch from 0daeaa8 to c1888c9 Compare May 17, 2024 00:30

deemp added 2 commits May 17, 2024 03:51

fix(scripts): generate eo files only when lock changed

998fbac

fix(ci): run tests on a separate step

f7e575d

deemp force-pushed the 87-cache-results-of-tests-without-normalization branch from cfbb6d7 to f7e575d Compare May 17, 2024 01:14

fizruk approved these changes May 17, 2024

View reviewed changes

fizruk merged commit ef486c6 into master May 17, 2024
7 of 8 checks passed

fizruk deleted the 87-cache-results-of-tests-without-normalization branch May 17, 2024 03:01

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Speed up pipeline #340

Speed up pipeline #340

deemp commented May 14, 2024 •

edited by pr-codex bot

Speed up pipeline #340

Speed up pipeline #340

Conversation

deemp commented May 14, 2024 • edited by pr-codex bot

PR-Codex overview

Detailed summary

deemp commented May 14, 2024 •

edited by pr-codex bot