Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CI: Add cpp caching into CI build workflows #2718

Closed
wants to merge 31 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
31 commits
Select commit Hold shift + click to select a range
7113c21
Revert genbuild
sieniven Nov 23, 2023
99806f5
Initial dev on build-dev ci workflow
sieniven Nov 23, 2023
856b070
Fix dubious repo
sieniven Nov 23, 2023
7f4cac7
Shift docker defi build out of container
sieniven Nov 23, 2023
1f90cc3
Fix pkg name
sieniven Nov 23, 2023
30940e4
Add zip dep
sieniven Nov 23, 2023
01073f6
Fix populate env
sieniven Nov 24, 2023
dc75853
Remove incorrect copy command in defi dockerfile
sieniven Nov 24, 2023
1eabb7f
Revert dockerfile
sieniven Nov 24, 2023
830cb94
Fix sh check
sieniven Nov 24, 2023
852ce79
Clean up package dir
sieniven Nov 24, 2023
ff59f8f
Upgrade all checkout action to v4
sieniven Nov 24, 2023
9ca4a88
Add rust caching to build dev, release and staging
sieniven Nov 24, 2023
2aba3d1
Resolve copying package into docker image
sieniven Nov 24, 2023
1eb8f22
Remove dockerignore, fix release and staging builds
sieniven Nov 24, 2023
6c75047
Add git config to build dev
sieniven Nov 24, 2023
0469cf8
Better rename
sieniven Nov 24, 2023
2ae88f5
Remove sudo privilege
sieniven Nov 24, 2023
3dfb40d
remove relative path
sieniven Nov 24, 2023
219f8f2
Better rename
sieniven Nov 24, 2023
a8987c1
Debug log
sieniven Nov 24, 2023
b504033
Fix package path
sieniven Nov 24, 2023
16eefad
Better refactor
sieniven Nov 24, 2023
090fc7c
Fix docker build
sieniven Nov 24, 2023
bcbaa6f
Fix relative path
sieniven Nov 24, 2023
9575ca2
Use ccache in build workflow, add caching into build-dev
sieniven Nov 24, 2023
418d24a
Only save cache on merge to master, clean up workflows to restore cache
sieniven Nov 24, 2023
a9b2242
Disable saving cache only on master to test caching
sieniven Nov 24, 2023
f284f6f
Add build type to ci vars
sieniven Nov 24, 2023
05fe58a
Enable save cache
sieniven Nov 24, 2023
e29f75f
Merge branch 'master' of github.com:DeFiCh/ain into niven/add-cpp-cache
sieniven Nov 24, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
1 change: 1 addition & 0 deletions .dockerignore
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
.code
.cache
.ccache
.idea

# Editors.
Expand Down
39 changes: 39 additions & 0 deletions .github/workflows/build-dev.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -47,11 +47,29 @@ jobs:
- name: Setup user dependencies
run: ./make.sh ci-setup-user-deps

- name: Restore build depends
uses: actions/cache/restore@v3
with:
path: ./build/depends
key: ${{ env.TARGET }}-${{ env.BUILD_TYPE }}-depends

- uses: Swatinem/rust-cache@v2
with:
workspaces: lib -> ../build/lib/target
save-if: ${{ github.ref == 'refs/heads/master' }}

- name: Restore cpp build
uses: actions/cache/restore@v3
with:
path: ./build/src
key: ${{ env.TARGET }}-${{ env.BUILD_TYPE }}-cpp

- name: Restore ccache
uses: actions/cache/restore@v3
with:
path: ./build/.ccache
key: ${{ env.TARGET }}-${{ env.BUILD_TYPE }}-ccache

- name: Build and package
run: ./make.sh release

Expand All @@ -61,6 +79,27 @@ jobs:
name: defichain-${{ env.BUILD_VERSION }}-${{ env.TARGET }}
path: ./build/defichain-${{ env.BUILD_VERSION }}-${{ env.TARGET }}.${{ env.PKG_TYPE }}

- name: Cache build depends
# if: ${{ github.ref == 'refs/heads/master' }}
uses: actions/cache/save@v3
with:
path: ./build/depends
key: ${{ env.TARGET }}-${{ env.BUILD_TYPE }}-depends

- name: Cache cpp build
# if: ${{ github.ref == 'refs/heads/master' }}
uses: actions/cache/save@v3
with:
path: ./build/src
key: ${{ env.TARGET }}-${{ env.BUILD_TYPE }}-cpp

- name: Cache ccache
# if: ${{ github.ref == 'refs/heads/master' }}
uses: actions/cache/save@v3
with:
path: ./build/.cacche
key: ${{ env.TARGET }}-${{ env.BUILD_TYPE }}-ccache

docker-build:
runs-on: ubuntu-latest
needs: [build]
Expand Down
14 changes: 13 additions & 1 deletion .github/workflows/lint.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -54,14 +54,26 @@ jobs:
- name: Setup dependencies
run: ./make.sh ci-setup-deps

- name: Setup dependencies for user
- name: Setup user dependencies
run: ./make.sh ci-setup-user-deps

- name: Restore build depends
uses: actions/cache/restore@v3
with:
path: ./build/depends
key: ${{ env.TARGET }}-${{ env.BUILD_TYPE }}-depends

- uses: Swatinem/rust-cache@v2
with:
workspaces: lib -> ../build/lib/target
save-if: ${{ github.ref == 'refs/heads/master' }}

- name: Restore ccache
uses: actions/cache/restore@v3
with:
path: ./build/.ccache
key: ${{ env.TARGET }}-${{ env.BUILD_TYPE }}-ccache

- name: Build depends and configure
run: ./make.sh build-deps && ./make.sh build-conf

Expand Down
18 changes: 18 additions & 0 deletions .github/workflows/tests-ethlibs.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -38,11 +38,29 @@ jobs:
- name: Setup user dependencies
run: ./make.sh ci-setup-user-deps

- name: Restore build depends
uses: actions/cache/restore@v3
with:
path: ./build/depends
key: ${{ env.TARGET }}-${{ env.BUILD_TYPE }}-depends

- uses: Swatinem/rust-cache@v2
with:
workspaces: lib -> ../build/lib/target
save-if: ${{ github.ref == 'refs/heads/master' }}

- name: Restore cpp build
uses: actions/cache/restore@v3
with:
path: ./build/src
key: ${{ env.TARGET }}-${{ env.BUILD_TYPE }}-cpp

- name: Restore ccache
uses: actions/cache/restore@v3
with:
path: ./build/.ccache
key: ${{ env.TARGET }}-${{ env.BUILD_TYPE }}-ccache

- name: Build binaries
run: ./make.sh build

Expand Down
22 changes: 22 additions & 0 deletions .github/workflows/tests-frontier.yml
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,10 @@ concurrency:
group: ${{ github.workflow }}-${{ github.ref || github.run_id }}
cancel-in-progress: ${{ github.ref != 'refs/heads/master' }}

env:
TARGET: x86_64-pc-linux-gnu
MAKE_DEBUG: 0

jobs:
build:
runs-on: ubuntu-latest
Expand All @@ -30,11 +34,29 @@ jobs:
- name: Setup user dependencies
run: ./make.sh ci-setup-user-deps

- name: Restore build depends
uses: actions/cache/restore@v3
with:
path: ./build/depends
key: ${{ env.TARGET }}-${{ env.BUILD_TYPE }}-depends

- uses: Swatinem/rust-cache@v2
with:
workspaces: lib -> ../build/lib/target
save-if: ${{ github.ref == 'refs/heads/master' }}

- name: Restore cpp build
uses: actions/cache/restore@v3
with:
path: ./build/src
key: ${{ env.TARGET }}-${{ env.BUILD_TYPE }}-cpp

- name: Restore ccache
uses: actions/cache/restore@v3
with:
path: ./build/.ccache
key: ${{ env.TARGET }}-${{ env.BUILD_TYPE }}-ccache

- name: Build binaries
run: ./make.sh build

Expand Down
4 changes: 3 additions & 1 deletion .github/workflows/tests-jellyfish.yml
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,8 @@ concurrency:

env:
BUILD_VERSION: latest # Computed
MAKE_DEBUG: 0
GIT_VERSION: 1

jobs:
test:
Expand All @@ -33,7 +35,7 @@ jobs:
node-version: '18'

- name: Populate environment
run: cd defichain && GIT_VERSION=1 ./make.sh ci-export-vars
run: cd defichain && ./make.sh ci-export-vars

- name: Build and setup
run: "cd defichain && docker build -t test-build-container
Expand Down
22 changes: 22 additions & 0 deletions .github/workflows/tests-sync.yml
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,10 @@ on:
- master
types: [labeled, opened, reopened, synchronize]

env:
TARGET: x86_64-pc-linux-gnu
MAKE_DEBUG: 0

concurrency:
group: ${{ github.workflow }}-${{ github.ref || github.run_id }}
cancel-in-progress: ${{ github.ref != 'refs/heads/master' }}
Expand All @@ -40,11 +44,29 @@ jobs:
- name: Setup user dependencies
run: ./make.sh ci-setup-user-deps

- name: Restore build depends
uses: actions/cache/restore@v3
with:
path: ./build/depends
key: ${{ env.TARGET }}-${{ env.BUILD_TYPE }}-depends

- uses: Swatinem/rust-cache@v2
with:
workspaces: lib -> ../build/lib/target
save-if: ${{ github.ref == 'refs/heads/master' }}

- name: Restore cpp build
uses: actions/cache/restore@v3
with:
path: ./build/src
key: ${{ env.TARGET }}-${{ env.BUILD_TYPE }}-cpp

- name: Restore ccache
uses: actions/cache/restore@v3
with:
path: ./build/.ccache
key: ${{ env.TARGET }}-${{ env.BUILD_TYPE }}-ccache

- name: Build binaries
run: ./make.sh build

Expand Down
38 changes: 36 additions & 2 deletions .github/workflows/tests.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,10 @@ on:
branches:
- master

env:
TARGET: x86_64-pc-linux-gnu
MAKE_DEBUG: 0

concurrency:
group: ${{ github.workflow }}-${{ github.ref || github.run_id }}
cancel-in-progress: ${{ github.ref != 'refs/heads/master' }}
Expand Down Expand Up @@ -45,11 +49,29 @@ jobs:
- name: Setup user dependencies
run: ./make.sh ci-setup-user-deps

- name: Restore build depends
uses: actions/cache/restore@v3
with:
path: ./build/depends
key: ${{ env.TARGET }}-${{ env.BUILD_TYPE }}-depends

- uses: Swatinem/rust-cache@v2
with:
workspaces: lib -> ../build/lib/target
save-if: ${{ github.ref == 'refs/heads/master' }}

- name: Restore cpp build
uses: actions/cache/restore@v3
with:
path: ./build/src
key: ${{ env.TARGET }}-${{ env.BUILD_TYPE }}-cpp

- name: Restore ccache
uses: actions/cache/restore@v3
with:
path: ./build/.ccache
key: ${{ env.TARGET }}-${{ env.BUILD_TYPE }}-ccache

- name: Build binaries
run: ./make.sh build

Expand All @@ -73,16 +95,28 @@ jobs:
- name: Setup dependencies
run: ./make.sh ci-setup-deps

- name: Setup dependencies for user
- name: Setup user dependencies
run: ./make.sh ci-setup-user-deps

- name: Restore build depends
uses: actions/cache/restore@v3
with:
path: ./build/depends
key: ${{ env.TARGET }}-${{ env.BUILD_TYPE }}-depends

- uses: Swatinem/rust-cache@v2
with:
workspaces: lib -> ../build/lib/target
save-if: ${{ github.ref == 'refs/heads/master' }}

- name: Restore ccache
uses: actions/cache/restore@v3
with:
path: ./build/.ccache
key: ${{ env.TARGET }}-${{ env.BUILD_TYPE }}-ccache

- name: Build deps and configure
run: ./make.sh build-deps && ./make.sh build-conf

- name: Rust tests
run: ./make.sh lib test
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
.code
.cache
.ccache
.idea

# Editors.
Expand Down
12 changes: 11 additions & 1 deletion make.sh
Original file line number Diff line number Diff line change
Expand Up @@ -689,7 +689,7 @@ pkg_install_deps() {
software-properties-common build-essential git libtool autotools-dev automake \
pkg-config bsdmainutils python3 python3-pip python3-venv libssl-dev libevent-dev libboost-system-dev \
libboost-filesystem-dev libboost-chrono-dev libboost-test-dev libboost-thread-dev \
libminiupnpc-dev libzmq3-dev libqrencode-dev wget \
libminiupnpc-dev libzmq3-dev libqrencode-dev wget ccache \
libdb-dev libdb++-dev libdb5.3 libdb5.3-dev libdb5.3++ libdb5.3++-dev \
curl cmake zip unzip libc6-dev gcc-multilib locales locales-all

Expand Down Expand Up @@ -1140,11 +1140,21 @@ _nproc() {

# shellcheck disable=SC2129
ci_export_vars() {
local build_dir="${BUILD_DIR}"

if [[ -n "${GITHUB_ACTIONS-}" ]]; then
# GitHub Actions
echo "BUILD_VERSION=${IMAGE_VERSION}" >> "$GITHUB_ENV"
echo "PATH=$HOME/.cargo/bin:$PATH" >> "$GITHUB_ENV"
echo "CCACHE_DIR=${build_dir}/.ccache" >> "$GITHUB_ENV"
echo "CARGO_INCREMENTAL=0" >> "$GITHUB_ENV"

if [[ "${MAKE_DEBUG}" == "1" ]]; then
echo "BUILD_TYPE=debug" >> "$GITHUB_ENV"
else
echo "BUILD_TYPE=release" >> "$GITHUB_ENV"
fi

if [[ "${TARGET}" == "x86_64-w64-mingw32" ]]; then
echo "PKG_TYPE=zip" >> "$GITHUB_ENV"
else
Expand Down