This "HelloWorld" Helm Operator component was shamelessly copied (with permission) from Swati Nair of IBM and lays down a simple BusyBox component through a Helm chart via an operator. This component is an example of a fully onboarded component in the new CICD pipeline including build, test, deploy, and publishing to Quay. This README will walk you through onboarding an existing or new component onto the new CICD Pipeline.
If you need to access RedHat infrastructure, check out our Onboarding Guide.
In order to interface your component with the pipeline, you need to include some artifacts produced by CICD in your repo:
Makefileentries to bootstrap the build harness + build harness extensions. This line will pull build-harness-extensions which will also pull and configure the build-harness..travis.ymltemplate that defines the CICD provided travis job interface. Any input required from a squad will be parameterized and noted.
You'll also need to produce some artifacts that are specific to your component, required for interface with the component and pipeline build harness module:
COMPONENT_NAMEfile containing the name of your component to be produced by your build.COMPONENT_VERSIONfile containing the version of your component to be produced by your build.- Within the template
.travis.yml, scripts that you are expected to supply in the root of your component directory will be called as appropriate:install-build-dependencies.sh- install any build dependencies you havebuild.sh- execute any build actionsunit-test.sh- execute unit testse2e-test.sh- execute integration tests, assumptions can be made that the broader environment will be up
Your component should have a branch for each release of the product, for example this first release should have an associated branch release-1.0.0. All work towards that product release should go in that branch.
The branch name will be used to match with the destination branch in the pipeline repo. If your component isn't in a release version of the pipeline repo, it won't be picked up by the pipeline! If you have a release-versioned branch that doesn't match one in the pipeline repo, your build will fail on the publish step!
Using the .travis.yml template provided, your jobs will have the following behavior:
PR builds will run the build, unit-test, and test-e2e stages. These stages will build your component and publish a PR-tagged image to quay.io, run your unit tests, deploy your component to an OCP cluster, and run your e2e tests.
Builds on release verisoned branches will run the build and publish stages. These stages will build your component and push to quay.io with final tags (<version>-<git-sha>) and update the CICD pipeline repo's integration manifest to point to your new image. Note: the Pipeline Manifest will be updated in the git branch with a name matching your current git branch. If there is no matching release-<version> branch in the pipeline repo, the publish stage will error and you should contact CICD!
If a branch name is not present in the list of publish branches in your travis.yml:
- stage: publish
name: "Publish the image to quay with an official version/sha tag and publish entry to integration pipeline stage"
if: type = push AND branch IN ( master )
under branch in (<branches>) the travis job will simply build an image tagged with <version>-<git-sha> and push it to quay so you can test!
First, start by copying the .travis.yml from this repository into your repository. We'll update/build this this throughout the following sections. We won't have to make too many edits, and almost all of the edits will be in the env: section.
Ask the CICD team to onboard your project in SonarCloud. You will get three things back:
- SonarCloud encrypted token
- SonarCloud project key
- SonarCloud project name
The encrypted token will be added to the .travis.yml file. The project key and project name will be added to the sonar-project.properties file.
Add the encrypted token to the .travis.yml file by adding this snippet just before the env: section:
addons:
sonarcloud:
organization: "open-cluster-management"
token:
secure: "ENCRYPTED_TOKEN"
For a Go project, create a sonar-project.properties file at the top of the repo using this template:
sonar.projectKey=PROJECT_KEY
sonar.projectName=PROJECT_NAME
sonar.sources=.
sonar.exclusions=**/*_test.go,**/*_generated*.go,**/*_generated/**,**/vendor/**
sonar.tests=.
sonar.test.inclusions=**/*_test.go
sonar.test.exclusions=**/*_generated*.go,**/*_generated/**,**/vendor/**
sonar.go.tests.reportPaths=report.json
sonar.go.coverage.reportPaths=coverage.out
sonar.externalIssuesReportPaths=gosec.json
This template is in this repo as the file sonar-project.properties.go.example.
Next, in the .travis.yml add this line to the end of the script in the unit-test stage:
make sonar/go
The stage should look something like this:
- stage: unit-test
name: "Run unit tests"
if: type = pull_request
script:
# Set the image tag differently for PRs
- if [ "$TRAVIS_PULL_REQUEST" != "false" ]; then export COMPONENT_TAG_EXTENSION="-PR${TRAVIS_PULL_REQUEST}-${TRAVIS_COMMIT}"; fi;
# Bootstrap the build harness, pull test image, and run unit tests.
- |
make
make component/pull
make component/test/unit
make sonar/go
Note that this repo is a python project and so its unit-test stage has the sonar-scanner --debug line as the last statement in the script. Be sure to remove this line for a Go project as it's part of the make sonar/go recipe.
For a Node project that uses Jest for testing, create a sonar-project.properties file at the top of the repo using this template:
sonar.projectKey=PROJECT_KEY
sonar.projectName=PROJECT_NAME
sonar.sources=.
sonar.exclusions=tests/**/*,test-output/**/*,node_modules/**/*
sonar.tests=tests
sonar.javascript.lcov.reportPaths=test-output/coverage/lcov.info
sonar.testExecutionReportPaths=test-report.xml
This template is in this repo as the file sonar-project.properties.js-jest.example.
Next, in the jest.config.js file, add this line to the JS object being stored as const jestConfig:
testResultsProcessor: 'jest-sonar-reporter',
Finally, in the .travis.yml file, add this line to the script in the unit-test
stage just after make component/test/unit to install the jest-sonar-reporter module:
make sonar/js/jest-init
and this line to the end of the script in the unit-test stage:
make sonar/js
The stage should look something like this:
- stage: unit-test
name: "Run unit tests"
if: type = pull_request
script:
# Set the image tag differently for PRs
- if [ "$TRAVIS_PULL_REQUEST" != "false" ]; then export COMPONENT_TAG_EXTENSION="-PR${TRAVIS_PULL_REQUEST}-${TRAVIS_COMMIT}"; fi;
# Bootstrap the build harness, pull test image, and run unit tests.
- |
make
make component/pull
make sonar/js/jest-init
make component/test/unit
make sonar/js
For a project that is not covered above, create a sonar-project.properties file at the top of the repo using this template:
sonar.projectKey=PROJECT_KEY
sonar.projectName=PROJECT_NAME
Note that this project will not have code coverage or unit test results. Those reports will need to be configured on a case by case basis. Contact the CICD team for help implementing these reports for other languages.
Next, in the .travis.yml add this line to the end of the script in the unit-test stage:
sonar-scanner --debug
The stage should look something like this:
- stage: unit-test
name: "Run unit tests"
if: type = pull_request
script:
# Set the image tag differently for PRs
- if [ "$TRAVIS_PULL_REQUEST" != "false" ]; then export COMPONENT_TAG_EXTENSION="-PR${TRAVIS_PULL_REQUEST}-${TRAVIS_COMMIT}"; fi;
# Bootstrap the build harness, pull test image, and run unit tests.
- |
make
make component/pull
make component/test/unit
sonar-scanner --debug
To onboard a new repo to SonarCloud:
If the project is not in the list:
- Click the "+" in the top right and select "Analyze new project"
- Check the box next to the repo in the list. If the repo is not here, verify it is present in GitHub.
- Click "Set Up" on the right side of the page to add the repo as a SonarCloud project.
Once the project is in the list:
- Find the project in the list and click "Configure Analysis"
- Click the "With Travis CI" link
- Copy the
travis encrypt a1b2c3d4command shown. - Paste the command in the command line and add the option
-r open-cluster-management/<repo-name> - Example:
travis encrypt -r open-cluster-management/grc-ui a1b2c3d4 - Run the command to generate an encrypted token and save this token.
- Click the "Continue" link.
- Under "What is your build technology?" click "Other".
- Click "Continue"
- In the sample
sonar-project.propertiesfile shown, copy the values forsonar.projectKeyandsonar.projectName - Send the encrypted token, project key, and project name back to the requester.
You'll need to identify which scripts you need to implement out of the following list one-by-one. CICD is working to provide generalized scripts to accomodate some common components types for component build and deploy, but we won't capture all of the possibilities.
You can find the generalized component scripts available through the build-harness-extensions component module here. If you're able to find a bin of scripts that partially or fully matches your component, update COMPONENT_TYPE in .travis.yml to that bin name. As an example, this component is a helmoperator.
Once you've reviewed the available bins of scripts and set COMPONENT_TYPE if relevant, walk through each of the next subsections:
The install-dependencies script is used to install all of the dependencies for your build.
If you found a component in the component module bin with an install-dependencies script that matches your needs, make sure COMPONENT_INIT_COMMAND isn't set in your .travis.yml and skip to the next subsection.
If you didn't find a component in the component module bin matching your needs, implement a script (we recommend calling it install-dependencies.sh for clarity) and put it in the base directory of your component's git repo. Set COMPONENT_INIT_COMMAND in .travis.yml to ${TRAVIS_BUILD_DIR}/<script-name>.
In this component, the install-dependenices script for this repo is defined by the helmoperator bin
The build script is used by the build harness component module to build your component. The script will be called with an argument that is the <repo>/<component>:<tag> of the component your build script needs to output. This component will be referenced by that input identifier in the following steps, so the naming/tagging must be correct, if not your component won't enter the pipeline!
If you found a component in the component module bin with a build script that matches your needs, make sure COMPONENT_BUILD_COMMAND isn't set in your .travis.yml and COMPONENT_TYPE is set to that component bin, and skip to the next subsection.
If you didn't find a component in the component module bin matching your needs, implement a script (we recommend calling it build.sh for clarity) and put it in the base directory of your component's git repo. Set COMPONENT_BUILD_COMMAND in .travis.yml to ${TRAVIS_BUILD_DIR}/<script-name>.
In this component, the build.sh script for this repo is defined by the helmoperator bin
The unit-test script is used by the build harness component module to run your unit tests. The script will be called with an argument that is the <repo>/<component>:<tag> of the component that your unit tests scripts need to test. If your tests occur as part of the build, you'll need to implement a script that for this stage that is a no-op and passes cleanly!
If you found a component in the component module bin with a unit-test script that matches your needs, make sure COMPONENT_UNIT_TEST_COMMAND isn't set in your .travis.yml and COMPONENT_TYPE is set to that component bin, and skip to the next subsection.
If you didn't find a component in the component module bin matching your needs, implement a script (we recommend calling it run-unit-tests.sh for clarity) and put it in the base directory of your component's git repo. Set COMPONENT_UNIT_TEST_COMMAND in .travis.yml to ${TRAVIS_BUILD_DIR}/<script-name>.
In this component, the run-unit-tests.sh script for this repo is defined by the helmoperator bin
The deploy script is used by the build harness component module to deploy your component to a running OCP 4.X cluster in preparation for e2e testing. The script will be called with an argument that is the <repo>/<component>:<tag> of the component that your script needs to deploy.
If you found a component in the component module bin with a deploy-to-cluster script that matches your needs, make sure COMPONENT_DEPLOY_COMMAND isn't set in your .travis.yml and COMPONENT_TYPE is set to that component bin, and skip to the next subsection.
If you didn't find a component in the component module bin matching your needs, implement a script (we recommend calling it deploy-to-cluster.sh for clarity) and put it in the base directory of your component's git repo. Set COMPONENT_DEPLOY_COMMAND in .travis.yml to ${TRAVIS_BUILD_DIR}/<script-name>.
In this component, the deploy-to-cluster.sh script for this repo is defined in the repository and is passed into the .travis.yml as an environment variable:
env:
global:
...
- COMPONENT_DEPLOY_COMMAND=${TRAVIS_BUILD_DIR}/deploy-to-cluster.sh
...
The e2e-test script is used by the build harness component module to run your e2e tests on your PR component deployed on an OCP cluster. The script will be called with an argument that is the <repo>/<component>:<tag> of the component that your e2e tests scripts need to test.
If you found a component in the component module bin with a run-e2e-tests script that matches your needs, make sure COMPONENT_E2E_TEST_COMMAND isn't set in your .travis.yml and COMPONENT_TYPE is set to that component bin, and skip to the next subsection.
If you didn't find a component in the component module bin matching your needs, implement a script (we recommend calling it run-e2e-tests.sh for clarity) and put it in the base directory of your component's git repo. Set COMPONENT_E2E_TEST_COMMAND in .travis.yml to ${TRAVIS_BUILD_DIR}/<script-name>.
In this component, the run-e2e-tests.sh script for this repo is defined by the helmoperator bin
COMPONENT_NAME should be a file containing the name of your component to be produced by your build.
COMPONENT_VERSION file containing the version of your component to be produced by your build.
Make sure you either copy the Makefile from this repository, or place the contents of this Makefile within your own Makefile. We rely on the makefile to bootstrap the build-harness and build-harness-extensions.
After following all of the above, ensure that your copy of the .travis.yml has all of the "Component Specific" environment variables defined to your liking, and ensure that all "Required" environment variables are properly defined.
You need to properly set the following environment variables under "Settings" for the Travis job for your component repository:
DOCKER_USERandDOCKER_PASS: the username and token for a robot account for the destination Quay repository. You can learn more about creating a Robot Account hereGITHUB_USERandGITHUB_TOKEN: the username and token for a GitHub user with access to your component repository and the build harness and extensions. Generate an OAuth token for your GitHub user according to GitHub's official documentation on the subject, giving it admin permission for repo and repo hooks, and place that token in theGITHUB_TOKENvariable in your travis job.
If you've made it this far, your component should now be prepared for the CICD pipeline. You can verify your .travis.yml using the travis CLI's linter. Then its time to start up your Travis job and try it out. Contact CICD at #acm-devops in CoreOS Slack with any questions.