Goal

Provide a full Bazel example using multi-languages of a simple data pipeline with sensors, consumers, queues, dispatchers, databases, api(back), web(front). The choices of languages and technologies apart from bazel are arbitrary.

In this repository you can found an integration between Bazel and these technologies.

Services

Each service corresponds to a folder at the root repository.

• ./senror Go : Send an random number to dispatcher.

• ./dispatcher Go : Wait data from sensor(s) and send them to rabbitmqp.

• ./rabbitmq Rabbitmq services

• ./consumer Rust : Retrieved from rabbitmqp store data to Postgresql.

• ./postgresql Postgresql services

• ./graphql Node/Apollo : Create api from Postgresql data.

• ./web (JS & Node) : Display data from graphql endpoint. (Can be outsourced of the cluster, hosted by Vercel for example).

Bazel one command for all code deliveries steps.

Bazel allow to describe and bring together all steps of code delivery with multiple languages.

Install

• Install bazel and ibazel for live reload.

Add this env variables to your .bashrc or .profil or others init shell file.

export REGISTRY=docker.local:5000/pipeline-demo
export CLUSTER=minikube
export REGISTRY_AUTH=False
export POSTGRES_PASSWORD=admin123
export RABBITMQ_DEFAULT_PASS=guest

• Build all $> bazel build //...

Get started (locally)

• Start a local docker registry

docker run -d -p 5000:5000 --name registry registry:2

• Install minikube and kubectl

• Start minikube with insecure docker registry (local registry)

minikube start --insecure-registry "docker.local:5000"

• Go inside minikube with $> minikube ssh. We have to allow services inside minikube to reach docker.local:5000.

  • Install vim $> sudo apt update && sudo apt install -y vim
  • Edit sudo vim /etc/hosts
  • Add docker.local entry with the ip of the host host.minikube.internal
    • Entry example : 192.168.49.1 docker.local
  • ⚠️ You must do it after each time that minikube start.

• Add ingress addon $> minikube addons enable ingress. Ingress allow to reach http services inside k8s.

• Open the dashboard with $> minikube dashboard. This web app allow to display the big picture of the k8s state.

• Run all bazel run //:all.create

• Reach nginx-ingress IP, the entry point of all applications.

# Command :
$> kubectl get ingress nginx-ingress
# Ouput : 
NAME            CLASS    HOSTS   ADDRESS        PORTS   AGE
nginx-ingress   <none>   *       192.168.49.2   80      13m

Tests

• Requirements google-auth, install with command pip install google-auth

• Run tests bazel test //...

Test live (use ibazel) and debug (use --test_output=all)

# Example with dispatcher
ibazel test //dispatcher:test --test_output=all

Development

Build, deploy and test (unit, and integration test)

bazel run //:apps.apply && bazel test //...

Need more faster 🚀! You can restrict the scope of build and deployment. Example by deploying only the dispatcher.

bazel run //dispatcher:app.apply && bazel test //...

You can also restrict tests. Example by executing only dispatcher unit tests.

bazel run //dispatcher:app.apply && bazel test //dispatcher:test

• Development front standalone. Use NextJS with endpoint mapping from localcluster.

$> ./web/run.sh
...
ready - started server on http://localhost:3000
...

Rust

Add dependencies

• Add dependecies to Cargo.toml
• Run cargo raze in the package folder

Go

Add dependencies

• Add dependencies to go.mod
• Run bazel run //:gazelle -- update-repos -from_file=<service directory>/go.mod

Tools

Log

Local/Remote stream log : Stern display pod log, example stern <podname>.

Cli

Package //util/tools provide some shortcuts for get service urls or cluster urls in development mode.

$> cd <project WORKSPACE path>
$> bazel build //util/tools
$> ./bazel-bin/util/tools/tools
Usage: tools.py [OPTIONS] COMMAND [ARGS]...

Options:
  --help  Show this message and exit.

Commands:
  get-cluster-url
  get-service-url

Troubleshooting

JS

Lock file update and generation have to do manually, emaple with front package.

cd ./front/
npm i --package-lock-only