Vagrant-hashistack

Hashistack in one click for development & testing

This vagrant box aims to make it dead simple to start a hashistack and emulate how services will be deployed to production.

---

🚧 - current vagrant box runs consul, nomad and vault in dev (development) mode.

• consul development mode
• nomad development mode
• vault development mode

---

Content

1. Description - what & why
   1. Submodules
   2. Services
   3. Why does this exist?
   4. Installed stack
   5. Versions
2. Install prerequisites
   1. General requirements
      1. Proxy
         1. MacOS
   2. Linux requirements
   3. MacOS requirements
   4. Windows requirements
      1. GNU Make and Git Bash
3. Build the vagrant box
4. Configuration
   1. Default Configuration
   2. Override default configuration
      1. Option 1 - Change environment variables
      2. Option 2 - Add/Override the configuration files
      3. Option 3 - Add/Override the -pre and -post tasks
5. Usage
   1. Option-1 Starting a plain default box
      1. Port collisions
         1. Option 1 Shut down the running machine
         2. Option 2 Use the auto_correct feature to dynamically allocate ports
   2. Option-2 Starting a new project based on the template
   3. Custom port forwarding
6. Test
   1. Local run
   2. CI pipeline run
      1. CI test configuration
7. Diagram
8. Contribute

Description - what & why

This repository will build a base-box for different projects to extend on. The base-box contains components, and a setup that makes it ideal for working with the hashistack.

Hashistack, in current repository context, is a set of software products by HashiCorp.

Submodules

Current repository contains submodules

• vagrant-hashistack-template

Check Using submodules in Git - Tutorial

Current repository points to concrete commit in submodule(s)

To point to the latest commit in submodule master

make update-submodule

Services

The default box will start Nomad, Vault, Consul and MinIO bound to loopback and advertising on the IP 10.0.3.10, which should be available on your local machine. Port-forwarding for nomad on port 4646 should bind to 127.0.0.1 and should allow you to use the nomad binary to post jobs directly. Consul and Vault have also been port-forwarded and are available on 127.0.0.1 on ports 8500 and 8200 respectively. Minio is started on port 9000 and shares the /vagrant (your repo) from within the vagrant box.

Service	URL	Token(s)
Nomad	http://10.0.3.10:4646
Consul	http://10.0.3.10:8500	master
Vault	http://10.0.3.10:8200	master
Minio	http://10.0.3.10:9000	minioadmin : minioadmin

Why does this exist?

We needed a Vagrant box with the complete hashistack to use for demo, development and testing. In order to build cloud native, security minded and dependable services, there exists a killer combination;

• Containers - (Docker)
• Simple&Powerful Orchestrator - (Nomad)
• Service-mesh mTLS - (Consul connect)
• Secrets management - (Vault)

Installed stack

• Consul
• Nomad
• Vault
• Terraform
• Docker CE

with a side-play of

• Ansible
• Packer
• Consul-template
• Minio

Versions

vagrant-hashistack	terraform	consul	nomad	vault	packer	consul-template	remarks
0.11.1	1.0.5	1.9.8	1.1.3	1.7.3	1.7.4	0.27.0
0.11.0	1.0.5	1.9.8	1.1.3	1.7.3	1.7.4	0.27.0
0.10.0	0.14.1	1.9.1	1.0.2	1.6.1	1.6.6	0.25.1
0.9.0	0.13.5	1.8.5	0.12.8	1.6.0	1.6.5	0.25.1
0.7.1	0.13.4	1.8.4	0.12.5	1.5.4	1.6.4	0.25.1
0.7.0	0.13.4	1.8.4	0.12.5	1.5.4	1.6.4	0.25.1
0.6.0	0.13.4	1.8.4	0.12.5	1.5.4	1.6.4	0.25.1
0.5.0	0.13.2	1.8.4	0.12.4	1.5.3	1.6.1	0.25.1
0.4.3	0.13.2	1.8.4	0.12.4	1.5.3	1.6.1	0.25.1
0.4.2	0.13.2	1.8.4	0.12.4	1.5.3	1.6.1	0.25.1
0.4.1	0.13.2	1.8.4	0.12.3	1.5.3	1.6.1	0.25.1
0.4.0	0.13.1	1.8.3	0.12.3	1.5.2	1.6.0	0.25.1	vault 1.5.2.1+ent

Install prerequisites

make install

The command, will install:

• VirtualBox
• Packer
• Vagrant with additional plugins
• Additional software dependent on the OS (Linux, MacOS)

General requirements

You will need to have the following binaries pre-installed:

• Make
• Git CLI

---

NB Post installation you might need to reboot your system in order to start the virtual-provider (VirtualBox)

The rest of the Requirements are operative system dependent

---

Proxy

If you for any reason find yourself behind a transparent proxy you need to set the environment variables SSL_CERT_FILE and CURL_CA_BUNDLE. You have three options:

• Prefix vagrant up; SSL_CERT_FILE=<path/to/ca-certificates-file> CURL_CA_BUNDLE=<path/to/ca-certificates-file> vagrant up
• Set the environment variables in your current session by running export SSL_CERT_FILE=<path/to/ca-certificates-file> and export CURL_CA_BUNDLE=<path/to/ca-certificates-file> in the terminal. Eg:export SSL_CERT_FILE=/etc/ssl/certs/ca-certificates.crt CURL_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt
• Set the environment variables permanently by adding the above export commands to your ~/.bashrc or equivalent.

MacOS

MacOS uses its own keychain, which means you have no certificates-file to point to. You can solve this by exporting all certificates present in the keychain to a file.

security find-certificate -a -p > ~/certs.pem

The command above will export all system and personal certificates in your keychain to certs.pem in ~ (your user home)

 printf '\n### You added this for vagrant box certs ###\nexport SSL_CERT_FILE=~/certs.pem CURL_CA_BUNDLE=~/certs.pem\n' >> ~/.zshrc
 source ~/.zshrc

This will permanently export paths to your shell expected by tools used in this vagrant box.

Linux requirements

• Virtualization must be enabled. Error if it is not.
• Packages gpg and apt must be installed.

MacOS requirements

• Virtualization must be enabled. This is enabled by default on MacOS.
• Homebrew must be installed.

Windows requirements

Most developers are likely to be using Linux or Mac as their development box, however the install procedure should also work on Windows.

GNU Make and Git Bash

• Git. Use the download link and follow the steps.

⚠️ Make sure to install Git Bash when installing Git

• GNU Make. Run the commands below in your Git Bash, or any other terminal.

curl -O https://managedway.dl.sourceforge.net/project/ezwinports/make-4.3-without-guile-w32-bin.zip
unzip -d "/c/Program Files/Git/mingw64" make-4.3-without-guile-w32-bin.zip

Next, make sure to start git bash as admin shell (start > Git Bash > rightclick > Run as administrator). Now you are all set to run

make install

Build the vagrant box

Make build will build a vagrant box based on ubuntu/hirsute64. The packaged box will be locally available at ´packer/output-hashistack/package.box´:

make build

Make dev will build a vagrant box as make build, but without running the tests

make dev

Make test runs through your ansible playbook test, you can give the box different attributes by adding them in the.env file under the test/ directory:

make test

Make clean will destroy and take down your box if there is any:

make clean

💡 You can refer to the configuration section in order to get a comprehensive overview of the default configurations with which the system is set up. The section also provides information about steps for overriding the default system configuration.

Configuration

The vagrant-box is set up with a default configuration (see subsection Default Configuration), but users may want to customize the vagrant-box services configuration to fit their needs.

In order to simplify making such changes in the configuration, we provide switches. These switches are controlled by environment-variables, and provide the user with the opportunity to quickly switch between different configurations. The next sections will describe the defaults, and how to change them.

💡 All supported switches are listed under # Control box features section in the .env_default.j2 file.

Default Configuration

Each of the following links lead to the configuration file and is the default values when setting up the box.

Default configuration files

• .env_default.j2
• hashistack component versions
• nomad policies
• consul policies

Defaults Consul

• Using open source version
• ACL enabled=true
• default_policy=allow

Defaults Nomad

• Using open source version
• ACL enabled=false
• Integrated with Consul, using consul master token
• Integrated with Vault, using token issued by Vault

💡 There is opportunity to edit nomad-server policy in Vault UI.

Defaults Vault

• Using open source version
• Unsealed
• PKI enabled at /pki

Override default configuration

To override the default configuration you have several options:

• Change the environment variables
• Add/Override the configuration files
• Add/Override the -pre and -post tasks

⚠️ Overriding the configuration files will take effect last. In other words, using config files (Option 2) will override any configuration which were setup by the env variables (Option 1)

Option 1 - Change environment variables

When the vagrant box is provisioned, it reads the data from the following environment file /home/vagrant/.env_default in order to set up the system. If you wish to override any of the default values then you can do so by adding that variable name and value in .env file. The property values in the .env file override the property values present in the .env_default file and thus makes it simple to provision systems that suffice the relevant development needs.

Say you want to do some tweaks and use Consul Enterprise, an open-source Nomad and Vault Enterprise. Then you would need to override the following variables:

• Consul Enterprise:
  1. consul_enterprise=true
• Nomad open source:
  1. nomad_enterprise=false
• Vault Enterprise:
  1. vault_enterprise=true

In order to override the default configuration, the following needs to be added to the .env file:

consul_enterprise=true
nomad_enterprise=false
vault_enterprise=true

Option 2 - Add/Override the configuration files

It is possible to add and/or override the hashistack components configuration files. See documentation here

Option 3 - Add/Override the -pre and -post tasks

It is possible to add and/or override the hashistack components -pre and -post tasks. See documentation here

Usage

Vagrant-hashistack provides these features:

• Deploy & test terraform modules
• Deploy & test nomad jobs
• Upload files to Minio
• Test automation

Option-1 Starting a plain default box

To get a running VM using the latest release of this box run

vagrant init Skatteetaten/hashistack
vagrant up --provision

The first command will add a file called Vagrantfile to your directory, and vagrant up will start a box based on the specifications of that file.

NB If you are behind a transparent proxy, follow proxy documentation

Port collisions

If you get the error message

Vagrant cannot forward the specified ports on this VM, since they
would collide with some other application that is already listening
on these ports. The forwarded port to 8500 is already in use
on the host machine.

you do most likely have another version of the vagrant-box already running and using the ports. You can solve this in one of two ways:

Option 1 Shut down the running machine

Run

vagrant status

to see all running boxes. Then run

vagrant destroy <box-name>

to take it down. Doc on what vagrant destroy does.

Option 2 Use the auto_correct feature to dynamically allocate ports

Vagrant has a configuration option called auto_correct which will use another port if the port specified is already taken. To enable it you can add the lines below to the bottom of your Vagrantfile.

Vagrant.configure("2") do |config|
    # Hashicorp consul ui
    config.vm.network "forwarded_port", guest: 8500, host: 8500, host_ip: "127.0.0.1", auto_correct: true
    # Hashicorp nomad ui
    config.vm.network "forwarded_port", guest: 4646, host: 4646, host_ip: "127.0.0.1", auto_correct: true
    # Hashicorp vault ui
    config.vm.network "forwarded_port", guest: 8200, host: 8200, host_ip: "127.0.0.1", auto_correct: true
end

This will enable the autocorrect-feature on the ports used by consul, nomad, and vault.

💡 You can find out more about Vagrantfiles here

Option-2 Starting a new project based on the template

To see a full example of how to start a new project based on this box go to template-repo.

NB If you are behind a transparent proxy, follow proxy documentation

Custom port forwarding

There is opportunity to forward ports from vagrant box instance to localhost using vagrant ssh --

vagrant ssh -- --help

usage: ssh [-46AaCfGgKkMNnqsTtVvXxYy] [-B bind_interface]
           [-b bind_address] [-c cipher_spec] [-D [bind_address:]port]
           [-E log_file] [-e escape_char] [-F configfile] [-I pkcs11]
           [-i identity_file] [-J [user@]host[:port]] [-L address]
           [-l login_name] [-m mac_spec] [-O ctl_cmd] [-o option] [-p port]
           [-Q query_option] [-R address] [-S ctl_path] [-W host:port]
           [-w local_tun[:remote_tun]] destination [command]

The syntax is vagrant ssh -- -L <local machine port>:<host inside machine>:<port inside machine>

The example below shows how to forward MinIO (which is used as an artifact manager) port :9000 from vagrant box to localhost port :6666

vagrant ssh -- -L 6666:localhost:9000

💡 Custom port forwarding vagrant ssh -- command could be used in cases when you need to forward ports of consul-connect proxies from vagrant box to localhost machine

💡 cli command could take several args

Example

vagrant ssh -- -L 7001:localhost:9999 -L 7002:localhost:9000

Test

There are two options how to run tests:

• run all tests on local machine
• run all tests in CI (env variable CI)

Options are controlled by environment variable CI.

Local run

CI env variable is not set.

make test

The above command runs the tests by starting the countdash consul-connect example. If ´packer/output-hashistack/package.box´ does not exist, it will run ´make build´.

CI pipeline run

CI env variable set to any non-null value.

make test

The tests are run using Github Actions feature which makes it possible to automate, customize, and execute the software development workflows right in the repository.

We utilize the matrix testing strategy to cover all the possible and logical combinations of the different properties and values that the components support. The .env_override file is used by the tests to override the values that are available in the .env_default file, as well as the user configurable .env file.

Pay attention that we pass extra-vars ci_test=true to the ansible provisioner. Full example

CI test configuration

As of today, the following tests are executed whenever a Pull request is created :

Test name	Consul Acl	Consul Acl Policy	Nomad Acl	Hashicorp binary
test (consul_acl_enabled, consul_acl_deny, nomad_acl_enabled, hashicorp_oss)	true	deny	true	Open source
test (consul_acl_enabled, consul_acl_deny, nomad_acl_enabled, hashicorp_enterprise)	true	deny	true	enterprise
test (consul_acl_enabled, consul_acl_deny, nomad_acl_disabled, hashicorp_oss)	true	deny	false	Open source
test (consul_acl_enabled, consul_acl_deny, nomad_acl_disabled, hashicorp_enterprise)	true	deny	false	enterprise
test (consul_acl_disabled, consul_acl_deny, nomad_acl_enabled, hashicorp_oss)	false	deny	true	Open source
test (consul_acl_disabled, consul_acl_deny, nomad_acl_enabled, hashicorp_enterprise)	false	deny	true	enterprise
test (consul_acl_disabled, consul_acl_deny, nomad_acl_disabled, hashicorp_oss)	false	deny	false	Open source
test (consul_acl_disabled, consul_acl_deny, nomad_acl_disabled, hashicorp_enterprise)	false	deny	false	enterprise

The latest test results can be looked up under the Actions tab Actions

Diagram

Contribute

See here