Omicron is the control plane for an Oxide rack. It expects to execute on Helios systems, and Sleds use Helios-specific interfaces to manage resources.
If you’re interested in running the control plane on other platforms, including Linux and Mac, refer to the guide on running simulated Omicron.
A major prerequisite is to have a machine already running Helios. An easy way to do this is by using a Helios VM. ISOs are also available for download here.
Any additional prerequisite software may be installed with the following script:
$ ./tools/install_prerequisites.shThis script expects that you are both attempting to compile code and execute it on the same machine. If you’d like to have a different machine for a "builder" and a "runner", you can use the two more fine-grained scripts:
# To be invoked on the machine building Omicron
$ ./tools/install_builder_prerequisites.sh
# To be invoked on the machine running Omicron
$ ./tools/install_runner_prerequisites.shThe sled agent expects to manage a real Gimlet. However, until those are built,
developers generally make do with something else, usually a commodity machine.
To make your machine "look" like a Gimlet, the
./tools/create_virtual_hardware.sh script can be used. This creates a few
file-based ZFS vdevs and ZFS zpools on top of those, and a couple of VNICs. The
vdevs model the actual U.2s that will be in a Gimlet, and the VNICs model the
two Chelsio NIC ports.
You can clean up these resources with ./tools/destroy_virtual_hardware.sh.
This script requires Omicron be uninstalled, e.g., with pfexec
./target/release/omicron-package uninstall, and a warning will be printed if
that is not the case. The script will then remove the file-based vdevs and the
VNICs created by create_virtual_hardware.sh.
Nexus’s external interface will typically be served using public-facing x.509
certificate. While we are still configuring the mechanism to integrate this real
certificate into the package system, ./tools/create_self_signed_cert.sh can be
used to generate an equivalent self-signed certificate.
The control plane repository contains a packaging tool which bundles binaries and SMF manifests. After building the expected binaries, they can be packaged in a format which lets them be transferred to a Helios machine.
This tool acts on a package-manifest.toml file which describes the packages to be
bundled in the build.
Configuration files are used to select IP addresses, and to manage Zpools
utilized by the Sled Agent. These configuration files are located within
smf/, and likely need to be modified to use addresses and zpools which match
your hardware. Much of this configuration will be automated in the future
(e.g., IP addresses will be inferred and posted to a DNS system, Zpools will
automatically be detected on discovered disks), but for now it remains
hard-coded.
Omicron currently implements a bit of a hack to allow external connectivity into
guest instances. This overrides the default behavior of the Oxide Packet
Transformation Engine (OPTE, the kernel module which provides the private
virtual networking to guests). To make this work, OPTE needs to know about the
local networking configuration, in this case the MAC address of the local
internet gateway. To be able to get into your instances, you must specify this
in the gateway.mac field of the config file smf/sled-agent/config-rss.toml.
The value there is correct for the lab environment, so if you’re running there, no changes are needed. If you’re running elsewhere, you find the value with:
$ arp -an | grep $(netstat -rn | awk '{ if ($1 == "default") print $2 }')
igb0 192.168.1.1 255.255.255.255 74:ac:b9:a4:dc:02That MAC address in the far right column should be provided in place of the
current value. Note that it can be specified as a string or an array of hex
values. If you have multiple gateways, e.g., different gateways for different
data links, you must choose the value for the physical link over which
create_virtual_hardware.sh will create the underlay VNICs. That is, this must
be a value in the same L2 segment as OPTE.
$ cargo run --release --bin omicron-package -- package|
Note
|
Running in release mode isn’t strictly required, but improves
the performance of the packaging tools significantly.
|
The aforementioned package command fills a target directory of choice
(by default, out/ within the omicron repository) with tarballs ready
to be unpacked as services.
To install the services on a target machine, the following command may be executed:
$ cargo build --release --bin omicron-package
$ pfexec ./target/release/omicron-package install|
Note
|
Do not use If you’ve done this already, and you wish to recover, run from the root of this repository |
This service installs a bootstrap service, which itself loads other requested services. The bootstrap service is currently the only service which is "persistent" across reboots - although it will initialize other service as part of its setup sequence anyway.
# List all services:
$ svcs
# View zones managed by Omicron (prefixed with "oxz_"):
$ zoneadm list -cv
# View logs for a service:
$ pfexec tail -f $(pfexec svcs -z oxz_nexus -L nexus)To uninstall all Omicron services from a machine, the following may be executed:
$ cargo build --release --bin omicron-package
$ pfexec ./target/release/omicron-package uninstallWhen we deploy, we’re effectively creating a number of different zones
for all the components that make up Omicron (Nexus, Clickhouse, Crucible, etc).
Since all these services run in different zones they cannot communicate with
each other (and Sled Agent in the global zone) via localhost. In practice,
we’ll assign addresses as per RFD 63 as well as incorporating DNS based
service discovery.
For the purposes of local development today, we specify some hardcoded IPv6
unique local addresses in the subnet of the first Sled Agent: fd00:1122:3344:1::/64.
If you’d like to modify these values to suit your local network, you can modify
them within the smf/ subdirectory.
Notably, Nexus is being served from IPv4 address, which may be configured to be
external. By default, it uses a private IPv4 address and no Internet gateway, but may
be configured to use a public-facing IP address with an Internet gateway that may
be set as a default route for the Nexus zone.
| Service | Endpoint |
|---|---|
Sled Agent: Bootstrap |
Derived from MAC address of physical data link. |
Sled Agent: Dropshot API |
|
Switch Zone |
|
Cockroach DB |
|
Nexus: Internal API |
|
Oximeter |
|
Clickhouse |
|
Crucible Downstairs 1 |
|
Crucible Downstairs 2 |
|
Crucible Downstairs 3 |
|
Internal DNS Service |
|
Nexus: External API |
|
Internet Gateway |
None, but can be set in |
Note that Sled Agent runs in the global zone and is the one responsible for bringing up all the other other services and allocating them with vNICs and IPv6 addresses.
Here are the current steps to provision an instance using the oxide
command line interface. Note that the jq command is required. In addition, the examples build on each other, so a previous name (or org, or project) are used in later steps.
-
Create an organization and project that the resources will live under:
oxide org create myorg oxide project create -o myorg myproj
-
Create an IP Pool, for providing external connectivity to the instance later. We need to create an IP Pool itself, and a range of IP addresses in that pool. Important: The addresses used here are appropriate for the Oxide lab environment, but not for an arbitrary environment. The actual IP range must currently be something that matches the physical network that the host is running in, at least if you’d like to be able to SSH into the guest. This is most often a private address range, like
10.0.0.0/8or192.168.0.0/16, but the exact addresses that are available depends on the environment.oxide api /system/ip-pools/default/ranges/add --method POST --input - <<EOF { "first": "172.20.15.227", "last": "172.20.15.239" } EOF -
Define a global image that will be used as initial disk contents.
-
This can be the alpine.iso image that ships with propolis:
oxide api /system/images --method POST --input - <<EOF { "name": "alpine", "description": "boot from propolis zone blob!", "block_size": 512, "distribution": { "name": "alpine", "version": "propolis-blob" }, "source": { "type": "you_can_boot_anything_as_long_as_its_alpine" } } EOF -
Or an ISO / raw disk image / etc hosted at a URL:
oxide api /system/images --method POST --input - <<EOF { "name": "crucible-tester-sparse", "description": "boot from a url!", "block_size": 512, "distribution": { "name": "debian", "version": "9" }, "source": { "type": "url", "url": "http://[fd00:1122:3344:101::15]/crucible-tester-sparse.img" } } EOF
-
-
Create a disk from that global image (note that disk size must be greater than or equal to image size and a 1GiB multiple!). The example below creates a disk using the image made from the alpine ISO that ships with propolis, and sets the size to the next 1GiB multiple of the original alpine source:
oxide api /organizations/myorg/projects/myproj/disks/ --method POST --input - <<EOF { "name": "alpine", "description": "alpine.iso blob", "block_size": 512, "size": 1073741824, "disk_source": { "type": "global_image", "image_id": "$(oxide api /system/images/alpine | jq -r .id)" } } EOF -
Create an instance, attaching the alpine disk created above:
oxide api /organizations/myorg/projects/myproj/instances --method POST --input - <<EOF { "name": "myinst", "description": "my inst", "hostname": "myinst", "memory": 1073741824, "ncpus": 2, "disks": [ { "type": "attach", "name": "alpine" } ], "external_ips": [{"type": "ephemeral"}] } EOF -
Optionally, attach to the proxied propolis server serial console (this requires this commit or newer of the CLI.)
oxide instance serial --interactive -p myproj -o myorg myinst