Proof of concept for OTel tracing

[punya]

This only covers non-queued execute requests. If the direction makes sense, we could

• add support for other request types,
• improve code organization,
• add tests,
• add documentation.

Fixes #727.

[otoolep]

The Trace you showed me looks about right. Clustered systems running on local host do about 10-100 INSERTs per second (because the disks saturate for all the nodes). rqlite won't win any prizes for speed.

[otoolep]

So why do this? I am wondering if I had use context objects properly and throughout rqlite (I started coding rqlite long before contexts were introduced) would this be necessary?

[punya]

Yes it would still be necessary.

Context objects propagate trace context within a process; this mechanism is analogous to HTTP headers such as the W3C trace-parent header, and is used to propagate context across process boundaries.

[otoolep]

Just so we're clear, what did you set this to at launch time?

[punya]

$ cat << EOF > config.yml
receivers:
  otlp:
    protocols:
      grpc:
exporters:
  googlecloud:
    project: some-project-name
service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [googlecloud]
EOF

$ otelcol-contrib \
  --config ./config.yml &

$ OTEL_SERVICE_NAME=rqlited.1 go run ./cmd/rqlited \
  --otlp-dest localhost:4317 \
  ~/node1 &

$ OTEL_SERVICE_NAME=rqlited.2 go run ./cmd/rqlited \
  --http-addr localhost:14001 \
  --raft-addr localhost:14002 \
  --otlp-dest localhost:4317 \
  --join localhost:4002 \
  ~/node2 &

$ OTEL_SERVICE_NAME=rqlite go run ./cmd/rqlite \
  --otlp-dest localhost:4317

[punya]

I should mention that this command-line argument is short and explicit, but doesn't expose the full feature set of the OTLP exporter. Before this PR is merged, I'd want to switch over to the autoexport package which is configured using these environment variables.