SpectatorD is a high-performance telemetry daemon that listens for metrics specified by a text-based protocol and publishes updates periodically to an Atlas aggregator service. It consolidates the logic required to apply common tagging to all metrics received, maintain metric lifetimes, and route metrics to the correct backend.
spectatord --help
spectatord: A daemon that listens for metrics and reports them to Atlas.
--age_gauge_limit (The maximum number of age gauges that may be reported by
this process.); default: 1000;
--common_tags (Common tags: nf.app=app,nf.cluster=cluster. Override the
default common tags. If empty, then spectatord will use the default set.
This flag should only be used by experts who understand the risks.);
default: "";
--debug (Debug spectatord. All values will be sent to a dev aggregator and
dropped.); default: false;
--enable_socket (Enable UNIX domain socket support. Default is true on Linux
and false on MacOS.); default: true;
--enable_statsd (Enable statsd support.); default: false;
--meter_ttl (Meter TTL: expire meters after this period of inactivity.);
default: 15m;
--port (Port number for the UDP socket.); default: 1234;
--socket_path (Path to the UNIX domain socket.);
default: "/run/spectatord/spectatord.unix";
--statsd_port (Port number for the statsd socket.); default: 8125;
--admin_port (Port number for the admin server.); default: 1234;
--uri (Optional override URI for the aggregator.); default: "";
--verbose (Use verbose logging.); default: false;
--verbose_http (Output debug info for HTTP requests.); default: false;
Try --helpfull to get a list of all flags or --help=substring shows help for
flags which include specified substring in either in the name, or description or
path.
By default, the daemon will listen on the following endpoints:
- Metrics Message Protocol
1234/udp
(~430K reqs/sec with 16MB buffers)/run/spectatord/spectatord.unix
Domain Socket (~1M reqs/sec with batching)
- Admin Server:
1234/tcp
The choice of which endpoint to use is determined by your performance and access requirements; the Unix domain socket offers higher performance, but requires filesystem access, which may not be tenable under some container configurations. See Performance Numbers for more details.
⚠️ In container environments, the-w0
option may not work and-w1
should be used instead.
echo "c:server.numRequests,id=failed:1" | nc -u -w0 localhost 1234
echo "t:server.requestLatency:0.042" | nc -u -w0 localhost 1234
echo "d:server.responseSizes:1024" | nc -w0 -uU /run/spectatord/spectatord.unix
echo "g:someGauge:60" | nc -w0 -uU /run/spectatord/spectatord.unix
echo "g,300:anotherGauge:60" | nc -w0 -uU /run/spectatord/spectatord.unix
echo "X,1543160297100:monotonic.Source:42" | nc -w0 -uU /run/spectatord/spectatord.unix
echo "X,1543160298100:monotonic.Source:43" | nc -w0 -uU /run/spectatord/spectatord.unix
echo "A:age.gauge:0" | nc -u -w0 localhost 1234
The message sent to the server has the following format, where the ,options
and ,tags
portions
are optional:
metric-type,options:name,tags:value
Multiple lines may be sent in the same packet, separated by newlines (\n
):
echo -e "t:server.requestLatency:0.042\nd:server.responseSizes:1024" | nc -u -w0 localhost 1234
Symbol | Metric Type | Description |
---|---|---|
c |
Counter | The value is the number of increments that have occurred since the last time it was recorded. |
d |
Distribution Summary | The value tracks the distribution of events. It is similar to a Timer, but more general, because the size does not have to be a period of time. For example, it can be used to measure the payload sizes of requests hitting a server or the number of records returned from a query. |
g |
Gauge | The value is a number that was sampled at a point in time. The default time-to-live (TTL) for gauges is 900 seconds (15 minutes) - they will continue reporting the last value set for this duration of time. Optionally, the TTL may be specified in seconds, with a minimum TTL of 5 seconds. For example, g,120:gauge:42.0 spcifies a gauge with a 120 second (2 minute) TTL. |
m |
Max Gauge | The value is a number that was sampled at a point in time, but it is reported as a maximum gauge value to the backend. |
t |
Timer | The value is the number of seconds that have elapsed for an event. |
A |
Age Gauge | The value is the time in seconds since the epoch at which an event has successfully occurred, or 0 to use the current time in epoch seconds. After an Age Gauge has been set, it will continue reporting the number of seconds since the last time recorded, for as long as the spectatord process runs. The purpose of this metric type is to enable users to more easily implement the Time Since Last Success alerting pattern. To set a specific time as the last success: A:time.sinceLastSuccess:1611081000 . To set now() as the last success: A:time.sinceLastSuccess:0 . By default, a maximum of 1000 Age Gauges are allowed per spectatord process, because there is no mechanism for cleaning them up. This value may be tuned with the --age_gauge_limit flag on the spectatord binary. |
C |
Monotonic Counter | The value is a monotonically increasing number. A minimum of two samples must be received in order for spectatord to calculate a delta value and report it to the backend. A variety of networking metrics may be reported monotically and this metric type provides a convenient means of recording these values, at the expense of a slower time-to-first metric. |
D |
Percentile Distribution Summary | The value tracks the distribution of events, with percentile estimates. It is similar to a Percentile Timer, but more general, because the size does not have to be a period of time. For example, it can be used to measure the payload sizes of requests hitting a server or the number of records returned from a query. In order to maintain the data distribution, they have a higher storage cost, with a worst-case of up to 300X that of a standard Distribution Summary. Be diligent about any additional dimensions added to Percentile Distribution Summaries and ensure that they have a small bounded cardinality. |
T |
Percentile Timer | The value is the number of seconds that have elapsed for an event, with percentile estimates. This metric type will track the data distribution by maintaining a set of Counters. The distribution can then be used on the server side to estimate percentiles, while still allowing for arbitrary slicing and dicing based on dimensions. In order to maintain the data distribution, they have a higher storage cost, with a worst-case of up to 300X that of a standard Timer. Be diligent about any additional dimensions added to Percentile Timers and ensure that they have a small bounded cardinality. |
X |
Monotonic Counter with Millisecond Timestamps | The value is a monotonically increasing number, sampled at a specified number of milliseconds since the epoch. A minimum of two samples must be received in order for spectatord to calculate a delta value and report it to the backend. This is an experimental metric type that can be used to track monotonic sources that were sampled in the recent past, with the value normalized over the reported time period. The timestamp in milliseconds since the epoch when the value was sampled must be included as a metric option: X,1543160297100:monotonic.Source:42 |
The metric name and tags must follow Atlas restrictions, which are described in the sections below.
Tags are optional. They may be specified as comma-separated key=value
pairs after the metric name.
For example:
fooIsTheName,some.tag=val1,some.otherTag=val2
See Atlas Naming Conventions for recommendations on naming metrics.
Limit | Min | Max |
---|---|---|
Length of name |
1 | 255 |
Tag key length | 2 | 60 |
Tag value length | 1 | 120 |
The metric name, tag keys and values may only use characters in the following set: -._A-Za-z0-9
.
All others characters will be converted to an underscore (_
) by the client.
To avoid issues with parsing metrics, avoid using the SpectatorD protocol delimiter characters
(,=:
) rather than relying on the client to rewrite them to _
.
A double value. The meaning of the value depends on the metric type.
See METRICS for a list of metrics published by this service.
An administrative server is provided with SpectatorD, so that debugging information and few
data management tasks may be completed. By default, this server listens on port 1234/TCP
,
but this can be modified with the --admin_port
flag. The endpoints which change data may
only be accessed from localhost.
GET /
- Returns a service description and list of available endpoints.
GET /config
- Returns the current SpectatorD configuration, including the current set of common tags.
GET /config/common_tags
- Returns a description of how to use this endpoint to modify common tags.
POST /config/common_tags
- Create, modify or delete common tags from the allowed set of Mantis common tags. No other common tags may be modified. Create or update a tag by setting it to a string. Delete a tag by setting the value to an empty string.
- Allowed tags:
mantisJobId
mantisJobName
mantisUser
mantisWorkerIndex
mantisWorkerNumber
mantisWorkerStageNumber
- Example:
curl -X POST \ -d '{"mantisJobId": "foo", "mantisJobName": "bar", "mantisUser": ""}' \ -w " %{http_code}\n" \ http://localhost:1234/config/common_tags
GET /metrics
- Return an object containing lists of all metrics currently known to the Registry, grouped by type.
DELETE /metrics/A
- Delete all AgeGauge metrics from the Registry.
DELETE /metrics/A/{id}
- Delete one AgeGauge metric from the Registry, identified by the
id
. - Example:
curl -X DELETE \ -w " %{http_code}\n" \ http://localhost:1234/metrics/A/fooIsTheName,some.tag=val1,some.otherTag=val2
- Delete one AgeGauge metric from the Registry, identified by the
DELETE /metrics/g
- Delete all Gauge metrics from the Registry.
DELETE /metrics/g/{id}
- Delete one Gauge metric from the Registry, identified by the
id
. - Example:
curl -X DELETE \ -w " %{http_code}\n" \ http://localhost:1234/metrics/g/fooIsTheName,some.tag=val1,some.otherTag=val2
- Delete one Gauge metric from the Registry, identified by the
A key goal of this project is to deliver high performance. This means that we need to use few resources for the common use case, where the number of metric updates is relatively small (< 10k reqs/sec), and it also needs to be able to handle hundreds of thousands of updates per second when required.
Using Unix domain sockets, we can handle close to 1M metric updates per second, assuming the client batches the updates and sends a few at a time. Sending every single metric update requires a lot of context switching, but is something that works well for the majority of our use cases. This simplicity means the user does not have to maintain any local state.
Transport Batch Size First 10M Second 10M
Unix Dgram 1 22.98s (435k rps) 20.58s (486k rps)
Unix Dgram 8 11.46s (873k rps) 9.89s (1011k rps)
Unix Dgram 32 10.38s (963k rps) 8.49s (1178k rps)
The UDP transport is particularly sensitive the max receive buffer size (16MB on our systems).
Our tests indicate that sending 430K rps to the UDP port did not drop packets, but if there is a
need for higher throughput, then tweaking /proc/sys/net/unix/max_dgram_qlen
is recommended.
- Start
spectatord
in debug mode (--debug
) to send metrics to a dev stack of the Atlas aggregator, which will perform validation and return the correct HTTP status codes for payloads, then drop the metrics on the floor. Alternatively, you can also configure it to send metrics to/dev/null
. - Use the
metrics_gen
binary to generate and send a stream of metrics to a running spectatord binary. - Use the
perf-record
andperf-report
Linux utilities to measure the performance of the running binary. - The
udp_numbers.pl
script is used to automate runningmetrics_gen
with different kernel settings for UDP sockets.
# setup python venv and activate, to gain access to conan cli
./setup-venv.sh
source venv/bin/activate
# link clion default build directory to our build directory
ln -s cmake-build cmake-build-debug
./build.sh # [clean|clean --force|skiptest]
- CLion > Preferences > Plugins > Marketplace > Conan > Install
- CLion > Preferences > Build, Execution, Deploy > Conan > Conan Executable: $PROJECT_HOME/venv/bin/conan