Thanks to UnderSiege for getting this project off the ground.
- Maven
- Java 7
Check which metrics library cassandra is using by looking in cassandra/lib/metrics-core*, verify that pom.xml points to the same exact version. For example, if you have metrics-core-2.2.0.jar, make sure pom.xml has 2.2.0.
mvn package
Alternatively, grab the binary from bintray:
curl -L http://dl.bintray.com/lookout/systems/com/github/lookout/metrics/agent/1.2/agent-1.2.jar -o agent-1.2.jar
Copy the statsd library from the .m2 folder to cassandra/lib. Add the following to your cassandra startup script:
Copy the agent-1.2.jar to a new directory cassandra/plugins
Change cassandra startup to add this agent. This can be done in a stock install by adding the following to /etc/default/cassandra:
export JVM_OPTS="-javaagent:/usr/share/cassandra/plugins/agent-1.2.jar=localhost"
Note the '=localhost' at the end. This supports the following syntaxes:
hostname:port@interval
For example:
your.statsd.host.com:9999@60
The default port is 8125 and the default interval is 10 (seconds); these
can be omitted. IPV6 is also supported with the following syntax:
[2001:db8::1]:8888@300
which means connect to 2001:db8::1 on port 8888 every 300 seconds.
A log message will be added to the system.log at startup to confirm that everything is running, it looks like this:
INFO [metrics-statsd-thread-1] 2014-12-19 19:05:37,120 StatsdReporter.java:65 - Statsd reporting to host localhost port 8125 every 10 seconds
Lots of stuff:
- Gossip statistics: gossip.score., which help decide who is closer/faster for queries gossip.severity, which indicates how busy this node is self-reporting to others
- Per table statistics: cfstats...ReadCount cfstats...WriteCount cfstats...RecentReadLatencyMicros cfstats...RecentWriteLatencyMicros cfstats...TombstonesPerSlice cfstats...estimatedKeys The last one is great for monitoring general trends, but of course don't rely on that number to be very accurate.
- PHI reporter Also supported is the currently-experimental PHI reporter, in PHI., coming to a Cassandra cluster near you soon.
- JVM GC metrics
- Anything else registered with yammer-metrics
Not working? There's a lot of tracing and debugging available. Change the log4j-server.properties and add something like this to get extremely detailed traces of what it's doing in the server.log.
log4j.logger.com.github.lookout.metrics.agent.generators=TRACE
Errors that happen during startup are not reported as well as they should be, mostly because the logging system is not active during startup. The log message is only generated when the actual metrics collector has run for the first time.