Zerolog is a distributed, decentralized log transport originally developed as a replacement for Scribe.
Below, you will find the “quick start” directions for setting up and using Zerolog. For more information, browse the following files:
- README: this file
- apps/zerolog/src/ the source tree for Zerolog
- rel/zerolog/etc/app.config: Zerolog configuration file
For obvious reasons Zerolog uses a client-server architecture where clients sends the logs as Protocol Buffers over ZeroMQ or as HTTP POST to a REST service to one or many Zerolog servers. Zerolog can be setup in a cluster to avoid single point of failures. Also the ZeroMQ transport layer will keep track of send messages and even buffer to memory and then to disk if the Zerolog server can’t be reached.
Zerolog uses a clustered caching database (Riak) to buffer incoming log messages and
when reaching a set threshold it will push the whole set to disco as one file.
After struggling with Scribe to get it to compile and work with Hadoop for log transportation we felt that enough is enough. Sure these are complex issues but done right they should be easy enough to install and use. After looking into Chukwa, Flume and others we felt that we could accomplish basically the same but with 1/10th the code.
Around the same time we finally gave up on Hadoop and moved to Disco and ironically none of the above solutions work with Disco so rolling our own made all the more sense.
This section assumes that you have copy of the Zerolog source tree. If not just
$ git clone git://github.com/kivra/zerolog.git
To get started, you need to:
- Build Zerolog
- Configure Zerolog for your log analysis backend
- Start the Zerolog server
- Connect a client and store log data
Assuming you have a working Erlang installation, building Zerolog should be as simple as:
$ cd $ZEROLOG $ make rel
Everything needed to configure Zerolog is in the rel/zerolog/etc/app.config file.
[
{zerolog, [
{enabled_backends, [zerolog_tty, zerolog_disco]},
{enabled_receivers, [zerolog_rest, zerolog_zeromq]},
{zerolog_tty, []},
{zerolog_disco, [
% 64MB (64x1024x1024)
{threshold, 67108864},
% Which nodes in the cluster
%{nodes, ['zerolog@127.0.0.1', 'zerolog@an.ot.her.ip']},
{master, "http://localhost:8989"},
{prefix, "0b"},
{tag, "data:zerolog"}
]},
{zerolog_zeromq, [{addr, "tcp://*:2121"}]},
{zerolog_rest, [
{ip, "127.0.0.1"},
{port, 2122}
]}
]},
%% Nothing needs to be edited below.
%% ===================================================================
[...]
].Zerolog is configured to use one or many backends, they are enabled when put
under enabled_backends and configured in each of their entries. Zerolog can
also use different receivers for incoming log messages, they are enabled when
put under enabled_receivers.
Once you have successfully built Zerolog, you can start the server with the following commands:
$ cd $ZEROLOG/rel/zerolog $ bin/zerolog start
Now that you have a functional server, let’s try storing some logs in it.
$ cd $ZEROLOG $ erl -pa deps/*/ebin -pa apps/zerolog/ebin Erlang R15B (erts-5.9) [source] [64-bit] [smp:2:2] [async-threads:0] [hipe] [kernel-poll:false] Eshell V5.9 (abort with ^G) 1> zerolog_client:run(1).
zerolog_client is a small client test library with one function run/1 which takes one argument
number of time to send, so zerolog_client:run(10) would send 10 messages.
$ curl -X POST -d "testdata" http://127.0.0.1:2122/zerolog ok
Zerolog is designed and meant to be run in a cluster. When run in a cluster you can configure your client to send to any of the configured Zerolog servers. They will (when using the Disco backend) keep a clustered database in sync where all the log messages get stored until the specified threshold has been reached after which it pushes everything in to Disco and clears the pushed log messages from the database.
Setting up a Zerolog cluster is meant to be dirt simple and robust. When running in a cluster make sure you
have correct ip adresses and erlang cookies set to be able to for the cluster to talk to each other. Editing
the file rel/zerolog/etc/vm.args and change the parameters -name zerolog@127.0.0.1 and
-setcookie zerolog to match the ip of your machineand secret cookie used to secure your erlang nodes.
With default settings in your app.config you can start up the first zerolog server in the cluster. Then you can go on to configure and start up the remaining machines in the Zerolog cluster. When starting up the first machine you should see something like:
=INFO REPORT==== 5-Jan-2012::14:02:51 === Starting Zerolog =INFO REPORT==== 5-Jan-2012::14:02:51 === 'zerolog@10.0.0.1' elected
On the next machine open app.config and edit the section below to match your cluster. Also, don’t forget
to set the correct values in vm.args as well.
[
{zerolog_disco, [
% Which nodes in the cluster
{nodes, ['zerolog@10.0.0.1', 'zerolog@10.0.0.2']},
[...]
].The nodes specified under {nodes, []} are the other nodes in the cluster. When you startup the second
node you should something like:
=INFO REPORT==== 5-Jan-2012::14:02:51 === Starting Zerolog =INFO REPORT==== 5-Jan-2012::14:02:51 === 'zerolog@10.0.0.2' surrendered to 'zerolog@10.0.0.1'. =INFO REPORT==== 5-Jan-2012::13:47:26 === Sent join request to Riak node: 'zerolog@10.0.0.1'
you can now execute riak_core_status:ringready(). which should print all the nodes in the database cluster.
As of this writing these are the ready made options:
- Alogger (with the Zerolog backend configured): https://github.com/bipthelin/alogger
- REST interface at http://zerolog.ip/zerolog accepts POST and the message