add AVRO/Flume logger driver

[sebglon]

- What I did
Adding ne logger plugin for Flume Avro protocol
- How I did it
#23978

Send log container message on Apache flume with Avro protocol
add new library sebglon/goavro and snappy library
- How to verify it
Run docker with new flume-avro logger
dockerd --log-driver=flume-avro --log-opt avro-host=<IP of flume> --log-opt avro-port=<flume port>
Run container
docker run --rm hello-world
And show log message on your flume

Example AvroFlumeEvent:

{ headers:{image_name=hello-world, container_name=/boring_heyrovsky, container_created=1472212849, daemon_name=docker, host=439c0ae46a3d, source=stdout, image_id=sha256:c54a2cc56cbb2f04003c1cd4507e118af7c0d340fe7e2720f70976c4b75237dc, partial=false, command=/hello, container_id=40682c62d234bf3b2218e5536973253afdbc68f06bcc5428e819a36e3bf8d015, timestamp=2016-08-26 12:00:49.935212928 +0000 UTC} body: 54 68 69 73 20 6D 65 73 73 61 67 65 20 73 68 6F This message sho }

- Description for the changelog

New logger driver for Apache Flume

[sebglon]

@vdemeester, @cpuguy83 @LK4D4
Error to rebase branch on PR #26042

Issue for external logging plugin #18604

[sebglon]

@thaJeztah can you help pe to validate
ping @bfirsh @sirupsen

[sebglon]

End of work;

[thaJeztah]

ping @cpuguy83 @tiborvass WDYT?

[sebglon]

What is the next step. How can i plan to use them on production?

[thaJeztah]

Thanks for being patient @sebglon. I brought this PR up in our maintainers meeting / review session. We are still looking for ways to make logging drivers "pluggable" so that we don't have to include all the dependencies in the main docker binary (which we will have to maintain).

Since I was not familiar with "Flume", and not sure how popular it is as a choice, I asked around, and was told that Flume also has an option to integrate with syslog; would that be an option?

In addition, I've understood that Flume is popular for Hadoop users, but Kafka is getting more and more the choice for those users.

/cc @nathanleclaire @justincormack @stevvooe wdyt?

[nathanleclaire]

Thanks for the PR @sebglon . As Sebastiaan noted, why not use something like the approach here: http://blog.thilinamb.com/2014/06/how-to-pump-syslog-data-into-flume.html with Docker's syslog log driver to send the logs to Flume? The daemon already has quite a bit of complexity/dependencies, so if we can make your use case work without having to modify the code directly, that'd be ideal.

[sebglon]

Kafka can use avro message to send or consule log
syslog has message length limit that is not compatible with Java, python or Javascript stack (rsyslog truncated at 2097 characters)

I think i can make an evolution to send on kafka data. But now this is not supported by docker

[sebglon]

Avro is a message protocol that can be use to exchange formated data with consumer.
Now i have implemented transceiver for Flume/Netty Transceiver.

In a few time i can test to send on kafka without another dependencies

[sebglon]

Fluentd, Flume, Kafka can consume Avro message
Apache Avro™ is a data serialization system.

My code has not dependency for flume or another consumer but only implement Avro message and protocol.

This can be a good solution to limit Custom logger and dependencies to use Avro

[thaJeztah]

Would be good to have this merged upstream if we decide to accept this PR

[stevvooe]

Also, what exactly is in this fork?

[thaJeztah]

see linkedin/goavro#62

[stevvooe]

While I don't see anything wrong with adding a new log driver, I'd like to better understand the make up of the avro-flume ecosystem. Is this approach enough or are we going to add new protocols? Is the structure of this driver flexible enough to add these changes in the future or does this hit a single company's use case?

The other issue I see is that the goavro implementation is a significant fork from upstream (linkedin/goavro, as far as I can tell). The fork doesn't seemed to be pass golint or follow the Go styleguide. Why that is not a show stopper, is does make me concerned about adopting this as a dependency.

Also, into the future, who would maintain this integration? How can we ensure this tracks the upstream implementations to be sufficiently useful and relevant?

[sebglon]

Avro protocol is a big protocol that can be used to send formated and serialized data in many consumer.
This implementation is a need of my company but my personal objectif is to extend this driver to many consumer that can accept Avro message.

I think this is a good choice to externalise logging without custom product library.

Avro start exchange by sending an handshake to validate exchange format. But for me docker can define this format and all consumer can deserialisze this and transformt then to their need.

Actualy my client work with Flume but many client company use another Log consumer that accept avro.

[thaJeztah]

We discussed this in the maintainers meeting, and after further discussion, we don't think we're comfortable with adding this driver. There's a first PoC of having external plugins for logging drivers #28403. which would be a great option to implement this, once ready

Thanks for contributing!

[cpuguy83]

Log plugin support is now merged and should be part of 17.05.
I do not think we'll be accepting any new built-in log drivers.

Closing this. Any new log drivers should be implemented as a plugin at this point. If anyone wants to discuss this feel free to ping me on Slack, or come see me @ DockerCon!