cppkafka: high level C++ wrapper for rdkafka

cppkafka allows C++ applications to consume and produce messages using the Apache Kafka protocol. The library is built on top of librdkafka, and provides a high level API that uses modern C++ features to make it easier to write code while keeping the wrapper's performance overhead to a minimum.

Features

• cppkafka is a high level C++ wrapper for rdkafka, aiming to allow using rdkafka in a simple, less error prone way.

• cppkafka provides an API to produce messages as well as consuming messages, but the latter is only supported via the high level consumer API. cppkafka requires rdkafka >= 0.9.4 in order to use it. Other wrapped functionalities are also provided, like fetching metadata, offsets, etc.

• cppkafka provides message header support. This feature requires rdkafka >= 0.11.4.

• cppkafka tries to add minimal overhead over librdkafka. A very thin wrapper for librdkafka messages is used for consumption so there's virtually no overhead at all.

It's simple!

cppkafka's API is simple to use. For example, this code creates a producer that writes a message into some partition:

#include <cppkafka/cppkafka.h>

using namespace std;
using namespace cppkafka;

int main() {
    // Create the config
    Configuration config = {
        { "metadata.broker.list", "127.0.0.1:9092" }
    };

    // Create the producer
    Producer producer(config);

    // Produce a message!
    string message = "hey there!";
    producer.produce(MessageBuilder("my_topic").partition(0).payload(message));
    producer.flush();
}

Compiling

In order to compile cppkafka you need:

• librdkafka >= 0.9.4
• CMake
• A compiler with good C++11 support (e.g. gcc >= 4.8). This was tested successfully on g++ 4.8.3.
• The boost library.

Now, in order to build, just run:

mkdir build
cd build
cmake <OPTIONS> ..
make

CMake options

The following cmake options can be specified:

• RDKAFKA_ROOT_DIR : Specify a different librdkafka install directory.
• BOOST_ROOT : Specify a different Boost install directory.
• CPPKAFKA_CMAKE_VERBOSE : Generate verbose output. Default is OFF.
• CPPKAFKA_BUILD_SHARED : Build cppkafka as a shared library. Default is ON.
• CPPKAFKA_DISABLE_TESTS : Disable build of cppkafka tests. Default is OFF.
• CPPKAFKA_DISABLE_EXAMPLES : Disable build of cppkafka examples. Default is OFF.
• CPPKAFKA_BOOST_STATIC_LIBS : Link with Boost static libraries. Default is ON.
• CPPKAFKA_BOOST_USE_MULTITHREADED : Use Boost multi-threaded libraries. Default is ON.
• CPPKAFKA_RDKAFKA_STATIC_LIB : Link to Rdkafka static library. Default is OFF.

Example:

cmake -DRDKAFKA_ROOT_DIR=/some/other/dir -DCPPKAFKA_BUILD_SHARED=OFF ...

The RDKAFKA_ROOT_DIR must contain the following structure. If the system architecture is 64-bit and both lib and lib64 folders are available, the lib64 folder location will be selected by cmake.

${RDKAFKA_ROOT_DIR}/
                   |
                   + include/librdkafka/rdkafka.h
                   |
                   + lib/librdkafka.a
                   |
                   + lib64/librdkafka.a (optional)

Using

If you want to use cppkafka, you'll need to link your application with:

• cppkafka
• rdkafka

Documentation

You can generate the documentation by running make docs inside the build directory. This requires Doxygen to be installed. The documentation will be written in html format at <build-dir>/docs/html/.

Make sure to check the wiki which includes some documentation about the project and some of its features.