Skip to content

Latest commit

 

History

History

kafka-connect

Folders and files

NameName
Last commit message
Last commit date

parent directory

..

plugin

plugin

 
 

README.md

README.md

 
 

docker-compose.yml

docker-compose.yml

 
 

twitter-source.json

twitter-source.json

 
 

README.md

Kafka Connect for Twitter

This contains a Compose file used to create a Kafka/ZK Confluent cluster with a single broker.

It also contains:

  • Kafka Connect
  • Kafka Connect UI

Setup

To run the cluster use:

docker-compose up

Then you can access:

  • Kafka broker - localhost:9092
  • Zookeper - localhost:2181
  • Kafka Connect - localhost:8083
  • Kafka Connect UI - http://localhost:8003

Kafka Connect

Kafka Connect uses a Twitter connector (https://github.com/jcustenborder/kafka-connect-twitter) to stream tweets and dump them into Kafka.

The JARs of the Twitter connector can be found in the plugin folder. These will be placed in the plugins directory so Kafka Connect can pick it up.

Once the Kafka stack is started and you have populated twitter-source.json with the relevant values, you will have to create the connector by running the following cURL command:

curl -X POST -H "Content-Type: application/json" -H "Accept: application/json" \
    -d @twitter-source.json \
    localhost:8003/api/kafka-connect-1/connectors