Spring, Kafka Streams, gRPC with TestContainers - demo application

Account Service - Spring, Kafka Streams sample application that allows users to Sign up and retrieve account details and statistics.

This project demonstrates how to run multiple Kafka Streams instances (with different consumer groups) inside single Spring Boot application.

Table of Content

• Patterns, technologies and concepts demonstrated here
• Architecture
• Getting Started
  • Prerequisites
  • Installation
    • Local
  • Usage
  • Testing Kafka Stream
• Deployment
  • Docker
• References
• License

Patterns, technologies and concepts demonstrated here

• Testcontainers for Kafka streams integration tests.
• Multiple KStreams inside single boot application.
• Kafka protobuf integration.
• Dynamic property binding in Spring Framework.
• gRPC for handling commands (CreateAccount).
• gRPC client via BloomRPC application.
• Interactive queries (kafka state store integration).
• Query endpoints for account details and statistics.
• REST endpoints for printing kstream topology details.
• Separating unit and integration tests with gradle.
• Docker, docker compose integration.
• Custom gradle task for printing docker image name and sample docker-compose file.
• Faker with scheduling setup for generating dummy data, activated by dummy_data_gen spring profile.
• Custom lombok configuration.

Architecture

Getting Started

Prerequisites

• Java 11
• Docker
• Kafka
• Schema Registry

Installation

Local

• Start Kafka broker and Schema Registry using Confluent Community or Confluent Platform components.

  • Confluent community:

      docker-compose -f ./docker/local/docker-compose.yml up -d

  • Confluent Platform:

    docker-compose -f ./docker/confluent-platform/docker-compose.yml up -d

• Check if all containers started, using docker-compose ps command.

  # Sample output for Confluent Platform
      Name                  Command            State                       Ports                     
  ----------------------------------------------------------------------------------------------------
  broker            /etc/confluent/docker/run   Up      0.0.0.0:9092->9092/tcp, 0.0.0.0:9101->9101/tcp
  control-center    /etc/confluent/docker/run   Up      0.0.0.0:9021->9021/tcp                        
  rest-proxy        /etc/confluent/docker/run   Up      0.0.0.0:8082->8082/tcp                        
  schema-registry   /etc/confluent/docker/run   Up      0.0.0.0:8081->8081/tcp                        
  zookeeper         /etc/confluent/docker/run   Up      0.0.0.0:2181->2181/tcp, 2888/tcp, 3888/tcp    

• Start Account Service

  ./gradlew bootRun

Usage

• Check Create Account flow

  • Import proto files into BloomRPC client.
  • Choose 6565 as gRPC server port.
  • Create new account using AccountService.Create rpc call.

    # Sample request body
    {
      "email": "newAccount@test.com",
      "countryCode": "GB",
      "type": 2
    }

  • Get account id from response

    # Sample response body
    {
      "id": "519faeb7-ccdf-49dd-95a4-3e2a6ac8ad13",
      "email": "newAccount@test.com",
      "countryCode": "GB",
      "type": "PREMIUM",
      "createdDate": {
      "seconds": "1616860904",
      "nanos": 644947000
      }
    }

  • Test interactive queries using REST endpoints.
    • Get account details from accounts.store state store (use account id retrieved before):

        curl -X GET --location "http://localhost:8080/api/v1/accounts/519faeb7-ccdf-49dd-95a4-3e2a6ac8ad13"

    • Get account statistics for 'GB' countryCode:

      curl -X GET --location "http://localhost:8080/api/v1/statistics/countries/GB/accounts"

• Analise kafka streams topology:

  • Get all available topologies.

    curl -X GET --location "http://localhost:8080/topologies"

  • Print topology for accounts-per-country kstream.

    curl -X GET --location "http://localhost:8080/topologies/accounts-per-country"

  • Print topology diagram, visit https://zz85.github.io/kafka-streams-viz/ page.

Testing Kafka Streams

• Unit test (TopologyTestDriver) examples may be found under src/test/.
• Integration test (Testcontainers) examples may be found under src/integrationTest/.

Deployment

Docker

Docker compose file with instructions are kept in docker subdirectory.

References

• Kafka Streams Topology Visualizer
• Spring for Apache Kafka
• Quick Start for Apache Kafka using Confluent Platform (Docker)
• DynamicPropertySource in Spring Framework
• Testcontainers: singleton-containers
• Testcontainers: Kafka Modules Reference Guide
• Testing a streams application
• BloomRPC
• Java Faker
• Gradle Test Logger Plugin
• Protobuf Plugin for Gradle

License

Distributed under the MIT License. See LICENSE for more information.