springboot-protobuf-kafkakms

Sample application

First a normal application

For purposes of demonstration, the application is developed/designed without the personal data concepts at first.

There is a tag springboot-protobuf-kafka-kms-before for this state.

It uses the apache kafka and google protobuf for its long term data durability.

The project contains two sub-components within the same codebase and toggled by simple spring profiles, for simplicity reasons.

Components

• Onboarding: Rest api and publishes FarmerRegistered Event (profiles: onboarding, onboarding-auto-publish)
• Newsletter: Listens to farmer registered events and sends weekly newsletter emails (profiles: newsletter)

Adding pi2schema

Gradle Dependency

Currently, by adding the dependency pi2schema:serialization-kafka-protobuf:0.1.0 will bring all the required dependencies, which are schema metadata, key manamement system and serializer In the current sample, as it is a subproject it is just:

	compile(project(":serialization-kafka-protobuf"))

Schema definition - PII metadata annotation.

⚠️ Draft api: Most likely the following definitions will change until version 1.0

👂 Feedback is welcome. Please give your feedback here.

The schema must define the Subject Identifier as well as which part of the payload contains personal data.

To identify the Subject Identifier we add the [(pi2schema.subject_identifier) = true] annotation to the uuid field:

string uuid = 1[(pi2schema.subject_identifier) = true];

For defining a personal data field we wrap it with an oneOf tag and simply add a new element pi2schema.EncryptedPersonalData encryptedPersonalData = 6; as part of the oneOf. The library will identify this pattern and do the replacement on the serialization / deserialization.

 oneof personalData {
   ContactInfo contactInfo = 2;
   pi2schema.EncryptedPersonalData encryptedPersonalData = 6;
 }

Key management

⚠️ Not safe for production. Please for the time being consider integrating a 3rd party kms as aws kms, gcp kms or Hashicorp Vault

The key management that is currently used out of the box is a simple JCE based AES-256 local encryptor and decryptor . The secret key is stored in a kafka topic for durability.

Running the example

Local environment

Start the local environment with:

docker compose -f examples/docker-compose.yaml up

Running the services

Producer:

./gradlew examples:springboot-protobuf-kafkakms:bootRun --args='--spring.profiles.active=registration'

Listener:

./gradlew examples:springboot-protobuf-kafkakms:bootRun --args='--spring.profiles.active=newsletter --server.port=8180'

Simulating a registration of a farmer with his personal data

http -v  :8080/api/v1/farmers name=Test4 email=test4@email.com phone=15780807079