⭐ - The star motivates me a lot!

Transactional box is an implementation of the outbox and inbox pattern in .NET.
Ensures reliable network communication (eventual consistency) between services.

All complexity is taken over by the transactional box and simplifies communication between services to the maximum extent possible.
It is designed for a low entry threshold and quick learning.

Examples of problems that occur during network communication:

• Lost message

Amount was taken from bank account and transfer was never executed.

• The same message was processed again

Transfer was ordered and amount was debited from bank account twice.

• Unavailable service

Transfer order attempt fails.

For more information, see the documentation📖.
Packages are hosted by nuget.org.

🎬 Run Sample

Note

Docker is required.

Clone this repo and open TransactionalBox.sln via Visual Studio 2022. Set the TransactionalBox.Sample.WebApi as startup and then run. You should see the following view.

Project is at an early stage of life cycle, if you find some bug, let me know ☎️.

Have fun 😃!

✨ Features

Actions

• Add a message to send to the outbox
• Add a message to publish to the outbox
• Get messages from outbox and add them to transport
• Get a message from transport and add them to the inbox
• Get a message from inbox and process it
• Get messages from inbox and process them

Storage

• InMemory (Default)
• Entity Framework (Relational)
  • Migrations
  • Distributed Lock (Based on atomic write operation, Standalone Package)
• MongoDB

Transport

• InMemory (Default)
• Apache Kafka
• RabbitMQ
• Iggy

Scalability & Fault Tolerance

• Support for multiple outbox worker instances
  • Multiple instances of the same service
  • Multiple processes in the same service
• Support for multiple inbox instances
  • Multiple instances of the same service
  • Multiple processes in the same service
• Standalone inbox
• Error handling in background services
• Dead messages
• Hook processing
  • Outbox
  • Inbox

Observability

• Support for OpenTelemetry
• Outbox size
• Inbox size
• Message failure rate
• Message delivery latency
• Number of duplicated messages
• Message duplication rate

Maintenecne

• Remove processed messages from the outbox
• Remove processed messages from the inbox
• Remove expired idempotency keys
• Archive processed messages from the outbox
• Archive processed messages from the inbox
• Correlation ID

Other

• Modular package architecture
• Support for TimeProvider
• Unordered messages
• Internal high-performance logging (Source Generators)
• Execution context in Inbox
• High-performance invoking of inbox message handlers from assemblies (Compiled Lambda Expressions)
• Grouping of messages
  • Group by topic outbox messages to a single transport message from batch (better compression)
  • Adjusting optimal transport message size
• Messages serialization and deserialization
  • System.Text.Json (default)
• Messages compression and decompression
  • No compression (default)
  • Brotli
  • GZip
• Message streaming
• Package configuration using appsetings.json
• Own transport message serialier and deseralizer (with StringBuilder)
• Idempotent messages
• Keyed in memory lock (based on SemaphoreSlim and ConcurrentDictionary)
• Transport discriminator (one outbox many transport, tagged message)
• Inbox based on the header, selects the appropriate algorithm to:
  • Decompression
  • Deserialization
  • Decryption

🏰 Architecture

The transactional box consists of four basic components. The following diagrams show the basic flow (omits details).

Outbox

Outbox is responsible for adding messages to the storage.

Outbox Worker

Outbox worker is responsible for getting the messages from storage and adding them to the transport.

Inbox Worker

Inbox worker is responsible for getting messages from transport and adding them to the storage.

Inbox

Inbox is responsible for processing messages from the storage.

🗺️ Roadmap

Name of TransactionalBox fits perfectly with the future of the project. It will be possible to use different transactional boxes.

Outbox and Inbox

No guarantee of message order between services.

Implementation under the competition.

e.g. Payment service asynchronously sends notification of payment to user.

Improvements:

• Code refactor
• More tests
• Support for more storage providers (e.g. Marten, RavenDB)
• Support for more transport providers (e.g. HTTP, gRPC, Azure Service Bus)
• Performance optimization
• Encrypted transport messages

StreamOubox and StreamInbox

Guarantee of message order in stream between services.

Idea
Ensuring the order of messages within a stream. Messages are sent by StreamOutbox and the order is respected in StreamInbox. Transport provider does not have to support message order.

e.g. (CQRS Pattern) When transfer is made in the write service, the event asynchronously refreshes the account balance in the read service.

InternalBox

Guarantee of message order in stream inside the service.

Idea
Ensuring the order of messages within a stream insite the service. Messages are added to storage provider and then processed. Transport provider is unnecessary.

e.g. Asynchronous internal communication between Aggregate Roots using domain events in the same service.

🏅 Competition '100commitow'

The project is part of the competition 100 commitow.

Topics

• Distributed lock (prevent race condition)
• Hook processing
• Concurrency control
• Scaling and parallel processing (distributed processing)
• Synchronization primitives
• Idempotency
• Retry Pattern