Clarification

[cloudcompute]

Does this library continually check the Outbox table, probably after every 10 ms or so ?

How can an application interact with this library?

I have an app that is writing domain events (internal.. not exposed to other microservices and Kafka) to PostgreSQL db. In order to use this library, I need to create an Outbox table and the app will save the integration events (external.. whose ultimate destination is Kafka) in the Outbox table in the same transaction i.e. write both domain and integration events in a single transaction.

Now this library will read the event(s) from the Outbox table as soon as they are written to it and push them to Kafka.

Is this how the flow like?

Getting used in production?

Thank you.

[magro]

Yes, your understanding is correct. The interval that the outbox table is checked is configured/specified via the parameter processingInterval.

The interaction of your application with the library is only writing to the outbox table via the OutboxService (apart from the setup of the OutboxProcessor).

And yes, it's used in production.

Does this answer your questions?

[cloudcompute]

Very much, thank you !

One more Question (for now).. Does it delete a record from Outbox after it is able to successfully write the record to Kafka?

[magro]

No. It marks it as "processed" by updating this column. I.e. in case you'd need to re-publish a message you could set this column to null.

[magro]

There's also not yet any automatic mechanism to regularly cleanup the outbox table - still a potential improvement or contribution 🙂

[cloudcompute]

There's also not yet any automatic mechanism to regularly cleanup the outbox table - still a potential improvement or contribution 🙂

I will try to implement

[cloudcompute]

Okay .. Thank you for the quick responses.

I am wondering how does this Usage fit in the scenario that I mentioned above? My app that is writing domain events is Node/TypeScript based and I will add the integration events to Outbox functionality to the same app. Is not the library itself the Kafka producer?

Also, does it impose any constraint on the schema of the Outbox table?

[magro]

You can also write to the outbox table from an app not written in Java.

The schema of the outbox table is documented here: https://github.com/tomorrow-one/transactional-outbox#prepare-database

[magro]

Maybe you want to read the README again 😉

[magro]

Another thought: If you're writing a truly event sourced application you're probably using some event sourcing framework (if I were you I'd do this). This framework should also have a concept of an event processor or an event steam that you could process - so that you could process the domain events to integration events to Kafka. So maybe the transactional-outbox is not that valuable to you.
Not sure, more knowledge about your context and requirements etc would be needed to reach a good decision, so I can't really give a good advice here (this would need more time/consulting)...

[cloudcompute]

Yes, I need to go through the code as well in order to understand it properly.

Initially, I assumed it is a standalone process. But.. probably, this library was written with an intention that the users can embed it in their Java applications (Kafka producers). If this is so, the Usage section is meant for them.

[magro]

Yes, that's right, it was written with this assumption. If you have an idea how the README could be improved to make this more clear I'd also be thankful for a PR 😁

[cloudcompute]

You're probably using some event sourcing framework

As of now, I am not using any. I came across several and the ones I have liked are:
https://github.com/hnaderi/edomata
https://github.com/fraktalio
https://github.com/johanhaleby/occurrent/

As a side note.. event sourcing is very hard and expensive to implement.

This framework should also have a concept of an event processor or an event steam that you could process - so that you could process the domain events to integration events to Kafka

Very true. After looking at this library of yours, I have reached to the same conclusion. In case, I decide to use NodeJS/TypeScript for writing events to the database, I need to find (or write myself) the code what your library is doing right now. I want to use NodeJS because it really shines in I/O.

If you have an idea how the README could be improved to make this more clear..

I think we can add a section named "Purpose" at the very beginning of Readme

• Explain the Outbox table pattern.

You may Copy/Paste the text and diagram from here:
https://debezium.io/blog/2019/02/19/reliable-microservices-data-exchange-with-the-outbox-pattern/

• How can we achieve this?

A single process should do all this
a. Process the domain events and extract the integration events
b. Write the domain events to the event store and write the integration events to the Outbox in a transaction
c. Write the integration events to Kafka

Then at the very beginning of the Usage section, we can write something like this:

We do not recommend using this library as a standalone process. Instead, embed it in your application. The points (b) and (c) are taken care by this library. You need to code the point (a) yourself depending on your domain logic.

Regards

[magro]

Thanks for your proposal!

Btw, this library was written to support transactional creation/updates/deletes of domain entities (not domain events) and creation of integration events. In an event sourced application I would have relied on event sourcing / processing stuff.

Regarding event sourcing I agree that's it's quite a change of the programming paradigm (or how software works), and it also requires new/additional tooling for operating an event sourced application (to follow and analyse user/system behavior and debug issues). I also only know few engineers who are experienced with event sourcing and the consequences and challenges that come with it. I.e. it depends also on the organisational context if it's a good technological choice.

[cloudcompute]

support transactional creation/updates/deletes of domain entities (not domain events) and creation of integration events

Okay

event sourcing and the consequences and challenges that come with it. I.e. it depends also on the organisational context

I agree. I read a lot about Event Sourcing and also went through the code of several open source libraries. As of now, my viewpoint is that we should begin with a small open-source ES library (instead of using a huge Axon-like framework) and keep adding features on top of it depending upon our requirements.

For example, I'd consider this feature as either intermediate or advanced:
https://github.com/castore-dev/castore/tree/main#event-groups

They have something called Event Groups. Is this really needed especially if we are writing a simple application.. I doubt.

[magro]

I'm not sure either. Their concept of an Event Group shows one of the challenges that may come with ES. If this is a good solution is a different question though. I'm also a bit irritated when reading EventAlreadyExistsError because this raises the question of consistency guarantees and the overall model to achieve them. Because I haven't looked deeper into how this library works I don't know the trade-offs they're choosing.

[cloudcompute]

reading EventAlreadyExistsError because this raises the question of consistency guarantees and the overall model to achieve them

I agree.