[RFC] Batch

[henrikbjorn]

This is a first attempt at creating a Batch overview system in Bernard. It implements some new concepts with fun names. Everything is open for change based on comments.

When you create a batch of messages first you create a Batch. This generates an id for the batch and also keeps track of its status (failed, total, successful).

I order to have messages be part of the batch they are assigned though Batch::assign(). At this point you have a Batch with assigned messages, theese have not yet been persisted. To persist them give the whole Batch to the producer.

$batch = new Batch;
$batch->assign(new DefaultMessage('Email'));

$producer->produce($batch);

It is also possible to assign messages to an already existing batch. This is done by creating a Batch using a specific id, or retrieving a batch for its storage. By producing the batch again its internal list of messages are flushed and the internal list is reset.

$batch = new Batch('existing-id');
$batch->assign(new DefaultMessage('Email'));

$producer->produce($batch);

As earlier mentioned it is possible to retrieve a Batch status from storage. Currently there is only a Redis implementation.

$storage = new RedisStorage($redis);

$batch = $storage->find('batch-id');
$batches = $storage->all();

// reload it status
$batch  = $storage->reload($batch);

// will block until all messages in batch is complete
while ($storage->reload($batch)->isRunning()) {
    continue;
}

Reloading will return a NEW batch object, so if you have assigned messages that are not yet been produced they will be lost.

Every batch is stored with its storage. This is done in events, one for the producer which updates the Batch total and one for the consumer wich increments failed and/or successful counts.

Theese are registered as event subscribers.

$storage = new RedisStorage($redis);
$eventDispatcher->addSubscriber(new BatchSubscriber($storage));

For bernard internals this is only possible by adding metadata to Envelope's which is called stamps. This is a hash of additional metadata given to an Envelope when created.

Does this make sense?

[henrikbjorn]

Integration for Juno and the Bundle will be done when this is mature enough to be used and have been tested in a real application.

[Baachi]

This should be private because the class is marked as final.

[henrikbjorn]

Will be fixed in next push, thanks :)

[henrikbjorn]

In the context of this it makes sense for the Producer to publish instead of produce. So what do people think of changing "Producer::produce()" into "Producer::publish()" (we will provide a proxy for produce).

[henrikbjorn]

A note on expiry. A Batch will be expired after a certain time, this is currently just cosmetic and means it won't be listed when calling all() on the storage.

[henrikbjorn]

Okay so we have now merged the Event pull request and this one have gotten its middleware updated to a EventSubscriber.

The remaining things are i am not quite sure "stamps" is the right way to go, also i facing an issue by not having the full Batch object when working with an Envelope. The "perfect" world would have the stamp deserialized/serialized when dequeuing or enqueuing the envelope. Doing that is "problematic" as every serializer implementation must support it and also have access to the storage in order to sets it status. Unless be just decide a deserialized batch does not automatically get its status updated.

Whatever we decide will also be the way a Retry mechanish is done, aka adding a Retry object containing the metadata.

[estahn]

@henrikbjorn Do you have a real world use case scenario?

[henrikbjorn]

Hey @estahn

Our usecase is we will push 10 messages to the queue at once, and then use Redis as a temporary storage. That storage needs to be cleaned up and with Batches it would be easier to track if all messages that use that data are done.

[henrikbjorn]

I think this is close to being done, if the thought behind it can be validated.