Programmable Event Target

[nkreiger]

It would be really powerful to have a target that will emit the received event at a programmed time to simulate/create scenarios like cache expiration, delayed notifications, etc.

The challenges, layed our perfectly by @odacremolbap

The challenges are

• if this is developed as a target it would live in memory. The event received at the target should be ACKed to avoid blocking the sender for a long time, but if the target instance goes down, the event is lost.

• if developed as a target, there is a good chance of OOM due to too many events living in memory.

• there is a much better way of implementing this counting on the broker, because it is the stateful piece and will store whatever pending programmed events exist, but it requires a combination of ordering the pending events per launch condition (something that data streams do not usually do), polling the list of events that might be ready to launch, and keeping the next to be sent in memory. In certain cases that can be very resource consuming.

Personally, I'm leaning towards using the broker for implementation. I have one setup where I cycle the events back through the broker until a process finishes, which works well and protects against losing the event in case of a crash. However, are their performance concerns for doing that at scale? I would think this would not be the best at scale with 100,000s events just constantly cycling through endlessly for a long running process.

Is there a performance concern for not sending an ACK to the broker? And the HTTPTarget returns an appropriate error when it hits its memory limit? Maybe this can be solved by autoscaling?

Can the logic live in the broker itself? triggered by a special cloud event header? Order would then be determined by expiration, not by the stream