Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.

Already on GitHub? Sign in to your account

Initial cut of AWS Kinesis streaming #8967

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

dpbevin
Copy link
Contributor

@dpbevin dpbevin commented May 1, 2024

Relates to #8966

An initial attempt at producing a persistent stream implementation for AWS Kinesis streams. This is 100% an early PR to solicit feedback (and gauge interest?).

Tested locally against: https://github.com/etspaceman/kinesis-mock

This approach uses the AmazonKinesisClient from the .NET SDK, not the "KCL Consumer" that requires a running JVM 馃ぎ

Includes

  • Kinesis stream provider
  • A grain-based queue checkpointer (instead of needing to require DynamoDB, EF Core, or some other approach). This checkpointer could be used by EventHub too (instead of requiring Azure Table Storage).

Remaining work

  • A few "TODO"s
  • Tests

Useful references:

Microsoft Reviewers: Open in CodeFlow

@scalalang2
Copy link

scalalang2 commented May 15, 2024

this is what I've been waiting for. thanks :)

[PreferLocalPlacement]
public class StreamCheckpointerGrainGrain : Grain, IStreamCheckpointerGrain
{
private readonly IPersistentState<StreamCheckpointerGrainState> _state;

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm here to drop a very trivial question : )
Is there any reason not to use the native checkpoint of Kinesis?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants