Late arriving data - by design - an IoT use case #2503
Recodify
started this conversation in
Show and tell
Replies: 1 comment
-
This will probably be a non-issue in the near future. We are working on the WAL (write ahead log) functionality to allow for read replicas. The way WAL works, all the data we receive is first stored in the WAL files, then committed to the TableWriter. This means that non committed data will still be stored in the WAL, and also means when TableWriter is committing data, ingestion is decoupled, so less affected by potential out of order problems. WAL is (hopefully) not even a few months away. For the time being, I don't see any options other than the ones you described. |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
Hi everyone,
Not really a show and tell, but more a show and ask. I toyed with putting this discussion in slack, but I think what I'm going to ask might be useful to others and things tend to get buried in slack.
My Use Case.
So, in summary, delivery of data across the network is "laggy" by design. This lag is obviously significantly longer than the type of lag you would expect to see with typical TCP based delivery due to network jitter and out-of-order (OOO) async processing.
My Use Case - In QuestDb
I see a few possible approaches.
Option 1 - long commit lag
Set a commit lag in QuestDb to cover the 2 hour window + a little extra to account for the standard jitter once the message arrives on a TCP network.
This feels like the best approach for optimal data ingest but feels like it might have some potential pain points.
Option 2 - slower ingest
Accept that all my ingests will incur the penalty of data file resorts
Option 3 - I'm missing something
I hope this is the case! Is there a third option which I've not considered?
Would love to get everyones thoughts.
Beta Was this translation helpful? Give feedback.
All reactions