Create DataLink for Kafka [HZ-1985] #23886
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
For details on KafkaDataLink see the Javadoc.
frant-hartm
added
Type: Enhancement
Source: Internal
frant-hartm
changed the title
TomaszGaweda
approved these changes
Mar 9, 2023
ldziedziul
approved these changes
Mar 9, 2023
Comment on lines
+320
to
+352
| * The behavior depends on the job's processing guarantee: | ||
| * <ul> | ||
| * <li><em>EXACTLY_ONCE:</em> the sink will use Kafka transactions to | ||
| * commit the messages. This brings some overhead on the broker side, | ||
| * slight throughput reduction (we don't send messages between snapshot | ||
| * phases) and, most importantly, increases the latency of the messages | ||
| * because they are only visible to consumers after they are committed. | ||
| * <p> | ||
| * When using transactions pay attention to your {@code | ||
| * transaction.timeout.ms} config property. It limits the entire | ||
| * duration of the transaction since it is begun, not just inactivity | ||
| * timeout. It must not be smaller than your snapshot interval, | ||
| * otherwise the Kafka broker will roll the transaction back before Jet | ||
| * is done with it. Also it should be large enough so that Jet has time | ||
| * to restart after a failure: a member can crash just before it's | ||
| * about to commit, and Jet will attempt to commit the transaction | ||
| * after the restart, but the transaction must be still waiting in the | ||
| * broker. The default in Kafka 2.4 is 1 minute. | ||
| * | ||
| * <li><em>AT_LEAST_ONCE:</em> messages are committed immediately, the | ||
| * sink ensure that all async operations are done at 1st snapshot | ||
| * phase. This ensures that each message is written if the job fails, | ||
| * but might be written again after the job restarts. | ||
| * </ul> | ||
| * | ||
| * If you want to avoid the overhead of transactions, you can reduce the | ||
| * guarantee just for the sink by calling {@link | ||
| * Builder#exactlyOnce(boolean) exactlyOnce(false)} on the returned builder. | ||
| * <p> | ||
| * IO failures are generally handled by Kafka producer and do not cause the | ||
| * processor to fail. Refer to Kafka documentation for details. | ||
| * <p> | ||
| * Default local parallelism for this processor is 1. |
There was a problem hiding this comment.
Maybe we could reference this javadoc to avoid duplicated description
There was a problem hiding this comment.
We have a mix of approaches here, I kept what was the case for the Kafka source/sinks which repeats the javadoc.
| if (properties != null) { | ||
| metaSupplier = writeKafkaP(properties, topic, extractKeyFn1, extractValueFn1, exactlyOnce); | ||
| } else { | ||
| assert dataLinkRef != null; |
There was a problem hiding this comment.
asserts are disabled by default, couldn't we throw IllegalStateException here?
There was a problem hiding this comment.
if you look at the constructors of the builder one is always non-null and the other null, I actually added these asserts to make intellij shut up about possible NPE and it's nicer than a suppression
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
For details on KafkaDataLink see the Javadoc.
Checklist:
Team:,Type:,Source:,Module:) and Milestone setAdd to Release NotesorNot Release Notes contentset@Nonnull/@Nullableannotations@sincetags in Javadoc