SERVER-81719: add new kafka loader actor #1029
Conversation
Aadeshp
force-pushed
the
aa/kafka-loader
branch
from
ed18626
to
d96ffd0
Compare
Aadeshp
force-pushed
the
aa/kafka-loader
branch
from
d96ffd0
to
b478046
Compare
Aadeshp
changed the title
|
I have added myself and @PyGeek03 as reviewers on this. I have added @johndaniels and @lgfang to CC them in on this review, but they're optional (unless they are requesting changes) |
|
Hi @Aadeshp, can you update the TOOLCHAIN_BUILD_ID to be |
| auto inserts = _totalInserts.start(); | ||
| BOOST_LOG_TRIVIAL(debug) << " KafkaLoader Inserting " << json; |
There was a problem hiding this comment.
Logging might introduce unpredictable performance variance, so I think we should do any logging outside of the code path being measured as much as possible:
| auto inserts = _totalInserts.start(); | |
| BOOST_LOG_TRIVIAL(debug) << " KafkaLoader Inserting " << json; | |
| BOOST_LOG_TRIVIAL(debug) << " KafkaLoader Inserting " << json; | |
| auto inserts = _totalInserts.start(); |
| using namespace genny::testing; | ||
| namespace bson_stream = bsoncxx::builder::stream; | ||
|
|
||
| TEST_CASE_METHOD(KafkaTestFixture, "KafkaLoader successfully connects to a Kafka broker.", "[streams][KafkaLoader]") { |
There was a problem hiding this comment.
An Actor test case would only be run if one of the square-bracketed tags match one of the tags defined in a YAML file in this directory. The existing defined tags are [sharded], [single_node_replset], and [three_node_replset] ([SelfTestActor] is to test Genny code, not Actor code, so it should not be used). The square-bracketed tags are not arbitrary. So we have 2 options here:
- Add one of the three existing defined tags above.
- Add a new YAML file under the above directory to define either
[streams]or[KafkaLoader].
On a related note, I found that this actor test case is also not being run because neither [streams] nor [StreamStatsReporter] are defined tags. But I think this should be fixed in another PR instead of in this PR.
|
Hey @Aadeshp, this PR has been sitting in draft for a while, what are the plans for this one? |
Thanks for submitting a PR to the Genny repo. Please include the following fields (if relevant) prior to submitting your PR.
Jira Ticket: SERVER-81719
Whats Changed:
Introducing a new actor that inserts data into a specified kafka cluster. This won't be used in DSI automatically right now since DSI doesn't support kafka at the moment, but this is moreso for local testing.
Related PRs:
vcpkg PR to install librdkafka for the genny toolchain (https://github.com/10gen/vcpkg/pull/29)