Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add in memory Original Source to Message class #19220

Open
ryan-carroll-graylog opened this issue May 1, 2024 · 0 comments
Open

Add in memory Original Source to Message class #19220

ryan-carroll-graylog opened this issue May 1, 2024 · 0 comments

Comments

@ryan-carroll-graylog
Copy link
Contributor

What?

We add #setOriginalSource() and #getOriginalSource() methods to the Message class that set and return the original message as bytes or string. In addition to that, we add has_original_source and get_original_source (with an optional default return value) pipeline functions to check and retrieve the original source value.

In input codecs, we can add the original source to each message object. Not into the message fields map, but either as a separate field or the existing metadata map. That way the original source doesn't get indexed into OpenSearch.

The main drawback is that the memory consumption of the in-flight Message object increases. We can benchmark that to see how big of an impact that is. We can also add an option to inputs to disable the storage of the original source data in the in-memory message object. (default is enabled)

Why?

Proposed as potential solution to #18416, but would be useful for many instances where Illuminate processing would benefit from access to the original message source but we would not want to store all data on the indexed message.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants