Open-telemetry data pipline(ETL) use case discussion #1344
Replies: 2 comments 1 reply
-
Hi, What I was hoping to get out of something like distributed tracing for this kind of use case is to be able to track the full lifetime of a data object, as well as get information about propagation delays across the whole system. I'm still trying out a few ideas, but in the end I believe this may be just a visualisation problem - using OpenTelemetry APIs for distributed tracing I believe I can collect all the needed information, it's just that regular tracing UIs are not ready for long lived traces - this use case is a bit different from a client-facing HTTP API. There is also the issue of systems that do aggregations - if we think of starting the trace when the first message is produced, and a system downstream needs to aggregate N different messages, we have a problem: we can either publish spans to N different traces, or be able to create a new trace referencing the all original ones. This is still something I need to work out. |
Beta Was this translation helpful? Give feedback.
-
Hey! 🤗 @danielbcorreia @MarsSu0618 |
Beta Was this translation helpful? Give feedback.
-
Hi, everyone.
I am a newbie in open-telemetry and as a data engineer in my team.
Because our team would like to introduce this open source to implement observability of microservice.
Today I want to survey whether it have related data pipeline use case or architecture which can be referred.
By the way, my team's objective is to build AIOps Platform in the future.
Based on this reason, we need to buildup completed data pipeline to process and store related data so that next step can use ML algorithm to apply in advanced.
If someone have any idea or related experiences, hope you can share it to me. I will be appreciated in advanced.
Beta Was this translation helpful? Give feedback.
All reactions