New component: Failover Connector

[djaglowski]

The purpose and use-cases of the new component

A connector that routes data based on the current health status of a downstream component, typically an exporter.

I have heard several users ask for the ability to send data to a backup exporter, if a primary exporter fails. I believe this could be implemented as a routing connector.

The user would specify at least one pipeline to which data would typically be routed. Additionally, the user must specify at least one backup pipeline or pipelines which would be used when an error is encountered.

Initially, I think the trigger for routing to a backup pipeline could be based on backpropogated errors, though this is not yet very robust (See open-telemetry/opentelemetry-collector#7460). At a later time, I imagine this could be based on the health status of an exporter (See open-telemetry/opentelemetry-collector#6344).

Example configuration for the component

receivers:
  foo:

exporters:
  bar/main:
  bar/backup:

connectors:
  failover:
    primary: logs/main
    secondary: logs/backup

service:
  pipelines:
    logs/in:
       receivers: [foo]
       exporters: [failover]
    logs/main:
      receivers: [failover]
      exporters: [bar/main]
    logs/backup:
      receivers: [failover]
      exporters: [bar/backup]

Telemetry data types supported

traces->traces
metrics->metrics
logs->logs

Is this a vendor-specific component?

• This is a vendor-specific component
• If this is a vendor-specific component, I am proposing to contribute this as a representative of the vendor.

Sponsor (optional)

No response

Additional context

No response

[sethallen]

I'm glad you made this @djaglowski! I was just chatting with @atoulme about adding failover and circuit breaker support for exporters a couple days ago. The connector seems like a great method to add broad failover support.

How about tweaking this slightly to support 1..N entries as a yaml flow sequence? It would reduce complexity in the failover connector by removing the need for keys (primary, secondary, etc.) in order to choose the next pipeline to failover to.

Example:

receivers:
  foo:

exporters:
  bar/main:
  bar/backup:
  bar/backup2:

connectors:
  failover: [logs/main, logs/backup, logs/backup2, .. n]
#    primary: logs/main
#    secondary: logs/backup

service:
  pipelines:
    logs/in:
       receivers: [foo]
       exporters: [failover]
    logs/main:
      receivers: [failover]
      exporters: [bar/main]
    logs/backup:
      receivers: [failover]
      exporters: [bar/backup]
    logs/backup2:
      receivers: [failover]
      exporters: [bar/backup2]

[fatsheep9146]

I'd like to sponsor this.

[djaglowski]

@sethallen, I like the idea of allowing a priority list, but I think we should leave room for other parameters as well. I also think we need to allow multiple pipelines per "level".

connectors:
  failover: 
    priority:
      - [logs/main]
      - [logs/backup, logs/backup2]
      - [logs/backup/3]
    min_failover_interval: 2m # Possibly would add this in future

[cparkins]

@djaglowski How would the multiple pipelines be used? In a fan-out or Priority 1, Priority 2-1, Priority 2-2, ... Priority N

[djaglowski]

@cparkins, when there are multiple pipelines at the same priority level, it would fan out data to those pipelines.

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

[akats7]

@djaglowski I also had this feature request, I'd be happy to work on it/ support any way I can.

[djaglowski]

@akats7, any help moving this forward would be great. I'll be happy to review any PRs.

[akats7]

@djaglowski sounds good, can I please be assigned this issue.

[sethallen]

@djaglowski / @akats7 / @atoulme - Perhaps this work effort can be merged with what @cparkins has been working on internally for us over the last few months. He added resiliency features (Failover, Circuit Breaker) to the Splunk HEC Exporter for the OTel Collector and submitted them in the PR below:

• Add Resiliency Features to the Splunk HEC Exporter #23821

[djaglowski]

@sethallen, I'm supportive of the idea. In my opinion, failover at least should be implemented as a connector because in many cases it may be appropriate to failover to a different type of exporter. If I recall correctly, you and/or @cparkins looked into the idea of implementing other resiliency features into a connector. Do you still see that as a viable path? Either way, I think the failover connector should move forward and we can add additional capabilities based on a proposal.

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

[akats7]

^ I was able to begin looking into this recently and will open a first pass PR for this shortly.

[sethallen]

That's exciting @akats7. We've been maintaining an internal fork of resiliency features added to the Splunk HEC Exporter and would love to get these features somewhere into the mainline collector. Your PR for a Connector will be great to see and hopefully help with. Cheers!

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

[verejoel]

Would love to revive this, definitely interested in this topic.

[djaglowski]

Thanks for pinging this @verejoel.

An implementation is in place but stability is still marked as development. @akats7, do you recall what is left to do? If we have a minimally functional component then I think we should move it to alpha status, close this issue and open issues for any additional functionality we would like.

[akats7]

Hey @djaglowski @verejoel,

Yep the MVP functionality is in place, I did have one more change I've been planning to push so I'll push that along with the update to Alpha.