add ComponentLoader to support more auto configuration scenarios #6217

zeitlinger · 2024-02-09T14:14:14Z

Part of open-telemetry/opentelemetry-java-instrumentation#10409

Here's a PoC to show that the functionality actually works for spring boot: open-telemetry/opentelemetry-java-instrumentation#10453

the new component loader: https://github.com/zeitlinger/opentelemetry-java/blob/1a6f0016521bfbdc7c7fc3900655efdd4df2fd16/sdk-extensions/autoconfigure/src/main/java/io/opentelemetry/sdk/autoconfigure/ComponentLoader.java#L28-L42
this is the spring implementation of the component loader: https://github.com/open-telemetry/opentelemetry-java-instrumentation/pull/10453/files#diff-077c09a6a48b1e63a0d47867f73142466872586cf04fb2de15eb16c8da11d221R19-R72

Why do we need the new component loader public interface?

Let's say a spring user wants to create a new span exporter - then they would create the following spring class:

@Configuration
public class OtelConfig {
  @Bean
  ConfigurableSpanExporterProvider mySpanExporter() {
    Return new MySpanExporterProvider()
  }
}

public class MySpanExporterProvider implements ConfigurableSpanExporterProvider {

}

(They would also need to set otel.spans.exporter to use the span exporter.)

Normally this would not have any effect on the SDK auto-config, because Spring beans are different from the SPI way of loading things.

The component loader allows the spring starter to create an adapter, which is shown here

zeitlinger · 2024-02-09T14:47:28Z

@jack-berg please take a look 😄

jack-berg

I think I can get behind this but need to think about it a little more. If we go in this direction, I think we'll want to incubate the ComponentLoader class in an internal class for some period of time.

.../src/main/java/io/opentelemetry/sdk/autoconfigure/AutoConfiguredOpenTelemetrySdkBuilder.java

...toconfigure/src/main/java/io/opentelemetry/sdk/autoconfigure/MeterProviderConfiguration.java

codecov · 2024-02-19T13:49:00Z

Codecov Report

Attention: Patch coverage is 88.23529% with 4 lines in your changes are missing coverage. Please review.

Project coverage is 91.04%. Comparing base (1d4a2f4) to head (22996e0).
Report is 2 commits behind head on main.

Files	Patch %	Lines
.../sdk/autoconfigure/internal/AutoConfigureUtil.java	71.42%	4 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #6217      +/-   ##
============================================
- Coverage     91.06%   91.04%   -0.02%     
- Complexity     5695     5705      +10     
============================================
  Files           621      621              
  Lines         16667    16703      +36     
  Branches       1707     1709       +2     
============================================
+ Hits          15177    15208      +31     
- Misses          997     1002       +5     
  Partials        493      493

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

zeitlinger · 2024-02-19T16:11:04Z

@jack-berg added all suggested changes - and even figured out how to appease jCmpApi (hopefully correctly).

zeitlinger · 2024-02-19T18:27:35Z

An additional SPI is needed for ConfigProperties - this is needed to create an adapter for spring properties: https://github.com/zeitlinger/opentelemetry-java/blob/c1534b5513b29c6ce6b8020bbf05a5bead68e5a2/sdk-extensions/autoconfigure/src/main/java/io/opentelemetry/sdk/autoconfigure/AutoConfiguredOpenTelemetrySdkBuilder.java#L571-L585

zeitlinger · 2024-02-20T09:09:30Z

An additional SPI is needed for ConfigProperties - this is needed to create an adapter for spring properties: https://github.com/zeitlinger/opentelemetry-java/blob/c1534b5513b29c6ce6b8020bbf05a5bead68e5a2/sdk-extensions/autoconfigure/src/main/java/io/opentelemetry/sdk/autoconfigure/AutoConfiguredOpenTelemetrySdkBuilder.java#L571-L585

sorry, just saw that there is already setConfig

...autoconfigure/src/main/java/io/opentelemetry/sdk/autoconfigure/internal/ComponentLoader.java

...configure-spi/src/main/java/io/opentelemetry/sdk/autoconfigure/spi/ConfigurableProvider.java

.../src/main/java/io/opentelemetry/sdk/autoconfigure/AutoConfiguredOpenTelemetrySdkBuilder.java

...oconfigure/src/main/java/io/opentelemetry/sdk/autoconfigure/TracerProviderConfiguration.java

…. spring boot

…e it

This reverts commit 1a867b1.

This reverts commit c1534b5.

This reverts commit 655d2e6.

remove SPI for SpanProcessor

jack-berg · 2024-02-28T15:42:17Z

.../src/main/java/io/opentelemetry/sdk/autoconfigure/AutoConfiguredOpenTelemetrySdkBuilder.java

@@ -368,6 +369,13 @@ public AutoConfiguredOpenTelemetrySdkBuilder setServiceClassLoader(
    return this;
  }

+  /** Sets the {@link ComponentLoader} to be used to load SPI implementations. */
+  public AutoConfiguredOpenTelemetrySdkBuilder setComponentLoader(ComponentLoader componentLoader) {


Actually this is problematic. Can't have a public API reference an internal class, since this would break when we move ComponentLoader to a public package. The way we've implemented this stuff in the past is:

ComponentLoader is in an internal package

AutoConfiguredOpenTelemetrySdkBuilder#setComponentLoader is package private

A public static method is added to a class in an internal package which reflectively calls AutoConfiguredOpenTelemetrySdkBuilder#setComponentLoader. The natural place for this is AutoConfigureUtil. If a user wants to use the experimental functionality, they can call the public static method on the internal AutoConfigureUtil until we promote the functionality to the public API.

ah, that's why we have those reflective methods 😄

Its strange but it works 😅

jack-berg · 2024-02-28T15:44:49Z

.../src/main/java/io/opentelemetry/sdk/autoconfigure/AutoConfiguredOpenTelemetrySdkBuilder.java

@@ -115,7 +116,7 @@ public final class AutoConfiguredOpenTelemetrySdkBuilder implements AutoConfigur
   * {@link #addPropertiesSupplier(Supplier)} and {@link #addPropertiesCustomizer(Function)} will
   * have no effect if this method is used.
   */
-  AutoConfiguredOpenTelemetrySdkBuilder setConfig(ConfigProperties config) {
+  public AutoConfiguredOpenTelemetrySdkBuilder setConfig(ConfigProperties config) {


Can you explain the reasoning for this? Its pretty drastic to want to call this and negate calls to addPropertiesSupplier and addPropertiesCustomizer.

Its currently only being used in tests, and even those usages aren't necessary. I honestly think we ought to delete it altogether rather than promote it to the public API.

Spring needs a different implementation of ConfigProperties, because of limitations in list and map handling - see https://github.com/open-telemetry/opentelemetry-java-instrumentation/blob/d8aa0f5b48e2dee28498b6f95d3d0017eb673cbf/instrumentation/spring/spring-boot-autoconfigure/src/main/java/io/opentelemetry/instrumentation/spring/autoconfigure/resources/SpringConfigProperties.java#L102-L117

The alternative would be to stringify maps and lists - and then let them be parsed using DefaultConfigProperties - that doesn't look right, IMO.

My suggestion: Make this method experimental like setComponentLoader

My suggestion: Make this method experimental like setComponentLoader

What would be the path to stabilizing? I think we should stabilize setComponentLoader after we integrate it into spring autoconfigure and confirm there we didn't miss anything. But overriding the config here conflicts with autoconfigure patterns by erasing property suppliers and customizers provided via SPI. Overriding ConfigProperties means that spring autoconfigure would interact with most SPIs normally, except those involving customizing config properties. Would be surprising to users.

What about an experimental method to customize ConfigProperties? Something like:

AutoConfiguredOpenTelemetrySdkBuilder setConfigPropertiesCustomizer(Function<ConfigProperties, ConfigProperties> configPropertiesCustomizer)

The idea is that it follows a similar pattern to the customization methods in AutoConfigurationCustomizer like AutoConfigurationCustomizer#addPropertiesCustomizer(Function<ConfigProperties, Map<String, String>>), but is able to return a custom ConfigProperties instance instead of Map<String, String>. We can limit this method to AutoConfigurationOpenTelemetrySdkBuilder (i.e. do not add it to AutoConfigurationCustomizer). Spring autoconfiguration's implementation of ConfigProperties can fallback to the ConfigProperties instance provided by autoconfigure if a particular property is undefined in spring's configuration system.

That idea is quite elegant.

I don't think a spring user would expect that capability, because spring can already do that: https://stackoverflow.com/questions/29072628/how-can-i-override-spring-boot-application-properties-programmatically

I think your proposal could be good for distros of the spring starter, so I'll implement it.

jack-berg · 2024-02-28T15:45:16Z

buildSrc/src/main/kotlin/otel.japicmp-conventions.gradle.kts

@@ -115,6 +115,8 @@ if (!project.hasProperty("otel.release") && !project.name.startsWith("bom")) {
        // Reproduce defaults from https://github.com/melix/japicmp-gradle-plugin/blob/09f52739ef1fccda6b4310cf3f4b19dc97377024/src/main/java/me/champeau/gradle/japicmp/report/ViolationsGenerator.java#L130
        // with some changes.
        val exclusions = mutableListOf<String>()
+        // Generics are not detected correctly
+        exclusions.add("CLASS_GENERIC_TEMPLATE_CHANGED")


Shouldn't need this anymore.

not for this PR - but it's still true that jApiCmp will trip over generics - I'm fine to take it out, of course.

jack-berg

Couple more comments but I think we're close.

trask · 2024-02-29T01:26:28Z

...oconfigure/src/main/java/io/opentelemetry/sdk/autoconfigure/TracerProviderConfiguration.java

-        {
-          double ratio =
-              config.getDouble("otel.traces.sampler.arg", DEFAULT_TRACEIDRATIO_SAMPLE_RATIO);
-          return Sampler.parentBased(Sampler.traceIdRatioBased(ratio));
-        }
+        return Sampler.parentBased(ratioSampler(config));


this is a nice change, but seems unrelated to this PR? in the future splitting out unrelated changes can make life easier for reviewers

trask · 2024-02-29T01:31:09Z

...autoconfigure/src/main/java/io/opentelemetry/sdk/autoconfigure/internal/ComponentLoader.java

+  /**
+   * Load implementations of an ordered SPI (i.e. implements {@link Ordered}).
+   *
+   * @param spiClass the SPI class
+   * @param <T> the SPI type
+   * @return list of SPI implementations, in order
+   */
+  default <T extends Ordered> List<T> loadOrdered(Class<T> spiClass) {
+    return StreamSupport.stream(load(spiClass).spliterator(), false)
+        .sorted(Comparator.comparing(Ordered::order))
+        .collect(Collectors.toList());
+  }


do we need this method? (can inline as private method in SpiHelper?)

the method is meant for overriding.
In spring, the spring beans are given a higher priority - https://github.com/open-telemetry/opentelemetry-java-instrumentation/pull/10453/files#diff-8c1885af89564bfbd461edbbf02fe19e23e40a2e54f6e264d41d786b760e5d26R40-R53.

I'm still unsure if that's what users actually expect though.

It affects

AutoConfigurationCustomizerProvider

ResourceProvider

can't find a really compelling reason - I'll remove it.

zeitlinger · 2024-03-05T08:12:33Z

@jack-berg can you check again?

zeitlinger · 2024-03-11T13:59:37Z

Thanks @jack-berg for getting it into this release 🎉

zeitlinger mentioned this pull request Feb 9, 2024

Spring use sdk autoconfig open-telemetry/opentelemetry-java-instrumentation#10453

Merged

2 tasks

zeitlinger force-pushed the componet-loader branch from 1c89890 to 1a6f001 Compare February 9, 2024 14:43

zeitlinger marked this pull request as ready for review February 9, 2024 14:47

zeitlinger requested a review from a team as a code owner February 9, 2024 14:47

zeitlinger force-pushed the componet-loader branch from b2539b2 to 703e039 Compare February 13, 2024 14:20

jack-berg reviewed Feb 15, 2024

View reviewed changes

.../src/main/java/io/opentelemetry/sdk/autoconfigure/AutoConfiguredOpenTelemetrySdkBuilder.java Outdated Show resolved Hide resolved

...toconfigure/src/main/java/io/opentelemetry/sdk/autoconfigure/MeterProviderConfiguration.java Outdated Show resolved Hide resolved

zeitlinger force-pushed the componet-loader branch from ba945fc to 5c9e395 Compare February 19, 2024 12:44

zeitlinger force-pushed the componet-loader branch from 1d8b60c to 441ca55 Compare February 25, 2024 12:27

jack-berg reviewed Feb 27, 2024

View reviewed changes

zeitlinger added 16 commits February 28, 2024 07:05

add ComponentLoader to support more auto configuration scenarios, e.g…

0531869

…. spring boot

format

6d51896

add loadOptional

2b9fb28

customizers are sufficient

e1352d6

ExemplarFilter is going to be removed from public API, so don't expos…

091b19b

…e it

customizers are sufficient

eac57ca

add exception for jcmpapi

ae2c633

jcmpapi

2bec27f

test component loader is used

ad4047f

Revert "customizers are sufficient"

5336d59

This reverts commit 1a867b1.

add loader for ConfigProperties

44df716

Revert "add loader for ConfigProperties"

74322cd

This reverts commit c1534b5.

Revert "Revert "customizers are sufficient""

5799589

This reverts commit 655d2e6.

add loader for ConfigProperties

3f574fb

make component loader internal to give it time to incubate

6d06755

remove ConfigurableProvider

42d6e17

remove SPI for SpanProcessor

add javadoc

d03a67f

zeitlinger force-pushed the componet-loader branch from 441ca55 to d03a67f Compare February 28, 2024 06:39

jack-berg reviewed Feb 28, 2024

View reviewed changes

make new methods experimental

517d773

trask reviewed Feb 29, 2024

View reviewed changes

zeitlinger added 3 commits February 29, 2024 09:09

don't make loadOrdered overridable

e97c4f9

add setConfigPropertiesCustomizer

4519a7e

add setConfigPropertiesCustomizer

22996e0

jack-berg approved these changes Mar 5, 2024

View reviewed changes

jack-berg merged commit 56c11e5 into open-telemetry:main Mar 6, 2024
18 checks passed

zeitlinger deleted the componet-loader branch March 11, 2024 13:59

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add ComponentLoader to support more auto configuration scenarios #6217

add ComponentLoader to support more auto configuration scenarios #6217

zeitlinger commented Feb 9, 2024 •

edited

zeitlinger commented Feb 9, 2024

jack-berg left a comment

codecov bot commented Feb 19, 2024 •

edited

zeitlinger commented Feb 19, 2024

zeitlinger commented Feb 19, 2024

zeitlinger commented Feb 20, 2024

jack-berg Feb 28, 2024

zeitlinger Feb 28, 2024

jack-berg Feb 28, 2024

jack-berg Feb 28, 2024

zeitlinger Feb 28, 2024

zeitlinger Feb 28, 2024

jack-berg Feb 28, 2024

jack-berg Feb 28, 2024

zeitlinger Feb 28, 2024

zeitlinger Feb 29, 2024

jack-berg Feb 28, 2024

zeitlinger Feb 28, 2024

jack-berg left a comment

trask Feb 29, 2024

trask Feb 29, 2024

zeitlinger Feb 29, 2024

zeitlinger Feb 29, 2024

zeitlinger commented Mar 5, 2024

zeitlinger commented Mar 11, 2024

add ComponentLoader to support more auto configuration scenarios #6217

add ComponentLoader to support more auto configuration scenarios #6217

Conversation

zeitlinger commented Feb 9, 2024 • edited

Why do we need the new component loader public interface?

zeitlinger commented Feb 9, 2024

jack-berg left a comment

Choose a reason for hiding this comment

codecov bot commented Feb 19, 2024 • edited

Codecov Report

zeitlinger commented Feb 19, 2024

zeitlinger commented Feb 19, 2024

zeitlinger commented Feb 20, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jack-berg left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zeitlinger commented Mar 5, 2024

zeitlinger commented Mar 11, 2024

zeitlinger commented Feb 9, 2024 •

edited

codecov bot commented Feb 19, 2024 •

edited