Agent mode support for bootstrap output

[michel-laterman]

What is the problem this PR solves?

When running under the elastic-agent fleet-server is not able to use output settings from it's policy.

How does this PR solve the problem?

This PR requires elastic/elastic-agent#4643 to work.

The elastic-agent will inject enrollment configuration options in output.elasticsearch.bootstrap instead of overwriting matching keys in output.elasticsearch.
When running under agent mode, fleet-server will inject specific keys in bootstrap that are not in output.elasticsearch, then test the resulting output to see if it can connect to Elasticsearch, if so it is used. If not then bootstrap is used instead and the output is periodically retested in case the failure was caused by a temporary network issue.

How to test this PR locally

Create an elastic-agent package from: elastic/elastic-agent#4643
replace the fleet-server component using one generated from this pr.

If testing with docker images docker.elastic.co/observability-ci/elastic-agent:8.15.0-SNAPSHOT-dd4c89e-1715633206 can be used as the BASE_IMAGE for generating a new image/deployment with the make cloud-deploy target in dev-tools/cloud.
Or the docker.elastic.co/observability-ci/elastic-agent:8.15.0-SNAPSHOT-laterman-1715705581 image can be used as it contains the changes from both PRs.

I've verified that the following behaviours work:

When deployed to ESS:

• fleet-server is available/healthy
• logging level for fleet-server can be changed with no issues
• diagnostics can be collected
• e2e cypress tests for defender succeed (thanks @tomsonpl!)

When deployed locally:

• fleet-server is available/healthy
• logging level for fleet-server can be changed with no issues
• diagnostics can be collected
• multiple hosts can be added to Elasticasearch output (in Kibana) and show up in fleet-server.yml when a diagnostics bundle is collected

Design Checklist

• I have ensured my design is stateless and will work when multiple fleet-server instances are behind a load balancer.
• I have or intend to scale test my changes, ensuring it will work reliably with 100K+ agents connected.
• I have included fail safe mechanisms to limit the load on fleet-server: rate limiting, circuit breakers, caching, load shedding, etc.

Checklist

• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in ./changelog/fragments using the changelog tool

Related issues

• Closes separate "bootstrap" from retrieved config when running in agent mode #3464
• Relates Fleet Server configuration does not contain all the hosts available in the Elasticsearch cluster elastic-agent#2784

[michel-laterman]

Testing progress: elastic/elastic-agent#4643 (comment)

i'll now try to implement the areas of improvement above:

1. skip injecting verification_mode: none if a CA or CA fingerprint is in the retrieved policy.
2. async period output testing if the bootstap block has been passed

[pchila]

Just a couple of comments, looks good overall

[cmacknz]

I think we need an automated test proving that Elastic Agent can bootstrap Fleet Server in one of the repositories before this or elastic/elastic-agent#4643 are merged.

If the coordination of the two PRs with the test is annoying enough I'd be fine with the test being added a separate PR, but not closing the implementation issue until it exists.

[michel-laterman]

buildkite test this

[elastic-sonarqube]

Quality Gate passed

Issues
2 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
77.4% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube