v1.0.3

2021-12-20 Version 1.0.3

---

Security

• Fixes CVE-2021-45105 by using Apache Log4J 2.17.0. #949

v1.0.1

2021-12-10 Version 1.0.1

---

Security

• Fixes CVE-2021-44228 by using Apache Log4J 2.15.0. #942, #944
• Run yum update on Docker images to get all security patches at the time of the Data Prepper Docker build. #943

Version 1.0.0

2021-05-11 Version 1.0.0

Highlights

• Now builds using version 1.0+ of the OpenTelemetry tracing specification
• Additional TraceGroup fields are emitted for enhanced searching and filtering

Compatibility

• No compatibility issues with previous versions of Data Prepper.

Version 0.8.0-beta

2021-03-25 Version 0.8.0 (beta)

Highlights:

• Horizontally-scaling clusters can now be deployed by using the new Peer Forwarder plugin.
  • Refer to the README file for usage instructions
  • See the deployment templates directory for new container-based deployment strategies.
• Prometheus-friendly metrics can now be scraped via a new /metrics endpoint.
  • See the monitoring docs for more information

---

Peer Forwarder Plugin

To enable horizontal scaling, it is expected to run Data Prepper instances in a cluster with requests distributed by a load balancer. This complicates preppers which maintain state between requests, specifically those which require that only one instance process all spans for a given trace. Peer forwarding facilitates this usecase by ensuring all spans for a given trace arrive at a specific Data Prepper instance.

The Peer Forwarding plugin enables instances in a Data Prepper cluster to forward requests to other specific instances within the cluster. Peer forwarding is accomplished by a consistent hashing algorithm to ensure that all requests for a given trace are routed to the same Data Prepper instance. The algorithm maps requests to addresses of known instances - these instances are provided via a peer discovery mechanism. As of version 0.8.0, peer discovery can be provided by either a static address list or by dynamic DNS lookups.

Refer to the README file for usage instructions.

---

Metrics

Version 0.8.0-beta offers APIs which provide JVM/system metrics and Plugin metrics.

Users can use these endpoints to connect to their existing Prometheus instance to start aggregating metrics. Additional tools such as Grafana can then be used to provide dashboards and alerting.

See the monitoring docs for a list of default metrics emitted by each component type (buffers, sinks, etc.)

Version 0.7.1-alpha

This is a patch release to Data Prepper 0.7.0. In this release we have added the ability to use Data Prepper to send trace data to Amazon Elasticsearch domains.

Version 0.7.0-alpha

2020-16-10 Version 0.7.0-alpha (Current)

This is the first official release of Data Prepper.

Data Prepper is a server side data collector with abilities to filter, enrich, transform, normalize and aggregate data for downstream analytics and visualization. The broader vision for Data Prepper is to enable an end-to-end data analysis life cycle from gathering raw logs to facilitating sophisticated and actionable interactive ad-hoc analyses on the data.
In the first release of Data Prepper, it only supports plugins that are required for Trace Analytics feature. More details check the documentations below.

Documentation

• Overview: Overview of Data Prepper
• Trace Overview: Trace overview