Skip to content

Log4j 2.x plugin for customizable and Logstash-friendly JSON layout.

License

Notifications You must be signed in to change notification settings

vy/log4j2-logstash-layout

Repository files navigation

log4j2-logstash-layout is not maintained anymore! Since Log4j 2.14.0, it is superseded by log4j-layout-template-json shipping JsonTemplateLayout, which is a successor of LogstashLayout. We strongly advise all LogstashLayout users to migrate to JsonTemplateLayout. We will do our best to bring bug fixes to LogstashLayout, but all the new development efforts will be focused on JsonTemplateLayout.

Actions Status Maven Central License

LogstashLayout is the fastest Log4j 2 JSON layout allowing schema customization and Logstash-friendly output.

By default, LogstashLayout ships the official JSONEventLayoutV1 stated by log4j-jsonevent-layout Log4j 1.x plugin. Compared to JSONLayout included in Log4j 2 and log4j-jsonevent-layout, LogstashLayout provides the following additional features:

  • Superior performance
  • Customizable JSON schema (see eventTemplate[Uri] and stackTraceElementTemplate[Uri] parameters)
  • Customizable timestamp formatting (see dateTimeFormatPattern and timeZoneId parameters)

Table of Contents

Usage

Add the log4j2-logstash-layout dependency to your POM file

<dependency>
    <groupId>com.vlkan.log4j2</groupId>
    <artifactId>log4j2-logstash-layout</artifactId>
    <version>${log4j2-logstash-layout.version}</version>
</dependency>

in combination with a valid log4j-core dependency:

<dependency>
    <groupId>org.apache.logging.log4j</groupId>
    <artifactId>log4j-core</artifactId>
    <version>${log4j2.version}</version>
</dependency>

(Note that the Java 9 module name is com.vlkan.log4j2.logstash.layout.)

Below you can find a sample log4j2.xml snippet employing LogstashLayout.

<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="warn">
    <Appenders>
        <Console name="CONSOLE" target="SYSTEM_OUT">
            <LogstashLayout dateTimeFormatPattern="yyyy-MM-dd'T'HH:mm:ss.SSSZZZ"
                            eventTemplateUri="classpath:LogstashJsonEventLayoutV1.json"
                            prettyPrintEnabled="true"
                            stackTraceEnabled="true"/>
        </Console>
    </Appenders>
    <Loggers>
        <Root level="info">
            <AppenderRef ref="CONSOLE"/>
        </Root>
    </Loggers>
</Configuration>

Or using the log4j2.properties file instead:

status = warn

appender.console.name = CONSOLE
appender.console.type = CONSOLE
appender.console.target = SYSTEM_OUT

appender.console.logstash.type = LogstashLayout
appender.console.logstash.dateTimeFormatPattern = yyyy-MM-dd'T'HH:mm:ss.SSSZZZ
appender.console.logstash.eventTemplateUri = classpath:LogstashJsonEventLayoutV1.json
appender.console.logstash.prettyPrintEnabled = true
appender.console.logstash.stackTraceEnabled = true

rootLogger.level = info
rootLogger.appenderRef.stdout.ref = CONSOLE

This generates an output as follows:

{
  "exception": {
    "exception_class": "java.lang.RuntimeException",
    "exception_message": "test",
    "stacktrace": "java.lang.RuntimeException: test\n\tat com.vlkan.log4j2.logstash.layout.demo.LogstashLayoutDemo.main(LogstashLayoutDemo.java:11)\n"
  },
  "line_number": 12,
  "class": "com.vlkan.log4j2.logstash.layout.demo.LogstashLayoutDemo",
  "@version": 1,
  "source_host": "varlik",
  "message": "Hello, error!",
  "thread_name": "main",
  "@timestamp": "2017-05-25T19:56:23.370+02:00",
  "level": "ERROR",
  "file": "LogstashLayoutDemo.java",
  "method": "main",
  "logger_name": "com.vlkan.log4j2.logstash.layout.demo.LogstashLayoutDemo"
}

LogstashLayout is configured with the following parameters:

Parameter Name Type Description
prettyPrintEnabled boolean enables pretty-printer (defaults to false)
locationInfoEnabled boolean includes the filename and line number in the output (defaults to false)
stackTraceEnabled boolean includes stack traces (defaults to false)
emptyPropertyExclusionEnabled boolean exclude empty and null properties (defaults to false)
dateTimeFormatPattern String timestamp formatter pattern (defaults to yyyy-MM-dd'T'HH:mm:ss.SSSZZZ)
timeZoneId String time zone id (defaults to TimeZone.getDefault().getID())
locale String locale in one of the following forms: <language>, <language>_<country>, or <language>_<country>_<variant> (defaults to Locale.getDefault())
mdcKeyPattern String regex to filter MDC keys (does not apply to direct mdc:key access)
ndcPattern String regex to filter NDC items
eventTemplate String inline JSON template for rendering LogEvents (has priority over eventTemplateUri)
eventTemplateUri String JSON template for rendering LogEvents (defaults to classpath:LogstashJsonEventLayoutV1.json)
eventTemplateAdditionalFields1 KeyValuePair additional key-value pairs appended to the root of the event template
stackTraceElementTemplate String inline JSON template for rendering StackTraceElements (has priority over stackTraceElementTemplateUri)
stackTraceElementTemplateUri String JSON template for rendering StackTraceElements (defaults to classpath:Log4j2StackTraceElementLayout.json)
lineSeparator String used to separate log outputs (defaults to System.lineSeparator())
maxByteCount int used to cap the internal byte[] buffer used for serialization (defaults to 16 KiB)
maxStringLength2 int truncate string values longer than the specified limit (defaults to 0)
objectMapperFactoryMethod String custom object mapper factory method (defaults to com.fasterxml.jackson.databind.ObjectMapper.new)
mapMessageFormatterIgnored boolean as a temporary work around for LOG4J2-2703, serialize MapMessages using Jackson rather than MapMessage#getFormattedMessage() (defaults to true)

1 One can configure additional event template fields as follows:

<LogstashLayout ...>
    <EventTemplateAdditionalFields>
        <KeyValuePair key="serviceName" value="auth-service"/>
        <KeyValuePair key="containerId" value="6ede3f0ca7d9"/>
    </EventTemplateAdditionalFields>
</LogstashLayout>

2 Note that string value truncation via maxStringLength can take place both in object keys and values, and this operation does not leave any trace behind. maxStringLength is intended as a soft protection against bogus input and one should always rely on maxByteCount for a hard limit.

eventTemplateUri denotes the URI pointing to the JSON template that will be used while formatting the LogEvents. By default, LogstashLayout ships LogstashJsonEventLayoutV1.json providing the official Logstash JSONEventLayoutV1.

{
  "mdc": "${json:mdc}",
  "ndc": "${json:ndc}",
  "exception": {
    "exception_class": "${json:exception:className}",
    "exception_message": "${json:exception:message}",
    "stacktrace": "${json:exception:stackTrace:text}"
  },
  "line_number": "${json:source:lineNumber}",
  "class": "${json:source:className}",
  "@version": 1,
  "source_host": "${hostName}",
  "message": "${json:message}",
  "thread_name": "${json:thread:name}",
  "@timestamp": "${json:timestamp}",
  "level": "${json:level}",
  "file": "${json:source:fileName}",
  "method": "${json:source:methodName}",
  "logger_name": "${json:logger:name}"
}

Similarly, stackTraceElementUri denotes the URI pointing to the JSON template that will be used while formatting the StackTraceElements. By default, LogstashLayout ships classpath:Log4j2StackTraceElementLayout.json providing an identical stack trace structure produced by Log4j 2 JSONLayout.

{
  "class": "${json:stackTraceElement:className}",
  "method": "${json:stackTraceElement:methodName}",
  "file": "${json:stackTraceElement:fileName}",
  "line": "${json:stackTraceElement:lineNumber}"
}

In case of need, you can create your own templates with a structure tailored to your needs. That is, you can add new fields, remove or rename existing ones, change the structure, etc. Please note that eventTemplateUri parameter only supports file and classpath URI schemes.

Below is the list of allowed LogEvent template variables that will be replaced while rendering the JSON output.

Variable Name Description
endOfBatch logEvent.isEndOfBatch()
exception:className logEvent.getThrown().getClass().getCanonicalName()
exception:message logEvent.getThrown().getMessage()
exception:stackTrace logEvent.getThrown().getStackTrace() (inactive when stackTraceEnabled=false)
exception:stackTrace:text logEvent.getThrown().printStackTrace() (inactive when stackTraceEnabled=false)
exceptionRootCause:className the innermost exception:className in causal chain
exceptionRootCause:message the innermost exception:message in causal chain
exceptionRootCause:stackTrace[:text] the innermost exception:stackTrace[:text] in causal chain
level logEvent.getLevel()
level:severity Syslog severity keyword of logEvent.getLevel()
level:severity:code Syslog severity code of logEvent.getLevel()
logger:fqcn logEvent.getLoggerFqcn()
logger:name logEvent.getLoggerName()
main:<key> performs Main Argument Lookup for the given key
map:<key> performs Map Lookup for the given key
marker:name logEvent.getMarker.getName()
mdc Mapped Diagnostic Context Map<String, String> returned by logEvent.getContextData()
mdc:<key> Mapped Diagnostic Context String associated with key (mdcKeyPattern is discarded)
message logEvent.getFormattedMessage()
message:json if logEvent.getMessage() is of type MultiformatMessage and supports JSON, its read value, otherwise, {"message": <formattedMessage>} object
ndc Nested Diagnostic Context String[] returned by logEvent.getContextStack()
source:className logEvent.getSource().getClassName()
source:fileName logEvent.getSource().getFileName() (inactive when locationInfoEnabled=false)
source:lineNumber logEvent.getSource().getLineNumber() (inactive when locationInfoEnabled=false)
source:methodName logEvent.getSource().getMethodName()
thread:id logEvent.getThreadId()
thread:name logEvent.getThreadName()
thread:priority logEvent.getThreadPriority()
timestamp logEvent.getTimeMillis() formatted using dateTimeFormatPattern and timeZoneId
timestamp:epoch epoch nanoseconds derived from logEvent.getInstant()
timestamp:epoch:divisor=<divisor> epoch nanoseconds derived from logEvent.getInstant() divided by provided divisor (of type double)
timestamp:epoch:divisor=<divisor>,integral epoch nanoseconds derived from logEvent.getInstant() divided by provided divisor (of type double) and casted to long

JSON field lookups are performed using the ${json:<variable-name>} scheme where <variable-name> is defined as <resolver-name>[:<resolver-key>]. Characters following colon (:) are treated as the resolver-key.

Log4j 2 Lookups (e.g., ${java:version}, ${env:USER}, ${date:MM-dd-yyyy}) are supported in templates too. Though note that while ${json:...} template variables are expected to occupy an entire field, that is, "level": "${json:level}", a lookup can be mixed within a regular text: "myCustomField": "Hello, ${env:USER}!".

Similarly, below is the list of allowed StackTraceElement template variables:

Variable Name Description
stackTraceElement:className stackTraceElement.getClassName()
stackTraceElement:methodName stackTraceElement.getMethodName()
stackTraceElement:fileName stackTraceElement.getFileName()
stackTraceElement:lineNumber stackTraceElement.getLineNumber()

As in LogEvent templates, StackTraceElement templates support Log4j 2 lookups too.

See layout-demo directory for a sample application demonstrating the usage of LogstashLayout.

Predefined Templates

log4j2-logstash-layout artifact contains the following predefined templates:

Features

Below is a feature comparison matrix between LogstashLayout and alternatives.

Feature LogstashLayout JsonLayout EcsLayout
Java version 8 73 6
Dependencies Jackson Jackson None
Full schema customization?
Timestamp customization?
(Almost) garbage-free?
Custom typed Message serialization? 4
Custom typed MDC value serialization?
Rendering stack traces as array?
Enabling/Disabling JSON pretty print?
Additional fields?

3 Log4j 2.4 and greater requires Java 7, versions 2.0-alpha1 to 2.3 required Java 6.

4 Only for ObjectMessages and if Jackson is in the classpath.

Fat JAR

Project also contains a log4j2-logstash-layout-fatjar artifact which includes all its transitive dependencies in a separate shaded package (to avoid the JAR Hell) with the exception of log4j-core, that you need to include separately.

This might come handy if you want to use this plugin along with already compiled applications, e.g., Elasticsearch 5.x and 6.x versions, which requires Log4j 2.

Appender Support

log4j2-logstash-layout is all about providing a highly customizable JSON schema for your logs. Though this does not necessarily mean that all of its features are expected to be supported by every appender in the market. For instance, while prettyPrintEnabled=true works fine with log4j2-redis-appender, it should be turned off for Logstash's log4j-json file input type. (See Pretty printing in Logstash issue.) Make sure you configure log4j2-logstash-layout properly in a way that is aligned with your appender of preference.

Performance

The source code contains a benchmark comparing LogstashLayout performance with JsonLayout (shipped by default in Log4j 2) and EcsLayout (shipped by Elastic). There JMH is used to assess the rendering performance of these layouts. In the tests, different LogEvent profiles are employed:

  • full: LogEvent contains MDC, NDC, and an exception.
  • lite: LogEvent has no MDC, NDC, or exception attachment.

To give an idea, we ran the benchmark with the following settings:

  • CPU: Intel i7 2.70GHz (x86-64, confined java process to a single core using taskset -c 0)
  • JVM: OpenJDK 64-Bit, AdoptOpenJDK, build 25.232-b09
    • -XX:+TieredCompilation
    • -Dlog4j2.garbagefreeThreadContextMap=true
    • -Dlog4j2.enableDirectEncoders=true
    • -Dlog4j2.enable.threadlocals=true
    • -Dlog4j2.is.webapp=false
  • OS: Xubuntu 18.04.3 (4.15.0-70-generic, x86-64)
  • LogstashLayout4{Ecs,Json,Gelf}Layout used default settings with the following exceptions:
    • stackTraceEnabled: true
    • maxByteCount: (4096) 4KiB
  • JsonLayout used in two different flavors:
    • DefaultJsonLayout: default settings
    • CustomJsonLayout: default settings with an additional "@version": 1 field (this forces instantiation of a wrapper class to obtain the necessary Jackson view)
  • EcsLayout used with the following configurations:
    • serviceName: benchmark
    • additionalFields: new KeyValuePair[0]
  • GelfLayout used with the following configurations:
    • compressionType: off

The figures for serializing 1,000 LogEvents at each operation are as follows. (See layout-benchmark directory for the full report.)

Benchmark ops/sec5 B/op5
liteLogstashLayout4GelfLayout 1,517,062 ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ (100%) 0.0
liteLogstashLayout4EcsLayout 1,196,255 ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ (79%) 0.0
liteGelfLayout 1,184,922 ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ (78%) 0.0
liteLogstashLayout4JsonLayout 870,012 ▉▉▉▉▉▉▉▉▉▉▉ (57%) 0.0
liteEcsLayout 836,648 ▉▉▉▉▉▉▉▉▉▉▉ (55%) 0.0
liteDefaultJsonLayout 506,985 ▉▉▉▉▉▉▉ (33%) 5,331,680.0
liteCustomJsonLayout 446,243 ▉▉▉▉▉▉ (29%) 5,740,400.0
fullLogstashLayout4JsonLayout 118,294 ▉▉ (8%) 104,000.1
fullLogstashLayout4GelfLayout 73,102 ▉ (5%) 35,663,200.3
fullLogstashLayout4EcsLayout 60,569 ▉ (4%) 35,631,200.4
fullEcsLayout 27,887 ▉ (2%) 46,479,200.5
fullGelfLayout 21,458 ▉ (1%) 58,911,200.7
fullDefaultJsonLayout 13,513 ▉ (1%) 234,102,401.5
fullCustomJsonLayout 13,511 ▉ (1%) 234,238,401.5

5 99th percentile

Let us try to answer some common questions:

  • How come log4j2-logstash-layout can yield superior performance compared to Log4j 2 JsonLayout? Log4j 2 JsonLayout employs a single Jackson view to generate JSON, XML, and YAML outputs. For this purpose, it uses Jackson ObjectMapper, which needs to walk over the class fields via reflection and perform heavy branching and intermediate object instantiation. On the contrary, log4j2-logstash-layout parses the given template once and compiles a (mostly) garbage- and (to a certain extent) branching-free JSON generator employing Jackson JsonGenerator.

  • Why is log4j2-logstash-layout is not totally garbage-free?

    • Since Throwable#getStackTrace() clones the original StackTraceElement[], accesses to (and hence rendering) stack traces can never be garbage-free.

    • Rendering of context data (that is, MDC) field values is garbage-free if the value is either null, or of type String, Short, Integer, Long, or byte[].

  • How can one run the benchmark on his/her machine? After a fresh mvn clean verify within the source directory, run layout-benchmark/benchmark.py.

  • What about thread-local allocations? Even though Log4j 2 exposes a log4j2.enable.threadlocals flag to toggle TLAs, neither EcsLayout, nor Log4j 2 JsonLayout and GelfLayout honor it. Historically, LogstashLayout used to have TLAs taking the log4j2.enable.threadlocals flag into account. In version 0.18, we switched to simple memory-efficient object pools, though that incurred extra synchronization costs. Since version 0.22, LogstashLayout switched back to TLAs taking log4j2.enable.threadlocals into account.

F.A.Q.

  • How can one enable thread-local allocations? For performance reasons, it is highly recommended to turn TLAs on. For this purpose, you need to make sure log4j2.enable.threadlocals=true and log4j2.is.webapp=false.

  • Is there a fat JAR of the plugin? Project also contains a log4j2-logstash-layout-fatjar artifact which includes all its transitive dependencies in a separate shaded package (to avoid the JAR Hell) with the exception of log4j-core, that you need to include separately. Fat JAR might come handy if you want to use this plugin along with certain applications, e.g., Elasticsearch 5.x and 6.x versions, which requires Log4j 2.

  • Why do I get irrelevant stack traces from exception[RootCause]:stackTrace[:text] directives? LogstashLayout uses Throwable#printStackTrace(PrintWriter) and Throwable#getStackTrace() methods to resolve these directives. Under certain circumstances, these methods may throw exception as well. Put another way, trying to access the stack trace of an exception might cause another exception. In such a case, LogstashLayout continues the directive resolution using the new exception and keeps on repeating until it manages to resolve a stack trace to at least provide some insight into the underlying cause.

Contributors

License

Copyright © 2017-2020 Volkan Yazıcı

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.