How to get rid of debug log "Detected ES 8.x or above: document type will not be used"

[dwidyna]

(check apply)

• read the contribution guideline

Problem

I was using fluentd with your plugin sending logs to Elasticsearch 5.6 . Now I am upgrading ES to 8.8.
I changed fluent/plugin configuration and mapping templates to get rid of "types" that are no longer present in ES 8.8.
I cannot get rid of this log message "[debug]: #0 [match_es_fluent_all_copy_aws] Detected ES 8.x or above: document type will not be used." (I'm getting same message regarding other tags as well).

Steps to replicate

fluent.conf

<system>
  rpc_endpoint 0.0.0.0:24444
  log_level debug
</system>

<source>
  @type forward
  @id source_forward_generic
  port 24224
</source>

<label @FLUENT_LOG>
  <filter fluent.**>
    @type record_transformer
    @id filter_record_transformer_fluent_all
    renew_record true
    enable_ruby true
    <record>
      timestamp ${time.utc.strftime('%Y-%m-%d %H:%M:%S.%6N')}
      severity ${tag_parts.last}
      message ${record.to_json}
      tag_text ${record.dig("tag")}
      tag_keyword ${record.dig("tag")}
    </record>
  </filter>

  <match>
    @type elasticsearch
    @id match_es_fluent_all_copy_aws

    @include _common_elasticsearch.conf

    #below index name is not used:
    index_name fluentd_logs_fallback

    #index data into yearly rotated index:
    logstash_format true
    logstash_prefix fluentd_logs_v_1_0_0
    logstash_prefix_separator _
    logstash_dateformat %Y

    #take time from record itself but don't index @timestamp field:
    time_key timestamp
    time_key_format "%Y-%m-%d %H:%M:%S.%N"
    time_precision 6
    include_timestamp false
    time_key_exclude_timestamp true

    #load specific index template:
    templates {
      "template_fluentd_logs_v_1_0_0": "/fluentd/etc/es_mappings/template_fluentd_logs.json"
    }
  </match>
</label>

_common_elasticsearch.conf

    #general settings:
    scheme http
    host elasticsearch
    port 9200
    user elasticsearch
    password es_password

    #Managed Elasticsearch service in AWS needs it false (shield case)
    reload_connections true

    validate_client_version true
    verify_es_version_at_startup true
    suppress_type_name true

    max_retry_putting_template 100

    <buffer>
      flush_interval 1s
    </buffer>

/fluentd/etc/es_mappings/template_fluentd_logs.json

{
    "order": 0,
    "index_patterns": [
        "fluentd_logs_v_1_0_0_*"
    ],
    "settings": {
        "index": {
            "number_of_shards": "1",
            "number_of_replicas": "0"
        }
    },
    "mappings": {
        "dynamic": "strict",
        "properties": {
            "timestamp": {
                "format": "yyyy-MM-dd HH:mm:ss.SSSSSS",
                "type": "date"
            },
            "severity": {
                "type": "keyword"
            },
            "tag_keyword": {
                "type": "keyword"
            },
            "tag_text": {
                "type": "text"
            },
            "message": {
                "type": "text"
            }
        }
    },
    "aliases": {}
}

Expected Behavior or What you need to ask

No logs regarding the type as they are polluting logs and I believe no "type" resides in my config.

Using Fluentd and ES plugin versions

• OS version Ubuntu 22 with Docker 24
• Environment: ES 8.8, fluentd 1.16.1 (I'm using docker image fluent/fluentd:v1.16.1-debian-amd64-1.0)
• ES plugin 5.2.5 / 5.3.0
  • paste boot log of fluentd or td-agent
  • paste result of fluent-gem list

async (1.31.0)
async-http (0.60.1)
async-io (1.34.3)
async-pool (0.4.0)
base64 (default: 0.1.1)
benchmark (default: 0.2.0)
bigdecimal (default: 3.1.1)
bundler (default: 2.3.26)
cgi (default: 0.3.6)
concurrent-ruby (1.2.2)
console (1.16.2)
cool.io (1.7.1)
csv (default: 3.2.5)
date (default: 3.2.2)
debug (1.6.3)
delegate (default: 0.2.0)
did_you_mean (default: 1.6.1)
digest (default: 3.1.0)
drb (default: 2.1.0)
elastic-transport (8.2.1)
elasticsearch (8.8.0)
elasticsearch-api (8.8.0)
english (default: 0.7.1)
erb (default: 2.2.3)
error_highlight (default: 0.3.0)
etc (default: 1.3.0)
excon (0.100.0)
faraday (2.7.8)
faraday-excon (2.1.0)
faraday-net_http (3.0.2)
fcntl (default: 1.0.1)
fiber-local (1.0.0)
fiddle (default: 1.1.0)
fileutils (default: 1.6.0)
find (default: 0.1.1)
fluent-config-regexp-type (1.0.0)
fluent-plugin-elasticsearch (5.3.0)
fluent-plugin-rewrite-tag-filter (2.4.0)
fluentd (1.16.1)
forwardable (default: 1.3.2)
getoptlong (default: 0.1.1)
http_parser.rb (0.8.0)
io-console (default: 0.5.11)
io-nonblock (default: 0.1.0)
io-wait (default: 0.2.1)
ipaddr (default: 1.2.4)
irb (default: 1.4.1)
json (2.6.3, default: 2.6.1)
logger (default: 1.5.0)
matrix (0.4.2)
minitest (5.15.0)
msgpack (1.7.1, 1.7.0)
multi_json (1.15.0)
mutex_m (default: 0.1.1)
net-ftp (0.1.3)
net-http (default: 0.3.0)
net-imap (0.2.3)
net-pop (0.1.1)
net-protocol (default: 0.1.2)
net-smtp (0.3.1)
nio4r (2.5.9)
nkf (default: 0.1.1)
observer (default: 0.1.1)
oj (3.14.3)
open-uri (default: 0.2.0)
open3 (default: 0.1.1)
openssl (default: 3.0.1)
optparse (default: 0.2.0)
ostruct (default: 0.5.2)
pathname (default: 0.2.0)
power_assert (2.0.1)
pp (default: 0.3.0)
prettyprint (default: 0.1.1)
prime (0.1.2)
protocol-hpack (1.4.2)
protocol-http (0.24.1)
protocol-http1 (0.15.0)
protocol-http2 (0.15.1)
pstore (default: 0.1.1)
psych (default: 4.0.4)
racc (default: 1.6.0)
rake (13.0.6)
rbs (2.7.0)
rdoc (default: 6.4.0)
readline (default: 0.0.3)
readline-ext (default: 0.1.4)
reline (default: 0.3.1)
resolv (default: 0.2.1)
resolv-replace (default: 0.1.0)
rexml (3.2.5)
rinda (default: 0.1.1)
rss (0.2.9)
ruby2_keywords (default: 0.0.5)
securerandom (default: 0.2.0)
serverengine (2.3.2)
set (default: 1.0.2)
shellwords (default: 0.1.0)
sigdump (0.2.4)
singleton (default: 0.1.1)
stringio (default: 3.0.1)
strptime (0.2.5)
strscan (default: 3.0.1)
syslog (default: 0.1.0)
tempfile (default: 0.1.2)
test-unit (3.5.3)
time (default: 0.2.2)
timeout (default: 0.2.0)
timers (4.3.5)
tmpdir (default: 0.1.2)
traces (0.9.1)
tsort (default: 0.1.0)
typeprof (0.21.3)
tzinfo (2.0.6)
tzinfo-data (1.2023.3)
un (default: 0.2.0)
uri (default: 0.12.1)
weakref (default: 0.1.1)
webrick (1.8.1)
yajl-ruby (1.4.3)
yaml (default: 0.2.0)
zlib (default: 2.1.1)

• ES version 8.8.1 (docker docker.elastic.co/elasticsearch/elasticsearch:8.8.1-amd64)
• ES template(s) pasted above

[dwidyna]

Hello guys, any chance for solution ?

[psandeep09]

Hello, any update

[Floppe]

Some months later, any solution?

[prateekshetty-eox]

Any updates here?