Investigation comments:

High level comment for the situation

• We shouldn't compare JSON and LogStash:Json serialization behavior, instead focus on what is the expectation and desired behavior with LogStash:Json. JSON clearly rejects the payload if it is invalid UTF-8. However, it will not be suitable in Logstash ecosystem with number of reasons such as event.original may contain invalid unicode and still claim UTF-8 (will provide tech details below) as it may come from various upstream sources (file, elastic-agent, UDP, TCP, etc...).
• The expected behavior we need with current state for the JSON payloads, is to safely deliver the message to the downstream assuming the unicode.

Tech details

• Logstash internally uses jrjackson library (bases on FasterXML jackson modules) to convert the payload to the JSON format (JrJackson::Base.generate(o, options)). jrjackson expects the byte stream to be shape of UTF-8, generates Ruby#String (which has internal Encoding, defaults to UTF-8) from the given bytes of the input regardless of different encoding of the input.

• Ruby has its own representational layer for the encodings, how-to store the bytes for unicodes. See this blog to understand the evolution of the encoding in the ruby environment. That means, the bytes of A encoding and the bytes of B encoding differ, to support the string operations like String#revere. To support with an example, "\xEFs" in Windows-1252 will be with [239, 115] two bytes, identical to "\u{EF}s" in unicode, which is three bytes in ruby representation. Blindly copying the bytes without taking care the source encoding brings an invalid outcome. This is where ruby introduced Encoding::Converter. When we use the converter, it figures out the representation format (or whichever internal details) and creates a claimed UTF-8 encoding, which will be identical three bytes, equals to "\u{EF}s".

  [13] pry(#<LogStash::Runner>)> 
  [14] pry(#<LogStash::Runner>)> "\u{EF}s".bytes
   => [195, 175, 115]
  [15] pry(#<LogStash::Runner>)> "\xEFs".b.force_encoding(Encoding::WINDOWS_1252).bytes
   => [239, 115]
  [16] pry(#<LogStash::Runner>)> ec = Encoding::Converter.new("WINDOWS-1252", "UTF-8")
   => #<Encoding::Converter: Windows-1252 to UTF-8>
  [17] pry(#<LogStash::Runner>)> ec.convert("\xEFs".b.force_encoding(Encoding::WINDOWS_1252)).bytes
    => [195, 175, 115]
  

• However, as ruby is a strong scripting language which provides alot flexibilities, it may also keep the invalid unicode byte stream in its representational layer. Example, ruby stores invalid UTF-8 "\x8A" and unless we claim it as UTF-8, it doesn't reject but still defaults to UTF-8.

  [20] pry(#<LogStash::Runner>)> "\x8A".encoding
   => #<Encoding:UTF-8>
  [21] pry(#<LogStash::Runner>)> "\x8A".unicode_normalize
   ArgumentError: invalid byte sequence in UTF-8 from org/jruby/RubyString.java:3192:in `gsub'
  

  

To wrap
With this situation in Logstash, we have two areas to improve (both should happen before JrJackson::Base.generate(o, options)):

1. Make sure it considers the source input encoding when generating UTF-8 outputs.
2. Make sure to validate the unicode data before telling the downstreams that is UTF-8, if not as a best practice in current internalization world, use replace character to express the invalid bytes.