Crash on latin1 character

[Glandos]

Welcome to the rude world of character encoding:

~ ❯ echo "é" | iconv -f utf8 -t latin1 | /tmp/tailspin/spin 
Error reading from stdin: stream did not contain valid UTF-8
thread 'main' panicked at 'Could not receive EOF signal from oneshot channel: RecvError(())', /home/compil/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tailspin-1.2.1/src/main.rs:107:10
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

Here, the error message is clear, but I have logs with Latin1 encoded value, and the application crashes. It shouldn't :)
I suspect other encoding not compatible with UTF-8 generate the same crash.

[bensadeh]

Thanks for opening this issue.

How does your terminal handle non-utf8 encodings in general? Have you set any of these env variables: LANG, LC_ALL, LC_CTYPE, LC_COLLATE? And if so, what are they set to?

Are you able to cat or less the log files with no issues?

[Glandos]

/tmp/tailspin ❯ echo "é" | iconv -t latin1 > test.log
/tmp/tailspin ❯ cat test.log 
�
/tmp/tailspin ❯ less test.log 
/tmp/tailspin ❯ xxd test.log 
00000000: e90a                                     ..
/tmp/tailspin ❯ ./spin test.log 
thread 'main' panicked at 'Could not receive EOF signal from oneshot channel: RecvError(())', /home/compil/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tailspin-1.2.1/src/main.rs:107:10
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
/tmp/tailspin [101] ❯ echo $LANG , $LC_ALL , $LC_CTYPE , $LC_COLLATE .
fr_FR.UTF-8 , , , .

You already guessed I'm French, giving the "chosen character" 😉

I hope this test is long enough. I have the same behavior with LANG=C by the way.

[bensadeh]

Ahh, I see now, thanks for explaining it to me in detail. I was afraid to start implementing specific encodings for non-utf8 types at this stage, but I agree that non utf8 encodings should at the very least not crash.

This is indeed a bug, and we should handle foreign characters gracefully. My goal is to have the same behavior as both cat and less, which in this case is "don't crash".

[Glandos]

cat and less do nothing with encoding. cat transliterate with UNKNOWN_CHARACTER and less output an escape sequence with <E9> on a white background.

I guess the best behavior is:

• detect it's not a valid UTF-8 character
• in this case display the escaped byte sequence

[bensadeh]

Since 1.3.0, tailspin gracefully handles different encodings similar to cat when using stdin. So this should resolve the explicit example above.

For reading files on disk, we use a library that provides the actual reading and tailing functionality, and at the moment it crashes on these non-utf-8 encodings. I've opened a ticket in the repo upstream and will implement a fix on our end once a solution is in place.