Using MultiGzDecoder for file with garbage after gzip data

[jinschoi]

I have a gzip json file that I did not create that I am using flate2 and serde_json to parse and transform. When I run my code over the unzipped file, everything is file. When I run it on the gzipped file, it throws an unexpected end of file error. I am trying to figure out what is going on.

My working assumption is that the file, which is a multi-member gzip file, has some extra garbage after the end of the last member; and indeed, when I use gzcat to uncompress it, it does say "trailing garbage ignored". The section about multi-member files in the introduction says "If a file contains contains non-gzip data after the gzip data, MultiGzDecoder will emit an error after decoding the gzip data. This behavior matches the gzip, gunzip, and zcat command line tools."

I would like some way of decoding such a file without an error being returned, and just having any trailing garbage be ignored. What would be the simplest way of doing this?

[Byron]

Thanks for bringing this up! Could you provide a testfile along with a test-program that shows the current behaviour?
My assumption is that it's not actually possible to decode the last member of such a file as it emits an error on the last read, but it would be great to validate. Maybe it will also decode the last member successfully, but fail on the next read as it can't decode garbage, which would mean all zipped data could be read with the current API, and it's really a question on how to indicate a 'garbage tail' occurred.

With such example, it would be possible to setup a test-case and eventually find a solution.

[jinschoi]

Sure. Here is a test file I created with the following commands:

echo "Hello, world" | gzip > hello.gz
echo x >> hello.gz

hello.gz

Running gzcat on the file:

% gzcat hello.gz
Hello, world
gzcat: hello.gz: trailing garbage ignored

A short program that demonstrates the issue:

use flate2::read::MultiGzDecoder;
use std::fs::File;
use std::io::{BufReader, Read, Write};

fn main() {
    let file = File::open("hello.gz").unwrap();
    let mut buf_reader = BufReader::new(MultiGzDecoder::new(file));
    let mut buf = [0_u8; 100];
    loop {
        match buf_reader.read(&mut buf) {
            Ok(0) => break,
            Ok(n) => {
                println!("read {} bytes:", n);
                std::io::stdout().write_all(&buf[0..n]).unwrap();
                std::io::stdout().flush().unwrap();
            }
            Err(e) => {
                println!("{:?}", e);
                break;
            }
        }
    }
}

And the output:

read 13 bytes:
Hello, world
Kind(UnexpectedEof)

[Byron]

Thanks a lot for your help!

Thoughts

It looks like the question here is of UnexpectedEOF could also be another kind of error which would indicate more clearly that indeed, there was nothing to read, or that the stream it tried to read (x here) isn't actually a deflated stream.

Another guess is that if the trailing bytes are longer, the error message will be a different, maybe more specific one. After all, the magic signature seems to be 2 bytes long, so we'd run out of data before reading this even.

Next Steps

I think it would be worth setting up a couple of test cases with varying length of trailing garbage to get an idea of how it can be detected on application level right now.

From there, we might be able to figure out how to optionally adjust or enhance the API in a backward compatible manner to make detecting this situation easier.