Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ordered Map #209

Open
tomberek opened this issue Feb 16, 2020 · 1 comment
Open

Ordered Map #209

tomberek opened this issue Feb 16, 2020 · 1 comment

Comments

@tomberek
Copy link

When using rq with avro, there is a re-ordering of keys causing a validation error. A avro file and schema that works perfectly with avro-tools, fails to write with rq. Reading works fine and outputting to JSON or another format is okay, but during write, the schema validation fails because the way Maps are stored internally in rq does not maintain the original order, which is significant for some formats like Avro.

Something like:

cat data.avro | rq -aA avro.avsc
[ERROR] [rq] Encountered: Avro error
[ERROR] [rq] Caused by: validation error
[ERROR] [rq] Caused by: Decoding error: value does not match schema
[ERROR] [rq] (Re-run with --trace or RUST_BACKTRACE=1 for a backtrace)

Manually adjusting the schema to match the resulting order works. There is a TODO here that seems relevant. I attempted to use https://github.com/bluss/indexmap and making the Map a tuple of Vec and BTree hoping to store the order in Vec. I'd like to help (and learn more Rust on the way), but my rust-fu was not good enough to create a working PR.

@jcaesar
Copy link
Contributor

jcaesar commented Feb 28, 2023

Does this persist on current master? If so, do you maybe have a example avro files for testing for me?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants