Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Custom metadata in data files #106

Open
mccheah opened this issue Nov 16, 2018 · 0 comments
Open

Custom metadata in data files #106

mccheah opened this issue Nov 16, 2018 · 0 comments

Comments

@mccheah
Copy link
Contributor

mccheah commented Nov 16, 2018

It would be useful for consumers of Iceberg tables to be able to specify additional metadata in data files that enable them to know how to read the files. Some examples of custom metadata include:

  • Encryption keys required to read the file,
  • Compression codecs specified on the file without needing to have a specific file extension,
  • Metadata that's specific to a custom file format. Suppose we supported CSV tables in Iceberg down the road. It would be nice to attach the column delimiter on a per-file basis so that a table can be comprised of multiple files that may not necessarily be uniform in terms of the exact layout, but have compatible schemas.

The custom metadata field should be of type Map<String, String> and can be an optional column.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant