Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support mutating BigQuery nested types without dropping #2321

Open
plaflamme opened this issue Mar 21, 2024 · 0 comments
Open

Support mutating BigQuery nested types without dropping #2321

plaflamme opened this issue Mar 21, 2024 · 0 comments
Labels
Engine: BigQuery issues related to BigQuery Improvement Improves existing functionality

Comments

@plaflamme
Copy link
Contributor

This is a feature request for the BigQuery engine.

Currently, sqlmesh supports nested types (i.e.: STRUCT) within models, e.g.:

SELECT STRUCT(foo, bar) AS my_struct FROM underlying

It also supports changes to these, e.g.:

SELECT STRUCT(foo, bar, new_field) AS my_struct FROM underlying

But because BigQuery does not support mutating nested types using SQL, the way this is implemented in sqlmesh requires dropping the column and creating a new one (which truncates the data).

The feature request is to fallback on using the BigQuery API directly for doing these mutations. Specifically, it should be possible to add columns to a nested type without dropping it.

The caveat is that adding columns to a nested type can only be done "at the end" (i.e.: STRUCT(foo, new_field, bar) is not possible, only STRUCT(foo, bar, new_field)).

Ideally, sqlmesh would allow appending fields to a STRUCT as long as the query also matches the resulting schema.

@izeigerman izeigerman added Improvement Improves existing functionality Engine: BigQuery issues related to BigQuery labels Mar 21, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Engine: BigQuery issues related to BigQuery Improvement Improves existing functionality
Projects
None yet
Development

No branches or pull requests

2 participants