Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature: compiler can read metadata without resolving values #434

Merged
merged 1 commit into from
May 27, 2024

Conversation

csansoon
Copy link
Contributor

@csansoon csansoon commented May 16, 2024

Describe your changes

Some procedures require extracting metadata from the query, like the defined configuration or knowing which functions are being used, without actually having to resolve the query.

The current way to do this is just compiling the query, which will return some additional metadata along with other compiled output. The problem with this approach is that compiling a query is not always free, as some queries may use runQuery. This function is executed in compile time, since we need the result to be able to resolve other parts of the query. Since we need to get some of the query metadata instantly in some cases, we cannot depend on running queries just to read its configs.

To fix this issue, I have divided the current compilation process in two independent functions: compile and readMetadata:

  • compile resolves all variables, parameters and functions in a query, which are dynamic and dependent on context and may take some time depending on its content.
  • readMetadata in the other hand, returns information that is static in the query, like the defined configuration or the methods that are being called inside of it.

In order to read something as "metadata", it must be static and not depend on other values, parameters or functions resolved during run-time. Since I want the config to be part of the query metadata, I had to add these restrictions to the {@config} tag:

  • Config must be defined using only literal values. This means it cannot use variables or functions.
  • Config tags must be in the root level of the query. This means it cannot be used inside {#if} or {#each} blocks.

Checklist before requesting a review

  • I have added thorough tests
  • I have updated the documentation if necessary
  • I have added a human-readable description of the changes for the release notes
  • I have included a recorded video capture of the feature manually tested

馃殏 TRENECITO.
馃殙 Previous station: #439
馃殙 Next station: #444

@csansoon csansoon added the WIP label May 16, 2024
Copy link

changeset-bot bot commented May 16, 2024

馃 Changeset detected

Latest commit: 40017c3

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 17 packages
Name Type
@latitude-data/source-manager Minor
@latitude-data/sql-compiler Major
@latitude-data/server Patch
@latitude-data/cli Patch
@latitude-data/athena-connector Patch
@latitude-data/bigquery-connector Patch
@latitude-data/clickhouse-connector Patch
@latitude-data/databricks-connector Patch
@latitude-data/duckdb-connector Patch
@latitude-data/materialized-connector Patch
@latitude-data/mssql-connector Patch
@latitude-data/mysql-connector Patch
@latitude-data/postgresql-connector Patch
@latitude-data/snowflake-connector Patch
@latitude-data/sqlite-connector Patch
@latitude-data/test-connector Patch
@latitude-data/trino-connector Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@csansoon csansoon changed the base branch from main to canary May 16, 2024 12:30
@csansoon csansoon force-pushed the feature/fast-config-compiler branch from c9eadc8 to 20e32cb Compare May 16, 2024 16:46
@csansoon csansoon force-pushed the feature/fast-config-compiler branch 7 times, most recently from cd7c7f6 to d2fe6b9 Compare May 17, 2024 15:10
@csansoon csansoon changed the title Feature/fast config compiler Feature: compiler can read metadata without resolving values May 17, 2024
@csansoon csansoon force-pushed the feature/fast-config-compiler branch from d2fe6b9 to e6a3b96 Compare May 17, 2024 15:19
@csansoon csansoon changed the base branch from canary to fix/compiler-can-run-methods-from-scoped-variables-canary May 17, 2024 15:20
@csansoon csansoon force-pushed the feature/fast-config-compiler branch 2 times, most recently from dd54722 to 79e3096 Compare May 17, 2024 15:23
@csansoon csansoon added WIP and removed WIP labels May 17, 2024
@csansoon csansoon force-pushed the feature/fast-config-compiler branch 2 times, most recently from bf001cd to 9f64e4f Compare May 21, 2024 08:48
@csansoon csansoon changed the base branch from fix/compiler-can-run-methods-from-scoped-variables-canary to restructured-compiler-logic May 21, 2024 08:48
@csansoon csansoon force-pushed the feature/fast-config-compiler branch 5 times, most recently from a9391b0 to 5099d40 Compare May 21, 2024 10:58
@csansoon csansoon removed the WIP label May 22, 2024
@csansoon csansoon force-pushed the restructured-compiler-logic branch 2 times, most recently from c8a9d19 to 2eb5531 Compare May 22, 2024 12:07
@csansoon csansoon force-pushed the feature/fast-config-compiler branch from 5099d40 to c238bea Compare May 22, 2024 12:14
@csansoon csansoon force-pushed the restructured-compiler-logic branch 2 times, most recently from cd48840 to f155f00 Compare May 22, 2024 12:17
@csansoon csansoon force-pushed the feature/fast-config-compiler branch from c238bea to 72c954d Compare May 22, 2024 12:29
@csansoon csansoon force-pushed the restructured-compiler-logic branch from f155f00 to 0931006 Compare May 22, 2024 12:30
@csansoon csansoon force-pushed the feature/fast-config-compiler branch 3 times, most recently from dce3aa5 to f055e9f Compare May 22, 2024 12:34
@csansoon csansoon changed the base branch from restructured-compiler-logic to canary May 24, 2024 10:01
@csansoon csansoon force-pushed the feature/fast-config-compiler branch from f055e9f to 40017c3 Compare May 24, 2024 10:01
@csansoon csansoon merged commit 9e2dd26 into canary May 27, 2024
4 checks passed
@github-actions github-actions bot locked and limited conversation to collaborators May 27, 2024
@andresgutgon andresgutgon deleted the feature/fast-config-compiler branch May 27, 2024 08:03
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants