dbt-materialize: support generate_cluster_name

[chaas]

Outcome: support generate_cluster_name so users can prefix a cluster name per dbt target. E.g. so that when running in dev, the dev_ cluster is used and in prod, the prod_ cluster is used.

Context - slack thread:

Introduce a macro called generate_cluster_name, akin to generate_schema_name, that was invoked on every model’s cluster with the following default definition:

{% macro generate_cluster_name(custom_cluster_name, node) -%}

    {%- set default_cluster = target.cluster -%}
    {%- if custom_cluster_name is none -%}

        {{ default_cluster }}

    {%- else -%}

        {{ custom_cluster_name }}

    {%- endif -%}

{%- endmacro %}

We should automatically call generate_cluster_name(model.cluster) on each model’s cluster to determine the true cluster name to use.
The user can plug in their own version of the macro, like one that automatically prefixes each cluster with the target name:

{% macro generate_cluster_name(custom_cluster_name, node) -%}

    {%- set default_cluster = target.cluster -%}
    {%- if custom_cluster_name is none -%}

        {{ default_cluster }}

    {%- else -%}

        {{target.name}}_{{ custom_cluster_name }}

    {%- endif -%}

{%- endmacro %}

[sjwiesman]

Just as an FYI, we already have a macro called generate_cluster_name that is invoked in all the places described in this ticket.

https://github.com/MaterializeInc/materialize/blob/main/misc/dbt-materialize/dbt/include/materialize/macros/utils/generate_cluster_name.sql

[chaas]

Just as an FYI, we already have a macro called generate_cluster_name that is invoked in all the places described in this ticket.

Ah ok thanks, Seth, so then we should just update the macro to support this new option?

[bobbyiliev]

I was just looking into doing this after the dbt adapter migration to v1.8.0 is done.

Just to make sure that I have the full picture here, as we already have the generate_cluster_name macro that is invoked in all of the required places, there is technically no work on the adapter side that has to be done as users should already be able to override the macro and extend it in their own projects, eg. they could use something like this:

{% macro generate_cluster_name(custom_cluster_name) -%}
    {%- set default_cluster = target.cluster -%}
    {%- set deploy_suffix = "_dbt_deploy" if var('deploy', False) else "" -%}

    {%- if custom_cluster_name is none -%}
        {{ default_cluster }}{{ deploy_suffix }}
    {%- else -%}
        {{ target.name }}_{{ custom_cluster_name | trim }}{{ deploy_suffix }}
    {%- endif -%}
{%- endmacro %}

That way users who want to override the logic, will be able to do that, but also users who prefer the default behavior (eg. just using the default cluster name without prefixing it with the target name) would still have that.

Or do we prefer to actually extend the macro? Maybe introduce another var that users could pass to prefix cluster names?

[sjwiesman]

Because of the away dependency resolution works, I think you'll need to update the callers to use the adapter dispatch method ¹.

And then more broadly, there's a question of if the deploy suffix handling should happen inside this macro or not. If users override it, they will break blue/green if they don't copy the suffix handling exactly.

Maybe what we should do it create an internal macro that handles deploy suffixes and calls generate_cluster_name.

{% macro generate_cluster_name_internal(custom_cluster_name) -%}
      {%- set cluster_name = adapter.dispatch('generate_cluster_name')(custom_cluster_name) -%}
      {%- set deploy_suffix = "_dbt_deploy" if var('deploy', False) else "" -%}
      {{ cluster_name }}{{ deploy_suffix }}
{%- endmacro %}

Footnotes

1. https://docs.getdbt.com/reference/dbt-jinja-functions/dispatch ↩

[bobbyiliev]

Thanks @sjwiesman this is very helpful!

I started putting together a PR for this.