User-defined custom incremental strategies

[dbeatty10]

Preview

What are you changing in this pull request and why?

This addresses the "For end users" portion of #1761.

The feature request in dbt-labs/dbt-core#5245 describes the value proposition as well as the previous and new behavior:

Functional Requirement

• Advanced users that wish to specify a custom incremental strategy must be able to do so.

Previous behavior

• Advanced dbt users who wished to specify a custom incremental strategy must override the same boilerplate Jinja macro by copy pasting it into their dbt project.

New behavior

• Advanced dbt users who wish to specify a custom incremental strategy will only need to create a macro that conforms to the naming convention get_incremental_NAME_sql that produces the correct SQL for the target warehouse.

Also

To address the questions raised in dbt-labs/dbt-core#8769, we also want to document how to utilize custom incremental macros that come from a package.

For example, to use the merge_null_safe custom incremental strategy from the example package, first install the package, then add this macro to your project:

{% macro get_incremental_merge_null_safe_sql(arg_dict) %}
    {% do return(example.get_incremental_merge_null_safe_sql(arg_dict)) %}
{% endmacro %}

🎩

Checklist

• Review the Content style guide so my content adheres to these guidelines.
• For docs versioning, review how to version a whole page and version a block of content.

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
docs-getdbt-com	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Jan 8, 2024 3:58pm

[mirnawong1]

hey @dbeatty10 thanks for opening this up! two small suggestions on my end but are non blockers for approval.

I'm not sure if I'm understanding this correctly: the 2nd suggestion is whether we need to add the built in strategies - this table seems helpful?

[mirnawong1]

are we saying the macro should always be named get_incremental_STRATEGY_sql?

[mirnawong1]

or is STRATEGY a placeholder? if its a placeholder, we should maybe be more clear on that.

[dbeatty10]

Yep, STRATEGY is a placeholder!

Any ideas how to make it more clear?

Along the lines of your suggestion about the table, you think adding adding it would make it more clear?

[mirnawong1]

ok gotcha, thank you for clarifying! waht do you think about the following suggestion @dbeatty10 :

Built-in strategies and their corresponding macros

Before diving into custom strategies, it's important to understand the built-in incremental strategies in dbt and their corresponding macros:

incremental_strategy	Corresponding macro
append	get_incremental_append_sql
delete+insert	get_incremental_delete_insert_sql
merge	get_incremental_merge_sql
insert_overwrite	get_incremental_insert_overwrite_sql

For example, a built-in strategy for the append can be defined and used with the following files:

{% macro get_incremental_append_sql(arg_dict) %}

  {% do return(some_custom_macro_with_sql(arg_dict["target_relation"], arg_dict["temp_relation"], arg_dict["unique_key"], arg_dict["dest_columns"], arg_dict["incremental_predicates"])) %}

{% endmacro %}


{% macro some_custom_macro_with_sql(target_relation, temp_relation, unique_key, dest_columns, incremental_predicates) %}

    {%- set dest_cols_csv = get_quoted_csv(dest_columns | map(attribute="name")) -%}

    insert into {{ target_relation }} ({{ dest_cols_csv }})
    (
        select {{ dest_cols_csv }}
        from {{ temp_relation }}
    )

{% endmacro %}

Define a model models/my_model.sql:

{{ config(
    materialized="incremental",
    incremental_strategy="append",
) }}

select * from {{ ref("some_model") }}

Custom strategies

Custom incremental strategies can be defined beginning in dbt v1.2.

As an easier alternative to creating an entirely new materialization,
users can define and use their own "custom" user-defined incremental strategies by:

1. defining a macro named get_incremental_STRATEGY_sql. Note that STRATEGY is a placeholder and you should replace it with the name of your custom incremental strategy.
2. configuring incremental_strategy: STRATEGY within an incremental model

dbt won't validate user-defined strategies, it will just look for the macro by that name, and raise an error if it can't find one.

For example,.....

[mirnawong1]

ok @dbeatty10 proposing the above adn before i make the changes or sugg - please let me know if that makes sense to you 🙏

[dbeatty10]

This looks good 🚀

[mirnawong1]

ok cool, will make that change!