-
Notifications
You must be signed in to change notification settings - Fork 944
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
User-defined custom incremental strategies #4716
Conversation
The latest updates on your projects. Learn more about Vercel for Git ↗︎
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
hey @dbeatty10 thanks for opening this up! two small suggestions on my end but are non blockers for approval.
I'm not sure if I'm understanding this correctly: the 2nd suggestion is whether we need to add the built in strategies - this table seems helpful?
|
||
As an easier alternative to [creating an entirely new materialization](/guides/create-new-materializations), users can define and use their own "custom" user-defined incremental strategies by: | ||
|
||
1. defining a macro named `get_incremental_STRATEGY_sql` |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
are we saying the macro should always be named get_incremental_STRATEGY_sql
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
or is STRATEGY
a placeholder? if its a placeholder, we should maybe be more clear on that.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ok gotcha, thank you for clarifying! waht do you think about the following suggestion @dbeatty10 :
Built-in strategies and their corresponding macros
Before diving into custom strategies, it's important to understand the built-in incremental strategies in dbt and their corresponding macros:
incremental_strategy |
Corresponding macro |
---|---|
append |
get_incremental_append_sql |
delete+insert |
get_incremental_delete_insert_sql |
merge |
get_incremental_merge_sql |
insert_overwrite |
get_incremental_insert_overwrite_sql |
For example, a built-in strategy for the append
can be defined and used with the following files:
{% macro get_incremental_append_sql(arg_dict) %}
{% do return(some_custom_macro_with_sql(arg_dict["target_relation"], arg_dict["temp_relation"], arg_dict["unique_key"], arg_dict["dest_columns"], arg_dict["incremental_predicates"])) %}
{% endmacro %}
{% macro some_custom_macro_with_sql(target_relation, temp_relation, unique_key, dest_columns, incremental_predicates) %}
{%- set dest_cols_csv = get_quoted_csv(dest_columns | map(attribute="name")) -%}
insert into {{ target_relation }} ({{ dest_cols_csv }})
(
select {{ dest_cols_csv }}
from {{ temp_relation }}
)
{% endmacro %}
Define a model models/my_model.sql:
{{ config(
materialized="incremental",
incremental_strategy="append",
) }}
select * from {{ ref("some_model") }}
Custom strategies
Custom incremental strategies can be defined beginning in dbt v1.2.
As an easier alternative to creating an entirely new materialization,
users can define and use their own "custom" user-defined incremental strategies by:
- defining a macro named
get_incremental_STRATEGY_sql
. Note thatSTRATEGY
is a placeholder and you should replace it with the name of your custom incremental strategy. - configuring
incremental_strategy: STRATEGY
within an incremental model
dbt won't validate user-defined strategies, it will just look for the macro by that name, and raise an error if it can't find one.
For example,.....
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ok @dbeatty10 proposing the above adn before i make the changes or sugg - please let me know if that makes sense to you 🙏
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This looks good 🚀
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ok cool, will make that change!
Preview
What are you changing in this pull request and why?
This addresses the "For end users" portion of #1761.
The feature request in dbt-labs/dbt-core#5245 describes the value proposition as well as the previous and new behavior:
Functional Requirement
Previous behavior
New behavior
get_incremental_NAME_sql
that produces the correct SQL for the target warehouse.Also
To address the questions raised in dbt-labs/dbt-core#8769, we also want to document how to utilize custom incremental macros that come from a package.
For example, to use the
merge_null_safe
custom incremental strategy from theexample
package, first install the package, then add this macro to your project:🎩
Checklist