Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

The guide "Incremental models in-depth" is not valid for BigQuery with partitioned tables #4955

Open
1 task done
b-per opened this issue Feb 20, 2024 · 2 comments
Open
1 task done
Labels
content Improvements or additions to content improvement Use this when an area of the docs needs improvement as it's currently unclear

Comments

@b-per
Copy link
Contributor

b-per commented Feb 20, 2024

Contributions

  • I have read the contribution docs, and understand what's expected of me.

Link to the page on docs.getdbt.com requiring updates

https://docs.getdbt.com/best-practices/materializations/4-incremental-models

What part(s) of the page would you like to see updated?

This code

{% if is_incremental() %}

where
  updated_at > (select max(updated_at) from {{ this }})

{% endif %}

won't filter on partitions in BigQuery and can be extremely expensive 🤑 . See docs here on partition pruning in BQ.

For BQ, with partitioned tables, we should calculate the value first with a run_query() and then use it.

Example:

{%- set max_date = dbt_utils.get_single_value("select max(updated_at) from {{ this }}", default="'2020-01-01'") -%}

{% if is_incremental() %}

where
  updated_at > {{ max_date }}

{% endif %}

Additional information

No response

@b-per b-per added content Improvements or additions to content improvement Use this when an area of the docs needs improvement as it's currently unclear labels Feb 20, 2024
@runleonarun
Copy link
Collaborator

Hi @b-per does this feel high priority?

@b-per
Copy link
Contributor Author

b-per commented Feb 22, 2024

Last week only I had chats with 2 different customers who were using incrementals the "wrong way" with BigQuery.

Would they have done differently if the relevant info/callout for BQ was in the guide? I don't know.

So, I don't think that this is the highest priority (this is costing some people money though), but I'd like to have it somewhere at some point so I can send a link to the section to any person with the same problem.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
content Improvements or additions to content improvement Use this when an area of the docs needs improvement as it's currently unclear
Projects
None yet
Development

No branches or pull requests

2 participants