The guide "Incremental models in-depth" is not valid for BigQuery with partitioned tables

[b-per]

Contributions

• I have read the contribution docs, and understand what's expected of me.

Link to the page on docs.getdbt.com requiring updates

https://docs.getdbt.com/best-practices/materializations/4-incremental-models

What part(s) of the page would you like to see updated?

This code

{% if is_incremental() %}

where
  updated_at > (select max(updated_at) from {{ this }})

{% endif %}

won't filter on partitions in BigQuery and can be extremely expensive 🤑 . See docs here on partition pruning in BQ.

For BQ, with partitioned tables, we should calculate the value first with a run_query() and then use it.

Example:

{%- set max_date = dbt_utils.get_single_value("select max(updated_at) from {{ this }}", default="'2020-01-01'") -%}

{% if is_incremental() %}

where
  updated_at > {{ max_date }}

{% endif %}

Additional information

No response

[runleonarun]

Hi @b-per does this feel high priority?

[b-per]

Last week only I had chats with 2 different customers who were using incrementals the "wrong way" with BigQuery.

Would they have done differently if the relevant info/callout for BQ was in the guide? I don't know.

So, I don't think that this is the highest priority (this is costing some people money though), but I'd like to have it somewhere at some point so I can send a link to the section to any person with the same problem.