Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Update databricks-configs.md (insert_overwrite incompatibility) #4198

Merged
merged 4 commits into from
Nov 29, 2023
Merged

Update databricks-configs.md (insert_overwrite incompatibility) #4198

Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
9910639
Update databricks-configs.md
benc-db Oct 5, 2023
fdc8795
Merge branch 'current' into patch-2
matthewshaver Nov 2, 2023
1937313
Update databricks-configs.md
benc-db Nov 29, 2023
8eef376
Merge branch 'current' into patch-2
dataders Nov 29, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions website/docs/reference/resource-configs/databricks-configs.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -100,6 +100,10 @@ insert into table analytics.databricks_incremental

### The `insert_overwrite` strategy

:::caution
This strategy is currently only compatible with All Purpose Clusters, not SQL Warehouses.
:::

This strategy is most effective when specified alongside a `partition_by` clause in your model config. dbt will run an [atomic `insert overwrite` statement](https://spark.apache.org/docs/3.0.0-preview/sql-ref-syntax-dml-insert-overwrite-table.html) that dynamically replaces all partitions included in your query. Be sure to re-select _all_ of the relevant data for a partition when using this incremental strategy.

If no `partition_by` is specified, then the `insert_overwrite` strategy will atomically replace all contents of the table, overriding all existing data with only the new records. The column schema of the table remains the same, however. This can be desirable in some limited circumstances, since it minimizes downtime while the table contents are overwritten. The operation is comparable to running `truncate` + `insert` on other databases. For atomic replacement of Delta-formatted tables, use the `table` materialization (which runs `create or replace`) instead.
Expand Down