From 50998fe803b6c48f46dab4a8cc930d8c33a41b01 Mon Sep 17 00:00:00 2001 From: Natalie Fiann Date: Fri, 30 Aug 2024 11:41:12 +0100 Subject: [PATCH 01/74] Updated constraints doc following slack --- .../resource-properties/constraints.md | 61 +++++++++++++++++++ 1 file changed, 61 insertions(+) diff --git a/website/docs/reference/resource-properties/constraints.md b/website/docs/reference/resource-properties/constraints.md index b8111ef0adb..74078d7541f 100644 --- a/website/docs/reference/resource-properties/constraints.md +++ b/website/docs/reference/resource-properties/constraints.md @@ -303,6 +303,67 @@ select +### Custom constraints on models for advanced configuration of tables + +In dbt Cloud, you can use custom constraints on models for advanced configuration of tables. Custom constraints allow you to set [masking policies](https://docs.snowflake.com/en/user-guide/security-column-intro#what-are-masking-policies) in Snowflake when using a Create Table As Select (CTAS). + +Masking policies conceal sensitive data from unauthorized access, while still allowing authorized users to retrieve the data during query execution. + +Contracts and constraints support tag-based masking policies using the following syntax: + + + +```shell + +models: + +- name: my_model +config: +contract: {enforced: true} +materialized: table +columns: + - name: id + data_type: int + constraints: + - type: custom + expression: "tag (my_tag = 'my_value')" # A custom SQL expression used to enforce a specific constraint on a column. + +``` + + + +Using this syntax requires defining all the columns and their types as it’s the only way where to send a create or replace ` mytable as ...`. It’s not possible to do it with just a partial list of columns. + +To generate a YAML with all the columns, you could also use `generate_model_yaml` from [dbt-codegen](https://github.com/dbt-labs/dbt-codegen/tree/0.12.1/?tab=readme-ov-file#generate_model_yaml-source). + +Alternatively, you can add a masking policy without tags: + + + +```shell + +models: + - name: my_model + config: + contract: {enforced: true} + materialized: table + columns: + - name: id + data_type: int + constraints: + - type: custom + expression: "masking policy my_policy" + +``` + + + +:::info + +Each data warehouse has its own set of parameters that can be set for columns in their CTAS statements. For example for [DBX](https://docs.databricks.com/en/sql/language-manual/sql-ref-syntax-ddl-create-table-using.html) for [BigQuery](https://cloud.google.com/bigquery/docs/reference/standard-sql/data-definition-language#column_name_and_column_schema). + +::: +
From dc70f56591115867f24672ef262d98e30e93558a Mon Sep 17 00:00:00 2001 From: nataliefiann <120089939+nataliefiann@users.noreply.github.com> Date: Fri, 30 Aug 2024 13:14:12 +0100 Subject: [PATCH 02/74] Update website/docs/reference/resource-properties/constraints.md --- website/docs/reference/resource-properties/constraints.md | 1 - 1 file changed, 1 deletion(-) diff --git a/website/docs/reference/resource-properties/constraints.md b/website/docs/reference/resource-properties/constraints.md index 74078d7541f..7d1e5d0a9e5 100644 --- a/website/docs/reference/resource-properties/constraints.md +++ b/website/docs/reference/resource-properties/constraints.md @@ -307,7 +307,6 @@ select In dbt Cloud, you can use custom constraints on models for advanced configuration of tables. Custom constraints allow you to set [masking policies](https://docs.snowflake.com/en/user-guide/security-column-intro#what-are-masking-policies) in Snowflake when using a Create Table As Select (CTAS). -Masking policies conceal sensitive data from unauthorized access, while still allowing authorized users to retrieve the data during query execution. Contracts and constraints support tag-based masking policies using the following syntax: From 1732a00296018a2c42410ed049f306fe0b906722 Mon Sep 17 00:00:00 2001 From: nataliefiann <120089939+nataliefiann@users.noreply.github.com> Date: Fri, 30 Aug 2024 13:14:28 +0100 Subject: [PATCH 03/74] Update website/docs/reference/resource-properties/constraints.md Co-authored-by: Mirna Wong <89008547+mirnawong1@users.noreply.github.com> --- website/docs/reference/resource-properties/constraints.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/reference/resource-properties/constraints.md b/website/docs/reference/resource-properties/constraints.md index 7d1e5d0a9e5..f3388febc43 100644 --- a/website/docs/reference/resource-properties/constraints.md +++ b/website/docs/reference/resource-properties/constraints.md @@ -312,7 +312,7 @@ Contracts and constraints support tag-based masking policies using the following -```shell +```yaml models: From 9368790e51b9f8fde212a574e3298887b2da9954 Mon Sep 17 00:00:00 2001 From: nataliefiann <120089939+nataliefiann@users.noreply.github.com> Date: Fri, 30 Aug 2024 14:40:01 +0100 Subject: [PATCH 04/74] Update website/docs/reference/resource-properties/constraints.md --- website/docs/reference/resource-properties/constraints.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/reference/resource-properties/constraints.md b/website/docs/reference/resource-properties/constraints.md index f3388febc43..8129535eb72 100644 --- a/website/docs/reference/resource-properties/constraints.md +++ b/website/docs/reference/resource-properties/constraints.md @@ -331,7 +331,7 @@ columns: -Using this syntax requires defining all the columns and their types as it’s the only way where to send a create or replace ` mytable as ...`. It’s not possible to do it with just a partial list of columns. +Using this syntax requires configuring all the columns and their types as it’s the only way where to send a create or replace ` mytable as ...`. It’s not possible to do it with just a partial list of columns. This means making sure the columns and constraints fields are fully defined. To generate a YAML with all the columns, you could also use `generate_model_yaml` from [dbt-codegen](https://github.com/dbt-labs/dbt-codegen/tree/0.12.1/?tab=readme-ov-file#generate_model_yaml-source). From 3ac1d4d2822ac63474dc131426d60dc331983a0c Mon Sep 17 00:00:00 2001 From: nataliefiann <120089939+nataliefiann@users.noreply.github.com> Date: Fri, 30 Aug 2024 14:42:57 +0100 Subject: [PATCH 05/74] Update website/docs/reference/resource-properties/constraints.md --- website/docs/reference/resource-properties/constraints.md | 1 - 1 file changed, 1 deletion(-) diff --git a/website/docs/reference/resource-properties/constraints.md b/website/docs/reference/resource-properties/constraints.md index 8129535eb72..956d03dfa72 100644 --- a/website/docs/reference/resource-properties/constraints.md +++ b/website/docs/reference/resource-properties/constraints.md @@ -357,7 +357,6 @@ models: -:::info Each data warehouse has its own set of parameters that can be set for columns in their CTAS statements. For example for [DBX](https://docs.databricks.com/en/sql/language-manual/sql-ref-syntax-ddl-create-table-using.html) for [BigQuery](https://cloud.google.com/bigquery/docs/reference/standard-sql/data-definition-language#column_name_and_column_schema). From 70460bbfd1a712c0387a236f1d7fe67a3a3a07c2 Mon Sep 17 00:00:00 2001 From: nataliefiann <120089939+nataliefiann@users.noreply.github.com> Date: Fri, 30 Aug 2024 14:43:40 +0100 Subject: [PATCH 06/74] Update website/docs/reference/resource-properties/constraints.md --- website/docs/reference/resource-properties/constraints.md | 1 - 1 file changed, 1 deletion(-) diff --git a/website/docs/reference/resource-properties/constraints.md b/website/docs/reference/resource-properties/constraints.md index 956d03dfa72..b181340f337 100644 --- a/website/docs/reference/resource-properties/constraints.md +++ b/website/docs/reference/resource-properties/constraints.md @@ -358,7 +358,6 @@ models: -Each data warehouse has its own set of parameters that can be set for columns in their CTAS statements. For example for [DBX](https://docs.databricks.com/en/sql/language-manual/sql-ref-syntax-ddl-create-table-using.html) for [BigQuery](https://cloud.google.com/bigquery/docs/reference/standard-sql/data-definition-language#column_name_and_column_schema). ::: From 3ea40bd01a7cded1f7df68734c020f98ab197685 Mon Sep 17 00:00:00 2001 From: nataliefiann <120089939+nataliefiann@users.noreply.github.com> Date: Fri, 30 Aug 2024 14:47:46 +0100 Subject: [PATCH 07/74] Update website/docs/reference/resource-properties/constraints.md --- website/docs/reference/resource-properties/constraints.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/website/docs/reference/resource-properties/constraints.md b/website/docs/reference/resource-properties/constraints.md index b181340f337..f4694c1875b 100644 --- a/website/docs/reference/resource-properties/constraints.md +++ b/website/docs/reference/resource-properties/constraints.md @@ -305,7 +305,9 @@ select ### Custom constraints on models for advanced configuration of tables -In dbt Cloud, you can use custom constraints on models for advanced configuration of tables. Custom constraints allow you to set [masking policies](https://docs.snowflake.com/en/user-guide/security-column-intro#what-are-masking-policies) in Snowflake when using a Create Table As Select (CTAS). +In dbt Cloud, you can use custom constraints on models for advanced configuration of tables. For example, custom constraints allow you to set [masking policies](https://docs.snowflake.com/en/user-guide/security-column-intro#what-are-masking-policies) in Snowflake when using a Create Table As Select (CTAS). + +Each data warehouse has its own set of parameters that can be set for columns in their CTAS statements. For example [Databricks](https://docs.databricks.com/en/sql/language-manual/sql-ref-syntax-ddl-create-table-using.html) and [BigQuery](https://cloud.google.com/bigquery/docs/reference/standard-sql/data-definition-language#column_name_and_column_schema)". Contracts and constraints support tag-based masking policies using the following syntax: From fa43fff6590f3ac861dcdeafc1e3973980865c49 Mon Sep 17 00:00:00 2001 From: nataliefiann <120089939+nataliefiann@users.noreply.github.com> Date: Fri, 30 Aug 2024 16:07:58 +0100 Subject: [PATCH 08/74] Update website/docs/reference/resource-properties/constraints.md --- website/docs/reference/resource-properties/constraints.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/reference/resource-properties/constraints.md b/website/docs/reference/resource-properties/constraints.md index f4694c1875b..5af9c90528f 100644 --- a/website/docs/reference/resource-properties/constraints.md +++ b/website/docs/reference/resource-properties/constraints.md @@ -320,7 +320,7 @@ models: - name: my_model config: -contract: {enforced: true} + contract: {enforced: true} materialized: table columns: - name: id From 4a176427ed7df522a7c9cb751231f74cc350705a Mon Sep 17 00:00:00 2001 From: nataliefiann <120089939+nataliefiann@users.noreply.github.com> Date: Fri, 30 Aug 2024 16:08:25 +0100 Subject: [PATCH 09/74] Update website/docs/reference/resource-properties/constraints.md --- website/docs/reference/resource-properties/constraints.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/reference/resource-properties/constraints.md b/website/docs/reference/resource-properties/constraints.md index 5af9c90528f..f6098f08a2c 100644 --- a/website/docs/reference/resource-properties/constraints.md +++ b/website/docs/reference/resource-properties/constraints.md @@ -321,7 +321,7 @@ models: - name: my_model config: contract: {enforced: true} -materialized: table + materialized: table columns: - name: id data_type: int From 0d834a73ef0085ee42e681a68b3e1e6e61a5d89b Mon Sep 17 00:00:00 2001 From: nataliefiann <120089939+nataliefiann@users.noreply.github.com> Date: Fri, 30 Aug 2024 16:10:28 +0100 Subject: [PATCH 10/74] Update website/docs/reference/resource-properties/constraints.md --- website/docs/reference/resource-properties/constraints.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/reference/resource-properties/constraints.md b/website/docs/reference/resource-properties/constraints.md index f6098f08a2c..5359327008d 100644 --- a/website/docs/reference/resource-properties/constraints.md +++ b/website/docs/reference/resource-properties/constraints.md @@ -322,7 +322,7 @@ models: config: contract: {enforced: true} materialized: table -columns: + columns: - name: id data_type: int constraints: From a8e26c4bab36051dfb4779a98d704779a2b5616d Mon Sep 17 00:00:00 2001 From: nataliefiann <120089939+nataliefiann@users.noreply.github.com> Date: Fri, 30 Aug 2024 16:12:23 +0100 Subject: [PATCH 11/74] Update website/docs/reference/resource-properties/constraints.md --- website/docs/reference/resource-properties/constraints.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/reference/resource-properties/constraints.md b/website/docs/reference/resource-properties/constraints.md index 5359327008d..082eae9f18d 100644 --- a/website/docs/reference/resource-properties/constraints.md +++ b/website/docs/reference/resource-properties/constraints.md @@ -324,7 +324,7 @@ config: materialized: table columns: - name: id - data_type: int + data_type: int constraints: - type: custom expression: "tag (my_tag = 'my_value')" # A custom SQL expression used to enforce a specific constraint on a column. From e6fe96b5b809552c32328a6cb5cca75bfdf0c6aa Mon Sep 17 00:00:00 2001 From: nataliefiann <120089939+nataliefiann@users.noreply.github.com> Date: Fri, 30 Aug 2024 16:14:11 +0100 Subject: [PATCH 12/74] Update website/docs/reference/resource-properties/constraints.md --- website/docs/reference/resource-properties/constraints.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/reference/resource-properties/constraints.md b/website/docs/reference/resource-properties/constraints.md index 082eae9f18d..1e56862fcec 100644 --- a/website/docs/reference/resource-properties/constraints.md +++ b/website/docs/reference/resource-properties/constraints.md @@ -325,7 +325,7 @@ config: columns: - name: id data_type: int - constraints: + constraints: - type: custom expression: "tag (my_tag = 'my_value')" # A custom SQL expression used to enforce a specific constraint on a column. From cdf0aded955f87f324091608a10d0eda4ff559d4 Mon Sep 17 00:00:00 2001 From: nataliefiann <120089939+nataliefiann@users.noreply.github.com> Date: Fri, 30 Aug 2024 16:14:37 +0100 Subject: [PATCH 13/74] Update website/docs/reference/resource-properties/constraints.md --- website/docs/reference/resource-properties/constraints.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/reference/resource-properties/constraints.md b/website/docs/reference/resource-properties/constraints.md index 1e56862fcec..7d8895ad7d6 100644 --- a/website/docs/reference/resource-properties/constraints.md +++ b/website/docs/reference/resource-properties/constraints.md @@ -327,7 +327,7 @@ config: data_type: int constraints: - type: custom - expression: "tag (my_tag = 'my_value')" # A custom SQL expression used to enforce a specific constraint on a column. + expression: "tag (my_tag = 'my_value')" # A custom SQL expression used to enforce a specific constraint on a column. ``` From 9da9e5d34429a040ab68ec0ec7758199717eabe3 Mon Sep 17 00:00:00 2001 From: nataliefiann <120089939+nataliefiann@users.noreply.github.com> Date: Fri, 30 Aug 2024 16:22:41 +0100 Subject: [PATCH 14/74] Update website/docs/reference/resource-properties/constraints.md --- website/docs/reference/resource-properties/constraints.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/reference/resource-properties/constraints.md b/website/docs/reference/resource-properties/constraints.md index 7d8895ad7d6..6f06d848fd8 100644 --- a/website/docs/reference/resource-properties/constraints.md +++ b/website/docs/reference/resource-properties/constraints.md @@ -310,7 +310,7 @@ In dbt Cloud, you can use custom constraints on models for advanced configuratio Each data warehouse has its own set of parameters that can be set for columns in their CTAS statements. For example [Databricks](https://docs.databricks.com/en/sql/language-manual/sql-ref-syntax-ddl-create-table-using.html) and [BigQuery](https://cloud.google.com/bigquery/docs/reference/standard-sql/data-definition-language#column_name_and_column_schema)". -Contracts and constraints support tag-based masking policies using the following syntax: +Here are a few examples of how to implement tag-based masking policies with contracts and constraints using the following syntax: From 85abbd2e92bac540dbeb6ad94ce384c9c9ced458 Mon Sep 17 00:00:00 2001 From: nataliefiann <120089939+nataliefiann@users.noreply.github.com> Date: Fri, 30 Aug 2024 16:29:48 +0100 Subject: [PATCH 15/74] Update website/docs/reference/resource-properties/constraints.md --- website/docs/reference/resource-properties/constraints.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/reference/resource-properties/constraints.md b/website/docs/reference/resource-properties/constraints.md index 6f06d848fd8..aaba6a3c851 100644 --- a/website/docs/reference/resource-properties/constraints.md +++ b/website/docs/reference/resource-properties/constraints.md @@ -341,7 +341,7 @@ Alternatively, you can add a masking policy without tags: -```shell +```yaml models: - name: my_model From 03fc2a2f704a4617a0044006a1e34c3fca72cc78 Mon Sep 17 00:00:00 2001 From: nataliefiann <120089939+nataliefiann@users.noreply.github.com> Date: Fri, 30 Aug 2024 16:30:11 +0100 Subject: [PATCH 16/74] Update website/docs/reference/resource-properties/constraints.md --- website/docs/reference/resource-properties/constraints.md | 1 - 1 file changed, 1 deletion(-) diff --git a/website/docs/reference/resource-properties/constraints.md b/website/docs/reference/resource-properties/constraints.md index aaba6a3c851..6fe9b199c08 100644 --- a/website/docs/reference/resource-properties/constraints.md +++ b/website/docs/reference/resource-properties/constraints.md @@ -361,7 +361,6 @@ models: -:::
From 8851b141655181f4e17011cdd26a022718f4a26a Mon Sep 17 00:00:00 2001 From: Natalie Fiann Date: Mon, 2 Sep 2024 16:29:05 +0100 Subject: [PATCH 17/74] Updated doc, moved section to new location --- .../resource-properties/constraints.md | 116 +++++++++--------- 1 file changed, 57 insertions(+), 59 deletions(-) diff --git a/website/docs/reference/resource-properties/constraints.md b/website/docs/reference/resource-properties/constraints.md index 6fe9b199c08..d65398ae005 100644 --- a/website/docs/reference/resource-properties/constraints.md +++ b/website/docs/reference/resource-properties/constraints.md @@ -303,65 +303,6 @@ select -### Custom constraints on models for advanced configuration of tables - -In dbt Cloud, you can use custom constraints on models for advanced configuration of tables. For example, custom constraints allow you to set [masking policies](https://docs.snowflake.com/en/user-guide/security-column-intro#what-are-masking-policies) in Snowflake when using a Create Table As Select (CTAS). - -Each data warehouse has its own set of parameters that can be set for columns in their CTAS statements. For example [Databricks](https://docs.databricks.com/en/sql/language-manual/sql-ref-syntax-ddl-create-table-using.html) and [BigQuery](https://cloud.google.com/bigquery/docs/reference/standard-sql/data-definition-language#column_name_and_column_schema)". - - -Here are a few examples of how to implement tag-based masking policies with contracts and constraints using the following syntax: - - - -```yaml - -models: - -- name: my_model -config: - contract: {enforced: true} - materialized: table - columns: - - name: id - data_type: int - constraints: - - type: custom - expression: "tag (my_tag = 'my_value')" # A custom SQL expression used to enforce a specific constraint on a column. - -``` - - - -Using this syntax requires configuring all the columns and their types as it’s the only way where to send a create or replace ` mytable as ...`. It’s not possible to do it with just a partial list of columns. This means making sure the columns and constraints fields are fully defined. - -To generate a YAML with all the columns, you could also use `generate_model_yaml` from [dbt-codegen](https://github.com/dbt-labs/dbt-codegen/tree/0.12.1/?tab=readme-ov-file#generate_model_yaml-source). - -Alternatively, you can add a masking policy without tags: - - - -```yaml - -models: - - name: my_model - config: - contract: {enforced: true} - materialized: table - columns: - - name: id - data_type: int - constraints: - - type: custom - expression: "masking policy my_policy" - -``` - - - - - -
@@ -577,3 +518,60 @@ alter table schema_name.my_model add constraint 472394792387497234 check (id > 0
+ +### Custom constraints on models for advanced configuration of tables + +In dbt Cloud, you can use custom constraints on models for advanced configuration of tables. For example, custom constraints allow you to set [masking policies](https://docs.snowflake.com/en/user-guide/security-column-intro#what-are-masking-policies) in Snowflake when using a Create Table As Select (CTAS). + +Each data warehouse has its own set of parameters that can be set for columns in their CTAS statements. For example [Databricks](https://docs.databricks.com/en/sql/language-manual/sql-ref-syntax-ddl-create-table-using.html) and [BigQuery](https://cloud.google.com/bigquery/docs/reference/standard-sql/data-definition-language#column_name_and_column_schema)". + +Here's an example of how to implement tag-based masking policies with contracts and constraints using the following syntax: + + + +```shell + +models: + +- name: my_model +config: + contract: {enforced: true} + materialized: table + columns: + - name: id + data_type: int + constraints: + - type: custom + expression: "tag (my_tag = 'my_value')" # A custom SQL expression used to enforce a specific constraint on a column. + +``` + + + +Using this syntax requires configuring all the columns and their types as it’s the only way where to send a create or replace ` mytable as ...`. It’s not possible to do it with just a partial list of columns. This means making sure the columns and constraints fields are fully defined. + +To generate a YAML with all the columns, you could also use `generate_model_yaml` from [dbt-codegen](https://github.com/dbt-labs/dbt-codegen/tree/0.12.1/?tab=readme-ov-file#generate_model_yaml-source). + +Alternatively, you can add a masking policy without tags: + + + +```shell + +models: + - name: my_model + config: + contract: {enforced: true} + materialized: table + columns: + - name: id + data_type: int + constraints: + - type: custom + expression: "masking policy my_policy" + +``` + + + + From 7fc710aa35dd444ca18ebddca666487d1f769d6a Mon Sep 17 00:00:00 2001 From: nataliefiann <120089939+nataliefiann@users.noreply.github.com> Date: Mon, 2 Sep 2024 16:56:43 +0100 Subject: [PATCH 18/74] Update website/docs/reference/resource-properties/constraints.md --- website/docs/reference/resource-properties/constraints.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/reference/resource-properties/constraints.md b/website/docs/reference/resource-properties/constraints.md index d65398ae005..7098a55e311 100644 --- a/website/docs/reference/resource-properties/constraints.md +++ b/website/docs/reference/resource-properties/constraints.md @@ -523,7 +523,7 @@ alter table schema_name.my_model add constraint 472394792387497234 check (id > 0 In dbt Cloud, you can use custom constraints on models for advanced configuration of tables. For example, custom constraints allow you to set [masking policies](https://docs.snowflake.com/en/user-guide/security-column-intro#what-are-masking-policies) in Snowflake when using a Create Table As Select (CTAS). -Each data warehouse has its own set of parameters that can be set for columns in their CTAS statements. For example [Databricks](https://docs.databricks.com/en/sql/language-manual/sql-ref-syntax-ddl-create-table-using.html) and [BigQuery](https://cloud.google.com/bigquery/docs/reference/standard-sql/data-definition-language#column_name_and_column_schema)". +Each data warehouse has its own set of parameters that can be set for columns in their CTAS statements. For example [Databricks](https://docs.databricks.com/en/sql/language-manual/sql-ref-syntax-ddl-create-table-using.html) and [BigQuery](https://cloud.google.com/bigquery/docs/reference/standard-sql/data-definition-language#column_name_and_column_schema). Here's an example of how to implement tag-based masking policies with contracts and constraints using the following syntax: From 8f7cd16960c42f05dc1e054ea04b621d8aba18c3 Mon Sep 17 00:00:00 2001 From: nataliefiann <120089939+nataliefiann@users.noreply.github.com> Date: Mon, 2 Sep 2024 16:57:15 +0100 Subject: [PATCH 19/74] Update website/docs/reference/resource-properties/constraints.md --- website/docs/reference/resource-properties/constraints.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/reference/resource-properties/constraints.md b/website/docs/reference/resource-properties/constraints.md index 7098a55e311..301dd2f7ed0 100644 --- a/website/docs/reference/resource-properties/constraints.md +++ b/website/docs/reference/resource-properties/constraints.md @@ -529,7 +529,7 @@ Here's an example of how to implement tag-based masking policies with contracts -```shell +```yaml models: From 43d6cf5de4dee15f8c04e387df3ac9a73927e707 Mon Sep 17 00:00:00 2001 From: nataliefiann <120089939+nataliefiann@users.noreply.github.com> Date: Mon, 2 Sep 2024 16:57:32 +0100 Subject: [PATCH 20/74] Update website/docs/reference/resource-properties/constraints.md --- website/docs/reference/resource-properties/constraints.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/reference/resource-properties/constraints.md b/website/docs/reference/resource-properties/constraints.md index 301dd2f7ed0..673af5277c4 100644 --- a/website/docs/reference/resource-properties/constraints.md +++ b/website/docs/reference/resource-properties/constraints.md @@ -556,7 +556,7 @@ Alternatively, you can add a masking policy without tags: -```shell +```yaml models: - name: my_model From 8295508329a32349ec8c8f6cb11c6d03f297c433 Mon Sep 17 00:00:00 2001 From: nataliefiann <120089939+nataliefiann@users.noreply.github.com> Date: Tue, 3 Sep 2024 09:18:21 +0100 Subject: [PATCH 21/74] Update website/docs/reference/resource-properties/constraints.md --- website/docs/reference/resource-properties/constraints.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/reference/resource-properties/constraints.md b/website/docs/reference/resource-properties/constraints.md index 673af5277c4..09f5f6076e9 100644 --- a/website/docs/reference/resource-properties/constraints.md +++ b/website/docs/reference/resource-properties/constraints.md @@ -527,7 +527,7 @@ Each data warehouse has its own set of parameters that can be set for columns in Here's an example of how to implement tag-based masking policies with contracts and constraints using the following syntax: - + ```yaml From f8cce3344bbc924b9b646c0e89583f4f77e7b1b7 Mon Sep 17 00:00:00 2001 From: nataliefiann <120089939+nataliefiann@users.noreply.github.com> Date: Tue, 3 Sep 2024 09:18:54 +0100 Subject: [PATCH 22/74] Update website/docs/reference/resource-properties/constraints.md --- website/docs/reference/resource-properties/constraints.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/reference/resource-properties/constraints.md b/website/docs/reference/resource-properties/constraints.md index 09f5f6076e9..079dd88106e 100644 --- a/website/docs/reference/resource-properties/constraints.md +++ b/website/docs/reference/resource-properties/constraints.md @@ -554,7 +554,7 @@ To generate a YAML with all the columns, you could also use `generate_model_yaml Alternatively, you can add a masking policy without tags: - + ```yaml From 04bf56d9daf18e3a0d3ea6f7abe58e014ed7e93c Mon Sep 17 00:00:00 2001 From: nataliefiann <120089939+nataliefiann@users.noreply.github.com> Date: Tue, 3 Sep 2024 09:34:33 +0100 Subject: [PATCH 23/74] Update website/docs/reference/resource-properties/constraints.md --- website/docs/reference/resource-properties/constraints.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/reference/resource-properties/constraints.md b/website/docs/reference/resource-properties/constraints.md index 079dd88106e..b31923c2693 100644 --- a/website/docs/reference/resource-properties/constraints.md +++ b/website/docs/reference/resource-properties/constraints.md @@ -533,7 +533,7 @@ Here's an example of how to implement tag-based masking policies with contracts models: -- name: my_model + - name: my_model config: contract: {enforced: true} materialized: table From 2455fb5804538906d8d09a20dcf8ed1bd687ce70 Mon Sep 17 00:00:00 2001 From: nataliefiann <120089939+nataliefiann@users.noreply.github.com> Date: Tue, 3 Sep 2024 09:35:26 +0100 Subject: [PATCH 24/74] Update website/docs/reference/resource-properties/constraints.md --- website/docs/reference/resource-properties/constraints.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/reference/resource-properties/constraints.md b/website/docs/reference/resource-properties/constraints.md index b31923c2693..838e502d714 100644 --- a/website/docs/reference/resource-properties/constraints.md +++ b/website/docs/reference/resource-properties/constraints.md @@ -534,7 +534,7 @@ Here's an example of how to implement tag-based masking policies with contracts models: - name: my_model -config: + config: contract: {enforced: true} materialized: table columns: From 70ad2605a99021fafb8bd5649ce1bbf913e1ca25 Mon Sep 17 00:00:00 2001 From: nataliefiann <120089939+nataliefiann@users.noreply.github.com> Date: Tue, 3 Sep 2024 09:41:58 +0100 Subject: [PATCH 25/74] Update website/docs/reference/resource-properties/constraints.md --- website/docs/reference/resource-properties/constraints.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/reference/resource-properties/constraints.md b/website/docs/reference/resource-properties/constraints.md index 838e502d714..d7c5c53dd61 100644 --- a/website/docs/reference/resource-properties/constraints.md +++ b/website/docs/reference/resource-properties/constraints.md @@ -537,7 +537,7 @@ models: config: contract: {enforced: true} materialized: table - columns: + columns: - name: id data_type: int constraints: From 72420bdf2a90f7aafd1638e9cafe4db1b2a91be5 Mon Sep 17 00:00:00 2001 From: nataliefiann <120089939+nataliefiann@users.noreply.github.com> Date: Tue, 3 Sep 2024 09:43:31 +0100 Subject: [PATCH 26/74] Update website/docs/reference/resource-properties/constraints.md --- website/docs/reference/resource-properties/constraints.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/reference/resource-properties/constraints.md b/website/docs/reference/resource-properties/constraints.md index d7c5c53dd61..a502a0b87d4 100644 --- a/website/docs/reference/resource-properties/constraints.md +++ b/website/docs/reference/resource-properties/constraints.md @@ -533,7 +533,7 @@ Here's an example of how to implement tag-based masking policies with contracts models: - - name: my_model + - name: my_model config: contract: {enforced: true} materialized: table From 7d2287d7e2443896cf362a08a6234744b67f2c4d Mon Sep 17 00:00:00 2001 From: nataliefiann <120089939+nataliefiann@users.noreply.github.com> Date: Tue, 3 Sep 2024 09:44:08 +0100 Subject: [PATCH 27/74] Update website/docs/reference/resource-properties/constraints.md --- website/docs/reference/resource-properties/constraints.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/reference/resource-properties/constraints.md b/website/docs/reference/resource-properties/constraints.md index a502a0b87d4..a26a2ebb917 100644 --- a/website/docs/reference/resource-properties/constraints.md +++ b/website/docs/reference/resource-properties/constraints.md @@ -537,7 +537,7 @@ models: config: contract: {enforced: true} materialized: table - columns: + columns: - name: id data_type: int constraints: From e10b8269d95f6443b1f0b821feafc07338fe01b9 Mon Sep 17 00:00:00 2001 From: nataliefiann <120089939+nataliefiann@users.noreply.github.com> Date: Tue, 3 Sep 2024 09:51:47 +0100 Subject: [PATCH 28/74] Update website/docs/reference/resource-properties/constraints.md --- website/docs/reference/resource-properties/constraints.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/reference/resource-properties/constraints.md b/website/docs/reference/resource-properties/constraints.md index a26a2ebb917..b044b4ba670 100644 --- a/website/docs/reference/resource-properties/constraints.md +++ b/website/docs/reference/resource-properties/constraints.md @@ -535,7 +535,7 @@ models: - name: my_model config: - contract: {enforced: true} + contract: {enforced: true} materialized: table columns: - name: id From 244e2f407c1e5ef57ea95674114a85106c84945a Mon Sep 17 00:00:00 2001 From: nataliefiann <120089939+nataliefiann@users.noreply.github.com> Date: Tue, 3 Sep 2024 09:52:02 +0100 Subject: [PATCH 29/74] Update website/docs/reference/resource-properties/constraints.md --- website/docs/reference/resource-properties/constraints.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/reference/resource-properties/constraints.md b/website/docs/reference/resource-properties/constraints.md index b044b4ba670..d038465a687 100644 --- a/website/docs/reference/resource-properties/constraints.md +++ b/website/docs/reference/resource-properties/constraints.md @@ -536,7 +536,7 @@ models: - name: my_model config: contract: {enforced: true} - materialized: table + materialized: table columns: - name: id data_type: int From 8258226b8a62e226e159181558df39cd6fbceb62 Mon Sep 17 00:00:00 2001 From: nataliefiann <120089939+nataliefiann@users.noreply.github.com> Date: Tue, 3 Sep 2024 09:53:15 +0100 Subject: [PATCH 30/74] Update website/docs/reference/resource-properties/constraints.md --- website/docs/reference/resource-properties/constraints.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/reference/resource-properties/constraints.md b/website/docs/reference/resource-properties/constraints.md index d038465a687..555766e13d7 100644 --- a/website/docs/reference/resource-properties/constraints.md +++ b/website/docs/reference/resource-properties/constraints.md @@ -537,7 +537,7 @@ models: config: contract: {enforced: true} materialized: table - columns: + columns: - name: id data_type: int constraints: From 2cf8771dfd8afd71bc59d7fc54ea9fde3cab3133 Mon Sep 17 00:00:00 2001 From: nataliefiann <120089939+nataliefiann@users.noreply.github.com> Date: Tue, 3 Sep 2024 10:35:17 +0100 Subject: [PATCH 31/74] Update website/docs/reference/resource-properties/constraints.md --- website/docs/reference/resource-properties/constraints.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/website/docs/reference/resource-properties/constraints.md b/website/docs/reference/resource-properties/constraints.md index 555766e13d7..891cff8c9e8 100644 --- a/website/docs/reference/resource-properties/constraints.md +++ b/website/docs/reference/resource-properties/constraints.md @@ -521,7 +521,9 @@ alter table schema_name.my_model add constraint 472394792387497234 check (id > 0 ### Custom constraints on models for advanced configuration of tables -In dbt Cloud, you can use custom constraints on models for advanced configuration of tables. For example, custom constraints allow you to set [masking policies](https://docs.snowflake.com/en/user-guide/security-column-intro#what-are-masking-policies) in Snowflake when using a Create Table As Select (CTAS). +In dbt Cloud, you can use custom constraints on models for advanced configuration of tables. Different data warehouses will support different syntax and different capabilities, but custom constraints will always allow you to add configuration to specific columns. + +For example, custom constraints allow you to set [masking policies](https://docs.snowflake.com/en/user-guide/security-column-intro#what-are-masking-policies) in Snowflake when using a Create Table As Select (CTAS). Each data warehouse has its own set of parameters that can be set for columns in their CTAS statements. For example [Databricks](https://docs.databricks.com/en/sql/language-manual/sql-ref-syntax-ddl-create-table-using.html) and [BigQuery](https://cloud.google.com/bigquery/docs/reference/standard-sql/data-definition-language#column_name_and_column_schema). From 15f76640be92a19273610878ca94cb3f8302d308 Mon Sep 17 00:00:00 2001 From: nataliefiann <120089939+nataliefiann@users.noreply.github.com> Date: Tue, 3 Sep 2024 12:00:36 +0100 Subject: [PATCH 32/74] Update website/docs/reference/resource-properties/constraints.md --- website/docs/reference/resource-properties/constraints.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/website/docs/reference/resource-properties/constraints.md b/website/docs/reference/resource-properties/constraints.md index 891cff8c9e8..2457972ad86 100644 --- a/website/docs/reference/resource-properties/constraints.md +++ b/website/docs/reference/resource-properties/constraints.md @@ -527,6 +527,11 @@ For example, custom constraints allow you to set [masking policies](https://docs Each data warehouse has its own set of parameters that can be set for columns in their CTAS statements. For example [Databricks](https://docs.databricks.com/en/sql/language-manual/sql-ref-syntax-ddl-create-table-using.html) and [BigQuery](https://cloud.google.com/bigquery/docs/reference/standard-sql/data-definition-language#column_name_and_column_schema). +There are some different scenarios of implementing contracts and constraints: + +- [Tag-based masking policies](https://github.com/dbt-labs/dbt-adapters/issues/85#issuecomment-2310772737). +- [Add a masking policy without a tag](https://github.com/dbt-labs/dbt-adapters/issues/85#issuecomment-2314547162). + Here's an example of how to implement tag-based masking policies with contracts and constraints using the following syntax: From 05afe4ece7b40f71b58c63f9416f08af3a034568 Mon Sep 17 00:00:00 2001 From: Natalie Fiann Date: Wed, 4 Sep 2024 13:46:37 +0100 Subject: [PATCH 33/74] updated doc --- .../resource-properties/constraints.md | 35 ++++++++++--------- 1 file changed, 18 insertions(+), 17 deletions(-) diff --git a/website/docs/reference/resource-properties/constraints.md b/website/docs/reference/resource-properties/constraints.md index 2457972ad86..8f2746c497c 100644 --- a/website/docs/reference/resource-properties/constraints.md +++ b/website/docs/reference/resource-properties/constraints.md @@ -521,13 +521,13 @@ alter table schema_name.my_model add constraint 472394792387497234 check (id > 0 ### Custom constraints on models for advanced configuration of tables -In dbt Cloud, you can use custom constraints on models for advanced configuration of tables. Different data warehouses will support different syntax and different capabilities, but custom constraints will always allow you to add configuration to specific columns. +In dbt Cloud, you can use custom constraints on models for the advanced configuration of tables. Different data warehouses will support different syntax and capabilities, but custom constraints will always allow you to add configuration to specific columns. For example, custom constraints allow you to set [masking policies](https://docs.snowflake.com/en/user-guide/security-column-intro#what-are-masking-policies) in Snowflake when using a Create Table As Select (CTAS). -Each data warehouse has its own set of parameters that can be set for columns in their CTAS statements. For example [Databricks](https://docs.databricks.com/en/sql/language-manual/sql-ref-syntax-ddl-create-table-using.html) and [BigQuery](https://cloud.google.com/bigquery/docs/reference/standard-sql/data-definition-language#column_name_and_column_schema). +[Databricks](https://docs.databricks.com/en/sql/language-manual/sql-ref-syntax-ddl-create-table-using.html) and [BigQuery](https://cloud.google.com/bigquery/docs/reference/standard-sql/data-definition-language#column_name_and_column_schema) are also other examples of data warehouses having their own set of parameters that can be set for columns in their CTAS statements. -There are some different scenarios of implementing contracts and constraints: +There are different scenarios of implementing contracts and constraints: - [Tag-based masking policies](https://github.com/dbt-labs/dbt-adapters/issues/85#issuecomment-2310772737). - [Add a masking policy without a tag](https://github.com/dbt-labs/dbt-adapters/issues/85#issuecomment-2314547162). @@ -539,25 +539,25 @@ Here's an example of how to implement tag-based masking policies with contracts ```yaml models: - - name: my_model - config: - contract: {enforced: true} - materialized: table - columns: - - name: id - data_type: int - constraints: - - type: custom - expression: "tag (my_tag = 'my_value')" # A custom SQL expression used to enforce a specific constraint on a column. +config: null +contract: + enforced: true +materialized: table +columns: + - name: id + data_type: int + constraints: + - type: custom + expression: tag (my_tag = 'my_value') # A custom SQL expression used to enforce a specific constraint on a column. ``` -Using this syntax requires configuring all the columns and their types as it’s the only way where to send a create or replace ` mytable as ...`. It’s not possible to do it with just a partial list of columns. This means making sure the columns and constraints fields are fully defined. +Using this syntax requires configuring all the columns and their types as it’s the only way to send a create or replace ` mytable as ...`. It’s not possible to do it with just a partial list of columns. This means making sure the columns and constraints fields are fully defined. -To generate a YAML with all the columns, you could also use `generate_model_yaml` from [dbt-codegen](https://github.com/dbt-labs/dbt-codegen/tree/0.12.1/?tab=readme-ov-file#generate_model_yaml-source). +To generate a YAML with all the columns, you can use `generate_model_yaml` from [dbt-codegen](https://github.com/dbt-labs/dbt-codegen/tree/0.12.1/?tab=readme-ov-file#generate_model_yaml-source). Alternatively, you can add a masking policy without tags: @@ -568,14 +568,15 @@ Alternatively, you can add a masking policy without tags: models: - name: my_model config: - contract: {enforced: true} + contract: + enforced: true materialized: table columns: - name: id data_type: int constraints: - type: custom - expression: "masking policy my_policy" + expression: masking policy my_policy ``` From d9290f0da08641875d7534e6a4cd2db24381baf3 Mon Sep 17 00:00:00 2001 From: nataliefiann <120089939+nataliefiann@users.noreply.github.com> Date: Wed, 4 Sep 2024 15:17:35 +0100 Subject: [PATCH 34/74] Update website/docs/reference/resource-properties/constraints.md Co-authored-by: Mirna Wong <89008547+mirnawong1@users.noreply.github.com> --- website/docs/reference/resource-properties/constraints.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/reference/resource-properties/constraints.md b/website/docs/reference/resource-properties/constraints.md index 8f2746c497c..fc4b52103eb 100644 --- a/website/docs/reference/resource-properties/constraints.md +++ b/website/docs/reference/resource-properties/constraints.md @@ -521,7 +521,7 @@ alter table schema_name.my_model add constraint 472394792387497234 check (id > 0 ### Custom constraints on models for advanced configuration of tables -In dbt Cloud, you can use custom constraints on models for the advanced configuration of tables. Different data warehouses will support different syntax and capabilities, but custom constraints will always allow you to add configuration to specific columns. +In dbt Cloud, you can use custom constraints on models for the advanced configuration of tables. Different data warehouses support different syntax and capabilities. However, custom constraints allow you to add configuration to specific columns. For example, custom constraints allow you to set [masking policies](https://docs.snowflake.com/en/user-guide/security-column-intro#what-are-masking-policies) in Snowflake when using a Create Table As Select (CTAS). From 3fea39018e7741ae6af661becd2b12a828ff3254 Mon Sep 17 00:00:00 2001 From: nataliefiann <120089939+nataliefiann@users.noreply.github.com> Date: Wed, 4 Sep 2024 15:17:48 +0100 Subject: [PATCH 35/74] Update website/docs/reference/resource-properties/constraints.md Co-authored-by: Mirna Wong <89008547+mirnawong1@users.noreply.github.com> --- website/docs/reference/resource-properties/constraints.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/reference/resource-properties/constraints.md b/website/docs/reference/resource-properties/constraints.md index fc4b52103eb..2aeaeb37e3c 100644 --- a/website/docs/reference/resource-properties/constraints.md +++ b/website/docs/reference/resource-properties/constraints.md @@ -527,7 +527,7 @@ For example, custom constraints allow you to set [masking policies](https://docs [Databricks](https://docs.databricks.com/en/sql/language-manual/sql-ref-syntax-ddl-create-table-using.html) and [BigQuery](https://cloud.google.com/bigquery/docs/reference/standard-sql/data-definition-language#column_name_and_column_schema) are also other examples of data warehouses having their own set of parameters that can be set for columns in their CTAS statements. -There are different scenarios of implementing contracts and constraints: +You can implement constraints in a couple of different ways: - [Tag-based masking policies](https://github.com/dbt-labs/dbt-adapters/issues/85#issuecomment-2310772737). - [Add a masking policy without a tag](https://github.com/dbt-labs/dbt-adapters/issues/85#issuecomment-2314547162). From 278f3d5da3a6a993f81fd39899511654da0d0465 Mon Sep 17 00:00:00 2001 From: nataliefiann <120089939+nataliefiann@users.noreply.github.com> Date: Wed, 4 Sep 2024 15:18:45 +0100 Subject: [PATCH 36/74] Update website/docs/reference/resource-properties/constraints.md Co-authored-by: Mirna Wong <89008547+mirnawong1@users.noreply.github.com> --- website/docs/reference/resource-properties/constraints.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/website/docs/reference/resource-properties/constraints.md b/website/docs/reference/resource-properties/constraints.md index 2aeaeb37e3c..b8b857405ce 100644 --- a/website/docs/reference/resource-properties/constraints.md +++ b/website/docs/reference/resource-properties/constraints.md @@ -523,7 +523,9 @@ alter table schema_name.my_model add constraint 472394792387497234 check (id > 0 In dbt Cloud, you can use custom constraints on models for the advanced configuration of tables. Different data warehouses support different syntax and capabilities. However, custom constraints allow you to add configuration to specific columns. -For example, custom constraints allow you to set [masking policies](https://docs.snowflake.com/en/user-guide/security-column-intro#what-are-masking-policies) in Snowflake when using a Create Table As Select (CTAS). + - For example, custom constraints allow you to set [masking policies](https://docs.snowflake.com/en/user-guide/security-column-intro#what-are-masking-policies) in Snowflake when using a Create Table As Select (CTAS). + + - [Databricks](https://docs.databricks.com/en/sql/language-manual/sql-ref-syntax-ddl-create-table-using.html) and [BigQuery](https://cloud.google.com/bigquery/docs/reference/standard-sql/data-definition-language#column_name_and_column_schema) ar other examples of data warehouses having their own set of parameters that can be set for columns in their CTAS statements. [Databricks](https://docs.databricks.com/en/sql/language-manual/sql-ref-syntax-ddl-create-table-using.html) and [BigQuery](https://cloud.google.com/bigquery/docs/reference/standard-sql/data-definition-language#column_name_and_column_schema) are also other examples of data warehouses having their own set of parameters that can be set for columns in their CTAS statements. From f3bced0a19ac7908d8dab997ad9f932c03cd5219 Mon Sep 17 00:00:00 2001 From: nataliefiann <120089939+nataliefiann@users.noreply.github.com> Date: Wed, 4 Sep 2024 15:19:29 +0100 Subject: [PATCH 37/74] Update website/docs/reference/resource-properties/constraints.md Co-authored-by: Mirna Wong <89008547+mirnawong1@users.noreply.github.com> --- website/docs/reference/resource-properties/constraints.md | 1 - 1 file changed, 1 deletion(-) diff --git a/website/docs/reference/resource-properties/constraints.md b/website/docs/reference/resource-properties/constraints.md index b8b857405ce..abde4c13f94 100644 --- a/website/docs/reference/resource-properties/constraints.md +++ b/website/docs/reference/resource-properties/constraints.md @@ -527,7 +527,6 @@ In dbt Cloud, you can use custom constraints on models for the advanced configur - [Databricks](https://docs.databricks.com/en/sql/language-manual/sql-ref-syntax-ddl-create-table-using.html) and [BigQuery](https://cloud.google.com/bigquery/docs/reference/standard-sql/data-definition-language#column_name_and_column_schema) ar other examples of data warehouses having their own set of parameters that can be set for columns in their CTAS statements. -[Databricks](https://docs.databricks.com/en/sql/language-manual/sql-ref-syntax-ddl-create-table-using.html) and [BigQuery](https://cloud.google.com/bigquery/docs/reference/standard-sql/data-definition-language#column_name_and_column_schema) are also other examples of data warehouses having their own set of parameters that can be set for columns in their CTAS statements. You can implement constraints in a couple of different ways: From 88f16f07773fbbdb38ff0e852f3433b4d016c8fc Mon Sep 17 00:00:00 2001 From: nataliefiann <120089939+nataliefiann@users.noreply.github.com> Date: Wed, 4 Sep 2024 15:23:25 +0100 Subject: [PATCH 38/74] Update website/docs/reference/resource-properties/constraints.md Co-authored-by: Mirna Wong <89008547+mirnawong1@users.noreply.github.com> --- website/docs/reference/resource-properties/constraints.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/reference/resource-properties/constraints.md b/website/docs/reference/resource-properties/constraints.md index abde4c13f94..1b3ba2db7a3 100644 --- a/website/docs/reference/resource-properties/constraints.md +++ b/website/docs/reference/resource-properties/constraints.md @@ -577,7 +577,7 @@ models: data_type: int constraints: - type: custom - expression: masking policy my_policy + expression: "masking policy my_policy" ``` From bb6c5ea042d0edc74c2fb9e84207870de6c659cd Mon Sep 17 00:00:00 2001 From: nataliefiann <120089939+nataliefiann@users.noreply.github.com> Date: Wed, 4 Sep 2024 15:32:43 +0100 Subject: [PATCH 39/74] Update website/docs/reference/resource-properties/constraints.md --- website/docs/reference/resource-properties/constraints.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/reference/resource-properties/constraints.md b/website/docs/reference/resource-properties/constraints.md index 1b3ba2db7a3..ab5a83aa64a 100644 --- a/website/docs/reference/resource-properties/constraints.md +++ b/website/docs/reference/resource-properties/constraints.md @@ -530,7 +530,7 @@ In dbt Cloud, you can use custom constraints on models for the advanced configur You can implement constraints in a couple of different ways: -- [Tag-based masking policies](https://github.com/dbt-labs/dbt-adapters/issues/85#issuecomment-2310772737). +- ### Custom constraints with tags - [Add a masking policy without a tag](https://github.com/dbt-labs/dbt-adapters/issues/85#issuecomment-2314547162). Here's an example of how to implement tag-based masking policies with contracts and constraints using the following syntax: From fb6e44d5cb12350a29a6e21230d8ffe9f6202156 Mon Sep 17 00:00:00 2001 From: nataliefiann <120089939+nataliefiann@users.noreply.github.com> Date: Wed, 4 Sep 2024 15:33:23 +0100 Subject: [PATCH 40/74] Update website/docs/reference/resource-properties/constraints.md --- website/docs/reference/resource-properties/constraints.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/reference/resource-properties/constraints.md b/website/docs/reference/resource-properties/constraints.md index ab5a83aa64a..702da092b73 100644 --- a/website/docs/reference/resource-properties/constraints.md +++ b/website/docs/reference/resource-properties/constraints.md @@ -531,7 +531,7 @@ In dbt Cloud, you can use custom constraints on models for the advanced configur You can implement constraints in a couple of different ways: - ### Custom constraints with tags -- [Add a masking policy without a tag](https://github.com/dbt-labs/dbt-adapters/issues/85#issuecomment-2314547162). +- ### Custom constraints without tags Here's an example of how to implement tag-based masking policies with contracts and constraints using the following syntax: From 9cfc253db79a8a40c35b30ce266c74205a4e2f35 Mon Sep 17 00:00:00 2001 From: nataliefiann <120089939+nataliefiann@users.noreply.github.com> Date: Wed, 4 Sep 2024 15:54:08 +0100 Subject: [PATCH 41/74] Update website/docs/reference/resource-properties/constraints.md --- website/docs/reference/resource-properties/constraints.md | 1 - 1 file changed, 1 deletion(-) diff --git a/website/docs/reference/resource-properties/constraints.md b/website/docs/reference/resource-properties/constraints.md index 702da092b73..2d4ac244715 100644 --- a/website/docs/reference/resource-properties/constraints.md +++ b/website/docs/reference/resource-properties/constraints.md @@ -531,7 +531,6 @@ In dbt Cloud, you can use custom constraints on models for the advanced configur You can implement constraints in a couple of different ways: - ### Custom constraints with tags -- ### Custom constraints without tags Here's an example of how to implement tag-based masking policies with contracts and constraints using the following syntax: From 1a7b84e618769cf783c41a76ed3ff91b0276ac7c Mon Sep 17 00:00:00 2001 From: nataliefiann <120089939+nataliefiann@users.noreply.github.com> Date: Wed, 4 Sep 2024 15:55:13 +0100 Subject: [PATCH 42/74] Update website/docs/reference/resource-properties/constraints.md --- website/docs/reference/resource-properties/constraints.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/website/docs/reference/resource-properties/constraints.md b/website/docs/reference/resource-properties/constraints.md index 2d4ac244715..552ffbd918d 100644 --- a/website/docs/reference/resource-properties/constraints.md +++ b/website/docs/reference/resource-properties/constraints.md @@ -559,6 +559,9 @@ Using this syntax requires configuring all the columns and their types as it’s To generate a YAML with all the columns, you can use `generate_model_yaml` from [dbt-codegen](https://github.com/dbt-labs/dbt-codegen/tree/0.12.1/?tab=readme-ov-file#generate_model_yaml-source). + +### Custom constraints without tags + Alternatively, you can add a masking policy without tags: From 84ec68c10a816d9320fd9f6c8a0cf0b7ff844680 Mon Sep 17 00:00:00 2001 From: nataliefiann <120089939+nataliefiann@users.noreply.github.com> Date: Wed, 4 Sep 2024 15:55:36 +0100 Subject: [PATCH 43/74] Update website/docs/reference/resource-properties/constraints.md --- website/docs/reference/resource-properties/constraints.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/reference/resource-properties/constraints.md b/website/docs/reference/resource-properties/constraints.md index 552ffbd918d..58fa5b999a6 100644 --- a/website/docs/reference/resource-properties/constraints.md +++ b/website/docs/reference/resource-properties/constraints.md @@ -530,7 +530,7 @@ In dbt Cloud, you can use custom constraints on models for the advanced configur You can implement constraints in a couple of different ways: -- ### Custom constraints with tags +### Custom constraints with tags Here's an example of how to implement tag-based masking policies with contracts and constraints using the following syntax: From b4285e79dade96dd4fb6aa307494f17bfa0e129c Mon Sep 17 00:00:00 2001 From: Natalie Fiann Date: Wed, 4 Sep 2024 16:24:12 +0100 Subject: [PATCH 44/74] update yaml --- .../resource-properties/constraints.md | 20 +++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/website/docs/reference/resource-properties/constraints.md b/website/docs/reference/resource-properties/constraints.md index 8f2746c497c..9170f23dbd2 100644 --- a/website/docs/reference/resource-properties/constraints.md +++ b/website/docs/reference/resource-properties/constraints.md @@ -540,16 +540,16 @@ Here's an example of how to implement tag-based masking policies with contracts models: - name: my_model -config: null -contract: - enforced: true -materialized: table -columns: - - name: id - data_type: int - constraints: - - type: custom - expression: tag (my_tag = 'my_value') # A custom SQL expression used to enforce a specific constraint on a column. + config: null + contract: + enforced: true + materialized: table + columns: + - name: id + data_type: int + constraints: + - type: custom + expression: tag (my_tag = 'my_value') # A custom SQL expression used to enforce a specific constraint on a column. ``` From 5217d03149f33f96a39a697a163559f668e28bb9 Mon Sep 17 00:00:00 2001 From: nataliefiann <120089939+nataliefiann@users.noreply.github.com> Date: Wed, 4 Sep 2024 16:41:08 +0100 Subject: [PATCH 45/74] Update website/docs/reference/resource-properties/constraints.md --- website/docs/reference/resource-properties/constraints.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/reference/resource-properties/constraints.md b/website/docs/reference/resource-properties/constraints.md index 0d4c71c7b69..bcae091afde 100644 --- a/website/docs/reference/resource-properties/constraints.md +++ b/website/docs/reference/resource-properties/constraints.md @@ -525,7 +525,7 @@ In dbt Cloud, you can use custom constraints on models for the advanced configur - For example, custom constraints allow you to set [masking policies](https://docs.snowflake.com/en/user-guide/security-column-intro#what-are-masking-policies) in Snowflake when using a Create Table As Select (CTAS). - - [Databricks](https://docs.databricks.com/en/sql/language-manual/sql-ref-syntax-ddl-create-table-using.html) and [BigQuery](https://cloud.google.com/bigquery/docs/reference/standard-sql/data-definition-language#column_name_and_column_schema) ar other examples of data warehouses having their own set of parameters that can be set for columns in their CTAS statements. + - [Databricks](https://docs.databricks.com/en/sql/language-manual/sql-ref-syntax-ddl-create-table-using.html) and [BigQuery](https://cloud.google.com/bigquery/docs/reference/standard-sql/data-definition-language#column_name_and_column_schema) are other examples of data warehouses having their own set of parameters that can be set for columns in their CTAS statements. You can implement constraints in a couple of different ways: From 3885b93c00c3aaec9dfadb9314fd0319dd0ac3e8 Mon Sep 17 00:00:00 2001 From: nataliefiann <120089939+nataliefiann@users.noreply.github.com> Date: Wed, 4 Sep 2024 17:37:19 +0100 Subject: [PATCH 46/74] Update website/docs/reference/resource-properties/constraints.md Co-authored-by: Mirna Wong <89008547+mirnawong1@users.noreply.github.com> --- website/docs/reference/resource-properties/constraints.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/website/docs/reference/resource-properties/constraints.md b/website/docs/reference/resource-properties/constraints.md index bcae091afde..f047ecadfff 100644 --- a/website/docs/reference/resource-properties/constraints.md +++ b/website/docs/reference/resource-properties/constraints.md @@ -530,6 +530,8 @@ In dbt Cloud, you can use custom constraints on models for the advanced configur You can implement constraints in a couple of different ways: +- [Custom constraints with tags](#custom-constraints-with-tags) +- [Custom constraints without tags](#custom-constraints-without-tags) ### Custom constraints with tags Here's an example of how to implement tag-based masking policies with contracts and constraints using the following syntax: From ecff6de1c6800e423173e0ff58021ba79a203be5 Mon Sep 17 00:00:00 2001 From: Natalie Fiann Date: Tue, 10 Sep 2024 17:00:11 +0100 Subject: [PATCH 47/74] Updated project splitting section --- .../how-we-structure/5-the-rest-of-the-project.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/website/docs/best-practices/how-we-structure/5-the-rest-of-the-project.md b/website/docs/best-practices/how-we-structure/5-the-rest-of-the-project.md index c7522bf12eb..07489ab67cd 100644 --- a/website/docs/best-practices/how-we-structure/5-the-rest-of-the-project.md +++ b/website/docs/best-practices/how-we-structure/5-the-rest-of-the-project.md @@ -102,9 +102,9 @@ We’ve focused heavily thus far on the primary area of action in our dbt projec ### Project splitting -One important, growing consideration in the analytics engineering ecosystem is how and when to split a codebase into multiple dbt projects. Our present stance on this for most projects, particularly for teams starting out, is straightforward: you should avoid it unless you have no other option or it saves you from an even more complex workaround. If you do have the need to split up your project, it’s completely possible through the use of private packages, but the added complexity and separation is, for most organizations, a hindrance, not a help, at present. That said, this is very likely subject to change! [We want to create a world where it’s easy to bring lots of dbt projects together into a cohesive lineage](https://github.com/dbt-labs/dbt-core/discussions/5244). In a world where it’s simple to break up monolithic dbt projects into multiple connected projects, perhaps inside of a modern mono repo, the calculus will be different, and the below situations we recommend against may become totally viable. So watch this space! +One important, growing consideration in the analytics engineering ecosystem is how and when to split a codebase into multiple dbt projects. Currently, our advice for most teams, especially those just starting out, is fairly simple: in most cases, we recommend doing so! With the introduction of [dbt Mesh](/best-practices/how-we-mesh/mesh-1-intro), organizations can more effectively handle complexity by utilizing multiple interconnected dbt projects rather than a single large, monolithic one. This approach is designed to speed up development while maintaining governance. As breaking up monolithic dbt projects into smaller, connected projects, potentially within a modern mono repo becomes easier, the scenarios we currently advise against may soon become feasible. So watch this space! -- ❌ **Business groups or departments.** Conceptual separations within the project are not a good reason to split up your project. Splitting up, for instance, marketing and finance modeling into separate projects will not only add unnecessary complexity but destroy the unifying effect of collaborating across your organization on cohesive definitions and business logic. +- ✅ **Business groups or departments.** Conceptual separations within the project are a good reason to split up your project. For more information about dbt mesh, please refer to our [dbt Mesh FAQs](/best-practices/how-we-mesh/mesh-5-faqs). - ❌ **ML vs Reporting use cases.** Similarly to the point above, splitting a project up based on different use cases, particularly more standard BI versus ML features, is a common idea. We tend to discourage it for the time being. As with the previous point, a foundational goal of implementing dbt is to create a single source of truth in your organization. The features you’re providing to your data science teams should be coming from the same marts and metrics that serve reports on executive dashboards. - ✅ **Data governance.** Structural, organizational needs — such as data governance and security — are one of the few worthwhile reasons to split up a project. If, for instance, you work at a healthcare company with only a small team cleared to access raw data with PII in it, you may need to split out your staging models into their own projects to preserve those policies. In that case, you would import your staging project into the project that builds on those staging models as a [private package](https://docs.getdbt.com/docs/build/packages/#private-packages). - ✅ **Project size.** At a certain point, your project may grow to have simply too many models to present a viable development experience. If you have 1000s of models, it absolutely makes sense to find a way to split up your project. From 55f9a4c757f5fbb1251a9afc6443be0b02b312f7 Mon Sep 17 00:00:00 2001 From: nataliefiann <120089939+nataliefiann@users.noreply.github.com> Date: Tue, 10 Sep 2024 21:05:42 +0100 Subject: [PATCH 48/74] Update website/docs/best-practices/how-we-structure/5-the-rest-of-the-project.md Co-authored-by: dave-connors-3 <73915542+dave-connors-3@users.noreply.github.com> --- .../how-we-structure/5-the-rest-of-the-project.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/best-practices/how-we-structure/5-the-rest-of-the-project.md b/website/docs/best-practices/how-we-structure/5-the-rest-of-the-project.md index 07489ab67cd..172dbd149b4 100644 --- a/website/docs/best-practices/how-we-structure/5-the-rest-of-the-project.md +++ b/website/docs/best-practices/how-we-structure/5-the-rest-of-the-project.md @@ -104,7 +104,7 @@ We’ve focused heavily thus far on the primary area of action in our dbt projec One important, growing consideration in the analytics engineering ecosystem is how and when to split a codebase into multiple dbt projects. Currently, our advice for most teams, especially those just starting out, is fairly simple: in most cases, we recommend doing so! With the introduction of [dbt Mesh](/best-practices/how-we-mesh/mesh-1-intro), organizations can more effectively handle complexity by utilizing multiple interconnected dbt projects rather than a single large, monolithic one. This approach is designed to speed up development while maintaining governance. As breaking up monolithic dbt projects into smaller, connected projects, potentially within a modern mono repo becomes easier, the scenarios we currently advise against may soon become feasible. So watch this space! -- ✅ **Business groups or departments.** Conceptual separations within the project are a good reason to split up your project. For more information about dbt mesh, please refer to our [dbt Mesh FAQs](/best-practices/how-we-mesh/mesh-5-faqs). +- ✅ **Business groups or departments.** Conceptual separations within the project are the primary reason to split up your project. This allows your business domains to own their own data products and still collaborate using dbt Mesh. For more information about dbt Mesh, please refer to our [dbt Mesh FAQs](/best-practices/how-we-mesh/mesh-5-faqs). - ❌ **ML vs Reporting use cases.** Similarly to the point above, splitting a project up based on different use cases, particularly more standard BI versus ML features, is a common idea. We tend to discourage it for the time being. As with the previous point, a foundational goal of implementing dbt is to create a single source of truth in your organization. The features you’re providing to your data science teams should be coming from the same marts and metrics that serve reports on executive dashboards. - ✅ **Data governance.** Structural, organizational needs — such as data governance and security — are one of the few worthwhile reasons to split up a project. If, for instance, you work at a healthcare company with only a small team cleared to access raw data with PII in it, you may need to split out your staging models into their own projects to preserve those policies. In that case, you would import your staging project into the project that builds on those staging models as a [private package](https://docs.getdbt.com/docs/build/packages/#private-packages). - ✅ **Project size.** At a certain point, your project may grow to have simply too many models to present a viable development experience. If you have 1000s of models, it absolutely makes sense to find a way to split up your project. From b5d2a4514909b06ed760ecdb5a396fe41b6097c4 Mon Sep 17 00:00:00 2001 From: nataliefiann <120089939+nataliefiann@users.noreply.github.com> Date: Tue, 10 Sep 2024 21:06:45 +0100 Subject: [PATCH 49/74] Update website/docs/best-practices/how-we-structure/5-the-rest-of-the-project.md Co-authored-by: Mirna Wong <89008547+mirnawong1@users.noreply.github.com> --- .../how-we-structure/5-the-rest-of-the-project.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/website/docs/best-practices/how-we-structure/5-the-rest-of-the-project.md b/website/docs/best-practices/how-we-structure/5-the-rest-of-the-project.md index 172dbd149b4..f60e0c844f2 100644 --- a/website/docs/best-practices/how-we-structure/5-the-rest-of-the-project.md +++ b/website/docs/best-practices/how-we-structure/5-the-rest-of-the-project.md @@ -102,7 +102,9 @@ We’ve focused heavily thus far on the primary area of action in our dbt projec ### Project splitting -One important, growing consideration in the analytics engineering ecosystem is how and when to split a codebase into multiple dbt projects. Currently, our advice for most teams, especially those just starting out, is fairly simple: in most cases, we recommend doing so! With the introduction of [dbt Mesh](/best-practices/how-we-mesh/mesh-1-intro), organizations can more effectively handle complexity by utilizing multiple interconnected dbt projects rather than a single large, monolithic one. This approach is designed to speed up development while maintaining governance. As breaking up monolithic dbt projects into smaller, connected projects, potentially within a modern mono repo becomes easier, the scenarios we currently advise against may soon become feasible. So watch this space! +One important, growing consideration in the analytics engineering ecosystem is how and when to split a codebase into multiple dbt projects. Currently, our advice for most teams, especially those just starting, is fairly simple: in most cases, we recommend doing so with [dbt Mesh](/best-practices/how-we-mesh/mesh-1-intro)! dbt Mesh allows organizations to handle complexity by connecting several dbt projects rather than relying on one big, monolithic project. This approach is designed to speed up development while maintaining governance. + +As breaking up monolithic dbt projects into smaller, connected projects, potentially within a modern mono repo becomes easier, the scenarios we currently advise against may soon become feasible. So watch this space! - ✅ **Business groups or departments.** Conceptual separations within the project are the primary reason to split up your project. This allows your business domains to own their own data products and still collaborate using dbt Mesh. For more information about dbt Mesh, please refer to our [dbt Mesh FAQs](/best-practices/how-we-mesh/mesh-5-faqs). - ❌ **ML vs Reporting use cases.** Similarly to the point above, splitting a project up based on different use cases, particularly more standard BI versus ML features, is a common idea. We tend to discourage it for the time being. As with the previous point, a foundational goal of implementing dbt is to create a single source of truth in your organization. The features you’re providing to your data science teams should be coming from the same marts and metrics that serve reports on executive dashboards. From f40d5b3d1ea7e1a456f0e6c1ad0068e1749ae9bb Mon Sep 17 00:00:00 2001 From: Natalie Fiann Date: Tue, 10 Sep 2024 21:12:59 +0100 Subject: [PATCH 50/74] Changed order of paragraph around --- .../how-we-structure/5-the-rest-of-the-project.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/best-practices/how-we-structure/5-the-rest-of-the-project.md b/website/docs/best-practices/how-we-structure/5-the-rest-of-the-project.md index 07489ab67cd..117a78d8c67 100644 --- a/website/docs/best-practices/how-we-structure/5-the-rest-of-the-project.md +++ b/website/docs/best-practices/how-we-structure/5-the-rest-of-the-project.md @@ -105,9 +105,9 @@ We’ve focused heavily thus far on the primary area of action in our dbt projec One important, growing consideration in the analytics engineering ecosystem is how and when to split a codebase into multiple dbt projects. Currently, our advice for most teams, especially those just starting out, is fairly simple: in most cases, we recommend doing so! With the introduction of [dbt Mesh](/best-practices/how-we-mesh/mesh-1-intro), organizations can more effectively handle complexity by utilizing multiple interconnected dbt projects rather than a single large, monolithic one. This approach is designed to speed up development while maintaining governance. As breaking up monolithic dbt projects into smaller, connected projects, potentially within a modern mono repo becomes easier, the scenarios we currently advise against may soon become feasible. So watch this space! - ✅ **Business groups or departments.** Conceptual separations within the project are a good reason to split up your project. For more information about dbt mesh, please refer to our [dbt Mesh FAQs](/best-practices/how-we-mesh/mesh-5-faqs). -- ❌ **ML vs Reporting use cases.** Similarly to the point above, splitting a project up based on different use cases, particularly more standard BI versus ML features, is a common idea. We tend to discourage it for the time being. As with the previous point, a foundational goal of implementing dbt is to create a single source of truth in your organization. The features you’re providing to your data science teams should be coming from the same marts and metrics that serve reports on executive dashboards. - ✅ **Data governance.** Structural, organizational needs — such as data governance and security — are one of the few worthwhile reasons to split up a project. If, for instance, you work at a healthcare company with only a small team cleared to access raw data with PII in it, you may need to split out your staging models into their own projects to preserve those policies. In that case, you would import your staging project into the project that builds on those staging models as a [private package](https://docs.getdbt.com/docs/build/packages/#private-packages). - ✅ **Project size.** At a certain point, your project may grow to have simply too many models to present a viable development experience. If you have 1000s of models, it absolutely makes sense to find a way to split up your project. +- ❌ **ML vs Reporting use cases.** Similarly to the point above, splitting a project up based on different use cases, particularly more standard BI versus ML features, is a common idea. We tend to discourage it for the time being. As with the previous point, a foundational goal of implementing dbt is to create a single source of truth in your organization. The features you’re providing to your data science teams should be coming from the same marts and metrics that serve reports on executive dashboards. ## Final considerations From 63c9b4630ea44afd81e5d1120090de023c5ba759 Mon Sep 17 00:00:00 2001 From: nataliefiann <120089939+nataliefiann@users.noreply.github.com> Date: Tue, 10 Sep 2024 21:25:54 +0100 Subject: [PATCH 51/74] Update website/docs/best-practices/how-we-structure/5-the-rest-of-the-project.md --- .../best-practices/how-we-structure/5-the-rest-of-the-project.md | 1 - 1 file changed, 1 deletion(-) diff --git a/website/docs/best-practices/how-we-structure/5-the-rest-of-the-project.md b/website/docs/best-practices/how-we-structure/5-the-rest-of-the-project.md index 81151582c17..9358b507acc 100644 --- a/website/docs/best-practices/how-we-structure/5-the-rest-of-the-project.md +++ b/website/docs/best-practices/how-we-structure/5-the-rest-of-the-project.md @@ -107,7 +107,6 @@ One important, growing consideration in the analytics engineering ecosystem is h As breaking up monolithic dbt projects into smaller, connected projects, potentially within a modern mono repo becomes easier, the scenarios we currently advise against may soon become feasible. So watch this space! - ✅ **Business groups or departments.** Conceptual separations within the project are the primary reason to split up your project. This allows your business domains to own their own data products and still collaborate using dbt Mesh. For more information about dbt Mesh, please refer to our [dbt Mesh FAQs](/best-practices/how-we-mesh/mesh-5-faqs). -- ❌ **ML vs Reporting use cases.** Similarly to the point above, splitting a project up based on different use cases, particularly more standard BI versus ML features, is a common idea. We tend to discourage it for the time being. As with the previous point, a foundational goal of implementing dbt is to create a single source of truth in your organization. The features you’re providing to your data science teams should be coming from the same marts and metrics that serve reports on executive dashboards. - ✅ **Data governance.** Structural, organizational needs — such as data governance and security — are one of the few worthwhile reasons to split up a project. If, for instance, you work at a healthcare company with only a small team cleared to access raw data with PII in it, you may need to split out your staging models into their own projects to preserve those policies. In that case, you would import your staging project into the project that builds on those staging models as a [private package](https://docs.getdbt.com/docs/build/packages/#private-packages). - ✅ **Project size.** At a certain point, your project may grow to have simply too many models to present a viable development experience. If you have 1000s of models, it absolutely makes sense to find a way to split up your project. - ❌ **ML vs Reporting use cases.** Similarly to the point above, splitting a project up based on different use cases, particularly more standard BI versus ML features, is a common idea. We tend to discourage it for the time being. As with the previous point, a foundational goal of implementing dbt is to create a single source of truth in your organization. The features you’re providing to your data science teams should be coming from the same marts and metrics that serve reports on executive dashboards. From 1207ce20f26acf13c0a29da846e38b80bd8355c3 Mon Sep 17 00:00:00 2001 From: nataliefiann <120089939+nataliefiann@users.noreply.github.com> Date: Wed, 11 Sep 2024 12:27:47 +0100 Subject: [PATCH 52/74] Update website/docs/reference/resource-properties/constraints.md --- website/docs/reference/resource-properties/constraints.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/reference/resource-properties/constraints.md b/website/docs/reference/resource-properties/constraints.md index f047ecadfff..229d83bbc58 100644 --- a/website/docs/reference/resource-properties/constraints.md +++ b/website/docs/reference/resource-properties/constraints.md @@ -551,7 +551,7 @@ models: data_type: int constraints: - type: custom - expression: tag (my_tag = 'my_value') # A custom SQL expression used to enforce a specific constraint on a column. + expression: "tag (my_tag = 'my_value')" # A custom SQL expression used to enforce a specific constraint on a column. ``` From b7cd5067c76a17fde71625c3898740563163052b Mon Sep 17 00:00:00 2001 From: nataliefiann <120089939+nataliefiann@users.noreply.github.com> Date: Wed, 11 Sep 2024 13:37:16 +0100 Subject: [PATCH 53/74] Update website/docs/reference/resource-properties/constraints.md --- website/docs/reference/resource-properties/constraints.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/reference/resource-properties/constraints.md b/website/docs/reference/resource-properties/constraints.md index 229d83bbc58..3a4d432dfe8 100644 --- a/website/docs/reference/resource-properties/constraints.md +++ b/website/docs/reference/resource-properties/constraints.md @@ -521,7 +521,7 @@ alter table schema_name.my_model add constraint 472394792387497234 check (id > 0 ### Custom constraints on models for advanced configuration of tables -In dbt Cloud, you can use custom constraints on models for the advanced configuration of tables. Different data warehouses support different syntax and capabilities. However, custom constraints allow you to add configuration to specific columns. +In dbt Cloud and dbt Core, you can use custom constraints on models for the advanced configuration of tables. Different data warehouses support different syntax and capabilities. However, custom constraints allow you to add configuration to specific columns. - For example, custom constraints allow you to set [masking policies](https://docs.snowflake.com/en/user-guide/security-column-intro#what-are-masking-policies) in Snowflake when using a Create Table As Select (CTAS). From d7eb83035f145aca1a3f3e2caa6279acfcd5ab38 Mon Sep 17 00:00:00 2001 From: nataliefiann <120089939+nataliefiann@users.noreply.github.com> Date: Wed, 11 Sep 2024 13:51:30 +0100 Subject: [PATCH 54/74] Update website/docs/reference/resource-properties/constraints.md Co-authored-by: Mirna Wong <89008547+mirnawong1@users.noreply.github.com> --- website/docs/reference/resource-properties/constraints.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/reference/resource-properties/constraints.md b/website/docs/reference/resource-properties/constraints.md index 3a4d432dfe8..880b56b6710 100644 --- a/website/docs/reference/resource-properties/constraints.md +++ b/website/docs/reference/resource-properties/constraints.md @@ -519,7 +519,7 @@ alter table schema_name.my_model add constraint 472394792387497234 check (id > 0 -### Custom constraints on models for advanced configuration of tables +### Custom constraints In dbt Cloud and dbt Core, you can use custom constraints on models for the advanced configuration of tables. Different data warehouses support different syntax and capabilities. However, custom constraints allow you to add configuration to specific columns. From 6f7b622081718e8cf88ae98e1143fb6120a24d3f Mon Sep 17 00:00:00 2001 From: nataliefiann <120089939+nataliefiann@users.noreply.github.com> Date: Wed, 11 Sep 2024 13:51:45 +0100 Subject: [PATCH 55/74] Update website/docs/reference/resource-properties/constraints.md Co-authored-by: Mirna Wong <89008547+mirnawong1@users.noreply.github.com> --- website/docs/reference/resource-properties/constraints.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/website/docs/reference/resource-properties/constraints.md b/website/docs/reference/resource-properties/constraints.md index 880b56b6710..3c0b3683ab3 100644 --- a/website/docs/reference/resource-properties/constraints.md +++ b/website/docs/reference/resource-properties/constraints.md @@ -521,7 +521,9 @@ alter table schema_name.my_model add constraint 472394792387497234 check (id > 0 ### Custom constraints -In dbt Cloud and dbt Core, you can use custom constraints on models for the advanced configuration of tables. Different data warehouses support different syntax and capabilities. However, custom constraints allow you to add configuration to specific columns. +In dbt Cloud and dbt Core, you can use custom constraints on models for the advanced configuration of tables. Different data warehouses support different syntax and capabilities. + +Custom constraints allow you to add configuration to specific columns: - For example, custom constraints allow you to set [masking policies](https://docs.snowflake.com/en/user-guide/security-column-intro#what-are-masking-policies) in Snowflake when using a Create Table As Select (CTAS). From b72d530a0c71ede771f5261ced43338acaae76b7 Mon Sep 17 00:00:00 2001 From: nataliefiann <120089939+nataliefiann@users.noreply.github.com> Date: Wed, 11 Sep 2024 14:02:37 +0100 Subject: [PATCH 56/74] Update website/docs/reference/resource-properties/constraints.md Co-authored-by: Mirna Wong <89008547+mirnawong1@users.noreply.github.com> --- website/docs/reference/resource-properties/constraints.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/reference/resource-properties/constraints.md b/website/docs/reference/resource-properties/constraints.md index 3c0b3683ab3..a53842dc5e9 100644 --- a/website/docs/reference/resource-properties/constraints.md +++ b/website/docs/reference/resource-properties/constraints.md @@ -544,7 +544,7 @@ Here's an example of how to implement tag-based masking policies with contracts models: - name: my_model - config: null + config: contract: enforced: true materialized: table From 261d61f12ced36ec78f10c263bbba42334d765a3 Mon Sep 17 00:00:00 2001 From: nataliefiann <120089939+nataliefiann@users.noreply.github.com> Date: Wed, 11 Sep 2024 14:03:24 +0100 Subject: [PATCH 57/74] Update website/docs/reference/resource-properties/constraints.md Co-authored-by: Mirna Wong <89008547+mirnawong1@users.noreply.github.com> --- website/docs/reference/resource-properties/constraints.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/reference/resource-properties/constraints.md b/website/docs/reference/resource-properties/constraints.md index a53842dc5e9..34bc56b947a 100644 --- a/website/docs/reference/resource-properties/constraints.md +++ b/website/docs/reference/resource-properties/constraints.md @@ -523,7 +523,7 @@ alter table schema_name.my_model add constraint 472394792387497234 check (id > 0 In dbt Cloud and dbt Core, you can use custom constraints on models for the advanced configuration of tables. Different data warehouses support different syntax and capabilities. -Custom constraints allow you to add configuration to specific columns: +Custom constraints allow you to add configuration to specific columns. For example: - For example, custom constraints allow you to set [masking policies](https://docs.snowflake.com/en/user-guide/security-column-intro#what-are-masking-policies) in Snowflake when using a Create Table As Select (CTAS). From 42746780b317074fcfc8a624c1a90b6b9863a24b Mon Sep 17 00:00:00 2001 From: nataliefiann <120089939+nataliefiann@users.noreply.github.com> Date: Wed, 11 Sep 2024 14:03:39 +0100 Subject: [PATCH 58/74] Update website/docs/reference/resource-properties/constraints.md Co-authored-by: Mirna Wong <89008547+mirnawong1@users.noreply.github.com> --- website/docs/reference/resource-properties/constraints.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/reference/resource-properties/constraints.md b/website/docs/reference/resource-properties/constraints.md index 34bc56b947a..0e7c3d25cc1 100644 --- a/website/docs/reference/resource-properties/constraints.md +++ b/website/docs/reference/resource-properties/constraints.md @@ -525,7 +525,7 @@ In dbt Cloud and dbt Core, you can use custom constraints on models for the adva Custom constraints allow you to add configuration to specific columns. For example: - - For example, custom constraints allow you to set [masking policies](https://docs.snowflake.com/en/user-guide/security-column-intro#what-are-masking-policies) in Snowflake when using a Create Table As Select (CTAS). + - Set [masking policies](https://docs.snowflake.com/en/user-guide/security-column-intro#what-are-masking-policies) in Snowflake when using a Create Table As Select (CTAS). - [Databricks](https://docs.databricks.com/en/sql/language-manual/sql-ref-syntax-ddl-create-table-using.html) and [BigQuery](https://cloud.google.com/bigquery/docs/reference/standard-sql/data-definition-language#column_name_and_column_schema) are other examples of data warehouses having their own set of parameters that can be set for columns in their CTAS statements. From 86038b67d5c617eaf5268b0104d3e2ae1c570fbc Mon Sep 17 00:00:00 2001 From: nataliefiann <120089939+nataliefiann@users.noreply.github.com> Date: Wed, 11 Sep 2024 14:04:18 +0100 Subject: [PATCH 59/74] Update website/docs/reference/resource-properties/constraints.md Co-authored-by: Mirna Wong <89008547+mirnawong1@users.noreply.github.com> --- website/docs/reference/resource-properties/constraints.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/reference/resource-properties/constraints.md b/website/docs/reference/resource-properties/constraints.md index 0e7c3d25cc1..99a232f4c23 100644 --- a/website/docs/reference/resource-properties/constraints.md +++ b/website/docs/reference/resource-properties/constraints.md @@ -527,7 +527,7 @@ Custom constraints allow you to add configuration to specific columns. For examp - Set [masking policies](https://docs.snowflake.com/en/user-guide/security-column-intro#what-are-masking-policies) in Snowflake when using a Create Table As Select (CTAS). - - [Databricks](https://docs.databricks.com/en/sql/language-manual/sql-ref-syntax-ddl-create-table-using.html) and [BigQuery](https://cloud.google.com/bigquery/docs/reference/standard-sql/data-definition-language#column_name_and_column_schema) are other examples of data warehouses having their own set of parameters that can be set for columns in their CTAS statements. + - Other data warehouses (such as [Databricks](https://docs.databricks.com/en/sql/language-manual/sql-ref-syntax-ddl-create-table-using.html) and [BigQuery](https://cloud.google.com/bigquery/docs/reference/standard-sql/data-definition-language#column_name_and_column_schema)) have their own set of parameters that can be set for columns in their CTAS statements. You can implement constraints in a couple of different ways: From a9b4418a3b5f95add69bf0c802b85d6a80f5c2d3 Mon Sep 17 00:00:00 2001 From: nataliefiann <120089939+nataliefiann@users.noreply.github.com> Date: Wed, 11 Sep 2024 14:04:44 +0100 Subject: [PATCH 60/74] Update website/docs/reference/resource-properties/constraints.md --- website/docs/reference/resource-properties/constraints.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/reference/resource-properties/constraints.md b/website/docs/reference/resource-properties/constraints.md index 99a232f4c23..54391ed58f4 100644 --- a/website/docs/reference/resource-properties/constraints.md +++ b/website/docs/reference/resource-properties/constraints.md @@ -527,7 +527,7 @@ Custom constraints allow you to add configuration to specific columns. For examp - Set [masking policies](https://docs.snowflake.com/en/user-guide/security-column-intro#what-are-masking-policies) in Snowflake when using a Create Table As Select (CTAS). - - Other data warehouses (such as [Databricks](https://docs.databricks.com/en/sql/language-manual/sql-ref-syntax-ddl-create-table-using.html) and [BigQuery](https://cloud.google.com/bigquery/docs/reference/standard-sql/data-definition-language#column_name_and_column_schema)) have their own set of parameters that can be set for columns in their CTAS statements. + - Other data warehouses (such as [Databricks](https://docs.databricks.com/en/sql/language-manual/sql-ref-syntax-ddl-create-table-using.html) and [BigQuery](https://cloud.google.com/bigquery/docs/reference/standard-sql/data-definition-language#column_name_and_column_schema) have their own set of parameters that can be set for columns in their CTAS statements. You can implement constraints in a couple of different ways: From c9b93bf24b6ed7e262137d73f92d1244ed96b304 Mon Sep 17 00:00:00 2001 From: Mirna Wong <89008547+mirnawong1@users.noreply.github.com> Date: Thu, 12 Sep 2024 17:58:57 +0100 Subject: [PATCH 61/74] Update website/docs/reference/resource-properties/constraints.md --- website/docs/reference/resource-properties/constraints.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/reference/resource-properties/constraints.md b/website/docs/reference/resource-properties/constraints.md index 54391ed58f4..12691158e73 100644 --- a/website/docs/reference/resource-properties/constraints.md +++ b/website/docs/reference/resource-properties/constraints.md @@ -519,7 +519,7 @@ alter table schema_name.my_model add constraint 472394792387497234 check (id > 0 -### Custom constraints +## Custom constraints In dbt Cloud and dbt Core, you can use custom constraints on models for the advanced configuration of tables. Different data warehouses support different syntax and capabilities. From 44b5c24415b7b01ca9a0cd60642567ccb2522138 Mon Sep 17 00:00:00 2001 From: Doug Beatty Date: Thu, 3 Oct 2024 20:13:56 -0600 Subject: [PATCH 62/74] `state_modified_compare_more_unrendered_values` behavior change flag --- website/docs/docs/dbt-versions/release-notes.md | 2 +- website/docs/reference/global-configs/behavior-changes.md | 4 ++-- .../docs/reference/node-selection/state-comparison-caveats.md | 4 ++-- 3 files changed, 5 insertions(+), 5 deletions(-) diff --git a/website/docs/docs/dbt-versions/release-notes.md b/website/docs/docs/dbt-versions/release-notes.md index 96e8a7de37a..9030ca8e722 100644 --- a/website/docs/docs/dbt-versions/release-notes.md +++ b/website/docs/docs/dbt-versions/release-notes.md @@ -26,7 +26,7 @@ Release notes are grouped by month for both multi-tenant and virtual private clo - **New**: In dbt Cloud Versionless, [Snapshots](/docs/build/snapshots) have been updated to use YAML configuration files instead of SQL snapshot blocks. This new feature simplifies snapshot management and improves performance, and will soon be released in dbt Core 1.9. - Who does this affect? New user on Versionless can define snapshots using the new YAML specification. Users upgrading to Versionless who use snapshots can keep their existing configuration or can choose to migrate their snapshot definitions to YAML. - Users on dbt 1.8 and earlier: No action is needed; existing snapshots will continue to work as before. However, we recommend upgrading to Versionless to take advantage of the new snapshot features. -- **Behavior change:** Set [`state_modified_compare_more_unrendered`](/reference/global-configs/behavior-changes#source-definitions-for-state) to true to reduce false positives for `state:modified` when configs differ between `dev` and `prod` environments. +- **Behavior change:** Set [`state_modified_compare_more_unrendered_values`](/reference/global-configs/behavior-changes#source-definitions-for-state) to true to reduce false positives for `state:modified` when configs differ between `dev` and `prod` environments. - **Behavior change:** Set the [`skip_nodes_if_on_run_start_fails`](/reference/global-configs/behavior-changes#failures-in-on-run-start-hooks) flag to `True` to skip all selected resources from running if there is a failure on an `on-run-start` hook. - **Enhancement**: In dbt Cloud Versionless, snapshots defined in SQL files can now use `config` defined in `schema.yml` YAML files. This update resolves the previous limitation that required snapshot properties to be defined exclusively in `dbt_project.yml` and/or a `config()` block within the SQL file. This will also be released in dbt Core 1.9. - **New**: In dbt Cloud Versionless, the `snapshot_meta_column_names` config allows for customizing the snapshot metadata columns. This feature allows an organization to align these automatically-generated column names with their conventions, and will be included in the upcoming dbt Core 1.9 release. diff --git a/website/docs/reference/global-configs/behavior-changes.md b/website/docs/reference/global-configs/behavior-changes.md index f7c2344ae05..d35b83765e3 100644 --- a/website/docs/reference/global-configs/behavior-changes.md +++ b/website/docs/reference/global-configs/behavior-changes.md @@ -69,7 +69,7 @@ When we use dbt Cloud in the following table, we're referring to accounts that h | source_freshness_run_project_hooks | 2024.03 | TBD* | 1.8.0 | 1.9.0 | | [Redshift] [restrict_direct_pg_catalog_access](/reference/global-configs/redshift-changes#the-restrict_direct_pg_catalog_access-flag) | 2024.09 | TBD* | dbt-redshift v1.9.0 | 1.9.0 | | skip_nodes_if_on_run_start_fails | 2024.10 | TBD* | 1.9.0 | TBD* | -| state_modified_compare_more_unrendered | 2024.10 | TBD* | 1.9.0 | TBD* | +| state_modified_compare_more_unrendered_values | 2024.10 | TBD* | 1.9.0 | TBD* | When the dbt Cloud Maturity is "TBD," it means we have not yet determined the exact date when these flags' default values will change. Affected users will see deprecation warnings in the meantime, and they will receive emails providing advance warning ahead of the maturity date. In the meantime, if you are seeing a deprecation warning, you can either: - Migrate your project to support the new behavior, and then set the flag to `True` to stop seeing the warnings. @@ -85,7 +85,7 @@ Set the `skip_nodes_if_on_run_start_fails` flag to `True` to skip all selected r The flag is `False` by default. -Set `state_modified_compare_more_unrendered` to `True` to reduce false positives during `state:modified` checks (especially when configs differ by target environment like `prod` vs. `dev`). +Set `state_modified_compare_more_unrendered_values` to `True` to reduce false positives during `state:modified` checks (especially when configs differ by target environment like `prod` vs. `dev`). Setting the flag to `True` changes the `state:modified` comparison from using rendered values to unrendered values instead. It accomplishes this by persisting `unrendered_config` during model parsing and `unrendered_database` and `unrendered_schema` configs during source parsing. diff --git a/website/docs/reference/node-selection/state-comparison-caveats.md b/website/docs/reference/node-selection/state-comparison-caveats.md index 4d5593b7331..25301656539 100644 --- a/website/docs/reference/node-selection/state-comparison-caveats.md +++ b/website/docs/reference/node-selection/state-comparison-caveats.md @@ -46,7 +46,7 @@ dbt test -s "state:modified" --exclude "test_name:relationships" -To reduce false positives during `state:modified` selection due to env-aware logic, you can set the `state_modified_compare_more_unrendered` [behavior flag](/reference/global-configs/behavior-changes#behavior-change-flags) to `True`. +To reduce false positives during `state:modified` selection due to env-aware logic, you can set the `state_modified_compare_more_unrendered_values` [behavior flag](/reference/global-configs/behavior-changes#behavior-change-flags) to `True`. @@ -54,7 +54,7 @@ To reduce false positives during `state:modified` selection due to env-aware log State comparison works by identifying discrepancies between two manifests. Those discrepancies could be the result of: 1. Changes made to a project in development -2. Env-aware logic that causes different behavior based on the `target`, env vars, etc., which can be avoided if you upgrade to dbt Core 1.9 and set the `state_modified_compare_more_unrendered` [behavior flag](/reference/global-configs/behavior-changes#behavior-change-flags) to `True`. +2. Env-aware logic that causes different behavior based on the `target`, env vars, etc., which can be avoided if you upgrade to dbt Core 1.9 and set the `state_modified_compare_more_unrendered_values` [behavior flag](/reference/global-configs/behavior-changes#behavior-change-flags) to `True`. State comparison detects env-aware config in `dbt_project.yml`. This target-based config won't register as a modification: From ec8fa541db29bba5e1c3f8b1b8bdf118b514c6c3 Mon Sep 17 00:00:00 2001 From: Matt Shaver <60105315+matthewshaver@users.noreply.github.com> Date: Thu, 3 Oct 2024 22:50:48 -0400 Subject: [PATCH 63/74] Adding version blocks to iceberg (#6230) Adding version blocks to Iceberg for Snowflake content ## Checklist - [ ] I have reviewed the [Content style guide](https://github.com/dbt-labs/docs.getdbt.com/blob/current/contributing/content-style-guide.md) so my content adheres to these guidelines. - [ ] The topic I'm writing about is for specific dbt version(s) and I have versioned it according to the [version a whole page](https://github.com/dbt-labs/docs.getdbt.com/blob/current/contributing/single-sourcing-content.md#adding-a-new-version) and/or [version a block of content](https://github.com/dbt-labs/docs.getdbt.com/blob/current/contributing/single-sourcing-content.md#versioning-blocks-of-content) guidelines. - [ ] I have added checklist item(s) to this list for anything anything that needs to happen before this PR is merged, such as "needs technical review" or "change base branch." --- website/docs/reference/resource-configs/snowflake-configs.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/website/docs/reference/resource-configs/snowflake-configs.md b/website/docs/reference/resource-configs/snowflake-configs.md index b0a0ca18bea..342e8290458 100644 --- a/website/docs/reference/resource-configs/snowflake-configs.md +++ b/website/docs/reference/resource-configs/snowflake-configs.md @@ -9,6 +9,8 @@ To-do: - use the reference doc structure for this article / split into separate articles ---> + + ## Iceberg table format The dbt-snowflake adapter supports the Iceberg table format. It is available for three of the Snowflake materializations: @@ -95,6 +97,7 @@ There are some limitations to the implementation you need to be aware of: - Using Iceberg tables with dbt, the result is that your query is materialized in Iceberg. However, often, dbt creates intermediary objects as temporary and transient tables for certain materializations, such as incremental ones. It is not possible to configure these temporary objects also to be Iceberg-formatted. You may see non-Iceberg tables created in the logs to support specific materializations, but they will be dropped after usage. - You cannot incrementally update a preexisting incremental model to be an Iceberg table. To do so, you must fully rebuild the table with the `--full-refresh` flag. + ## Dynamic tables From 3ae24bd57d0f80dc6d8e7c59feb61f96d236cadc Mon Sep 17 00:00:00 2001 From: Mirna Wong <89008547+mirnawong1@users.noreply.github.com> Date: Fri, 4 Oct 2024 09:36:13 +0100 Subject: [PATCH 64/74] Update constraints.md --- .../docs/reference/resource-properties/constraints.md | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/website/docs/reference/resource-properties/constraints.md b/website/docs/reference/resource-properties/constraints.md index de429ea71bf..353e56cd1f8 100644 --- a/website/docs/reference/resource-properties/constraints.md +++ b/website/docs/reference/resource-properties/constraints.md @@ -15,7 +15,7 @@ Constraints require the declaration and enforcement of a model [contract](/refer Constraints may be defined for a single column, or at the model level for one or more columns. As a general rule, we recommend defining single-column constraints directly on those columns. -If you are defining multiple `primary_key` constraints for a single model, those _must_ be defined at the model level. Defining multiple `primary_key` constraints at the column level is not supported. +If you define multiple `primary_key` constraints for a single model, those _must_ be defined at the model level. Defining multiple `primary_key` constraints at the column level is not supported. The structure of a constraint is: - `type` (required): one of `not_null`, `unique`, `primary_key`, `foreign_key`, `check`, `custom` @@ -588,7 +588,8 @@ You can implement constraints in a couple of different ways: - [Custom constraints with tags](#custom-constraints-with-tags) - [Custom constraints without tags](#custom-constraints-without-tags) -### Custom constraints with tags + + Here's an example of how to implement tag-based masking policies with contracts and constraints using the following syntax: @@ -616,9 +617,9 @@ models: Using this syntax requires configuring all the columns and their types as it’s the only way to send a create or replace ` mytable as ...`. It’s not possible to do it with just a partial list of columns. This means making sure the columns and constraints fields are fully defined. To generate a YAML with all the columns, you can use `generate_model_yaml` from [dbt-codegen](https://github.com/dbt-labs/dbt-codegen/tree/0.12.1/?tab=readme-ov-file#generate_model_yaml-source). + - -### Custom constraints without tags + Alternatively, you can add a masking policy without tags: @@ -642,5 +643,5 @@ models: ``` - + From 06384c611188838795be6617dcd17ed43aacc3bd Mon Sep 17 00:00:00 2001 From: Mirna Wong <89008547+mirnawong1@users.noreply.github.com> Date: Fri, 4 Oct 2024 09:44:13 +0100 Subject: [PATCH 65/74] Update constraints.md --- website/docs/reference/resource-properties/constraints.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/reference/resource-properties/constraints.md b/website/docs/reference/resource-properties/constraints.md index 353e56cd1f8..4dced11f337 100644 --- a/website/docs/reference/resource-properties/constraints.md +++ b/website/docs/reference/resource-properties/constraints.md @@ -619,7 +619,7 @@ Using this syntax requires configuring all the columns and their types as it’s To generate a YAML with all the columns, you can use `generate_model_yaml` from [dbt-codegen](https://github.com/dbt-labs/dbt-codegen/tree/0.12.1/?tab=readme-ov-file#generate_model_yaml-source). - + Alternatively, you can add a masking policy without tags: From 06bf37d154702208a41a51c7bc6acc1d26910fb3 Mon Sep 17 00:00:00 2001 From: Mirna Wong <89008547+mirnawong1@users.noreply.github.com> Date: Fri, 4 Oct 2024 14:19:34 +0100 Subject: [PATCH 66/74] Update constraints.md remove redundant bullets --- website/docs/reference/resource-properties/constraints.md | 3 --- 1 file changed, 3 deletions(-) diff --git a/website/docs/reference/resource-properties/constraints.md b/website/docs/reference/resource-properties/constraints.md index 4dced11f337..ed38132c367 100644 --- a/website/docs/reference/resource-properties/constraints.md +++ b/website/docs/reference/resource-properties/constraints.md @@ -586,9 +586,6 @@ Custom constraints allow you to add configuration to specific columns. For examp You can implement constraints in a couple of different ways: -- [Custom constraints with tags](#custom-constraints-with-tags) -- [Custom constraints without tags](#custom-constraints-without-tags) - Here's an example of how to implement tag-based masking policies with contracts and constraints using the following syntax: From 9845a0579a4c2c5d55d9135dd90cb3dc0eb71d5a Mon Sep 17 00:00:00 2001 From: mirnawong1 Date: Fri, 4 Oct 2024 15:07:26 +0100 Subject: [PATCH 67/74] fix typos --- website/docs/docs/build/snapshots.md | 30 ++++++++++++++++--------- website/snippets/_snapshot-yaml-spec.md | 4 +++- 2 files changed, 22 insertions(+), 12 deletions(-) diff --git a/website/docs/docs/build/snapshots.md b/website/docs/docs/build/snapshots.md index c17350ab368..0762ea7b086 100644 --- a/website/docs/docs/build/snapshots.md +++ b/website/docs/docs/build/snapshots.md @@ -52,20 +52,25 @@ It is not possible to "preview data" or "compile sql" for snapshots in dbt Cloud -In dbt Cloud Versionless and dbt Core v1.9 and later, snapshots are configurations defined in YAML files (typically in your snapshots directory). You'll configure your snapshot to tell dbt how to detect record changes. +Configure your snapshots in YAML files (typically in your snapshots directory) to tell dbt how to detect record changes. Define snapshots configurations. Configuring in YAML files (alongside your models) provides a cleaner and more consistent set up. - + ```yaml snapshots: - - name: orders_snapshot - relation: source('jaffle_shop', 'orders') + - name: + relation: # source('my_source', 'my_table') config: - schema: snapshots - database: analytics - unique_key: id - strategy: timestamp - updated_at: updated_at + [database](/reference/resource-configs/database): + [schema](/reference/resource-configs/schema): + [alias](/reference/resource-configs/alias): + [strategy](/reference/resource-configs/strategy): timestamp | check + [unique_key](/reference/resource-configs/unique_key): + [check_cols](/reference/resource-configs/check_cols): [] | all + [updated_at](/reference/resource-configs/updated_at): + [invalidate_hard_deletes](/reference/resource-configs/invalidate_hard_deletes): true | false + [snapshot_meta_column_names](/reference/resource-configs/snapshot_meta_column_names): {} + ``` @@ -82,6 +87,7 @@ The following table outlines the configurations available for snapshots: | [check_cols](/reference/resource-configs/check_cols) | If using the `check` strategy, then the columns to check | Only if using the `check` strategy | ["status"] | | [updated_at](/reference/resource-configs/updated_at) | If using the `timestamp` strategy, the timestamp column to compare | Only if using the `timestamp` strategy | updated_at | | [invalidate_hard_deletes](/reference/resource-configs/invalidate_hard_deletes) | Find hard deleted records in source and set `dbt_valid_to` to current time if the record no longer exists | No | True | +| [snapshot_meta_column_names](/reference/resource-configs/snapshot_meta_column_names) | Customize the names of the snapshot meta fields | No | {} | - In versions prior to v1.9, the `target_schema` (required) and `target_database` (optional) configurations defined a single schema or database to build a snapshot across users and environment. This created problems when testing or developing a snapshot, as there was no clear separation between development and production environments. In v1.9, `target_schema` became optional, allowing snapshots to be environment-aware. By default, without `target_schema` or `target_database` defined, snapshots now use the `generate_schema_name` or `generate_database_name` macros to determine where to build. Developers can still set a custom location with [`schema`](/reference/resource-configs/schema) and [`database`](/reference/resource-configs/database) configs, consistent with other resource types. - A number of other configurations are also supported (for example, `tags` and `post-hook`). For the complete list, refer to [Snapshot configurations](/reference/snapshot-configs). @@ -160,7 +166,7 @@ To add a snapshot to your project follow these steps. For users on versions 1.8 ### Configuration best practices - + This strategy handles column additions and deletions better than the `check` strategy. @@ -188,7 +194,7 @@ Snapshots can't be rebuilt. Because of this, it's a good idea to put snapshots i - + If you need to clean or transform your data before snapshotting, create an ephemeral model (or a staging model) that applies the necessary transformations. Then, reference this model in your snapshot configuration. This approach keeps your snapshot definitions clean and allows you to test and run transformations separately. @@ -203,6 +209,8 @@ When you run the [`dbt snapshot` command](/reference/commands/snapshot): - The `dbt_valid_to` column will be updated for any existing records that have changed - The updated record and any new records will be inserted into the snapshot table. These records will now have `dbt_valid_to = null` +Note, these column names can be customized to your team or organizational conventions using the [snapshot_meta_column_names](#snapshot-meta-fields) config. + Snapshots can be referenced in downstream models the same way as referencing models — by using the [ref](/reference/dbt-jinja-functions/ref) function. ## Detecting row changes diff --git a/website/snippets/_snapshot-yaml-spec.md b/website/snippets/_snapshot-yaml-spec.md index 8bbdc6be72e..cb1675ce5bd 100644 --- a/website/snippets/_snapshot-yaml-spec.md +++ b/website/snippets/_snapshot-yaml-spec.md @@ -1,4 +1,6 @@ :::info Use the latest snapshot syntax -In Versionless and dbt v1.9 and later, snapshots are defined in an updated syntax using a YAML file within your `snapshots/` directory (as defined by the [`snapshot-paths` config](/reference/project-configs/snapshot-paths)). For faster and more efficient management, consider the updated snapshot YAML syntax, [available in Versionless](/docs/dbt-versions/versionless-cloud) or [dbt Core v1.9 and later](/docs/dbt-versions/core). +In [dbt Cloud Versionless](/docs/dbt-versions/versionless-cloud) or [dbt Core v1.9 and later](/docs/dbt-versions/core), you can configure snapshots in YAML files using the updated syntax within your `snapshots/` directory (as defined by the [`snapshot-paths` config](/reference/project-configs/snapshot-paths)). + +This syntax allows for faster, more efficient snapshot management. To use it, upgrade to Versionless or dbt v1.9 or newer. ::: From e5b10a400c113afb92f9af19c0f8f6f787920d9a Mon Sep 17 00:00:00 2001 From: mirnawong1 Date: Fri, 4 Oct 2024 15:09:47 +0100 Subject: [PATCH 68/74] update staging --- website/docs/docs/build/snapshots.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/docs/build/snapshots.md b/website/docs/docs/build/snapshots.md index 0762ea7b086..49f479fa1f9 100644 --- a/website/docs/docs/build/snapshots.md +++ b/website/docs/docs/build/snapshots.md @@ -196,7 +196,7 @@ Snapshots can't be rebuilt. Because of this, it's a good idea to put snapshots i - If you need to clean or transform your data before snapshotting, create an ephemeral model (or a staging model) that applies the necessary transformations. Then, reference this model in your snapshot configuration. This approach keeps your snapshot definitions clean and allows you to test and run transformations separately. + If you need to clean or transform your data before snapshotting, create an ephemeral model or a staging model that applies the necessary transformations. Then, reference this model in your snapshot configuration. This approach keeps your snapshot definitions clean and allows you to test and run transformations separately. From dcb2ce4872272402cadf28a538c1e98be516efbc Mon Sep 17 00:00:00 2001 From: mirnawong1 Date: Fri, 4 Oct 2024 15:15:15 +0100 Subject: [PATCH 69/74] update --- website/docs/docs/build/snapshots.md | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/website/docs/docs/build/snapshots.md b/website/docs/docs/build/snapshots.md index 49f479fa1f9..a83ff06de06 100644 --- a/website/docs/docs/build/snapshots.md +++ b/website/docs/docs/build/snapshots.md @@ -59,17 +59,17 @@ Configure your snapshots in YAML files (typically in your snapshots directory) t ```yaml snapshots: - name: - relation: # source('my_source', 'my_table') + relation: source> # source('my_source', 'my_table') config: - [database](/reference/resource-configs/database): - [schema](/reference/resource-configs/schema): - [alias](/reference/resource-configs/alias): + [database](/reference/resource-configs/database): string + [schema](/reference/resource-configs/schema): string + [alias](/reference/resource-configs/alias):string [strategy](/reference/resource-configs/strategy): timestamp | check - [unique_key](/reference/resource-configs/unique_key): - [check_cols](/reference/resource-configs/check_cols): [] | all - [updated_at](/reference/resource-configs/updated_at): + [unique_key](/reference/resource-configs/unique_key): column_name_or_expression + [check_cols](/reference/resource-configs/check_cols): [column_name] | all + [updated_at](/reference/resource-configs/updated_at): column_name [invalidate_hard_deletes](/reference/resource-configs/invalidate_hard_deletes): true | false - [snapshot_meta_column_names](/reference/resource-configs/snapshot_meta_column_names): {} + [snapshot_meta_column_names](/reference/resource-configs/snapshot_meta_column_names): {dictionary} ``` From a93cb2fa0849ea6ad9bfaadec4ae8bef6576c0ea Mon Sep 17 00:00:00 2001 From: mirnawong1 Date: Fri, 4 Oct 2024 15:15:44 +0100 Subject: [PATCH 70/74] remove arrows --- website/docs/docs/build/snapshots.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/website/docs/docs/build/snapshots.md b/website/docs/docs/build/snapshots.md index a83ff06de06..4b68dfb074b 100644 --- a/website/docs/docs/build/snapshots.md +++ b/website/docs/docs/build/snapshots.md @@ -58,8 +58,8 @@ Configure your snapshots in YAML files (typically in your snapshots directory) t ```yaml snapshots: - - name: - relation: source> # source('my_source', 'my_table') + - name: string + relation: source # source('my_source', 'my_table') config: [database](/reference/resource-configs/database): string [schema](/reference/resource-configs/schema): string @@ -87,7 +87,7 @@ The following table outlines the configurations available for snapshots: | [check_cols](/reference/resource-configs/check_cols) | If using the `check` strategy, then the columns to check | Only if using the `check` strategy | ["status"] | | [updated_at](/reference/resource-configs/updated_at) | If using the `timestamp` strategy, the timestamp column to compare | Only if using the `timestamp` strategy | updated_at | | [invalidate_hard_deletes](/reference/resource-configs/invalidate_hard_deletes) | Find hard deleted records in source and set `dbt_valid_to` to current time if the record no longer exists | No | True | -| [snapshot_meta_column_names](/reference/resource-configs/snapshot_meta_column_names) | Customize the names of the snapshot meta fields | No | {} | +| [snapshot_meta_column_names](/reference/resource-configs/snapshot_meta_column_names) | Customize the names of the snapshot meta fields | No | {dictionary} | - In versions prior to v1.9, the `target_schema` (required) and `target_database` (optional) configurations defined a single schema or database to build a snapshot across users and environment. This created problems when testing or developing a snapshot, as there was no clear separation between development and production environments. In v1.9, `target_schema` became optional, allowing snapshots to be environment-aware. By default, without `target_schema` or `target_database` defined, snapshots now use the `generate_schema_name` or `generate_database_name` macros to determine where to build. Developers can still set a custom location with [`schema`](/reference/resource-configs/schema) and [`database`](/reference/resource-configs/database) configs, consistent with other resource types. - A number of other configurations are also supported (for example, `tags` and `post-hook`). For the complete list, refer to [Snapshot configurations](/reference/snapshot-configs). From 0a3b4ae40764f2e2479a5e5498376d7673e69750 Mon Sep 17 00:00:00 2001 From: Mirna Wong <89008547+mirnawong1@users.noreply.github.com> Date: Fri, 4 Oct 2024 15:28:52 +0100 Subject: [PATCH 71/74] Update website/docs/docs/build/snapshots.md --- website/docs/docs/build/snapshots.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/docs/build/snapshots.md b/website/docs/docs/build/snapshots.md index 4b68dfb074b..910a58ab2cd 100644 --- a/website/docs/docs/build/snapshots.md +++ b/website/docs/docs/build/snapshots.md @@ -63,7 +63,7 @@ snapshots: config: [database](/reference/resource-configs/database): string [schema](/reference/resource-configs/schema): string - [alias](/reference/resource-configs/alias):string + [alias](/reference/resource-configs/alias): string [strategy](/reference/resource-configs/strategy): timestamp | check [unique_key](/reference/resource-configs/unique_key): column_name_or_expression [check_cols](/reference/resource-configs/check_cols): [column_name] | all From 5289c4c3f75fdbac67ed0f26ed1a3e9806fe04ab Mon Sep 17 00:00:00 2001 From: mirnawong1 Date: Fri, 4 Oct 2024 15:35:56 +0100 Subject: [PATCH 72/74] fix error --- website/docs/docs/build/snapshots.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/website/docs/docs/build/snapshots.md b/website/docs/docs/build/snapshots.md index 910a58ab2cd..c11b7c4b76c 100644 --- a/website/docs/docs/build/snapshots.md +++ b/website/docs/docs/build/snapshots.md @@ -69,7 +69,7 @@ snapshots: [check_cols](/reference/resource-configs/check_cols): [column_name] | all [updated_at](/reference/resource-configs/updated_at): column_name [invalidate_hard_deletes](/reference/resource-configs/invalidate_hard_deletes): true | false - [snapshot_meta_column_names](/reference/resource-configs/snapshot_meta_column_names): {dictionary} + [snapshot_meta_column_names](/reference/resource-configs/snapshot_meta_column_names): dictionary ``` @@ -87,7 +87,7 @@ The following table outlines the configurations available for snapshots: | [check_cols](/reference/resource-configs/check_cols) | If using the `check` strategy, then the columns to check | Only if using the `check` strategy | ["status"] | | [updated_at](/reference/resource-configs/updated_at) | If using the `timestamp` strategy, the timestamp column to compare | Only if using the `timestamp` strategy | updated_at | | [invalidate_hard_deletes](/reference/resource-configs/invalidate_hard_deletes) | Find hard deleted records in source and set `dbt_valid_to` to current time if the record no longer exists | No | True | -| [snapshot_meta_column_names](/reference/resource-configs/snapshot_meta_column_names) | Customize the names of the snapshot meta fields | No | {dictionary} | +| [snapshot_meta_column_names](/reference/resource-configs/snapshot_meta_column_names) | Customize the names of the snapshot meta fields | No | dictionary | - In versions prior to v1.9, the `target_schema` (required) and `target_database` (optional) configurations defined a single schema or database to build a snapshot across users and environment. This created problems when testing or developing a snapshot, as there was no clear separation between development and production environments. In v1.9, `target_schema` became optional, allowing snapshots to be environment-aware. By default, without `target_schema` or `target_database` defined, snapshots now use the `generate_schema_name` or `generate_database_name` macros to determine where to build. Developers can still set a custom location with [`schema`](/reference/resource-configs/schema) and [`database`](/reference/resource-configs/database) configs, consistent with other resource types. - A number of other configurations are also supported (for example, `tags` and `post-hook`). For the complete list, refer to [Snapshot configurations](/reference/snapshot-configs). From a3f893b1594a669c57271acc255cee9df296d75a Mon Sep 17 00:00:00 2001 From: mirnawong1 Date: Fri, 4 Oct 2024 15:54:50 +0100 Subject: [PATCH 73/74] reword --- website/docs/docs/build/snapshots.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/docs/build/snapshots.md b/website/docs/docs/build/snapshots.md index c11b7c4b76c..4a20742eb8f 100644 --- a/website/docs/docs/build/snapshots.md +++ b/website/docs/docs/build/snapshots.md @@ -52,7 +52,7 @@ It is not possible to "preview data" or "compile sql" for snapshots in dbt Cloud -Configure your snapshots in YAML files (typically in your snapshots directory) to tell dbt how to detect record changes. Define snapshots configurations. Configuring in YAML files (alongside your models) provides a cleaner and more consistent set up. +Configure your snapshots in YAML files to tell dbt how to detect record changes. Define snapshots configurations in YAML files, alongside your models, for a cleaner, faster, and more consistent set up. From 6fc9a8996e22935f1ebb8ccb00343286c4a7dd62 Mon Sep 17 00:00:00 2001 From: Mirna Wong <89008547+mirnawong1@users.noreply.github.com> Date: Fri, 4 Oct 2024 16:22:35 +0100 Subject: [PATCH 74/74] Update website/docs/docs/build/snapshots.md Co-authored-by: Doug Beatty <44704949+dbeatty10@users.noreply.github.com> --- website/docs/docs/build/snapshots.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/docs/build/snapshots.md b/website/docs/docs/build/snapshots.md index 4a20742eb8f..f5321aa626a 100644 --- a/website/docs/docs/build/snapshots.md +++ b/website/docs/docs/build/snapshots.md @@ -59,7 +59,7 @@ Configure your snapshots in YAML files to tell dbt how to detect record changes. ```yaml snapshots: - name: string - relation: source # source('my_source', 'my_table') + relation: relation # source('my_source', 'my_table') or ref('my_model') config: [database](/reference/resource-configs/database): string [schema](/reference/resource-configs/schema): string