Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Updated constraints doc following slack #5995

Merged
merged 80 commits into from
Oct 4, 2024
Merged
Show file tree
Hide file tree
Changes from 65 commits
Commits
Show all changes
80 commits
Select commit Hold shift + click to select a range
50998fe
Updated constraints doc following slack
nataliefiann Aug 30, 2024
d738c03
Merge branch 'current' into nfiann-snowflake-constraints
nataliefiann Aug 30, 2024
dc70f56
Update website/docs/reference/resource-properties/constraints.md
nataliefiann Aug 30, 2024
1732a00
Update website/docs/reference/resource-properties/constraints.md
nataliefiann Aug 30, 2024
9368790
Update website/docs/reference/resource-properties/constraints.md
nataliefiann Aug 30, 2024
3ac1d4d
Update website/docs/reference/resource-properties/constraints.md
nataliefiann Aug 30, 2024
70460bb
Update website/docs/reference/resource-properties/constraints.md
nataliefiann Aug 30, 2024
3ea40bd
Update website/docs/reference/resource-properties/constraints.md
nataliefiann Aug 30, 2024
fa43fff
Update website/docs/reference/resource-properties/constraints.md
nataliefiann Aug 30, 2024
4a17642
Update website/docs/reference/resource-properties/constraints.md
nataliefiann Aug 30, 2024
0d834a7
Update website/docs/reference/resource-properties/constraints.md
nataliefiann Aug 30, 2024
a8e26c4
Update website/docs/reference/resource-properties/constraints.md
nataliefiann Aug 30, 2024
e6fe96b
Update website/docs/reference/resource-properties/constraints.md
nataliefiann Aug 30, 2024
cdf0ade
Update website/docs/reference/resource-properties/constraints.md
nataliefiann Aug 30, 2024
9da9e5d
Update website/docs/reference/resource-properties/constraints.md
nataliefiann Aug 30, 2024
85abbd2
Update website/docs/reference/resource-properties/constraints.md
nataliefiann Aug 30, 2024
03fc2a2
Update website/docs/reference/resource-properties/constraints.md
nataliefiann Aug 30, 2024
8851b14
Updated doc, moved section to new location
nataliefiann Sep 2, 2024
6141055
Merge branch 'current' into nfiann-snowflake-constraints
mirnawong1 Sep 2, 2024
7fc710a
Update website/docs/reference/resource-properties/constraints.md
nataliefiann Sep 2, 2024
8f7cd16
Update website/docs/reference/resource-properties/constraints.md
nataliefiann Sep 2, 2024
43d6cf5
Update website/docs/reference/resource-properties/constraints.md
nataliefiann Sep 2, 2024
8295508
Update website/docs/reference/resource-properties/constraints.md
nataliefiann Sep 3, 2024
f8cce33
Update website/docs/reference/resource-properties/constraints.md
nataliefiann Sep 3, 2024
04bf56d
Update website/docs/reference/resource-properties/constraints.md
nataliefiann Sep 3, 2024
2455fb5
Update website/docs/reference/resource-properties/constraints.md
nataliefiann Sep 3, 2024
70ad260
Update website/docs/reference/resource-properties/constraints.md
nataliefiann Sep 3, 2024
72420bd
Update website/docs/reference/resource-properties/constraints.md
nataliefiann Sep 3, 2024
7d2287d
Update website/docs/reference/resource-properties/constraints.md
nataliefiann Sep 3, 2024
e10b826
Update website/docs/reference/resource-properties/constraints.md
nataliefiann Sep 3, 2024
244e2f4
Update website/docs/reference/resource-properties/constraints.md
nataliefiann Sep 3, 2024
8258226
Update website/docs/reference/resource-properties/constraints.md
nataliefiann Sep 3, 2024
2cf8771
Update website/docs/reference/resource-properties/constraints.md
nataliefiann Sep 3, 2024
15f7664
Update website/docs/reference/resource-properties/constraints.md
nataliefiann Sep 3, 2024
05afe4e
updated doc
nataliefiann Sep 4, 2024
eb1b5bf
Merge branch 'current' into nfiann-snowflake-constraints
nataliefiann Sep 4, 2024
d9290f0
Update website/docs/reference/resource-properties/constraints.md
nataliefiann Sep 4, 2024
3fea390
Update website/docs/reference/resource-properties/constraints.md
nataliefiann Sep 4, 2024
278f3d5
Update website/docs/reference/resource-properties/constraints.md
nataliefiann Sep 4, 2024
f3bced0
Update website/docs/reference/resource-properties/constraints.md
nataliefiann Sep 4, 2024
88f16f0
Update website/docs/reference/resource-properties/constraints.md
nataliefiann Sep 4, 2024
bb6c5ea
Update website/docs/reference/resource-properties/constraints.md
nataliefiann Sep 4, 2024
fb6e44d
Update website/docs/reference/resource-properties/constraints.md
nataliefiann Sep 4, 2024
9cfc253
Update website/docs/reference/resource-properties/constraints.md
nataliefiann Sep 4, 2024
1a7b84e
Update website/docs/reference/resource-properties/constraints.md
nataliefiann Sep 4, 2024
84ec68c
Update website/docs/reference/resource-properties/constraints.md
nataliefiann Sep 4, 2024
b4285e7
update yaml
nataliefiann Sep 4, 2024
35cc43e
Merge branch 'nfiann-snowflake-constraints' of https://github.com/dbt…
nataliefiann Sep 4, 2024
2ff1603
Merge branch 'current' into nfiann-snowflake-constraints
nataliefiann Sep 4, 2024
5217d03
Update website/docs/reference/resource-properties/constraints.md
nataliefiann Sep 4, 2024
3885b93
Update website/docs/reference/resource-properties/constraints.md
nataliefiann Sep 4, 2024
8f7afb7
Merge branch 'current' into nfiann-snowflake-constraints
nataliefiann Sep 9, 2024
ecff6de
Updated project splitting section
nataliefiann Sep 10, 2024
55f9a4c
Update website/docs/best-practices/how-we-structure/5-the-rest-of-the…
nataliefiann Sep 10, 2024
b5d2a45
Update website/docs/best-practices/how-we-structure/5-the-rest-of-the…
nataliefiann Sep 10, 2024
f40d5b3
Changed order of paragraph around
nataliefiann Sep 10, 2024
b31bf86
Merge branch 'nfiann-project-splitting' of https://github.com/dbt-lab…
nataliefiann Sep 10, 2024
63c9b46
Update website/docs/best-practices/how-we-structure/5-the-rest-of-the…
nataliefiann Sep 10, 2024
0e78924
Updated project splitting section (#6043)
nataliefiann Sep 10, 2024
62870a4
Merge branch 'current' into nfiann-snowflake-constraints
mirnawong1 Sep 11, 2024
1207ce2
Update website/docs/reference/resource-properties/constraints.md
nataliefiann Sep 11, 2024
b7cd506
Update website/docs/reference/resource-properties/constraints.md
nataliefiann Sep 11, 2024
ee54034
Merge branch 'current' into nfiann-snowflake-constraints
mirnawong1 Sep 11, 2024
d7eb830
Update website/docs/reference/resource-properties/constraints.md
nataliefiann Sep 11, 2024
6f7b622
Update website/docs/reference/resource-properties/constraints.md
nataliefiann Sep 11, 2024
b72d530
Update website/docs/reference/resource-properties/constraints.md
nataliefiann Sep 11, 2024
261d61f
Update website/docs/reference/resource-properties/constraints.md
nataliefiann Sep 11, 2024
4274678
Update website/docs/reference/resource-properties/constraints.md
nataliefiann Sep 11, 2024
86038b6
Update website/docs/reference/resource-properties/constraints.md
nataliefiann Sep 11, 2024
a9b4418
Update website/docs/reference/resource-properties/constraints.md
nataliefiann Sep 11, 2024
bb94cfe
Merge branch 'current' into nfiann-snowflake-constraints
mirnawong1 Sep 12, 2024
c9b93bf
Update website/docs/reference/resource-properties/constraints.md
mirnawong1 Sep 12, 2024
fce1c45
Merge branch 'current' into nfiann-snowflake-constraints
mirnawong1 Oct 2, 2024
e9b8223
Merge branch 'current' into nfiann-snowflake-constraints
mirnawong1 Oct 3, 2024
1997af4
Merge branch 'current' into nfiann-snowflake-constraints
mirnawong1 Oct 3, 2024
fa0da23
Merge branch 'current' into nfiann-snowflake-constraints
mirnawong1 Oct 3, 2024
d05793c
Merge branch 'current' into nfiann-snowflake-constraints
mirnawong1 Oct 3, 2024
749dc8a
Merge branch 'current' into nfiann-snowflake-constraints
mirnawong1 Oct 4, 2024
3ae24bd
Update constraints.md
mirnawong1 Oct 4, 2024
06384c6
Update constraints.md
mirnawong1 Oct 4, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -102,12 +102,14 @@ We’ve focused heavily thus far on the primary area of action in our dbt projec

### Project splitting

One important, growing consideration in the analytics engineering ecosystem is how and when to split a codebase into multiple dbt projects. Our present stance on this for most projects, particularly for teams starting out, is straightforward: you should avoid it unless you have no other option or it saves you from an even more complex workaround. If you do have the need to split up your project, it’s completely possible through the use of private packages, but the added complexity and separation is, for most organizations, a hindrance, not a help, at present. That said, this is very likely subject to change! [We want to create a world where it’s easy to bring lots of dbt projects together into a cohesive lineage](https://github.com/dbt-labs/dbt-core/discussions/5244). In a world where it’s simple to break up monolithic dbt projects into multiple connected projects, perhaps inside of a modern mono repo, the calculus will be different, and the below situations we recommend against may become totally viable. So watch this space!
One important, growing consideration in the analytics engineering ecosystem is how and when to split a codebase into multiple dbt projects. Currently, our advice for most teams, especially those just starting, is fairly simple: in most cases, we recommend doing so with [dbt Mesh](/best-practices/how-we-mesh/mesh-1-intro)! dbt Mesh allows organizations to handle complexity by connecting several dbt projects rather than relying on one big, monolithic project. This approach is designed to speed up development while maintaining governance.

- ❌ **Business groups or departments.** Conceptual separations within the project are not a good reason to split up your project. Splitting up, for instance, marketing and finance modeling into separate projects will not only add unnecessary complexity but destroy the unifying effect of collaborating across your organization on cohesive definitions and business logic.
- ❌ **ML vs Reporting use cases.** Similarly to the point above, splitting a project up based on different use cases, particularly more standard BI versus ML features, is a common idea. We tend to discourage it for the time being. As with the previous point, a foundational goal of implementing dbt is to create a single source of truth in your organization. The features you’re providing to your data science teams should be coming from the same marts and metrics that serve reports on executive dashboards.
As breaking up monolithic dbt projects into smaller, connected projects, potentially within a modern mono repo becomes easier, the scenarios we currently advise against may soon become feasible. So watch this space!

- ✅ **Business groups or departments.** Conceptual separations within the project are the primary reason to split up your project. This allows your business domains to own their own data products and still collaborate using dbt Mesh. For more information about dbt Mesh, please refer to our [dbt Mesh FAQs](/best-practices/how-we-mesh/mesh-5-faqs).
- ✅ **Data governance.** Structural, organizational needs — such as data governance and security — are one of the few worthwhile reasons to split up a project. If, for instance, you work at a healthcare company with only a small team cleared to access raw data with PII in it, you may need to split out your staging models into their own projects to preserve those policies. In that case, you would import your staging project into the project that builds on those staging models as a [private package](https://docs.getdbt.com/docs/build/packages/#private-packages).
- ✅ **Project size.** At a certain point, your project may grow to have simply too many models to present a viable development experience. If you have 1000s of models, it absolutely makes sense to find a way to split up your project.
- ❌ **ML vs Reporting use cases.** Similarly to the point above, splitting a project up based on different use cases, particularly more standard BI versus ML features, is a common idea. We tend to discourage it for the time being. As with the previous point, a foundational goal of implementing dbt is to create a single source of truth in your organization. The features you’re providing to your data science teams should be coming from the same marts and metrics that serve reports on executive dashboards.

## Final considerations

Expand Down
72 changes: 72 additions & 0 deletions website/docs/reference/resource-properties/constraints.md
Original file line number Diff line number Diff line change
Expand Up @@ -518,3 +518,75 @@ alter table schema_name.my_model add constraint 472394792387497234 check (id > 0
</div>

</WHCode>

### Custom constraints
mirnawong1 marked this conversation as resolved.
Show resolved Hide resolved

In dbt Cloud and dbt Core, you can use custom constraints on models for the advanced configuration of tables. Different data warehouses support different syntax and capabilities.

Custom constraints allow you to add configuration to specific columns:
nataliefiann marked this conversation as resolved.
Show resolved Hide resolved

- For example, custom constraints allow you to set [masking policies](https://docs.snowflake.com/en/user-guide/security-column-intro#what-are-masking-policies) in Snowflake when using a Create Table As Select (CTAS).
nataliefiann marked this conversation as resolved.
Show resolved Hide resolved

- [Databricks](https://docs.databricks.com/en/sql/language-manual/sql-ref-syntax-ddl-create-table-using.html) and [BigQuery](https://cloud.google.com/bigquery/docs/reference/standard-sql/data-definition-language#column_name_and_column_schema) are other examples of data warehouses having their own set of parameters that can be set for columns in their CTAS statements.
nataliefiann marked this conversation as resolved.
Show resolved Hide resolved


You can implement constraints in a couple of different ways:
nataliefiann marked this conversation as resolved.
Show resolved Hide resolved

nataliefiann marked this conversation as resolved.
Show resolved Hide resolved
- [Custom constraints with tags](#custom-constraints-with-tags)
- [Custom constraints without tags](#custom-constraints-without-tags)
### Custom constraints with tags

nataliefiann marked this conversation as resolved.
Show resolved Hide resolved
Here's an example of how to implement tag-based masking policies with contracts and constraints using the following syntax:
nataliefiann marked this conversation as resolved.
Show resolved Hide resolved

<File name='models/constraints_example.yml'>

```yaml

models:
nataliefiann marked this conversation as resolved.
Show resolved Hide resolved
- name: my_model
config: null
nataliefiann marked this conversation as resolved.
Show resolved Hide resolved
contract:
enforced: true
materialized: table
columns:
nataliefiann marked this conversation as resolved.
Show resolved Hide resolved
- name: id
data_type: int
constraints:
- type: custom
expression: "tag (my_tag = 'my_value')" # A custom SQL expression used to enforce a specific constraint on a column.

```

</File>

Using this syntax requires configuring all the columns and their types as it’s the only way to send a create or replace `<cols_info_with_masking> mytable as ...`. It’s not possible to do it with just a partial list of columns. This means making sure the columns and constraints fields are fully defined.

To generate a YAML with all the columns, you can use `generate_model_yaml` from [dbt-codegen](https://github.com/dbt-labs/dbt-codegen/tree/0.12.1/?tab=readme-ov-file#generate_model_yaml-source).


### Custom constraints without tags

Alternatively, you can add a masking policy without tags:
nataliefiann marked this conversation as resolved.
Show resolved Hide resolved
nataliefiann marked this conversation as resolved.
Show resolved Hide resolved

<File name='models/constraints_example.yml'>

```yaml

models:
nataliefiann marked this conversation as resolved.
Show resolved Hide resolved
- name: my_model
config:
contract:
enforced: true
materialized: table
columns:
- name: id
data_type: int
constraints:
- type: custom
expression: "masking policy my_policy"

```

</File>


Loading