Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

clarify build on docs content and move doc to dag section #5633

Merged
merged 60 commits into from
Jun 12, 2024
Merged
Show file tree
Hide file tree
Changes from 16 commits
Commits
Show all changes
60 commits
Select commit Hold shift + click to select a range
00d3861
rephrase
mirnawong1 Jun 6, 2024
aca3fa0
add explorer info
mirnawong1 Jun 7, 2024
0bd46db
add img
mirnawong1 Jun 7, 2024
f4968db
updates to build your docs
mirnawong1 Jun 10, 2024
b0f07cb
add redirect and update links
mirnawong1 Jun 10, 2024
483bfe3
add links
mirnawong1 Jun 10, 2024
887f10e
update screenshots
mirnawong1 Jun 10, 2024
da380bb
Merge branch 'current' into mwong-update-build-docs
mirnawong1 Jun 10, 2024
414be7d
Update website/docs/docs/deploy/artifacts.md
mirnawong1 Jun 10, 2024
50d1d28
Update website/docs/docs/deploy/artifacts.md
mirnawong1 Jun 10, 2024
1cb86a7
Update website/docs/docs/deploy/artifacts.md
mirnawong1 Jun 10, 2024
f3a6ce0
Update artifacts.md
mirnawong1 Jun 10, 2024
a8a44c6
Update website/docs/docs/deploy/job-commands.md
mirnawong1 Jun 10, 2024
5715d3d
Update artifacts.md
mirnawong1 Jun 10, 2024
d2125e2
Update website/docs/docs/deploy/artifacts.md
mirnawong1 Jun 10, 2024
14cd7c7
Update website/docs/docs/deploy/artifacts.md
mirnawong1 Jun 10, 2024
be268ba
Update website/docs/docs/collaborate/legacy-dbt-docs.md
mirnawong1 Jun 10, 2024
131576d
add
mirnawong1 Jun 10, 2024
b2583ae
Merge branch 'mwong-update-build-docs' of https://github.com/dbt-labs…
mirnawong1 Jun 10, 2024
0a021c7
katherine's feedback
mirnawong1 Jun 10, 2024
e39c75d
Merge branch 'current' into mwong-update-build-docs
mirnawong1 Jun 10, 2024
4181ee5
add
mirnawong1 Jun 10, 2024
844ca28
Merge branch 'mwong-update-build-docs' of https://github.com/dbt-labs…
mirnawong1 Jun 10, 2024
b7a07c2
cameron's feedback
mirnawong1 Jun 10, 2024
c64658f
Merge branch 'current' into mwong-update-build-docs
mirnawong1 Jun 10, 2024
5573db7
typo
mirnawong1 Jun 10, 2024
b3a737b
add
mirnawong1 Jun 11, 2024
a7f7bac
Merge branch 'current' into mwong-update-build-docs
mirnawong1 Jun 11, 2024
5e4cea3
remove legacy dbt docs file and readd build your docs
mirnawong1 Jun 11, 2024
db6371a
update links
mirnawong1 Jun 11, 2024
9b5f349
update doc
mirnawong1 Jun 11, 2024
43fae64
rework
mirnawong1 Jun 11, 2024
588a6d5
fix links
mirnawong1 Jun 11, 2024
acd92b9
Update documentation.md
mirnawong1 Jun 11, 2024
5a87091
Update website/sidebars.js
mirnawong1 Jun 11, 2024
055c28a
Update website/sidebars.js
mirnawong1 Jun 11, 2024
0434e44
Update collaborate-with-others.md
mirnawong1 Jun 11, 2024
abe6c22
Update website/docs/docs/build/documentation.md
mirnawong1 Jun 11, 2024
dd3eeed
Update website/docs/docs/collaborate/explore-projects.md
mirnawong1 Jun 11, 2024
2e906a0
Update website/docs/docs/collaborate/explore-projects.md
mirnawong1 Jun 11, 2024
9484e45
update titles
mirnawong1 Jun 11, 2024
e016b66
update to discover data
mirnawong1 Jun 11, 2024
33607a6
Merge branch 'current' into mwong-update-build-docs
mirnawong1 Jun 12, 2024
4ed3bd3
Update documentation.md
mirnawong1 Jun 12, 2024
3427e7f
Update build-and-view-your-docs.md
mirnawong1 Jun 12, 2024
d9d21de
Update explore-projects.md
mirnawong1 Jun 12, 2024
a2f9870
Update website/docs/docs/build/documentation.md
mirnawong1 Jun 12, 2024
167ec1c
Update website/docs/docs/build/documentation.md
mirnawong1 Jun 12, 2024
96d8f4b
update
mirnawong1 Jun 12, 2024
cf55222
Update website/docs/docs/build/documentation.md
mirnawong1 Jun 12, 2024
b929a93
Update website/docs/docs/build/documentation.md
mirnawong1 Jun 12, 2024
1b85ed9
updates
mirnawong1 Jun 12, 2024
1ad82b7
Merge branch 'mwong-update-build-docs' of https://github.com/dbt-labs…
mirnawong1 Jun 12, 2024
ec09a7e
Update website/docs/docs/build/documentation.md
mirnawong1 Jun 12, 2024
8414445
Merge branch 'current' into mwong-update-build-docs
mirnawong1 Jun 12, 2024
779505c
adding redirect
mirnawong1 Jun 12, 2024
38da545
Merge branch 'current' into mwong-update-build-docs
mirnawong1 Jun 12, 2024
246e41f
Update sharing-documentation.md (#5644)
mirnawong1 Jun 12, 2024
50609eb
update
mirnawong1 Jun 12, 2024
2390941
add plan
mirnawong1 Jun 12, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions website/blog/2021-02-05-dbt-project-checklist.md
Original file line number Diff line number Diff line change
Expand Up @@ -173,8 +173,8 @@ This post is the checklist I created to guide our internal work, and I’m shari

Useful Links

* [FAQs for documentation](/docs/collaborate/documentation#faqs)
* [Doc blocks](/docs/collaborate/documentation#using-docs-blocks)
* [FAQs for documentation](/docs/build/documentation#faqs)
* [Doc blocks](/docs/build/documentation#using-docs-blocks)

## ✅ dbt Cloud specifics
----------------------------------------------------------------------------------------------------------------------------------------------------------
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -87,7 +87,7 @@ The most important thing we’re introducing when your project is an infant is t

* Introduce modularity with [{{ ref() }}](/reference/dbt-jinja-functions/ref) and [{{ source() }}](/reference/dbt-jinja-functions/source)

* [Document](/docs/collaborate/documentation) and [test](/docs/build/data-tests) your first models
* [Document](/docs/build/documentation) and [test](/docs/build/data-tests) your first models

![image alt text](/img/blog/building-a-mature-dbt-project-from-scratch/image_3.png)

Expand Down
2 changes: 1 addition & 1 deletion website/blog/2022-09-28-analyst-to-ae.md
Original file line number Diff line number Diff line change
Expand Up @@ -133,7 +133,7 @@ It’s much easier to keep to a naming guide when the writer has a deep understa

If we want to know how certain logic was built technically, then we can reference the SQL code in dbt docs. If we want to know *why* a certain logic was built into that specific model, then that’s where we’d turn to the documentation.

- Example of not-so-helpful documentation ([dbt docs can](https://docs.getdbt.com/docs/collaborate/documentation) build this dynamically):
- Example of not-so-helpful documentation ([dbt docs can](https://docs.getdbt.com/docs/build/documentation) build this dynamically):
- `Case when Zone = 1 and Level like 'A%' then 'True' else 'False' end as GroupB`
- Example of better, more descriptive documentation (add to your dbt markdown file or column descriptions):
- Group B is defined as Users in Zone 1 with a Level beginning with the letter 'A'. These users are accessing our new add-on product that began in Beta in August 2022. It's recommended to filter them out of the main Active Users metric.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ In this article, two Montreal Analytics consultants, Jade and Callie, discuss th

**J:** To prepare for the exam, I built up a practice dbt project. All consultants do this as part of Montreal Analytics onboarding process, and this project allowed me to practice implementing sources and tests, refactoring SQL models, and debugging plenty of error messages. Additionally, I reviewed the [Certification Study Guide](https://www.getdbt.com/assets/uploads/dbt_certificate_study_guide.pdf) and attended group learning sessions.

**C:** To prepare for the exam I reviewed the official dbt Certification Study Guide and the [official dbt docs](https://docs.getdbt.com/), and attended group study and learning sessions that were hosted by Montreal Analytics for all employees interested in taking the exam. As a group, we prioritized subjects that we felt less familiar with; for the first cohort of test takers this was mainly newer topics that haven’t yet become integral to a typical dbt project, such as [doc blocks](https://docs.getdbt.com/docs/collaborate/documentation#using-docs-blocks) and [configurations versus properties](https://docs.getdbt.com/reference/configs-and-properties). These sessions mainly covered the highlights and common “gotchas” that are experienced using these techniques. The sessions were moderated by a team member who had already successfully completed the dbt Certification, but operated in a very collaborative environment, so everyone could provide additional information, ask questions to the group, and provide feedback to other members of our certification taking group.
**C:** To prepare for the exam I reviewed the official dbt Certification Study Guide and the [official dbt docs](https://docs.getdbt.com/), and attended group study and learning sessions that were hosted by Montreal Analytics for all employees interested in taking the exam. As a group, we prioritized subjects that we felt less familiar with; for the first cohort of test takers this was mainly newer topics that haven’t yet become integral to a typical dbt project, such as [doc blocks](https://docs.getdbt.com/docs/build/documentation#using-docs-blocks) and [configurations versus properties](https://docs.getdbt.com/reference/configs-and-properties). These sessions mainly covered the highlights and common “gotchas” that are experienced using these techniques. The sessions were moderated by a team member who had already successfully completed the dbt Certification, but operated in a very collaborative environment, so everyone could provide additional information, ask questions to the group, and provide feedback to other members of our certification taking group.

I felt comfortable with the breadth of my dbt knowledge and had familiarity with most topics. However in my day-to-day implementation, I am often reliant on documentation or copying and pasting specific configurations in order to get the correct settings. Therefore, my focus was on memorizing important criteria for *how to use* certain features, particularly on the order/nesting of how the key YAML files are configured (dbt_project.yml, table.yml, source.yml).

Expand Down Expand Up @@ -75,4 +75,4 @@ Now, the first thing you must do when you’ve passed a test is to get yourself
Standards and best practices are very important, but a test is a measure at a single point in time of a rapidly evolving industry. It’s also a measure of my test-taking abilities, my stress levels, and other things unrelated to my skill in data modeling; I wouldn’t be a good analyst if I didn’t recognize the faults of a measurement. I’m glad to have this check mark completed, but I will continue to stay up to date with changes, learn new data skills and techniques, and find ways to continue being a holistically helpful teammate to my colleagues and clients.


You can learn more about the dbt Certification [here](https://www.getdbt.com/blog/dbt-certification-program/).
You can learn more about the dbt Certification [here](https://www.getdbt.com/blog/dbt-certification-program/).
2 changes: 1 addition & 1 deletion website/blog/2023-05-04-generating-dynamic-docs.md
Original file line number Diff line number Diff line change
Expand Up @@ -215,7 +215,7 @@ Which in turn can be copy-pasted into a new `.yml` file. In our example, we writ

## Create docs blocks for the new columns

[Docs blocks](https://docs.getdbt.com/docs/collaborate/documentation#using-docs-blocks) can be utilized to write more DRY and robust documentation. To use docs blocks, update your folder structure to contain a `.md` file. Your file structure should now look like this:
[Docs blocks](https://docs.getdbt.com/docs/build/documentation#using-docs-blocks) can be utilized to write more DRY and robust documentation. To use docs blocks, update your folder structure to contain a `.md` file. Your file structure should now look like this:

```
models/core/activity_based_interest
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@ When structuring your YAML configuration files in a dbt project, you want to bal
- The leading underscore ensures your YAML files will be sorted to the top of every folder to make them easy to separate from your models.
- YAML files don’t need unique names in the way that SQL model files do, but including the directory (instead of simply `_sources.yml` in each folder), means you can fuzzy find the right file more quickly.
- We’ve recommended several different naming conventions over the years, most recently calling these `schema.yml` files. We’ve simplified to recommend that these simply be labelled based on the YAML dictionary that they contain.
- If you utilize [doc blocks](https://docs.getdbt.com/docs/collaborate/documentation#using-docs-blocks) in your project, we recommend following the same pattern, and creating a `_[directory]__docs.md` markdown file per directory containing all your doc blocks for that folder of models.
- If you utilize [doc blocks](https://docs.getdbt.com/docs/build/documentation#using-docs-blocks) in your project, we recommend following the same pattern, and creating a `_[directory]__docs.md` markdown file per directory containing all your doc blocks for that folder of models.
- ❌ **Config per project.** Some people put _all_ of their source and model YAML into one file. While you can technically do this, and while it certainly simplifies knowing what file the config you’re looking for will be in (as there is only one file), it makes it much harder to find specific configurations within that file. We recommend balancing those two concerns.
- ⚠️ **Config per model.** On the other end of the spectrum, some people prefer to create one YAML file per model. This presents less of an issue than a single monolith file, as you can quickly search for files, know exactly where specific configurations exist, spot models without configs (and thus without tests) by looking at the file tree, and various other advantages. In our opinion, the extra files, tabs, and windows this requires creating, copying from, pasting to, closing, opening, and managing creates a somewhat slower development experience that outweighs the benefits. Defining config per directory is the most balanced approach for most projects, but if you have compelling reasons to use config per model, there are definitely some great projects that follow this paradigm.
- ✅ **Cascade configs.** Leverage your `dbt_project.yml` to set default configurations at the directory level. Use the well-organized folder structure we’ve created thus far to define the baseline schemas and materializations, and use dbt’s cascading scope priority to define variations to this. For example, as below, define your marts to be materialized as tables by default, define separate schemas for our separate subfolders, and any models that need to use incremental materialization can be defined at the model level.
Expand Down
Original file line number Diff line number Diff line change
@@ -1,10 +1,12 @@
---
title: "About documentation"
title: "Documentation"
description: "Learn how good documentation for your dbt models helps stakeholders discover and understand your datasets."
id: "documentation"
pagination_next: "docs/collaborate/build-and-view-your-docs"
pagination_prev: null
---
<p style={{ color: '#808080', fontSize: '1.1em' }}>
Good documentation for your dbt models will help downstream consumers discover and understand the datasets which you curate for them.
mirnawong1 marked this conversation as resolved.
Show resolved Hide resolved
dbt provides a way to generate documentation for your dbt project and render it as a website.
</p>

## Related documentation
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we move this to the bottom while we're here, since that's where we usually place it?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

tbh i did think that but the other 'build' docs have this format so left it so it would be consistent for users.


Expand All @@ -19,18 +21,14 @@ pagination_prev: null

## Overview

Good documentation for your dbt models will help downstream consumers discover and understand the datasets which you curate for them.
The default documentation experience in dbt Cloud is [dbt Explorer](/docs/collaborate/explore-projects), available on [Team or Enterprise plans](https://www.getdbt.com/pricing/), which provides a richer and more interactive experience for understanding your project's resources and lineage. dbt Cloud developer and dbt Core users can use [dbt Docs](/docs/collaborate/legacy-dbt-docs), a legacy feature to generate documentation, and is accessible through dbt Explorer or the **Documentation** link.
mirnawong1 marked this conversation as resolved.
Show resolved Hide resolved

dbt provides a way to generate documentation for your dbt project and render it as a website. The documentation for your project includes:
The documentation for your project includes:
* **Information about your project**: including model code, a DAG of your project, any tests you've added to a column, and more.
* **Information about your <Term id="data-warehouse" />**: including column data types, and <Term id="table" /> sizes. This information is generated by running queries against the information schema.

Importantly, dbt also provides a way to add **descriptions** to models, columns, sources, and more, to further enhance your documentation.

Here's an example docs site:

<Lightbox src="/img/docs/building-a-dbt-project/dbt-docs-screenshot.png" title="Auto-generated dbt documentation website"/>

## Adding descriptions to your project
To add descriptions to your project, use the `description:` key in the same files where you declare [tests](/docs/build/data-tests), like so:

Expand Down Expand Up @@ -60,13 +58,14 @@ models:

</File>


## Generating project documentation
You can generate a documentation site for your project (with or without descriptions) using the CLI.
You can generate a documentation site for your project (with or without descriptions) in development using your editor or with [dbt Explorer](/docs/collaborate/explore-projects) in dbt Cloud after a successful job run in the deployment environment.
mirnawong1 marked this conversation as resolved.
Show resolved Hide resolved

First, run `dbt docs generate` — this command tells dbt to compile relevant information about your dbt project and warehouse into `manifest.json` and `catalog.json` files respectively. To see the documentation for all columns and not just columns described in your project, ensure that you have created the models with `dbt run` beforehand.

Then, run `dbt docs serve` to use these `.json` files to populate a local website.
If you're developing locally, run `dbt docs serve` to use these `.json` files to populate a local website.

To set up a documentation job for dbt Explorer, refer to [Set up a documentation job](/docs/collaborate/set-up-doc-job).

## FAQs
<FAQ path="Project/example-projects" alt_header="Are there any example dbt documentation sites?"/>
Expand All @@ -75,8 +74,7 @@ Then, run `dbt docs serve` to use these `.json` files to populate a local websit
<FAQ path="Docs/sharing-documentation" />
<FAQ path="Docs/document-other-resources" />


## Using Docs Blocks
## Using docs blocks
### Syntax
To declare a docs block, use the jinja `docs` tag. Docs blocks can contain arbitrary markdown, but they must be uniquely named. Their names may contain uppercase and lowercase letters (A-Z, a-z), digits (0-9), and underscores (_), but can't start with a digit.

Expand Down Expand Up @@ -128,9 +126,13 @@ models:

In the resulting documentation, `'{{ doc("table_events") }}'` will be expanded to the markdown defined in the `table_events` docs block.

## Setting a custom overview
## Legacy dbt Docs

The following additional configurations apply to [dbt Docs](/docs/collaborate/legacy-dbt-docs), a legacy feature that enables you to generate documentation for your project and data platform, rendering it as a website. The documentation is only updated with new information after a fully successful job run, ensuring accuracy and relevance.

### Setting a custom overview

The "overview" shown in the documentation website can be overridden by supplying your own docs block called `__overview__`. By default, dbt supplies an overview with helpful information about the docs site itself. Depending on your needs, it may be a good idea to override this docs block with specific information about your company style guide, links to reports, or information about who to contact for help. To override the default overview, create a docs block that looks like this:
The "overview" shown in the legacy dbt Docs website can be overridden by supplying your own docs block called `__overview__`. By default, dbt supplies an overview with helpful information about the docs site itself. Depending on your needs, it may be a good idea to override this docs block with specific information about your company style guide, links to reports, or information about who to contact for help. To override the default overview, create a docs block that looks like this:

<File name='models/overview.md'>

Expand Down Expand Up @@ -173,22 +175,27 @@ up to page views and sessions.

</File>

## Navigating the documentation site
Using the docs interface, you can navigate to the documentation for a specific model. That might look something like this:
### Navigating the documentation site

:::tip Use dbt Explorer for a richer documentation experience
For a richer and more interactive experience for understanding your project's resources and lineage, try out [dbt Explorer](/docs/collaborate/explore-projects), available on [Team or Enterprise plans](https://www.getdbt.com/pricing/).
:::

Using the legacy dbt Docs interface, you can navigate to the documentation for a specific model. That might look something like this:

<Lightbox src="/img/docs/building-a-dbt-project/testing-and-documentation/f2221dc-Screen_Shot_2018-08-14_at_6.29.55_PM.png" title="Auto-generated documentation for a dbt model"/>

Here, you can see a representation of the project structure, a markdown description for a model, and a list of all of the columns (with documentation) in the model.

From a docs page, you can click the green button in the bottom-right corner of the webpage to expand a "mini-map" of your DAG. This pane (shown below) will display the immediate parents and children of the model that you're exploring.
From a legacy dbt Docs page, you can click the green button in the bottom-right corner of the webpage to expand a "mini-map" of your DAG. This pane (shown below) will display the immediate parents and children of the model that you're exploring.

<Lightbox src="/img/docs/building-a-dbt-project/testing-and-documentation/ec77c45-Screen_Shot_2018-08-14_at_6.31.56_PM.png" title="Opening the DAG mini-map"/>

In this example, the `fct_subscription_transactions` model only has one direct parent. By clicking the "Expand" button in the top-right corner of the window, we can pivot the graph horizontally and view the full <Term id="data-lineage">lineage</Term> for our model. This lineage is filterable using the `--select` and `--exclude` flags, which are consistent with the semantics of [model selection syntax](/reference/node-selection/syntax). Further, you can right-click to interact with the DAG, jump to documentation, or share links to your graph visualization with your coworkers.

<Lightbox src="/img/docs/building-a-dbt-project/testing-and-documentation/ac97fba-Screen_Shot_2018-08-14_at_6.35.14_PM.png" title="The full lineage for a dbt model"/>

## Deploying the documentation site
### Deploying the documentation site

:::caution Security

Expand All @@ -198,7 +205,7 @@ The `dbt docs serve` command is only intended for local/development hosting of t

dbt's documentation website was built to make it easy to host on the web. The site is "static,” meaning you don't need any "dynamic" servers to serve the docs. You can host your documentation in several ways:

* Use [dbt Cloud](/docs/collaborate/documentation)
* Use [dbt Cloud](/docs/build/documentation)
* Host on [Amazon S3](https://docs.aws.amazon.com/AmazonS3/latest/dev/WebsiteHosting.html) (optionally [with IP access restrictions](https://docs.aws.amazon.com/AmazonS3/latest/dev/example-bucket-policies.html#example-bucket-policies-use-case-3))
* Publish with [Netlify](https://discourse.getdbt.com/t/publishing-dbt-docs-to-netlify/121)
* Use your own web server like Apache/Nginx
2 changes: 1 addition & 1 deletion website/docs/docs/build/exposures.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ id: "exposures"

Exposures make it possible to define and describe a downstream use of your dbt project, such as in a dashboard, application, or data science pipeline. By defining exposures, you can then:
- run, test, and list resources that feed into your exposure
- populate a dedicated page in the auto-generated [documentation](/docs/collaborate/documentation) site with context relevant to data consumers
- populate a dedicated page in the auto-generated [documentation](/docs/build/documentation) site with context relevant to data consumers

### Declaring an exposure

Expand Down
Loading
Loading