Skip to content

Commit

Permalink
Updates to merge jobs (#5612)
Browse files Browse the repository at this point in the history
## What are you changing in this pull request and why?

Updates to new merge jobs

closes https://github.com/dbt-labs/docs-internal/issues/578 

## Checklist
- [x] Review the [Content style
guide](https://github.com/dbt-labs/docs.getdbt.com/blob/current/contributing/content-style-guide.md)
so my content adheres to these guidelines.
- [x] For [docs
versioning](https://github.com/dbt-labs/docs.getdbt.com/blob/current/contributing/single-sourcing-content.md#about-versioning),
review how to [version a whole
page](https://github.com/dbt-labs/docs.getdbt.com/blob/current/contributing/single-sourcing-content.md#adding-a-new-version)
and [version a block of
content](https://github.com/dbt-labs/docs.getdbt.com/blob/current/contributing/single-sourcing-content.md#versioning-blocks-of-content).
- [x] Needs review from eng
- [x] Needs review from PM

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Leona B. Campbell <3880403+runleonarun@users.noreply.github.com>
Co-authored-by: Matt Shaver <60105315+matthewshaver@users.noreply.github.com>
  • Loading branch information
4 people authored Jun 28, 2024
1 parent cd9cc79 commit ccaf6b9
Show file tree
Hide file tree
Showing 13 changed files with 48 additions and 21 deletions.
2 changes: 2 additions & 0 deletions website/docs/docs/dbt-versions/release-notes.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,8 @@ Release notes are grouped by month for both multi-tenant and virtual private clo
- **Fix:** Fixed Private Link connections for Databricks.

#### Also available this month:

- **Enhancement:** Updates to the UI when [creating merge jobs](/docs/deploy/merge-jobs) are now available. The updates include improvements to helper text, new deferral settings, and performance improvements.
- **New**: The dbt Semantic Layer now offers a seamless integration with Microsoft Excel, available in [preview](/docs/dbt-versions/product-lifecycles#dbt-cloud). Build semantic layer queries and return data on metrics directly within Excel, through a custom menu. To learn more and install the add-on, check out [Microsoft Excel](/docs/cloud-integrations/semantic-layer/excel).
- **New:** [Job warnings](/docs/deploy/job-notifications) are now GA. Previously, you could receive email or Slack alerts about your jobs when they succeeded, failed, or were canceled. Now with the new **Warns** option, you can also receive alerts when jobs have encountered warnings from tests or source freshness checks during their run. This gives you more flexibility on _when_ to be notified.
- **New:** A [preview](/docs/dbt-versions/product-lifecycles#dbt-cloud) of the dbt Snowflake Native App is now available. With this app, you can access dbt Explorer, the **Ask dbt** chatbot, and orchestration observability features, extending your dbt Cloud experience into the Snowflake UI. To learn more, check out [About the dbt Snowflake Native App](/docs/cloud-integrations/snowflake-native-app) and [Set up the dbt Snowflake Native App](/docs/cloud-integrations/set-up-snowflake-native-app).
Expand Down
6 changes: 3 additions & 3 deletions website/docs/docs/deploy/jobs.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,10 +15,10 @@ Below is a comparison table that describes the behaviors of the different job ty

| | **Deploy jobs** | **CI jobs** | **Merge jobs** |
| --- | --- | --- | --- |
| Purpose | Builds production data assets. | Builds and tests new code before merging changes into production. | Build merged changes into production. |
| Trigger types | Triggered by a schedule or by API. | Triggered by a commit to a PR or by API. | Triggered by a merge request or by API. |
| Purpose | Builds production data assets. | Builds and tests new code before merging changes into production. | Build merged changes into production or update state for deferral. |
| Trigger types | Triggered by a schedule or by API. | Triggered by a commit to a PR or by API. | Triggered by a successful merge into the environment's branch or by API.|
| Destination | Builds into a production database and schema. | Builds into a staging database and ephemeral schema, lived for the lifetime of the PR. | Builds into a production database and schema. |
| Execution mode | Runs execute sequentially, so as to not have collisions on the underlying DAG. | Runs execute in parallel to promote team velocity. | Runs execute sequentially, so as to not have collisions on the underlying DAG. |
| Efficiency run savings | Detects over-scheduled jobs and cancels unnecessary runs to avoid queue clog. | Cancels existing runs when a newer commit is pushed to avoid redundant work. | N/A |
| State comparison | Only sometimes needs to detect state. | Almost always needs to compare state against the production environment to build on modified code and its dependents. | Does not need to detect state. |
| State comparison | Only sometimes needs to detect state. | Almost always needs to compare state against the production environment to build on modified code and its dependents. | Almost always needs to compare state against the production environment to build on modified code and its dependents. |
| Job run duration | Limit is 24 hours. | Limit is 24 hours. | Limit is 24 hours. |
44 changes: 35 additions & 9 deletions website/docs/docs/deploy/merge-jobs.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,35 +5,61 @@ description: "Learn how to trigger a dbt job run when a Git pull request merges.
---


You can set up a merge job to implement a continuous development (CD) workflow in dbt Cloud. The merge job triggers a dbt job to run when someone merges Git pull requests into production. This creates a seamless development experience where changes made in code will automatically update production data.
You can set up a merge job to implement a continuous development (CD) workflow in dbt Cloud. The merge job triggers a dbt job to run when someone merges Git pull requests into production. This workflow creates a seamless development experience where changes made in code will automatically update production data. Also, you can use this workflow for running `dbt compile` to update your environment's manifest so subsequent CI job runs are more performant.

By using CD in dbt Cloud, you can take advantage of deferral to build only the edited model and any downstream changes. With merge jobs, state will be updated almost instantly, always giving the most up-to-date state information in [dbt Explorer](/docs/collaborate/explore-projects).

## Prerequisites
- You have a dbt Cloud account.
- You have set up a [connection with your Git provider](/docs/cloud/git/git-configuration-in-dbt-cloud). This integration lets dbt Cloud run jobs on your behalf for job triggering.
- If you're using a native [GitLab](/docs/cloud/git/connect-gitlab) integration, you need a paid or self-hosted account that includes support for GitLab webhooks and [project access tokens](https://docs.gitlab.com/ee/user/project/settings/project_access_tokens.html). If you're using GitLab Free, merge requests will trigger CI jobs but CI job status updates (success or failure of the job) will not be reported back to GitLab.
- You are set up to receive push notifications from your Git provider. For details, refer to [Job notifications](/docs/deploy/job-notifications).
- For deferral (which is the default), make sure there has been at least one successful job run in the environment you defer to.

## Set up job trigger on Git merge {#set-up-merge-jobs}

1. On your deployment environment page, click **Create Job** > **Merge Job**.
1. On your deployment environment page, click **Create job** > **Merge job**.
1. Options in the **Job settings** section:
- **Job name** &mdash; Specify the name for the merge job.
- **Description** &mdash; Provide a descripion about the job.
- **Environment** &mdash; By default, it’s set to the environment you created the job from.
1. In the **Git trigger** section, the **Run on merge** option is enabled by default. Every time a PR merges (to the main branch) in your Git repo, this job will get triggered to run.
1. In the **Git trigger** section, the **Run on merge** option is enabled by default. Every time a PR merges (to a base
branch configured in the environment) in your Git repo, this job will get triggered to run.
1. Options in the **Execution settings** section:
- **Commands** &mdash; By default, it includes the `dbt build --select state:modified+` command. This informs dbt Cloud to build only new or changed models and their downstream dependents. Importantly, state comparison can only happen when there is a deferred environment selected to compare state to. Click **Add command** to add more [commands](/docs/deploy/job-commands) that you want to be invoked when this job runs.
- **Compare changes against an environment (Deferral)** &mdash; By default, it's set to the environment you created the job from. This option allows dbt Cloud to check the state of the code in the PR against the code running in the deferred environment, so as to only check the modified code, instead of building the full table or the entire DAG.
- **Compare changes against** &mdash; By default, it's set to compare changes against the environment you created the job from. This option allows dbt Cloud to check the state of the code in the PR against the code running in the deferred environment, so as to only check the modified code, instead of building the full table or the entire DAG. To change the default settings, you can select **No deferral**, **This job** for self-deferral, or choose a different environment.
1. (optional) Options in the **Advanced settings** section:
- **Environment variables** &mdash; Define [environment variables](/docs/build/environment-variables) to customize the behavior of your project when this job runs.
- **Target name** &mdash; Define the [target name](/docs/build/custom-target-names). Similar to **Environment Variables**, this option lets you customize the behavior of the project.
- **Target name** &mdash; Define the [target name](/docs/build/custom-target-names). Similar to environment variables, this option lets you customize the behavior of the project.
- **Run timeout** &mdash; Cancel this job if the run time exceeds the timeout value.
- **dbt version** &mdash; By default, it’s set to inherit the [dbt version](/docs/dbt-versions/core) from the environment. dbt Labs strongly recommends that you don't change the default setting. This option to change the version at the job level is useful only when you upgrade a project to the next dbt version; otherwise, mismatched versions between the environment and job can lead to confusing behavior.
- **Threads** &mdash; By default, it’s set to 4 [threads](/docs/core/connect-data-platform/connection-profiles#understanding-threads). Increase the thread count to increase model execution concurrency.
- **Run source freshness** &mdash; Enable this option to invoke the `dbt source freshness` command before running this job. Refer to [Source freshness](/docs/deploy/source-freshness) for more details.

## Example
<Lightbox src="/img/docs/dbt-cloud/using-dbt-cloud/example-create-merge-job.png" title="Example of creating a merge job"/>

<Lightbox src="/img/docs/dbt-cloud/using-dbt-cloud/example-create-merge-job.png" width="90%" title="Example of creating a merge job"/>
## Verify push events in Git

Merge jobs require push events so make sure they've been enabled in your Git provider, especially if you have an already-existing Git integration. However, for a new integration setup, you can skip this check since push events are typically enabled by default.

<Expandable alt_header="GitHub example" >

The following is a GitHub example of when the push events are already set:

<Lightbox src="/img/docs/dbt-cloud/using-dbt-cloud/example-github-push-events.png" title="Example of the Pushes option enabled in the GitHub settings"/>

</Expandable>

<Expandable alt_header="GitLab example" >

The following is a GitLab example of when the push events are already set:

<Lightbox src="/img/docs/dbt-cloud/using-dbt-cloud/example-gitlab-push-events.png" title="Example of the Push events option enabled in the GitLab settings"/>

</Expandable>

<Expandable alt_header="Azure DevOps example" >

The following is an example of creating a new **Code pushed** trigger in Azure DevOps. Create a new service hooks subscription when code pushed events haven't been set:

<Lightbox src="/img/docs/dbt-cloud/using-dbt-cloud/example-azuredevops-new-event.png" title="Example of creating a new trigger to push events in Azure Devops"/>

</Expandable>
2 changes: 1 addition & 1 deletion website/docs/docs/deploy/monitor-jobs.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ This portion of our documentation will go over dbt Cloud's various capabilities

<Lightbox src="/img/docs/dbt-cloud/deployment/deploy-scheduler.jpg" width="98%" title="An overview of a dbt Cloud job run which contains Run Summary, Job Trigger, Run Duration, and more."/>

<Lightbox src="/img/docs/dbt-cloud/deployment/run-history.jpg" width="95%" title="Run History dashboard allows you to monitor the health of your dbt project and displays jobs, job status, environment, timing, and more."/>
<Lightbox src="/img/docs/dbt-cloud/deployment/run-history.png" width="95%" title="Run History dashboard allows you to monitor the health of your dbt project and displays jobs, job status, environment, timing, and more."/>


<Lightbox src="/img/docs/dbt-cloud/deployment/access-logs.gif" width="85%" title="Access logs for run steps" />
Expand Down
15 changes: 7 additions & 8 deletions website/docs/docs/deploy/run-visibility.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ dbt Cloud developers can access their run history for the last 365 days through

We limit self-service retrieval of run history metadata to 365 days to improve dbt Cloud's performance.

<Lightbox src="/img/docs/dbt-cloud/deployment/run-history.jpg" width="85%" title="Run history dashboard allows you to monitor the health of your dbt project and displays jobs, job status, environment, timing, and more."/>
<Lightbox src="/img/docs/dbt-cloud/deployment/run-history.png" width="85%" title="Run history dashboard allows you to monitor the health of your dbt project and displays jobs, job status, environment, timing, and more."/>

## Job run details

Expand All @@ -27,19 +27,18 @@ Click **Rerun now** to rerun the job immediately.

An example of a completed run with a configuration for a [job completion trigger](/docs/deploy/deploy-jobs#trigger-on-job-completion):

<Lightbox src="/img/docs/dbt-cloud/deployment/example-job-details.png" width="70%" title="Example of run details" />
<Lightbox src="/img/docs/dbt-cloud/deployment/example-job-details.png" width="65%" title="Example of run details" />

## Access logs
### Access logs

You can view or download in-progress and historical logs for your dbt runs. This makes it easier for the team to debug errors more efficiently.

<Lightbox src="/img/docs/dbt-cloud/deployment/access-logs.gif" width="85%" title="Access logs for run steps" />

## Model timing
> Available on [multi-tenant](/docs/cloud/about-cloud/access-regions-ip-addresses) dbt Cloud accounts on the [Team or Enterprise plans](https://www.getdbt.com/pricing/).
### Model timing <Lifecycle status="team,enterprise" />

The model timing dashboard on dbt Cloud displays the composition, order, and time taken by each model in a job run. The visualization appears for successful jobs and highlights the top 1% of model durations. This helps you identify bottlenecks in your runs, so you can investigate them and potentially make changes to improve their performance.
The **Model timing** dashboard displays the composition, order, and time taken by each model in a job run. The visualization appears for successful jobs and highlights the top 1% of model durations. This helps you identify bottlenecks in your runs, so you can investigate them and potentially make changes to improve their performance.

You can find the dashboard on the **Run Overview** page.
You can find the dashboard on the [job's run details](#job-run-details).

<Lightbox src="/img/docs/dbt-cloud/model-timing.jpg" width="85%" title="The model timing tab displays the top 1% of model durations and visualizes model bottlenecks" />
<Lightbox src="/img/docs/dbt-cloud/model-timing.png" width="85%" title="The Model timing tab displays the top 1% of model durations and visualizes model bottlenecks" />
Binary file not shown.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file removed website/static/img/docs/dbt-cloud/model-timing.jpg
Binary file not shown.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.

0 comments on commit ccaf6b9

Please sign in to comment.