Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add dbt_valid_to_current #6308

Merged
merged 43 commits into from
Nov 19, 2024
Merged

add dbt_valid_to_current #6308

merged 43 commits into from
Nov 19, 2024

Conversation

mirnawong1
Copy link
Contributor

@mirnawong1 mirnawong1 commented Oct 17, 2024

adding new config for dbt_valid_to_current, which allows users to set a future date for dbt_valid_to using the dbt_valid_to_current config in yaml/config/project.

Resolves #6275

Outstanding questions:
✅ - Can date be hardcoded using this syntax? "to_date('2024-05-10')" or can they use "to_date('2024, 05, 10')" ?
✅ - is the syntax just a string (e.g. dbt_valid_to_current: '9999, 12, 31' ) or must they use to_date?
✅ - can user use var to return a SQL statement '{{ var('my_future_date') }}'
✅ - can they use a macro that returns a SQL statement '{{ dbt.date(9999, 12, 31) }}'?
✅ - how does this new config work with deferral/state:modified? will we warn users that the config has been updated and they need to manually update their snapshot?


🚀 Deployment available! Here are the direct links to the updated files:

@mirnawong1 mirnawong1 requested a review from a team as a code owner October 17, 2024 15:07
Copy link

vercel bot commented Oct 17, 2024

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name Status Preview Updated (UTC)
docs-getdbt-com ✅ Ready (Inspect) Visit Preview Nov 19, 2024 8:01pm

@github-actions github-actions bot added content Improvements or additions to content Docs team Authored by the Docs team @dbt Labs size: medium This change will take up to a week to address labels Oct 17, 2024
@dbeatty10
Copy link
Contributor

@mirnawong1 grouping your outstanding questions into a handful of categories:

Hardcoded SQL expressions

Question:

  • Can date be hardcoded using this syntax? "to_date('2024-05-10')" or can they use "to_date('2024, 05, 10')" ?

👍 Hardcoded expressions that work within the data platform is the only option that is available currently. The precise syntax will vary per warehouse, so to_date('2024-05-10') may work for some, but others may require something like date(2024, 5, 10).

Jinja within dbt_valid_to_current

Questions:

  • is the syntax just a string (e.g. dbt_valid_to_current: '9999, 12, 31' ) or must they use to_date?
  • can user use var to return a SQL statement '{{ var('my_future_date') }}'
  • can they use a macro that returns a SQL statement '{{ dbt.date(9999, 12, 31) }}'?

👎 Jinja is not currently available for dbt_valid_to_current. So none of the above are possible, and the only option is to supply a full SQL expression that is specific to the users data warehouse.

Deferral / state:modified

Question:

  • how does this new config work with deferral/state:modified? will we warn users that the config has been updated and they need to manually update their snapshot?

👍 Based on hands-on experimentation, it does work as expected with deferral / state:modified. So if this config changes, than it will show up when using --select state:modified.

@dbeatty10 dbeatty10 added the blocked_by_dev Awaiting merge of PR with associated functionality label Oct 28, 2024
@dbeatty10
Copy link
Contributor

@mirnawong1 I added the blocked_by_dev label until this is resolved:

@mirnawong1
Copy link
Contributor Author

@mirnawong1 I added the blocked_by_dev label until this is resolved:

thank you @dbeatty10 ! it looks like this is resolved now right?

Copy link
Collaborator

@graciegoheen graciegoheen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

a few nits!

website/docs/docs/build/snapshots.md Outdated Show resolved Hide resolved
website/docs/docs/build/snapshots.md Outdated Show resolved Hide resolved
website/docs/docs/build/snapshots.md Outdated Show resolved Hide resolved
website/docs/docs/build/snapshots.md Outdated Show resolved Hide resolved
website/docs/docs/dbt-versions/release-notes.md Outdated Show resolved Hide resolved
mirnawong1 and others added 6 commits November 19, 2024 11:44
Co-authored-by: Grace Goheen <53586774+graciegoheen@users.noreply.github.com>
Co-authored-by: Grace Goheen <53586774+graciegoheen@users.noreply.github.com>
Co-authored-by: Grace Goheen <53586774+graciegoheen@users.noreply.github.com>
Co-authored-by: Grace Goheen <53586774+graciegoheen@users.noreply.github.com>
Co-authored-by: Grace Goheen <53586774+graciegoheen@users.noreply.github.com>
@mirnawong1 mirnawong1 enabled auto-merge November 19, 2024 19:56
@mirnawong1 mirnawong1 merged commit 175ad2b into current Nov 19, 2024
9 checks passed
@mirnawong1 mirnawong1 deleted the add-custom-date branch November 19, 2024 20:01
Copy link
Contributor

Check your created Crawler

Check your created index on your Algolia Application

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
content Improvements or additions to content Docs team Authored by the Docs team @dbt Labs size: medium This change will take up to a week to address trigger-crawl
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Core] Allow custom date for dbt_valid_to in snapshots
3 participants