Docs updates for Applied Source Metadata

[mikealfare]

Contributions

• I have read the contribution docs, and understand what's expected of me.

Link to the page on docs.getdbt.com requiring updates

I'll find this, but it would probably be a page about docs generate that lists what adapters support what features.

What part(s) of the page would you like to see updated?

Add that dbt-redshift supports filtered get_catalog and metadata-based source freshness.

Additional information

No response

[nghi-ly]

hey @mikealfare : if basically we just want a table that lists what's supported with what, i'm thinking that info can live on this page:

https://docs.getdbt.com/docs/dbt-cloud-apis/project-state

since it's the place where "Applied State" is talked about. thoughts?

for the table, do we want the rows to be adapter names and the columns to be the metadata?

[mikealfare]

Discovery API sounds right to me. I think that's where we ultimately surface this information anyway. @graciegoheen / @dbeatty10, does that sound right to you too? I think we would want the rows to be adapters as you mention, and then maybe a column for catalog information and a column for source freshness, each indicating the "level" of support for the information. So something like:

| Adapter       | Catalog     | Source Freshness      |
|===============|=============|=======================|
| dbt-snowflake | incremental | metadata-based        |
| dbt-spark     | manual run  | loaded_at field-based |

And then we should have something that explains the "level"s:

Source Freshness can be measured via metadata when the platform and adapter support it. This allows for calculating source freshness without a loaded_at field and without querying the table directly. This is much faster and more flexible. This can also be overridden if desired by providing a loaded_at field in the model config. If the platform/adapter do not support this, then using a loaded_at field is still an option.

The catalog can be built incrementally for adapters that support it. This allows for the catalog to be built along with the model, which eliminates the need to run a lengthy dbt docs generate at the end of a dbt run. For adapters that do not support incremental catalog generation, you will need to run dbt docs generate to build the catalog.

I would want someone else, either @nghi-ly, @graciegoheen, or @dbeatty10 to review the verbiage I'm using here, including the terms (e.g. loaded_at, "manual" vs. "incremental", "metadata-based" vs "field-based", etc.).