You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
When ingestion happens from an operation database in snapshot mode (no event/info dates), and the database was unavailable for a specific date, it might not make sense to load that data with previous information date since the new data with thew new information date is going to be loaded anyway, and it is going to have the same contents.
Feature
Add an option to not track late data for a table.
Example
pramen.metastore {
tables = [
{
name = "table_name"# ...# If true, Pramen will try loading all previous partition/info dates # If false, Pramen won't try filling gaps, will just load data to the current partition/information datetrack.late.data = false
}
]
The text was updated successfully, but these errors were encountered:
Background
When ingestion happens from an operation database in snapshot mode (no event/info dates), and the database was unavailable for a specific date, it might not make sense to load that data with previous information date since the new data with thew new information date is going to be loaded anyway, and it is going to have the same contents.
Feature
Add an option to not track late data for a table.
Example
The text was updated successfully, but these errors were encountered: