You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The current harvest validation is based on the prior AWS deployment and retrieves data from S3, the task needs to be refactored to use the local data harvest location and archive data to validate the harvest.
Compare the harvest output dataframe to the most recent archive dataframe
If they are identical, no new data was harvested and the remainder of the ETL pipeline should be skipped
If there are changes, update the harvested output data.csv with the diff from the new harvest
The text was updated successfully, but these errors were encountered:
The current harvest validation is based on the prior AWS deployment and retrieves data from S3, the task needs to be refactored to use the local data harvest location and archive data to validate the harvest.
data.csv
with the diff from the new harvestThe text was updated successfully, but these errors were encountered: