You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
A common operation that takes place with data used by hydrological models is what I refer to as a 3 way sync.
Data exists remotely by a data provider (most often federal gov datamart )
Local copy is created for rapid access / processing
The data gets persisted in object storage.
If a process should crash, It looks to object storage and will preferentially pull the data from object storage vs the original remote.
Operations to support this work have been wrapped up into a python module. To simplify the development of future work thinking that all data pipelines that require this flow would use the same code.
This ticket would see the extraction of the object storage sync into its own pipy installable module.
Could either create a new repo, or potentially add to the existing NR object storage lib.
(if make it part of NR object storage lib https://github.com/bcgov/nr-objectstore-util)
The text was updated successfully, but these errors were encountered:
A common operation that takes place with data used by hydrological models is what I refer to as a 3 way sync.
If a process should crash, It looks to object storage and will preferentially pull the data from object storage vs the original remote.
Operations to support this work have been wrapped up into a python module. To simplify the development of future work thinking that all data pipelines that require this flow would use the same code.
This ticket would see the extraction of the object storage sync into its own pipy installable module.
Could either create a new repo, or potentially add to the existing NR object storage lib.
(if make it part of NR object storage lib https://github.com/bcgov/nr-objectstore-util)
The text was updated successfully, but these errors were encountered: