Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Extract the 3 way synk into pypi installable lib #32

Open
franTarkenton opened this issue Sep 11, 2023 · 0 comments
Open

Extract the 3 way synk into pypi installable lib #32

franTarkenton opened this issue Sep 11, 2023 · 0 comments

Comments

@franTarkenton
Copy link
Member

A common operation that takes place with data used by hydrological models is what I refer to as a 3 way sync.

  • Data exists remotely by a data provider (most often federal gov datamart )
  • Local copy is created for rapid access / processing
  • The data gets persisted in object storage.

If a process should crash, It looks to object storage and will preferentially pull the data from object storage vs the original remote.

Operations to support this work have been wrapped up into a python module. To simplify the development of future work thinking that all data pipelines that require this flow would use the same code.

This ticket would see the extraction of the object storage sync into its own pipy installable module.
Could either create a new repo, or potentially add to the existing NR object storage lib.
(if make it part of NR object storage lib https://github.com/bcgov/nr-objectstore-util)

@franTarkenton franTarkenton moved this to In Progress in RFC Backlog Dec 5, 2023
@franTarkenton franTarkenton moved this from In Progress to New Issues in RFC Backlog Dec 5, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: New Issues
Development

No branches or pull requests

1 participant