Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

should all stores have a path attribute that functions like a URLs path? #2266

Open
d-v-b opened this issue Sep 28, 2024 · 1 comment
Open

Comments

@d-v-b
Copy link
Contributor

d-v-b commented Sep 28, 2024

We are not consistent about the path attribute of stores right now. localstore has a root attribute that's basically a path (with type pathlib.path), remotestore has a v2-style explicit path attribute (with type str), zipstore has a path attribute but I think its semantics are not like remotestore.path, since zipstore.path points to a location outside the zip file system, etc.

I think we should normalize the store.path attribute along the following lines:

  • store.path is a string that names a location inside the filesystem modeled by the store class. All stores have such a path attribute.
  • new store class instances can be created via methods like store_instance.with_path('new_path') (completely new path) or store_instance.join_path('relative_path') (old path + relative_path), or store_instance / 'relative_path'` as an abbreviation of the last one.
  • The StorePath class goes away entirely, because we are giving stores their own path.

(This API is largely influenced by the yarl.URL class, which I have been using quite a bit recently)

Thoughts?

@d-v-b
Copy link
Contributor Author

d-v-b commented Sep 29, 2024

after looking into this a bit more and starting a draft implementation of this idea, I noticed that the semantics of some of the store operations varies too much across the different store implementations. If we compare MemoryStore and RemoteStore:

  • MemoryStore has no path attribute; MemoryStore.list returns all keys in the store. This is not scalable for stores with many keys.
  • RemoteStore has a path attribute; RemoteStore.list only returns the keys that have Remotestore.path as a leading prefix. This makes sense, because attempting to list all the keys in s3 would not be very useful.

I'm increasingly convinced that all stores should have an interface that works for RemoteStore, i.e. a path attribute that serves to narrow the scope of list / create / get actions in the key space of the store.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant