Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use container names to disambiguate backend stores #371

Closed
timuralp opened this issue Aug 27, 2021 · 3 comments · Fixed by #378
Closed

Use container names to disambiguate backend stores #371

timuralp opened this issue Aug 27, 2021 · 3 comments · Fixed by #378

Comments

@timuralp
Copy link
Collaborator

s3proxy disambiguates backend stores by different credentials, which means that from the application side typically a new client needs to be created to connect to the different backends. To reduce friction, it would be great to be able to configure s3proxy with multiple backends with the same credentials and specify the buckets associated with them.

The specific use case is Azure, where multiple storage accounts are common, but being able to hide that complexity from the application would simplify access.

@gaul
Copy link
Owner

gaul commented Aug 27, 2021

How to do these mappings since the existing properties file is too simple? #365 suggests putting them in a SQL database but maybe a JSON file or similar is more appropriate? One advantage of the properties is that its simplicity allows easy configuration via environment variables for Docker containers.

Related to #364. cc @berlinsaint

@timuralp
Copy link
Collaborator Author

I propose to still use the properties file and add a new configuration option, e.g. s3proxy.bucket-locator.buckets.<N>=<bucket name>. When s3proxy compiles the properties, it can add mapping entries (<identity>, <bucket name>) -> blobstore, as opposed to using only the identity as the key.

I recognize that the properties file is limiting, however, it is simple and if we can make this work without introducing a whole new configuration storage mechanism, I think that would be preferred.

@whybeyoung
Copy link

#372

timuralp added a commit to timuralp/s3proxy that referenced this issue Sep 12, 2021
Allows differentiating the backend blobstores by configured buckets.
The commit allows using the same s3proxy credentials across the
different backends and relies on the bucket names to disambiguate them.

The "default" blobstore is the one that occurs first.

Fixes: gaul#371
timuralp added a commit to timuralp/s3proxy that referenced this issue Sep 17, 2021
Allows differentiating the backend blobstores by configured buckets.
The commit allows using the same s3proxy credentials across the
different backends and relies on the bucket names to disambiguate them.

The "default" blobstore is the one that occurs first.

Fixes: gaul#371
timuralp added a commit to timuralp/s3proxy that referenced this issue Sep 17, 2021
Allows differentiating the backend blobstores by configured buckets.
The commit allows using the same s3proxy credentials across the
different backends and relies on the bucket names to disambiguate them.

The "default" blobstore is the one that occurs first.

Fixes: gaul#371
timuralp added a commit to timuralp/s3proxy that referenced this issue Oct 19, 2021
Allows differentiating the backend blob stores by configured buckets.
The commit allows using the same s3proxy credentials across the
different backends and relies on the bucket names to disambiguate them.

If no buckets are configured, the prior blob store selection algorithm
is used, which returns the first configured blob store for the specified
identity.

The patch supports the glob syntax, which allows specifying groups of
buckets more easily. However, there is no checking for overlapping
globs.

Fixes: gaul#371
@gaul gaul closed this as completed in #378 Oct 20, 2021
gaul pushed a commit that referenced this issue Oct 20, 2021
Allows differentiating the backend blob stores by configured buckets.
The commit allows using the same s3proxy credentials across the
different backends and relies on the bucket names to disambiguate them.

If no buckets are configured, the prior blob store selection algorithm
is used, which returns the first configured blob store for the specified
identity.

The patch supports the glob syntax, which allows specifying groups of
buckets more easily. However, there is no checking for overlapping
globs.

Fixes: #371
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
3 participants