Plugin for Kong as API Gateway to proxy S3

[liu0013]

Summary

Whether or not kong can make a plugin to upload file into S3 or download file from S3 and cache it?
For Example:
https://github.com/jamesmarlowe/lua-resty-s3
Kong can upload the file and send s3 private url to backend, receive a s3 url from backend to download the file and cache it.
I am not sure that the idea make sense as kong plugin?

[peterbsmyth]

Who would use it and why?

[sheshkovsky]

We've recently used same approach (nginx + lua scripts) to implement a s3 proxy so we don't need to include aws sdk on client-side anymore to upload files to s3. So I would say that could be a useful plugin. But I'm not sure about caching, maybe using a cdn would be a better choice.

[jeremyjpj0916]

Could be a neat Kong core integration feature for Cassandra/Postgres DB backups(which is what we use S3, for but we use a tool called cassandra-snapshotter that is opensource to actually orchestrate that).

But your use case is to live serve a file from S3 with an API proxy call to achieve a sort of authn/authz @ the gateway? I find that a bit indirect because S3 offerings are designed to handle the authn/authz with id/keys but I suppose it could be done.

[sheshkovsky]

@jeremyjpj0916 we generate temp auth credentials (on server-side where we don't have problem to import aws sdk) to grant s3 access for client, then client calls the proxy sending these credentials and we generate aws signature using slightly modified version of this script : https://github.com/Kong/kong/blob/master/kong/plugins/aws-lambda/v4.lua but that would be the idea.

[arulrajnet]

Anyone knows about, how to integrate https://github.com/anomalizer/ngx_aws_auth this with kong?

[carnei-ro]

We have a scenario involving Kong and S3 buckets too, although is to Get objects.

Imagine you have a bucket with a bunch of objects, then you want to put Kong in front of it to be able to expose the objects tweaking the request (maybe Kong plugins to do authn/authz).
Although, when Kong sits in front of a bucket, either Kong gets through s3 "website" endpoint or directly with s3 endpoint (kind of the same behavior). Those requests are unsigned GETs, it means you cannot write a bucket policy to cover "let only that IAM role to get the objects" policy, because those unsigned GETs does not carry information to authenticate the caller (to be simply: the unsigned GETs does not have "authorization" header).
So, with my plugin https://github.com/carnei-ro/kong-plugin-s3, based on aws-lambda, I am signing the GET requests, solving this headache.

So far, I only cover GETs... it is an alpha stage. PRs are welcome to cover e.g. "POST"/"PUT" and "DELETE" requests/operations.