Onyx plugin for Amazon S3.
In your project file:
[org.onyxplatform/onyx-amazon-s3 "0.9.10.0"]
In your peer boot-up namespace:
(:require [onyx.plugin.s3-output])
Catalog entry:
{:onyx/name <<TASK_NAME>>
:onyx/plugin :onyx.plugin.s3-output/output
:s3/bucket <<BUCKET_NAME>>
:s3/serializer-fn :clojure.core/pr-str
:s3/key-naming-fn :onyx.plugin.s3-output/default-naming-fn
:onyx/type :output
:onyx/medium :s3
:onyx/batch-size 20
:onyx/doc "Writes segments to s3 files, one file per batch"}
Segments received by this task will be serialized by the :s3/serializer-fn
,
into a file per batch, placed at a key in the bucket which is named via the
function defined at :s3/key-naming-fn
. This function takes an event map and
returns a string. Using the default naming function, :onyx.plugin.s3-output/default-naming-fn
,
will name keys in the following format in UTC time format:
"yyyy-MM-dd-hh.mm.ss.SSS_batch_BATCH_UUID".
Lifecycle entry:
{:lifecycle/task <<TASK_NAME>>
:lifecycle/calls :onyx.plugin.s3-output/s3-output-calls}
key | type | description |
---|---|---|
:s3/bucket |
string |
The name of the s3 bucket to write the file to |
:s3/serializer-fn |
keyword |
A keyword pointing to a fully qualified function that will serialize the batch of segments to bytes |
:s3/key-naming-fn |
keyword |
A keyword pointing to a fully qualified function that be supplied with the Onyx event map, and produce an s3 key for the batch. |
Many thanks to AdGoji for allowing this work to be open sourced and contributed back to the Onyx Platform community.
Pull requests into the master branch are welcomed.
Copyright © 2016 Distributed Masonry LLC
Distributed under the Eclipse Public License, the same as Clojure.