A Git LFS server using an OpenStack Swift cluster as backend storage. Implemented in python with flask.
Git LFS is a git command line extension for managing large files. It uses a Git LFS server for retrieving URLs, which are then used for uploading/downloading the large files. git-lfs-swift-server is an implementation of a Git LFS server, which uses a configurable OpenStack Swift cluster to store the data. This server implements the batch API. As web server framework, Flask is used.
git clone https://github.com/cbartz/git-lfs-swift-server
cd git-lfs-swift-server
python setup.py install
In a python file:
AUTH_URL = "https://example.com/auth/v1.0" # Required for token Auth
BASE_URL = "https://example.com" # Required for tempURL Auth
TOKEN_EXPIRY = 3600
LOGFILE = /tmp/logfile
LOGLEVEL = INFO
AUTH_KWARGS = {}
Ensure to set the environment variable GIT_LFS_SWIFT_SETTINGS_FILE=/path/to/file.py, so that flask knows where to retrieve the config.
You can also set directly the settings as environment variables:
export GIT_LFS_SWIFT_AUTH_URL="https://example.com/auth/v1.0"
export GIT_LFS_SWIFT_TOKEN_EXPIRY=3600
export GIT_LFS_SWIFT_LOGFILE=/tmp/logfile
export GIT_LFS_SWIFT_LOGLEVEL=INFO
For testing purposes, flask ships a web server. You can just call:
GIT_LFS_SWIFT_SETTINGS_FILE=/path/to/file.py python -m git_lfs_swift_server.server
For production, use the webserver of your choice. See the Flask documentation for more information.
The swift container and account used for storing the files will be determined by the client. GIT-LFS uses HTTP Basic authentication, and its required to use the same username/password combination as when authenticating against the swift cluster. It is common to use the form user:account as username to authenticate against a swift cluster. This does not work well with HTTP Basic authentication, because the colon : is already used as a delimiter. Therefore, replace : by ; when authenticating against a git-lfs-swift server.
The swift account is automatically retrieved by the username (technically: The storage url the auth middleware is returning is used), if the account is not specified explicitly in the LFS Server URL. Specifying the account explicitly makes sense if the user is accessing a space which does not belong to his own (e.g. using the ACL mechanism).
Beyond that, it is possible to use prefix-based temporary URLs with an empty prefix (thus valid for a whole container). They have the advantage that people without credentials for the swift cluster can access the LFS objects. In this case, no username and password is required and HTTP Basic auth can be disabled in the LFS config file. Authentication with temporary URLs require that the variable BASE_URL is defined in the server config file.
In summary, there are three kinds of a URL:
<host>/<container>
<host>/<account>/<container>
<host>/<account>/<container>/read_<readsig>/write_<writesig>/<expires_at>
Example: If all the files should end up in a container called mycontainer and the domain of the git-lfs-swift server is example.com use following command to setup the LFS URL:
git config lfs.url https://example.com/mycontainer
If mycontainer lies within a different account, specify it before the container part:
git config lfs.url https://example.com/AUTH_otheraccount/mycontainer
And, if a prefix-based temporary URL is used, the command could look like:
git config lfs.url https://example.com/AUTH_account/mycontainer/read_eb1566dd06c757566a46f46134404b6a047913e1/write_45e76be84e45ed9f0c08b5ed63bde3ea64f41100/1503915711/
The git-lfs-swift server supports the required basic transfer mode. But there is an issue with that: Swift clusters have a maximum object size (defaults to 5 GiB). Files larger then this size have to be split up into multiple segments. The basic transfer mode does not support this mechanism. Therefore, the server supports the custom transfer mode called swift, too. This mode is currently not compatible with prefix-based temporary URL authentication.
The server has been only tested with auth version 1.0 . It is possible to add additional kwargs to the auth call, if you specify a dict AUTH_KWARGS in the config file. Therefore in theory, it should be possible to use keystone, too.