We need to find a way to be able to copy binary files into our backup system

[ypriverol]

@osallou

One of the ideas of having the ftp binary folder is to find a way to copy all the binaries needed to generate the containers into our internal file system. We need to find out a way to allow users or administrators of Biocontainers upon request to copy to the binary folder https://containers.biocontainers.pro/s3/ the binary files.

Best Regards
Yasset

[bgruening]

to copy to the binary folder https://containers.biocontainers.pro/s3/ the binary files.

You mean copy tarballs and binaries into this folder?

[ypriverol]

We have now the backup from your side, which is what we aim at the very beginning. However, we don't have a way to copy a tarball or binary in that folder in the s3. one possible option is that you @bgruening give us access to copy in the original folder and this will be finally reflected in the s3.

[bgruening]

This already exists for years. Its called cargo-port and you can find it here: https://github.com/galaxyproject/cargo-port

[ypriverol]

How can I copy a binary to the ftp using cargo?

Thanks @bgruening for the feedback

[bgruening]

There is documentation in this repo. You need to edit a tsv file add a temporary location of your binary and create a PR. On merge cargo-port will download the binary and archive it.

This is done in this way to give all people the same change to add files and to prevent uploading malicious content. GitHub acts here as a review place for the links and binaries.

[ypriverol]

Ok, we can formally use this mechanism to add the binaries/ tarballs to the system thanks. Can we give to @BioContainers/core members rights to merge there?

[bgruening]

Can we give to @BioContainers/core members rights to merge there?

I think so. If it turns out to be useful and Biocontainers/core keeps maintaining it, why not. Please note it is not my project.

[ypriverol]

Quick Question. I have seen that the url.tsv file hasn't been modified in 3 months. How the bioconda that rely on binaries has been released recently?

Yasset

[ypriverol]

@bgruening I have check one recently added container and you use this path https://github.com/bgruening/download_store/raw/master/thermorawfileparser/thermorawfileparser2018_09_07.zip

Why you didn't use the cargo project? Any special reason?

[bgruening]

Because you told me its not official :-)

[ypriverol]

Then, we go for that PR in that system.

[osallou]

@ypriverol, sorry but i do not understand what you mean here by 'binaries needed to generate the containers'
What do you need to save here for biocontainers?
Admin (we) can upload things there, using s3 cmds...
Users, who and what would they upload there?

[ypriverol]

@osallou I have a binary now and I would like to generate my container with it. First thing, I will need to copy the binary to a server like the s3 and then create the container. This probably can be achieved by a similar mechanism that @bgruening proposed before using a controlled tsv and PRs.

[osallou]

Generate a container from a binary? How differebt is it from creating your dockerfile and putting necessary files in git repo? Le dim. 23 sept. 2018 à 09:15, Yasset Perez-Riverol < notifications@github.com> a écrit :

…

I have a binary now and I would like to generate my container with it. First thing, I will need to copy the binary to a server like the s3 and then create the container. This probably can be achieved by a similar mechanism that @bgruening <https://github.com/bgruening> proposed before using a controlled *tsv* and PRs. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#2 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AA-gYvK0snHHeIjNpzwjC62f-KxOvNGgks5udzUFgaJpZM4W1U86> .

[ypriverol]

We can force the creator to create a git repo with the binary. But when we accept the PR we should copy the file to our internal S3 folder. Whit this we guarantee that we can reproduce the container in case the binary/repo in git is deleted. Is that ok @osallou ?

[osallou]

But binaries could be put in containers repo itself. Furthermore, should we accept containers from binaries only? Are sources not mandatory (open source)? Le dim. 23 sept. 2018 à 10:23, Yasset Perez-Riverol < notifications@github.com> a écrit :

…

We can force the creator to create a git repo with the binary. But when we accept the PR we should copy the file to our internal S3 folder. Whit this we guarantee that we can reproduce the container in case the binary/repo in git is deleted. Is that ok @osallou <https://github.com/osallou> ? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#2 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AA-gYrLAbCoJGLMYqgjK4OOjg-0_cQSVks5ud0UGgaJpZM4W1U86> .

[ypriverol]

But binaries could be put in containers repo itself.
This is when the container has been created by us. But int some cases the binaries are already in github and we don't need to copy them again into containers repo. But we should keep the copy of all the binaries in the S3 because if for some reason the binaries can't be store in the future, we have already them in S3.

Furthermore, should we accept containers from binaries only? Are sources
not mandatory (open source)?

We have containers already from binaries. Even if the code is available some specific arch is needed to produce the binary, and it will take times in our side.

[osallou]

as cargo-port is already here and matches those requirements, I think we can close this issue.
Maybe a link to this repo (and explaining why) in biocontainers web site would be enough