using snakedeploy with repos not on github

[nate-d-olson]

• snakedeploy version: 0.3.0
• Python version: 3.10.1
• Operating System: Debian GNU/Linux 9

Description

Would like to use snakedelopy with either a local git repository or an internal gitlab server.

I get the following error that the object has no attribute get_repo_name

I understand that this repo is under activate development and you might not be able to support this functionality at this time.

What I Did

When I tried to use the gitlab url

snakedeploy deploy-workflow --tag v0.002 https://gitlab.nist.gov/gitlab/njd2/giab-asm-bench-whole-genome /working/geneteam/testdeploy
Writing Snakefile with module definition...
Traceback (most recent call last):
  File "/working/geneteam/mambaforge/envs/defrabb/bin/snakedeploy", line 10, in <module>
    sys.exit(main())
  File "/working/geneteam/mambaforge/envs/defrabb/lib/python3.10/site-packages/snakedeploy/client.py", line 156, in main
    deploy(
  File "/working/geneteam/mambaforge/envs/defrabb/lib/python3.10/site-packages/snakedeploy/deploy.py", line 36, in deploy
    name=name or provider.get_repo_name().replace("-", "_"),
AttributeError: 'NoneType' object has no attribute 'get_repo_name'

When I tried to use a local path

 snakedeploy deploy-workflow --tag v0.002 /working/geneteam/giab-asm-bench-whole-genome testdeploy_local
Writing Snakefile with module definition...
Traceback (most recent call last):
  File "/working/geneteam/mambaforge/envs/defrabb/bin/snakedeploy", line 10, in <module>
    sys.exit(main())
  File "/working/geneteam/mambaforge/envs/defrabb/lib/python3.10/site-packages/snakedeploy/client.py", line 156, in main
    deploy(
  File "/working/geneteam/mambaforge/envs/defrabb/lib/python3.10/site-packages/snakedeploy/deploy.py", line 36, in deploy
    name=name or provider.get_repo_name().replace("-", "_"),
AttributeError: 'NoneType' object has no attribute 'get_repo_name'

[vsoch]

@nate-d-olson I don't see that this is publicly available - do you have a public repo at the same url to test? If it's an issue of permissions we will need to add that. If it's an issue of parsing the gitlab url we can test that, but I'd like to figure out if it's just that it's private first. If it's the URL, then what I'll need to help and test is a public GitLab repository to develop from.

[nate-d-olson]

@vsoch Thank you for the quick response. Unfortunately, all the repositories or https://gitlab.nist.gov/gitlab, are not publically accessible as this gitlab instance is hosted on an internal server. I am happy to help update and test the codebase so that the tool works with similar internal gitlab servers. Using a local file path works for us as well and maybe easier to implement and test and useful to more users.

[vsoch]

okay - I can't work on this during the workday but let me give a shot at adding a GitLab provider. It might be that simple! I should have some time this evening.

[vsoch]

See #27

[cpauvert]

@nate-d-olson I don't see that this is publicly available - do you have a public repo at the same url to test? If it's an issue of permissions we will need to add that. If it's an issue of parsing the gitlab url we can test that, but I'd like to figure out if it's just that it's private first. If it's the URL, then what I'll need to help and test is a public GitLab repository to develop from.

Hi,
I have the same problem (with snakedeploy version: 0.3.0 as well) and believe it is a question of parsing the url as well. Using the reprex below, I reproduce this error on a publicly available repository:

# Mimic a local git repository
git clone https://github.com/snakemake-workflows/rna-seq-star-deseq2
# Deploy from a local git repository (with various syntax)
snakedeploy deploy-workflow rna-seq-star-deseq2 use-workflow-as-module --tag v.1.2.0
snakedeploy deploy-workflow rna-seq-star-deseq2/ use-workflow-as-module --tag v.1.2.0
snakedeploy deploy-workflow ./rna-seq-star-deseq2 use-workflow-as-module --tag v.1.2.0
snakedeploy deploy-workflow ./rna-seq-star-deseq2/ use-workflow-as-module --tag v.1.2.0

All the syntaxes above generate the same error message (displayed first by @nate-d-olson)

AttributeError: 'NoneType' object has no attribute 'get_repo_name'

Best,

[vsoch]

Ah I don't think we parse local repos - that can be added! Could someone please test #27 before I do another PR?

[cpauvert]

Ok sorry @vsoch, I based my comment on the first message of @nate-d-olson which mentioned local repos. Unfortunately I do not have a gitlab repo to test #27, when I use the original code from @nate-d-olson with your PR (snakedeploy v0.3.0+1.g6e83087):

snakedeploy deploy-workflow --tag v0.002 https://gitlab.nist.gov/gitlab/njd2/giab-asm-bench-whole-genome testdeply27

I do not have the access as well so ended up with a time out:

Writing Snakefile with module definition...
Obtaining source repository...
Cloning into '.'...
fatal: unable to access 'https://gitlab.nist.gov/gitlab/njd2/giab-asm-bench-whole-genome/': Failed to connect to gitlab.nist.gov port 443: Connection timed out
('Failed to clone repository {}:\n{}', 'https://gitlab.nist.gov/gitlab/njd2/giab-asm-bench-whole-genome', CalledProcessError(128, ['git', 'clone', 'https://gitlab.nist.gov/gitlab/njd2/giab-asm-bench-whole-genome', '.']))

When used locally, I have the error with get_repo_name.

[vsoch]

@cpauvert it's no worry! I think we need @nate-d-olson to test since he has access to clone the repo, and if we need to add credentials in there somewhere we can discuss that! This is an extra project I work on in free time so I move a bit serially with respect to feature request -> testing -> merge (and then onto the next!)

[nate-d-olson]

@vsoch and @cpauvert https://gitlab.nist.gov/gitlab/njd2/giab-asm-bench-whole-genome is behind an internal firewall and password protected. This is probably a rare use case. I am happy to help with testing but think implementing functionality for using snakedeploy with local repositories would help more snakemake deploy users. Thanks for your help!

[vsoch]

@nate-d-olson could you point me on an open GitLab repo to test? I don't really use GitLab beyond a few repos.

[nate-d-olson]

@vsoch sorry for the back and forth. I cannot create an open GitLab.nist.gov repository. I set up a free gitlab account and want to set up a snakemake repo for testing can you recommend a simple snakemake pipeline repository for me to fork for use in testing snakedeploy?

[vsoch]

Sure thing, and that would be so great! So there is the workflow catalog that has a ton: https://snakemake.github.io/snakemake-workflow-catalog/ What I usually do is use the default tutorial on the website. What you can do that might be easiest is to start with my spack + snakemake tutorial: https://github.com/snakemake/snakemake-spack-tutorial but:

1. don't clone the add/spack branch just use default
2. use this Snakefile instead (without spack) https://snakemake.readthedocs.io/en/stable/tutorial/basics.html#summary

You can also just start from the tutorial repo and grab that same Snakefile: https://github.com/snakemake/snakemake-tutorial-data

[cpauvert]

@vsoch Do I open a new issue regarding the local repos to not mess up the work and discussions here?

[vsoch]

A local repo is much easier to test so I can probably just take a shot at this point and open another PR - will add to my TODO list for the weekend.

[nate-d-olson]

@vsoch here is the repo I set up for testing, https://gitlab.com/natedolson/snaketestworkflow. I am happy to run some tests myself or help you setup a testing framework using this example public gitlab repo. Thanks!

[vsoch]

perfecto! I'll work on these both this evening and weekend. Happy Friday y'all!

[vsoch]

@nate-d-olson can you double check if that repo is public? It's asking me to login to see it!

[nate-d-olson]

@nate-d-olson can you double check if that repo is public? It's asking me to login to see it!

Sorry, the gitlab repos are private by default. I recreated the repo as public, let me know if you are still unable to access it. https://gitlab.com/nate-d-olson/snaketestworkflow.git

[vsoch]

okay I see it now! Thank you!

[vsoch]

hey @nate-d-olson ! Okay I think I'm almost done, but I did mess up in my instructions to you. I'm looking at the https://snakemake.github.io/snakemake-workflow-catalog/ and I think an "official" snakemake workflow has a workflow/ and config/ directory - that seems to be the pattern we expect when we generate the deployment. I'm wondering if snakedeploy could also support a Snakefile in the root - @johanneskoester what do you think?

[vsoch]

I don't see any reason we couldn't find the Snakefile in the repo and customize the path for that - I'm going to give it a shot and see what happens :)

[vsoch]

Okay so if I'm understanding this correctly, snakedeploy is only going to work for workflows for which the data / assets outside of the snakefile and config are at remotes (e.g., no local data directories). I guess I'm wondering what the advantage is over just a clone and then run with snakemake?

[vsoch]

okay refactor is done for GitLab and local #27 but we do need a bit of discussion about the details. We probably need to pull in @johanneskoester for some of the discussion - if he doesn't hop into discussion by next weekend I'll ping him via email.

[cpauvert]

Okay so if I'm understanding this correctly, snakedeploy is only going to work for workflows for which the data / assets outside of the snakefile and config are at remotes (e.g., no local data directories). I guess I'm wondering what the advantage is over just a clone and then run with snakemake?

IMHO, you can use a whole chunk of the workflow and include it in your own workflow in a modular way to do something with the output of the worklow without having to have all the directory/file structure (envs, schemas, scripts, rules etc.). But it depends on the case I guess.