Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

added background subtraction #48

Open
wants to merge 24 commits into
base: dev
Choose a base branch
from
Open

added background subtraction #48

wants to merge 24 commits into from

Conversation

RobJY
Copy link
Contributor

@RobJY RobJY commented Jul 25, 2024 •
edited
Loading

PR checklist

  • This comment contains a description of changes (with reason).
  • If you've fixed a bug or added code that should be tested, add tests!
  • If you've added a new tool - have you followed the pipeline conventions in the contribution docs
  • If necessary, also make a PR on the nf-core/mcmicro branch on the nf-core/test-datasets repository.
  • Make sure your code lints (nf-core lint).
  • Ensure the test suite passes (nextflow run . -profile test,docker --outdir <OUTDIR>).
  • Check for unexpected warnings in debug mode (nextflow run . -profile debug,test,docker --outdir <OUTDIR>).
  • Usage Documentation in docs/usage.md is updated.
  • Output Documentation in docs/output.md is updated.
  • CHANGELOG.md is updated.
  • README.md is updated (including new tool citations and authors/contributors).

@github-actions GitHub Actions
Copy link

github-actions bot commented Jul 25, 2024 •
edited
Loading

nf-core pipelines lint overall result: Failed ❌

Posted for pipeline commit 4f3921b

+| ✅ 169 tests passed       |+
#| ❔   1 tests were ignored |#
!| ❗  28 tests had warnings |!
-| ❌  10 tests failed       |-

❌ Test failures:

  • nextflow_config - Config variable (incorrectly) found: params.max_cpus
  • nextflow_config - Config variable (incorrectly) found: params.max_memory
  • nextflow_config - Config variable (incorrectly) found: params.max_time
  • nextflow_config - Old lines for loading custom profiles found. File should contain: ```groovy
    // Load nf-core custom profiles from different Institutions
    includeConfig !System.getenv('NXF_OFFLINE') && params.custom_config_base ? "${params.custom_config_base}/nfcore_custom.config" : "/dev/null"
  • files_unchanged - .github/CONTRIBUTING.md does not match the template
  • files_unchanged - .github/PULL_REQUEST_TEMPLATE.md does not match the template
  • files_unchanged - .github/workflows/linting_comment.yml does not match the template
  • files_unchanged - .github/workflows/linting.yml does not match the template
  • files_unchanged - .prettierignore does not match the template
  • actions_awsfulltest - .github/workflows/awsfulltest.yml is not triggered correctly

❗ Test warnings:

  • files_exist - File not found: conf/igenomes_ignored.config
  • nextflow_config - nf-validation has been detected in the pipeline. Please migrate to nf-schema: https://nextflow-io.github.io/nf-schema/latest/migration_guide/
  • nextflow_config - Config manifest.version should end in dev: 2.0.0
  • readme - README contains the placeholder zenodo.XXXXXXX. This should be replaced with the zenodo doi (after the first release).
  • pipeline_todos - TODO string in README.md: TODO nf-core:
  • pipeline_todos - TODO string in README.md: Include a figure that guides the user through the major workflow steps. Many nf-core
  • pipeline_todos - TODO string in README.md: Fill in short bullet-pointed list of the default steps in the pipeline
  • pipeline_todos - TODO string in README.md: Describe the minimum required steps to execute the pipeline, e.g. how to prepare samplesheets.
  • pipeline_todos - TODO string in README.md: update the following command to include all required parameters for a minimal example
  • pipeline_todos - TODO string in README.md: If applicable, make list of people who have also contributed
  • pipeline_todos - TODO string in README.md: Add citation for pipeline after first release. Uncomment lines below and update Zenodo doi and badge at the top of this file.
  • pipeline_todos - TODO string in README.md: Add bibliography of tools and data used in your pipeline
  • pipeline_todos - TODO string in methods_description_template.yml: #Update the HTML below to your preferred methods description, e.g. add publication citation for this pipeline
  • pipeline_todos - TODO string in test.config: Specify the paths to your test data on nf-core/test-datasets
  • pipeline_todos - TODO string in test.config: Give any required params for the test so that command line flags are not needed
  • pipeline_todos - TODO string in test_full.config: Specify the paths to your full test data ( on nf-core/test-datasets or directly in repositories, e.g. SRA)
  • pipeline_todos - TODO string in test_full.config: Give any required params for the test so that command line flags are not needed
  • pipeline_todos - TODO string in base.config: Check the defaults for all processes
  • pipeline_todos - TODO string in base.config: Customise requirements for specific processes.
  • pipeline_todos - TODO string in main.nf: Optionally add in-text citation tools to this list.
  • pipeline_todos - TODO string in main.nf: Optionally add bibliographic entries to this list.
  • pipeline_todos - TODO string in main.nf: Only uncomment below if logic in toolCitationText/toolBibliographyText has been filled!
  • pipeline_todos - TODO string in ci.yml: You can customise CI pipeline run tests as required
  • pipeline_todos - TODO string in awsfulltest.yml: You can customise AWS full pipeline tests as required
  • pipeline_todos - TODO string in output.md: Write this documentation describing your workflow's output
  • pipeline_todos - TODO string in usage.md: Add documentation about anything specific to running your pipeline. For general topics, please point to (and add to) the main nf-core website.
  • schema_lint - Parameter input not found in schema
  • nfcore_yml - nf-core version in .nf-core.yml is not set to the latest version. Should be 3.0.0 but was 2.14.1

❔ Tests ignored:

✅ Tests passed:

Run details

  • nf-core/tools version 3.0.0
  • Run at 2024-10-08 18:27:41

@RobJY RobJY requested a review from jmuhlich July 25, 2024 19:40
@RobJY RobJY changed the title added bacground subtraction added background subtraction Jul 25, 2024
kbestak
kbestak reviewed Aug 7, 2024
View reviewed changes
Copy link
Contributor

@kbestak kbestak left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me! I would only change the exposure time to be a number to allow for floats as we have a dataset with such exposure times.

I'll be hopefully updating the nf-core module for Backsub soon so it adheres to nf-core standards more with conda support.

I'm leaning towards having a different step name than the tool name, so for the step name, I think it should be background/af_correction/channel_subtraction, with Backsub as the tool to do it. What are the preferences from your side?

assets/schema_marker.json Outdated Show resolved Hide resolved
@jmuhlich
Copy link
Member

jmuhlich commented Sep 6, 2024

I'm leaning towards having a different step name than the tool name, so for the step name, I think it should be background/af_correction/channel_subtraction, with Backsub as the tool to do it. What are the preferences from your side?

From what I see in other nf-core pipelines, modules are imported and used under their original names (unless a module is used multiple times and must be aliased for uniqueness) and publishDir folder names also generally follow the module name. Sometimes publishDir paths are grouped under a higher level folder if there are many alternatives for the same step like we have for segmentation. The mcmicro-legacy module names and publishDir layout did not use this style, rather naming everything by its purpose, e.g "registration", "segmentation", "quantification", and we did start to use that same publishDir structure here in mcmicro 2.0. I think we should pivot to adopting the nf-core style for consistency though.

jmuhlich
jmuhlich requested changes Sep 6, 2024
View reviewed changes
nextflow.config Show resolved Hide resolved
subworkflows/local/utils_nfcore_mcmicro_pipeline/main.nf Show resolved Hide resolved
tests/main.nf.test Outdated Show resolved Hide resolved
workflows/mcmicro.nf Outdated Show resolved Hide resolved
workflows/mcmicro.nf Outdated Show resolved Hide resolved
@kbestak
Copy link
Contributor

kbestak commented Sep 6, 2024

Sometimes publishDir paths are grouped under a higher level folder if there are many alternatives for the same step like we have for segmentation. The mcmicro-legacy module names and publishDir layout did not use this style, rather naming everything by its purpose, e.g "registration", "segmentation", "quantification", and we did start to use that same publishDir structure here in mcmicro 2.0. I think we should pivot to adopting the nf-core style for consistency though.

I was under the impression our goal is to still keep the multiple choice aspect based on steps - especially when it will come to defining the "execution stream" - e.g. staging, registration, AF_correction, segmentation and quantification. The broader steps are defined and used to determine the "stream", but for each step (or several steps), we would allow for multiple options (e.g. staging, registration*, segmentation). I think it would be somewhat confusing to start mixing process names into the broader execution stream which is why I made the point above.

jmuhlich
jmuhlich requested changes Sep 25, 2024
View reviewed changes
Copy link
Member

@jmuhlich jmuhlich left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure about the direction this is going -- the code is way too complex and it's actually incorrect as it runs segmentation and quantification on BOTH the un-backsubbed images and the backsubbed images. Let's revert to the original implementation with backsub in params for now.

subworkflows/local/utils_nfcore_mcmicro_pipeline/main.nf Outdated Show resolved Hide resolved
subworkflows/local/utils_nfcore_mcmicro_pipeline/main.nf Outdated Show resolved Hide resolved
@RobJY
Copy link
Contributor Author

RobJY commented Sep 26, 2024

Oh, I didn't realize I was getting both of those channels. I was so happy to get it to pass the tests, I didn't pay close enough attention to the number of output files I was generating. I'll keep a better eye on that in the future. Thank you for pointing it out!
I've reverted back to the input parameter for backsub as you suggested and it seems to be working now.

@RobJY
Copy link
Contributor Author

RobJY commented Sep 30, 2024

I made a branch as suggested of the attempt to run backsub without an input parameter.

@RobJY
Copy link
Contributor Author

RobJY commented Sep 30, 2024

I think this is ready to merge.
Please let me know if I've missed anything or if you see any new issues. Thanks!

jmuhlich
jmuhlich requested changes Oct 4, 2024
View reviewed changes
tests/main.nf.test Outdated Show resolved Hide resolved
tests/main.nf.test Outdated Show resolved Hide resolved
jmuhlich
jmuhlich requested changes Oct 7, 2024
View reviewed changes
tests/main.nf.test Show resolved Hide resolved
@RobJY
Copy link
Contributor Author

RobJY commented Oct 8, 2024

I fixed a lint error with the schema URL, but now we're getting new lint errors. Looks like the latest update has made some changes.
I'll get to work on these.

@RobJY
Copy link
Contributor Author

RobJY commented Nov 7, 2024

The combination of pulling in the template update in dev and adding the logo file to the files_unchanged section of .nf-core.yml seems to have fixed our lint errors.
Please let me know if you see any other issues. Thanks!

@kbestak
Copy link
Contributor

kbestak commented Nov 14, 2024 •
edited
Loading

Hi,

I've noticed a few issues with the Backsub and changed them: RobJY#5

  • segmentation would always take ASHLAR outputs since the tma section always defaults to it
  • I added an error and set the minimal value to 0.001 for the exposure column as it should be a positive number.
  • validation of the marker sheet
    • was position dependant (e.g. having the background column in position 4 meant it would not be validated)
    • was missing some backsub-specific cases

Right now I would suggest the remove functionality of Backsub to be ignored, as it is not taken into account (if added, for segmentation channel specification this would be an essential check), but right now I don't think it is essential.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@kbestak kbestak kbestak left review comments

@jmuhlich jmuhlich jmuhlich requested changes

Requested changes must be addressed to merge this pull request.

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@RobJY @jmuhlich @kbestak