Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

added background subtraction #48

Open
wants to merge 24 commits into
base: dev
Choose a base branch
from
Open

added background subtraction #48

Show file tree
Hide file tree
Changes from 17 commits
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
ca57e14
seems to be working but needs more tests and clean-up
RobJY Jul 24, 2024
aabcf52
clean-up
RobJY Jul 24, 2024
fc9dc82
clean-up
RobJY Jul 24, 2024
8e54128
changed storeDir to the workDir
RobJY Jul 24, 2024
6f2ebfc
clean-up
RobJY Jul 25, 2024
0ffa4d6
added backsub to config
RobJY Jul 25, 2024
3056e07
fixed backsub command with suggestions from PR
RobJY Sep 16, 2024
cc37008
fixed ch_segmentation_input and cleaned-up some
RobJY Sep 18, 2024
dc67b4d
removed params.backsub
RobJY Sep 18, 2024
c430f14
added commented out dev code
RobJY Sep 18, 2024
3f0363c
added commented out attempts to make code more compact for discussion
RobJY Sep 19, 2024
9539963
reverted to backsub input parameter
RobJY Sep 26, 2024
0c939b7
made changes suggested in PR
RobJY Sep 26, 2024
d0136ad
merged coreograph updates and fixed conflicts
RobJY Sep 30, 2024
341126a
removed whitespace causing lint error
RobJY Sep 30, 2024
12f6a69
Update assets/schema_marker.json
RobJY Sep 30, 2024
decacc3
updated backsub test with suggested changes from the PR
RobJY Oct 4, 2024
07ec4d0
bug fix for backsub test
RobJY Oct 8, 2024
4f3921b
lint fix for schema
RobJY Oct 8, 2024
5492813
accepting automatic lint fixes
RobJY Oct 9, 2024
2961517
updated config from new template
RobJY Oct 9, 2024
32558e2
added suggested template changes
RobJY Oct 10, 2024
950c896
merged in dev template update
RobJY Nov 7, 2024
4589851
added nf-core-mcmicro_logo_dark.png to files_unchanged section of .nf…
RobJY Nov 7, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 5 additions & 5 deletions assets/markers_1_sp.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
channel_number,cycle_number,marker_name,filter,excitation_wavelength,emission_wavelength,background
21,1,DNA_6,DAPI,395,431,21
22,1,ELA NE,FITC,485,525,21
23,1,CD57,Sy tox,555,590,21
24,1,CD45,Cy5,640,690,21
channel_number,cycle_number,marker_name
21,1,DNA_6
22,1,ELA NE
23,1,CD57
24,1,CD45
9 changes: 9 additions & 0 deletions assets/schema_marker.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,15 @@
"emission_wavelength": {
"type": "integer",
"errorMessage": ""
},
"exposure": {
"type": "number"
},
"background": {
"type": "string"
},
"remove": {
"type": "boolean"
}
},
"required": ["channel_number", "cycle_number", "marker_name"]
Expand Down
3 changes: 3 additions & 0 deletions nextflow.config
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,9 @@ params {
// Illumination correction
illumination = null

// Background subtraction
backsub = false
RobJY marked this conversation as resolved.
Show resolved Hide resolved

// MultiQC options
multiqc_config = null
multiqc_title = null
Expand Down
4 changes: 4 additions & 0 deletions nextflow_schema.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -79,6 +79,10 @@
"type": "string",
"description": "optional model file for cellpose segmentation"
},
"backsub": {
"type": "boolean",
"description": "boolean to flag whether or not to apply background subtraction"
},
"pixel_size": {
"type": "number",
"description": "Pixel width of input image data, in microns",
Expand Down
25 changes: 24 additions & 1 deletion subworkflows/local/utils_nfcore_mcmicro_pipeline/main.nf
View file
Open in desktop
RobJY marked this conversation as resolved.
Show resolved Hide resolved
Original file line number Diff line number Diff line change
Expand Up @@ -224,12 +224,35 @@ def validateInputMarkersheet( markersheet_data ) {
error("Duplicate [channel, cycle] pairs: ${dups}")
}

// validate backsub columns if present
def exposure_list = markersheet_data.findResults{ _1, _2, _3, _4, _5, _6, exposure, _8, _9 -> exposure ?: null }
def background_list = markersheet_data.findResults{ _1, _2, _3, _4, _5, _6, _7, background, _9 -> background ?: null }
def remove_list = markersheet_data.findResults{ _1, _2, _3, _4, _5, _6, _7, _8, remove -> remove ?: null }

if (!background_list && (exposure_list || remove_list)) {
error("No values in background column, but values in either exposure or remove columns. Must have background column values to perform background subtraction.")
} else if (background_list) {
inter_list = marker_name_list.intersect(background_list)
if (inter_list.size() != background_list.size()) {
outliers_list = background_list - inter_list
error('background column values must exist in the marker_name column. The following background column values do not exist in the marker_name column: ' + outliers_list)
}

if (!exposure_list) {
error('You must have at least one value in the exposure column to perform background subtraction')
}

if (!remove_list) {
error ('You must have at least one value in the remove column to perform background subtraction')
}
}

return markersheet_data
}

def validateInputSamplesheetMarkersheet ( samples, markers ) {
def sample_cycles = samples.collect{ meta, image_tiles, dfp, ffp -> meta.cycle_number }
def marker_cycles = markers.collect{ channel_number, cycle_number, marker_name, filter, excitation_wavelength, emission_wavelength -> cycle_number }
def marker_cycles = markers.collect{ channel_number, cycle_number, marker_name, _1, _2, _3, _4, _5, _6 -> cycle_number }

if (marker_cycles.unique(false) != sample_cycles.unique(false) ) {
error("cycle_number values must match between sample and marker sheets")
Expand Down
193 changes: 128 additions & 65 deletions tests/main.nf.test
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,10 +24,10 @@ nextflow_workflow {
)
input[1] = Channel.of(
[
[1,1,'DNA_6',[],[],[]],
[2,1,'ELANE',[],[],[]],
[3,1,'CD57',[],[],[]],
[4,1,'CD45',[],[],[]],
[1,1,'DNA_6',[],[],[],[],[],[]],
[2,1,'ELANE',[],[],[],[],[],[]],
[3,1,'CD57',[],[],[],[],[],[]],
[4,1,'CD45',[],[],[],[],[],[]],
],
)
"""
Expand Down Expand Up @@ -68,10 +68,10 @@ nextflow_workflow {
)
input[1] = Channel.of(
[
[1,1,'DNA_6',[],[],[]],
[2,1,'ELANE',[],[],[]],
[3,1,'CD57',[],[],[]],
[4,1,'CD45',[],[],[]],
[1,1,'DNA_6',[],[],[],[],[],[]],
[2,1,'ELANE',[],[],[],[],[],[]],
[3,1,'CD57',[],[],[],[],[],[]],
[4,1,'CD45',[],[],[],[],[],[]],
],
)
"""
Expand Down Expand Up @@ -113,10 +113,10 @@ nextflow_workflow {
)
input[1] = Channel.of(
[
[1,1,'DNA_6',[],[],[]],
[2,1,'ELANE',[],[],[]],
[3,1,'CD57',[],[],[]],
[4,1,'CD45',[],[],[]],
[1,1,'DNA_6',[],[],[],[],[],[]],
[2,1,'ELANE',[],[],[],[],[],[]],
[3,1,'CD57',[],[],[],[],[],[]],
[4,1,'CD45',[],[],[],[],[],[]],
],
)
"""
Expand Down Expand Up @@ -167,18 +167,18 @@ nextflow_workflow {
)
input[1] = Channel.of(
[
[1,1,'DNA_6',[],[],[]],
[2,1,'ELANE',[],[],[]],
[3,1,'CD57',[],[],[]],
[4,1,'CD45',[],[],[]],
[5,1,'DNA_7',[],[],[]],
[6,1,'ELANE7',[],[],[]],
[7,1,'CD577',[],[],[]],
[8,1,'CD457',[],[],[]],
[9,1,'DNA_8',[],[],[]],
[10,1,'ELANE8',[],[],[]],
[11,1,'CD578',[],[],[]],
[12,1,'CD458',[],[],[]],
[1,1,'DNA_6',[],[],[],[],[],[]],
[2,1,'ELANE',[],[],[],[],[],[]],
[3,1,'CD57',[],[],[],[],[],[]],
[4,1,'CD45',[],[],[],[],[],[]],
[5,1,'DNA_7',[],[],[],[],[],[]],
[6,1,'ELANE7',[],[],[],[],[],[]],
[7,1,'CD577',[],[],[],[],[],[]],
[8,1,'CD457',[],[],[],[],[],[]],
[9,1,'DNA_8',[],[],[],[],[],[]],
[10,1,'ELANE8',[],[],[],[],[],[]],
[11,1,'CD578',[],[],[],[],[],[]],
[12,1,'CD458',[],[],[],[],[],[]],
],
)
"""
Expand Down Expand Up @@ -235,14 +235,14 @@ nextflow_workflow {
)
input[1] = Channel.of(
[
[1,1,'DNA_6',[],[],[]],
[2,1,'ELANE',[],[],[]],
[3,1,'CD57',[],[],[]],
[4,1,'CD45',[],[],[]],
[5,1,'DNA_7',[],[],[]],
[6,1,'ELANE7',[],[],[]],
[7,1,'CD577',[],[],[]],
[8,1,'CD457',[],[],[]],
[1,1,'DNA_6',[],[],[],[],[],[]],
[2,1,'ELANE',[],[],[],[],[],[]],
[3,1,'CD57',[],[],[],[],[],[]],
[4,1,'CD45',[],[],[],[],[],[]],
[5,1,'DNA_7',[],[],[],[],[],[]],
[6,1,'ELANE7',[],[],[],[],[],[]],
[7,1,'CD577',[],[],[],[],[],[]],
[8,1,'CD457',[],[],[],[],[],[]],
],
)
"""
Expand Down Expand Up @@ -302,14 +302,14 @@ nextflow_workflow {
)
input[1] = Channel.of(
[
[1,1,'DNA_6',[],[],[]],
[2,1,'ELANE',[],[],[]],
[3,1,'CD57',[],[],[]],
[4,1,'CD45',[],[],[]],
[5,1,'DNA_7',[],[],[]],
[6,1,'ELANE7',[],[],[]],
[7,1,'CD577',[],[],[]],
[8,1,'CD457',[],[],[]],
[1,1,'DNA_6',[],[],[],[],[],[]],
[2,1,'ELANE',[],[],[],[],[],[]],
[3,1,'CD57',[],[],[],[],[],[]],
[4,1,'CD45',[],[],[],[],[],[]],
[5,1,'DNA_7',[],[],[],[],[],[]],
[6,1,'ELANE7',[],[],[],[],[],[]],
[7,1,'CD577',[],[],[],[],[],[]],
[8,1,'CD457',[],[],[],[],[],[]],
],
)
"""
Expand Down Expand Up @@ -368,18 +368,18 @@ nextflow_workflow {
)
input[1] = Channel.of(
[
[1,1,'DNA_6',[],[],[]],
[2,1,'ELANE',[],[],[]],
[3,1,'CD57',[],[],[]],
[4,1,'CD45',[],[],[]],
[5,1,'DNA_7',[],[],[]],
[6,1,'ELANE7',[],[],[]],
[7,1,'CD577',[],[],[]],
[8,1,'CD457',[],[],[]],
[9,1,'DNA_8',[],[],[]],
[10,1,'ELANE8',[],[],[]],
[11,1,'CD578',[],[],[]],
[12,1,'CD458',[],[],[]],
[1,1,'DNA_6',[],[],[],[],[],[]],
[2,1,'ELANE',[],[],[],[],[],[]],
[3,1,'CD57',[],[],[],[],[],[]],
[4,1,'CD45',[],[],[],[],[],[]],
[5,1,'DNA_7',[],[],[],[],[],[]],
[6,1,'ELANE7',[],[],[],[],[],[]],
[7,1,'CD577',[],[],[],[],[],[]],
[8,1,'CD457',[],[],[],[],[],[]],
[9,1,'DNA_8',[],[],[],[],[],[]],
[10,1,'ELANE8',[],[],[],[],[],[]],
[11,1,'CD578',[],[],[],[],[],[]],
[12,1,'CD458',[],[],[],[],[],[]],
],
)
"""
Expand Down Expand Up @@ -443,14 +443,14 @@ nextflow_workflow {
)
input[1] = Channel.of(
[
[1,1,'DNA_6',[],[],[]],
[2,1,'ELANE',[],[],[]],
[3,1,'CD57',[],[],[]],
[4,1,'CD45',[],[],[]],
[5,1,'DNA_7',[],[],[]],
[6,1,'ELANE7',[],[],[]],
[7,1,'CD577',[],[],[]],
[8,1,'CD457',[],[],[]],
[1,1,'DNA_6',[],[],[],[],[],[]],
[2,1,'ELANE',[],[],[],[],[],[]],
[3,1,'CD57',[],[],[],[],[],[]],
[4,1,'CD45',[],[],[],[],[],[]],
[5,1,'DNA_7',[],[],[],[],[],[]],
[6,1,'ELANE7',[],[],[],[],[],[]],
[7,1,'CD577',[],[],[],[],[],[]],
[8,1,'CD457',[],[],[],[],[],[]],
],
)
"""
Expand Down Expand Up @@ -498,10 +498,10 @@ nextflow_workflow {
)
input[1] = Channel.of(
[
[1,1,'DNA_6',[],[],[]],
[2,1,'ELANE',[],[],[]],
[3,1,'CD57',[],[],[]],
[4,1,'CD45',[],[],[]],
[1,1,'DNA_6',[],[],[],[],[],[]],
[2,1,'ELANE',[],[],[],[],[],[]],
[3,1,'CD57',[],[],[],[],[],[]],
[4,1,'CD45',[],[],[],[],[],[]],
],
)
"""
Expand Down Expand Up @@ -543,7 +543,7 @@ nextflow_workflow {
)
input[1] = Channel.of(
[
[1,1,'DNA_6',[],[],[]],
[1,1,'DNA_6',[],[],[],[],[],[]],
],
)
"""
Expand All @@ -564,5 +564,68 @@ nextflow_workflow {
}
},

test("cycle: no illumination correction, backsub") {

when {
params {
segmentation = "mesmer"
backsub = true
}
workflow {
"""
input[0] = Channel.of(
[
[id:"TEST1", cycle_number:1, channel_count:4],
"https://raw.githubusercontent.com/nf-core/test-datasets/modules/data/imaging/background_subtraction/cycif_tonsil_registered.ome.tif",
RobJY marked this conversation as resolved.
Show resolved Hide resolved
[],
[],
],
[
[id:"TEST1", cycle_number:2, channel_count:4],
"https://raw.githubusercontent.com/nf-core/test-datasets/modules/data/imaging/background_subtraction/cycif_tonsil_registered.ome.tif",
[],
[],
],
)
input[1] = Channel.of(
[
[1,1,'DNA 1',[],[],[],100,[],[]],
[2,1,'Na/K ATPase',[],[],[],100,[],[]],
[3,1,'CD3',[],[],[],100,[],[]],
[4,1,'CD45RO',[],[],[],100,[],[]],
[5,1,'DNA 2',[],[],[],100,[],[]],
RobJY marked this conversation as resolved.
Show resolved Hide resolved
[6,1,'Antigen Ki67',[],[],[],100,[],[]],
[7,1,'Pan-cytokeratin',[],[],[],100,[],[]],
[8,1,'Aortic smooth muscle actin',[],[],[],100,[],[]],
[9,2,'DNA 1 1',[],[],[],100,[],[]],
[10,2,'Na/K ATPase 2',[],[],[],100,[],[]],
[11,2,'CD3 2',[],[],[],100,'Na/K ATPase',[]],
[12,2,'CD45RO 2',[],[],[],100,[],[]],
[13,2,'DNA 2 2',[],[],[],100,[],'TRUE'],
[14,2,'Antigen Ki67 2',[],[],[],100,[],[]],
[15,2,'Pan-cytokeratin 2',[],[],[],100,'CD3',[]],
[16,2,'Aortic smooth muscle actin 2',[],[],[],100,'CD45RO 2',[]]
],
)
"""
}
}

then {
assertAll (
{
assert snapshot (
path("$outputDir/registration/ashlar/TEST1.ome.tif"),
path("$outputDir/segmentation/deepcell_mesmer/mask_TEST1.tif"),
CsvUtils.summarizeCsv("$outputDir/quantification/mcquant/mesmer/TEST1_mask_TEST1.csv"),
path("$outputDir/backsub/TEST1.backsub.ome.tif"),
).match()
},
{ assert workflow.success }
)
}

},

]
}
Loading