Replace publishDir with workflow output definition

[bentsherman]

Prototype for nextflow-io/nextflow#4784

[bentsherman]

I fixed the bug to make the index file work with single files, try again?

[pditommaso]

Now, i'm getting a different error. Please, add tests

java.nio.file.NoSuchFileException: /Users/pditommaso/Projects/rnaseq-nf/results/fastqc/index.csv
	at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)
	at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)
	at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
	at java.base/sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:218)
	at java.base/java.nio.file.spi.FileSystemProvider.newOutputStream(FileSystemProvider.java:484)
	at java.base/java.nio.file.Files.newOutputStream(Files.java:228)
	at org.apache.groovy.nio.extensions.NioExtensions.append(NioExtensions.java:705)
	at org.apache.groovy.nio.extensions.NioExtensions.append(NioExtensions.java:586)
	at org.apache.groovy.nio.extensions.NioExtensions.leftShift(NioExtensions.java:500)
	at nextflow.util.CsvWriter.apply(CsvWriter.groovy:66)
	at nextflow.extension.PublishIndexOp.onComplete(PublishIndexOp.groovy:83)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
	at java.base/java.lang.reflect.Method.invoke(Method.java:578)
	at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:343)
	at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:328)
	at groovy.lang.MetaClassImpl.doInvokeMethod(MetaClassImpl.java:1333)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1088)
	at groovy.lang.MetaClassImpl.invokeMethodClosure(MetaClassImpl.java:1017)
	at groovy.lang.MetaClassImpl.doInvokeMethod(MetaClassImpl.java:1207)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1088)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1007)
	at groovy.lang.Closure.call(Closure.java:433)
	at groovy.lang.Closure.call(Closure.java:422)
	at org.codehaus.groovy.vmplugin.v8.IndyInterface.fromCache(IndyInterface.java:321)
	at nextflow.extension.DataflowHelper$3.afterStop(DataflowHelper.groovy:257)
	at groovyx.gpars.dataflow.operator.DataflowProcessor.fireAfterStop(DataflowProcessor.java:324)
	at groovyx.gpars.dataflow.operator.DataflowProcessorActor.afterStop(DataflowProcessorActor.java:59)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
	at java.base/java.lang.reflect.Method.invoke(Method.java:578)
	at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:343)
	at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:328)
	at groovy.lang.MetaClassImpl.doInvokeMethod(MetaClassImpl.java:1333)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1088)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1007)
	at org.codehaus.groovy.runtime.InvokerHelper.invokePojoMethod(InvokerHelper.java:633)
	at org.codehaus.groovy.runtime.InvokerHelper.invokeMethod(InvokerHelper.java:624)
	at groovyx.gpars.actor.Actor.callDynamic(Actor.java:369)
	at groovyx.gpars.actor.Actor.handleTermination(Actor.java:320)
	at groovyx.gpars.actor.AbstractLoopingActor.terminate(AbstractLoopingActor.java:204)
	at groovyx.gpars.dataflow.operator.DataflowProcessor.terminate(DataflowProcessor.java:147)
	at groovyx.gpars.dataflow.operator.DataflowProcessorActor.checkPoison(DataflowProcessorActor.java:115)
	at groovyx.gpars.dataflow.operator.DataflowOperatorActor.onMessage(DataflowOperatorActor.java:83)
	at groovyx.gpars.actor.impl.SDAClosure$1.call(SDAClosure.java:43)
	at groovyx.gpars.actor.AbstractLoopingActor.runEnhancedWithoutRepliesOnMessages(AbstractLoopingActor.java:293)
	at groovyx.gpars.actor.AbstractLoopingActor.access$400(AbstractLoopingActor.java:30)
	at groovyx.gpars.actor.AbstractLoopingActor$1.handleMessage(AbstractLoopingActor.java:93)
	at groovyx.gpars.util.AsyncMessagingCore.run(AsyncMessagingCore.java:132)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1589)

[bentsherman]

Can you push your changes so I can see how you are configuring the index file?

[pditommaso]

Just using this in the rnaseq-nf workflow-output-dsl branch

output {
  directory params.outdir
  mode 'copy'

  fastqc {
    index {
      path 'index.csv'
    }
  }
}

[bentsherman]

That is strange. There is an e2e test called output-dsl.nf which does the same thing and has never failed with this error. But trying your snippet with rnaseq-nf, it only fails if the output directory is deleted beforehand. Must be a race condition with the directory creation.

[bentsherman]

Working on my end now

[bentsherman]

I refactored the outputs to be a channel of samples, where each sample contains all associated files that were created throughout the pipeline. In this case, each sample has salmon results and fastqc logs.

The output directory looks like this:

$ find results/ | sort
results/
results/ggal_gut
results/ggal_gut/fastqc
results/ggal_gut/quant
results/index.json
results/multiqc
results/multiqc/multiqc_report.html

And the index file looks like this:

[
    {
        "id": "ggal_gut",
        "quant": "/home/bent/projects/nextflow-io_rnaseq-nf/results/quant",
        "fastqc": "/home/bent/projects/nextflow-io_rnaseq-nf/results/fastqc"
    }
]

It looks like there are some issues with the publish destination paths, the index file does not match the actual paths.

[pditommaso]

let's rebase this changes over workflow-output-2nd branch, and keep workflow-output-dsl for review 1