Workflow output definition

[bentsherman]

Close #4670

This PR adds a DSL for defining workflow outputs. It is meant to completely replace the publishDir directive, and centralize the definition of published outputs into one location.

This first iteration is essentially a builder DSL for a directory structure, which also collects the essential elements of the existing publish mechanism: process selectors to select process outputs, and publish options like pattern and enabled (other options like mode can also be supported this way).

Under the hood, there is now a WorkflowPublisher which contains all of the publish "selectors" and uses PublishDir just like the directive. So the publishing itself is also handled in one place rather than during the task finalization.

See nf-core/fetchngs#275 for a prototype with nf-core/fetchngs

[netlify]

✅ Deploy Preview for nextflow-docs-staging canceled.

Name	Link
🔨 Latest commit	0db73c8
🔍 Latest deploy log	https://app.netlify.com/sites/nextflow-docs-staging/deploys/664665b20da96700083892ac

[marcodelapierre]

Let me try and clarify, in the case of rnaseq-nf we need to give an emit name to a module channel to be able to publish it, e.g. see this diff:

      output:
-    path "fastqc_${sample_id}_logs" 
+    path "fastqc_${sample_id}_logs", emit: logs
+
+    publish:
+    logs >> '.'

So my thinking point is whether we should have a syntax for the publish section that does not require giving the emit name logs to the channel to be published. As I was saying above, my personal answer is that we don't need it, but still wanted to leave this for your consideration.

[bentsherman]

With static types we are moving towards a syntax where every process output will have a name by definition (<type> <name> = <value>), so in time this will not be an issue

[marcodelapierre]

thanks for the reminder Ben

[marcodelapierre]

I am happy with the testing of the feature with the Fetchngs POC (both linux vm and Slurm cluster); there I disabled nf-validation.

For the rnaseq POC, how do I get around the following error?

ERROR ~ Install is not supported on dev mode - Missing plugin nf-validation 1.1.3

(nf-validation here seems needed to use fromSamplesheet)

[bentsherman]

You can install the local build with make install. It will replace the installation of your current Nextflow version and allow you to run pipelines with third-party plugins. It won't use the local build of core plugins like nf-amazon, but that's okay because this PR doesn't modify any core plugins.

[marcodelapierre]

Testing Rnaseq was also successful. As Ben expected, output directory layout is not identical, but output files are there.

[pditommaso]

While testing the output index I've got this

ERROR ~ Index file record must be a list or map: /Users/pditommaso/Projects/rnaseq-nf/work/10/f200de889f31fcba450bab2d2964a3/fastqc_ggal_gut_logs [UnixPath]

 -- Check '.nextflow.log' file for details

What about instead creating a simple index as "file name", "file path" ?

[pditommaso]

Made a few tests and works like a charm. I was expecting a much bigger impact in the codebase, great work Ben.

I've commented a few things to address above

[pditommaso]

Better, but it's not an index file

"/Users/pditommaso/Projects/rnaseq-nf/results/fastqc/fastqc_ggal_gut_logs"

I was expecting as

"fastqc_ggal_gut_logs", "/Users/pditommaso/Projects/rnaseq-nf/results/fastqc/fastqc_ggal_gut_logs"

[bentsherman]

That can be done with the index mapper:

index {
  mapper { file -> [file.name, file] }
}

[pditommaso]

I know, but it would be nice to have it as default behaviour

[pditommaso]

and it's ready!