Separate reference docs from manual pages

[bentsherman]

This PR reorganizes the docs in a number of ways:

• reorganize manual pages into "Running pipelines" and "Developing pipelines" to be more aligned with the user journey
• separate reference content (config scopes, process directives, etc) from the main docs
• improve section on managing dependencies

The original exploration was around refactoring the docs into a user guide vs reference manual. After a few iterations, the main focus now is the first two items, separating the reference docs and reorganizing the sections to be more user friendly.

[netlify]

✅ Deploy Preview for nextflow-docs-staging ready!

Name	Link
🔨 Latest commit	870c4b2
🔍 Latest deploy log	https://app.netlify.com/sites/nextflow-docs-staging/deploys/6631139fda053c0008d54582
😎 Deploy Preview	https://deploy-preview-4766--nextflow-docs-staging.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site configuration.

[bentsherman]

After discussing with Paolo and the team, we are thinking more to organize the docs like the Docker Docs.

The key implication for this PR is that there will not be one general "user guide" and "reference". Instead, there will be a "Guides" section which contains many guides targeted to specific use cases, such as a quickstart for each cloud provider, HPC schedulers, etc.

The existing "reference" docs are more accurately described as a "manual" which also contains reference material. Instead of trying to factor out the user guide content, I will keep the docs as they are but just try to better separate the reference material from the "manual" pages.

I will refactor this PR accordingly and continue to restructure the existing docs, but also carve out a space for the team to start writing the guides. These efforts can be pursued independently, but since they both depend on a new deployment framework, no harm in keeping them together for now.

[bentsherman]

Some notes on the Nextflow training and nf-core docs after a brief review:

• nf-core docs seems to be mostly about nf-core conventions, really only the installation page has some duplication that could be trimmed

• the advanced and "applied" training guides can be adapted into guides under our proposed structure, pretty straightforward

• the basic training duplicates a lot of nextflow docs, it should be condensed significantly and become another guide (see notes below)

Because I have pulled out the reference material (config scopes, channel factories, process directives, etc), the manual pages for config, channels, process, etc have a lot more breathing room to give some illustrative examples. As a result, I think the basic training can be condensed significantly by just linking to the docs more instead of duplicating it. Instead it can focus on walking through the writing and testing of a specific pipeline example.

[bentsherman]

After discussing with Paolo, we would like to only keep the manual + reference docs in the Nextflow repo. Additional content like targeted guides and tutorials should be maintained somewhere else and can be combined with the reference docs at build time.

To that end I have trimmed this PR to focus on the internal improvements to the existing docs. These changes can be merged independently of other efforts like writing guides, deployment framework, website overhaul, etc.

[bentsherman]

This PR is ready for review. Going to do one more pass to iron our any rough edges.

This PR has evolved in scope since it was created. To clarify, the purpose of this PR now is to reorganize some pages and sections to be more user friendly, and to make it easier to pull out higher-level content for the docs team in future iterations.

[bentsherman]

Found some rough edges with the process docs, should have it resolved today

[bentsherman]

@ewels @pditommaso shall we move this forward? Everything looks good to me, but as long as you agree with the basic structure, we can refine things in future (smaller) PRs. This one is getting difficult to maintain

[pditommaso]

I'm not sure we should do this; we are continuing refactoring this doc, making links unstable. I assume new docs should go into docs.seqera.io, after that we can progressively reorganize this as needed.

[bentsherman]

This restructure is a prerequisite for moving docs to seqera docs because it makes it easier to pull out the user guide content

[pditommaso]

Let's solve the conflict and I'll give this another try

[bentsherman]

Ready 👍

[pditommaso]

Has something changed in the deps? I'm getting this while building the docs

Configuration error:
There is a programmable error in your configuration file:

Traceback (most recent call last):
  File "/opt/conda/lib/python3.7/site-packages/sphinx/config.py", line 350, in eval_config_file
    exec(code, namespace)
  File "/tmp/conf.py", line 75, in <module>
    process = subprocess.Popen(shlex.split("git tag --points-at HEAD"), stdout=subprocess.PIPE)
  File "/opt/conda/lib/python3.7/subprocess.py", line 800, in __init__
    restore_signals, start_new_session)
  File "/opt/conda/lib/python3.7/subprocess.py", line 1551, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'git': 'git'

[bentsherman]

Looks like the git command is missing wherever you are running that

You know you can just use the deploy preview: https://deploy-preview-4766--nextflow-docs-staging.netlify.app/

[pditommaso]

it's running in the container make via

nextflow/docs/Dockerfile

Line 1 in e2f74b4

FROM mambaorg/micromamba:1.3.1

[bentsherman]

Think you need to add git to the container, it is being used by the docs build now:

nextflow/docs/conf.py

Lines 70 to 72 in ea8246f

	# AUTO-GENERATED BY GIT
	# Check if we are checked out at a specific version or not
	process = subprocess.Popen(shlex.split("git tag --points-at HEAD"), stdout=subprocess.PIPE)