Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix for Azure certificates compatibility #4888

Draft
wants to merge 3 commits into
base: master
Choose a base branch
from

Conversation

adamrtalbot
Copy link
Collaborator

@adamrtalbot adamrtalbot commented Apr 5, 2024

When running a CentOS container on a Debian based virtual machine on Azure (default), an error was thrown while mounting the certificates. This was because we tried to add them by default. This PR changes the behaviour so this no longer happens. A user can still renable these options using the container.runOptions as per any other Docker runner.

Fixes #4828

Signed-off-by: Adam Talbot 12817534+adamrtalbot@users.noreply.github.com

Copy link

netlify bot commented Apr 5, 2024

Deploy Preview for nextflow-docs-staging canceled.

Name Link
🔨 Latest commit 72abcc5
🔍 Latest deploy log https://app.netlify.com/sites/nextflow-docs-staging/deploys/66570d3bdc20410008ff033e

@adamrtalbot
Copy link
Collaborator Author

adamrtalbot commented Apr 5, 2024

Steps to show it works:

main.nf:

process HELLO {
    container "docker.io/redhat/ubi8:8.9"

    output:
        stdout
    
    script:
    """
    echo "Hello, World!"
    """
}

workflow {
    HELLO()
}

Config:

process {
    executor = 'azurebatch'
}

azure {
    storage {
        accountName = "$AZURE_STORAGE_ACCOUNT_NAME"
        accountKey = "$AZURE_STORAGE_ACCOUNT_KEY"
    }
    batch {
        location = "$AZURE_LOCATION"
        accountName = "$AZURE_BATCH_ACCOUNT_NAME"
        accountKey = "$AZURE_BATCH_ACCOUNT_KEY"
        autoPoolMode = true
        deletePoolsOnCompletion = true
        deleteTasksOnCompletion = false
    }
}

Command:

nextflow-dev run . -c nextflow.config

Output:

> nextflow-dev run . -c nextflow.config -w az://scidev-useast/
N E X T F L O W  ~  version 24.02.0-edge
Launching `./main.nf` [silly_waddington] DSL2 - revision: 11eb8eab2e
[5c/75b8a6] Submitted process > HELLO

From Batch Explorer (note no container run options):
image

@adamrtalbot
Copy link
Collaborator Author

Shoot. Integration tests fail:

Launching `./test-complexpaths.nf` [thirsty_bohr] DSL2 - revision: 44828720b8
[47/0ffbb5] Submitted process > foo
ERROR ~ Error executing process > 'foo'

Caused by:
  Missing output file(s) `*.fa` expected by process `foo`

Command executed:

  echo A > hello.txt
  echo B > sample.zip 
  echo C > sample.html
  echo D > 01_A\(R1\).fastq
  echo E > 01_A\(R2\).fastq
  echo F > sample_\(1\ 2\).vcf
  echo 1 > f1.fa
  echo 2 > f2.fa
  echo 3 > f3.fa
  mkdir .alpha
  echo "Hello world!" > .alpha/hello.txt

Command exit status:
  0

Command output:
  (empty)

@pditommaso
Copy link
Member

Comment on lines 12 to 13
docker.runOptions = '-v /etc/ssl/certs:/etc/ssl/certs:ro -v /etc/pki:/etc/pki:ro'

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This doesn't look nice

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It also doesn't work.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed.

When running a CentOS container on a Debian based virtual machine on Azure (default), an error was thrown while mounting the certificates. This was because we tried to add them by default. This PR changes the behaviour so this no longer happens. A user can still renable these options using the `container.runOptions` as per any other Docker runner.

Signed-off-by: Adam Talbot <12817534+adamrtalbot@users.noreply.github.com>
@adamrtalbot adamrtalbot force-pushed the 4828_fix_azure_credentials_compatibility branch from aadb67c to 0c3ce63 Compare April 9, 2024 12:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants