-
Notifications
You must be signed in to change notification settings - Fork 595
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
symlinks for staged files in directories are not removed in .command.run #4971
Comments
This is likely because you have many staged files, see here nextflow/modules/nextflow/src/main/groovy/nextflow/executor/SimpleFileCopyStrategy.groovy Lines 122 to 133 in aa9e127
|
Thanks @pditommaso for pointing that out! What is the problem with including possibly a few 1000 more lines in the runner script? |
It's explained in the comment. To contain the script file size. You can delete all symlink using a Bash oneliner like |
Why does the file size need to be contained to <100 lines of removing symlinks? Extending to 1000's of lines will not add much size to the file.
Why not just use |
lol. need to think it there could be other links. @bentsherman opinion? |
I thought all symlinks were (re)created by the runner script, but maybe I'm mistaken? |
Deleting all links should be fine, I can't think of any other links that are created. But Nick also suggested using |
Bug report
For a failed job in my Nextflow pipeline, I'm manually running bash .command.run and I'm getting
ln: failed to create symbolic link 'DIRECTORY_NAME/FILE_NAME.txt': File exists
.The nxf_stage() function includes:
The symlinks are not removed via
rm -f
prior to recreating them in thenxf_stage()
function, andln -s
is used instead ofln -sf
. This results in the error when manually re-running.command.run
. This make troubleshooting failed jobs harder, since I manually have to delete existing symlinks or comment-out all of theln -s
commands innxf_stage()
.This issue does not occur for files not in staged directories, just for
mkdir -p new_directory && ln -s new_directory/new_file.txt
.Expected behavior and actual behavior
See above
Steps to reproduce the problem
This should occur for any pipeline that creates staged files in directories:
mkdir -p new_directory && ln -s new_directory/new_file.txt
Program output
See above
Environment
Additional context
See this slack thread
The text was updated successfully, but these errors were encountered: