file deletion fails: invalid cross-device link

[lvankampenhout]

Describe the bug
When working remotely on the Cheyenne compute facility I can make files but not delete them. The error message is

Delete Failed
Invalid response: 500 Internal Server Error

The preceding question is:

Are you sure you want to permanently delete: Circulation_Temperature.ipynb?

The console output shows:

    OSError: [Errno 18] Invalid cross-device link: (...)

See full output below.

This may be related to the fact that the space where I'm working is a symbolic link, deep down:

$ pwd
/glade/u/home/lvank/analysis/cesm2_cmip/notebooks_paper
$ readlink -f .
/gpfs/u/home/lvank/analysis/cesm2_cmip/notebooks_paper
$ ls -l /glade/u
lrwxrwxrwx 1 root root 7 Dec  4 20:45 /glade/u -> /gpfs/u

To Reproduce
On my workstation, I can delete files from a symbolically linked folder without problems.

Expected behavior
The file should be deleted.

Desktop (please complete the following information):
server:

• Linux CentOS 7 (link)
• JupyterLab 0.35.4 from conda.

client:

• Firefox 64 on OSX

Additional context
I wonder why the question reads: "permanently delete" but nevertheless tries to move the file to the Trash. This is not permanent deletion IMO.

Command Line Output

In the terminal windows where JupyterLab is running I see this:

[W 07:16:47.339 LabApp] delete /analysis/cesm2_cmip/notebooks_paper/Circulation_Temperature.ipynb
[E 07:16:47.340 LabApp] Uncaught exception DELETE /api/contents/analysis/cesm2_cmip/notebooks_paper/Circulation_Temperature.ipynb?1545142607236 (10.12.202.21)
    HTTPServerRequest(protocol='http', host='localhost:8889', method='DELETE', uri='/api/contents/analysis/cesm2_cmip/notebooks_paper/Circulation_Temperature.ipynb?1545142607236', version='HTTP/1.1', remote_ip='10.12.202.21')
    Traceback (most recent call last):
      File "/glade/u/home/lvank/miniconda3/lib/python3.7/site-packages/tornado/web.py", line 1592, in _execute
        result = yield result
      File "/glade/u/home/lvank/miniconda3/lib/python3.7/site-packages/tornado/gen.py", line 1133, in run
        value = future.result()
      File "/glade/u/home/lvank/miniconda3/lib/python3.7/site-packages/tornado/gen.py", line 326, in wrapper
        yielded = next(result)
      File "/glade/u/home/lvank/miniconda3/lib/python3.7/site-packages/notebook/services/contents/handlers.py", line 235, in delete
        yield gen.maybe_future(cm.delete(path))
      File "/glade/u/home/lvank/miniconda3/lib/python3.7/site-packages/notebook/services/contents/manager.py", line 279, in delete
        self.delete_file(path)
      File "/glade/u/home/lvank/miniconda3/lib/python3.7/site-packages/notebook/services/contents/filemanager.py", line 540, in delete_file
        send2trash(os_path)
      File "/glade/u/home/lvank/miniconda3/lib/python3.7/site-packages/send2trash/plat_other.py", line 192, in send2trash
        trash_move(path_b, dest_trash, topdir)
      File "/glade/u/home/lvank/miniconda3/lib/python3.7/site-packages/send2trash/plat_other.py", line 103, in trash_move
        os.rename(src, op.join(filespath, destname))
    OSError: [Errno 18] Invalid cross-device link: b'/gpfs/u/home/lvank/analysis/cesm2_cmip/notebooks_paper/Circulation_Temperature.ipynb' -> b'/glade/u/home/lvank/.local/share/Trash/files/Circulation_Temperature.ipynb'
[W 07:16:47.341 LabApp] Unhandled error
[E 07:16:47.341 LabApp] {
      "Host": "localhost:8889",
      "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.13; rv:64.0) Gecko/20100101 Firefox/64.0",
      "Accept": "*/*",
      "Accept-Language": "en-US,en;q=0.5",
      "Accept-Encoding": "gzip, deflate",
      "Referer": "http://localhost:8889/lab?",
      "Authorization": "token fed73392a59e58a72d35391074180b19027f62d14f8d1322",
      "Content-Type": "application/json",
      "Origin": "http://localhost:8889",
      "Dnt": "1",
      "Connection": "keep-alive",
      "Cookie": "username-localhost-8888=\"2|1:0|10:1544705686|23:username-localhost-8888|44:MDQ4NjNhMWMzN2ZhNDI2OWE3OTUyNDY0MjBhOTEwOWE=|3fea2a7b0ea9ea96686cdf32f90f97913977a4e7f789bd7862cc7bb112798319\"; username-localhost-8889=\"2|1:0|10:1545142607|23:username-localhost-8889|44:ZWI2NmRjYjJkZTRiNGQ0ZGFiMDM3MGZkMjBmNjllZDM=|83ae6b39dfd59d56b3393291e1e3a149019257cd9290023b8db55babdabeafb1\"; username-localhost-8890=\"2|1:0|10:1544623875|23:username-localhost-8890|44:YmUxMDBhYzIzMGYwNGZhNDk1NzM1NDI1ZjBiZWJmZmY=|85f390c16647fd5e89029fa1ddba60a64f333ed1a04cd89e36f84773e0a70b10\"; _xsrf=2|9004da89|b63cb8eb2aee10408acab12f0a79f02b|1545048934",
      "Pragma": "no-cache",
      "Cache-Control": "no-cache"
    }

[vidartf]

A general google of the exception gives this SO answer: https://stackoverflow.com/questions/42392600/oserror-errno-18-invalid-cross-device-link

os.rename only works if source and destination are on the same file system. You should use shutil.move instead.

So, maybe the solution is for send2trash to also use shutil.move?

[vidartf]

Xref arsenetar/send2trash#26

[michaelkarlcoleman]

@vidartf Using shutil.move might be a good first step. (Though I thought I saw that in older versions it simply does os.remove.)

More generally, though, for cross-device and permissions errors while trying to put a file in the "Trash" directory, it seems like it would be better to just delete (perhaps with a prompt).

Note that this is related to #3793 and #4318 and jupyter/notebook#3304 .

[jasongrout]

It sounds like there is nothing for us to do in JupyterLab, but it is an error in the server-side libraries? Is that right?

[michaelkarlcoleman]

@jasongrout I think it depends on the resolution. If it's decided that there should be a dialog box saying something like "This can't be moved to the Trash, but instead will be permanently deleted--OK?", perhaps that has to go in the the Lab code? (I'm not sure how it all fits together.)

[alfonsomhc]

It is unknown if os.rename will ever be fixed (maybe they consider this a feature and not a bug!). Therefore my suggestion is that trash_move is implemented with shutil.copy() followed by os.remove().

[jasongrout]

That is a fix for the server then. Do you want to open an issue in the notebook server repo?

[alfonsomhc]

@jasongrout , thanks for your answer. I assumed you meant the repo https://github.com/jupyterlab/jupyterlab_server/ and has created an issue there:
jupyterlab/jupyterlab_server#80

[michaelkarlcoleman]

@alfonsomhc The behavior of os.rename is presumably intended to closely match the rename(2) system call. I'd call it a feature. :-)

[michaelkarlcoleman]

In tracking down a user problem just now, realized that it's this same bug. :-(

[arsenetar]

Since this was linked in recent issues/PRs over at https://github.com/arsenetar/send2trash letting you guys know there is a pre-release package published at https://pypi.org/project/Send2Trash/1.8.1b0/ which should address the issues reported here based on the information I have. If there are still issues, I will need more information on the environments (mount points and paths involved).

I will move this to a release if I can get some confirmation on if this resolves the issues over here as jupyter seems to be the main source of these reports.

[michaelkarlcoleman]

@arsenetar It might be a bit before I can test this, but eyeballing the commit (arsenetar/send2trash@5e4517a), it appears reasonable.

I'm wondering, though, why not just invoke shutil.move in all cases? It appears that it already uses os.rename or not internally, as appropriate.

[arsenetar]

@michaelkarlcoleman The reason as to why shutil.move is not used in all cases is due to it potentially not being appropriate in all cases. I'll expand on this with the flow of how trashing determines where to trash (in order):

1. User's home trash if the file is on the same device
2. The top directory trash of the device (.Trash) where the file to trash exists if the top directory trash is usable
3. The user specific top directory trash of the device (.Trash-uid) where the file to trash exists
4. The user's home trash

Until the recent change (4) was not part of the implementation by send2trash although part of the freedesktop.org Trash spec. Some other programs implementing trashing do not appear to support (4) either in some cases. Additionally, the behavior of other programs that send to trash on similar platforms appear to follow the restrictions of os.rename more than not.

With this flow if we use shutil.move(src, trash) directly it will use os.rename then fallback to the copyfunction copy2 automatically. In some cases (due to the behavior of os.path.ismount) going directly from (1) to (4) would be appropriate and this behavior would be fine. In other cases this could cause an issue with a file trying to be handled by (2) or (3) to move across devices resulting in a file being trashed to the top directory trash of a different device than where it came from. Cross device trashing into top directory trash locations is problematic.

The change made is attempting to follow the spec without entering the gray area of what to do where os.rename and os.path.ismount disagree. I can optimize the case of (1) to (4) by just going directly with shutil.move, however the code still needs to manage the flow for (2) or (3) to (4).

[michaelkarlcoleman]

@arsenetar I see. Thanks for the detailed explanation.

FWIW, our use case is a multi-user Linux system (OOD on HPC). At least in our shop, the only plausible locations for a .Trash directory would be in $HOME or $PWD. On a single-user system, other things could work, as you say.

One troublesome case would be if the trashed file is huge, especially if a copy attempt is made (though even renames might lead to quota issues). That might be more of a problem for users of "send2trash" to deal with, though.

[arsenetar]

@michaelkarlcoleman I would be okay with adding a flag to the send2trash call for this implementation to only allow trashing to the user's home trash if that would be helpful.

Of course depending on the usage environment this may cause copies to happen more "aggressively" into the user's home trash. I am also considering a flag to prevent copies (i.e. the current behavior without this fix) as well as potentially throwing a better error up to allow calling programs to handle this (like asking the user if they would be okay with hard delete since it could not go to trash).