get_open_files might hang forever

[giampaolo]

From amo...@google.com on November 24, 2012 15:39:51

What steps will reproduce the problem?  


1. Create a process_iter()
2. Call get_open_files on each of the processes 



What is the expected output?  
The list of open files for each process. 



What do you see instead?  
The program hangs forever.

This has already been encountered, there are some exception in the code for 
exactly this problem. I have found on more access mask where the problem arises 
- please add it to the list of exceptions. Patch is attached.

Attachment: patch

Original issue: http://code.google.com/p/psutil/issues/detail?id=340

[giampaolo]

From g.rodola on November 24, 2012 11:48:21

Could you provide some info as to where 0x00120089 comes from?

[giampaolo]

From amo...@google.com on November 24, 2012 11:52:36

sadly, I can't. We ran it, it hung on a couple machines, this value was the
access mask.

Do you have any idea where the other values come from? Maybe we can explain
this one then. I looked at the others but the flags they represent are
completely independent, I can't see a pattern.

Sorry, that I don't have more info.

[giampaolo]

From g.rodola on November 24, 2012 12:04:55

What I'd like to know is how exactly you came up with that value (you HAVE to know =)).
Did you debug the problem by printing handle.GrantedAccess and then you noticed 
that when it was ==  0x00120089 the function would hang?

If that's the case then I'm ok with applying the patch.
It's just that I don't feel comfortable in blindly accepting patches for an 
area of the code which is already pretty messy, that's all.

[giampaolo]

From amo...@google.com on November 24, 2012 12:09:07

yeah, exactly. I just printed the value and it hung after getting
0x00120089.

I don't really like the code there either to be honest and I have disabled
this completely locally. There might be more values that hang that we don't
know about. It would be really good to come up with a pattern to determine
if the calls are dangerous...

[giampaolo]

From g.rodola on November 25, 2012 14:38:39

Committed in r1548 .
I agree it would be great if that part was refactored.
Unfortunately it was borrowed from somewhere I can't even remember.
Anyway, closing this out as fixed for now and thanks for the patch.

Status: FixedInSVN
Labels: OpSys-Windows Milestone-0.6.2

[giampaolo]

From wj32...@gmail.com on November 25, 2012 14:42:42

This limits the usefulness of the get_open_files function, since it won't show 
a lot of open files/documents. The only way to completely fix this is to use a 
kernel-mode driver. There might also be a new API in Windows 8 that allows you 
to query the file name properly, but I can't remember what it is...

[giampaolo]

From g.rodola on November 25, 2012 14:56:01

Status: ReOpened

[giampaolo]

From g.rodola on November 26, 2012 03:22:54

wj32.64, thanks for signaling.
I will revert r1548 for now.
If the prospect is to skip files which might be valid then I'd rather risk it 
and signal in the doc that get_open_files() on Windows is not fully reliable.

[giampaolo]

From amo...@google.com on November 26, 2012 12:53:11

So I don't have a strong opinion on this issue since I have just stopped
using get_open_files altogether for now. I just want you to know that I ran
a simple get_open_files for all processes on 50 boxes and it locked up
exactly 20 of them so it's not a rare occurrence.

[giampaolo]

From g.rodola on November 26, 2012 13:54:30

That's weird as despite I have your same configuration (Win7 64bit) I've never 
bumped into this issue before.
Just a thought which occurred to me just now: might it be the case that 
0x00120089 refers to a file living on a network filesystem (e.g. SMB)? In that 
case it might be ok to just skip it.

I still couldn't find the original blog entry where I borrowed that code, nor I 
was able to figure out the the meaning of those codes (0x00120089, etc.).
This is one of those cases where I regret about copying and pasting code 
without understanding it but I had no chance. For things such as this one 
Windows requires to go so close to the kernel that the resulting code is just a 
mess which makes no sense.

[giampaolo]

From jlo...@gmail.com on November 28, 2012 06:17:32

Given what wj32.64 noted above, I would recommend pulling the change back in to 
eliminate the known hang at this point. 

There's not a whole lot we can do to avoid skipping some open files unless 
we're willing to create a kernel-mode driver component for psutil. That's 
something I'm not really eager to do; it would provide more power and 
functionality but it would also be a bunch of added work and complexity to the 
project that I'm not sure is worth the trade-off. 

For now I'd say we skip those we know cause a hang, and note in the 
documentation that due to access rights some files may not show up on Windows.

[giampaolo]

From wj32...@gmail.com on December 12, 2012 19:56:13

You can use Process Hacker's driver which is signed (for 64-bit). But users of 
your library might not like a Python module loading drivers without permission.

[giampaolo]

From g.rodola on December 13, 2012 06:10:05

I reverted the original r1548 commit because it turns out that that change 
basically skips *all* regular (legitimate) files, making get_open_files() 
basically useless on Windows.

@amoser@google.com: I'd like to know a bit more about the circumstances causing the hang.
For which processes do you have this problem?
The ones not owned by your user?
If that's the case we might check permissions up front and raise AccessDenied 
preemptively.

[giampaolo]

From g.rodola on February 24, 2013 11:49:00

@amoser: 
Up. 
Can you tell whether the problem occurs only for processes owned by another user?

Labels: -Milestone-0.6.2

[giampaolo]

From amo...@google.com on February 26, 2013 07:54:45

Sorry for the late response.

I have done some more testing and I have narrowed down the problem a bit. I can 
reproduce by running:

    > PsExec.exe -i -s C:\Windows\system32\cmd.exe

to get system and then

    for p in psutil.process_iter():
       with open("o", "a") as f:
             f.write(p.name)
       try:
          print p.get_open_files()
       except psutil.AccessDenied:
          pass

This will hang on the PsExec.exe process. All other processes including those 
owned by other users and my own cmd.exe work just fine. I hope this helps 
debugging?!

[giampaolo]

From g.rodola on February 27, 2013 06:56:15

Thanks for your suggestion: I'm now able to reproduce the issue.
I researched a bit and as wj32.64 suggested it seems there's no way to avoid 
the hanging on NtQueryObject() aside from using a driver.

Another solution seems to be using GetFileInformationByHandleEx(): 
http://0memory.blogspot.it/2008/08/getfinalpathnamebyhandle-api-hungs.html 
...which is supported on Windows >= Vista only.
If it works we might think about dropping support for get_open_files() on 
previous Windows versions.
I will research into that.

[giampaolo]

From g.rodola on March 02, 2013 04:13:54

Updated csets after the SVN -> Mercurial migration: r1548 == revision 
9ee5c1e61dea

[giampaolo]

From gtmacdon...@gmail.com on July 08, 2013 15:37:05

I think I've found an easy fix for this. You can use the windows api call 
GetFileType to figure out what the file handle type is, one of which is 
FILE_TYPE_PIPE. I've fixed it for myself locally by adding an if statement in 
process_handles.c:185 to continue if it's not the disk file type. Seems to fix 
the problem I was having earlier with it blocking on a cygwin terminal. Hope 
this helps. Thanks for the nice process library!

    if (handle.ProcessId != pid)
        continue;

    if (GetFileType(handle.Handle) != 0x0001 ) //FILE_TYPE_DISK
        continue;

[giampaolo]

From g.rodola on July 09, 2013 04:37:57

With that change I get this failure:

    ======================================================================
    FAIL: test_get_open_files (__main__.TestProcess)
    ----------------------------------------------------------------------
    Traceback (most recent call last):
      File "test\test_psutil.py", line 1380, in test_get_open_files
        self.assertIn(TESTFN, filenames)
    AssertionError: 'C:\\Users\\user\\cygwin\\home\\user\\psutil\\$testfile' 
not found in [u'C:\\Windows\\SysWOW64\\en-US\\KernelBase.dll.mui']

[giampaolo]

From gtmacdon...@gmail.com on July 09, 2013 10:31:43

GetFileType should just skip over those handles that aren't disk files. Perhaps 
the tests need to be updated?

[giampaolo]

From g.rodola on July 09, 2013 10:38:37

No, (unfortunately) the test is correct.

[giampaolo]

From gtmacdon...@gmail.com on July 09, 2013 11:09:08

test_get_open_files passes for me. I had failures for three others though. I'm 
running Python 2.7, 64 bit, on Windows 7, psutil-0.7.1.

    ======================================================================
    FAIL: test_disk_partitions (__main__.TestCase)
    ----------------------------------------------------------------------
    Traceback (most recent call last):
      File "test_psutil.py", line 566, in test_disk_partitions
        assert os.path.exists(disk.device), disk
    AssertionError: partition(device='Q:\\', mountpoint='Q:\\', fstype='', 
opts='fixed')

    ======================================================================
    FAIL: test_get_set_ionice (__main__.TestCase)
    ----------------------------------------------------------------------
    Traceback (most recent call last):
      File "test_psutil.py", line 206, in inner
        return fun(self, *args, **kwargs)
      File "test_psutil.py", line 918, in test_get_set_ionice
        self.assertEqual(p.get_ionice(), value)
    AssertionError: 2 != 0

    ======================================================================
    FAIL: test_sys_cpu_times_percent (__main__.TestCase)
    ----------------------------------------------------------------------
    Traceback (most recent call last):
      File "test_psutil.py", line 527, in test_sys_cpu_times_percent
        self._test_cpu_percent(percent)
      File "test_psutil.py", line 506, in _test_cpu_percent
        assert percent >= 0.0, percent
    AssertionError: -800.0

    ----------------------------------------------------------------------

[giampaolo]

From g.rodola on July 09, 2013 15:55:42

Not sure what to say. :(

[giampaolo]

From gtmacdon...@gmail.com on July 10, 2013 03:10:31

:) Computers are a pain in the ass. I'm happy it works on my end though. Cheers.

[giampaolo]

From jt...@vahna.net on February 28, 2014 10:17:20

I am able to replicate this bug reliably.  Hopefully here's some more 
information.

OS: Windows 2012 (x64)
Python: Python 2.7.6 (default, Nov 10 2013, 19:24:24) [MSC v.1500 64 bit 
(AMD64)] on win32
psutil: 1.2.1 (from PyPi)

Test case:
User: NT Authority\SYSTEM (must be SYSTEM to replicate)
Process: vds.exe (PID: 964)
Code: psutil.Process(964).get_open_files()

As an elevated Administrator:

    >>> psutil.Process(964).get_open_files()
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
      File "C:\Python27\lib\site-packages\psutil\__init__.py", line 808, in get_open
    _files
        return self._platform_impl.get_open_files()
      File "C:\Python27\lib\site-packages\psutil\_psmswindows.py", line 199, in wrap
    per
        raise AccessDenied(self.pid, self._process_name)
    psutil._error.AccessDenied: (pid=964)

As SYSTEM: Hangs

[giampaolo]

From g.rodola on February 28, 2014 10:18:48

I'm starting to think whether it makes sense to get rid of get_open_files() on Windows.

[giampaolo]

From jt...@vahna.net on February 28, 2014 15:05:43

Just came across this:

"Call CreateFileMapping on the duplicated handle (assuming it has GENERIC_READ 
or better access).  It doesn't seem to block on those troublesome named pipes.  
If the call succeeds, you have a filesystem-backed file and you can immediately 
CloseHandle on the resulting map handle and successfully query for the file 
handle's name.  If the function fails, it's not a filesystem-backed handle (or 
you have insufficient privileges anyway), and you don't need to fetch the name 
at all." ( 
http://forum.sysinternals.com/discussion-howto-enumerate-handles_topic19403_page10.html )

[giampaolo]

From jt...@vahna.net on March 03, 2014 11:27:31

I'm able to fix the issue using the CreateFileMapping workaround.  I've 
attached my diff against the 1.2.1 release.

Attachment: get_open_files.patch

[giampaolo]

From g.rodola on March 03, 2014 11:31:54

Wow! These are very good news.
Do tests pass?
I will try your patch later during this week.
If this works it will be a great fix for the upcoming 2.0 version.

Labels: -Priority-Medium Priority-High

[giampaolo]

From jt...@vahna.net on March 03, 2014 14:21:27

I've modified the patch to use the original GrantedAccess check as well as 
CreateFileMapping with checks to GetLastError.

The tests pass on Windows XP running as Administrator and SYSTEM.

It also passes my test case of vds.exe on Windows 2012 as SYSTEM.

Attachment: get_open_files2.patch

[giampaolo]

@daveajones can you please test the pull request in #519?

[giampaolo]

I merged the PR #591 into master. Please someone confirm whether the hanging issue is fixed or not. If you have troubles compiling psutil on Windows just ask and I'll send you a pre-compiled binary.

[daveajones]

I'll need a binary to test. I can test today I think. Just can't compile.

[giampaolo]

OK, what python version do you need for? 32 or 64 bit?

[daveajones]

64-bit python 3.3.5

[giampaolo]

Here. Rename it to .exe (github won't let me upload exes).

[daveajones]

Test successful. I ran my script before installing your new binary and it hung on the first process enumeration pass. I then installed the patched code you sent and now the script runs fine with no hanging. :-)

[giampaolo]

Cool! This was a very long standing issue and I hope it will remain closed for good this time. =)
Kudos to @mrjefftang who provided the patch.

[jimbo1969]

Note that as of 15-Oct-2014, it still hangs on Windows XP (and others, I assume) when you run the following python code (hangs at open_files() on the third process it tries in my case):

import psutil
for proc in psutil.process_iter():
    try:
        print(proc.pid.numerator, proc.open_files())
    except psutil.NoSuchProcess as err:
        print("process gone",err)
    except psutil.AccessDenied as e:
        print("cannot access", e.pid, e.name)

First I tried it after using pip to install psutil, and I assumed pip had installed a version that did not yet have the fix integrated. So I uninstalled with pip and confirmed it was cleaned up in my site-packages. I then cloned the source from github, confirmed the changes in process_handles.c, and compiled manually to make sure I had this fix. Tried again and the hang-problem persists in Windows (it works fine on *nix).

[mrjefftang]

I've never been able to reproduce this on Windows XP. What is the process that it hangs on?

[giampaolo]

If this bug is confirmed I think I will give up and deprecate and/or remove open_files() functionality on Windows for good, also because it appears this is not possible to do reliably unless a Windows system driver or service is used.

[mrjefftang]

I think perhaps we should just deprecate it for Windows XP and rewrite the code to use the appropriate non-hanging API calls supported in the newer versions of Windows.

Or maintain this hacked up fix and update it as necessary for WinXP.

There are only 2 ways to do this on Windows XP without running into the issue:
1 - Install a driver to query for open file handles
2 - Create a new thread to run the API calls but there is no way to clean up the hanging thread

[giampaolo]

I kind of lost track of the previous progress: what are the appropriate non-hanging APIs to use on Windows > XP exactly? If there are such APIs and it is proven that they work (the hanging problem is fixed) it would be perfectly fine to drop this functionality for Windows XP. We're already doing something like that for Process.ionice_get/set which is supported on Windows >= Vista only. Also I suspect the currently "fixed" implementation is skipping some legitimate open files already.

1 . Install a driver to query for open file handles

We cannot use a system driver for various reasons. A quick search on the ml led to these posts:
https://groups.google.com/forum/#!searchin/psutil/driver/psutil/cAAHCOMgQmw/ttzDwqEOefgJ
https://groups.google.com/forum/#!searchin/psutil/driver/psutil/EbdkIGlb4ls/KRe4tMDVPc8J
Other references are available on the bug tracker but long story short is that a system driver is not really suitable for a Python library.

2 - Create a new thread to run the API calls but there is no way to clean up the hanging thread

This is not really a solution.

[mrjefftang]

I agree options 1 and 2 are not good solutions given the context of being a python library.

This blog post (http://0memory.blogspot.com/2008/08/getfinalpathnamebyhandle-api-hungs.html) suggests using GetFileInformationByHandleEx (http://msdn.microsoft.com/en-us/library/aa364953(VS.85).aspx) which is introduced in WIndows Vista+

[jimbo1969]

Thanks for responding, guys! Jeff, to answer your question I wrote the following test script, which returned the following results before hanging forever on 32-bit Windows XP (I cannot say how it acts on newer versions) (short answer= "BTTray.exe"):
import psutil
try:
__procs = filter(lambda p: p.name, psutil.process_iter()) # get all processes
__for proc in procs:
____print("Try: ", str(proc.name)[45:-2])
____try:
______l = proc.open_files() # call suspect method to get list of all files that the process has open
____except psutil.NoSuchProcess as e: # catch when process closes before we get open files
______print("Process terminated.\n", e)
____except psutil.AccessDenied as e: # catch access denied inside open_files()
______print("Denied open_files() for ", e)
____except Exception as e:
______print("other error.\n", e)
except Exception as e: # catch access denied to process
__print("Error in outer try:\n", e.args)

output:
Try: (pid=0, name='System Idle Process') at 23730928
Try: (pid=4, name='System') at 23731728
Try: (pid=236, name='ScanToPCActivationApp.exe') at 23803760
Try: (pid=256, name='pythonw.exe') at 24046736
Try: (pid=352) at 24046864
Denied open_files() for (pid=352)
Try: (pid=404, name='ati2evxx.exe') at 24046800
Denied open_files() for (pid=404, name='ati2evxx.exe')
Try: (pid=432, name='svchost.exe') at 24046992
Denied open_files() for (pid=432, name='svchost.exe')
Try: (pid=536) at 24047056
Denied open_files() for (pid=536)
Try: (pid=768, name='AnyDVDtray.exe') at 24047024
Try: (pid=792, name='iexplore.exe') at 24047280
Try: (pid=864, name='svchost.exe') at 24046672
Denied open_files() for (pid=864, name='svchost.exe')
Try: (pid=896) at 24047376
Denied open_files() for (pid=896)
Try: (pid=908, name='svchost.exe') at 24046640
Denied open_files() for (pid=908, name='svchost.exe')
Try: (pid=972, name='QBCFMonitorService.exe') at 24047472
Denied open_files() for (pid=972, name='QBCFMonitorService.exe')
Try: (pid=1000, name='MsMpEng.exe') at 24047536
Denied open_files() for (pid=1000, name='MsMpEng.exe')
Try: (pid=1012, name='schedul2.exe') at 24047600
Denied open_files() for (pid=1012, name='schedul2.exe')
Try: (pid=1036, name='svchost.exe') at 24047568
Denied open_files() for (pid=1036, name='svchost.exe')
Try: (pid=1092, name='notepad.exe') at 24046512
Try: (pid=1128, name='msseces.exe') at 24046448
Try: (pid=1132, name='AppleMobileDeviceService.exe') at 24133904
Denied open_files() for (pid=1132, name='AppleMobileDeviceService.exe')
Try: (pid=1160, name='mDNSResponder.exe') at 24133968
Denied open_files() for (pid=1160, name='mDNSResponder.exe')
Try: (pid=1280, name='acrotray.exe') at 24134032
Try: (pid=1308, name='spoolsv.exe') at 24134096
Denied open_files() for (pid=1308, name='spoolsv.exe')
Try: (pid=1400, name='svchost.exe') at 24134160
Denied open_files() for (pid=1400, name='svchost.exe')
Try: (pid=1412, name='MSOSYNC.EXE') at 24134224
Try: (pid=1440, name='jqs.exe') at 24134288
Denied open_files() for (pid=1440, name='jqs.exe')
Try: (pid=1468, name='svchost.exe') at 24134352
Denied open_files() for (pid=1468, name='svchost.exe')
Try: (pid=1492, name='iTunesHelper.exe') at 24134448
Try: (pid=1496, name='inetinfo.exe') at 24134576
Denied open_files() for (pid=1496, name='inetinfo.exe')
Try: (pid=1500, name='tvnserver.exe') at 24134704
Denied open_files() for (pid=1500, name='tvnserver.exe')
Try: (pid=1536, name='smss.exe') at 24134768
Denied open_files() for (pid=1536, name='smss.exe')
Try: (pid=1624, name='NBService.exe') at 24134832
Denied open_files() for (pid=1624, name='NBService.exe')
Try: (pid=1820) at 24134896
Denied open_files() for (pid=1820)
Try: (pid=1876, name='OUTLOOK.EXE') at 24134864
Try: (pid=1912, name='ati2evxx.exe') at 24135152
Denied open_files() for (pid=1912, name='ati2evxx.exe')
Try: (pid=1924) at 24135280
Denied open_files() for (pid=1924)
Try: (pid=1956, name='winlogon.exe') at 24135216
Denied open_files() for (pid=1956, name='winlogon.exe')
Try: (pid=2000, name='services.exe') at 24135408
Denied open_files() for (pid=2000, name='services.exe')
Try: (pid=2012, name='lsass.exe') at 24135472
Denied open_files() for (pid=2012, name='lsass.exe')
Try: (pid=2052, name='iexplore.exe') at 24135536
Try: (pid=2088, name='wscntfy.exe') at 24135696
Try: (pid=2264, name='UsbClientService.exe') at 24135824
Denied open_files() for (pid=2264, name='UsbClientService.exe')
Try: (pid=2276, name='vmh.exe') at 24135888
Denied open_files() for (pid=2276, name='vmh.exe')
Try: (pid=2304, name='searchindexer.exe') at 24135952
Denied open_files() for (pid=2304, name='searchindexer.exe')
Try: (pid=2364) at 24136048
Denied open_files() for (pid=2364)
Try: (pid=2528, name='APAgent.exe') at 24136016
Try: (pid=2628, name='PCSuite.exe') at 24136272
Try: (pid=2660, name='TrueImageMonitor.exe') at 24136496
Try: (pid=2696, name='btwdins.exe') at 24136624
Denied open_files() for (pid=2696, name='btwdins.exe')
Try: (pid=2856, name='schedhlp.exe') at 24136688
Try: (pid=2868, name='ETDCtrlHelper.exe') at 24136752
Try: (pid=2944, name='tvnserver.exe') at 24136816
Try: (pid=3104, name='hpqtra08.exe') at 24136880
Try: (pid=3152, name='pythonw.exe') at 24136944
Try: (pid=3212, name='HPNetworkCommunicator.exe') at 24137072
Try: (pid=3252, name='pythonw.exe') at 24137296
Try: (pid=3320) at 24137424
Denied open_files() for (pid=3320)
Try: (pid=3400, name='BTTray.exe') at 24137360

[giampaolo]

This is now supposed to be fixed by PR #597 which spawns a thread.

[daveajones]

thanks @giampaolo @mrjefftang . much appreciated.

[giampaolo]

From original comment:

This blog post (http://0memory.blogspot.com/2008/08/getfinalpathnamebyhandle-api-hungs.html) suggests using GetFileInformationByHandleEx (http://msdn.microsoft.com/en-us/library/aa364953(VS.85).aspx) which is introduced in WIndows Vista+

...here's an archived copy in case that blog disappears:
https://web.archive.org/web/20201023075814/https://0memory.blogspot.com/2008/08/getfinalpathnamebyhandle-api-hungs.html

[PolarGoose]

From original comment:

This blog post (http://0memory.blogspot.com/2008/08/getfinalpathnamebyhandle-api-hungs.html) suggests using GetFileInformationByHandleEx (http://msdn.microsoft.com/en-us/library/aa364953(VS.85).aspx) which is introduced in WIndows Vista+

...here's an archived copy in case that blog disappears: https://web.archive.org/web/20201023075814/https://0memory.blogspot.com/2008/08/getfinalpathnamebyhandle-api-hungs.html

I have tried GetFileInformationByHandleEx:

• It hangs the same way NtQueryObject(ObjectNameInformation) does.
• The information in the blog post you mentioned is incorrect. GetFileInformationByHandleEx doesn't use a driver. It uses NtQueryObject under the hood, like GetFileInformationByHandle.
• The file name it returns doesn't contain a drive name: \Windows\System32\en-US\mswsock.dll.mui