Research: Capturing output from interactive tool e.g. fzf
#5342
Comments
anki-code
changed the title
anki-code
changed the title
anki-code
changed the title
fzf
|
Example for playing with Popen and fzf (it works perfect on macOS and also when ran from xonsh): # This is the same as bash `ls / | fzf 2> /tmp/fzf_stderr`
import subprocess as sp
ls = sp.Popen(('ls','/'), stdout=sp.PIPE)
fzf = sp.Popen(["fzf"], stdin=ls.stdout, stdout=sp.PIPE) # stderr=sp.PIPE - hide the fzf UI
head = sp.Popen(["head"], stdin=fzf.stdout, stdout=sp.PIPE)
stdout, stderr = head.communicate()
print('stdout:', repr(stdout))
print('stderr:', repr(stderr))💡 I found that if we set 🤔 Hypothesis: when we capture stderr from fzf we have the case with hidden fzf UI. |
|
I got answer from fzf owner:
|
|
Found repeatable case. Minimal case: xonsh --no-rc
!(fzf)
# Ctrl-C - RecursionError: maximum recursion depth exceeded
# Back to the terminal.
$(echo 1)
# ExceptionHow it works:
The challenge: we need to carefully terminate We can do -- Two types of exceptions can be. First: Exception in thread Thread-11:
object address : 0x111c12bc0
object refcount : 2
object type : 0x104aefd00
object type name: ValueError
object repr : ValueError('I/O operation on closed file.')
lost sys.stderr
Exception in threading.excepthook:
Traceback (most recent call last):
File "/Users/pc/.local/xonsh-env/lib/python3.12/threading.py", line 1379, in invoke_excepthook
hook(args)
File "/Users/pc/.local/xonsh-env/lib/python3.12/site-packages/xonsh/base_shell.py", line 198, in flush
getattr(getattr(self, "mem", lambda: None), "flush", lambda: None)()
ValueError: I/O operation on closed file.Second: Exception in thread Thread-12:
Traceback (most recent call last):
File "/usr/local/lib/python3.11/threading.py", line 1038, in _bootstrap_inner
self.run()
File "/usr/local/lib/python3.11/site-packages/xonsh/procs/posix.py", line 180, in run
i = self._read_write(procout, stdout, sys.__stdout__)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/xonsh/procs/posix.py", line 223, in _read_write
writer.flush()
ValueError: I/O operation on closed file.-- Related code points: xonsh/xonsh/procs/pipelines.py Line 333 in 2419eaa Line 162 in 8ff5079 |
|
New knowledge (for me). Short list:
Сonclusion: We see the situation that interactive process was stopped after run in thread (background?). This looks like Terminal Access Control Mechanism in action. CommandPipeline and PopenThread don’t know how to handle this situation. I'm going to treat this as a feature to implement. UPD: see #5361 Any help are very welcome! @jnoortheen @gforsyth |
|
You are doing a great work. As I tried to remove the threaded subprocs, most of the code has used threads in some way and needed more time and haven't looked beyond the surface. |
### Motivation There is annoying behavior when you run command in the loop and can't interrupt e.g. [this report](#5371) and a bit #5342. After diving into this I see the issue around return code. ### The essence Basically ``p = subprocess.Popen()`` populates ``p.returncode`` after ``p.wait()``, ``p.poll()`` or ``p.communicate()`` ([doc](https://docs.python.org/3/library/os.html#os.waitpid)). But if you're using `os.waitpid()` BEFORE these functions you're capturing return code from a signal subsystem and ``p.returncode`` will be ``0`` like success but it's not success. So after ``os.waitid`` call you need to set return code manually ``p.returncode = -os.WTERMSIG(status)`` like in Popen. Example: ```xsh python # python interactive import os, signal, subprocess as sp p = sp.Popen(['sleep', '100']) p.pid # 123 p.wait() # Ctrl+C or `kill -SIGINT 123` from another terminal p.returncode # -2 # BUT: p = sp.Popen(['sleep', '100']) p.pid # 123 pid, status = os.waitpid(p.pid, os.WUNTRACED) # status=2 # From another terminal: kill -SIGINT 123 p.wait() # 0 p.returncode # 0 ``` ```xsh from xonsh.tools import describe_waitpid_status describe_waitpid_status(2) # WIFEXITED - False - Return True if the process returning status exited via the exit() system call. # WEXITSTATUS - 0 - Return the process return code from status. # WIFSIGNALED - True - Return True if the process returning status was terminated by a signal. # WTERMSIG - 2 - Return the signal that terminated the process that provided the status value. # WIFSTOPPED - False - Return True if the process returning status was stopped. # WSTOPSIG - 0 - Return the signal that stopped the process that provided the status value. ``` See also: [Helpful things for knight](#5361 (comment)). ### Before ```xsh $RAISE_SUBPROC_ERROR = True sleep 100 # Ctrl+C _.rtn # 0 # It's wrong and RAISE_SUBPROC_ERROR ignored. for i in range(5): print(i) sleep 5 # 0 # Ctrl+C # Can't interrupt # 1 # 2 ``` ### After ```xsh sleep 100 # Ctrl+C _.rtn # -2 # Like in default Popen $RAISE_SUBPROC_ERROR = True for i in range(5): print(i) sleep 5 # 0 # Ctrl+C # subprocess.CalledProcessError ``` ### Notes * We need to refactor `xonsh.jobs`. It's pretty uncomfortable work with module. * The logic is blurry between Specs, Pipelines and Jobs. We need to bring more clear functions. * The `captured` variable looks like "just the way of capturing (stdout, object)" but in fact it affects all logic very much. We need to create table where we can see the logic difference for every type of capturing. E.g. in `captured=stdout` mode we use `xonsh.jobs` to monitor the command but in `captured=object` we avoid this and some logic from `xonsh.jobs` applied in `stdout` mode but missed in `object` mode. We need clear map. ## For community ⬇️ **Please click the 👍 reaction instead of leaving a `+1` or 👍 comment** --------- Co-authored-by: a <1@1.1> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
|
Current state. I made a bunch of merged pull requests during the looking around (see above). UPD: OS suspends these processes and posix shells keep them tracked in |
|
@anki-code This might be of your interest. junegunn/fzf@d274d09 |
anki-code
changed the title
fzffzf
…5422) ### Motivation There is no way to access to the CommandPipeline of executed command in uncaptured mode. This causes: * No chance to get return code in one universal way. * Barrier to trace and inspection. ### Before ```xsh $(ls nofile) # No way to access to the CommandPipeline of executed command. ``` ### After ```xsh $(ls nofile) __xonsh__.last.rtn # In interactive or not interactive. # 2 ``` ```xsh # Sugar sir: last = type('LastCP', (object,), {'__getattr__':lambda self, name: getattr(__xonsh__.last, name) })() ls nofile last.rtn # 2 ``` JFYI #5342 ## For community ⬇️ **Please click the 👍 reaction instead of leaving a `+1` or 👍 comment** --------- Co-authored-by: a <1@1.1> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Motivation
fzfis a common tool to fuzzy search and commonly used to choosing file name. People expect that it works in captured object mode!()but this mode was made to capture everything (stdout/stderr) and the people expectations are not satisfied because fzf use stderr to show UI and produces unexpected exceptions. We need to have solid approach of how to operate with this kind of tools and fix exceptions.UPD: Instant answer
Use
$()to operate this this kind of tools. The captured mode operator!()only for commands that have no any interaction with terminal.Goals
Test cases
Pipe (#2404 #2159 #5003 #4444):
Alias:
PTK (#4700):
Script mode:
TODO: XonshMode (main.py): single_command, script_from_file, script_from_file, interactive.
Exceptions
Test cases could work when we run it in fresh session and once BUT if we randomly run different kind of capturing in one session (i.e.
$(),![], etc) we may have unexpected errors:I/O operation on closed file.RecursionError: maximum recursion depth exceeded(!(echo 123 | fzf))stty: 'standard input': unable to perform all requested operations.Interrupted system callBad file descriptorThis issue is to catch and fix them.
Docs
Basics:
Find the code examples:
Subprocess/popen/thread:
Signals in threads:
TTY/stdin/stdout/stderr:
Unthreading:
Tools
Uncompleted efforts around threads
For community
⬇️ Please click the 👍 reaction instead of leaving a
+1or 👍 commentThe text was updated successfully, but these errors were encountered: