Unthreadable aliases hang when piped from or to

[Qyriad]

I'm not totally clear on the trigger conditions here, but the xonsh built-in commands that are unthreadable (xonfig, source, trace, fg) hang when piped to or from. However callable aliases that I've defined with @xonsh.tools.unthreadable don't seem to hang, so I'm not quite sure what's going on. Some builtins that don't hang include disown, which, source-foreign, showcmd, and bg.

Neither --no-rc nor $THREAD_SUBPROCS affect this behavior.

xonfig

``` $ xonfig +------------------+-----------------------------------+ | xonsh | 0.15.1 | | Git SHA | da7add7 | | Commit Date | Dec 12 19:30:59 2023 | | Python | 3.11.8 | | PLY | 3.11 | | have readline | True | | prompt toolkit | 3.0.41 | | shell type | prompt_toolkit | | history backend | json | | pygments | 2.17.2 | | on posix | True | | on linux | True | | distro | unknown | | on wsl | False | | on darwin | False | | on windows | False | | on cygwin | False | | on msys2 | False | | is superuser | False | | default encoding | utf-8 | | xonsh encoding | utf-8 | | encoding errors | surrogateescape | | xontrib | [] | | RC file 1 | /etc/xonsh/xonshrc | | RC file 2 | /home/qyriad/.config/xonsh/rc.xsh | +------------------+-----------------------------------+ ```

Expected Behavior

Commands like echo foo | source or trace --help | rg . should not hang.

Current Behavior & Reproduction

In a checkout of the latest commit:

[qyriad@Yuki:~/code/builds/xonsh]$ env XONSH_DEBUG=1 python -m xonsh --no-rc -c 'echo foo | source'
<string>:1:5 - echo foo | source
<string>:1:5 + ![echo foo | source]

(Reproduces interactively or non-interactively).

Traceback

From within the shell, Ctrl-C does nothing, but if executed non-interactively as above, CtrlC produces the following traceback:

[qyriad@Yuki:~/code/builds/xonsh]$ env XONSH_DEBUG=1 XONSH_SHOW_TRACEBACK=1 python -m xonsh --no-rc -c 'echo foo | source'
<string>:1:5 - echo foo | source
<string>:1:5 + ![echo foo | source]
^Cxonsh: To log full traceback to a file set: $XONSH_TRACEBACK_LOGFILE = <filename>
Traceback (most recent call last):
  File "/home/qyriad/code/builds/xonsh/xonsh/jobs.py", line 270, in wait_for_active_job
    _, wcode = os.waitpid(obj.pid, os.WUNTRACED)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ChildProcessError: [Errno 10] No child processes

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/qyriad/code/builds/xonsh/xonsh/built_ins.py", line 206, in subproc_captured_hiddenobject
    return xonsh.procs.specs.run_subproc(cmds, captured="hiddenobject", envs=envs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/qyriad/code/builds/xonsh/xonsh/procs/specs.py", line 910, in run_subproc
    return _run_specs(specs, cmds)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/qyriad/code/builds/xonsh/xonsh/procs/specs.py", line 943, in _run_specs
    command.end()
  File "/home/qyriad/code/builds/xonsh/xonsh/procs/pipelines.py", line 459, in end
    self._end(tee_output=tee_output)
  File "/home/qyriad/code/builds/xonsh/xonsh/procs/pipelines.py", line 467, in _end
    for _ in self.tee_stdout():
  File "/home/qyriad/code/builds/xonsh/xonsh/procs/pipelines.py", line 369, in tee_stdout
    for line in self.iterraw():
  File "/home/qyriad/code/builds/xonsh/xonsh/procs/pipelines.py", line 254, in iterraw
    task = xj.wait_for_active_job()
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/qyriad/code/builds/xonsh/xonsh/jobs.py", line 275, in wait_for_active_job
    return _safe_wait_for_active_job(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/qyriad/code/builds/xonsh/xonsh/jobs.py", line 296, in _safe_wait_for_active_job
    rtn = wait_for_active_job(
          ^^^^^^^^^^^^^^^^^^^^
  File "/home/qyriad/code/builds/xonsh/xonsh/jobs.py", line 263, in wait_for_active_job
    active_task = get_next_task()
                  ^^^^^^^^^^^^^^^
  File "/home/qyriad/code/builds/xonsh/xonsh/jobs.py", line 308, in get_next_task
    _clear_dead_jobs()
  File "/home/qyriad/code/builds/xonsh/xonsh/jobs.py", line None, in _clear_dead_jobs
KeyboardInterrupt

---

Any suggestions on places to start digging for this one?

For community

⬇️ Please click the 👍 reaction instead of leaving a +1 or 👍 comment

[Qyriad]

I would presume this is also the same root issue as #2854, #5296, #5150, and #4884

[anki-code]

xunter tells me about the endless loop:

xunter --no-rc -c "xonfig | head" ++depth-lt 20

xonsh/xonsh/procs/pipelines.py

Lines 807 to 825 in 04cba39

	while proc.poll() is None:
	if not check_prev_done:
	# In the case of pipelines with more than one command
	# we should give the commands a little time
	# to start up fully. This is particularly true for
	# GNU Parallel, which has a long startup time.
	pass
	elif pipeline._prev_procs_done():
	pipeline._close_prev_procs()
	proc.prevs_are_closed = True
	break
	if not check_prev_done:
	# if we are piping...
	if prev_end_time is None:
	# or see if we already know that the next-to-last
	# proc in the pipeline has ended.
	if pipeline._prev_procs_done():
	# if it has, record the time
	prev_end_time = time.time()

Because _prev_procs_done never returns True for xonfig | head.

By putting import ipdb; ipdb.set_trace() in the appropriate place you can investigate the environment.

[anki-code]

Catched this for xonfig and for xontribs:

from xonsh.xonfig import XonfigAlias

aliases['xfg'] = XonfigAlias(threadable=True)
xfg | head
# Works as expected

aliases['xfg'] = XonfigAlias(threadable=False)  # current behavior in `xonfig`
xfg | head
# Hanging (the cause described above)


from xonsh.xontribs import XontribAlias
aliases['xnt'] = XontribAlias(threadable=True)
xnt list | grep load
# No hanging but grep is not working - we need to understand why

UPD: fixed in #5321

@gforsyth could you please comment why the aliases were marked as unthreadable? If you know or you can guess.

@Qyriad I think we need to understand the cause of hanging and as minimum create sane exception for the situation that hangs the execution.

I would presume this is also the same root issue as

Thanks for looking around! I've closed the same issues.

PS. Personally I would like to have a builtin tool to inspect the command pipeline (xonsh.pipelines.CommandPipeline) entirely. e.g. showcmd--full xonfig | head and output like:

cmd("xonfig"): alias=True, threadable=False, args=[],
cmd("head"): alias=False, threadable=True, args=[], stdin=pipe

OR improve CommandPipeline.__repr__ to show procs and specs.

It will help to debug the processes.

[gforsyth]

@gforsyth could you please comment why the aliases were marked as unthreadable? If you know or you can guess.

Well outside of memory, I'm afraid. But if they are working now with threadable=True we might as well change them.

I suspect some part of ArgParserAlias class is causing an issue here -- echo foo | source hanging is a bigger issue and also a more confusing one.

[anki-code]

Catched this for xonfig and for xontribs

Fixed in #5321

[jnoortheen]

I suspect some part of ArgParserAlias class is causing an issue here -- echo foo | source hanging is a bigger issue and also a more confusing one.

Yes that could be an issue. Especially how it keeps hold of stdin/stdout variables passed to it.

[anki-code]

Hey @jnoortheen! You wrote in #5321 (comment):

3. I suggested to run the aliases in a new process . I am not sure how other shells do that . Even though it will incur performance cost, it will get rid of all the stdin/pipe issues.

Could you please describe the machinery and pros/cons of this approach? May be if this requires just a few lines of code we can play with it in the branch.

[jnoortheen]

:1:5 - echo foo | source
:1:5 + ![echo foo | source]

The debug code reveals that , the default mode is being ![] instead of $[] .

[anki-code]

I debug this issue with this run.py:

from pprint import pprint

from xonsh.built_ins import XSH
XSH.load()

from xonsh.procs.specs import cmds_to_specs, run_subproc
from xonsh.tools import unthreadable, uncapturable


# Default alias - threadable by default.
@XSH.aliases.register('def')
def a_def(args, i, o):
    print('def', file=o)

# Unthreadable alias.
@XSH.aliases.register('unt')
@unthreadable
def a_unt(args, i, o):
    print('unt', file=o)


cmd = [['def'], '|', ['head']]  # This command is working well.
# cmd = [['unt'], '|', ['head']]  # This command is hanging without execution.
# cmd = [['unt']]  # This command executed well.

specs = cmds_to_specs(cmd, captured="stdout")
pprint([s.__dict__ for s in specs])

"""
https://github.com/xonsh/xonsh/blob/db83a3f35382b8f692c25d73740ddde11d5a17d3/xonsh/procs/specs.py#L314-L317
        captured : bool or str, optional
            The flag for if the subprocess is captured, may be one of:
            False for $[], 'stdout' for $(), 'hiddenobject' for ![], or
            'object' for !().
"""
r = run_subproc(cmd, captured="stdout")
print(r)

I found that threadable variable defines the class that will be used to run alias function:

xonsh/xonsh/procs/specs.py

Line 690 in db83a3f

cls = ProcProxyThread if thable else ProcProxy

xonsh/xonsh/procs/proxies.py

Line 762 in db83a3f

class ProcProxy:

xonsh/xonsh/procs/proxies.py

Line 351 in db83a3f

class ProcProxyThread(threading.Thread):

If we run threadable alias then ProcProxyThread runs parse_proxy_return and the loop based on proc.poll() can finish by getting the return code.

But if we run unthredable alias in pipe than ProcProxy avoid execution and parse_proxy_return will never be called and proc.poll() becomes endless loop because returncode=None forever. (Without pipe it works well because no waiting for finishing the previous process in pipe and ProcProxy.wait called.)

This is the cause why piping from unthreadable alias hang the execution.