Tab setting anomaly?

[vefatica]

I have a plugin command, TABS, for the TCC shell.

v:\> tabs /?
TABS -width    - set evenly-spaced tabstops; e.g., TABS -4
TABS N1 N2 ... - set tabstops at columns N1, N2, ... (1-based)
both clear previously set tabs; also clear with TABS 0

It first ensures VT mode is enabled, clears all tabs (ESC 3 g) and in the evenly-spaced case, successively moves the cursor to column 1, 1+width, 1+2*width, ... and sets a tab (ESC H).

This batch file works as desired.

tabs -16
do i=0 to 7 ( echos %i^^t )
echo ^r^n%_ruler

This batch file doesn't work as desired,

echo foo | grep bar > nul
tabs -16
do i=0 to 7 ( echos %i^^t )
echo ^r^n%_ruler

See how all the tabs after column 1 are in the wrong place (and rather inexplicably so)!

After a few hours of debugging, I'm at a complete loss. Regardless of what the pipe and redirection may do to the console, I figure we're starting all over again with the TABS command.

Any ideas?

[vefatica]

I should have noted that any intervening output fixes things. For example, this batch file works as desired.

echo foo | grep bar > nul
tabs -16
echo foo
do i=0 to 7 ( echos %i^^t )
echo ^r^n%_ruler

[vefatica]

Just a note ... in this one

echo foo | grep foo > t:\foo
tabs -16
do i=0 to 2 ( echos %i^^t )
echo ^r^n%_ruler

which does not work correctly,

the terminal is doing exactly what it was asked to do. According to the debug tap, the "do" command produces

0␛[9C1␛[15C2␛[15C

which is what we see.

Where did the "9" come from? It takes you to column 11 which does not correspond to an old (or new) tab stop. Before the experiment, the tab stops were at 1, 9, 17, et c.

And the debug tap does not show the tab_clear and tab_set sequences (though I doubt that seeing them would have been much help).

[DHowett]

So, the debug tap actually only shows off equivalent VT that should result in the same visual representation as the console's backing buffer. Unfortunately it will not be very useful to debug this.

[DHowett]

My biggest concern is that some process is mucking with the console output mode between when grep exits, when tabs starts, when tabs exits and when the rest of your script continues. The unfortunate truth is that tabstops are a VT-output-specific setting and and they may not be durable if something keeps jiggling the console modes.

I know that tools that use the Cygwin (or MSYS2 -- it's Cygwin-derived) runtime do do some console mode shenanigans. If you use a tool that isn't grep (like, findstr?)... does it reproduce?

[vefatica]

I doubt anyone is messing with the tab settings and recall you making a change a few years ago so that the tab settings would be remembered when the console goes out_of/back_into VT mode.

[j4james]

My guess is this is just something writing NUL values to the conhost buffer, which conpty can't handle. So instead of the tab line starting with "0 1", it's actually got something like "0\0\0\0\0\0 1", and when that's forwarded over conpty, the NUL values get dropped, and all the subsequent tab positions appear to be off by 5.

In the example where you perform the additional echo foo, that's probably overwriting the corrupted line, so when the tabs are output on the following line, everything works correctly.

[vefatica]

@j4james , I reckon you're right about that. I can get rid of the problem by, after the grep, clearing the whole buffer or by writing a line of spaces and returning the cursor to column 1; I'll find other ways. Have you any idea what grep (Gnu 32-bit or Cygwin, 64-bit, also Gnu) might be doing?

[j4james]

No idea what they're doing, and I may be wrong about the NUL thing anyway. But if it is that, then it's a Windows Terminal problem (it's being tracked in issues #6265 and #4363).

[vefatica]

I just noticed an ESC [ 3 J in my clear_the_whole_buffer command. '3' is not documented. What does it do?

And, for my purposes, this works to fix whatever grep does: echos ^e[1G^e[0J (cursor to BOL and clear thereafter).

[j4james]

I just noticed an ESC [ 3 J in my clear_the_whole_buffer command. '3' is not documented. What does it do?

That erases the scrollback buffer. It's an XTerm extension that's documented here.

[vefatica]

Thanks! I still go here (below). Is there a better place?

https://learn.microsoft.com/en-us/windows/console/console-virtual-terminal-sequences

With regard to whatever grep is doing, clearing one line helps. Are the tabs remembered in the buffer itself? My experience suggests that they're being altered.

[j4james]

If you want to know what a particular escape sequence does, the best place to look would be the DEC programming manuals, e.g. search for something like VT525 Programmer Information.

I you can't find the operation you're looking for there, then it's likely an XTerm extension, which will be documented on the page I linked above.

If you want to know which sequences are supported by Windows Terminal, you'll just have to try them out and see what works. There's no up to date list that I'm aware of (see MicrosoftDocs/Console-Docs/issues/91).

[j4james]

Are the tabs remembered in the buffer itself?

Tabs are control characters, so they don't store anything in the buffer - they just move the cursor position to the next tab stop.

[vefatica]

Yeah, but where is it remembered where the tab stops are? Is that somewhere that grep could have corrupted?

[j4james]

They are stored in the AdaptDispatch class, which is independent of the buffer. It's highly unlikely they would be corrupted by grep. The only way they can be modified is with escape sequences. And while I could imagine an app accidentally clearing them all, or accidentally resetting them to the 8-column defaults, you couldn't easily move all of them 5 columns to the left - that would take a lot of effort.

[j4james]

Btw, you can confirm whether this has anything to do with tabs by replacing your tab loop with CUF sequences. I'm not sure what the syntax is for this in TCC, but instead of outputting a tab character, just output the TCC equivalent of "\e[15C". If my NUL theory is correct, the digits should again be 5 columns left of where you expect them to be.

[vefatica]

It does not misbehave when I use CUF. And using CUF first fixes it if tabs are later used.

But (see post 3) ... when I was using echo 0^t1^t2 (tabs) I saw this in the debug tap.

So a tab stop was corrupted, at least temporarily.

[j4james]

Tabs stops are represented as column numbers, not relative offsets, so that's not just one tab stop that's corrupted - they're all in the wrong position. But looking at your debug tap again, my NUL theory is probably wrong. Visually I can reproduce your output by writing NUL values to the buffer, but I can actually see those values in my debug tap. I get something like this:

0␀␀␀␀␀␀␛[9C1␛[15C2␛[15C

And the fact that you're not seeing the same issue with CUF also suggests it must be something other than NUL. I have no idea what that might be though.

[vefatica]

I can't reproduce ANY erros in a console (Windows 10 ConHost or OpenConsole). It has nothing to do with grep or anything Gnu; a variety of apps used in the pipe will cause the same error (e.g., findstr.exe). I can change the behavior with command grouping. For example, there is no error if I use (echo foo | grep foo) > nul as opposed to echo foo | grep foo > nul which does produce the error.

So it has something to do with how TCC implements piping and redirection. I'm pretty sure that the console goes out of VT mode and back into it when TCC does piping. Is it true that the tab settings are remembered when that happens? Are you aware of any mechanisms that would thwart that? I doubt TCC is messing with the tabs directly.

[j4james]

Is it true that the tab settings are remembered when that happens?

If the tab settings weren't remembered, then you'd expect them to be reset to every 8 columns, or perhaps no tabs at all. What you're getting - visually at least - is all the tab positions are 5 columns to the left. So it's far more likely that the tabs are fine, but conpty is not able to reproduce the buffer content correctly. That's why you don't see the same problem in conhost or openconsole.

[vefatica]

Does that account for the bogus ESC[9C in the debug tap? Is there some fault here ... and whose is it?

[j4james]

Yes. That can happen if conpty's idea of the cursor position is out of sync with reality. It probably knows it needs to render the 1 in column 16, but it may think that the cursor is already in column 7, so it only moves forward by 9 columns. One way that can happen is with NUL values in the buffer, as I've explained above, but that doesn't seem to be the case here. We'd need to know exactly what TCC is doing to find the real cause.

[vefatica]

Yeah, there does seem to be a disconnect when it comes to where the cursor is. After the pipe/redirection, a mere ^e[1G (CHA) gets rid of the later error. I doubt we'll ever know what TCC is doing.

[vefatica]

Hmmm! That difference of 6 seems to be somehow built-in to TCC. At least I have something to ask about in TCC's forum. Below, _COLUMN in built-in to TCC and, no doubt, uses GetConsoleScreenBufferInfo[Ex] to get the current cursor position.

v:\> echo %_column & echos foo | findstr bar & echo %_column
0
6

That's a bit odd in that, if the cursor were really at 6, I'd expect the "6" to be echoed at column 6! :-)

[vefatica]

I posted in TCC's forum and almost immediately retracted my post because it's WT-specific, In a console, I see this.

v:\> echo %_column & echos foo | findstr bar & echo %_column
0
0

Regardless of the middle command, it's always 6 in WT and 0 in a console. I'm not sure what to make of it!