Terminal Emulation on Windows without ConPTY results in ANSI Escape Code flooding

[rowds15]

Describe the bug
Terminal Emulation on Windows 7 and Server 2008R2 not rendering correctly

To Reproduce
Steps to reproduce the behavior:

1. Go to 'Open Terminal'
2. See error

Expected behavior
A clear and concise description of what you expected to happen.

Server Software (please complete the following information):

• OS: [e.g. AWS OS]
• Network: LAN/WAN, Nginx reverse proxy
• Version: [e.g. 1.0.71]
• Node: [e.g. 16.17.0]

Client Device (please complete the following information):

• Device: Laptop
• OS: Windows 10
• Network: Local to Meshcentral, Remote over WAN]
• Browser: Firefox & Google Chrome]

Remote Device (please complete the following information):

• Device: PC/Server
• OS: Windows 7 and Server 2008
• Network: Local to Meshcentral, Remote over WAN]
• Current Core Version (if known): Apr 4 2022, 3863795360 & Mar 25 2022, 3863795360

Issue started with version 1.0.70 and yes I know Windows 7 and server 2008 are EOL but.....

[sblosser]

Experiencing this as well.

[krayon007]

The main issue with these two OS is they do not support MSFT's pseudo console API (ConPTY) which is why the behavior is wierd because we had to do some hacks to fake the terminal. But some apps aren't tolerable to how it works, which is why it renders incorrectly at time

What specific app are you running, and how does it render?

[Ylianst]

The screen shot above does not show any applications. It's just the Windows command shell.

[rowds15]

I'm not running any Applications it's just the command shell

this is what types out when i type ipconfig [4;29H[0mi[4;30H[0mp[4;31H[0mc[4;32H[0mo[4;33H[0mn[4;34H[0mf[4;35H[0mi[4;36H[0mg[6;1H

[MC-PM]

Same is happening since a couple of days on several Windows 10 and Windows Server 2012 devices. CMD is appearing kind of missing layout. This is also happening on the PS option:

[krayon007]

Can you run the osinfo console command, for your win 10 device that is messed up. I'm guessing it says ConPTY: No?

[rowds15]

osinfo from working and non working os's

OS: Microsoft Windows Server 2012 R2 Standard - 9600 [ConPTY: NO]
OS: Microsoft Windows Server 2008 R2 Standard - 7601 [ConPTY: NO]
OS: Microsoft Windows 7 Professional - 7601 [ConPTY: NO]
OS: Microsoft Windows 10 Pro - 21H2/19044 [ConPTY: YES]

[Matt-CyberGuy]

We've started noticing this issue as well more and more

I've definitely seen it on a few Windows 10 systems, and Windows 2019 server

[adsgreen]

It used to be "Ok" - whilst not perfect it could be used for basic tasks as you only generally encountered an issue when text scrolled off the bottom of the screen. However now it's not usable with the control codes being output.

[krayon007]

This is really interesting. I can't reproduce it right now on any of my machines that match match what you guys are running. It looks like it is an issue with the xterm environment in the browser, as that is what is supposed to be interpreting those vt100 codes. I'm looking into it to see if I can find a system that has this issue, then I can test with different server versions to see what happened, or if its a config issue, etc.

[krayon007]

Does it make any difference if you append the following query string to the url in your browser, before connecting the terminal?

&xterm=0

[sblosser]

Does it make any difference if you append the following query string to the url in your browser, before connecting the terminal?

&xterm=0

I checked a few systems and this did not help.

[adsgreen]

Same here. Didn’t help and I tried a number of clients including mobile.

[krayon007]

I'm curious if all the people having issue with this, are all running reverse proxies of some type? I'm having a hard time trying to reproduce this issue. I even built multiple test clients running the same versions of windows, and tried the versions of the servers mentioned above. For these systems, the agent side is very simplistic in how it sends the escape codes. It almost seems like something in between is escaping the escape codes... Are all the clients affected on your guys network, or just these windows machines that lack ConPTY?

[si458]

@krayon007 ive just tried an old Windows 7 Pro pc we still have around and im experiencing the same,
it just pours our garbage,
i am not using a proxy either, im running meshcentral directly on 80+443 and its handling LE too,
the server is external ip, and the remote client is also external, im using hybrid mode, also 1.0.85 MC
Agent Version: Aug 25 2022, 1360394467
OS: Microsoft Windows 7 Professional - 7601 [ConPTY: NO]

[sblosser]

No reverse proxies here. I'm running the same config as @si458.

[adsgreen]

No reverse proxy here either.
running the letsencrypt https service.

it doesn’t impact all our clients

[krayon007]

Ok, can someone try this:

Using Firefox, go to the terminal tab, then hit F-12 to bring up the browser console... Then select network
Then click connect on the terminal...
Then in the network tab of Firefox, select the one that says meshrelay, which should be one of the first ones.
Then on the right side, select response

And then show me what it receives on the websocket... I want to see if the escape codes are arriving correctly, or if they are escaped, etc... Here's what it looks like from one of my Windows machines that lacks ConPTY:

[sblosser]

Here is what a WIndows Server 2016 (no ConPTY) looks like (agent branding/folder redacted):

[sblosser]

In case it is needed, here is the output from a Server 2019 (with ConPTY):

[krayon007]

So from the same browser, some agents work and some dont?
This is very interesting, becuase your output looks identical to mine. Seems like the escape codes are arriving correctly, but for some reason aren't getting interpreted correctly by the web application... I'll have to dig deeper...

[si458]

@krayon007 ok im seeing the same as @sblosser for my Windows 7 machine

[krayon007]

Can you click on one of the entries, so it shows the raw data? Like this:

[si458]

[1;1H[0mMicrosoft Windows [Version 6.1.7601]                                            
[2;1H[0mCopyright (c) 2009 Microsoft Corporation.  All rights reserved.                 
[3;1H[0m                                                                                
[4;1H[0mC:\Program Files\Hestor Ltd\Mesh Agent>                                         

any use?

[adsgreen]

When connecting to terminal I get this in the browsers network debug on two messages:
OS: Microsoft Windows 10 Pro - 21H2/19044 [ConPTY: YES]

00000000: 1b5b 324a 1b5b 6d1b 5b48 4d69 6372 6f73  .[2J.[m.[HMicros
00000001: 6f66 7420 5769 6e64 6f77 7320 5b56 6572  oft Windows [Ver
00000002: 7369 6f6e 2031 302e 302e 3139 3034 342e  sion 10.0.19044.
00000003: 3138 3839 5d1b 5d30 3b43 3a5c 5749 4e44  1889].]0;C:\WIND
00000004: 4f57 535c 5379 7374 656d 3332 5c63 6d64  OWS\System32\cmd
00000005: 2e65 7865 071b 5b3f 3235 68              .exe..[?25h

00000000: 0d0a 2863 2920 4d69 6372 6f73 6f66 7420  ..(c) Microsoft 
00000001: 436f 7270 6f72 6174 696f 6e2e 2041 6c6c  Corporation. All
00000002: 2072 6967 6874 7320 7265 7365 7276 6564   rights reserved
00000003: 2e0d 0a1b 5b34 3758 0d0a 433a 5c50 726f  ....[47X..C:\Pro
00000004: 6772 616d 2046 696c 6573 5c4d 6573 6820  gram Files\Mesh 
00000005: 4167 656e 743e 1b5b 3139 581b 5d30 3b41  Agent>.[19X.]0;A
00000006: 646d 696e 6973 7472 6174 6f72 3a20 433a  dministrator: C:
00000007: 5c57 494e 444f 5753 5c53 7973 7465 6d33  \WINDOWS\System3
00000008: 325c 636d 642e 6578 6507                 2\cmd.exe.

However on the node that has weird behaviour :
OS: Microsoft Windows 10 Enterprise 2016 LTSB - 14393 [ConPTY: NO]

00000000: 5b32 3b31 485b 306d 2863 2920 3230 3136  [2;1H[0m(c) 2016
00000001: 204d 6963 726f 736f 6674 2043 6f72 706f   Microsoft Corpo
00000002: 7261 7469 6f6e 2e20 416c 6c20 7269 6768  ration. All righ
00000003: 7473 2072 6573 6572 7665 642e            ts reserved.

00000000: 5b34 3b31 485b 306d 433a 5c50 726f 6772  [4;1H[0mC:\Progr
00000001: 616d 2046 696c 6573 5c4d 6573 6820 4167  am Files\Mesh Ag
00000002: 656e 743e                                ent>

And this renders on the terminal browser screen as:
[1;1H[0mMicrosoft Windows [Version 10.0.14393] [2;1H[0m(c) 2016 Microsoft Corporation. All rights reserved. [3;1H[0m [4;1H[0mC:\Program Files\Mesh Agent> [5;1H[0m [6;1H[0m [7;1H[0m [8;1H[0m [9;1H[0m [10;1H[0m [11;1H[0m [12;1H[0m [13;1H[0m [14;1H[0m [15;1H[0m [16;1H[0m [17;1H[0m [18;1H[0m [19;1H[0m [20;1H[0m [21;1H[0m [22;1H[0m [23;1H[0m [24;1H[0m [25;1H[0m [1;1H[0mMicrosoft Windows [Version 10.0.14393][2;1H[0m(c) 2016 Microsoft Corporation. All rights reserved.[4;1H[0mC:\Program Files\Mesh Agent>

[krayon007]

It looks like the ESC character got dropped... I'll have to check to see if I can figure out where it got dropped...

[Matt-CyberGuy]

Ok... I can also confirm, when I rolled one of my personal mesh server back to 0.9.77, terminal for Windows 7 began functioning correctly.

I don't think I have a way to update to in between versions, so will have to take @rowds15 word for non-ConPTY terminal functioning in that version

[krayon007]

Interesting. I'll do some testing with these versions to see if I can spot something. Just for giggles I'll start with an older version and start upgrading to newer versions, as maybe something in the update process screwed something up.

[silversword411]

It will...of course....but something ridiculously inconsequential...like a comma, or a slash in the wrong place 🤣

[krayon007]

All of you guys that are having issues.... Do you all have minify set to true?

[silversword411]

"Minify": 1,

was in my config.

Removing that line...made no difference on my Windows Server 2012 R2 terminal...still had the [21;1H[0m 's on terminal Admin Shell session

[krayon007]

I figured it out. It was indeed related to minification. I'm not sure if when you restarted your server, if minification was undone, it may have still got the setting from the db. I think you need to set it to 0, not remove it...

But anyways, the problem was that when win-terminal.js was minified, it for some reason removed the ESC character sequence. I'll have to play around with the minifier he used to get it to not do that...

But anyways, there are a few ways to work around this... Either disable minification....
Or, go to your server, go to the agents folder, and go to the modules_meshcore_min folder.
In there, you'll find the problem win-terminal.min.js

If you edit that file, search for:
function GetEsc(e, t)

and you'll see the definition as:
return Buffer.from("[" + t.join(";") + e)

you'll need to change it to:
return Buffer.from("\x1B[" + t.join(";") + e)

Or, alternately, you can just copy agents\modules_meshcore\win-terminal.js
and save it into: agents\modules_meshcore_min\win-terminal.min.js

[krayon007]

On a side note, I tested this on Windows Server 2012 R2. Ylian made changes to the minifier a few versions ago, so this is probably how it broke in a recent version.

[si458]

@krayon007 I don't use minify and I have this issue? Are u sure it's a minify issue?

[krayon007]

@krayon007 I don't use minify and I have this issue? Are u sure it's a minify issue?

Well, in my case, I was never able to reproduce this issue, until I turned minification on. Then I was able to get it to go back and forth between working and not working, with the above mentioned change. From a usage point of view, everything seems to line up. The ESC sequence was deleted by the minifier, and the minifier was modified in version 1.0.70, which is when people reported this issue happening. And looking at the logs people posted earlier, looking at the raw sequences, it looked like the ESC character was missing, which would explain why the VT100 codes simply printed to the output, instead of being eaten by the console.

If you go to your server, do you have an agents\modules_meshcore_min folder? If you never enabled minification, that folder should not exist.

[si458]

If you go to your server, do you have an agents\modules_meshcore_min folder? If you never enabled minification, that folder should not exist.

the folder does exists at /home/simon/meshcentral/node_modules/meshcentral/agents/modules_meshcore_min
but i dont have minify listed in my config.json ? and never enabled it?

  "$schema": "http://info.meshcentral.com/downloads/meshcentral-config-schema.json",
  "settings": {
    "cert": "meshcentral.myserver.com",
    "port": 443,
    "noAgentUpdate": 1,
    "redirPort": 80,
    "AgentPong": 300,
    "authLog": "authlog.log",
    "relayDNS": "relay.meshcentral.myserver.com",
    "autoBackup": {
      "backupIntervalHours": 12,
      "keepLastDaysBackup": 7,
      "backupPath": "meshcentral-backup"
    }
  },
  "domains": {
    "": {
      "agentInviteCodes": true,
      "title": "myserver",
      "agentCustomization": {
        "companyName": "myserverLtd",
        "image": "128x128.png"
      },
      "assistantCustomization": {
        "image": "128x128.png"
      },
      "androidCustomization": {
        "image": "128x128.png"
      }
    }
  },
  "letsencrypt": {
    "email": "simon@myserver.com",
    "names": "meshcentral.myserver.com,relay.meshcentral.myserver.com",
    "skipChallengeVerification": false,
    "production": true
  }
}

which network url request should i be inspecting/looking out for with firefox? as it never asks for win-terminal.min.js ?

EDIT: on a fresh npm install meshcentral in a clean folder, its downloading the agents\modules_meshcore_min folder?

[krayon007]

which network url request should i be inspecting/looking out for with firefox? as it never asks for win-terminal.min.js ?

You won't see that in the browser, becuase it's used by the agent, not the browser. As far as npm, I'm not sure what the default state is from npm, but I know when pulled the server from GIT, it was never there... One way to test if it is a screwed up minification, is to copy the win-terminal.js file from the non minified folder on your server, and place it on your agent manually, in the same folder as the agent... Then either restart the agent, or go to the console tab, and select "clear core", and "load default core".... The agent should use the copy that you manually copied over if the timestamp is newer.

But like I said, maybe it will work, if you specify minify in your config, and specify the value as false.

[krayon007]

If you wanted to test what your agent was actually using, to verify your server config... Copy the modules\dbTool.js file from the agent repository, and place it in the same folder as your agent... Then run the following from an elevated command prompt:
MeshAgent dbTool.js export CoreModule
This will export the meshcore the agent received from the server, into a file called CoreModule.js. Open it in a text editor... If the file is human readable, it is not minified. If the whole file is on a single line, it was minified.

[Matt-CyberGuy]

I have success!

Unfortunately disabling minifi was not enough. I decided to go in and remove modules_meshcore_min. Once I did that and rebooted, I was able to get a prompt on one of our Windows 7 machines!!

I don't care what Ylian says, you're the best Brian!! ;)

Cheers!

[krayon007]

I have success!

Unfortunately disabling minifi was not enough. I decided to go in and remove modules_meshcore_min. Once I did that and rebooted, I was able to get a prompt on one of our Windows 7 machines!!

I don't care what Ylian says, you're the best Brian!! Cheers!

Sweet! I'll have to talk to Ylian when he gets back. It may be like the signed agents thing, where if the file exists, the server always uses that...

[si458]

I have success!

Unfortunately disabling minifi was not enough. I decided to go in and remove modules_meshcore_min. Once I did that and rebooted, I was able to get a prompt on one of our Windows 7 machines!!

I don't care what Ylian says, you're the best Brian!! Cheers!

beat me to it!
i just did the exact same thing!
just removed the modules_meshcore_min folder, then restarted meshcentral

[si458]

@krayon007 might be helpful

meshcentral.js:2661:            if (obj.args.minifycore !== false) { try { moduleDirPath = obj.path.join(meshcorePath, 'modules_meshcore_min'); modulesDir = obj.fs.readdirSync(moduleDirPath); } catch (ex) { } } // Favor minified modules if present.
pluginHandler.js:152:            //if (obj.args.minifycore !== false) { try { moduleDirPath = obj.path.join(obj.pluginPath, 'modules_meshcore_min'); modulesDir = obj.fs.readdirSync(moduleDirPath); } catch (e) { } } // Favor minified modules if present.

[krayon007]

I have success!

Unfortunately disabling minifi was not enough. I decided to go in and remove modules_meshcore_min. Once I did that and rebooted, I was able to get a prompt on one of our Windows 7 machines!!

I don't care what Ylian says, you're the best Brian!! ;)

Cheers!

By the way @Matt-CyberGuy earlier I installed an agent on your test system you gave me access to, called: BryanAgentTest, feel free to delete/uninstall that service and remove my access.

[Matt-CyberGuy]

Oh right haha! Thanks

[si458]

@krayon007 fixed the default for you! #4594
it was checking if it was 0 instead of null for some reason?
so if null (not set) then set to false, which then skips the minifycore!
unless minifycore is set to true then minifycore
@silversword411 could you please add minifycore to the scheme and the docs as its not listed

[rowds15]

@krayon007
I can confirm that,
1, changing Minify to 0 or false, alone, does not resolve the issue.
2, removing the modules_meshcore_min folder and restarting the server service does fix the issue.

[silversword411]

@si458 If you're referring to

https://ylianst.github.io/MeshCentral/meshcentral/config/

I literally just copy/paste the raw file from here: https://raw.githubusercontent.com/Ylianst/MeshCentral/master/meshcentral-config-schema.json

It's currently up to date, and I periodically copy/paste it over to update. I only created that page so that all the config items are automatically added to the search index in mkdocs.

I don't feel comfortable with updating the source, not sure what to modify about minify (other than typos: dee3f58 :) )

If we need another section about minifycore in here somewhere I'm not sure what to put in there.

[shadyslim2018]

There is still a problem with the terminal on Windows Server 2012 R2, even after modify win-terminal.min.js return Buffer.from("[" + t.join(";") + e) to return Buffer.from("\x1B[" + t.join(";") + e)

When I press the arrow keys or del key, backspace is fine. These are the characters that show up.

[3~ DEL
[A UP
[B DOWN
[C RIGHT
[D LEFT

I also changed if (obj.args.minifycore === 0) obj.args.minifycore = false; to if (obj.args.minifycore == null) obj.args.minifycore = false; and it didn't make a difference.

[silversword411]

There is still a problem

You did say you deleted the files necessary, did you?

[krayon007]

Hopefully with disabling minification fixes this issue for most people. The issue with the arrow keys is a separate issue caused by the terminal not being an actual pseudo console. That particular issue will require some investigation on my part to find a solution, but I'm not sure if that can be solved, because normally it's the console that handles that, but in this scenario we're not using a real console, which is why up-arrow to show previous commands doesn't work, etc.

[shadyslim2018]

There is still a problem

You did say you deleted the files necessary, did you?

Yes

[shadyslim2018]

Hopefully with disabling minification fixes this issue for most people. The issue with the arrow keys is a separate issue caused by the terminal not being an actual pseudo console. That particular issue will require some investigation on my part to find a solution, but I'm not sure if that can be solved, because normally it's the console that handles that, but in this scenario we're not using a real console, which is why up-arrow to show previous commands doesn't work, etc.

All other Windows OS is working fine other than Windows 2012 R2

[si458]

Hopefully with disabling minification fixes this issue for most people. The issue with the arrow keys is a separate issue caused by the terminal not being an actual pseudo console. That particular issue will require some investigation on my part to find a solution, but I'm not sure if that can be solved, because normally it's the console that handles that, but in this scenario we're not using a real console, which is why up-arrow to show previous commands doesn't work, etc.

All other Windows OS is working fine other than Windows 2012 R2

I have the arrow key issue with windows 7 but i think @krayon007 is right,
we can close this issue as its fixed but open another issue for the arrow keys to be investigated