Check for exclamation marks on Windows

[josevalim]

No description provided.

[josevalim]

@E14, unfortunately it didn't work. If you have suggestions, drop them and I can merge it to let CI give it a try!

[E14]

@josevalim, I did poke a bit at it, but local tests were more than enough.

The issue is related to delayed-expansion, quoting does not help at all and escaping is not possible in a useful way as far as I can tell. While it seems it's possible to work with such arguments in batch files by disabling delayed expansion, that immediately makes looping using goto's impossible, or at least if you want to use a variable with changing value, like !par! is.

Another example that's easy to reproduce:

elixir.bat -e "IO.puts 'nowhere! To hide!'"
:: nowhere

So, these are the possible solutions I see, but none are "great" IMO, apart from possibly rewriting the whole thing in powershell, which you already mentioned you were planning on doing anyway:

• Leave as is (maybe expand documentation)
• Abort or show warning on stderr on any argument/option with a ! (including escaped ones)
• Write the argument parsing logic in another language - You mentioned PowerShell, but Erlang could also work if you used it only for preparing arguments

Because I already have it, here's a snippet that should detect any !:

@echo off
setlocal disabledelayedexpansion
echo %* | findstr "!" > nul
if %ERRORLEVEL% neq 0 (
    echo no bang
) else (
    echo bang
)
endlocal

[josevalim]

Thank you for validating it. So it seems that rewriting to PowerShell is the best option forward, I assume it would have a lower boot cost compared to calling Erlang for preprocessing. I appreciate the inputs here.

[E14]

So it seems that rewriting to PowerShell is the best option forward, I assume it would have a lower boot cost compared to calling Erlang for preprocessing. I appreciate the inputs here.

I'm not so sure about the lower boot cost. While it would be one additional (short-lived) process, powershell is fairly heavy as shells go, it's essentially a .net program - a quick check puts an empty powershell.exe console well above 60M working set (20M+ private), conversely a cmd.exe has just 5M working set (1.2M private)

[josevalim]

Interesting, we will measure that then with more care. We also looked into VBScript but compiled VBScript almost always trigger security checks/firewall/antivirus. :(

[josevalim]

I guess the boot time will also depend if you are already inside powershell or not?

[E14]

Runnning powershell scripts or commandlets within powershell does not seem to start a new process, so if your expectation is that elixir would be run primarily in this way, then that should yield the best performance.

[josevalim]

Perhaps the best option is to keep the .bat versions around with some minimal parsing and, if it finds anything complex, we delegate to powershell.

The other option would be to write a native executable and ship precompiled versions, but Elixir does not ship executables at the moment, and doing so will likely introduce complexities around other areas.

[simonmcconnell]

The other option would be to write a native executable and ship precompiled versions, but Elixir does not ship executables at the moment, and doing so will likely introduce complexities around other areas.

I would prefer an executable. I took a stab at this a while ago but hit a hurdle with a wx application and haven't looked further. Which language do you think it should be in? C? Go? Zig? Rust?

I've had customers mention the fact that the application is started with a script. It seems a little unprofessional.

[josevalim]

Which language do you think it should be in? C? Go? Zig? Rust?

The one with the smallest runtime footprint and file size possible. Now that Windows supports ARM too, we either need to ship universal binaries, or have precompiled versions per OS, which either bloats for everyone or introduces a bunch of complexity.

I've had customers mention the fact that the application is started with a script. It seems a little unprofessional.

To be fair, this sounds more of an assessment of the state of scripting languages on Windows (perhaps before powershell) than of scripts. Not including executables means one less liability in the supply chain. Plus, small executables on Windows are constantly flagged as false positives by anti-virus software. We tried executables for Livebook and it consistently showed to be the hardest experience to iron out for users. :(

And there is still releases, which can't move away from executables. However, I had another idea of how we could solve this issue, I will give it a couple tries.

[simonmcconnell]

To be fair, this sounds more of an assessment of the state of scripting languages on Windows (perhaps before powershell) than of scripts. Not including executables means one less liability in the supply chain. Plus, small executables on Windows are constantly flagged as false positives by anti-virus software. We tried executables for Livebook and it consistently showed to be the hardest experience to iron out for users. :(

I'm talking about when the user has to launch a GUI application with a .vbs/.ps1/.bat, which I have done. I should probably do something like elixir-kit but there are the code signing issues you mention. I saw the code signing handled in CI on something recently, I'll try dig it up.

[josevalim]

@simonmcconnell I see. Whatever we ended up on Livebook was what we found the best for the GUI approach after months of experimentation, so I do recommend tagging along.

[josevalim]

It seems my idea worked! It now handles exclamation marks on filenames and eval. We should potentially still move away from .bat files in the long term though.

@E14 one additional question: if I am inside Powershell, and we have both bin/iex.bat and bin/iex.ps1, will it choose the powerscript version first? Or will it start a .bat environment and then instantiate a new powershell environment inside that?

[simonmcconnell]

iex is an alias for Invoke-Expression, so you have to type iex.bat or iex.ps1.

[E14]

It seems my idea worked! It now handles exclamation marks on filenames and eval. We should potentially still move away from .bat files in the long term though.

Good solution! Too bad you still have to handle the options in the script to parse out the erl options. I did try to come up with ways to break it, but for now I can't

@E14 one additional question: if I am inside Powershell, and we have both bin/iex.bat and bin/iex.ps1, will it choose the powerscript version first? Or will it start a .bat environment and then instantiate a new powershell environment inside that?

Well iex is a powershell builtin alias (Invoke-Expression), so you'll always have to add the extension, but otherwise I'd assume so - I did a quick test, with an elixir.ps1 and that ran the powershell script instead of the bat.

Sadly, when you enter iex and auto-complete it does suggest iex.bat before iex.ps1, so seems to be in alphabetical order there.

[josevalim]

Good solution! Too bad you still have to handle the options in the script to parse out the erl options. I did try to come up with ways to break it, but for now I can't

Yeah, it was inspired by your suggestion to "parse it in Erlang" and then I thought: "well, why not let the Erlang/Elixir code we already invoke it parse it?". We still need bat for the Erlang bits (and you can still have ! in those, which then won't work), but those are used quite less frequently.