Entering backslash character '\' on windows causes null byte input '\x00'

[pygrum]

I was debating whether to post this here or on the readline issue page.

Affects

Windows only - works fine on Ubuntu and MacOS
I'm using version 0.1.6 of console and 1.0.12 of readline.

Issue

Whenever entering the backslash character, nothing seems to happen. However invisible null chars are sent the prompt, so that when I hit 'enter' I'll get 'unknown command "\x00" for ...'.

Now Sliver uses the same library and doesn't seem to share this issue for Windows builds. What I've also noticed is that my typing prompt (on cmd.exe) is a block cursor, while Sliver's is a regular line cursor. So perhaps it is a setting that I haven't explored. Note too, that the same versions of both readline and console are in use. Your support would be appreciated!

[maxlandon]

Hello @pygrum,

Are you using the Windows Terminal (the new one) or the legacy cmd.exe prompt ? If you are using the latter, please know that there is pretty much no support for it by this library. Many things won't work at all.

Related to this, this non-printed backslash was already an issue here, but should have been solved since. I'm thus eager to know what causes this !

Cursors:
The readline library uses the default terminal cursor, which may vary from cmd.exe to Windows Terminal (I suspect the latter to have a beam cursor by default).

See the readline documentation (options/config page) for ways to change cursor styles.

Finally, between v0.1.6 and v0.1.13 lots of things have been fixed, so I would advise upgrading your dependencies (Sliver is going to do it as well).

[pygrum]

Hi @maxlandon Thanks for the response.

I'm not sure what causes this. I've just tested on Windows Terminal as well as Command Prompt (both my app and sliver), and the no backslash, null bytes issue persists after a new build. I'm doing basic initialisation of the console (console.New("abc"); Run()) so I'm not sure if it's something on my side.

Here it is, running on windows terminal.

Finally, I would love to update (especially using my syntax highlight change request lol) but I've had to downgrade to use carapace completion alongside your carapace patch. It might be a straightforward fix on my side but I'll have to look into it more.

Thanks for your work on this library.

[pygrum]

Okay, an update.

I updated to the latest console version and null bytes aren't being registered, which is nice! However, backslashes still aren't being registered or appearing. The fact that the beam cursor isn't adopted might be another symptom of the underlying issue. I'll keep testing around and get back if I've found a permanent fix.

[maxlandon]

Okay so a couple things:

• before all, grateful for the thanks, I hope this lib will be useful to many.

• dependency upgrade: so the patched carapace library is about to be a thing of the past. Solutions have been found on the carapace repo, and in a few days console will use the main branch. No more replace directives needed.

• Null bytes: it's probable I didn't test my fix on Windows, so I actually wouldn't be surprised if it does not work.
  I will check that tomorrow and fix it. Expect a fix and corresponding new version released by the end of the week.

[pygrum]

Great to hear those updates. Thanks again, I look forward to the release! Meanwhile, I'll look for solutions to the backslash issue.

PS: could you show me the logic behind this / where it occurs in code?

The readline library uses the default terminal cursor, which may vary from cmd.exe to Windows Terminal (I suspect the latter to have a beam cursor by default).

[pygrum]

Hello @maxlandon , an FYI I've just compiled and tested for darwin/amd64 (on a separate machine) and backslashes don't register here either.
I used the example project.
Although null byte response doesn't appear, it seems like there is something going on in the background - autocomplete consumes something if you've pressed backslash, meaning entering a command after the fact doesn't trigger the autocomplete.

If you have access to a non-linux machine maybe you could test that to make sure it's a real issue?

[maxlandon]

Ok unfortunately since a few days I don't have access to either Windows or MacOS.
So I'm gonna have to guide you and ask you to perform a couple tests:

Could you give me the result of the following things (on Windows and/or MacOS) ?

• \command then Enter: what is line given to execute ?
• \ without enter: does the backslash appears in the line ?
• readline bind --remove "\<TAB> (don't forget the quote): Does this work for completion ? It does on my side, and that's normal: normally a backslash is an escape sequence, so it should be treated as such by the shell, unless quoted.

Although null byte response doesn't appear, it seems like there is something going on in the background -
autocomplete consumes something if you've pressed backslash, meaning entering a command after the fact
doesn't trigger the autocomplete.

Could you give me an example please ?

[maxlandon]

I've opened a new branch (backspace) on which you will see this commented line:

To explain, so when the line is split and the last character is an unterminated escape sequence, I used not to return it to the completion engine. I've commented the line so that now, the backspace is also returned.

Normally (but I'm not positive at all), this should solve your completion problem when a backspace is inserted.
Could you confirm/infirm by testing with the backspace branch ?

[maxlandon]

Hello @pygrum , please be advised that in a week, I won't be available to do anything on this repository for several months (I will be strictly and unavoidably away from a laptop).

Thus it would be great if we can solve this this week !
Thanks a lot for your attention

[maxlandon]

By the way, quite unrelated, but I looked at your monarch repo, and I thought you might be interested in https://github.com/reeflective/team: it's the extracted teamclient/teamserver code of Sliver.
It factors out everything teamserver/teamclient related, and gives a dedicated tree of commands to integrate in any application.

[pygrum]

Ok unfortunately since a few days I don't have access to either Windows or MacOS. So I'm gonna have to guide you and ask you to perform a couple tests:

Could you give me the result of the following things (on Windows and/or MacOS) ?

• \command then Enter: what is line given to execute ?
• \ without enter: does the backslash appears in the line ?
• readline bind --remove "\<TAB> (don't forget the quote): Does this work for completion ? It does on my side, and that's normal: normally a backslash is an escape sequence, so it should be treated as such by the shell, unless quoted.

Although null byte response doesn't appear, it seems like there is something going on in the background -
autocomplete consumes something if you've pressed backslash, meaning entering a command after the fact
doesn't trigger the autocomplete.

Could you give me an example please ?

I could've been a bit clearer here.

• When entering a backslash and then a command, the command runs - the backslash is ignored. Which is good! However, if I for example made a command called ls and typed \l and hit TAB, it wouldn't complete the 'ls' for me.
• When typing a backslash and hitting enter , nothing happens - as if I just hit enter after no input.

Somehow. MacOS and Windows builds completely ignore backslash in most contexts. Maybe there's an odd build constraint somewhere?

[pygrum]

Hello @pygrum , please be advised that in a week, I won't be available to do anything on this repository for several months (I will be strictly and unavoidably away from a laptop).

Thus it would be great if we can solve this this week ! Thanks a lot for your attention

I will be using the console for only linux builds, as I'd planned to eventually make a GUI for non-linux hosts. So no worries, this issue isn't exactly a blocker for me!

And https://github.com/reeflective/team looks awesome! I'd definitely try it out when I next plan to use a client-server CLI infrastructure.

I'll leave this issue open in case I do find anything while you are away. Thanks for your help with this

[maxlandon]

Ok unfortunately since a few days I don't have access to either Windows or MacOS. So I'm gonna have to guide you and ask you to perform a couple tests:
Could you give me the result of the following things (on Windows and/or MacOS) ?

• \command then Enter: what is line given to execute ?
• \ without enter: does the backslash appears in the line ?
• readline bind --remove "\<TAB> (don't forget the quote): Does this work for completion ? It does on my side, and that's normal: normally a backslash is an escape sequence, so it should be treated as such by the shell, unless quoted.

Although null byte response doesn't appear, it seems like there is something going on in the background -
autocomplete consumes something if you've pressed backslash, meaning entering a command after the fact
doesn't trigger the autocomplete.

Could you give me an example please ?

I could've been a bit clearer here.

* When entering a backslash and then a command, the command runs - the backslash is ignored. Which is good! However, if I for example made a command called `ls` and typed `\l` and hit TAB, it wouldn't complete the 'ls' for me.

* When typing a backslash and hitting enter , nothing happens - as if I just hit enter after no input.

Somehow. MacOS and Windows builds completely ignore backslash in most contexts. Maybe there's an odd build constraint somewhere?

• For the \ alone and when pressing Enter, nothing happens: actually what happens is that your cursor goes onto the beginning of the next line below: that's expected behavior. In your sh/bash/zsh shell, backlslash+Enter adds a newline and lets you keep writing your command. At exec time, this newline is removed and all lines are considered to be a single one.

• For \l not tab-completing ls: that's half normal, since before generating the completions, the shell splits the output with this package, which will consider the character just after a backslash as an escape sequence. I still sometimes find an inconsistent behavior in my own completion stuff,
  especially when starting completion on a new line (after entering \ + Enter, then <tab>), but I don't have the time nor a good idea on how to uniformly fix these inconsistencies.

I'm probably waiting for some time or someone who will jump in when they feel the need for this to be fixed.

[maxlandon]

Hello @pygrum, so when actually writing a summary of what remained to do, the problems to solve and so on, I decided to open a new dedicated issue, since apparently the null bytes on Windows are not a problem anymore, and that the discussion on backslash treatment concerns all platforms/users.

So closing this issue, but opened this one #43. If you need anything or if you have any ideas, feel free to share !

(Also, I deleted your message and my response to it about the crash when completing: this is solved, please update to v0.1.15)