[CDT] gdbserver not found

[mcoufal]

I can't make debugger to work, I always end up with an error that command 'gdbserver' is not found.

How to reproduce:

1. create simple Hello World C++ Project with Linux GCC toolchain
2. build project
3. set a line breakpoint somewhere in main
4. debug as Local C/C++ Application

Console view now shows an error: '/bin/sh: gdbserver: command not found'.
Might be connected to [https://github.com//issues/35](issue 35).

My configuration:
Eclipse SDK
Version: 2021-06 (4.20)
Build id: I20210601-0600
Java runtime version: 15.0.2+7-27
Eclipse C/C++ Development Tools 10.3.0.202105311920

[vrubezhny]

When using a build made of PR #45 I see that gdbserver executable now can be found, but process still cannot be normally started.

I found that the command line that is used to launch gdbserver does look like:

flatpak-spawn -v --host --watch-bus /bin/sh -c gdbserver :2345 /home/user/projects/workspace/cdtCppTest/Debug/cdtCppTest

when starting a debugger ( Debug as -> C/C++ Application) it tries to execute the command line above and finally fails with a TimeoutException:

org.eclipse.core.runtime.CoreException: Timed out trying to launch GDB.
	at org.eclipse.cdt.dsf.concurrent.Sequence.abortExecution(Sequence.java:599)
	at org.eclipse.cdt.dsf.concurrent.Sequence$2.handleErrorOrWarning(Sequence.java:448)
	at org.eclipse.cdt.dsf.concurrent.RequestMonitor.handleFailure(RequestMonitor.java:433)
	at org.eclipse.cdt.dsf.concurrent.RequestMonitor.handleCompleted(RequestMonitor.java:393)
	at org.eclipse.cdt.dsf.concurrent.RequestMonitor$2.run(RequestMonitor.java:317)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
	at java.base/java.lang.Thread.run(Thread.java:831)

Normally, when a gdbserver process is started it outputs a text like the following:

Process /home/user/projects/workspace/cdtCppTest/Debug/cdtCppTest created; pid = 999092
Listening on port 2345

So I see a text like this in the console, but then it looks like nothing more is being communicated between server process and CDT GDBBackend. At least it looks like this.

[vrubezhny]

Probably, the real reason is in flatpak-spawn'ed gdb process:

At startup, GDB reads the following init files and executes their commands:
   * system-wide init files: /etc/gdbinit

The host /etc directory is not available inside a sandbox (it has its own '/etc' directory, so flatpak-spawn'ed gdb process has no access to host's /etc/gdbinit file.

[vrubezhny]

@jjohnstn @akurtakov
The problem with Flatpak CDT Debug is somewhere in flatpak-spawn'ed-GDB-to-GDBBackend communication... GDB id flatpak-spawned process with --interpreter mi2 argument and -ex "new-ui mi /dev/pts/1" command sent to it. It's started like this:

flatpak-spawn --verbose  --host --watch-bus  gdb --interpreter mi2  --nx  -q  --interpreter console -ex "new-ui mi /dev/pts/1" -ex "set pagination off" -ex "show version"

"/dev/pts/1" here is a PTY name that GDBBackend(GDBBackend_7_12) is created to listen for mi2 interpreter output and [probably] to send commands to.

When debug starts, GDBBackend tries to read all lines from "/dev/pts/1" until it received (dgb) prompt. And here it get stuck on inputReader.readLine() - it never receives any lines (even chars) from this stream.

When I manually start the similar commands (but replacing "/dev/pts/1" by a real terminal name), like:

• In terminal 1 I execute tty command to get know the terminal name:

$  tty
/dev/pts/14

• then in terminal 2 I execute (putting the /dev/pts/14 name received at the previous step):

flatpak-spawn --verbose  --host --watch-bus  gdb --interpreter mi2  --nx  -q  --interpreter console -ex "new-ui mi /dev/pts/14" -ex "set pagination off" -ex "show version"

In terminal 1 I start receiving the GDB prompt and the mi2 interpreter echo/log of the commands I run from the terminal 2, like:

 =thread-group-added,id="i1"
(gdb) 
=cmd-param-changed,param="pagination",value="off"
=cmd-param-changed,param="pagination",value="on"
=cmd-param-changed,param="pagination",value="off"

So, it looks like mi2 interpreter's output really exists (and I can read it by a terminal that I specified as the command argument).

But when I start CDT debug - mi2 interpreter output is never received... Everything gets stuck on inputReader.readLine() at: https://git.eclipse.org/c/cdt/org.eclipse.cdt.git/tree/dsf-gdb/org.eclipse.cdt.dsf.gdb/src/org/eclipse/cdt/dsf/gdb/service/GDBBackend.java?id=05c45c0bae9c85b917309712306aadee052f3d8f#n568

I tried to read flatpak-spawn'ed GDB process output - and it receives all the DGB console output (including the required prompt (gdb) but,off course, not in mi2 interpreter's format (so it cannot be used further in the debug session).

@jjohnstn @akurtakov Any thoughts on how this can be fixed? Maybe we should fix some terminal setting when we create it or I don't know.. some flatpak-spawn argument is to be set to make it flash the streams (however I don't think it controls that stream anyhow)?

PS: In order to run Flatpak CDT debugger I've installed the build made of PR #45 (needed to make gdbserver available on sandbox) and then executed the following commands to link Host's directories on sandbox'es /etc directory (needed to make GDB init configurations available on sandbox) and to start org.eclipse.Java:

$ flatpak run --command=sh org.eclipse.Java
[📦 org.eclipse.Java ~]$ ln -s /var/run/host/etc/gdbinit /etc
[📦 org.eclipse.Java ~]$ ln -s /var/run/host/etc/gdbinit.d /etc
[📦 org.eclipse.Java ~]$ /app/bin/eclipse  &

[vrubezhny]

I'm not sure, maybe I did a wrong experiment, but IMHO the problem is that Flatpak CDT GDBBackend (running under the sandbox) tries to read a stream that is not sandbox'ed....

flatpak-spawn is a kind of portal that runs a not-sandbox'ed application and re-transmits its input/output/error streams to/from other sandbox'ed application. But all the other custom streams opened by such application, like the one opened directly from GDB for mi2 interpreter, are blocked and cannot reach sandbox'ed clients - they can be read/written only outside of sandbox (like I did in previous test with terminal).

To demo it I did almost the same steps I did in previous comment (#36 (comment)) but in first step I've used a terminal opened in flatpak:

• In terminal 1 I executed tty command in sandbox' sh to get know the terminal name:

$ flatpak run --command=sh org.eclipse.Java
[📦 org.eclipse.Java ~]$ tty
/dev/console

• then when in terminal 2 I executed (putting the /dev/console name received at the previous step):

flatpak-spawn --verbose  --host --watch-bus  gdb --interpreter mi2  --nx  -q  --interpreter console -ex "new-ui mi /dev/console" -ex "set pagination off" -ex "show version"

I did not received any (gdb) prompts nor any other output - looks exactly as I see it when I run CDT debug.

So, we need to find out a way to read/write a stream created by non-sandbox'ed application to/from a sandbox'ed one.

[vrubezhny]

There is some progress with build https://flathub.org/builds/#/builders/32/builds/53062, at least now we can start gdbserver and gdb and read the messages from gdb - not just hanging out on Mi-command reading until timeout happens.
But still it's not possible to debug due to the following error:

Error in final launch sequence:

Failed to execute MI command:
-target-select remote localhost:2345/tcp
Error message from debugger back end:
localhost:2345/tcp: No such file or directory.

[jjohnstn]

Hi Victor, I am running an old Flatpak 2020-09 SDK with CDT installed from the matching release repo and using my local gdb 7.6.1 and C/C++ debugging is working. The gdbserver gets kicked off and gdb appears to talk to it fine. There have been some mi issues reported lately in cdt-dev but they were for mac. Perhaps the problem is really tied to newest versions of gdb and/or CDT. My program is a simple hello world program that I created in my workspace but I also tested with a program that does a getchar() and that works as well.

…

-- Jeff J.

On Fri, Jul 9, 2021 at 12:02 PM Victor V. Rubezhny ***@***.***> wrote: @jjohnstn <https://github.com/jjohnstn> @akurtakov <https://github.com/akurtakov> The problem with Flatpak CDT Debug is somewhere in flatpak-spawn'ed-GDB-to-GDBBackend communication... GDB id flatpak-spawned process with --interpreter mi2 argument and -ex "new-ui mi /dev/pts/1" command sent to it. It's started like this: flatpak-spawn --verbose --host --watch-bus gdb --interpreter mi2 --nx -q --interpreter console -ex "new-ui mi /dev/pts/1" -ex "set pagination off" -ex "show version" "/dev/pts/1" here is a PTY name that GDBBackend(GDBBackend_7_12) is created to listen for mi2 interpreter output and [probably] to send commands to. When debug starts, GDBBackend tries to read all lines from "/dev/pts/1" until it received (dgb) prompt. And here it get stuck on inputReader.readLine() - it never receives any lines (even chars) from this stream. When I manually start the similar commands (but replacing "/dev/pts/1" by a real terminal name), like: - In terminal 1 I execute tty command to get know the terminal name: $ tty /dev/pts/14 - then in terminal 2 I execute (putting the /dev/pts/14 name received at the previous ster): flatpak-spawn --verbose --host --watch-bus gdb --interpreter mi2 --nx -q --interpreter console -ex "new-ui mi /dev/pts/14" -ex "set pagination off" -ex "show version" In terminal 1 I start receiving the GDB prompt and the mi2 interpreter echo/log of the commands I run from the terminal 2, like: =thread-group-added,id="i1" (gdb) =cmd-param-changed,param="pagination",value="off" =cmd-param-changed,param="pagination",value="on" =cmd-param-changed,param="pagination",value="off" So, it looks like mi2 interpreter's output really exists (and I can read it by a terminal that I specified as the command argument). But when I start CDT debug - mi2 interpreter output is never received... Everything gets stuck on inputReader.readLine() at: https://git.eclipse.org/c/cdt/org.eclipse.cdt.git/tree/dsf-gdb/org.eclipse.cdt.dsf.gdb/src/org/eclipse/cdt/dsf/gdb/service/GDBBackend.java?id=05c45c0bae9c85b917309712306aadee052f3d8f#n568 I tried to read flatpak-spawn'ed GDB process output - and it receives all the DGB console output (including the required prompt (gdb) but,off course, not in mi2 interpreter's format (so it cannot be used further in the debug session). Any thoughts on how this can be fixed? Maybe we should fix some terminal setting when we create it or I don't know.. some flatpak-spawn argument is to be set to make it flash the streams (however I don't think it controls that stream anyhow)? PS: In order to run Flatpak CDT debugger I've installed the build made of PR #45 <#45> (needed to make gdbserver available on sandbox) and then executed the following commands to link Host's directories on sandbox'es /etc directory (needed to make GDB init configurations available on sandbox) and to start org.eclipse.Java: $ flatpak run --command=sh org.eclipse.Java [📦 org.eclipse.Java ~]$ ln -s /var/run/host/etc/gdbinit /etc [📦 org.eclipse.Java ~]$ ln -s /var/run/host/etc/gdbinit.d /etc [📦 org.eclipse.Java ~]$ /app/bin/eclipse & — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#36 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAJJC3P626VCOPJDMSSNATDTW4MQNANCNFSM456XX3BQ> .

[vrubezhny]

@jjohnstn Yes, off course, the newer versions could be a reason.
I'm currently on:

• Flatpak 1.10.2
• Flatpak Eclipse 2021-06 (4.20.0), Build id: 20210612-2011
• GNU gdb/gdbserver (GDB) Fedora 10.1-4.fc33

Yet another problem s that (probably in case of problem on starting debugger) I have gdbserver running left in the system (until I restart eclipse or kill it), so every second debugger start ends with:

Can't bind address: Address already in use.
Exiting

Not sure if it needs to be loaded all the time (especially given that fact that it's given with the path to executable to debug at start) - but we're launching it on every debugger launch (and yes - that port number is blocked after the first attempt). But it probably another error.

[jjohnstn]

FWIW, multiple debugs work fine in my scenario as well. I can debug again and again and the gdbserver exits each time.

…

On Fri, Jul 9, 2021 at 8:33 PM Victor V. Rubezhny ***@***.***> wrote: @jjohnstn <https://github.com/jjohnstn> Yes, off course, the newer versions could be a reason. I'm currently on: - Flatpak Eclipse 2021-06 (4.20.0), Build id: 20210612-2011 - GNU gdb/gdbserver (GDB) Fedora 10.1-4.fc33 Yet another problem s that (probably in case of problem on starting debugger) I have gdbserver running left in the system (until I restart eclipse or kill it), so every second debugger start ends with: Can't bind address: Address already in use. Exiting Not sure if it needs to be loaded all the time (especially given that fact that it's given with the path to executable to debug at start) - but we're launching it on every debugger launch (and yes - that port number is blocked after the first attempt). But it probably another error. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#36 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAJJC3JCTC4T2ZMUKGOMYULTW6IMVANCNFSM456XX3BQ> .

[jjohnstn]

After discussing with Jonah, the problem appears to be that CDT will use a full gdb console if the level of gdb supports it (which started in 7.12). There is a CDT call isFullGdbConsoleSupported() which when it returns true: there are two PTYs that connect GDB and Eclipse CDT, one for MI and one for CLI. When !“full gdb” only a single PTY connects GDB and CDT and all that communication is MI. The CLI in that case is wrapped in MI commands. It may be possible to patch the code to always return false in the case of flatpak. I will look into creating such a patch.