Files Paths are Not Properly Quoted in platform.txt

[Rybec]

arduino/arduino-ide#2602

I originally filed this issue with Arduino. The verbose compile output is below. You'll notice that under "Detecting libraries used..." and "Generating function prototypes", there is no quoting of the file paths, which is resulting in them being split into 3 separate arguments (due to Windows deciding that it needed to include spaces in my username during installation). This causes ash to fail. The result of this appears to be malformed arguments to SDCC, which appears to interpret an extra argument as an additional source file argument, causing it to fail.

According to the dev who responded on the arduino-ide issue linked to above, the problem is lack of proper argument quoting in platform.txt, which should be very easily fixed.

Here's the verbose compile output:

FQBN: CH55xDuino:mcs51:ch552:clock=16internal
Using board 'ch552' from platform in folder: C:\Users\Rybec Thalin Arethda\AppData\Local\Arduino15\packages\CH55xDuino\hardware\mcs51\0.0.25
Using core 'ch55xduino' from platform in folder: C:\Users\Rybec Thalin Arethda\AppData\Local\Arduino15\packages\CH55xDuino\hardware\mcs51\0.0.25

Detecting libraries used...
C:\Users\Rybec Thalin Arethda\AppData\Local\Arduino15\packages\CH55xDuino\tools\MCS51Tools\2023.10.10/win/busybox ash C:\Users\Rybec Thalin Arethda\AppData\Local\Arduino15\packages\CH55xDuino\tools\MCS51Tools\2023.10.10/wrapper/sdcc.sh C:\Users\Rybec Thalin Arethda\AppData\Local\Arduino15\packages\CH55xDuino\tools\sdcc\build.13407_4/bin/sdcc C:\Users\Rybec Thalin Arethda\AppData\Local\arduino\sketches\F28B92F05926EE5B7BA4053F3A9A7367\sketch\sketch_dec20a.ino.cpp nul re12 -c -Ddouble=float -DUSE_STDINT -D__PROG_TYPES_COMPAT__ --model-large --int-long-reent -E -MC -mmcs51 -DCH552 -DF_CPU=16000000L -DF_EXT_OSC=0L -DARDUINO=10607 -DARDUINO_ch55x -DARDUINO_ARCH_MCS51 -DEP0_ADDR=0 -DEP1_ADDR=10 -DEP2_ADDR=20 -IC:\Users\Rybec Thalin Arethda\AppData\Local\Arduino15\packages\CH55xDuino\hardware\mcs51\0.0.25\cores\ch55xduino -IC:\Users\Rybec Thalin Arethda\AppData\Local\Arduino15\packages\CH55xDuino\hardware\mcs51\0.0.25\variants\ch552 -IC:\Users\Rybec Thalin Arethda\AppData\Local\Arduino15\packages\CH55xDuino\tools\sdcc\build.13407_4/include
Generating function prototypes...
C:\Users\Rybec Thalin Arethda\AppData\Local\Arduino15\packages\CH55xDuino\tools\MCS51Tools\2023.10.10/win/busybox ash C:\Users\Rybec Thalin Arethda\AppData\Local\Arduino15\packages\CH55xDuino\tools\MCS51Tools\2023.10.10/wrapper/sdcc.sh C:\Users\Rybec Thalin Arethda\AppData\Local\Arduino15\packages\CH55xDuino\tools\sdcc\build.13407_4/bin/sdcc C:\Users\Rybec Thalin Arethda\AppData\Local\arduino\sketches\F28B92F05926EE5B7BA4053F3A9A7367\sketch\sketch_dec20a.ino.cpp C:\Users\Rybec Thalin Arethda\AppData\Local\Temp\1859949367\sketch_merged.cpp re12 -c -Ddouble=float -DUSE_STDINT -D__PROG_TYPES_COMPAT__ --model-large --int-long-reent -E -MC -mmcs51 -DCH552 -DF_CPU=16000000L -DF_EXT_OSC=0L -DARDUINO=10607 -DARDUINO_ch55x -DARDUINO_ARCH_MCS51 -DEP0_ADDR=0 -DEP1_ADDR=10 -DEP2_ADDR=20 -IC:\Users\Rybec Thalin Arethda\AppData\Local\Arduino15\packages\CH55xDuino\hardware\mcs51\0.0.25\cores\ch55xduino -IC:\Users\Rybec Thalin Arethda\AppData\Local\Arduino15\packages\CH55xDuino\hardware\mcs51\0.0.25\variants\ch552 -IC:\Users\Rybec Thalin Arethda\AppData\Local\Arduino15\packages\CH55xDuino\tools\sdcc\build.13407_4/include
ash: Thalin: unknown operand
ash: Thalin: unknown operand
C:\Users\Rybec Thalin Arethda\AppData\Local\Arduino15\packages\builtin\tools\ctags\5.8-arduino11/ctags -u --language-force=c++ -f - --c++-kinds=svpf --fields=KSTtzns --line-directives C:\Users\Rybec Thalin Arethda\AppData\Local\Temp\1859949367\sketch_merged.cpp
Compiling sketch...
"C:\\Users\\Rybec Thalin Arethda\\AppData\\Local\\Arduino15\\packages\\CH55xDuino\\tools\\MCS51Tools\\2023.10.10/win/busybox" ash "C:\\Users\\Rybec Thalin Arethda\\AppData\\Local\\Arduino15\\packages\\CH55xDuino\\tools\\MCS51Tools\\2023.10.10/wrapper/sdcc.sh" "C:\\Users\\Rybec Thalin Arethda\\AppData\\Local\\Arduino15\\packages\\CH55xDuino\\tools\\sdcc\\build.13407_4/bin/sdcc" "C:\\Users\\Rybec Thalin Arethda\\AppData\\Local\\arduino\\sketches\\F28B92F05926EE5B7BA4053F3A9A7367\\sketch\\sketch_dec20a.ino.cpp" "C:\\Users\\Rybec Thalin Arethda\\AppData\\Local\\arduino\\sketches\\F28B92F05926EE5B7BA4053F3A9A7367\\sketch\\sketch_dec20a.ino.cpp.o" re2 -MMD -c -Ddouble=float -DUSE_STDINT -D__PROG_TYPES_COMPAT__ --model-large --int-long-reent --less-pedantic -mmcs51 -DCH552 -DF_CPU=16000000L -DF_EXT_OSC=0L -DARDUINO=10607 -DARDUINO_ch55x -DARDUINO_ARCH_MCS51 -DEP0_ADDR=0 -DEP1_ADDR=10 -DEP2_ADDR=20 "-IC:\\Users\\Rybec Thalin Arethda\\AppData\\Local\\Arduino15\\packages\\CH55xDuino\\hardware\\mcs51\\0.0.25\\cores\\ch55xduino" "-IC:\\Users\\Rybec Thalin Arethda\\AppData\\Local\\Arduino15\\packages\\CH55xDuino\\hardware\\mcs51\\0.0.25\\variants\\ch552" "-IC:\\Users\\Rybec Thalin Arethda\\AppData\\Local\\Arduino15\\packages\\CH55xDuino\\tools\\sdcc\\build.13407_4/include"
sdcpp.exe: fatal error: too many input files
compilation terminated.
subprocess error 1
ash: Thalin: unknown operand
ash: Thalin: unknown operand

exit status 1

Compilation error: exit status 1

[DeqingSun]

I can add quote \" around $SRC and $OBJ in sdcc.sh to pass the path with space as a whole to SDCC.
And I did change SDCC to the snapshot sdcc-snapshot-x86_64-w64-mingw32-20250103-15197
And I tested:

C:\TEMP\testSpace>"C:\\TEMP\\testSpace\\folder space test\\tools\\sdcc\\build.13407_4/bin/sdcc" -MMD -c -Ddouble=float -DUSE_STDINT -D__PROG_TYPES_COMPAT__ --model-large --int-long-reent --less-pedantic -mmcs51 -DCH552 -DF_CPU=16000000L -DF_EXT_OSC=0L -DARDUINO=10819 -DARDUINO_ch55x -DARDUINO_ARCH_MCS51 -DUSER_USB_RAM=266 -IC:\\Users\\DSun\\AppData\\Local\\Arduino15\\packages\\CH55xDuino\\hardware\\mcs51\\0.0.25\\cores\\ch55xduino -IC:\\Users\\DSun\\AppData\\Local\\Arduino15\\packages\\CH55xDuino\\hardware\\mcs51\\0.0.25\\variants\\ch552 -IC:\\Users\\DSun\\AppData\\Local\\Arduino15\\packages\\CH55xDuino\\tools\\sdcc\\build.13407_4/include -x c --include dummy_variable_main.h "C:\\TEMP\\testSpace\\folder space test\\sketch_jan03a.ino.cpp" -o "C:\\TEMP\\testSpace\\folder space test\\sketch_jan03a.ino.cpp.o"
sdcpp.exe: fatal error: too many input files
compilation terminated.
subprocess error 1

There is still some issue with space that I can not help any further, because all spaces are protected with quote.

C:\TEMP\testSpace>"C:\\TEMP\\testSpace\\folderspacetest\\tools\\sdcc\\build.13407_4/bin/sdcc" -MMD -c -Ddouble=float -DUSE_STDINT -D__PROG_TYPES_COMPAT__ --model-large --int-long-reent --less-pedantic -mmcs51 -DCH552 -DF_CPU=16000000L -DF_EXT_OSC=0L -DARDUINO=10819 -DARDUINO_ch55x -DARDUINO_ARCH_MCS51 -DUSER_USB_RAM=266 -IC:\\Users\\DSun\\AppData\\Local\\Arduino15\\packages\\CH55xDuino\\hardware\\mcs51\\0.0.25\\cores\\ch55xduino -IC:\\Users\\DSun\\AppData\\Local\\Arduino15\\packages\\CH55xDuino\\hardware\\mcs51\\0.0.25\\variants\\ch552 -IC:\\Users\\DSun\\AppData\\Local\\Arduino15\\packages\\CH55xDuino\\tools\\sdcc\\build.13407_4/include -x c --include dummy_variable_main.h "C:\\TEMP\\testSpace\\folderspacetest\\sketch_jan03a.ino.cpp" -o "C:\\TEMP\\testSpace\\folderspacetest\\sketch_jan03a.ino.cpp.o"

But when I copy the folder and remove all the spaces, the compile just runs fine, I don't think I've set anything wrong.

So you need to go back to SDCC to ask what get wrong.

[Rybec]

I'll try to dig around some when I have time. I've spent some time thinking about this, and I think it is two separate bugs. I can't see how the two error messages from ash are related to the sdcpp error message, because the commands listed by Arduino are entirely unquoted, while the compile command appears to be properly quoted. I'm not familiar enough with the inner workings of Arduino to figure it out on my own, but I've looked at platform.txt, and it looks like things are properly quoted. Further, the actual compile command does seem to have everything properly quoted but is still failing, so either there's something strange going on in sdcc.sh, or the SDCC bug is not fully fixed.

I'll dig around some more when I have time. I suspect the ash errors are due to a bug in Arduino itself. I'll run this by the guys on the Arduino side and see if they are willing to consider this. The actual compile error could still be in the shell script, or in SDCC. I happen to have come across a document from Microsoft about how no one knows how to do argument quoting right. I haven't finished reading it, but maybe the answer is in there somewhere.

In my personal opinion, the biggest problem here is that Arduino itself does not allow the user to set a custom build location. That would be a trivial solution to this problem (and pre-2.0 versions of Arduino did allow this), but evidently they think they know better than the user, which this issue proves wrong.

Anyhow, rant aside, I'll see if I can figure out what is going on. At this point I've posted bug reports on SDCC, here, and on Arduino, so I might as well be liaison between projects, so far as I have time to do so!

[Rybec]

Ok, I'm looking through sdcc.sh, and I've come across some code looking for cp. This has led me to a question. I'm using Windows, and I have MinGW installed. This means that I do have cp installed (and rm), so the script should find cp, despite the fact that I'm using Windows. Is there any chance this could cause any issues?

[Rybec]

I think I've worked out the issue. In the script, you use the first few arguments $1-$4 explicitly. These seem to be correctly quoted. The rest of the arguments are used as-is, with $@. Some of these arguments are not correctly quoted, and they appear to be fed to the script that way by Arduino. Most of them are fine, but these three are not:

-IC:\Users\Rybec Thalin Arethda\AppData\Local\Arduino15\packages\CH55xDuino\hardware\mcs51\0.0.25\cores\ch55xduino
-IC:\Users\Rybec Thalin Arethda\AppData\Local\Arduino15\packages\CH55xDuino\hardware\mcs51\0.0.25\variants\ch552
-IC:\Users\Rybec Thalin Arethda\AppData\Local\Arduino15\packages\CH55xDuino\tools\sdcc\build.13407_4/include

Could this be solved by quoting all of the arguments in $@ in a for loop? No, I tried this, and it didn't work. My shell scripting skills are not currently good enough to isolate and quote just the -I arguments, but experimenting at the command line suggests that fully quoting just the -I arguments would work.

I've explained this on the Arduino bug report on this issue, in hopes of learning more about how this stuff is supposed to work and why Arduino is doing it the way it is doing it. I don't know if Arduino should be quoting those arguments, if there is something that can be done in platform.txt to handle it, or if it needs to be handled in the shell script, but maybe someone over there can help me figure it out. I suspect a for loop that goes through $@ building an argument list and quoting just the -I arguments would solve this, but my shell scripting skills aren't good enough to do that without putting significant time into learning shell scripting better. (I generally use Python for more complex stuff like this, but I'm not sure that would be viable here, as you would have to include Python in your distribution.)

Anyhow, I'll report back once I've gotten a response from Arduino.

[Rybec]

It looks like I was wrong. They are properly quoted when sent to the shell script, but the quoting is stripped as normal. This means either they need a second layer of escaped quoting coming into the script (which might be possible in platform.txt; I just asked about this in the Arduino issue), or the script itself has to reapply the quotes.

[Rybec]

Testing: arduino/arduino-ide#2602 (comment)

Quoting line 76 in platform.txt to look like this seems to work for the third -I argument, but the other two are contained in a different variable that appears to be provided by Arduino, so it doesn't help them, and I don't see any way of fixing those in platform.txt:

compiler.systemincludes="-I"{compiler.syslibs.stdlib.path}/include""

[DeqingSun]

At this moment I suspect SDCC more than Arduino because running commands in cmd:

C:\TEMP\testSpace>"C:\\TEMP\\testSpace\\folder space test\\tools\\sdcc\\build.13407_4/bin/sdcc" -MMD -c -Ddouble=float -DUSE_STDINT -D__PROG_TYPES_COMPAT__ --model-large --int-long-reent --less-pedantic -mmcs51 -DCH552 -DF_CPU=16000000L -DF_EXT_OSC=0L -DARDUINO=10819 -DARDUINO_ch55x -DARDUINO_ARCH_MCS51 -DUSER_USB_RAM=266 -IC:\\Users\\DSun\\AppData\\Local\\Arduino15\\packages\\CH55xDuino\\hardware\\mcs51\\0.0.25\\cores\\ch55xduino -IC:\\Users\\DSun\\AppData\\Local\\Arduino15\\packages\\CH55xDuino\\hardware\\mcs51\\0.0.25\\variants\\ch552 -IC:\\Users\\DSun\\AppData\\Local\\Arduino15\\packages\\CH55xDuino\\tools\\sdcc\\build.13407_4/include -x c --include dummy_variable_main.h "C:\\TEMP\\testSpace\\folder space test\\sketch_jan03a.ino.cpp" -o "C:\\TEMP\\testSpace\\folder space test\\sketch_jan03a.ino.cpp.o"
sdcpp.exe: fatal error: too many input files
compilation terminated.
subprocess error 1

Does not work, but it will work after I remove the space.

[Rybec]

I'll look at that. My normal local SDCC isn't installed in a path with a space in it, so I haven't really tested that. I have an SDCC project outside of Arduino that I've been using for command line testing. I'll see if I can alter my makefile to use the ch55xdino installed SDCC that is on a path with a space. If I run into similar issues, and they only occur on the SDCC in the path with a space, I'll file another bug report with SDCC.

I'm pretty sure it's not just SDCC though. The -I arguments are not properly quoted coming out of the script. I'll figure out a for loop that adds quoting to just the -I arguments after I run through the tests on my non-Arduino project with the SDCC in Arduino. (The version of SDCC is the same for both, because I just copied the files from my local install over the ch55xduino ones. The only difference is location. The ch55xduino one has a space in the path, while the local install does not.)

[DeqingSun]

I haven't got to the -I part yet. I think the SDCC need to fix "sdcpp.exe: fatal error: too many input files" before the bug can be fully solved.

[Rybec]

Yeah, your command doesn't have spaces in the -I elements. The error message will be the same for the -I elements with spaces and no quotes though, because it interprets the parts after the spaces as additional input files.

That issue looks like the one I was seeing before I updated SDCC to the latest snapshot... I'm using 15177. Yours is slightly more recent. I'll try replacing my ch55xduino SDCC with that and see what happens.

Are you using the 32-bit or 64-bit version of that snapshot?

[DeqingSun]

64bit

[Rybec]

Well darn. I tried to reproduce your error with the following command, but instead it compiled without any problem.

The version you have in your test setup is 15197? The following is using the 64-bit build of that version (I just doubled checked with -v).

"C:\\Users\\Rybec Thalin Arethda\\rybec\\space_test\\test\\space folder\\tools\\sdcc\\build.13407_4/bin/sdcc" -MMD -c -Ddouble=float -DUSE_STDINT -D__PROG_TYPES_COMPAT__ --model-large --int-long-reent --less-pedantic -mmcs51 -DCH552 -DF_CPU=16000000L -DF_EXT_OSC=0L -DARDUINO=10819 -DARDUNIO_ch55x -DARDUINO_ARCH_MCS51 -DUSER_USB_RAM=266 "-IC:\\Users\\Rybec Thalin Arethda\\AppData\\Local\\Arduino15\\packages\\CH55xDuino\\hardware\\mcs51\\0.0.25\\cores\\ch55xduino" "-IC:\\Users\\Rybec Thalin Arethda\\AppData\\Local\\Arduino15\\packages\\CH55xDuino\\hardware\\mcs51\\0.0.25\\variants\\ch552" "-IC:\\Users\\Rybec Thalin Arethda\\AppData\\Local\\Arduino15\\packages\\CH55xDuino\\tools\\sdcc\\build.13407_4/include" main.c -o main.o

I normally work in Git Bash, but I did this with CMD, since it looks like that is what you are using.

[Rybec]

Hmm, I removed my local SDCC install from the path, to ensure it wasn't trying to use something from that, and now I get this error:

sdcpp.exe: fatal error: cannot execute 'cc1': CreateProcess: No such file or directory

The zip file with version 15197 does contain an executable named cc1.exe, but it's not in bin\. It's in libexec\sdcc\x86_64-w64-mingw32\12.1.0\. Copying it into the directory with the rest of the compiler tools solves this, and now it is back to compiling without any errors.

I don't know. I can't reproduce your issue, but when I quote the -I arguments on mine, it eliminates the problem I'm having and lets it compile fine (at least, from CMD).

I'll see if I can put together a for loop to quote the -I arguments and drop it here so we can eliminate that problem. If after doing that I still have issues like what you are having, I'll see if I can help with that. Otherwise, if I can't reproduce it, I'm not sure how I can help.

[DeqingSun]

If you find a solution, you may share the edit you've done with a PR and I'll test to see if it works without space.

Then the next time I update the compiler it will be fully fixed.

[Rybec]

Well, that doesn't work either, because it crashes the Detecting libraries used... phase.

I have no clue how to fix this at this point. If the -I arguments aren't quoted, SDCC can't handle it. If they are quoted, the process fails at the Detecting libraries used... phase, because somehow quoting them in the script causes them to be interpreted as separate arguments during that phase? I don't even know if that problem is something in ch55xduino or in Arduino itself.

I wonder...

Here's what I've done in the script:

I added this:

ARGS=
for arg in "$@"
do
	if [[ "${arg:0:2}" == "-I" ]]; then
		ARGS="$ARGS -I\"${arg:2}\""
	else
		ARGS="$ARGS $arg"
	fi
done

I changed the first compile line from this:

"$SDCC" "$@" -x c --include dummy_variable_main.h "$SRC" -o "$OBJ"

...to this:

"$SDCC" $ARGS -x c --include dummy_variable_main.h "$SRC" -o "$OBJ"

I changed the second compile line from this:

"$SDCC" "$@" "$SRC" -o "$OBJ"

...to this:

"$SDCC" $ARGS "$SRC" -o "$OBJ"

Notice that the $@ is quoted, but the $ARGS is not. As I understand it, the reason it failed when I quoted $ARGS is that $@ is an array of strings, while $ARGS is just a single string. Now, I tried to make $ARGS an array, but Bash arrays work differently from Ash arrays, and I don't much shell experience in general, and none with Ash, so I was unable to do that.

So I'm wondering: If I can figure out how to make $ARGS an array of arguments, maybe the Detecting libraries used... phase will correctly interpret the quoted -I arguments as independent arguments that are quoted, instead of as part of a single string of space delimited arguments.

Do you know Ash well enough to figure out what I need to do to make that for loop generate an array of strings rather than a single string?

[DeqingSun]

not really, the script was originally done by https://github.com/tenbaht/sduino

[Rybec]

Alright, I'll try to do more digging when I have time. I tried to figure it out yesterday, but it's hard to find anything on the topic that isn't for Bash.