Improved start.jar dry-run command line quoting

[gregw]

Use shell double quoting for arguments.
Replaces #9999 for jetty/jetty.docker#148

[joakime]

The list of characters for bash (haven't looked at other shells yet) that have special meaning ...

Character	Meaning
~	home directory
`	Single quote for command substitution
#	comment
$	variable
&	background
*	string wildcard
(	start substitution
)	end substitution
\	quote next char
|	pipe to other command
[	start character-set wildcard
]	end character-set wildcard
{	start command block
}	end command block
;	shell command separator
'	strong quote
"	weak quote
<	input redirection
>	output redirection
/	pathname separator
?	single character wildcard
!	pipeline logical not operator

Each of those should trigger quoting and in some cases escaping

[gregw]

@joakime I think we don't need every special character, we just need metacharacters that are used to split a command line and characters involved with quoting itself.....
Or do we need to also protect wild cards? Hmmm probably yes....

[gregw]

I wonder if it would be simpler to have a set of characters that we don't need to quote for: a-z A-Z 0-9 _-/

[joakime]

Here's a longer breakdown of special characters for the shells: csh, sh, bash, ksh, zsh

Character(s)	Shells	Meaning
~	sh, csh, bash, ksh, zsh	home directory
`	bash	Single quote for command substitution
#	csh, sh, bash	comment
#	ksh, bash	variable replacement
^	sh only	pipe character
^	csh, bash	previous command
$	csh, sh, bash, ksh, zsh	variable
&	csh, sh, bash, ksh, zsh	background job. or combine stdin and stdout
*	csh, sh, bash	string wildcard (match zero or more characters)
( )	bash, ksh	command substitution
\	csh, sh, bash, ksh, zsh	quote next char
|	csh, sh, bash, ksh	pipe to other command
|	zsh	job control
[ ]	csh, sh, bash, ksh	character-set wildcard (match range of characters)
{ }	csh, bash	in-line expansion
{ }	csh, bash, ksh, zsh	command block
:	sh	evaluate args. or separate values in paths.
:	csh	variable modifier
;	csh, sh, bash, ksh, zsh	shell command separator
;	sh	end of case element
'	bash	strong quote
"	csh, sh, bash	weak quote
<	csh, sh, bash, ksh, zsh	input redirection
>	csh, sh, bash, ksh, zsh	output redirection
/	csh, sh, bash, ksh, zsh	pathname separator
-	sh, bash, ksh, zsh	stdin or stdout reference
?	csh, sh, bash	single character wildcard
?	sh, bash	optional variable
!	csh, bash	pipeline logical not operator
!	ksh, zsh	exit status

[joakime]

I wonder if it would be simpler to have a set of characters that we don't need to quote for: a-z A-Z 0-9 _-/

I wouldn't include - (dash) or / (slash) in that list.

The - (dash) is used as stdin or stdout reference in many shells.
And the / (slash) is used as a hard path separator in some shells (forces the shell to intepret the text around the / in path terms, applying path behaviors to that string), and a soft separator in others (if present, the matching logic applies to the strings adjacent too)

[joakime]

For Windows, the rules are a bit different ...

You can only use the \ back-slash to escape non-control characters when executing in CMD (such as a space character).

In Powershell, the escaping is done with ^ character.

We will likely need separate quoting and escaping methods for Unix/Linux/OSX and Windows. (Maybe even a special method for those users on Powershell)

[gregw]

I think this can only really be output for unix shells, and then only the common ones. It is a tool for human debugging and for doing start scripts.
If we keep the output constant, then others can transform to any special usages

[gregw]

I've updated the usage.txt file to make it clear that the output is only suitable for the unix sh command. I do not think we can come up with an output format that will work on every shell on every platform. I also think that any attempt to cover multiple platforms is just going to result in hard to handle variable output. Best to just give fixed output and allows any users on other platforms either to use a sh port or do a transformation.

[gregw]

@joakime nudge

[joakime]

I don't think supporting only unix sh will work.

The complexity of some of our properties, as pointed out in #9309, means that you will have to address ALL of the special characters and how they interact with the various shells (as any one of them can appear in a property), quoting alone isn't enough, you'll also have to escape, and escaping is different on sh (rare anymore) vs bash (super common) vs zsh (default on OSX).

To break down this issue in a different way.

• The lions share of issues with quoting is with our properties. (which is difficult)
• Excluding properties, the rest of our quoting is limited to classpaths, and XML file references (which is easy)

So I question, do we need to pass properties "in the raw" on the command line?
I don't think so.

Hypothetical solution:

The --dry-run can have the ability to save all properties to a <name>.properties file (configurable location + name).
Then the users of the output can just reference that location, as our existing XmlConfiguration supports reading properties from a properties file (if passed in as a command line argument)

Perhaps something like --dry-run --properties-file=/var/run/jetty.properties
And then usage like java -cp <classpath> <jvm-args> org.eclipse.jetty.xml.XmlConfiguration /var/run/jetty.properties <xml-files> to be used.

WDYT?

BTW, This properties file approach is how the non-dry-run, forked JVM is executed now.

[joakime]

If we go this --dry-run --properties-file=<name> approach, then the xargs stuff done in the jetty.sh can be reverted too.

[gregw]

@joakime it is not just properties. It is any file, including possibly the location of a dynamic property for that might need to be quoted

We have previously only ever supported Unix sh, we just never said as much. Almost every fancy Unix shell supports normal sh quoting, so I think this is fine. We have never made any attempt to support windows shell

At the very least, this PR is better that what we currently have and should fix the problems we have with docker whilst not being a too big of a change (as this actually should be done in jetty 10/11/12).

This represents a minimal change to our existing behavior that correctly quotes when needed.

Let me rebase into 10...

[gregw]

@joakime I have backported this to 10 in #10099

We got by for decades without supporting any other shell than sh, mostly be not using any special shell features. Then we added quoting and we've been having problems:

• quoting added was single quote with escape, which is not any shell standard
• we always quote, so any quote issues will always be hit
• we quote entire args, which is less readable than quoting just names and/or values

So this PR and #10099 address this by:

• Applying sh double quoting as specified, which is the minimum common denominator of most shells
• Only quoting when we need to, using a simple set of safe characters to determine that
• quote names and/or values separately

So for most usages, our output now will be no different to before we added quoting. For deployments with special properties or directories with strange names, then quoting will be used minimally only where necessary.

This is a minimal change to long standing behaviour. I do not think we should be opening the can of worms that supporting arbitrary shells on arbitrary OS's would require.

[gregw]

This also fixes #10056

[sbordet]

Closing because #10099 has been merged forward, and deprecated methods removed during the merge to 12.