Windows issue fix

[pverkind]

Fix for issue #13

Proposed solution:

1. use quotation marks around "xxhash64(series) as gid" instead of escaping the spaces
2. use a placeholder "LESSTHAN" instead of the less than sign, which is not allowed with CALL, and replace it with the less than character "<" in seriatim.py

Possible improvements: replace less than, greater than and pipe in user-defined arguments with LESSTHAN, GREATERTHAN and PIPE.

[dasmiq]

I'm not entirely happy with the ad hoc symbol names in seriatim.py for just that argument. I think the problem is that, on Windows, passim.cmd calls the other command seriatim.cmd, so effectively you have to escape the escape characters to get it to work. What if you just copied the contents of seriatim.cmd to passim.cmd and then added the --all-pairs, --fields and --filterpairs arguments to it?

[pverkind]

I can see why you don't like that. I tried what you proposed now, but I don't seem to get it to work. The same problem seems to apply here.

When I copy the contents of seriatim.cmd into passim.cmd and change the last line to this, I get the same "The system cannot find the file specified" error as before:

spark-submit --repositories https://repos.spark-packages.org/ ^
  --packages graphframes:graphframes:0.8.0-spark3.0-s_2.12 ^
    %SPARK_SUBMIT_ARGS% %SCRIPT% %* --all-pairs --fields "xxhash64(series) as gid" -f gid<id2

(I tried also all the other combinations with caret escapes for the less than character and double quotation marks and backquotes)

But when I try with the LESSTHAN paramater, it works again:

spark-submit --repositories https://repos.spark-packages.org/ ^
  --packages graphframes:graphframes:0.8.0-spark3.0-s_2.12 ^
    %SPARK_SUBMIT_ARGS% %SCRIPT% %* --all-pairs --fields "xxhash64(series) as gid" -f gidLESSTHANid2

if you, @dasmiq or @mutherr , could give it a try and see if you can get it to work, that would be great.

[dasmiq]

Did you try "gid<gid2" or gid^<gid2 ? Can you escape it with backslash, e.g., "gid\<gid2" or gid\<gid2 ?

[pverkind]

I tried "gid^<gid2", "gid<gid2", gid^<gid2, gid<gid2, gid<gid2, gid<gid2 ; and because you asked, I now also tried gid<gid2 and "gid<gid2", all with the same effect

[pverkind]

Do I understand correctly that your problem with my earlier proposal is mostly that it works only for this one argument?
If so, it would perhaps be possible to do a replace operation for all offending characters (<, >, |) on all arguments in passim.cmd and seriatim.cmd, and then do the inverse replacement on all (relevant) arguments in seriatim.py?

(I looked into find and replace in Windows CMD, and it does not seem straightforward, though)

[pverkind]

I looked into the replace functionality within Windows batch scripts and it is not great; but I managed to add some code to passim.cmd that replaces all <, >, & and | characters with placeholders in all arguments, not just the built-in filterpairs argument.
I also added the reverse replacement in seriatim.py for the only other field I thought it was relevant for (more could be added of course).

Please ignore this if you think this is a horrible idea - but at least it makes passim work on Windows.