Implicit rules with unvarying prerequisites produce surprising results

[cspiel]

Adding unvarying prerequisites to implicit rules produces surprising
results with OMake. A simple example is

    %.q: %.p header
            cat header $<  > $@

which states that

• every .q file depends on the matching .p file,
• any .q file depends on header, too, and
• header gets prepended to the .p file producing the associated .q file.

Obviously header does not contain a %-sign and therefore is an unvarying prerequisite.
See also Section "Introduction to Pattern Rules" in the GNU Make Manual.
AFAICT, the OMake manual does not explicitly mention unvarying
prerequisites in implicit rules.

Necessary files to reproduce the problem can be found in attachment.zip.
There, the script surprise.dash uses the exemplary rule in OMakefile.
The script name already gives it away; the
findings are surprising. Indeed, omake executes the rule body
with every change to foo.p as expected. However, it does not
reliably pick up changes of header. As soon as the unvarying
prerequisite's hash is stored in OMake's database it is considered
unchanged. Running omake with the -U-switch circumvents the
problem, but of course undermines the purpose of using OMake in the
first place.

A somewhat edited output of surprise.dash looks like this
(explanations below):

header> Initial header.
foo.p> This is our initial source.
*** omake: changing directory to ...
foo.q> Initial header.
foo.q> This is our initial source.

header> Initial header.
foo.p> This is an updated source.
*** omake: changing directory to ...
foo.q> Initial header.
foo.q> This is an updated source.

header> Updated header.
foo.p> This is an updated source.
*** omake: changing directory to ...
foo.q> Initial header.
foo.q> This is an updated source.

header> Updated header.
foo.p> This is a twice updated source.
*** omake: changing directory to ...
foo.q> Updated header.
foo.q> This is a twice updated source.

The four blocks are the output of four consecutive runs of omake.
Run 1: OK - first production.
Run 2: OK - updated source makes it into output.
Run 3: Surprise - header was changed, but the rule body has not been run again.
Run 4: OK - confirmation run; changing foo.p a second time triggers the implicit rule.

If we let surprise.dash run omake always with the -U-switch,
the output changes for Run 3 and we arrive at the following unified
diff:

 header> Updated header.
 foo.p> This is an updated source.
-foo.q> Initial header.
+foo.q> Updated header.
 foo.q> This is an updated source.

which means the quirkiness is gone.

Script surprise.dash also allows for running GNU Make (make) in
the same setting as omake, but the results there are not a
surprise, they are scary and more reason to prefer OMake to GNUMake.

[ANogin]

It appears to be some weird race condition between filesystem and omake. If I add anything like "sync", "sleep 1", etc right before the $MAKE_COMMAND in make_and_show, the issue goes away.

[cspiel]

@ANogin: I can confirm "weird" as diagnosis and "sleep 1" as fix but not "sync". LOL.

If I run surprise.dash in a loop sometimes (but rarely) the issue shows up even
without any delays introduced before $MAKE_COMMAND.

./surprise.dash  > reference.out  # fingers crossed
while true; do ./surprise.dash > test.out; diff -u reference.out test.out; done

Shouldn't the checksum-based OMake algorithm exactly be immune
against race conditions like this?

[gerdstolpmann]

yeah, in theory, but then you want it also to be fast, and you start skipping checks now and then... That problem could easily be rooted in the part that I rewrote to improve performance. In particular, the question is when to even look at files (because computing the checksum is expensive). I wonder what happens when the mtime of header is identical to the mtime of the last run. Could be that we simply assume then that the file did not change, without looking at the contents. (That's just a hypothesis.)

[ANogin]

@gerdstolpmann: I was actually testing with my old install of 0.9.8.6 RC1, so it's not your rewrite.
@cspiel: Omake does not redo checksums, unless the file stat data have changed. I have not gone through -debug logs carefully, but it could be that the race is because omake is seeing stale fstat somehow.