Input existence checking

[zhengp0]

Description

Stop checking input existence if the input is part of the output from upstream stage that hasn't been produced yet.

Please let me know if I misunderstood something here.

[kels271828]

Were you getting errors?

The way I wrote it, the arg upstream_stages are the stages that you DO want to check (see description in docstring:

OneMod/src/onemod/io/base.py

Line 155 in aa648a9

upstream_stages : set of str, optional

)

When the function gets called in Pipeline.evaluate(), it only passes the stages that aren't being run (

OneMod/src/onemod/pipeline.py

Line 346 in aa648a9

dep for dep in stage.dependencies if dep not in stages

).

I also added check_exists() to Stage.evaluate() (

OneMod/src/onemod/stage/base.py

Line 262 in aa648a9

self.input.check_exists()

) and ModelStage.evaluate() (

OneMod/src/onemod/stage/base.py

Line 561 in aa648a9

self.input.check_exists()

). This is redundant if you've called Pipeline.evaluate(), so we can check the logic of when we check stage input.

The main reasons to have the check_exists() in the first place is to 1) have more informative error messages, and 2) to fail before running or scheduling anything.