Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

A first pass at storage #160

Merged
merged 203 commits into from
Feb 17, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
203 commits
Select commit Hold shift + click to select a range
9b3735c
Expose the set of package identifiers for child nodes on a composite
liamhuber Jan 10, 2024
c4a457d
Add methods for interacting with `tinybase.storage.H5ioStorage`
liamhuber Jan 10, 2024
5de74c5
Give public-facing save and load methods
liamhuber Jan 10, 2024
57b06d4
Fail early on Workflow saves if any children are not from a package
liamhuber Jan 10, 2024
8606a63
Load discovered save files automatically
liamhuber Jan 10, 2024
b299973
:bug: use correct method name
liamhuber Jan 10, 2024
c1d107b
Save list not set
liamhuber Jan 10, 2024
87ae9f9
Format black
pyiron-runner Jan 10, 2024
583cb33
Use renamed method
liamhuber Jan 15, 2024
f377971
Clean the working directory attribute
liamhuber Jan 16, 2024
9f6153a
Use H5ioStorage directly
liamhuber Jan 16, 2024
f541fa9
Don't rely on `return` behaviour on `DirectoryObject.delete`
liamhuber Jan 18, 2024
b6b47ce
Revert the return behaviour on `DirectoryObject.delete`
liamhuber Jan 18, 2024
b55702c
Remove comment
liamhuber Jan 18, 2024
ff0463d
No magic strings
liamhuber Jan 18, 2024
6309516
Revert name changes on the DirectoryObject removing files method
liamhuber Jan 18, 2024
8edc5a0
Use FileObject directly
liamhuber Jan 18, 2024
404d876
Store child nodes under their own label, not in a substorage
liamhuber Jan 18, 2024
359c3a5
Stick the signals information into properties
liamhuber Jan 18, 2024
53b3d7e
Pull the same trick with the node class name
liamhuber Jan 18, 2024
e288383
Modify remaining storage fields to use existing class fields
liamhuber Jan 18, 2024
8951528
Refactor: slide
liamhuber Jan 18, 2024
7777c34
Remove serialization placeholders
liamhuber Jan 18, 2024
c0ae035
Merge branch 'main' into simple_storage
liamhuber Jan 18, 2024
49924fb
Format black
pyiron-runner Jan 18, 2024
a5bea04
Point storage to that node's particular storage spot
liamhuber Jan 18, 2024
64f6d50
Break out the storage file path as a property
liamhuber Jan 19, 2024
afc2ff1
Use the save file property
liamhuber Jan 19, 2024
dd3bb73
Refactor: extract method for cleaning empty working directory
liamhuber Jan 19, 2024
b00fd9e
Refactor: extract method for deleting storage
liamhuber Jan 19, 2024
18f1477
Simplify logic clauses (a little)
liamhuber Jan 19, 2024
61470aa
Simplify logic clauses (actually)
liamhuber Jan 19, 2024
39e943d
Add comment
liamhuber Jan 19, 2024
681ea49
Rely on children loading themselves at instantiation
liamhuber Jan 19, 2024
e9cac1e
Refactor: extract method for readability
liamhuber Jan 19, 2024
02b416f
Add comment
liamhuber Jan 19, 2024
00ef492
Refactor: extract methods for readability
liamhuber Jan 19, 2024
5388e19
Tidy the working directory after checking for storage contents
liamhuber Jan 19, 2024
032642a
Tidy the working directory recursively for composites
liamhuber Jan 19, 2024
eda307a
Clean up at the very end too
liamhuber Jan 19, 2024
af0313a
Document saving as an alpha feature
liamhuber Jan 19, 2024
38b4b45
Test saving and loading a node
liamhuber Jan 19, 2024
3e604c7
:bug: hack around the fact we're not inheriting from storable
liamhuber Jan 19, 2024
133c6d0
Add a macro to the test demo nodes
liamhuber Jan 19, 2024
0504f31
Reload macro children explicitly
liamhuber Jan 19, 2024
a342d34
Test reloading composed workflows
liamhuber Jan 19, 2024
e40dafa
Clean up after test
liamhuber Jan 20, 2024
f8dbeea
Update node package test to reflect new demo node
liamhuber Jan 20, 2024
1a5b953
Only allow saving from the root
liamhuber Jan 20, 2024
dfd8217
Format black
pyiron-runner Jan 20, 2024
f579d24
Merge branch 'main' into simple_storage
liamhuber Jan 20, 2024
59c20b3
Add storage dependencies
liamhuber Jan 20, 2024
9055671
[dependabot skip] Update env file
pyiron-runner Jan 20, 2024
23b2145
Add boto to componsate for contrib
liamhuber Jan 20, 2024
3f47117
Merge remote-tracking branch 'origin/simple_storage' into simple_storage
liamhuber Jan 20, 2024
6d85b1f
[dependabot skip] Update env file
pyiron-runner Jan 20, 2024
b95d385
Add a flag to save at the end of a run
liamhuber Jan 20, 2024
2a61527
Cast known bools as bools on load
liamhuber Jan 20, 2024
f3df416
Merge remote-tracking branch 'origin/simple_storage' into simple_storage
liamhuber Jan 20, 2024
1c9d541
Update README
liamhuber Jan 20, 2024
7f2df33
Update deepdive
liamhuber Jan 21, 2024
5e0097f
Propagate change to environment-notebooks.yml
liamhuber Jan 22, 2024
3b405fd
[dependabot skip] Update env file
pyiron-runner Jan 22, 2024
4118fae
Merge branch 'main' into simple_storage
liamhuber Jan 22, 2024
181c884
Accumulate run signals with scoped labels not objects
liamhuber Jan 22, 2024
5ca844e
Merge branch 'store_scoped_labels' into purge_channel_state
liamhuber Jan 22, 2024
ffae639
Require the callback to always be a method of the owning node
liamhuber Jan 24, 2024
0e66e03
Just store the name of the callback instead of the object
liamhuber Jan 24, 2024
3fa2419
Add tests
liamhuber Jan 24, 2024
b93f585
Fix tests where callback was not owned by the node
liamhuber Jan 24, 2024
8b71f45
Merge branch 'main' into simple_storage
liamhuber Jan 24, 2024
db4629f
Merge branch 'simple_storage' into callbacks_as_strings
liamhuber Jan 24, 2024
c7975f2
Format black
pyiron-runner Jan 24, 2024
bf3f506
:bug: Fix typo in __getstate__ return
liamhuber Jan 24, 2024
d3b1a23
Fix typo (missing words) in test message
liamhuber Jan 24, 2024
e449583
Have composite pass its entire self for remote execution
liamhuber Jan 24, 2024
dc09c82
Remove unused methods
liamhuber Jan 24, 2024
6685d7a
Format black
pyiron-runner Jan 24, 2024
d736912
Be consistent about copying and updating dict
liamhuber Jan 24, 2024
1ce79ce
Unparent local nodes before taking remote
liamhuber Jan 24, 2024
1f3893d
:bug: stop running
liamhuber Jan 24, 2024
2f6972a
Don't reciprocally store the owning node in channel state
liamhuber Jan 24, 2024
e0d1f46
Don't store the owner-scope value receiver on channels
liamhuber Jan 24, 2024
1d0047f
:bug: workflows don't need to re-parent data IO
liamhuber Jan 24, 2024
e2dcd5c
Don't store the connections on channels
liamhuber Jan 24, 2024
95727c6
Format black
pyiron-runner Jan 25, 2024
dbbae00
:bug: Don't override __get/setstate__ for `OutputData`
liamhuber Jan 25, 2024
bd26858
Use a singleton for NotData instead of the class
liamhuber Jan 25, 2024
f3c3909
Update references to `NotData` in docs and strings
liamhuber Jan 25, 2024
81c7089
Re-execute the deepdive
liamhuber Jan 25, 2024
2bbc61d
Parent decorated nodes to the module their function came from
liamhuber Jan 25, 2024
ced66f7
Use h5io as the default storage routine
liamhuber Jan 26, 2024
b1adbd4
Move tinybase requirements to optional
liamhuber Jan 26, 2024
e396668
Depend on a version of h5io that doesn't exist yet
liamhuber Jan 26, 2024
b4f479f
[dependabot skip] Update env file
pyiron-runner Jan 26, 2024
5ff8424
Test tinybase backend
liamhuber Jan 29, 2024
721d44d
:bug: Get the test passing by propagating storage mode flag
liamhuber Jan 29, 2024
0d50670
Format black
pyiron-runner Jan 29, 2024
9fffaf8
Refactor: break out a storage interface
liamhuber Jan 29, 2024
d2ec42e
Check for allowable back ends
liamhuber Jan 29, 2024
262bdb6
Refactor: rename
liamhuber Jan 29, 2024
f80d7ab
Refactor to delay contrib import
liamhuber Jan 29, 2024
7aec157
Update docstring
liamhuber Jan 29, 2024
b030571
h5io storage file should always be in the working directory
liamhuber Jan 29, 2024
c1ca1d6
Add tests for modified macro storage
liamhuber Jan 29, 2024
0477cf7
Test modified macro storage for the backends separately
liamhuber Jan 29, 2024
189c204
NOT_DATA should work like None in the if-statement
samwaseda Jan 29, 2024
1908914
add error message
samwaseda Jan 29, 2024
9de61c8
Update spec
liamhuber Jan 29, 2024
63c7e63
Test claims about saving workflows with children from different places
liamhuber Jan 29, 2024
dd7f06c
Update deepdive
liamhuber Jan 29, 2024
4199a33
Format black
pyiron-runner Jan 29, 2024
e5311c1
Format black
pyiron-runner Jan 29, 2024
871b9e1
Merge pull request #185 from pyiron/notdata_is_false
liamhuber Jan 29, 2024
d20b24c
Merge branch 'notdata_as_singleton' into parent_decorated_nodes_to_th…
liamhuber Jan 29, 2024
19e1528
Merge branch 'parent_decorated_nodes_to_their_module' into store_with…
liamhuber Jan 29, 2024
d9cfcba
Fix expected package length
liamhuber Jan 29, 2024
819a8d5
Beautify _no_positional_args logic
liamhuber Jan 30, 2024
a3f4a75
Merge pull request #177 from pyiron/hotfix_set_running_false
liamhuber Jan 30, 2024
e4b1595
Merge pull request #176 from pyiron/copy_and_update
liamhuber Jan 30, 2024
97eab58
Merge pull request #175 from pyiron/composites_pass_self
liamhuber Jan 30, 2024
6c401ab
Merge pull request #174 from pyiron/callbacks_as_strings
liamhuber Jan 30, 2024
abaf22b
Merge branch 'simple_storage' into nonreflexive_state
liamhuber Jan 30, 2024
55bd2ad
Merge branch 'nonreflexive_state' into notdata_as_singleton
liamhuber Jan 30, 2024
121fc84
Merge branch 'notdata_as_singleton' into parent_decorated_nodes_to_th…
liamhuber Jan 30, 2024
cbb480f
Merge branch 'parent_decorated_nodes_to_their_module' into store_with…
liamhuber Jan 30, 2024
0457a18
Merge pull request #178 from pyiron/nonreflexive_state
liamhuber Jan 30, 2024
d614523
Merge pull request #179 from pyiron/notdata_as_singleton
liamhuber Jan 30, 2024
2669574
Merge pull request #180 from pyiron/parent_decorated_nodes_to_their_m…
liamhuber Jan 30, 2024
600b00c
Merge pull request #181 from pyiron/store_with_h5io
liamhuber Jan 30, 2024
6af0a7a
Merge branch 'main' into simple_storage
liamhuber Jan 30, 2024
df0c617
Clean up the save file even if the test fails
liamhuber Jan 30, 2024
991765d
Flatten the state path so it matches the semantic path
liamhuber Jan 30, 2024
8ff5181
Add comments for the other state modifications
liamhuber Jan 30, 2024
636dcb0
Format black
pyiron-runner Jan 30, 2024
26620b2
Purge and restore starting nodes from the state
liamhuber Jan 30, 2024
ff65a32
Add promise about child name namespace uniqueness
liamhuber Jan 31, 2024
60de0b8
Clarify availability for future devs
liamhuber Jan 31, 2024
9e389cb
Guarantee storage key availability
liamhuber Jan 31, 2024
4df004b
Merge pull request #187 from pyiron/semantic_storage_path
liamhuber Jan 31, 2024
be6ead0
Merge branch 'simple_storage' into purge_starting_node_instances
liamhuber Jan 31, 2024
eb862b1
Guarantee storage key availability
liamhuber Jan 31, 2024
48141ac
:bug: hotfix generator to tuple conversion
liamhuber Jan 31, 2024
edac929
Merge pull request #188 from pyiron/purge_starting_node_instances
liamhuber Jan 31, 2024
21cd42e
Add and test wrappers for sticking nodes in a pyiron job
liamhuber Feb 1, 2024
d7c7801
Introduce and test an importability property
liamhuber Feb 6, 2024
b361523
Extend node package count
liamhuber Feb 6, 2024
bd388b1
Fail saving early if h5io won't be able to import your nodes
liamhuber Feb 6, 2024
8ced849
try...finally changing the classname
liamhuber Feb 7, 2024
6b0bf50
Slide var used in finally outside the try
liamhuber Feb 7, 2024
604a8ce
Fix line length
liamhuber Feb 7, 2024
6d0903a
Remove debug print
liamhuber Feb 7, 2024
519f176
Fix test message
liamhuber Feb 7, 2024
284b9ca
Protect tinybase from object type-import type mismatches
liamhuber Feb 7, 2024
9af564a
Update the storage docs on Node
liamhuber Feb 7, 2024
a59e01f
Add a convenience method for reporting importability
liamhuber Feb 7, 2024
1a99af7
Good catch by Niklas
liamhuber Feb 7, 2024
8736693
Return a string instead of printing
liamhuber Feb 7, 2024
88a0f8a
Add the report to the error message
liamhuber Feb 7, 2024
efafe97
Refactor: rename method
liamhuber Feb 7, 2024
51ec2e4
Add back a property on the old name
liamhuber Feb 7, 2024
acb19db
Merge pull request #198 from pyiron/patch_tinybase_safe_save_dynamics
liamhuber Feb 7, 2024
2687a70
Merge branch 'safe_save_dynamics' into safe_save_report
liamhuber Feb 7, 2024
5f581c0
Merge pull request #199 from pyiron/safe_save_report
liamhuber Feb 7, 2024
8c463dc
:bug: Fix a string typo that snuck in on the github web merger
liamhuber Feb 7, 2024
8bbaf32
Format black
pyiron-runner Feb 7, 2024
3612ca8
Merge pull request #197 from pyiron/safe_save_dynamics
liamhuber Feb 7, 2024
8f56a2b
Bump pyiron_contrib
liamhuber Feb 7, 2024
ef40062
Merge branch 'simple_storage' into as_pyiron_job
liamhuber Feb 7, 2024
e6e3b83
Update docstring examples
liamhuber Feb 7, 2024
9cc5fa5
Format black
pyiron-runner Feb 7, 2024
b06997e
Merge branch 'main' into simple_storage
liamhuber Feb 14, 2024
7ab6c72
Merge branch 'simple_storage' into as_pyiron_job
liamhuber Feb 14, 2024
bb2647b
Move contrib and h5io_browser to the main dependencies
liamhuber Feb 14, 2024
4a8eec2
[dependabot skip] Update env file
pyiron-runner Feb 14, 2024
83b7db4
Update h5io_browser dep
liamhuber Feb 14, 2024
0b65f3d
Make sure the tinybase storage directory exists before saving there
liamhuber Feb 14, 2024
9392a1e
Make available backend depend on python version
liamhuber Feb 14, 2024
d874c87
Force save_after_run to carry the backend
liamhuber Feb 14, 2024
b04dc8b
Make the backend an attribute and return save_after_run to bool
liamhuber Feb 14, 2024
22954eb
Update tests
liamhuber Feb 14, 2024
aebd835
Rerun notebooks
liamhuber Feb 14, 2024
2871a7b
[dependabot skip] Update env file
pyiron-runner Feb 14, 2024
e3c609d
Bumpy h5io_browser
liamhuber Feb 15, 2024
4561724
[dependabot skip] Update env file
pyiron-runner Feb 15, 2024
92a0b17
Merge pull request #189 from pyiron/as_pyiron_job
liamhuber Feb 15, 2024
394ccd4
Format black
pyiron-runner Feb 15, 2024
e685c84
Bump atomistics and contrib
liamhuber Feb 16, 2024
648cfdc
[dependabot skip] Update env file
pyiron-runner Feb 16, 2024
0370f52
Make NodeJob compliant with the storage interface
liamhuber Feb 16, 2024
84c326c
Wrap tests that use storage in a version check
liamhuber Feb 17, 2024
f831420
:bug: de-double and de-bug the node test
liamhuber Feb 17, 2024
6abc23a
Fail hard and clean when the python version is too low
liamhuber Feb 17, 2024
740e236
Don't even try loading if python <3.11
liamhuber Feb 17, 2024
8fbd070
:bug: Add missed skipIf
liamhuber Feb 17, 2024
dc1ea55
Add a warning to the node job tests
liamhuber Feb 17, 2024
e03ab93
Only test node job docs on valid python versions
liamhuber Feb 17, 2024
afc7d90
Add the version requirement right to the top of the docs about storage
liamhuber Feb 17, 2024
c7474f3
Add the version requirement right to the top of the docs about storage
liamhuber Feb 17, 2024
d21e85a
Merge remote-tracking branch 'origin/patch_jobs_in_storage' into patc…
liamhuber Feb 17, 2024
fabf465
Add version filtering to the deepdive
liamhuber Feb 17, 2024
3f0f8d9
Format black
pyiron-runner Feb 17, 2024
2a887dc
Merge pull request #209 from pyiron/patch_jobs_in_storage
liamhuber Feb 17, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion .binder/environment.yml
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,10 @@ dependencies:
- bidict =0.22.1
- cloudpickle =3.0.0
- graphviz =8.1.0
- h5io =0.2.2
- h5io_browser =0.0.9
- matplotlib =3.8.2
- pyiron_contrib =0.1.15
- pympipool =0.7.13
- python-graphviz =0.20.1
- toposort =1.10
Expand All @@ -15,6 +18,6 @@ dependencies:
- atomistics =0.1.23
- lammps
- phonopy =2.21.0
- pyiron_atomistics =0.4.14
- pyiron_atomistics =0.4.15
- pyiron-data =0.0.27
- numpy =1.26.4
2 changes: 1 addition & 1 deletion .ci_support/environment-notebooks.yml
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,6 @@ dependencies:
- atomistics =0.1.23
- lammps
- phonopy =2.21.0
- pyiron_atomistics =0.4.14
- pyiron_atomistics =0.4.15
- pyiron-data =0.0.27
- numpy =1.26.4
3 changes: 3 additions & 0 deletions .ci_support/environment.yml
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,10 @@ dependencies:
- bidict =0.22.1
- cloudpickle =3.0.0
- graphviz =8.1.0
- h5io =0.2.2
- h5io_browser =0.0.9
- matplotlib =3.8.2
- pyiron_contrib =0.1.15
- pympipool =0.7.13
- python-graphviz =0.20.1
- toposort =1.10
Expand Down
4 changes: 3 additions & 1 deletion docs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,9 @@ Individual node computations can be shipped off to parallel processes for scalab

Once you're happy with a workflow, it can be easily turned it into a macro for use in other workflows. This allows the clean construction of increasingly complex computation graphs by composing simpler graphs.

Nodes (including macros) can be stored in plain text, and registered by future workflows for easy access. This encourages and supports an ecosystem of useful nodes, so you don't need to re-invent the wheel. (This is an alpha-feature, with full support of [FAIR](https://en.wikipedia.org/wiki/FAIR_data) principles for node packages planned.)
Nodes (including macros) can be stored in plain text, and registered by future workflows for easy access. This encourages and supports an ecosystem of useful nodes, so you don't need to re-invent the wheel. (This is a beta-feature, with full support of [FAIR](https://en.wikipedia.org/wiki/FAIR_data) principles for node packages planned.)

Executed or partially-executed graphs can be stored to file, either by explicit call or automatically after running. When creating a new node(/macro/workflow), the working directory is automatically inspected for a save-file and the node will try to reload itself if one is found. (This is an alpha-feature, so it is currently only possible to save entire graphs at once and not individual nodes within a graph, all the child nodes in a saved graph must have been instantiated by `Workflow.create` (or equivalent, i.e. their code lives in a `.py` file that has been registered), and there are no safety rails to protect you from changing the node source code between saving and loading (which may cause errors/inconsistencies depending on the nature of the changes).)

## Example

Expand Down
3 changes: 3 additions & 0 deletions docs/environment.yml
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,10 @@ dependencies:
- bidict =0.22.1
- cloudpickle =3.0.0
- graphviz =8.1.0
- h5io =0.2.2
- h5io_browser =0.0.9
- matplotlib =3.8.2
- pyiron_contrib =0.1.15
- pympipool =0.7.13
- python-graphviz =0.20.1
- toposort =1.10
Expand Down
12 changes: 6 additions & 6 deletions notebooks/atomistics_nodes.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -91,18 +91,18 @@
"name": "stderr",
"output_type": "stream",
"text": [
"/Users/huber/work/pyiron/pyiron_workflow/pyiron_workflow/channels.py:166: UserWarning: The channel accumulate_and_run was not connected to ran, andthus could not disconnect from it.\n",
"/Users/huber/work/pyiron/pyiron_workflow/pyiron_workflow/channels.py:168: UserWarning: The channel ran was not connected to accumulate_and_run, andthus could not disconnect from it.\n",
" warn(\n",
"/Users/huber/work/pyiron/pyiron_workflow/pyiron_workflow/channels.py:166: UserWarning: The channel ran was not connected to accumulate_and_run, andthus could not disconnect from it.\n",
"/Users/huber/work/pyiron/pyiron_workflow/pyiron_workflow/channels.py:168: UserWarning: The channel accumulate_and_run was not connected to ran, andthus could not disconnect from it.\n",
" warn(\n",
"/Users/huber/work/pyiron/pyiron_workflow/pyiron_workflow/channels.py:166: UserWarning: The channel run was not connected to ran, andthus could not disconnect from it.\n",
"/Users/huber/work/pyiron/pyiron_workflow/pyiron_workflow/channels.py:168: UserWarning: The channel run was not connected to ran, andthus could not disconnect from it.\n",
" warn(\n"
]
},
{
"data": {
"text/plain": [
"<matplotlib.collections.PathCollection at 0x14bcc6250>"
"<matplotlib.collections.PathCollection at 0x1300e1350>"
]
},
"execution_count": 4,
Expand Down Expand Up @@ -168,7 +168,7 @@
{
"data": {
"text/plain": [
"<matplotlib.collections.PathCollection at 0x14bdc2f10>"
"<matplotlib.collections.PathCollection at 0x14027d1d0>"
]
},
"execution_count": 6,
Expand Down Expand Up @@ -207,7 +207,7 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.4"
"version": "3.11.7"
}
},
"nbformat": 4,
Expand Down
556 changes: 368 additions & 188 deletions notebooks/deepdive.ipynb

Large diffs are not rendered by default.

Loading
Loading