Notebooks env patch #149

liamhuber · 2024-01-05T18:17:22Z

And updating the atomistics notebook to use the new API.

Basically a manual version of what should be automated, as discussed in #148

To use the new API

review-notebook-app · 2024-01-05T18:17:26Z

Check out this pull request on

See visual diffs & provide feedback on Jupyter Notebooks.

Powered by ReviewNB

github-actions · 2024-01-05T18:17:35Z

👈 Launch a binder notebook on branch pyiron/pyiron_workflow/notebooks_env_patch

codacy-production · 2024-01-05T18:19:13Z

Coverage summary from Codacy

See diff coverage on Codacy

Coverage variation	Diff coverage
✅ +0.00% (target: -1.00%)	✅ ∅

Coverage variation details

	Coverable lines	Covered lines	Coverage
Common ancestor commit (`1ea4ff6`)	2503	2136	85.34%
Head commit (`cb6a7e6`)	2503 (+0)	2136 (+0)	85.34% (+0.00%)

Coverage variation is the difference between the coverage for the head and common ancestor commits of the pull request branch: <coverage of head commit> - <coverage of common ancestor commit>

Diff coverage details

	Coverable lines	Covered lines	Diff coverage
Pull request (#149)	0	0	∅ (not applicable)

Diff coverage is the percentage of lines that are covered by tests out of the coverable lines that the pull request added or modified: <covered lines added or modified>/<coverable lines added or modified> * 100%

See your quality gate settings Change summary preferences

_{You may notice some variations in coverage metrics with the latest Coverage engine update. For more details, visit the documentation}

github-actions · 2024-01-05T18:19:15Z

Pull Request Test Coverage Report for Build 7426831220

0 of 0 changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 90.264%

Totals
Change from base Build 7415131425:	0.0%
Covered Lines:	4543
Relevant Lines:	5033

💛 - Coveralls

liamhuber · 2024-01-05T18:26:05Z

At least one Lammps job in the deepdive is crashing. Here's the tail of the errors:

File /usr/share/miniconda3/envs/my-env/lib/python3.11/site-packages/pyiron_base/jobs/job/runfunction.py:114, in run_job_with_status_created(job)
    112     run_job_with_runmode_manually(job=job, _manually_print=True)
    113 elif job.server.run_mode.modal:
--> 114     job.run_static()
    115 elif job.server.run_mode.srun:
    116     run_job_with_runmode_srun(job=job)

File /usr/share/miniconda3/envs/my-env/lib/python3.11/site-packages/pyiron_base/jobs/job/generic.py:765, in GenericJob.run_static(self)
    761 def run_static(self):
    762     """
    763     The run static function is called by run to execute the simulation.
    764     """
--> 765     execute_job_with_external_executable(job=self)

File /usr/share/miniconda3/envs/my-env/lib/python3.11/site-packages/pyiron_base/jobs/job/runfunction.py:608, in run_time_decorator.<locals>.wrapper(job)
    606 if not state.database.database_is_disabled and job.job_id is not None:
    607     job.project.db.item_update({"timestart": datetime.now()}, job.job_id)
--> 608     func(job)
    609     job.project.db.item_update(job._runtime(), job.job_id)
    610 else:

File /usr/share/miniconda3/envs/my-env/lib/python3.11/site-packages/pyiron_base/jobs/job/runfunction.py:646, in execute_job_with_external_executable(job)
    636     out = subprocess.run(
    637         executable,
    638         cwd=job.project_hdf5.working_directory,
   (...)
    643         check=True,
    644     ).stdout
    645 except subprocess.CalledProcessError as e:
--> 646     out, job_crashed = handle_failed_job(job=job, error=e)
    648 job._logger.info(
    649     "{}, status: {}, output: {}".format(job.job_info_str, job.status, out)
    650 )
    651 with open(
    652     posixpath.join(job.project_hdf5.working_directory, "error.out"), mode="w"
    653 ) as f_err:

File /usr/share/miniconda3/envs/my-env/lib/python3.11/site-packages/pyiron_base/jobs/job/runfunction.py:700, in handle_failed_job(job, error)
    698     if job.server.run_mode.non_modal:
    699         state.database.close_connection()
--> 700     raise RuntimeError("Job aborted")
    701 else:
    702     return True, out

RuntimeError: Job aborted

Error: Process completed with exit code 1

…oks_env_patch

liamhuber · 2024-01-05T18:35:54Z

Same error. It's in this cell, the first one where pyiron_lammps is used:

wf.register("pyiron_atomistics", "pyiron_workflow.node_library.pyiron_atomistics")
wf.register("plotting", "pyiron_workflow.node_library.plotting")

wf = Workflow("with_prebuilt")

wf.structure = wf.create.pyiron_atomistics.Bulk(cubic=True, name="Al")
wf.engine = wf.create.pyiron_atomistics.Lammps(structure=wf.structure)
wf.calc = wf.create.pyiron_atomistics.CalcMd(job=wf.engine)
wf.plot = wf.create.plotting.Scatter(
    x=wf.calc.outputs.steps, 
    y=wf.calc.outputs.temperature
)

out = wf.run()
out.plot__fig

It runs totally fine on my machine, and a quick human comparison shows no difference in the env (doesn't mean it's not there, just not in the packages I peeked at).

liamhuber · 2024-01-05T18:44:32Z

I took a closer look and my graphviz (certainly not an issue since the visualization cells are fine) and numpy (potentially an issue but seems unlikely) packages are out of date on my local env. My more thorough search showed everything else lining up fine -- in particular my local env has the same lammps and pyiron-data, along with pyiron_base, pyiron_atomistics, the h5* packages, and everything else I compared. Very strange.

liamhuber · 2024-01-05T18:55:42Z

Nope, updated those two and it's still fine locally.

The head of the error stack is in pyiron_base:

---------------------------------------------------------------------------
Exception encountered at "In [42]":
---------------------------------------------------------------------------
CalledProcessError                        Traceback (most recent call last)
File /usr/share/miniconda3/envs/my-env/lib/python3.11/site-packages/pyiron_base/jobs/job/runfunction.py:636, in execute_job_with_external_executable(job)
    635 try:
--> 636     out = subprocess.run(
    637         executable,
    638         cwd=job.project_hdf5.working_directory,
    639         shell=shell,
    640         stdout=subprocess.PIPE,
    641         stderr=subprocess.STDOUT,
    642         universal_newlines=True,
    643         check=True,
    644     ).stdout
    645 except subprocess.CalledProcessError as e:

So it really looks like a lammps executable crash but why?

liamhuber · 2024-01-05T19:00:03Z

CalledProcessError: Command '/usr/share/miniconda3/envs/my-env/share/pyiron/lammps/bin/run_lammps_2020.03.03.sh' returned non-zero exit status 127.

127 is that the command could not be found. Indeed, in my local env's directory .../share/pyiron/lammps/bin/run_lammps_2020.03.03.sh is there. So what part of the remote env is failing to set this up?

…oks_env_patch

codacy-production · 2024-01-05T19:27:10Z

Coverage summary from Codacy

See diff coverage on Codacy

Coverage variation	Diff coverage
✅ +0.00% (target: -1.00%)	✅ ∅

Coverage variation details

	Coverable lines	Covered lines	Coverage
Common ancestor commit (`1ea4ff6`)	2503	2136	85.34%
Head commit (`313794a`)	2503 (+0)	2136 (+0)	85.34% (+0.00%)

Coverage variation is the difference between the coverage for the head and common ancestor commits of the pull request branch: <coverage of head commit> - <coverage of common ancestor commit>

Diff coverage details

	Coverable lines	Covered lines	Diff coverage
Pull request (#149)	0	0	∅ (not applicable)

Diff coverage is the percentage of lines that are covered by tests out of the coverable lines that the pull request added or modified: <covered lines added or modified>/<coverable lines added or modified> * 100%

See your quality gate settings Change summary preferences

_{You may notice some variations in coverage metrics with the latest Coverage engine update. For more details, visit the documentation}

liamhuber · 2024-01-05T19:30:00Z

Even pyiron-data 0.0.24, which was the first one I used has run_lammps_2020.03.03.sh, so I'm super confused why the executable is being not found

liamhuber · 2024-01-05T19:33:11Z

I wondered if somehow the resource path is broken, but I modified the notebook to show me the resource paths before crashing and they look totally fine:

Cell In[42], line 19
     17 except:
     18     from pyiron_base import state
---> 19     raise RuntimeError(f"State configuration:{state.settings.configuration}")

RuntimeError: State configuration:{'user': 'pyiron', 'resource_paths': ['/home/runner/work/pyiron_workflow/pyiron_workflow/tests/static', '/usr/share/miniconda3/envs/my-env/share/pyiron', '/usr/share/miniconda3/envs/my-env/share/iprpy'], 'project_paths': ['/home/runner/work/pyiron_workflow/pyiron_workflow/'], 'connection_timeout': 60, 'sql_connection_string': None, 'sql_table_name': 'jobs_pyiron', 'sql_view_connection_string': None, 'sql_view_table_name': None, 'sql_view_user': None, 'sql_view_user_key': None, 'sql_file': '/home/runner/pyiron.db', 'sql_host': None, 'sql_type': 'SQLite', 'sql_user_key': None, 'sql_database': None, 'project_check_enabled': False, 'disable_database': False, 'credentials_file': None, 'write_work_dir_warnings': True, 'config_file_permissions_warning': True}

liamhuber · 2024-01-05T21:06:08Z

I couldn't even get things working over in #150 where I specified old versions of stuff.

I updated everything in my local env and find I can still run the deepdive notebook without trouble.

Forget spot-checking suspect packages, I copied the entire environment being installed here, and the entire environment on my local machine, and used some python to compare all the packages that are installed in both locations but don't have the same version. It is few:

PACKAGE	LOCAL	REMOTE
widgetsnbextension	3.6.1	4.0.9
jupyterlab	3.5.3	4.0.10
graphviz	9.0.0	8.1.0
ipywidgets	7.7.1	8.1.1
pyiron-data	0.0.27	0.0.26
notebook	6.5.4	7.0.6
jupyterlab_widgets	1.1.2	3.0.9

For completeness I'm going to bump pyiron-data one more time here, but I'm not optimistic it will actually fix things, since I also had no trouble back when I was on 0.0.26 locally.

liamhuber · 2024-01-05T21:16:38Z

Well, that is good but extremely frustrating. I have no idea why pyiron-data works with both 0.0.26 and 0.0.27 locally, but requires 0.0.27 on the CI.

…oks_env_patch

liamhuber added 2 commits January 5, 2024 10:14

Update extras_require and manually sync notebooks-env to that

6630461

Update atomistics demo

6054f24

To use the new API

[dependabot skip] Update env file

4af9419

liamhuber and others added 3 commits January 5, 2024 10:27

bump pyiron-data

17e77b5

Merge remote-tracking branch 'origin/notebooks_env_patch' into notebo…

1980489

…oks_env_patch

[dependabot skip] Update env file

cb6a7e6

liamhuber added 2 commits January 5, 2024 11:25

Debug resource path in deepdive

6724a0e

Merge remote-tracking branch 'origin/notebooks_env_patch' into notebo…

e72f2a2

…oks_env_patch

Check file status

dcc5d48

liamhuber mentioned this pull request Jan 5, 2024

Don't remove a readme that isn't there #150

Merged

liamhuber and others added 2 commits January 5, 2024 13:07

Bump pyiron-data

f02b8d5

[dependabot skip] Update env file

c6db33d

liamhuber added 2 commits January 5, 2024 13:18

Revert debug stuff in the deepdive notebook

6e6dcb2

Merge remote-tracking branch 'origin/notebooks_env_patch' into notebo…

313794a

…oks_env_patch

liamhuber mentioned this pull request Jan 5, 2024

Tinystore Draft pyiron/pyiron_contrib#948

Closed

liamhuber merged commit 18ae526 into main Jan 5, 2024
16 checks passed

liamhuber deleted the notebooks_env_patch branch January 5, 2024 21:23

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Notebooks env patch #149

Notebooks env patch #149

liamhuber commented Jan 5, 2024

review-notebook-app bot commented Jan 5, 2024

github-actions bot commented Jan 5, 2024

codacy-production bot commented Jan 5, 2024 •

edited

Loading

github-actions bot commented Jan 5, 2024 •

edited

Loading

liamhuber commented Jan 5, 2024

liamhuber commented Jan 5, 2024

liamhuber commented Jan 5, 2024

liamhuber commented Jan 5, 2024

liamhuber commented Jan 5, 2024

codacy-production bot commented Jan 5, 2024 •

edited

Loading

liamhuber commented Jan 5, 2024

liamhuber commented Jan 5, 2024

liamhuber commented Jan 5, 2024

liamhuber commented Jan 5, 2024

Notebooks env patch #149

Notebooks env patch #149

Conversation

liamhuber commented Jan 5, 2024

review-notebook-app bot commented Jan 5, 2024

github-actions bot commented Jan 5, 2024

codacy-production bot commented Jan 5, 2024 • edited Loading

Coverage summary from Codacy

See diff coverage on Codacy

See your quality gate settings Change summary preferences

github-actions bot commented Jan 5, 2024 • edited Loading

Pull Request Test Coverage Report for Build 7426831220

💛 - Coveralls

liamhuber commented Jan 5, 2024

liamhuber commented Jan 5, 2024

liamhuber commented Jan 5, 2024

liamhuber commented Jan 5, 2024

liamhuber commented Jan 5, 2024

codacy-production bot commented Jan 5, 2024 • edited Loading

Coverage summary from Codacy

See diff coverage on Codacy

See your quality gate settings Change summary preferences

liamhuber commented Jan 5, 2024

liamhuber commented Jan 5, 2024

liamhuber commented Jan 5, 2024

liamhuber commented Jan 5, 2024

codacy-production bot commented Jan 5, 2024 •

edited

Loading

github-actions bot commented Jan 5, 2024 •

edited

Loading

codacy-production bot commented Jan 5, 2024 •

edited

Loading