notebooks/hpc_example.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "fb567fa9-ba61-46d4-a645-c31ed30239a5",
   "metadata": {},
   "source": [
    "# `pyiron_workflow` on HPC\n",
    "\n",
    "## Running an entire graph on HPC\n",
    "\n",
    "We've seen how nodes can be saved and loaded, how we can tell a node to make a \"checkpoint\" save  of its graph when it finishes running, and how we (by default) get \"recovery\" save files if the workflow encounters an error. These are all the pieces we need to be able to send a workflow off for remote submission on high performance computing (HPC)!\n",
    "\n",
    "\n",
    "`pyiron_workflow` itself does not provide an explicit interface to HPC compute, although convenience functions built around `pysqa` are highly likely to appear in the future. In the meantime, running on resources that are remote (i.e. _not_ dependent on this local python process) is as simple as:\n",
    "- Saving the node locally\n",
    "- Remotely:\n",
    "  - Loading that save file\n",
    "  - Running the node\n",
    "  - Saving again\n",
    "\n",
    "Here we'll do this by building a super-simple wrapper function that saves, then writes a SLURM script that loads, runs, and saves, and submit that script with `sbatch`. This wrapper is specific to my particular cluster, and won't work out-of-the-box for you -- but it demonstrates the fundamental compatibility between `pyiron_workflow` and file-based execution."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "c9cd6f03-c461-41b8-9f84-0e41f549c3c1",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pyiron_workflow as pwf\n",
    "from time import sleep, perf_counter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "95f9f0a0-3eef-4753-8309-fe38ab778a7b",
   "metadata": {},
   "outputs": [],
   "source": [
    "def submit_to_slurm(\n",
    "    node,\n",
    "    /,\n",
    "    job_name=None,\n",
    "    output_file=None,\n",
    "    error_file=None,\n",
    "    time_limit=\"00:05:00\",\n",
    "    account=\"mpd\",\n",
    "    partition=\"s.cmmg\",\n",
    "    nodes=1,\n",
    "    ntasks=1,\n",
    "    cpus_per_task=1,\n",
    "    memory=\"1GB\",\n",
    "):\n",
    "    \"\"\"\n",
    "    An example of a helper function for running nodes on slurm.\n",
    "\n",
    "    - Saves the node\n",
    "    - Writes a slurm batch script that \n",
    "        - Loads the node\n",
    "        - Runs it\n",
    "        - Saves it again\n",
    "    - Runs the batch script\n",
    "    \"\"\"\n",
    "    if node.graph_root is not node:\n",
    "        raise ValueError(\n",
    "            f\"Can only submit parent-most nodes, but {node.full_label} \"\n",
    "            f\"has root {node.graph_root.full_label}\"\n",
    "        )\n",
    "        \n",
    "    node.save(backend=\"pickle\")\n",
    "    p = node.as_path()\n",
    "    \n",
    "    if job_name is None:\n",
    "        job_name = node.full_label \n",
    "        job_name = job_name.replace(node.semantic_delimiter, \"_\")\n",
    "        job_name = \"pwf\" + job_name\n",
    "        \n",
    "    script_content = f\"\"\"#!/bin/bash\n",
    "#SBATCH --job-name={job_name} \n",
    "#SBATCH --output={p.joinpath(\"slurm.out\").resolve() if output_file is None else output_file}\n",
    "#SBATCH --error={p.joinpath(\"slurm.err\").resolve() if error_file is None else error_file}\n",
    "#SBATCH --time={time_limit}\n",
    "#SBATCH --account={account}\n",
    "#SBATCH --partition={partition}\n",
    "#SBATCH --nodes={nodes}\n",
    "#SBATCH --ntasks={ntasks}\n",
    "#SBATCH --cpus-per-task={cpus_per_task}\n",
    "#SBATCH --mem={memory}\n",
    "\n",
    "# Execute Python script inline\n",
    "python - <<EOF\n",
    "from pyiron_workflow import PickleStorage\n",
    "node = PickleStorage().load(filename=\"{node.as_path().joinpath('picklestorage').resolve()}\")  # Load\n",
    "node.run()  # Run\n",
    "node.save(backend=\"pickle\")  # Save again\n",
    "EOF\n",
    "\"\"\"\n",
    "    submission_script = p.joinpath(\"node_submission.sh\")\n",
    "    submission_script.write_text(script_content)\n",
    "    import subprocess\n",
    "    submission = subprocess.run([\"sbatch\", submission_script.resolve()])\n",
    "    return submission\n",
    "\n",
    "def delete_path(path):\n",
    "    \"\"\"\n",
    "    Delete everything in a one-deep pathlib Path, then delete it too.\n",
    "\n",
    "    Just a little convenience so we can clean up the files we make here.\n",
    "    \"\"\"\n",
    "    for file in path.iterdir():\n",
    "        file.unlink()\n",
    "    path.rmdir()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cdf3aa9c-6d2f-4e6e-a942-a4ebe9be5537",
   "metadata": {},
   "source": [
    "### Running on slurm\n",
    "\n",
    "Here we have an easy but intentionally slow workflow so we can use `squeue` to quickly verify that the job really is running on the cluster.\n",
    "\n",
    "We'll follow the same six steps in each example:\n",
    "- Build the workflow\n",
    "- Send it off to the queue (includes a `.save()` call)\n",
    "- Reload it (make sure the `.save()` has finished!)\n",
    "- Keep reloading it until we can tell it's done (you could also just look in the SLURM queue, but we need a way to automate it here)\n",
    "- Discuss what we can see\n",
    "- Finally, clean up any files we made"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "e6a1199a-9cee-462e-acd2-b9d8122cd3a4",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Submitted batch job 9445778\n",
      "NOT_DATA\n",
      "NOT_DATA\n",
      "NOT_DATA\n",
      "NOT_DATA\n",
      "NOT_DATA\n",
      "NOT_DATA\n",
      "NOT_DATA\n",
      "NOT_DATA\n",
      "NOT_DATA\n",
      "NOT_DATA\n",
      "NOT_DATA\n",
      "NOT_DATA\n",
      "NOT_DATA\n",
      "NOT_DATA\n",
      "NOT_DATA\n",
      "NOT_DATA\n",
      "NOT_DATA\n",
      "NOT_DATA\n",
      "2\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'skipped_my_nap__rmul': 0, 'floating_dreams__float': 2.0}"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Build the workflow\n",
    "wf = pwf.Workflow(\"slurmflow\")\n",
    "wf.sleep_time = pwf.standard_nodes.UserInput(2)\n",
    "wf.skipped_my_nap = 0 * wf.sleep_time\n",
    "wf.sleep = pwf.standard_nodes.Sleep(wf.sleep_time)\n",
    "wf.floating_dreams = pwf.standard_nodes.Float(wf.sleep)\n",
    "\n",
    "# Send it off\n",
    "submit_to_slurm(wf)\n",
    "\n",
    "# Try reloading it (need to wait for `submit_to_slurm`)\n",
    "while not wf.has_saved_content():\n",
    "    sleep(0.2)\n",
    "reloaded = pwf.Workflow(wf.label)\n",
    "\n",
    "# Reload it until it's done\n",
    "while reloaded.floating_dreams.outputs.float.value is pwf.NOT_DATA:\n",
    "    reloaded.load()\n",
    "    print(reloaded.sleep.outputs.time.value)\n",
    "    sleep(1)\n",
    "reloaded.outputs.to_value_dict()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2f0a30b2-d532-4103-8afa-5aeb34617fc0",
   "metadata": {},
   "source": [
    "Here we see that after the initial `.save()` call in `submit_to_slurm` but before the job is finished, the final output is `NOT_DATA` -- we can tell the final `.save()` in the SLURM script got called once this is no longer the case and we load something new!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "d8da8468-93fd-4a8e-a3af-26a5b99f86cf",
   "metadata": {},
   "outputs": [],
   "source": [
    "delete_path(wf.as_path())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0f626ebb-01a2-423c-a471-52db4662fa9b",
   "metadata": {},
   "source": [
    "### Checkpoints\n",
    "\n",
    "This example is very similar, except one of our intermediate nodes is set as a \"checkpoint\" node, which will save the graph when it finishes running. This may be useful if you have a large workflow and want to be able to start examining intermediate results before the whole graph has finished running!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "f6c7f938-95aa-46de-9293-cec34cfa3c80",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Submitted batch job 9445782\n",
      "NOT_DATA NOT_DATA\n",
      "NOT_DATA NOT_DATA\n",
      "NOT_DATA NOT_DATA\n",
      "NOT_DATA NOT_DATA\n",
      "NOT_DATA NOT_DATA\n",
      "NOT_DATA NOT_DATA\n",
      "NOT_DATA NOT_DATA\n",
      "NOT_DATA NOT_DATA\n",
      "NOT_DATA NOT_DATA\n",
      "5 NOT_DATA\n",
      "5 NOT_DATA\n",
      "5 NOT_DATA\n",
      "5 NOT_DATA\n",
      "5 NOT_DATA\n",
      "5 5\n"
     ]
    }
   ],
   "source": [
    "wf = pwf.Workflow(\"checkpointing\")\n",
    "wf.sleep_time = pwf.standard_nodes.UserInput(5)\n",
    "wf.sleep = pwf.standard_nodes.Sleep(wf.sleep_time)\n",
    "wf.sleep_again = pwf.standard_nodes.Sleep(wf.sleep)\n",
    "\n",
    "wf.sleep.checkpoint = \"pickle\"\n",
    "submit_to_slurm(wf)\n",
    "\n",
    "while not wf.has_saved_content():\n",
    "    sleep(0.2)\n",
    "reloaded = pwf.Workflow(wf.label)\n",
    "    \n",
    "while reloaded.sleep_again.outputs.time.value is pwf.NOT_DATA:\n",
    "    reloaded.load()\n",
    "    print(reloaded.sleep.outputs.time.value, reloaded.sleep_again.outputs.time.value)\n",
    "    sleep(1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cdb78aba-a174-418f-b6f1-9a5427d33c57",
   "metadata": {},
   "source": [
    "We can see that initially there's a few seconds where all the (de)serialization is happening to get the job running, followed by five seconds of running where the checkpoint job `wf.sleep` hasn't finished yet -- both outputs are `NOT_DATA`. But then the checkpoint save is made _and the job keeps running_, so we can see the intermediate data `5`, while the later node hasn't finished yet. Finally, both nodes finish and the SLURM job makes its final save."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "7a9b3ee4-9a4c-4f96-a371-8ae3e8c54b0b",
   "metadata": {},
   "outputs": [],
   "source": [
    "delete_path(wf.as_path())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "85aff023-1ebd-4d99-ad16-ae3650841313",
   "metadata": {},
   "source": [
    "### Recovering from failure\n",
    "\n",
    "By default, any node that raises an exception will act as a checkpoint node at the time of failure `finally:` making a save after raising the exception. This means that if we have some expensive workflow and one of the nodes goes wrong and raises an error, all is not lost! The node will save a \"recovery\" file at the point of failure, which we can come back and investigate, extracting data and/or repairing it for a re-run (remember that non-failed nodes will (by default) have cached values, so re-running those parts is usually very fast)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "94cab20c-d129-400b-9ee5-67d1e63d840b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Submitted batch job 9445785\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 12.0.0 (20240803.0821)\n",
       " -->\n",
       "<!-- Title: clusterrecovery Pages: 1 -->\n",
       "<svg width=\"720pt\" height=\"187pt\"\n",
       " viewBox=\"0.00 0.00 720.00 187.22\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(0.28059 0.28059) rotate(0) translate(4 663.25)\">\n",
       "<title>clusterrecovery</title>\n",
       "<polygon fill=\"white\" stroke=\"none\" points=\"-4,4 -4,-663.25 2562.02,-663.25 2562.02,4 -4,4\"/>\n",
       "<text text-anchor=\"middle\" x=\"1279.01\" y=\"-7.95\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">recovery: Workflow</text>\n",
       "<g id=\"clust1\" class=\"cluster\">\n",
       "<title>clusterrecoveryInputs</title>\n",
       "<defs>\n",
       "<linearGradient id=\"clust1_l_0\" gradientUnits=\"userSpaceOnUse\" x1=\"8\" y1=\"-233.75\" x2=\"289.51\" y2=\"-233.75\" >\n",
       "<stop offset=\"0\" style=\"stop-color:#a5a4a5;stop-opacity:1.;\"/>\n",
       "<stop offset=\"1\" style=\"stop-color:#e4e3e4;stop-opacity:1.;\"/>\n",
       "</linearGradient>\n",
       "</defs>\n",
       "<polygon fill=\"url(#clust1_l_0)\" stroke=\"#a5a4a5\" points=\"8,-33.25 8,-434.25 289.51,-434.25 289.51,-33.25 8,-33.25\"/>\n",
       "<text text-anchor=\"middle\" x=\"148.76\" y=\"-416.95\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">Inputs</text>\n",
       "</g>\n",
       "<g id=\"clust2\" class=\"cluster\">\n",
       "<title>clusterrecoveryOutputs</title>\n",
       "<defs>\n",
       "<linearGradient id=\"clust2_l_1\" gradientUnits=\"userSpaceOnUse\" x1=\"2550.02\" y1=\"-341.75\" x2=\"2268.51\" y2=\"-341.75\" >\n",
       "<stop offset=\"0\" style=\"stop-color:#a5a4a5;stop-opacity:1.;\"/>\n",
       "<stop offset=\"1\" style=\"stop-color:#e4e3e4;stop-opacity:1.;\"/>\n",
       "</linearGradient>\n",
       "</defs>\n",
       "<polygon fill=\"url(#clust2_l_1)\" stroke=\"#a5a4a5\" points=\"2268.51,-195.25 2268.51,-488.25 2550.02,-488.25 2550.02,-195.25 2268.51,-195.25\"/>\n",
       "<text text-anchor=\"middle\" x=\"2409.27\" y=\"-470.95\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">Outputs</text>\n",
       "</g>\n",
       "<g id=\"clust3\" class=\"cluster\">\n",
       "<title>clusterrecoverya</title>\n",
       "<defs>\n",
       "<linearGradient id=\"clust3_l_2\" gradientUnits=\"userSpaceOnUse\" x1=\"301.51\" y1=\"-436.25\" x2=\"684.51\" y2=\"-436.25\" >\n",
       "<stop offset=\"0\" style=\"stop-color:#bfe2bf;stop-opacity:1.;\"/>\n",
       "<stop offset=\"1\" style=\"stop-color:#bfe2bf;stop-opacity:1.;\"/>\n",
       "</linearGradient>\n",
       "</defs>\n",
       "<polygon fill=\"url(#clust3_l_2)\" stroke=\"#2ca02c\" stroke-width=\"2\" points=\"301.51,-323.25 301.51,-549.25 684.51,-549.25 684.51,-323.25 301.51,-323.25\"/>\n",
       "<text text-anchor=\"middle\" x=\"493.01\" y=\"-531.95\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">a: UserInput</text>\n",
       "</g>\n",
       "<g id=\"clust4\" class=\"cluster\">\n",
       "<title>clusterrecoveryaInputs</title>\n",
       "<defs>\n",
       "<linearGradient id=\"clust4_l_3\" gradientUnits=\"userSpaceOnUse\" x1=\"309.51\" y1=\"-423.75\" x2=\"485.51\" y2=\"-423.75\" >\n",
       "<stop offset=\"0\" style=\"stop-color:#a5a4a5;stop-opacity:1.;\"/>\n",
       "<stop offset=\"1\" style=\"stop-color:#e4e3e4;stop-opacity:1.;\"/>\n",
       "</linearGradient>\n",
       "</defs>\n",
       "<polygon fill=\"url(#clust4_l_3)\" stroke=\"#a5a4a5\" points=\"309.51,-331.25 309.51,-516.25 485.51,-516.25 485.51,-331.25 309.51,-331.25\"/>\n",
       "<text text-anchor=\"middle\" x=\"397.51\" y=\"-498.95\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">Inputs</text>\n",
       "</g>\n",
       "<g id=\"clust5\" class=\"cluster\">\n",
       "<title>clusterrecoveryaOutputsWithInjection</title>\n",
       "<defs>\n",
       "<linearGradient id=\"clust5_l_4\" gradientUnits=\"userSpaceOnUse\" x1=\"676.51\" y1=\"-423.75\" x2=\"513.51\" y2=\"-423.75\" >\n",
       "<stop offset=\"0\" style=\"stop-color:#a5a4a5;stop-opacity:1.;\"/>\n",
       "<stop offset=\"1\" style=\"stop-color:#e4e3e4;stop-opacity:1.;\"/>\n",
       "</linearGradient>\n",
       "</defs>\n",
       "<polygon fill=\"url(#clust5_l_4)\" stroke=\"#a5a4a5\" points=\"513.51,-331.25 513.51,-516.25 676.51,-516.25 676.51,-331.25 513.51,-331.25\"/>\n",
       "<text text-anchor=\"middle\" x=\"595.01\" y=\"-498.95\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">OutputsWithInjection</text>\n",
       "</g>\n",
       "<g id=\"clust6\" class=\"cluster\">\n",
       "<title>clusterrecoveryb</title>\n",
       "<defs>\n",
       "<linearGradient id=\"clust6_l_5\" gradientUnits=\"userSpaceOnUse\" x1=\"692.51\" y1=\"-347.25\" x2=\"1075.51\" y2=\"-347.25\" >\n",
       "<stop offset=\"0\" style=\"stop-color:#bfe2bf;stop-opacity:1.;\"/>\n",
       "<stop offset=\"1\" style=\"stop-color:#bfe2bf;stop-opacity:1.;\"/>\n",
       "</linearGradient>\n",
       "</defs>\n",
       "<polygon fill=\"url(#clust6_l_5)\" stroke=\"#2ca02c\" stroke-width=\"2\" points=\"692.51,-234.25 692.51,-460.25 1075.51,-460.25 1075.51,-234.25 692.51,-234.25\"/>\n",
       "<text text-anchor=\"middle\" x=\"884.01\" y=\"-442.95\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">b: UserInput</text>\n",
       "</g>\n",
       "<g id=\"clust7\" class=\"cluster\">\n",
       "<title>clusterrecoverybInputs</title>\n",
       "<defs>\n",
       "<linearGradient id=\"clust7_l_6\" gradientUnits=\"userSpaceOnUse\" x1=\"700.51\" y1=\"-334.75\" x2=\"876.51\" y2=\"-334.75\" >\n",
       "<stop offset=\"0\" style=\"stop-color:#a5a4a5;stop-opacity:1.;\"/>\n",
       "<stop offset=\"1\" style=\"stop-color:#e4e3e4;stop-opacity:1.;\"/>\n",
       "</linearGradient>\n",
       "</defs>\n",
       "<polygon fill=\"url(#clust7_l_6)\" stroke=\"#a5a4a5\" points=\"700.51,-242.25 700.51,-427.25 876.51,-427.25 876.51,-242.25 700.51,-242.25\"/>\n",
       "<text text-anchor=\"middle\" x=\"788.51\" y=\"-409.95\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">Inputs</text>\n",
       "</g>\n",
       "<g id=\"clust8\" class=\"cluster\">\n",
       "<title>clusterrecoverybOutputsWithInjection</title>\n",
       "<defs>\n",
       "<linearGradient id=\"clust8_l_7\" gradientUnits=\"userSpaceOnUse\" x1=\"1067.51\" y1=\"-334.75\" x2=\"904.51\" y2=\"-334.75\" >\n",
       "<stop offset=\"0\" style=\"stop-color:#a5a4a5;stop-opacity:1.;\"/>\n",
       "<stop offset=\"1\" style=\"stop-color:#e4e3e4;stop-opacity:1.;\"/>\n",
       "</linearGradient>\n",
       "</defs>\n",
       "<polygon fill=\"url(#clust8_l_7)\" stroke=\"#a5a4a5\" points=\"904.51,-242.25 904.51,-427.25 1067.51,-427.25 1067.51,-242.25 904.51,-242.25\"/>\n",
       "<text text-anchor=\"middle\" x=\"986.01\" y=\"-409.95\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">OutputsWithInjection</text>\n",
       "</g>\n",
       "<g id=\"clust9\" class=\"cluster\">\n",
       "<title>clusterrecoveryc_fails</title>\n",
       "<defs>\n",
       "<linearGradient id=\"clust9_l_8\" gradientUnits=\"userSpaceOnUse\" x1=\"1083.51\" y1=\"-392.25\" x2=\"1466.51\" y2=\"-392.25\" >\n",
       "<stop offset=\"0\" style=\"stop-color:#bfe2bf;stop-opacity:1.;\"/>\n",
       "<stop offset=\"1\" style=\"stop-color:#bfe2bf;stop-opacity:1.;\"/>\n",
       "</linearGradient>\n",
       "</defs>\n",
       "<polygon fill=\"url(#clust9_l_8)\" stroke=\"#ff0000\" stroke-width=\"2\" points=\"1083.51,-252.25 1083.51,-532.25 1466.51,-532.25 1466.51,-252.25 1083.51,-252.25\"/>\n",
       "<text text-anchor=\"middle\" x=\"1275.01\" y=\"-514.95\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">c_fails: Add</text>\n",
       "</g>\n",
       "<g id=\"clust10\" class=\"cluster\">\n",
       "<title>clusterrecoveryc_failsInputs</title>\n",
       "<defs>\n",
       "<linearGradient id=\"clust10_l_9\" gradientUnits=\"userSpaceOnUse\" x1=\"1091.51\" y1=\"-379.75\" x2=\"1267.51\" y2=\"-379.75\" >\n",
       "<stop offset=\"0\" style=\"stop-color:#a5a4a5;stop-opacity:1.;\"/>\n",
       "<stop offset=\"1\" style=\"stop-color:#e4e3e4;stop-opacity:1.;\"/>\n",
       "</linearGradient>\n",
       "</defs>\n",
       "<polygon fill=\"url(#clust10_l_9)\" stroke=\"#a5a4a5\" points=\"1091.51,-260.25 1091.51,-499.25 1267.51,-499.25 1267.51,-260.25 1091.51,-260.25\"/>\n",
       "<text text-anchor=\"middle\" x=\"1179.51\" y=\"-481.95\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">Inputs</text>\n",
       "</g>\n",
       "<g id=\"clust11\" class=\"cluster\">\n",
       "<title>clusterrecoveryc_failsOutputsWithInjection</title>\n",
       "<defs>\n",
       "<linearGradient id=\"clust11_l_10\" gradientUnits=\"userSpaceOnUse\" x1=\"1458.51\" y1=\"-388.75\" x2=\"1295.51\" y2=\"-388.75\" >\n",
       "<stop offset=\"0\" style=\"stop-color:#a5a4a5;stop-opacity:1.;\"/>\n",
       "<stop offset=\"1\" style=\"stop-color:#e4e3e4;stop-opacity:1.;\"/>\n",
       "</linearGradient>\n",
       "</defs>\n",
       "<polygon fill=\"url(#clust11_l_10)\" stroke=\"#a5a4a5\" points=\"1295.51,-296.25 1295.51,-481.25 1458.51,-481.25 1458.51,-296.25 1295.51,-296.25\"/>\n",
       "<text text-anchor=\"middle\" x=\"1377.01\" y=\"-463.95\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">OutputsWithInjection</text>\n",
       "</g>\n",
       "<g id=\"clust12\" class=\"cluster\">\n",
       "<title>clusterrecoveryd_if_success</title>\n",
       "<defs>\n",
       "<linearGradient id=\"clust12_l_11\" gradientUnits=\"userSpaceOnUse\" x1=\"1474.51\" y1=\"-538.25\" x2=\"1865.51\" y2=\"-538.25\" >\n",
       "<stop offset=\"0\" style=\"stop-color:#bfe2bf;stop-opacity:1.;\"/>\n",
       "<stop offset=\"1\" style=\"stop-color:#bfe2bf;stop-opacity:1.;\"/>\n",
       "</linearGradient>\n",
       "</defs>\n",
       "<polygon fill=\"url(#clust12_l_11)\" stroke=\"#bfe2bf\" stroke-width=\"2\" points=\"1474.51,-425.25 1474.51,-651.25 1865.51,-651.25 1865.51,-425.25 1474.51,-425.25\"/>\n",
       "<text text-anchor=\"middle\" x=\"1670.01\" y=\"-633.95\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">d_if_success: UserInput</text>\n",
       "</g>\n",
       "<g id=\"clust13\" class=\"cluster\">\n",
       "<title>clusterrecoveryd_if_successInputs</title>\n",
       "<defs>\n",
       "<linearGradient id=\"clust13_l_12\" gradientUnits=\"userSpaceOnUse\" x1=\"1482.51\" y1=\"-525.75\" x2=\"1658.51\" y2=\"-525.75\" >\n",
       "<stop offset=\"0\" style=\"stop-color:#a5a4a5;stop-opacity:1.;\"/>\n",
       "<stop offset=\"1\" style=\"stop-color:#e4e3e4;stop-opacity:1.;\"/>\n",
       "</linearGradient>\n",
       "</defs>\n",
       "<polygon fill=\"url(#clust13_l_12)\" stroke=\"#a5a4a5\" points=\"1482.51,-433.25 1482.51,-618.25 1658.51,-618.25 1658.51,-433.25 1482.51,-433.25\"/>\n",
       "<text text-anchor=\"middle\" x=\"1570.51\" y=\"-600.95\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">Inputs</text>\n",
       "</g>\n",
       "<g id=\"clust14\" class=\"cluster\">\n",
       "<title>clusterrecoveryd_if_successOutputsWithInjection</title>\n",
       "<defs>\n",
       "<linearGradient id=\"clust14_l_13\" gradientUnits=\"userSpaceOnUse\" x1=\"1857.51\" y1=\"-525.75\" x2=\"1694.51\" y2=\"-525.75\" >\n",
       "<stop offset=\"0\" style=\"stop-color:#a5a4a5;stop-opacity:1.;\"/>\n",
       "<stop offset=\"1\" style=\"stop-color:#e4e3e4;stop-opacity:1.;\"/>\n",
       "</linearGradient>\n",
       "</defs>\n",
       "<polygon fill=\"url(#clust14_l_13)\" stroke=\"#a5a4a5\" points=\"1694.51,-433.25 1694.51,-618.25 1857.51,-618.25 1857.51,-433.25 1694.51,-433.25\"/>\n",
       "<text text-anchor=\"middle\" x=\"1776.01\" y=\"-600.95\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">OutputsWithInjection</text>\n",
       "</g>\n",
       "<g id=\"clust15\" class=\"cluster\">\n",
       "<title>clusterrecoveryd_if_failure</title>\n",
       "<defs>\n",
       "<linearGradient id=\"clust15_l_14\" gradientUnits=\"userSpaceOnUse\" x1=\"1474.51\" y1=\"-276.25\" x2=\"1865.51\" y2=\"-276.25\" >\n",
       "<stop offset=\"0\" style=\"stop-color:#bfe2bf;stop-opacity:1.;\"/>\n",
       "<stop offset=\"1\" style=\"stop-color:#bfe2bf;stop-opacity:1.;\"/>\n",
       "</linearGradient>\n",
       "</defs>\n",
       "<polygon fill=\"url(#clust15_l_14)\" stroke=\"#2ca02c\" stroke-width=\"2\" points=\"1474.51,-163.25 1474.51,-389.25 1865.51,-389.25 1865.51,-163.25 1474.51,-163.25\"/>\n",
       "<text text-anchor=\"middle\" x=\"1670.01\" y=\"-371.95\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">d_if_failure: UserInput</text>\n",
       "</g>\n",
       "<g id=\"clust16\" class=\"cluster\">\n",
       "<title>clusterrecoveryd_if_failureInputs</title>\n",
       "<defs>\n",
       "<linearGradient id=\"clust16_l_15\" gradientUnits=\"userSpaceOnUse\" x1=\"1482.51\" y1=\"-263.75\" x2=\"1658.51\" y2=\"-263.75\" >\n",
       "<stop offset=\"0\" style=\"stop-color:#a5a4a5;stop-opacity:1.;\"/>\n",
       "<stop offset=\"1\" style=\"stop-color:#e4e3e4;stop-opacity:1.;\"/>\n",
       "</linearGradient>\n",
       "</defs>\n",
       "<polygon fill=\"url(#clust16_l_15)\" stroke=\"#a5a4a5\" points=\"1482.51,-171.25 1482.51,-356.25 1658.51,-356.25 1658.51,-171.25 1482.51,-171.25\"/>\n",
       "<text text-anchor=\"middle\" x=\"1570.51\" y=\"-338.95\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">Inputs</text>\n",
       "</g>\n",
       "<g id=\"clust17\" class=\"cluster\">\n",
       "<title>clusterrecoveryd_if_failureOutputsWithInjection</title>\n",
       "<defs>\n",
       "<linearGradient id=\"clust17_l_16\" gradientUnits=\"userSpaceOnUse\" x1=\"1857.51\" y1=\"-263.75\" x2=\"1694.51\" y2=\"-263.75\" >\n",
       "<stop offset=\"0\" style=\"stop-color:#a5a4a5;stop-opacity:1.;\"/>\n",
       "<stop offset=\"1\" style=\"stop-color:#e4e3e4;stop-opacity:1.;\"/>\n",
       "</linearGradient>\n",
       "</defs>\n",
       "<polygon fill=\"url(#clust17_l_16)\" stroke=\"#a5a4a5\" points=\"1694.51,-171.25 1694.51,-356.25 1857.51,-356.25 1857.51,-171.25 1694.51,-171.25\"/>\n",
       "<text text-anchor=\"middle\" x=\"1776.01\" y=\"-338.95\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">OutputsWithInjection</text>\n",
       "</g>\n",
       "<g id=\"clust18\" class=\"cluster\">\n",
       "<title>clusterrecoverye_fails</title>\n",
       "<defs>\n",
       "<linearGradient id=\"clust18_l_17\" gradientUnits=\"userSpaceOnUse\" x1=\"1873.51\" y1=\"-249.25\" x2=\"2256.51\" y2=\"-249.25\" >\n",
       "<stop offset=\"0\" style=\"stop-color:#bfe2bf;stop-opacity:1.;\"/>\n",
       "<stop offset=\"1\" style=\"stop-color:#bfe2bf;stop-opacity:1.;\"/>\n",
       "</linearGradient>\n",
       "</defs>\n",
       "<polygon fill=\"url(#clust18_l_17)\" stroke=\"#ff0000\" stroke-width=\"2\" points=\"1873.51,-109.25 1873.51,-389.25 2256.51,-389.25 2256.51,-109.25 1873.51,-109.25\"/>\n",
       "<text text-anchor=\"middle\" x=\"2065.01\" y=\"-371.95\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">e_fails: Add</text>\n",
       "</g>\n",
       "<g id=\"clust19\" class=\"cluster\">\n",
       "<title>clusterrecoverye_failsInputs</title>\n",
       "<defs>\n",
       "<linearGradient id=\"clust19_l_18\" gradientUnits=\"userSpaceOnUse\" x1=\"1881.51\" y1=\"-236.75\" x2=\"2057.51\" y2=\"-236.75\" >\n",
       "<stop offset=\"0\" style=\"stop-color:#a5a4a5;stop-opacity:1.;\"/>\n",
       "<stop offset=\"1\" style=\"stop-color:#e4e3e4;stop-opacity:1.;\"/>\n",
       "</linearGradient>\n",
       "</defs>\n",
       "<polygon fill=\"url(#clust19_l_18)\" stroke=\"#a5a4a5\" points=\"1881.51,-117.25 1881.51,-356.25 2057.51,-356.25 2057.51,-117.25 1881.51,-117.25\"/>\n",
       "<text text-anchor=\"middle\" x=\"1969.51\" y=\"-338.95\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">Inputs</text>\n",
       "</g>\n",
       "<g id=\"clust20\" class=\"cluster\">\n",
       "<title>clusterrecoverye_failsOutputsWithInjection</title>\n",
       "<defs>\n",
       "<linearGradient id=\"clust20_l_19\" gradientUnits=\"userSpaceOnUse\" x1=\"2248.51\" y1=\"-263.75\" x2=\"2085.51\" y2=\"-263.75\" >\n",
       "<stop offset=\"0\" style=\"stop-color:#a5a4a5;stop-opacity:1.;\"/>\n",
       "<stop offset=\"1\" style=\"stop-color:#e4e3e4;stop-opacity:1.;\"/>\n",
       "</linearGradient>\n",
       "</defs>\n",
       "<polygon fill=\"url(#clust20_l_19)\" stroke=\"#a5a4a5\" points=\"2085.51,-171.25 2085.51,-356.25 2248.51,-356.25 2248.51,-171.25 2085.51,-171.25\"/>\n",
       "<text text-anchor=\"middle\" x=\"2167.01\" y=\"-338.95\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">OutputsWithInjection</text>\n",
       "</g>\n",
       "<!-- clusterrecoveryInputsrun -->\n",
       "<g id=\"node1\" class=\"node\">\n",
       "<title>clusterrecoveryInputsrun</title>\n",
       "<polygon fill=\"#21bfd8\" stroke=\"#21bfd8\" points=\"163.76,-71.25 121.76,-71.25 121.76,-47.25 163.76,-47.25 175.76,-59.25 163.76,-71.25\"/>\n",
       "<text text-anchor=\"middle\" x=\"148.76\" y=\"-54.58\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">run</text>\n",
       "</g>\n",
       "<!-- clusterrecoveryOutputsran -->\n",
       "<g id=\"node8\" class=\"node\">\n",
       "<title>clusterrecoveryOutputsran</title>\n",
       "<polygon fill=\"#21bfd8\" stroke=\"#21bfd8\" points=\"2424.27,-233.25 2382.27,-233.25 2382.27,-209.25 2424.27,-209.25 2436.27,-221.25 2424.27,-233.25\"/>\n",
       "<text text-anchor=\"middle\" x=\"2409.27\" y=\"-216.57\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">ran</text>\n",
       "</g>\n",
       "<!-- clusterrecoveryInputsrun&#45;&gt;clusterrecoveryOutputsran -->\n",
       "<!-- clusterrecoveryInputsaccumulate_and_run -->\n",
       "<g id=\"node2\" class=\"node\">\n",
       "<title>clusterrecoveryInputsaccumulate_and_run</title>\n",
       "<polygon fill=\"#21bfd8\" stroke=\"#21bfd8\" points=\"216.76,-125.25 68.76,-125.25 68.76,-101.25 216.76,-101.25 228.76,-113.25 216.76,-125.25\"/>\n",
       "<text text-anchor=\"middle\" x=\"148.76\" y=\"-108.58\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">accumulate_and_run</text>\n",
       "</g>\n",
       "<!-- clusterrecoveryInputsa__user_input -->\n",
       "<g id=\"node3\" class=\"node\">\n",
       "<title>clusterrecoveryInputsa__user_input</title>\n",
       "<ellipse fill=\"#edb22c\" stroke=\"#edb22c\" cx=\"148.76\" cy=\"-329.25\" rx=\"78.04\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"148.76\" y=\"-324.57\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">a__user_input</text>\n",
       "</g>\n",
       "<!-- clusterrecoveryaInputsuser_input -->\n",
       "<g id=\"node15\" class=\"node\">\n",
       "<title>clusterrecoveryaInputsuser_input</title>\n",
       "<ellipse fill=\"#edb22c\" stroke=\"#edb22c\" cx=\"397.51\" cy=\"-465.25\" rx=\"62.78\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"397.51\" y=\"-460.57\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">user_input</text>\n",
       "</g>\n",
       "<!-- clusterrecoveryInputsa__user_input&#45;&gt;clusterrecoveryaInputsuser_input -->\n",
       "<g id=\"edge16\" class=\"edge\">\n",
       "<title>clusterrecoveryInputsa__user_input&#45;&gt;clusterrecoveryaInputsuser_input</title>\n",
       "<path fill=\"none\" stroke=\"#edb22c\" d=\"M226.81,-327.71C249.87,-331.18 273.29,-339.28 289.51,-356.25 298.01,-365.14 299.46,-374.16 298.53,-383.24\"/>\n",
       "<path fill=\"none\" stroke=\"#edb22c\" d=\"M298.53,-383.24C296.67,-401.37 285.34,-419.8 301.51,-438.25 308.4,-446.11 317.39,-451.76 327.1,-455.8\"/>\n",
       "<polygon fill=\"#edb22c\" stroke=\"#edb22c\" points=\"325.82,-459.06 336.42,-459.08 328.15,-452.46 325.82,-459.06\"/>\n",
       "</g>\n",
       "<!-- clusterrecoveryInputsb__user_input -->\n",
       "<g id=\"node4\" class=\"node\">\n",
       "<title>clusterrecoveryInputsb__user_input</title>\n",
       "<ellipse fill=\"#edb22c\" stroke=\"#edb22c\" cx=\"148.76\" cy=\"-275.25\" rx=\"78.57\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"148.76\" y=\"-270.57\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">b__user_input</text>\n",
       "</g>\n",
       "<!-- clusterrecoverybInputsuser_input -->\n",
       "<g id=\"node21\" class=\"node\">\n",
       "<title>clusterrecoverybInputsuser_input</title>\n",
       "<ellipse fill=\"#edb22c\" stroke=\"#edb22c\" cx=\"788.51\" cy=\"-268.25\" rx=\"62.78\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"788.51\" y=\"-263.57\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">user_input</text>\n",
       "</g>\n",
       "<!-- clusterrecoveryInputsb__user_input&#45;&gt;clusterrecoverybInputsuser_input -->\n",
       "<g id=\"edge17\" class=\"edge\">\n",
       "<title>clusterrecoveryInputsb__user_input&#45;&gt;clusterrecoverybInputsuser_input</title>\n",
       "<path fill=\"none\" stroke=\"#edb22c\" d=\"M227.64,-274.4C289.68,-273.71 380.94,-272.71 471.85,-271.71\"/>\n",
       "<path fill=\"none\" stroke=\"#edb22c\" d=\"M471.85,-271.71C562.76,-270.72 653.32,-269.72 713.97,-269.06\"/>\n",
       "<polygon fill=\"#edb22c\" stroke=\"#edb22c\" points=\"713.85,-272.56 723.81,-268.95 713.78,-265.56 713.85,-272.56\"/>\n",
       "</g>\n",
       "<!-- clusterrecoveryInputsd_if_success__user_input -->\n",
       "<g id=\"node5\" class=\"node\">\n",
       "<title>clusterrecoveryInputsd_if_success__user_input</title>\n",
       "<ellipse fill=\"#edb22c\" stroke=\"#edb22c\" cx=\"148.76\" cy=\"-383.25\" rx=\"132.76\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"148.76\" y=\"-378.57\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">d_if_success__user_input</text>\n",
       "</g>\n",
       "<!-- clusterrecoveryd_if_successInputsuser_input -->\n",
       "<g id=\"node34\" class=\"node\">\n",
       "<title>clusterrecoveryd_if_successInputsuser_input</title>\n",
       "<ellipse fill=\"#edb22c\" stroke=\"#edb22c\" cx=\"1570.51\" cy=\"-567.25\" rx=\"62.78\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"1570.51\" y=\"-562.58\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">user_input</text>\n",
       "</g>\n",
       "<!-- clusterrecoveryInputsd_if_success__user_input&#45;&gt;clusterrecoveryd_if_successInputsuser_input -->\n",
       "<g id=\"edge18\" class=\"edge\">\n",
       "<title>clusterrecoveryInputsd_if_success__user_input&#45;&gt;clusterrecoveryd_if_successInputsuser_input</title>\n",
       "<path fill=\"none\" stroke=\"#edb22c\" d=\"M163.84,-401.49C188.27,-431.68 241.66,-491.56 301.51,-520.25 420.05,-577.07 462.06,-567.25 593.51,-567.25 593.51,-567.25 593.51,-567.25 618.76,-567.25\"/>\n",
       "<path fill=\"none\" stroke=\"#edb22c\" d=\"M618.76,-567.25C665.58,-567.25 799.2,-567.25 1180.51,-567.25 1290.58,-567.25 1418.17,-567.25 1496.36,-567.25\"/>\n",
       "<polygon fill=\"#edb22c\" stroke=\"#edb22c\" points=\"1496.01,-570.75 1506.01,-567.25 1496.01,-563.75 1496.01,-570.75\"/>\n",
       "</g>\n",
       "<!-- clusterrecoveryInputsd_if_failure__user_input -->\n",
       "<g id=\"node6\" class=\"node\">\n",
       "<title>clusterrecoveryInputsd_if_failure__user_input</title>\n",
       "<ellipse fill=\"#edb22c\" stroke=\"#edb22c\" cx=\"148.76\" cy=\"-221.25\" rx=\"124.86\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"148.76\" y=\"-216.57\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">d_if_failure__user_input</text>\n",
       "</g>\n",
       "<!-- clusterrecoveryd_if_failureInputsuser_input -->\n",
       "<g id=\"node40\" class=\"node\">\n",
       "<title>clusterrecoveryd_if_failureInputsuser_input</title>\n",
       "<ellipse fill=\"#edb22c\" stroke=\"#edb22c\" cx=\"1570.51\" cy=\"-197.25\" rx=\"62.78\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"1570.51\" y=\"-192.57\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">user_input</text>\n",
       "</g>\n",
       "<!-- clusterrecoveryInputsd_if_failure__user_input&#45;&gt;clusterrecoveryd_if_failureInputsuser_input -->\n",
       "<g id=\"edge19\" class=\"edge\">\n",
       "<title>clusterrecoveryInputsd_if_failure__user_input&#45;&gt;clusterrecoveryd_if_failureInputsuser_input</title>\n",
       "<path fill=\"none\" stroke=\"#edb22c\" d=\"M266.97,-215.13C356.42,-210.98 482.7,-206.25 593.51,-206.25 593.51,-206.25 593.51,-206.25 663.05,-206.25\"/>\n",
       "<path fill=\"none\" stroke=\"#edb22c\" d=\"M663.05,-206.25C735.1,-206.25 881.8,-206.25 1180.51,-206.25 1290.77,-206.25 1418.57,-202.59 1496.71,-199.94\"/>\n",
       "<polygon fill=\"#edb22c\" stroke=\"#edb22c\" points=\"1496.49,-203.45 1506.36,-199.61 1496.25,-196.45 1496.49,-203.45\"/>\n",
       "</g>\n",
       "<!-- clusterrecoveryInputse_fails__other -->\n",
       "<g id=\"node7\" class=\"node\">\n",
       "<title>clusterrecoveryInputse_fails__other</title>\n",
       "<ellipse fill=\"#edb22c\" stroke=\"#edb22c\" cx=\"148.76\" cy=\"-167.25\" rx=\"76.99\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"148.76\" y=\"-162.57\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">e_fails__other</text>\n",
       "</g>\n",
       "<!-- clusterrecoverye_failsInputsother -->\n",
       "<g id=\"node47\" class=\"node\">\n",
       "<title>clusterrecoverye_failsInputsother</title>\n",
       "<ellipse fill=\"#edb22c\" stroke=\"#edb22c\" cx=\"1969.51\" cy=\"-143.25\" rx=\"37\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"1969.51\" y=\"-138.57\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">other</text>\n",
       "</g>\n",
       "<!-- clusterrecoveryInputse_fails__other&#45;&gt;clusterrecoverye_failsInputsother -->\n",
       "<g id=\"edge20\" class=\"edge\">\n",
       "<title>clusterrecoveryInputse_fails__other&#45;&gt;clusterrecoverye_failsInputsother</title>\n",
       "<path fill=\"none\" stroke=\"#edb22c\" d=\"M221.95,-161.21C309.71,-154.39 462.36,-144.25 593.51,-144.25 593.51,-144.25 593.51,-144.25 707.38,-144.25\"/>\n",
       "<path fill=\"none\" stroke=\"#edb22c\" d=\"M707.38,-144.25C826.71,-144.25 1071.07,-144.25 1571.51,-144.25 1697.61,-144.25 1846.25,-143.74 1920.8,-143.45\"/>\n",
       "<polygon fill=\"#edb22c\" stroke=\"#edb22c\" points=\"1920.61,-146.95 1930.6,-143.41 1920.58,-139.95 1920.61,-146.95\"/>\n",
       "</g>\n",
       "<!-- clusterrecoveryOutputsfailed -->\n",
       "<g id=\"node9\" class=\"node\">\n",
       "<title>clusterrecoveryOutputsfailed</title>\n",
       "<polygon fill=\"#21bfd8\" stroke=\"#21bfd8\" points=\"2424.39,-287.25 2382.14,-287.25 2382.14,-263.25 2424.39,-263.25 2436.39,-275.25 2424.39,-287.25\"/>\n",
       "<text text-anchor=\"middle\" x=\"2409.27\" y=\"-270.57\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">failed</text>\n",
       "</g>\n",
       "<!-- clusterrecoveryOutputsc_fails__add -->\n",
       "<g id=\"node10\" class=\"node\">\n",
       "<title>clusterrecoveryOutputsc_fails__add</title>\n",
       "<ellipse fill=\"none\" stroke=\"#edb22c\" stroke-width=\"2\" cx=\"2409.27\" cy=\"-383.25\" rx=\"69.1\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"2409.27\" y=\"-378.57\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">c_fails__add</text>\n",
       "</g>\n",
       "<!-- clusterrecoveryOutputsd_if_success__user_input -->\n",
       "<g id=\"node11\" class=\"node\">\n",
       "<title>clusterrecoveryOutputsd_if_success__user_input</title>\n",
       "<ellipse fill=\"none\" stroke=\"#edb22c\" stroke-width=\"2\" cx=\"2409.27\" cy=\"-437.25\" rx=\"132.76\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"2409.27\" y=\"-432.57\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">d_if_success__user_input</text>\n",
       "</g>\n",
       "<!-- clusterrecoveryOutputse_fails__add -->\n",
       "<g id=\"node12\" class=\"node\">\n",
       "<title>clusterrecoveryOutputse_fails__add</title>\n",
       "<ellipse fill=\"none\" stroke=\"#edb22c\" stroke-width=\"2\" cx=\"2409.27\" cy=\"-329.25\" rx=\"69.62\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"2409.27\" y=\"-324.57\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">e_fails__add</text>\n",
       "</g>\n",
       "<!-- clusterrecoveryaInputsrun -->\n",
       "<g id=\"node13\" class=\"node\">\n",
       "<title>clusterrecoveryaInputsrun</title>\n",
       "<polygon fill=\"#21bfd8\" stroke=\"#21bfd8\" points=\"412.51,-369.25 370.51,-369.25 370.51,-345.25 412.51,-345.25 424.51,-357.25 412.51,-369.25\"/>\n",
       "<text text-anchor=\"middle\" x=\"397.51\" y=\"-352.57\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">run</text>\n",
       "</g>\n",
       "<!-- clusterrecoveryaOutputsWithInjectionran -->\n",
       "<g id=\"node16\" class=\"node\">\n",
       "<title>clusterrecoveryaOutputsWithInjectionran</title>\n",
       "<polygon fill=\"#21bfd8\" stroke=\"#21bfd8\" points=\"609.51,-369.25 567.51,-369.25 567.51,-345.25 609.51,-345.25 621.51,-357.25 609.51,-369.25\"/>\n",
       "<text text-anchor=\"middle\" x=\"594.51\" y=\"-352.57\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">ran</text>\n",
       "</g>\n",
       "<!-- clusterrecoveryaInputsrun&#45;&gt;clusterrecoveryaOutputsWithInjectionran -->\n",
       "<!-- clusterrecoveryaInputsaccumulate_and_run -->\n",
       "<g id=\"node14\" class=\"node\">\n",
       "<title>clusterrecoveryaInputsaccumulate_and_run</title>\n",
       "<polygon fill=\"#21bfd8\" stroke=\"#21bfd8\" points=\"465.51,-423.25 317.51,-423.25 317.51,-399.25 465.51,-399.25 477.51,-411.25 465.51,-423.25\"/>\n",
       "<text text-anchor=\"middle\" x=\"397.51\" y=\"-406.57\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">accumulate_and_run</text>\n",
       "</g>\n",
       "<!-- clusterrecoverybInputsrun -->\n",
       "<g id=\"node19\" class=\"node\">\n",
       "<title>clusterrecoverybInputsrun</title>\n",
       "<polygon fill=\"#21bfd8\" stroke=\"#21bfd8\" points=\"803.51,-388.25 761.51,-388.25 761.51,-364.25 803.51,-364.25 815.51,-376.25 803.51,-388.25\"/>\n",
       "<text text-anchor=\"middle\" x=\"788.51\" y=\"-371.57\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">run</text>\n",
       "</g>\n",
       "<!-- clusterrecoveryaOutputsWithInjectionran&#45;&gt;clusterrecoverybInputsrun -->\n",
       "<g id=\"edge8\" class=\"edge\">\n",
       "<title>clusterrecoveryaOutputsWithInjectionran&#45;&gt;clusterrecoverybInputsrun</title>\n",
       "<path fill=\"none\" stroke=\"#21bfd8\" d=\"M621.8,-359.85C638.42,-361.5 661.08,-363.74 684.3,-366.04\"/>\n",
       "<path fill=\"none\" stroke=\"#21bfd8\" d=\"M684.3,-366.04C707.52,-368.33 731.31,-370.69 750.19,-372.56\"/>\n",
       "<polygon fill=\"#21bfd8\" stroke=\"#21bfd8\" points=\"749.42,-376 759.72,-373.5 750.11,-369.03 749.42,-376\"/>\n",
       "</g>\n",
       "<!-- clusterrecoveryaOutputsWithInjectionfailed -->\n",
       "<g id=\"node17\" class=\"node\">\n",
       "<title>clusterrecoveryaOutputsWithInjectionfailed</title>\n",
       "<polygon fill=\"#21bfd8\" stroke=\"#21bfd8\" points=\"609.64,-423.25 567.39,-423.25 567.39,-399.25 609.64,-399.25 621.64,-411.25 609.64,-423.25\"/>\n",
       "<text text-anchor=\"middle\" x=\"594.51\" y=\"-406.57\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">failed</text>\n",
       "</g>\n",
       "<!-- clusterrecoveryaOutputsWithInjectionuser_input -->\n",
       "<g id=\"node18\" class=\"node\">\n",
       "<title>clusterrecoveryaOutputsWithInjectionuser_input</title>\n",
       "<ellipse fill=\"#edb22c\" stroke=\"#edb22c\" cx=\"594.51\" cy=\"-465.25\" rx=\"62.78\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"594.51\" y=\"-460.57\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">user_input</text>\n",
       "</g>\n",
       "<!-- clusterrecoveryc_failsInputsobj -->\n",
       "<g id=\"node27\" class=\"node\">\n",
       "<title>clusterrecoveryc_failsInputsobj</title>\n",
       "<ellipse fill=\"#edb22c\" stroke=\"#edb22c\" cx=\"1179.51\" cy=\"-448.25\" rx=\"27\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"1179.51\" y=\"-443.57\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">obj</text>\n",
       "</g>\n",
       "<!-- clusterrecoveryaOutputsWithInjectionuser_input&#45;&gt;clusterrecoveryc_failsInputsobj -->\n",
       "<g id=\"edge9\" class=\"edge\">\n",
       "<title>clusterrecoveryaOutputsWithInjectionuser_input&#45;&gt;clusterrecoveryc_failsInputsobj</title>\n",
       "<path fill=\"none\" stroke=\"#edb22c\" d=\"M657.35,-463.45C717.76,-461.69 814.02,-458.88 907.45,-456.15\"/>\n",
       "<path fill=\"none\" stroke=\"#edb22c\" d=\"M907.45,-456.15C1000.89,-453.43 1091.52,-450.79 1140.67,-449.35\"/>\n",
       "<polygon fill=\"#edb22c\" stroke=\"#edb22c\" points=\"1140.67,-452.85 1150.57,-449.06 1140.47,-445.86 1140.67,-452.85\"/>\n",
       "</g>\n",
       "<!-- clusterrecoverybOutputsWithInjectionran -->\n",
       "<g id=\"node22\" class=\"node\">\n",
       "<title>clusterrecoverybOutputsWithInjectionran</title>\n",
       "<polygon fill=\"#21bfd8\" stroke=\"#21bfd8\" points=\"1000.51,-280.25 958.51,-280.25 958.51,-256.25 1000.51,-256.25 1012.51,-268.25 1000.51,-280.25\"/>\n",
       "<text text-anchor=\"middle\" x=\"985.51\" y=\"-263.57\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">ran</text>\n",
       "</g>\n",
       "<!-- clusterrecoverybInputsrun&#45;&gt;clusterrecoverybOutputsWithInjectionran -->\n",
       "<!-- clusterrecoverybInputsaccumulate_and_run -->\n",
       "<g id=\"node20\" class=\"node\">\n",
       "<title>clusterrecoverybInputsaccumulate_and_run</title>\n",
       "<polygon fill=\"#21bfd8\" stroke=\"#21bfd8\" points=\"856.51,-334.25 708.51,-334.25 708.51,-310.25 856.51,-310.25 868.51,-322.25 856.51,-334.25\"/>\n",
       "<text text-anchor=\"middle\" x=\"788.51\" y=\"-317.57\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">accumulate_and_run</text>\n",
       "</g>\n",
       "<!-- clusterrecoveryc_failsInputsrun -->\n",
       "<g id=\"node25\" class=\"node\">\n",
       "<title>clusterrecoveryc_failsInputsrun</title>\n",
       "<polygon fill=\"#21bfd8\" stroke=\"#21bfd8\" points=\"1194.51,-298.25 1152.51,-298.25 1152.51,-274.25 1194.51,-274.25 1206.51,-286.25 1194.51,-298.25\"/>\n",
       "<text text-anchor=\"middle\" x=\"1179.51\" y=\"-281.57\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">run</text>\n",
       "</g>\n",
       "<!-- clusterrecoverybOutputsWithInjectionran&#45;&gt;clusterrecoveryc_failsInputsrun -->\n",
       "<g id=\"edge10\" class=\"edge\">\n",
       "<title>clusterrecoverybOutputsWithInjectionran&#45;&gt;clusterrecoveryc_failsInputsrun</title>\n",
       "<path fill=\"none\" stroke=\"#21bfd8\" d=\"M1012.8,-270.72C1029.42,-272.27 1052.08,-274.4 1075.3,-276.57\"/>\n",
       "<path fill=\"none\" stroke=\"#21bfd8\" d=\"M1075.3,-276.57C1098.52,-278.75 1122.31,-280.98 1141.19,-282.75\"/>\n",
       "<polygon fill=\"#21bfd8\" stroke=\"#21bfd8\" points=\"1140.44,-286.2 1150.72,-283.64 1141.09,-279.23 1140.44,-286.2\"/>\n",
       "</g>\n",
       "<!-- clusterrecoverybOutputsWithInjectionfailed -->\n",
       "<g id=\"node23\" class=\"node\">\n",
       "<title>clusterrecoverybOutputsWithInjectionfailed</title>\n",
       "<polygon fill=\"#21bfd8\" stroke=\"#21bfd8\" points=\"1000.64,-334.25 958.39,-334.25 958.39,-310.25 1000.64,-310.25 1012.64,-322.25 1000.64,-334.25\"/>\n",
       "<text text-anchor=\"middle\" x=\"985.51\" y=\"-317.57\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">failed</text>\n",
       "</g>\n",
       "<!-- clusterrecoverybOutputsWithInjectionuser_input -->\n",
       "<g id=\"node24\" class=\"node\">\n",
       "<title>clusterrecoverybOutputsWithInjectionuser_input</title>\n",
       "<ellipse fill=\"#edb22c\" stroke=\"#edb22c\" cx=\"985.51\" cy=\"-376.25\" rx=\"62.78\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"985.51\" y=\"-371.57\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">user_input</text>\n",
       "</g>\n",
       "<!-- clusterrecoveryc_failsInputsother -->\n",
       "<g id=\"node28\" class=\"node\">\n",
       "<title>clusterrecoveryc_failsInputsother</title>\n",
       "<ellipse fill=\"#edb22c\" stroke=\"#edb22c\" cx=\"1179.51\" cy=\"-394.25\" rx=\"37\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"1179.51\" y=\"-389.57\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">other</text>\n",
       "</g>\n",
       "<!-- clusterrecoverybOutputsWithInjectionuser_input&#45;&gt;clusterrecoveryc_failsInputsother -->\n",
       "<g id=\"edge11\" class=\"edge\">\n",
       "<title>clusterrecoverybOutputsWithInjectionuser_input&#45;&gt;clusterrecoveryc_failsInputsother</title>\n",
       "<path fill=\"none\" stroke=\"#edb22c\" d=\"M1045.68,-381.8C1059.44,-383.09 1074.39,-384.49 1089.13,-385.87\"/>\n",
       "<path fill=\"none\" stroke=\"#edb22c\" d=\"M1089.13,-385.87C1103.87,-387.25 1118.4,-388.61 1131.32,-389.83\"/>\n",
       "<polygon fill=\"#edb22c\" stroke=\"#edb22c\" points=\"1130.92,-393.3 1141.2,-390.75 1131.57,-386.33 1130.92,-393.3\"/>\n",
       "</g>\n",
       "<!-- clusterrecoveryc_failsOutputsWithInjectionran -->\n",
       "<g id=\"node29\" class=\"node\">\n",
       "<title>clusterrecoveryc_failsOutputsWithInjectionran</title>\n",
       "<polygon fill=\"#21bfd8\" stroke=\"#21bfd8\" points=\"1391.51,-442.25 1349.51,-442.25 1349.51,-418.25 1391.51,-418.25 1403.51,-430.25 1391.51,-442.25\"/>\n",
       "<text text-anchor=\"middle\" x=\"1376.51\" y=\"-425.57\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">ran</text>\n",
       "</g>\n",
       "<!-- clusterrecoveryc_failsInputsrun&#45;&gt;clusterrecoveryc_failsOutputsWithInjectionran -->\n",
       "<!-- clusterrecoveryc_failsInputsaccumulate_and_run -->\n",
       "<g id=\"node26\" class=\"node\">\n",
       "<title>clusterrecoveryc_failsInputsaccumulate_and_run</title>\n",
       "<polygon fill=\"#21bfd8\" stroke=\"#21bfd8\" points=\"1247.51,-352.25 1099.51,-352.25 1099.51,-328.25 1247.51,-328.25 1259.51,-340.25 1247.51,-352.25\"/>\n",
       "<text text-anchor=\"middle\" x=\"1179.51\" y=\"-335.57\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">accumulate_and_run</text>\n",
       "</g>\n",
       "<!-- clusterrecoveryd_if_successInputsrun -->\n",
       "<g id=\"node32\" class=\"node\">\n",
       "<title>clusterrecoveryd_if_successInputsrun</title>\n",
       "<polygon fill=\"#21bfd8\" stroke=\"#21bfd8\" points=\"1585.51,-471.25 1543.51,-471.25 1543.51,-447.25 1585.51,-447.25 1597.51,-459.25 1585.51,-471.25\"/>\n",
       "<text text-anchor=\"middle\" x=\"1570.51\" y=\"-454.57\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">run</text>\n",
       "</g>\n",
       "<!-- clusterrecoveryc_failsOutputsWithInjectionran&#45;&gt;clusterrecoveryd_if_successInputsrun -->\n",
       "<g id=\"edge12\" class=\"edge\">\n",
       "<title>clusterrecoveryc_failsOutputsWithInjectionran&#45;&gt;clusterrecoveryd_if_successInputsrun</title>\n",
       "<path fill=\"none\" stroke=\"#21bfd8\" d=\"M1403.8,-434.22C1420.42,-436.73 1443.08,-440.15 1466.3,-443.66\"/>\n",
       "<path fill=\"none\" stroke=\"#21bfd8\" d=\"M1466.3,-443.66C1489.52,-447.17 1513.31,-450.76 1532.19,-453.61\"/>\n",
       "<polygon fill=\"#21bfd8\" stroke=\"#21bfd8\" points=\"1531.32,-457.02 1541.73,-455.05 1532.36,-450.1 1531.32,-457.02\"/>\n",
       "</g>\n",
       "<!-- clusterrecoveryc_failsOutputsWithInjectionfailed -->\n",
       "<g id=\"node30\" class=\"node\">\n",
       "<title>clusterrecoveryc_failsOutputsWithInjectionfailed</title>\n",
       "<polygon fill=\"#21bfd8\" stroke=\"#21bfd8\" points=\"1391.64,-334.25 1349.39,-334.25 1349.39,-310.25 1391.64,-310.25 1403.64,-322.25 1391.64,-334.25\"/>\n",
       "<text text-anchor=\"middle\" x=\"1376.51\" y=\"-317.57\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">failed</text>\n",
       "</g>\n",
       "<!-- clusterrecoveryd_if_failureInputsrun -->\n",
       "<g id=\"node38\" class=\"node\">\n",
       "<title>clusterrecoveryd_if_failureInputsrun</title>\n",
       "<polygon fill=\"#21bfd8\" stroke=\"#21bfd8\" points=\"1585.51,-317.25 1543.51,-317.25 1543.51,-293.25 1585.51,-293.25 1597.51,-305.25 1585.51,-317.25\"/>\n",
       "<text text-anchor=\"middle\" x=\"1570.51\" y=\"-300.57\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">run</text>\n",
       "</g>\n",
       "<!-- clusterrecoveryc_failsOutputsWithInjectionfailed&#45;&gt;clusterrecoveryd_if_failureInputsrun -->\n",
       "<g id=\"edge13\" class=\"edge\">\n",
       "<title>clusterrecoveryc_failsOutputsWithInjectionfailed&#45;&gt;clusterrecoveryd_if_failureInputsrun</title>\n",
       "<path fill=\"none\" stroke=\"#21bfd8\" d=\"M1403.8,-319.92C1420.42,-318.45 1443.08,-316.44 1466.3,-314.39\"/>\n",
       "<path fill=\"none\" stroke=\"#21bfd8\" d=\"M1466.3,-314.39C1489.52,-312.33 1513.31,-310.23 1532.19,-308.55\"/>\n",
       "<polygon fill=\"#21bfd8\" stroke=\"#21bfd8\" points=\"1532.07,-312.08 1541.72,-307.71 1531.45,-305.11 1532.07,-312.08\"/>\n",
       "</g>\n",
       "<!-- clusterrecoveryc_failsOutputsWithInjectionadd -->\n",
       "<g id=\"node31\" class=\"node\">\n",
       "<title>clusterrecoveryc_failsOutputsWithInjectionadd</title>\n",
       "<ellipse fill=\"none\" stroke=\"#edb22c\" stroke-width=\"2\" cx=\"1376.51\" cy=\"-376.25\" rx=\"29.64\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"1376.51\" y=\"-371.57\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">add</text>\n",
       "</g>\n",
       "<!-- clusterrecoveryc_failsOutputsWithInjectionadd&#45;&gt;clusterrecoveryOutputsc_fails__add -->\n",
       "<g id=\"edge21\" class=\"edge\">\n",
       "<title>clusterrecoveryc_failsOutputsWithInjectionadd&#45;&gt;clusterrecoveryOutputsc_fails__add</title>\n",
       "<path fill=\"none\" stroke=\"#edb22c\" d=\"M1405.85,-379.99C1470.89,-388.23 1635.9,-407.25 1774.51,-407.25 1774.51,-407.25 1774.51,-407.25 1796.1,-407.25\"/>\n",
       "<path fill=\"none\" stroke=\"#edb22c\" d=\"M1796.1,-407.25C1819.55,-407.25 1868.46,-407.25 1970.51,-407.25 2097.44,-407.25 2244.76,-397.07 2332.04,-389.97\"/>\n",
       "<polygon fill=\"#edb22c\" stroke=\"#edb22c\" points=\"2332.24,-393.46 2341.92,-389.15 2331.67,-386.48 2332.24,-393.46\"/>\n",
       "</g>\n",
       "<!-- clusterrecoveryd_if_successOutputsWithInjectionran -->\n",
       "<g id=\"node35\" class=\"node\">\n",
       "<title>clusterrecoveryd_if_successOutputsWithInjectionran</title>\n",
       "<polygon fill=\"#21bfd8\" stroke=\"#21bfd8\" points=\"1790.51,-471.25 1748.51,-471.25 1748.51,-447.25 1790.51,-447.25 1802.51,-459.25 1790.51,-471.25\"/>\n",
       "<text text-anchor=\"middle\" x=\"1775.51\" y=\"-454.57\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">ran</text>\n",
       "</g>\n",
       "<!-- clusterrecoveryd_if_successInputsrun&#45;&gt;clusterrecoveryd_if_successOutputsWithInjectionran -->\n",
       "<!-- clusterrecoveryd_if_successInputsaccumulate_and_run -->\n",
       "<g id=\"node33\" class=\"node\">\n",
       "<title>clusterrecoveryd_if_successInputsaccumulate_and_run</title>\n",
       "<polygon fill=\"#21bfd8\" stroke=\"#21bfd8\" points=\"1638.51,-525.25 1490.51,-525.25 1490.51,-501.25 1638.51,-501.25 1650.51,-513.25 1638.51,-525.25\"/>\n",
       "<text text-anchor=\"middle\" x=\"1570.51\" y=\"-508.57\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">accumulate_and_run</text>\n",
       "</g>\n",
       "<!-- clusterrecoveryd_if_successOutputsWithInjectionfailed -->\n",
       "<g id=\"node36\" class=\"node\">\n",
       "<title>clusterrecoveryd_if_successOutputsWithInjectionfailed</title>\n",
       "<polygon fill=\"#21bfd8\" stroke=\"#21bfd8\" points=\"1790.64,-525.25 1748.39,-525.25 1748.39,-501.25 1790.64,-501.25 1802.64,-513.25 1790.64,-525.25\"/>\n",
       "<text text-anchor=\"middle\" x=\"1775.51\" y=\"-508.57\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">failed</text>\n",
       "</g>\n",
       "<!-- clusterrecoveryd_if_successOutputsWithInjectionuser_input -->\n",
       "<g id=\"node37\" class=\"node\">\n",
       "<title>clusterrecoveryd_if_successOutputsWithInjectionuser_input</title>\n",
       "<ellipse fill=\"none\" stroke=\"#edb22c\" stroke-width=\"2\" cx=\"1775.51\" cy=\"-567.25\" rx=\"62.78\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"1775.51\" y=\"-562.58\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">user_input</text>\n",
       "</g>\n",
       "<!-- clusterrecoveryd_if_successOutputsWithInjectionuser_input&#45;&gt;clusterrecoveryOutputsd_if_success__user_input -->\n",
       "<g id=\"edge22\" class=\"edge\">\n",
       "<title>clusterrecoveryd_if_successOutputsWithInjectionuser_input&#45;&gt;clusterrecoveryOutputsd_if_success__user_input</title>\n",
       "<path fill=\"none\" stroke=\"#edb22c\" d=\"M1827.41,-556.78C1881.53,-545.64 1972.12,-527 2065.84,-507.71\"/>\n",
       "<path fill=\"none\" stroke=\"#edb22c\" d=\"M2065.84,-507.71C2159.56,-488.43 2256.42,-468.5 2323.1,-454.77\"/>\n",
       "<polygon fill=\"#edb22c\" stroke=\"#edb22c\" points=\"2323.49,-458.27 2332.58,-452.83 2322.08,-451.41 2323.49,-458.27\"/>\n",
       "</g>\n",
       "<!-- clusterrecoveryd_if_failureOutputsWithInjectionran -->\n",
       "<g id=\"node41\" class=\"node\">\n",
       "<title>clusterrecoveryd_if_failureOutputsWithInjectionran</title>\n",
       "<polygon fill=\"#21bfd8\" stroke=\"#21bfd8\" points=\"1790.51,-209.25 1748.51,-209.25 1748.51,-185.25 1790.51,-185.25 1802.51,-197.25 1790.51,-209.25\"/>\n",
       "<text text-anchor=\"middle\" x=\"1775.51\" y=\"-192.57\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">ran</text>\n",
       "</g>\n",
       "<!-- clusterrecoveryd_if_failureInputsrun&#45;&gt;clusterrecoveryd_if_failureOutputsWithInjectionran -->\n",
       "<!-- clusterrecoveryd_if_failureInputsaccumulate_and_run -->\n",
       "<g id=\"node39\" class=\"node\">\n",
       "<title>clusterrecoveryd_if_failureInputsaccumulate_and_run</title>\n",
       "<polygon fill=\"#21bfd8\" stroke=\"#21bfd8\" points=\"1638.51,-263.25 1490.51,-263.25 1490.51,-239.25 1638.51,-239.25 1650.51,-251.25 1638.51,-263.25\"/>\n",
       "<text text-anchor=\"middle\" x=\"1570.51\" y=\"-246.57\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">accumulate_and_run</text>\n",
       "</g>\n",
       "<!-- clusterrecoverye_failsInputsrun -->\n",
       "<g id=\"node44\" class=\"node\">\n",
       "<title>clusterrecoverye_failsInputsrun</title>\n",
       "<polygon fill=\"#21bfd8\" stroke=\"#21bfd8\" points=\"1984.51,-263.25 1942.51,-263.25 1942.51,-239.25 1984.51,-239.25 1996.51,-251.25 1984.51,-263.25\"/>\n",
       "<text text-anchor=\"middle\" x=\"1969.51\" y=\"-246.57\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">run</text>\n",
       "</g>\n",
       "<!-- clusterrecoveryd_if_failureOutputsWithInjectionran&#45;&gt;clusterrecoverye_failsInputsrun -->\n",
       "<g id=\"edge14\" class=\"edge\">\n",
       "<title>clusterrecoveryd_if_failureOutputsWithInjectionran&#45;&gt;clusterrecoverye_failsInputsrun</title>\n",
       "<path fill=\"none\" stroke=\"#21bfd8\" d=\"M1802.8,-204.65C1819.42,-209.32 1842.08,-215.69 1865.3,-222.22\"/>\n",
       "<path fill=\"none\" stroke=\"#21bfd8\" d=\"M1865.3,-222.22C1888.52,-228.75 1912.31,-235.44 1931.19,-240.75\"/>\n",
       "<polygon fill=\"#21bfd8\" stroke=\"#21bfd8\" points=\"1930.19,-244.11 1940.77,-243.45 1932.09,-237.37 1930.19,-244.11\"/>\n",
       "</g>\n",
       "<!-- clusterrecoveryd_if_failureOutputsWithInjectionfailed -->\n",
       "<g id=\"node42\" class=\"node\">\n",
       "<title>clusterrecoveryd_if_failureOutputsWithInjectionfailed</title>\n",
       "<polygon fill=\"#21bfd8\" stroke=\"#21bfd8\" points=\"1790.64,-263.25 1748.39,-263.25 1748.39,-239.25 1790.64,-239.25 1802.64,-251.25 1790.64,-263.25\"/>\n",
       "<text text-anchor=\"middle\" x=\"1775.51\" y=\"-246.57\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">failed</text>\n",
       "</g>\n",
       "<!-- clusterrecoveryd_if_failureOutputsWithInjectionuser_input -->\n",
       "<g id=\"node43\" class=\"node\">\n",
       "<title>clusterrecoveryd_if_failureOutputsWithInjectionuser_input</title>\n",
       "<ellipse fill=\"#edb22c\" stroke=\"#edb22c\" cx=\"1775.51\" cy=\"-305.25\" rx=\"62.78\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"1775.51\" y=\"-300.57\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">user_input</text>\n",
       "</g>\n",
       "<!-- clusterrecoverye_failsInputsobj -->\n",
       "<g id=\"node46\" class=\"node\">\n",
       "<title>clusterrecoverye_failsInputsobj</title>\n",
       "<ellipse fill=\"#edb22c\" stroke=\"#edb22c\" cx=\"1969.51\" cy=\"-305.25\" rx=\"27\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"1969.51\" y=\"-300.57\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">obj</text>\n",
       "</g>\n",
       "<!-- clusterrecoveryd_if_failureOutputsWithInjectionuser_input&#45;&gt;clusterrecoverye_failsInputsobj -->\n",
       "<g id=\"edge15\" class=\"edge\">\n",
       "<title>clusterrecoveryd_if_failureOutputsWithInjectionuser_input&#45;&gt;clusterrecoverye_failsInputsobj</title>\n",
       "<path fill=\"none\" stroke=\"#edb22c\" d=\"M1838.58,-305.25C1853.78,-305.25 1870.29,-305.25 1886.27,-305.25\"/>\n",
       "<path fill=\"none\" stroke=\"#edb22c\" d=\"M1886.27,-305.25C1902.26,-305.25 1917.72,-305.25 1930.84,-305.25\"/>\n",
       "<polygon fill=\"#edb22c\" stroke=\"#edb22c\" points=\"1930.68,-308.75 1940.68,-305.25 1930.68,-301.75 1930.68,-308.75\"/>\n",
       "</g>\n",
       "<!-- clusterrecoverye_failsOutputsWithInjectionran -->\n",
       "<g id=\"node48\" class=\"node\">\n",
       "<title>clusterrecoverye_failsOutputsWithInjectionran</title>\n",
       "<polygon fill=\"#21bfd8\" stroke=\"#21bfd8\" points=\"2181.51,-209.25 2139.51,-209.25 2139.51,-185.25 2181.51,-185.25 2193.51,-197.25 2181.51,-209.25\"/>\n",
       "<text text-anchor=\"middle\" x=\"2166.51\" y=\"-192.57\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">ran</text>\n",
       "</g>\n",
       "<!-- clusterrecoverye_failsInputsrun&#45;&gt;clusterrecoverye_failsOutputsWithInjectionran -->\n",
       "<!-- clusterrecoverye_failsInputsaccumulate_and_run -->\n",
       "<g id=\"node45\" class=\"node\">\n",
       "<title>clusterrecoverye_failsInputsaccumulate_and_run</title>\n",
       "<polygon fill=\"#21bfd8\" stroke=\"#21bfd8\" points=\"2037.51,-209.25 1889.51,-209.25 1889.51,-185.25 2037.51,-185.25 2049.51,-197.25 2037.51,-209.25\"/>\n",
       "<text text-anchor=\"middle\" x=\"1969.51\" y=\"-192.57\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">accumulate_and_run</text>\n",
       "</g>\n",
       "<!-- clusterrecoverye_failsOutputsWithInjectionfailed -->\n",
       "<g id=\"node49\" class=\"node\">\n",
       "<title>clusterrecoverye_failsOutputsWithInjectionfailed</title>\n",
       "<polygon fill=\"#21bfd8\" stroke=\"#21bfd8\" points=\"2181.64,-263.25 2139.39,-263.25 2139.39,-239.25 2181.64,-239.25 2193.64,-251.25 2181.64,-263.25\"/>\n",
       "<text text-anchor=\"middle\" x=\"2166.51\" y=\"-246.57\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">failed</text>\n",
       "</g>\n",
       "<!-- clusterrecoverye_failsOutputsWithInjectionadd -->\n",
       "<g id=\"node50\" class=\"node\">\n",
       "<title>clusterrecoverye_failsOutputsWithInjectionadd</title>\n",
       "<ellipse fill=\"none\" stroke=\"#edb22c\" stroke-width=\"2\" cx=\"2166.51\" cy=\"-305.25\" rx=\"29.64\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"2166.51\" y=\"-300.57\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">add</text>\n",
       "</g>\n",
       "<!-- clusterrecoverye_failsOutputsWithInjectionadd&#45;&gt;clusterrecoveryOutputse_fails__add -->\n",
       "<g id=\"edge23\" class=\"edge\">\n",
       "<title>clusterrecoverye_failsOutputsWithInjectionadd&#45;&gt;clusterrecoveryOutputse_fails__add</title>\n",
       "<path fill=\"none\" stroke=\"#edb22c\" d=\"M2196.07,-308.1C2212.76,-309.76 2235.24,-312 2259.34,-314.4\"/>\n",
       "<path fill=\"none\" stroke=\"#edb22c\" d=\"M2259.34,-314.4C2283.44,-316.81 2309.16,-319.37 2332.32,-321.68\"/>\n",
       "<polygon fill=\"#edb22c\" stroke=\"#edb22c\" points=\"2331.75,-325.14 2342.05,-322.65 2332.44,-318.17 2331.75,-325.14\"/>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<graphviz.graphs.Digraph at 0x145f001f9210>"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "wf = pwf.Workflow(\"recovery\")\n",
    "wf.a = pwf.standard_nodes.UserInput(1)\n",
    "wf.b = pwf.standard_nodes.UserInput(\"two\")\n",
    "wf.c_fails = wf.a + wf.b  # Type error\n",
    "wf.d_if_success = pwf.standard_nodes.UserInput(0)\n",
    "wf.d_if_failure = pwf.standard_nodes.UserInput(\"But what's the question?\")\n",
    "wf.e_fails = pwf.standard_nodes.Add(wf.d_if_failure, 42)  # Type error\n",
    "\n",
    "wf.a >> wf.b >> wf.c_fails >> wf.d_if_success\n",
    "wf.c_fails.signals.output.failed >> wf.d_if_failure >> wf.e_fails\n",
    "wf.starting_nodes = [wf.a]\n",
    "wf.automate_execution = False\n",
    "\n",
    "submit_to_slurm(wf)\n",
    "\n",
    "while not wf.has_saved_content():\n",
    "    sleep(0.2)\n",
    "reloaded = pwf.Workflow(wf.label)\n",
    "    \n",
    "while not reloaded.e_fails.failed:\n",
    "    if reloaded.has_saved_content(\n",
    "        filename=wf.as_path().joinpath(\"recovery\")\n",
    "    ):\n",
    "        reloaded.load(\n",
    "            filename=wf.as_path().joinpath(\"recovery\")\n",
    "        )\n",
    "    else:\n",
    "        sleep(1)\n",
    "reloaded.draw(size=(10, 10))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "1fb2bfd3-4f85-4ef9-8882-c45cd22e70af",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "recovery failed? True\n",
      "c_fails failed\n",
      "e_fails failed\n"
     ]
    }
   ],
   "source": [
    "print(reloaded.label, \"failed?\", reloaded.failed)\n",
    "for child in reloaded:\n",
    "    if child.failed:\n",
    "        print(child.label, \"failed\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "72f119be-22da-4439-b6d6-9bb441250eee",
   "metadata": {},
   "source": [
    "We can see that the recovery file saved for us the final breath of our failed workflow."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "b2d3a39c-c4cf-4c88-a1e1-dd99b9b69c43",
   "metadata": {},
   "outputs": [],
   "source": [
    "delete_path(wf.as_path())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a510bb3e-761b-480c-929f-41884bc55f17",
   "metadata": {},
   "source": [
    "### Executors inside an HPC job\n",
    "\n",
    "Executor objects can't be saved and loaded -- consequently they get purged in the save process. So while we could write an entire `.py` script that defines and runs our workflow, including setting executors as usual, and ship the _whole_ `.py` script off as a slurm job, our little helper function won't help -- if we set executors in the naive way they'll be stripped off at the `.save()` step!\n",
    "\n",
    "To get around this, instead of setting an executor instance explicitly, `pyiron_workflow` instead allows us to set _instructions_ for how to make an executor, in the form `callable, tuple, dict` for the constructor, args, and kwargs. The simplest thing is just to pass an executor class, like `concurrent.futures.ProcessPoolExecutor, (), {\"max_workers\": 4}`, but any callable will do. Since we can use any callable we can even exploit this to give multiple nodes the same executor instance -- e.g. by having a separate `.py` file that instantiates a global executor, and pass a method that imports this. There is no hand-holding here, so if you want to do something fancy you need to do it yourself, but the framework is intended to give enough flexibility to get done what you need to get done.\n",
    "\n",
    "In our final example we'll explore this by running our `Sleep` nodes with a thread pool executor so they can all sleep at (more or less) the same time. We'll also do something else a bit different here: instead of sending a `Workflow` off, we'll send a particular node. In this case, a `For`-node, and we'll use the `body_node_executor` attribute to pass our executor (well, executor building instructions) to each child `Sleep` node individually."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "8dc56a18-59ac-45b1-ace6-efefae2d3ef0",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Submitted batch job 9445786\n",
      "Total duration (s) = 12.557957572862506\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>t</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>6</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>6</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>6</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>6</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>6</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>95</th>\n",
       "      <td>6</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>96</th>\n",
       "      <td>6</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>97</th>\n",
       "      <td>6</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>98</th>\n",
       "      <td>6</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>99</th>\n",
       "      <td>6</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>100 rows × 2 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "    t  time\n",
       "0   6     6\n",
       "1   6     6\n",
       "2   6     6\n",
       "3   6     6\n",
       "4   6     6\n",
       ".. ..   ...\n",
       "95  6     6\n",
       "96  6     6\n",
       "97  6     6\n",
       "98  6     6\n",
       "99  6     6\n",
       "\n",
       "[100 rows x 2 columns]"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from concurrent.futures import ThreadPoolExecutor\n",
    "import time\n",
    "\n",
    "# Build the workflow\n",
    "looped = pwf.standard_nodes.Sleep.for_node(\n",
    "    iter_on=(\"t\",),\n",
    "    t=100*[6],\n",
    ")\n",
    "looped.body_node_executor = (ThreadPoolExecutor, (), {})\n",
    "\n",
    "# Start timing \n",
    "t0 = perf_counter()\n",
    "\n",
    "# Send it off\n",
    "submit_to_slurm(looped)\n",
    "\n",
    "# Try reloading it (need to wait for `submit_to_slurm`\n",
    "while not looped.has_saved_content():\n",
    "    sleep(0.2)\n",
    "reloaded = pwf.standard_nodes.Sleep.for_node(\n",
    "    iter_on=(\"t\",),\n",
    "    label=looped.label,\n",
    ")\n",
    "reloaded.load()\n",
    "\n",
    "# Reload it until it's done\n",
    "while reloaded.outputs.df.value is pwf.NOT_DATA:\n",
    "    reloaded.load()\n",
    "    sleep(1)\n",
    "\n",
    "# Stop timing\n",
    "dt = perf_counter() - t0\n",
    "\n",
    "print(\"Total duration (s) =\", dt)\n",
    "reloaded.outputs.df.value"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "82094cad-00ee-4543-8f30-b9867f610798",
   "metadata": {},
   "source": [
    "With 100 six-second sleep nodes, in the absence of parallelization we would expect this workflow to take about 10 minutes to complete -- here it finishes in 10 seconds. There is some overhead to shipping a workflow off to SLURM, both in terms of (de)serialization and waiting in the queue, but if you need parallel processing beyond what you can get on the head node, this is the way to go.\n",
    "\n",
    "The example here was a simple python command with thread parallelization, but there is no barrier to exploiting nodes that `subprocess.run([\"mpiexec\", \"-n\", ...])` inside a `ProcessPoolExecutor` and really get cooking. You should be able to leverage multiple compute nodes with `executorlib` executors, but this is not tested yet.\n",
    "\n",
    "The only \"gotcha\" to remember is that _nesting_ executors is still not possible -- once an executor has been applied to a node, no executor can be applied to any of that node's children. (This may be forthcoming in `executorlib` under certain conditions, so watch there for independent developments.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "361ae81c-0ea8-4658-9d4f-61ca98522a4f",
   "metadata": {},
   "outputs": [],
   "source": [
    "delete_path(looped.as_path())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d4e1270c-84ac-4fb7-ae17-33cf8862f048",
   "metadata": {},
   "source": [
    "## Executing a single node remotely\n",
    "\n",
    "Instead of running the entire graph on remote resources, we might want to run just a single node remotely. This is the standard way of parallelizing graphs, and is done by the node's `.executor` attribute -- as long as we have an `Executor` that knows how to send calculations to remote compute! Conveniently, `executorlib.Executor` can do exactly this, e.g."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "033c4ea5-c54b-4e72-9b4c-38650e8a451e",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "srun: job 9445792 queued and waiting for resources\n",
      "srun: job 9445792 has been allocated resources\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'skipped_my_nap__rmul': 0, 'floating_dreams__float': 5.0}"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "wf = pwf.Workflow(\"living_process\")\n",
    "wf.sleep_time = pwf.standard_nodes.UserInput(5)\n",
    "wf.skipped_my_nap = 0 * wf.sleep_time\n",
    "wf.sleep = pwf.standard_nodes.Sleep(wf.sleep_time)\n",
    "wf.floating_dreams = pwf.standard_nodes.Float(wf.sleep)\n",
    "\n",
    "with pwf.Workflow.create.ExecutorlibExecutor(\n",
    "    max_cores=1, \n",
    "    backend=\"slurm\", \n",
    "    command_line_argument_lst=[\n",
    "        \"-n 1\",\n",
    "        \"-N 1\",\n",
    "        \"--account=mpd\",\n",
    "        \"--partition=s.cmmg\",\n",
    "        \"--time=00:00:30\",\n",
    "    ]\n",
    ") as exe:\n",
    "    wf.sleep.executor = exe\n",
    "    wf()\n",
    "\n",
    "wf.outputs.to_value_dict()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "51410311-f017-40bb-bf69-e266d208cf6b",
   "metadata": {},
   "source": [
    "If we go look at our slurm queue, we can see that `wf.sleep` did indeed get run on the cluster rather than locally in this python process. So running a node on the cluster really is that straightforward. But, here's the catch: this only works while the parent python process (i.e. this notebook) stays alive for the duration of the calculation?\n",
    "\n",
    "Since we're probably submitting things to HCP because they are computationally expensive, how do we handle submitting an individual node to remote processing and then _killing the parent python process?_ E.g. we start our workflow running, but shutdown our notebook once it's stuck waiting for expensive jobs in the cluster queue."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0f43b91a-6441-47bd-bb3a-4406c83d5f44",
   "metadata": {},
   "source": [
    "### Restarting the local processes\n",
    "\n",
    "From the `pyiron_workflow` side, the answer is relatively straightforward: we need to tell the executor not just to run the node as usual, but also to create a serialized dump of the run result.\n",
    "\n",
    "If the parent process stays alive until the `Future` returned by the executor finishes, we process it as usual and it's like nothing was different -- just like the simple `executorlib` example above. However, if the parent process _dies_, and we reload the checkpoint file, we'll find that the node is still in the \"running\" state. At this point, we want to see if the extra serialzied result we asked for is present (in which case we can load it and keep going), or not (in which case we can simply fail because the workflow is not ready to keep going). \n",
    "\n",
    "As before, a real and robust interface for this process should certainly leverage `pysqa`, and may also leverage `executorlib.FileExecutor`. In this case, we just make the bare minimal \"executor\" that will use cloudpickle to run our node's `on_run` in an independent python process:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "4b4e57ed-c1d8-4965-bd15-e44b185b76aa",
   "metadata": {},
   "outputs": [],
   "source": [
    "from concurrent.futures import Executor, Future\n",
    "import os\n",
    "from subprocess import Popen\n",
    "\n",
    "import cloudpickle\n",
    "\n",
    "class FakeExecutor(Executor):\n",
    "    \"\"\"\n",
    "    Just a silly executor that uses cloudpickle to start a function\n",
    "    on an independent python process -- the future will never even resolve!\n",
    "    \"\"\"\n",
    "    def submit(self, fn, /, *args, **kwargs):\n",
    "        with open(\"fake_executor_data.cpkl\", \"wb\") as f:\n",
    "            cloudpickle.dump((fn, args, kwargs), f)\n",
    "\n",
    "        commands = \"\"\"\n",
    "import cloudpickle\n",
    "with open(\"fake_executor_data.cpkl\", \"rb\") as f:\n",
    "    fn, args, kwargs = cloudpickle.load(f)\n",
    "fn(*args, **kwargs)\n",
    "\"\"\"\n",
    "        Popen([\"python\", \"-c\", commands], preexec_fn=os.setsid)\n",
    "        return Future()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "177d242c-394f-4c63-9706-49a87beee105",
   "metadata": {},
   "source": [
    "We'll build a super simple workflow, and then manually kill it while the slow, remotely executed node is still running. Since the node can't tell from the mere presence of an executor that this executor can survive the death of the parent python process, we need to manually let the relevant nodes know they should `._serialize_result`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "3e5c5e64-e723-43c3-931c-5990761f8884",
   "metadata": {},
   "outputs": [
    {
     "ename": "KeyboardInterrupt",
     "evalue": "",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mIndexError\u001b[0m                                Traceback (most recent call last)",
      "File \u001b[0;32m~/greyhaven/pyiron_workflow/pyiron_workflow/nodes/composite.py:168\u001b[0m, in \u001b[0;36mComposite._run_while_children_or_signals_exist\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m    167\u001b[0m \u001b[38;5;28;01mtry\u001b[39;00m:\n\u001b[0;32m--> 168\u001b[0m     firing, receiving \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43msignal_queue\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mpop\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;241;43m0\u001b[39;49m\u001b[43m)\u001b[49m\n\u001b[1;32m    169\u001b[0m     \u001b[38;5;28;01mtry\u001b[39;00m:\n",
      "\u001b[0;31mIndexError\u001b[0m: pop from empty list",
      "\nDuring handling of the above exception, another exception occurred:\n",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m                         Traceback (most recent call last)",
      "Cell \u001b[0;32mIn[16], line 10\u001b[0m\n\u001b[1;32m      8\u001b[0m wf\u001b[38;5;241m.\u001b[39msleep\u001b[38;5;241m.\u001b[39m_serialize_result \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;01mTrue\u001b[39;00m\n\u001b[1;32m      9\u001b[0m wf\u001b[38;5;241m.\u001b[39msleep\u001b[38;5;241m.\u001b[39mexecutor \u001b[38;5;241m=\u001b[39m exe\n\u001b[0;32m---> 10\u001b[0m \u001b[43mwf\u001b[49m\u001b[43m(\u001b[49m\u001b[43m)\u001b[49m\n",
      "File \u001b[0;32m~/greyhaven/pyiron_workflow/pyiron_workflow/node.py:758\u001b[0m, in \u001b[0;36mNode.__call__\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m    753\u001b[0m \u001b[38;5;28;01mdef\u001b[39;00m \u001b[38;5;21m__call__\u001b[39m(\u001b[38;5;28mself\u001b[39m, \u001b[38;5;241m*\u001b[39margs, \u001b[38;5;241m*\u001b[39m\u001b[38;5;241m*\u001b[39mkwargs) \u001b[38;5;241m-\u001b[39m\u001b[38;5;241m>\u001b[39m \u001b[38;5;28;01mNone\u001b[39;00m:\n\u001b[1;32m    754\u001b[0m \u001b[38;5;250m    \u001b[39m\u001b[38;5;124;03m\"\"\"\u001b[39;00m\n\u001b[1;32m    755\u001b[0m \u001b[38;5;124;03m    A shortcut for :meth:`pull` that automatically runs the entire set of upstream data\u001b[39;00m\n\u001b[1;32m    756\u001b[0m \u001b[38;5;124;03m    dependencies all the way to the parent-most graph object.\u001b[39;00m\n\u001b[1;32m    757\u001b[0m \u001b[38;5;124;03m    \"\"\"\u001b[39;00m\n\u001b[0;32m--> 758\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mpull\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43margs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mrun_parent_trees_too\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;28;43;01mTrue\u001b[39;49;00m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mkwargs\u001b[49m\u001b[43m)\u001b[49m\n",
      "File \u001b[0;32m~/greyhaven/pyiron_workflow/pyiron_workflow/workflow.py:378\u001b[0m, in \u001b[0;36mWorkflow.pull\u001b[0;34m(self, run_parent_trees_too, **kwargs)\u001b[0m\n\u001b[1;32m    376\u001b[0m \u001b[38;5;28;01mdef\u001b[39;00m \u001b[38;5;21mpull\u001b[39m(\u001b[38;5;28mself\u001b[39m, run_parent_trees_too\u001b[38;5;241m=\u001b[39m\u001b[38;5;28;01mFalse\u001b[39;00m, \u001b[38;5;241m*\u001b[39m\u001b[38;5;241m*\u001b[39mkwargs):\n\u001b[1;32m    377\u001b[0m \u001b[38;5;250m    \u001b[39m\u001b[38;5;124;03m\"\"\"Workflows are a parent-most object, so this simply runs without pulling.\"\"\"\u001b[39;00m\n\u001b[0;32m--> 378\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mrun\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mkwargs\u001b[49m\u001b[43m)\u001b[49m\n",
      "File \u001b[0;32m~/greyhaven/pyiron_workflow/pyiron_workflow/workflow.py:367\u001b[0m, in \u001b[0;36mWorkflow.run\u001b[0;34m(self, check_readiness, **kwargs)\u001b[0m\n\u001b[1;32m    358\u001b[0m \u001b[38;5;28;01mdef\u001b[39;00m \u001b[38;5;21mrun\u001b[39m(\n\u001b[1;32m    359\u001b[0m     \u001b[38;5;28mself\u001b[39m,\n\u001b[1;32m    360\u001b[0m     check_readiness: \u001b[38;5;28mbool\u001b[39m \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;01mTrue\u001b[39;00m,\n\u001b[0;32m   (...)\u001b[0m\n\u001b[1;32m    364\u001b[0m     \u001b[38;5;66;03m# worry about running their data trees first, fetching their input, nor firing\u001b[39;00m\n\u001b[1;32m    365\u001b[0m     \u001b[38;5;66;03m# their `ran` signal, hence the change in signature from Node.run\u001b[39;00m\n\u001b[0;32m--> 367\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28;43msuper\u001b[39;49m\u001b[43m(\u001b[49m\u001b[43m)\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mrun\u001b[49m\u001b[43m(\u001b[49m\n\u001b[1;32m    368\u001b[0m \u001b[43m        \u001b[49m\u001b[43mrun_data_tree\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;28;43;01mFalse\u001b[39;49;00m\u001b[43m,\u001b[49m\n\u001b[1;32m    369\u001b[0m \u001b[43m        \u001b[49m\u001b[43mrun_parent_trees_too\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;28;43;01mFalse\u001b[39;49;00m\u001b[43m,\u001b[49m\n\u001b[1;32m    370\u001b[0m \u001b[43m        \u001b[49m\u001b[43mfetch_input\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;28;43;01mFalse\u001b[39;49;00m\u001b[43m,\u001b[49m\n\u001b[1;32m    371\u001b[0m \u001b[43m        \u001b[49m\u001b[43mcheck_readiness\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mcheck_readiness\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    372\u001b[0m \u001b[43m        \u001b[49m\u001b[43memit_ran_signal\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;28;43;01mFalse\u001b[39;49;00m\u001b[43m,\u001b[49m\n\u001b[1;32m    373\u001b[0m \u001b[43m        \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mkwargs\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    374\u001b[0m \u001b[43m    \u001b[49m\u001b[43m)\u001b[49m\n",
      "File \u001b[0;32m~/greyhaven/pyiron_workflow/pyiron_workflow/node.py:483\u001b[0m, in \u001b[0;36mNode.run\u001b[0;34m(self, run_data_tree, run_parent_trees_too, fetch_input, check_readiness, raise_run_exceptions, emit_ran_signal, *args, **kwargs)\u001b[0m\n\u001b[1;32m    477\u001b[0m         \u001b[38;5;28;01mraise\u001b[39;00m \u001b[38;5;167;01mValueError\u001b[39;00m(\n\u001b[1;32m    478\u001b[0m             \u001b[38;5;124mf\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;132;01m{\u001b[39;00m\u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mfull_label\u001b[38;5;132;01m}\u001b[39;00m\u001b[38;5;124m is still waiting for a serialized result\u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[1;32m    479\u001b[0m         )\n\u001b[1;32m    481\u001b[0m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mset_input_values(\u001b[38;5;241m*\u001b[39margs, \u001b[38;5;241m*\u001b[39m\u001b[38;5;241m*\u001b[39mkwargs)\n\u001b[0;32m--> 483\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28;43msuper\u001b[39;49m\u001b[43m(\u001b[49m\u001b[43m)\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mrun\u001b[49m\u001b[43m(\u001b[49m\n\u001b[1;32m    484\u001b[0m \u001b[43m    \u001b[49m\u001b[43mcheck_readiness\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mcheck_readiness\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    485\u001b[0m \u001b[43m    \u001b[49m\u001b[43mraise_run_exceptions\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mraise_run_exceptions\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    486\u001b[0m \u001b[43m    \u001b[49m\u001b[43mbefore_run_kwargs\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43m{\u001b[49m\n\u001b[1;32m    487\u001b[0m \u001b[43m        \u001b[49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43mrun_data_tree\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m:\u001b[49m\u001b[43m \u001b[49m\u001b[43mrun_data_tree\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    488\u001b[0m \u001b[43m        \u001b[49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43mrun_parent_trees_too\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m:\u001b[49m\u001b[43m \u001b[49m\u001b[43mrun_parent_trees_too\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    489\u001b[0m \u001b[43m        \u001b[49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43mfetch_input\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m:\u001b[49m\u001b[43m \u001b[49m\u001b[43mfetch_input\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    490\u001b[0m \u001b[43m        \u001b[49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43memit_ran_signal\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m:\u001b[49m\u001b[43m \u001b[49m\u001b[43memit_ran_signal\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    491\u001b[0m \u001b[43m    \u001b[49m\u001b[43m}\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    492\u001b[0m \u001b[43m    \u001b[49m\u001b[43mrun_finally_kwargs\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43m{\u001b[49m\n\u001b[1;32m    493\u001b[0m \u001b[43m        \u001b[49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43memit_ran_signal\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m:\u001b[49m\u001b[43m \u001b[49m\u001b[43memit_ran_signal\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    494\u001b[0m \u001b[43m        \u001b[49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43mraise_run_exceptions\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m:\u001b[49m\u001b[43m \u001b[49m\u001b[43mraise_run_exceptions\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    495\u001b[0m \u001b[43m    \u001b[49m\u001b[43m}\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    496\u001b[0m \u001b[43m\u001b[49m\u001b[43m)\u001b[49m\n",
      "File \u001b[0;32m~/greyhaven/pyiron_workflow/pyiron_workflow/mixin/run.py:146\u001b[0m, in \u001b[0;36mRunnable.run\u001b[0;34m(self, check_readiness, raise_run_exceptions, before_run_kwargs, run_kwargs, run_exception_kwargs, run_finally_kwargs, finish_run_kwargs)\u001b[0m\n\u001b[1;32m    141\u001b[0m executor \u001b[38;5;241m=\u001b[39m (\n\u001b[1;32m    142\u001b[0m     \u001b[38;5;28;01mNone\u001b[39;00m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mexecutor \u001b[38;5;129;01mis\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m \u001b[38;5;28;01melse\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_parse_executor(\u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mexecutor)\n\u001b[1;32m    143\u001b[0m )\n\u001b[1;32m    145\u001b[0m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mrunning \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;01mTrue\u001b[39;00m\n\u001b[0;32m--> 146\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_run\u001b[49m\u001b[43m(\u001b[49m\n\u001b[1;32m    147\u001b[0m \u001b[43m    \u001b[49m\u001b[43mexecutor\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mexecutor\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    148\u001b[0m \u001b[43m    \u001b[49m\u001b[43mraise_run_exceptions\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mraise_run_exceptions\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    149\u001b[0m \u001b[43m    \u001b[49m\u001b[43mrun_exception_kwargs\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mrun_exception_kwargs\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    150\u001b[0m \u001b[43m    \u001b[49m\u001b[43mrun_finally_kwargs\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mrun_finally_kwargs\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    151\u001b[0m \u001b[43m    \u001b[49m\u001b[43mfinish_run_kwargs\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mfinish_run_kwargs\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    152\u001b[0m \u001b[43m    \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mrun_kwargs\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    153\u001b[0m \u001b[43m\u001b[49m\u001b[43m)\u001b[49m\n",
      "File \u001b[0;32m~/greyhaven/pyiron_workflow/pyiron_workflow/node.py:539\u001b[0m, in \u001b[0;36mNode._run\u001b[0;34m(self, executor, raise_run_exceptions, run_exception_kwargs, run_finally_kwargs, finish_run_kwargs)\u001b[0m\n\u001b[1;32m    537\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mparent \u001b[38;5;129;01mis\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m:\n\u001b[1;32m    538\u001b[0m     \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mparent\u001b[38;5;241m.\u001b[39mregister_child_starting(\u001b[38;5;28mself\u001b[39m)\n\u001b[0;32m--> 539\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28;43msuper\u001b[39;49m\u001b[43m(\u001b[49m\u001b[43m)\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_run\u001b[49m\u001b[43m(\u001b[49m\n\u001b[1;32m    540\u001b[0m \u001b[43m    \u001b[49m\u001b[43mexecutor\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mexecutor\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    541\u001b[0m \u001b[43m    \u001b[49m\u001b[43mraise_run_exceptions\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mraise_run_exceptions\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    542\u001b[0m \u001b[43m    \u001b[49m\u001b[43mrun_exception_kwargs\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mrun_exception_kwargs\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    543\u001b[0m \u001b[43m    \u001b[49m\u001b[43mrun_finally_kwargs\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mrun_finally_kwargs\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    544\u001b[0m \u001b[43m    \u001b[49m\u001b[43mfinish_run_kwargs\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mfinish_run_kwargs\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    545\u001b[0m \u001b[43m\u001b[49m\u001b[43m)\u001b[49m\n",
      "File \u001b[0;32m~/greyhaven/pyiron_workflow/pyiron_workflow/mixin/run.py:214\u001b[0m, in \u001b[0;36mRunnable._run\u001b[0;34m(self, executor, raise_run_exceptions, run_exception_kwargs, run_finally_kwargs, finish_run_kwargs, **kwargs)\u001b[0m\n\u001b[1;32m    212\u001b[0m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_run_finally(\u001b[38;5;241m*\u001b[39m\u001b[38;5;241m*\u001b[39mrun_finally_kwargs)\n\u001b[1;32m    213\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m raise_run_exceptions:\n\u001b[0;32m--> 214\u001b[0m     \u001b[38;5;28;01mraise\u001b[39;00m e\n\u001b[1;32m    215\u001b[0m \u001b[38;5;28;01melse\u001b[39;00m:\n\u001b[1;32m    216\u001b[0m     run_output \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;01mNone\u001b[39;00m\n",
      "File \u001b[0;32m~/greyhaven/pyiron_workflow/pyiron_workflow/mixin/run.py:209\u001b[0m, in \u001b[0;36mRunnable._run\u001b[0;34m(self, executor, raise_run_exceptions, run_exception_kwargs, run_finally_kwargs, finish_run_kwargs, **kwargs)\u001b[0m\n\u001b[1;32m    207\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m executor \u001b[38;5;129;01mis\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m:\n\u001b[1;32m    208\u001b[0m     \u001b[38;5;28;01mtry\u001b[39;00m:\n\u001b[0;32m--> 209\u001b[0m         run_output \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mon_run\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mon_run_args\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mon_run_kwargs\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m    210\u001b[0m     \u001b[38;5;28;01mexcept\u001b[39;00m (\u001b[38;5;167;01mException\u001b[39;00m, \u001b[38;5;167;01mKeyboardInterrupt\u001b[39;00m) \u001b[38;5;28;01mas\u001b[39;00m e:\n\u001b[1;32m    211\u001b[0m         \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_run_exception(\u001b[38;5;241m*\u001b[39m\u001b[38;5;241m*\u001b[39mrun_exception_kwargs)\n",
      "File \u001b[0;32m~/greyhaven/pyiron_workflow/pyiron_workflow/node.py:387\u001b[0m, in \u001b[0;36mNode.on_run\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m    385\u001b[0m save_result: \u001b[38;5;28mbool\u001b[39m \u001b[38;5;241m=\u001b[39m args[\u001b[38;5;241m0\u001b[39m]\n\u001b[1;32m    386\u001b[0m args \u001b[38;5;241m=\u001b[39m args[\u001b[38;5;241m1\u001b[39m:]\n\u001b[0;32m--> 387\u001b[0m result \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_on_run\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43margs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mkwargs\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m    388\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m save_result:\n\u001b[1;32m    389\u001b[0m     \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_temporary_result_pickle(result)\n",
      "File \u001b[0;32m~/greyhaven/pyiron_workflow/pyiron_workflow/nodes/composite.py:160\u001b[0m, in \u001b[0;36mComposite._on_run\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m    157\u001b[0m     \u001b[38;5;28;01mfor\u001b[39;00m node \u001b[38;5;129;01min\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mstarting_nodes:\n\u001b[1;32m    158\u001b[0m         node\u001b[38;5;241m.\u001b[39mrun()\n\u001b[0;32m--> 160\u001b[0m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_run_while_children_or_signals_exist\u001b[49m\u001b[43m(\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m    162\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28mself\u001b[39m\n",
      "File \u001b[0;32m~/greyhaven/pyiron_workflow/pyiron_workflow/nodes/composite.py:175\u001b[0m, in \u001b[0;36mComposite._run_while_children_or_signals_exist\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m    172\u001b[0m             errors[receiving\u001b[38;5;241m.\u001b[39mfull_label] \u001b[38;5;241m=\u001b[39m e\n\u001b[1;32m    173\u001b[0m     \u001b[38;5;28;01mexcept\u001b[39;00m \u001b[38;5;167;01mIndexError\u001b[39;00m:\n\u001b[1;32m    174\u001b[0m         \u001b[38;5;66;03m# The signal queue is empty, but there is still someone running...\u001b[39;00m\n\u001b[0;32m--> 175\u001b[0m         \u001b[43msleep\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_child_sleep_interval\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m    177\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;28mlen\u001b[39m(errors) \u001b[38;5;241m==\u001b[39m \u001b[38;5;241m1\u001b[39m:\n\u001b[1;32m    178\u001b[0m     \u001b[38;5;28;01mraise\u001b[39;00m FailedChildError(\n\u001b[1;32m    179\u001b[0m         \u001b[38;5;124mf\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;132;01m{\u001b[39;00m\u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mfull_label\u001b[38;5;132;01m}\u001b[39;00m\u001b[38;5;124m encountered error in child: \u001b[39m\u001b[38;5;132;01m{\u001b[39;00merrors\u001b[38;5;132;01m}\u001b[39;00m\u001b[38;5;124m\"\u001b[39m\n\u001b[1;32m    180\u001b[0m     ) \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01mnext\u001b[39;00m(\u001b[38;5;28miter\u001b[39m(errors\u001b[38;5;241m.\u001b[39mvalues()))\n",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m: "
     ]
    }
   ],
   "source": [
    "wf = pwf.Workflow(\"killed_process\")\n",
    "wf.sleep_time = pwf.standard_nodes.UserInput(2)\n",
    "wf.skipped_my_nap = 0 * wf.sleep_time\n",
    "wf.sleep = pwf.standard_nodes.Sleep(wf.sleep_time)\n",
    "wf.floating_dreams = pwf.standard_nodes.Float(wf.sleep)\n",
    "\n",
    "with FakeExecutor() as exe:\n",
    "    wf.sleep._serialize_result = True\n",
    "    wf.sleep.executor = exe\n",
    "    wf()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "db23e228-565f-4b99-96fb-f9747d425d25",
   "metadata": {},
   "source": [
    "Killing the process creates a recovery file we can load, where the workflow is in a failed state"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "9fcc4c53-3f48-4c08-bbc2-140469efa6c9",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "wf.load(filename=wf.as_path().joinpath(\"recovery\"))\n",
    "wf.failed"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "13e3d79d-e488-4d95-aee4-3b9bd7d9c9dd",
   "metadata": {},
   "source": [
    "We can see that at the time the graph process was killed, the `wf.sleep` node was still running (outlined in orange):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "f403fd47-1e43-42a6-95f2-cb7024d02b9c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 12.0.0 (20240803.0821)\n",
       " -->\n",
       "<!-- Title: clusterkilled_process Pages: 1 -->\n",
       "<svg width=\"720pt\" height=\"249pt\"\n",
       " viewBox=\"0.00 0.00 720.00 248.59\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(0.409378 0.409378) rotate(0) translate(4 603.25)\">\n",
       "<title>clusterkilled_process</title>\n",
       "<polygon fill=\"white\" stroke=\"none\" points=\"-4,4 -4,-603.25 1754.77,-603.25 1754.77,4 -4,4\"/>\n",
       "<text text-anchor=\"middle\" x=\"875.38\" y=\"-7.95\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">killed_process: Workflow</text>\n",
       "<g id=\"clust1\" class=\"cluster\">\n",
       "<title>clusterkilled_processInputs</title>\n",
       "<defs>\n",
       "<linearGradient id=\"clust1_l_0\" gradientUnits=\"userSpaceOnUse\" x1=\"8\" y1=\"-152.75\" x2=\"280.04\" y2=\"-152.75\" >\n",
       "<stop offset=\"0\" style=\"stop-color:#a5a4a5;stop-opacity:1.;\"/>\n",
       "<stop offset=\"1\" style=\"stop-color:#e4e3e4;stop-opacity:1.;\"/>\n",
       "</linearGradient>\n",
       "</defs>\n",
       "<polygon fill=\"url(#clust1_l_0)\" stroke=\"#a5a4a5\" points=\"8,-33.25 8,-272.25 280.04,-272.25 280.04,-33.25 8,-33.25\"/>\n",
       "<text text-anchor=\"middle\" x=\"144.02\" y=\"-254.95\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">Inputs</text>\n",
       "</g>\n",
       "<g id=\"clust2\" class=\"cluster\">\n",
       "<title>clusterkilled_processOutputs</title>\n",
       "<defs>\n",
       "<linearGradient id=\"clust2_l_1\" gradientUnits=\"userSpaceOnUse\" x1=\"1742.77\" y1=\"-204.75\" x2=\"1477.04\" y2=\"-204.75\" >\n",
       "<stop offset=\"0\" style=\"stop-color:#a5a4a5;stop-opacity:1.;\"/>\n",
       "<stop offset=\"1\" style=\"stop-color:#e4e3e4;stop-opacity:1.;\"/>\n",
       "</linearGradient>\n",
       "</defs>\n",
       "<polygon fill=\"url(#clust2_l_1)\" stroke=\"#a5a4a5\" points=\"1477.04,-85.25 1477.04,-324.25 1742.77,-324.25 1742.77,-85.25 1477.04,-85.25\"/>\n",
       "<text text-anchor=\"middle\" x=\"1609.9\" y=\"-306.95\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">Outputs</text>\n",
       "</g>\n",
       "<g id=\"clust3\" class=\"cluster\">\n",
       "<title>clusterkilled_processsleep_time</title>\n",
       "<defs>\n",
       "<linearGradient id=\"clust3_l_2\" gradientUnits=\"userSpaceOnUse\" x1=\"292.04\" y1=\"-298.25\" x2=\"675.04\" y2=\"-298.25\" >\n",
       "<stop offset=\"0\" style=\"stop-color:#bfe2bf;stop-opacity:1.;\"/>\n",
       "<stop offset=\"1\" style=\"stop-color:#bfe2bf;stop-opacity:1.;\"/>\n",
       "</linearGradient>\n",
       "</defs>\n",
       "<polygon fill=\"url(#clust3_l_2)\" stroke=\"#2ca02c\" stroke-width=\"2\" points=\"292.04,-185.25 292.04,-411.25 675.04,-411.25 675.04,-185.25 292.04,-185.25\"/>\n",
       "<text text-anchor=\"middle\" x=\"483.54\" y=\"-393.95\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">sleep_time: UserInput</text>\n",
       "</g>\n",
       "<g id=\"clust4\" class=\"cluster\">\n",
       "<title>clusterkilled_processsleep_timeInputs</title>\n",
       "<defs>\n",
       "<linearGradient id=\"clust4_l_3\" gradientUnits=\"userSpaceOnUse\" x1=\"300.04\" y1=\"-285.75\" x2=\"476.04\" y2=\"-285.75\" >\n",
       "<stop offset=\"0\" style=\"stop-color:#a5a4a5;stop-opacity:1.;\"/>\n",
       "<stop offset=\"1\" style=\"stop-color:#e4e3e4;stop-opacity:1.;\"/>\n",
       "</linearGradient>\n",
       "</defs>\n",
       "<polygon fill=\"url(#clust4_l_3)\" stroke=\"#a5a4a5\" points=\"300.04,-193.25 300.04,-378.25 476.04,-378.25 476.04,-193.25 300.04,-193.25\"/>\n",
       "<text text-anchor=\"middle\" x=\"388.04\" y=\"-360.95\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">Inputs</text>\n",
       "</g>\n",
       "<g id=\"clust5\" class=\"cluster\">\n",
       "<title>clusterkilled_processsleep_timeOutputsWithInjection</title>\n",
       "<defs>\n",
       "<linearGradient id=\"clust5_l_4\" gradientUnits=\"userSpaceOnUse\" x1=\"667.04\" y1=\"-285.75\" x2=\"504.04\" y2=\"-285.75\" >\n",
       "<stop offset=\"0\" style=\"stop-color:#a5a4a5;stop-opacity:1.;\"/>\n",
       "<stop offset=\"1\" style=\"stop-color:#e4e3e4;stop-opacity:1.;\"/>\n",
       "</linearGradient>\n",
       "</defs>\n",
       "<polygon fill=\"url(#clust5_l_4)\" stroke=\"#a5a4a5\" points=\"504.04,-193.25 504.04,-378.25 667.04,-378.25 667.04,-193.25 504.04,-193.25\"/>\n",
       "<text text-anchor=\"middle\" x=\"585.54\" y=\"-360.95\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">OutputsWithInjection</text>\n",
       "</g>\n",
       "<g id=\"clust6\" class=\"cluster\">\n",
       "<title>clusterkilled_processskipped_my_nap</title>\n",
       "<defs>\n",
       "<linearGradient id=\"clust6_l_5\" gradientUnits=\"userSpaceOnUse\" x1=\"683.04\" y1=\"-217.25\" x2=\"1074.04\" y2=\"-217.25\" >\n",
       "<stop offset=\"0\" style=\"stop-color:#bfe2bf;stop-opacity:1.;\"/>\n",
       "<stop offset=\"1\" style=\"stop-color:#bfe2bf;stop-opacity:1.;\"/>\n",
       "</linearGradient>\n",
       "</defs>\n",
       "<polygon fill=\"url(#clust6_l_5)\" stroke=\"#2ca02c\" stroke-width=\"2\" points=\"683.04,-77.25 683.04,-357.25 1074.04,-357.25 1074.04,-77.25 683.04,-77.25\"/>\n",
       "<text text-anchor=\"middle\" x=\"878.54\" y=\"-339.95\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">skipped_my_nap: RightMultiply</text>\n",
       "</g>\n",
       "<g id=\"clust7\" class=\"cluster\">\n",
       "<title>clusterkilled_processskipped_my_napInputs</title>\n",
       "<defs>\n",
       "<linearGradient id=\"clust7_l_6\" gradientUnits=\"userSpaceOnUse\" x1=\"691.04\" y1=\"-204.75\" x2=\"867.04\" y2=\"-204.75\" >\n",
       "<stop offset=\"0\" style=\"stop-color:#a5a4a5;stop-opacity:1.;\"/>\n",
       "<stop offset=\"1\" style=\"stop-color:#e4e3e4;stop-opacity:1.;\"/>\n",
       "</linearGradient>\n",
       "</defs>\n",
       "<polygon fill=\"url(#clust7_l_6)\" stroke=\"#a5a4a5\" points=\"691.04,-85.25 691.04,-324.25 867.04,-324.25 867.04,-85.25 691.04,-85.25\"/>\n",
       "<text text-anchor=\"middle\" x=\"779.04\" y=\"-306.95\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">Inputs</text>\n",
       "</g>\n",
       "<g id=\"clust8\" class=\"cluster\">\n",
       "<title>clusterkilled_processskipped_my_napOutputsWithInjection</title>\n",
       "<defs>\n",
       "<linearGradient id=\"clust8_l_7\" gradientUnits=\"userSpaceOnUse\" x1=\"1066.04\" y1=\"-177.75\" x2=\"903.04\" y2=\"-177.75\" >\n",
       "<stop offset=\"0\" style=\"stop-color:#a5a4a5;stop-opacity:1.;\"/>\n",
       "<stop offset=\"1\" style=\"stop-color:#e4e3e4;stop-opacity:1.;\"/>\n",
       "</linearGradient>\n",
       "</defs>\n",
       "<polygon fill=\"url(#clust8_l_7)\" stroke=\"#a5a4a5\" points=\"903.04,-85.25 903.04,-270.25 1066.04,-270.25 1066.04,-85.25 903.04,-85.25\"/>\n",
       "<text text-anchor=\"middle\" x=\"984.54\" y=\"-252.95\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">OutputsWithInjection</text>\n",
       "</g>\n",
       "<g id=\"clust9\" class=\"cluster\">\n",
       "<title>clusterkilled_processsleep</title>\n",
       "<defs>\n",
       "<linearGradient id=\"clust9_l_8\" gradientUnits=\"userSpaceOnUse\" x1=\"683.04\" y1=\"-478.25\" x2=\"1074.04\" y2=\"-478.25\" >\n",
       "<stop offset=\"0\" style=\"stop-color:#bfe2bf;stop-opacity:1.;\"/>\n",
       "<stop offset=\"1\" style=\"stop-color:#bfe2bf;stop-opacity:1.;\"/>\n",
       "</linearGradient>\n",
       "</defs>\n",
       "<polygon fill=\"url(#clust9_l_8)\" stroke=\"#ff7f0e\" stroke-width=\"2\" points=\"683.04,-365.25 683.04,-591.25 1074.04,-591.25 1074.04,-365.25 683.04,-365.25\"/>\n",
       "<text text-anchor=\"middle\" x=\"878.54\" y=\"-573.95\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">sleep: Sleep</text>\n",
       "</g>\n",
       "<g id=\"clust10\" class=\"cluster\">\n",
       "<title>clusterkilled_processsleepInputs</title>\n",
       "<defs>\n",
       "<linearGradient id=\"clust10_l_9\" gradientUnits=\"userSpaceOnUse\" x1=\"691.04\" y1=\"-465.75\" x2=\"867.04\" y2=\"-465.75\" >\n",
       "<stop offset=\"0\" style=\"stop-color:#a5a4a5;stop-opacity:1.;\"/>\n",
       "<stop offset=\"1\" style=\"stop-color:#e4e3e4;stop-opacity:1.;\"/>\n",
       "</linearGradient>\n",
       "</defs>\n",
       "<polygon fill=\"url(#clust10_l_9)\" stroke=\"#a5a4a5\" points=\"691.04,-373.25 691.04,-558.25 867.04,-558.25 867.04,-373.25 691.04,-373.25\"/>\n",
       "<text text-anchor=\"middle\" x=\"779.04\" y=\"-540.95\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">Inputs</text>\n",
       "</g>\n",
       "<g id=\"clust11\" class=\"cluster\">\n",
       "<title>clusterkilled_processsleepOutputsWithInjection</title>\n",
       "<defs>\n",
       "<linearGradient id=\"clust11_l_10\" gradientUnits=\"userSpaceOnUse\" x1=\"1066.04\" y1=\"-465.75\" x2=\"903.04\" y2=\"-465.75\" >\n",
       "<stop offset=\"0\" style=\"stop-color:#a5a4a5;stop-opacity:1.;\"/>\n",
       "<stop offset=\"1\" style=\"stop-color:#e4e3e4;stop-opacity:1.;\"/>\n",
       "</linearGradient>\n",
       "</defs>\n",
       "<polygon fill=\"url(#clust11_l_10)\" stroke=\"#a5a4a5\" points=\"903.04,-373.25 903.04,-558.25 1066.04,-558.25 1066.04,-373.25 903.04,-373.25\"/>\n",
       "<text text-anchor=\"middle\" x=\"984.54\" y=\"-540.95\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">OutputsWithInjection</text>\n",
       "</g>\n",
       "<g id=\"clust12\" class=\"cluster\">\n",
       "<title>clusterkilled_processfloating_dreams</title>\n",
       "<defs>\n",
       "<linearGradient id=\"clust12_l_11\" gradientUnits=\"userSpaceOnUse\" x1=\"1082.04\" y1=\"-424.25\" x2=\"1465.04\" y2=\"-424.25\" >\n",
       "<stop offset=\"0\" style=\"stop-color:#bfe2bf;stop-opacity:1.;\"/>\n",
       "<stop offset=\"1\" style=\"stop-color:#bfe2bf;stop-opacity:1.;\"/>\n",
       "</linearGradient>\n",
       "</defs>\n",
       "<polygon fill=\"url(#clust12_l_11)\" stroke=\"#bfe2bf\" stroke-width=\"2\" points=\"1082.04,-311.25 1082.04,-537.25 1465.04,-537.25 1465.04,-311.25 1082.04,-311.25\"/>\n",
       "<text text-anchor=\"middle\" x=\"1273.54\" y=\"-519.95\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">floating_dreams: Float</text>\n",
       "</g>\n",
       "<g id=\"clust13\" class=\"cluster\">\n",
       "<title>clusterkilled_processfloating_dreamsInputs</title>\n",
       "<defs>\n",
       "<linearGradient id=\"clust13_l_12\" gradientUnits=\"userSpaceOnUse\" x1=\"1090.04\" y1=\"-411.75\" x2=\"1266.04\" y2=\"-411.75\" >\n",
       "<stop offset=\"0\" style=\"stop-color:#a5a4a5;stop-opacity:1.;\"/>\n",
       "<stop offset=\"1\" style=\"stop-color:#e4e3e4;stop-opacity:1.;\"/>\n",
       "</linearGradient>\n",
       "</defs>\n",
       "<polygon fill=\"url(#clust13_l_12)\" stroke=\"#a5a4a5\" points=\"1090.04,-319.25 1090.04,-504.25 1266.04,-504.25 1266.04,-319.25 1090.04,-319.25\"/>\n",
       "<text text-anchor=\"middle\" x=\"1178.04\" y=\"-486.95\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">Inputs</text>\n",
       "</g>\n",
       "<g id=\"clust14\" class=\"cluster\">\n",
       "<title>clusterkilled_processfloating_dreamsOutputsWithInjection</title>\n",
       "<defs>\n",
       "<linearGradient id=\"clust14_l_13\" gradientUnits=\"userSpaceOnUse\" x1=\"1457.04\" y1=\"-411.75\" x2=\"1294.04\" y2=\"-411.75\" >\n",
       "<stop offset=\"0\" style=\"stop-color:#a5a4a5;stop-opacity:1.;\"/>\n",
       "<stop offset=\"1\" style=\"stop-color:#e4e3e4;stop-opacity:1.;\"/>\n",
       "</linearGradient>\n",
       "</defs>\n",
       "<polygon fill=\"url(#clust14_l_13)\" stroke=\"#a5a4a5\" points=\"1294.04,-319.25 1294.04,-504.25 1457.04,-504.25 1457.04,-319.25 1294.04,-319.25\"/>\n",
       "<text text-anchor=\"middle\" x=\"1375.54\" y=\"-486.95\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">OutputsWithInjection</text>\n",
       "</g>\n",
       "<!-- clusterkilled_processInputsrun -->\n",
       "<g id=\"node1\" class=\"node\">\n",
       "<title>clusterkilled_processInputsrun</title>\n",
       "<polygon fill=\"#21bfd8\" stroke=\"#21bfd8\" points=\"159.02,-71.25 117.02,-71.25 117.02,-47.25 159.02,-47.25 171.02,-59.25 159.02,-71.25\"/>\n",
       "<text text-anchor=\"middle\" x=\"144.02\" y=\"-54.58\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">run</text>\n",
       "</g>\n",
       "<!-- clusterkilled_processOutputsran -->\n",
       "<g id=\"node5\" class=\"node\">\n",
       "<title>clusterkilled_processOutputsran</title>\n",
       "<polygon fill=\"#21bfd8\" stroke=\"#21bfd8\" points=\"1624.9,-123.25 1582.9,-123.25 1582.9,-99.25 1624.9,-99.25 1636.9,-111.25 1624.9,-123.25\"/>\n",
       "<text text-anchor=\"middle\" x=\"1609.9\" y=\"-106.58\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">ran</text>\n",
       "</g>\n",
       "<!-- clusterkilled_processInputsrun&#45;&gt;clusterkilled_processOutputsran -->\n",
       "<!-- clusterkilled_processInputsaccumulate_and_run -->\n",
       "<g id=\"node2\" class=\"node\">\n",
       "<title>clusterkilled_processInputsaccumulate_and_run</title>\n",
       "<polygon fill=\"#21bfd8\" stroke=\"#21bfd8\" points=\"212.02,-125.25 64.02,-125.25 64.02,-101.25 212.02,-101.25 224.02,-113.25 212.02,-125.25\"/>\n",
       "<text text-anchor=\"middle\" x=\"144.02\" y=\"-108.58\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">accumulate_and_run</text>\n",
       "</g>\n",
       "<!-- clusterkilled_processInputssleep_time__user_input -->\n",
       "<g id=\"node3\" class=\"node\">\n",
       "<title>clusterkilled_processInputssleep_time__user_input</title>\n",
       "<ellipse fill=\"#edb22c\" stroke=\"#edb22c\" cx=\"144.02\" cy=\"-221.25\" rx=\"124.34\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"144.02\" y=\"-216.57\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">sleep_time__user_input</text>\n",
       "</g>\n",
       "<!-- clusterkilled_processsleep_timeInputsuser_input -->\n",
       "<g id=\"node11\" class=\"node\">\n",
       "<title>clusterkilled_processsleep_timeInputsuser_input</title>\n",
       "<ellipse fill=\"#edb22c\" stroke=\"#edb22c\" cx=\"388.04\" cy=\"-327.25\" rx=\"62.78\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"388.04\" y=\"-322.57\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">user_input</text>\n",
       "</g>\n",
       "<!-- clusterkilled_processInputssleep_time__user_input&#45;&gt;clusterkilled_processsleep_timeInputsuser_input -->\n",
       "<g id=\"edge12\" class=\"edge\">\n",
       "<title>clusterkilled_processInputssleep_time__user_input&#45;&gt;clusterkilled_processsleep_timeInputsuser_input</title>\n",
       "<path fill=\"none\" stroke=\"#edb22c\" d=\"M173.68,-239.06C192.32,-250.27 218.2,-265.18 245.69,-279.01\"/>\n",
       "<path fill=\"none\" stroke=\"#edb22c\" d=\"M245.69,-279.01C260.97,-286.7 276.74,-294.05 292.04,-300.25 303.48,-304.88 315.99,-309.05 328.07,-312.64\"/>\n",
       "<polygon fill=\"#edb22c\" stroke=\"#edb22c\" points=\"326.97,-315.96 337.55,-315.35 328.9,-309.23 326.97,-315.96\"/>\n",
       "</g>\n",
       "<!-- clusterkilled_processInputsskipped_my_nap__other -->\n",
       "<g id=\"node4\" class=\"node\">\n",
       "<title>clusterkilled_processInputsskipped_my_nap__other</title>\n",
       "<ellipse fill=\"#edb22c\" stroke=\"#edb22c\" cx=\"144.02\" cy=\"-167.25\" rx=\"128.02\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"144.02\" y=\"-162.57\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">skipped_my_nap__other</text>\n",
       "</g>\n",
       "<!-- clusterkilled_processskipped_my_napInputsother -->\n",
       "<g id=\"node18\" class=\"node\">\n",
       "<title>clusterkilled_processskipped_my_napInputsother</title>\n",
       "<ellipse fill=\"#edb22c\" stroke=\"#edb22c\" cx=\"779.04\" cy=\"-165.25\" rx=\"37\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"779.04\" y=\"-160.57\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">other</text>\n",
       "</g>\n",
       "<!-- clusterkilled_processInputsskipped_my_nap__other&#45;&gt;clusterkilled_processskipped_my_napInputsother -->\n",
       "<g id=\"edge13\" class=\"edge\">\n",
       "<title>clusterkilled_processInputsskipped_my_nap__other&#45;&gt;clusterkilled_processskipped_my_napInputsother</title>\n",
       "<path fill=\"none\" stroke=\"#edb22c\" d=\"M272.36,-166.85C342.98,-166.62 433.44,-166.34 518.27,-166.07\"/>\n",
       "<path fill=\"none\" stroke=\"#edb22c\" d=\"M518.27,-166.07C603.11,-165.8 682.33,-165.55 730.45,-165.4\"/>\n",
       "<polygon fill=\"#edb22c\" stroke=\"#edb22c\" points=\"730.33,-168.9 740.32,-165.37 730.3,-161.9 730.33,-168.9\"/>\n",
       "</g>\n",
       "<!-- clusterkilled_processOutputsfailed -->\n",
       "<g id=\"node6\" class=\"node\">\n",
       "<title>clusterkilled_processOutputsfailed</title>\n",
       "<polygon fill=\"#21bfd8\" stroke=\"#21bfd8\" points=\"1625.03,-177.25 1582.78,-177.25 1582.78,-153.25 1625.03,-153.25 1637.03,-165.25 1625.03,-177.25\"/>\n",
       "<text text-anchor=\"middle\" x=\"1609.9\" y=\"-160.57\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">failed</text>\n",
       "</g>\n",
       "<!-- clusterkilled_processOutputsskipped_my_nap__rmul -->\n",
       "<g id=\"node7\" class=\"node\">\n",
       "<title>clusterkilled_processOutputsskipped_my_nap__rmul</title>\n",
       "<ellipse fill=\"#edb22c\" stroke=\"#edb22c\" cx=\"1609.9\" cy=\"-219.25\" rx=\"124.86\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"1609.9\" y=\"-214.57\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">skipped_my_nap__rmul</text>\n",
       "</g>\n",
       "<!-- clusterkilled_processOutputsfloating_dreams__float -->\n",
       "<g id=\"node8\" class=\"node\">\n",
       "<title>clusterkilled_processOutputsfloating_dreams__float</title>\n",
       "<ellipse fill=\"none\" stroke=\"#edb22c\" stroke-width=\"2\" cx=\"1609.9\" cy=\"-273.25\" rx=\"120.13\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"1609.9\" y=\"-268.57\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">floating_dreams__float</text>\n",
       "</g>\n",
       "<!-- clusterkilled_processsleep_timeInputsrun -->\n",
       "<g id=\"node9\" class=\"node\">\n",
       "<title>clusterkilled_processsleep_timeInputsrun</title>\n",
       "<polygon fill=\"#21bfd8\" stroke=\"#21bfd8\" points=\"403.04,-231.25 361.04,-231.25 361.04,-207.25 403.04,-207.25 415.04,-219.25 403.04,-231.25\"/>\n",
       "<text text-anchor=\"middle\" x=\"388.04\" y=\"-214.57\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">run</text>\n",
       "</g>\n",
       "<!-- clusterkilled_processsleep_timeOutputsWithInjectionran -->\n",
       "<g id=\"node12\" class=\"node\">\n",
       "<title>clusterkilled_processsleep_timeOutputsWithInjectionran</title>\n",
       "<polygon fill=\"#21bfd8\" stroke=\"#21bfd8\" points=\"600.04,-231.25 558.04,-231.25 558.04,-207.25 600.04,-207.25 612.04,-219.25 600.04,-231.25\"/>\n",
       "<text text-anchor=\"middle\" x=\"585.04\" y=\"-214.57\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">ran</text>\n",
       "</g>\n",
       "<!-- clusterkilled_processsleep_timeInputsrun&#45;&gt;clusterkilled_processsleep_timeOutputsWithInjectionran -->\n",
       "<!-- clusterkilled_processsleep_timeInputsaccumulate_and_run -->\n",
       "<g id=\"node10\" class=\"node\">\n",
       "<title>clusterkilled_processsleep_timeInputsaccumulate_and_run</title>\n",
       "<polygon fill=\"#21bfd8\" stroke=\"#21bfd8\" points=\"456.04,-285.25 308.04,-285.25 308.04,-261.25 456.04,-261.25 468.04,-273.25 456.04,-285.25\"/>\n",
       "<text text-anchor=\"middle\" x=\"388.04\" y=\"-268.57\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">accumulate_and_run</text>\n",
       "</g>\n",
       "<!-- clusterkilled_processskipped_my_napInputsaccumulate_and_run -->\n",
       "<g id=\"node16\" class=\"node\">\n",
       "<title>clusterkilled_processskipped_my_napInputsaccumulate_and_run</title>\n",
       "<polygon fill=\"#21bfd8\" stroke=\"#21bfd8\" points=\"847.04,-231.25 699.04,-231.25 699.04,-207.25 847.04,-207.25 859.04,-219.25 847.04,-231.25\"/>\n",
       "<text text-anchor=\"middle\" x=\"779.04\" y=\"-214.57\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">accumulate_and_run</text>\n",
       "</g>\n",
       "<!-- clusterkilled_processsleep_timeOutputsWithInjectionran&#45;&gt;clusterkilled_processskipped_my_napInputsaccumulate_and_run -->\n",
       "<g id=\"edge6\" class=\"edge\">\n",
       "<title>clusterkilled_processsleep_timeOutputsWithInjectionran&#45;&gt;clusterkilled_processskipped_my_napInputsaccumulate_and_run</title>\n",
       "<path fill=\"none\" stroke=\"#21bfd8\" d=\"M612.33,-219.25C622.13,-219.25 634.02,-219.25 646.89,-219.25\"/>\n",
       "<path fill=\"none\" stroke=\"#21bfd8\" d=\"M646.89,-219.25C659.77,-219.25 673.62,-219.25 687.32,-219.25\"/>\n",
       "<polygon fill=\"#21bfd8\" stroke=\"#21bfd8\" points=\"687.08,-222.75 697.08,-219.25 687.08,-215.75 687.08,-222.75\"/>\n",
       "</g>\n",
       "<!-- clusterkilled_processsleepInputsaccumulate_and_run -->\n",
       "<g id=\"node23\" class=\"node\">\n",
       "<title>clusterkilled_processsleepInputsaccumulate_and_run</title>\n",
       "<polygon fill=\"#21bfd8\" stroke=\"#21bfd8\" points=\"847.04,-465.25 699.04,-465.25 699.04,-441.25 847.04,-441.25 859.04,-453.25 847.04,-465.25\"/>\n",
       "<text text-anchor=\"middle\" x=\"779.04\" y=\"-448.57\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">accumulate_and_run</text>\n",
       "</g>\n",
       "<!-- clusterkilled_processsleep_timeOutputsWithInjectionran&#45;&gt;clusterkilled_processsleepInputsaccumulate_and_run -->\n",
       "<g id=\"edge7\" class=\"edge\">\n",
       "<title>clusterkilled_processsleep_timeOutputsWithInjectionran&#45;&gt;clusterkilled_processsleepInputsaccumulate_and_run</title>\n",
       "<path fill=\"none\" stroke=\"#21bfd8\" d=\"M612.37,-219.53C632.93,-221.3 660.38,-227.38 675.04,-246.25 679.14,-251.53 680.01,-271.63 679.81,-296.64\"/>\n",
       "<path fill=\"none\" stroke=\"#21bfd8\" d=\"M679.81,-296.64C679.41,-346.47 674.77,-415.79 683.04,-426.25 685.08,-428.83 687.36,-431.18 689.81,-433.32\"/>\n",
       "<polygon fill=\"#21bfd8\" stroke=\"#21bfd8\" points=\"687.41,-435.9 697.57,-438.89 691.49,-430.21 687.41,-435.9\"/>\n",
       "</g>\n",
       "<!-- clusterkilled_processsleep_timeOutputsWithInjectionfailed -->\n",
       "<g id=\"node13\" class=\"node\">\n",
       "<title>clusterkilled_processsleep_timeOutputsWithInjectionfailed</title>\n",
       "<polygon fill=\"#21bfd8\" stroke=\"#21bfd8\" points=\"600.17,-285.25 557.92,-285.25 557.92,-261.25 600.17,-261.25 612.17,-273.25 600.17,-285.25\"/>\n",
       "<text text-anchor=\"middle\" x=\"585.04\" y=\"-268.57\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">failed</text>\n",
       "</g>\n",
       "<!-- clusterkilled_processsleep_timeOutputsWithInjectionuser_input -->\n",
       "<g id=\"node14\" class=\"node\">\n",
       "<title>clusterkilled_processsleep_timeOutputsWithInjectionuser_input</title>\n",
       "<ellipse fill=\"#edb22c\" stroke=\"#edb22c\" cx=\"585.04\" cy=\"-327.25\" rx=\"62.78\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"585.04\" y=\"-322.57\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">user_input</text>\n",
       "</g>\n",
       "<!-- clusterkilled_processskipped_my_napInputsobj -->\n",
       "<g id=\"node17\" class=\"node\">\n",
       "<title>clusterkilled_processskipped_my_napInputsobj</title>\n",
       "<ellipse fill=\"#edb22c\" stroke=\"#edb22c\" cx=\"779.04\" cy=\"-273.25\" rx=\"27\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"779.04\" y=\"-268.57\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">obj</text>\n",
       "</g>\n",
       "<!-- clusterkilled_processsleep_timeOutputsWithInjectionuser_input&#45;&gt;clusterkilled_processskipped_my_napInputsobj -->\n",
       "<g id=\"edge8\" class=\"edge\">\n",
       "<title>clusterkilled_processsleep_timeOutputsWithInjectionuser_input&#45;&gt;clusterkilled_processskipped_my_napInputsobj</title>\n",
       "<path fill=\"none\" stroke=\"#edb22c\" d=\"M631.06,-314.59C648.08,-309.8 668.13,-304.16 687.84,-298.62\"/>\n",
       "<path fill=\"none\" stroke=\"#edb22c\" d=\"M687.84,-298.62C707.56,-293.07 726.95,-287.62 742.67,-283.2\"/>\n",
       "<polygon fill=\"#edb22c\" stroke=\"#edb22c\" points=\"743.4,-286.63 752.08,-280.55 741.5,-279.89 743.4,-286.63\"/>\n",
       "</g>\n",
       "<!-- clusterkilled_processsleepInputst -->\n",
       "<g id=\"node24\" class=\"node\">\n",
       "<title>clusterkilled_processsleepInputst</title>\n",
       "<ellipse fill=\"#edb22c\" stroke=\"#edb22c\" cx=\"779.04\" cy=\"-507.25\" rx=\"27\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"779.04\" y=\"-502.57\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">t</text>\n",
       "</g>\n",
       "<!-- clusterkilled_processsleep_timeOutputsWithInjectionuser_input&#45;&gt;clusterkilled_processsleepInputst -->\n",
       "<g id=\"edge9\" class=\"edge\">\n",
       "<title>clusterkilled_processsleep_timeOutputsWithInjectionuser_input&#45;&gt;clusterkilled_processsleepInputst</title>\n",
       "<path fill=\"none\" stroke=\"#edb22c\" d=\"M592.51,-345.39C601.06,-367.53 618.06,-406.19 643.46,-439.79\"/>\n",
       "<path fill=\"none\" stroke=\"#edb22c\" d=\"M643.46,-439.79C654.94,-454.97 668.14,-469.12 683.04,-480.25 699.77,-492.74 722.07,-499.45 740.86,-503.06\"/>\n",
       "<polygon fill=\"#edb22c\" stroke=\"#edb22c\" points=\"740.09,-506.48 750.53,-504.68 741.25,-499.58 740.09,-506.48\"/>\n",
       "</g>\n",
       "<!-- clusterkilled_processskipped_my_napInputsrun -->\n",
       "<g id=\"node15\" class=\"node\">\n",
       "<title>clusterkilled_processskipped_my_napInputsrun</title>\n",
       "<polygon fill=\"#21bfd8\" stroke=\"#21bfd8\" points=\"794.04,-123.25 752.04,-123.25 752.04,-99.25 794.04,-99.25 806.04,-111.25 794.04,-123.25\"/>\n",
       "<text text-anchor=\"middle\" x=\"779.04\" y=\"-106.58\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">run</text>\n",
       "</g>\n",
       "<!-- clusterkilled_processskipped_my_napOutputsWithInjectionran -->\n",
       "<g id=\"node19\" class=\"node\">\n",
       "<title>clusterkilled_processskipped_my_napOutputsWithInjectionran</title>\n",
       "<polygon fill=\"#21bfd8\" stroke=\"#21bfd8\" points=\"999.04,-123.25 957.04,-123.25 957.04,-99.25 999.04,-99.25 1011.04,-111.25 999.04,-123.25\"/>\n",
       "<text text-anchor=\"middle\" x=\"984.04\" y=\"-106.58\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">ran</text>\n",
       "</g>\n",
       "<!-- clusterkilled_processskipped_my_napInputsrun&#45;&gt;clusterkilled_processskipped_my_napOutputsWithInjectionran -->\n",
       "<!-- clusterkilled_processskipped_my_napOutputsWithInjectionfailed -->\n",
       "<g id=\"node20\" class=\"node\">\n",
       "<title>clusterkilled_processskipped_my_napOutputsWithInjectionfailed</title>\n",
       "<polygon fill=\"#21bfd8\" stroke=\"#21bfd8\" points=\"999.17,-177.25 956.92,-177.25 956.92,-153.25 999.17,-153.25 1011.17,-165.25 999.17,-177.25\"/>\n",
       "<text text-anchor=\"middle\" x=\"984.04\" y=\"-160.57\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">failed</text>\n",
       "</g>\n",
       "<!-- clusterkilled_processskipped_my_napOutputsWithInjectionrmul -->\n",
       "<g id=\"node21\" class=\"node\">\n",
       "<title>clusterkilled_processskipped_my_napOutputsWithInjectionrmul</title>\n",
       "<ellipse fill=\"#edb22c\" stroke=\"#edb22c\" cx=\"984.04\" cy=\"-219.25\" rx=\"33.85\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"984.04\" y=\"-214.57\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">rmul</text>\n",
       "</g>\n",
       "<!-- clusterkilled_processskipped_my_napOutputsWithInjectionrmul&#45;&gt;clusterkilled_processOutputsskipped_my_nap__rmul -->\n",
       "<g id=\"edge14\" class=\"edge\">\n",
       "<title>clusterkilled_processskipped_my_napOutputsWithInjectionrmul&#45;&gt;clusterkilled_processOutputsskipped_my_nap__rmul</title>\n",
       "<path fill=\"none\" stroke=\"#edb22c\" d=\"M1018.14,-219.25C1060.06,-219.25 1137.14,-219.25 1222.1,-219.25\"/>\n",
       "<path fill=\"none\" stroke=\"#edb22c\" d=\"M1222.1,-219.25C1307.05,-219.25 1399.88,-219.25 1473.3,-219.25\"/>\n",
       "<polygon fill=\"#edb22c\" stroke=\"#edb22c\" points=\"1473.11,-222.75 1483.11,-219.25 1473.11,-215.75 1473.11,-222.75\"/>\n",
       "</g>\n",
       "<!-- clusterkilled_processsleepInputsrun -->\n",
       "<g id=\"node22\" class=\"node\">\n",
       "<title>clusterkilled_processsleepInputsrun</title>\n",
       "<polygon fill=\"#21bfd8\" stroke=\"#21bfd8\" points=\"794.04,-411.25 752.04,-411.25 752.04,-387.25 794.04,-387.25 806.04,-399.25 794.04,-411.25\"/>\n",
       "<text text-anchor=\"middle\" x=\"779.04\" y=\"-394.57\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">run</text>\n",
       "</g>\n",
       "<!-- clusterkilled_processsleepOutputsWithInjectionran -->\n",
       "<g id=\"node25\" class=\"node\">\n",
       "<title>clusterkilled_processsleepOutputsWithInjectionran</title>\n",
       "<polygon fill=\"#21bfd8\" stroke=\"#21bfd8\" points=\"999.04,-411.25 957.04,-411.25 957.04,-387.25 999.04,-387.25 1011.04,-399.25 999.04,-411.25\"/>\n",
       "<text text-anchor=\"middle\" x=\"984.04\" y=\"-394.57\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">ran</text>\n",
       "</g>\n",
       "<!-- clusterkilled_processsleepInputsrun&#45;&gt;clusterkilled_processsleepOutputsWithInjectionran -->\n",
       "<!-- clusterkilled_processfloating_dreamsInputsaccumulate_and_run -->\n",
       "<g id=\"node29\" class=\"node\">\n",
       "<title>clusterkilled_processfloating_dreamsInputsaccumulate_and_run</title>\n",
       "<polygon fill=\"#21bfd8\" stroke=\"#21bfd8\" points=\"1246.04,-411.25 1098.04,-411.25 1098.04,-387.25 1246.04,-387.25 1258.04,-399.25 1246.04,-411.25\"/>\n",
       "<text text-anchor=\"middle\" x=\"1178.04\" y=\"-394.57\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">accumulate_and_run</text>\n",
       "</g>\n",
       "<!-- clusterkilled_processsleepOutputsWithInjectionran&#45;&gt;clusterkilled_processfloating_dreamsInputsaccumulate_and_run -->\n",
       "<g id=\"edge10\" class=\"edge\">\n",
       "<title>clusterkilled_processsleepOutputsWithInjectionran&#45;&gt;clusterkilled_processfloating_dreamsInputsaccumulate_and_run</title>\n",
       "<path fill=\"none\" stroke=\"#21bfd8\" d=\"M1011.33,-399.25C1021.13,-399.25 1033.02,-399.25 1045.89,-399.25\"/>\n",
       "<path fill=\"none\" stroke=\"#21bfd8\" d=\"M1045.89,-399.25C1058.77,-399.25 1072.62,-399.25 1086.32,-399.25\"/>\n",
       "<polygon fill=\"#21bfd8\" stroke=\"#21bfd8\" points=\"1086.08,-402.75 1096.08,-399.25 1086.08,-395.75 1086.08,-402.75\"/>\n",
       "</g>\n",
       "<!-- clusterkilled_processsleepOutputsWithInjectionfailed -->\n",
       "<g id=\"node26\" class=\"node\">\n",
       "<title>clusterkilled_processsleepOutputsWithInjectionfailed</title>\n",
       "<polygon fill=\"#21bfd8\" stroke=\"#21bfd8\" points=\"999.17,-465.25 956.92,-465.25 956.92,-441.25 999.17,-441.25 1011.17,-453.25 999.17,-465.25\"/>\n",
       "<text text-anchor=\"middle\" x=\"984.04\" y=\"-448.57\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">failed</text>\n",
       "</g>\n",
       "<!-- clusterkilled_processsleepOutputsWithInjectiontime -->\n",
       "<g id=\"node27\" class=\"node\">\n",
       "<title>clusterkilled_processsleepOutputsWithInjectiontime</title>\n",
       "<ellipse fill=\"#edb22c\" stroke=\"#edb22c\" cx=\"984.04\" cy=\"-507.25\" rx=\"32.79\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"984.04\" y=\"-502.57\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">time</text>\n",
       "</g>\n",
       "<!-- clusterkilled_processfloating_dreamsInputsobj -->\n",
       "<g id=\"node30\" class=\"node\">\n",
       "<title>clusterkilled_processfloating_dreamsInputsobj</title>\n",
       "<ellipse fill=\"#edb22c\" stroke=\"#edb22c\" cx=\"1178.04\" cy=\"-453.25\" rx=\"27\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"1178.04\" y=\"-448.57\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">obj</text>\n",
       "</g>\n",
       "<!-- clusterkilled_processsleepOutputsWithInjectiontime&#45;&gt;clusterkilled_processfloating_dreamsInputsobj -->\n",
       "<g id=\"edge11\" class=\"edge\">\n",
       "<title>clusterkilled_processsleepOutputsWithInjectiontime&#45;&gt;clusterkilled_processfloating_dreamsInputsobj</title>\n",
       "<path fill=\"none\" stroke=\"#edb22c\" d=\"M1013.78,-499.17C1030.9,-494.35 1053.74,-487.93 1076.9,-481.41\"/>\n",
       "<path fill=\"none\" stroke=\"#edb22c\" d=\"M1076.9,-481.41C1100.07,-474.9 1123.55,-468.29 1141.95,-463.12\"/>\n",
       "<polygon fill=\"#edb22c\" stroke=\"#edb22c\" points=\"1142.53,-466.59 1151.21,-460.51 1140.64,-459.85 1142.53,-466.59\"/>\n",
       "</g>\n",
       "<!-- clusterkilled_processfloating_dreamsInputsrun -->\n",
       "<g id=\"node28\" class=\"node\">\n",
       "<title>clusterkilled_processfloating_dreamsInputsrun</title>\n",
       "<polygon fill=\"#21bfd8\" stroke=\"#21bfd8\" points=\"1193.04,-357.25 1151.04,-357.25 1151.04,-333.25 1193.04,-333.25 1205.04,-345.25 1193.04,-357.25\"/>\n",
       "<text text-anchor=\"middle\" x=\"1178.04\" y=\"-340.57\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">run</text>\n",
       "</g>\n",
       "<!-- clusterkilled_processfloating_dreamsOutputsWithInjectionran -->\n",
       "<g id=\"node31\" class=\"node\">\n",
       "<title>clusterkilled_processfloating_dreamsOutputsWithInjectionran</title>\n",
       "<polygon fill=\"#21bfd8\" stroke=\"#21bfd8\" points=\"1390.04,-357.25 1348.04,-357.25 1348.04,-333.25 1390.04,-333.25 1402.04,-345.25 1390.04,-357.25\"/>\n",
       "<text text-anchor=\"middle\" x=\"1375.04\" y=\"-340.57\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">ran</text>\n",
       "</g>\n",
       "<!-- clusterkilled_processfloating_dreamsInputsrun&#45;&gt;clusterkilled_processfloating_dreamsOutputsWithInjectionran -->\n",
       "<!-- clusterkilled_processfloating_dreamsOutputsWithInjectionfailed -->\n",
       "<g id=\"node32\" class=\"node\">\n",
       "<title>clusterkilled_processfloating_dreamsOutputsWithInjectionfailed</title>\n",
       "<polygon fill=\"#21bfd8\" stroke=\"#21bfd8\" points=\"1390.17,-411.25 1347.92,-411.25 1347.92,-387.25 1390.17,-387.25 1402.17,-399.25 1390.17,-411.25\"/>\n",
       "<text text-anchor=\"middle\" x=\"1375.04\" y=\"-394.57\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">failed</text>\n",
       "</g>\n",
       "<!-- clusterkilled_processfloating_dreamsOutputsWithInjectionfloat -->\n",
       "<g id=\"node33\" class=\"node\">\n",
       "<title>clusterkilled_processfloating_dreamsOutputsWithInjectionfloat</title>\n",
       "<ellipse fill=\"none\" stroke=\"#edb22c\" stroke-width=\"2\" cx=\"1375.04\" cy=\"-453.25\" rx=\"32.79\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"1375.04\" y=\"-448.57\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">float</text>\n",
       "</g>\n",
       "<!-- clusterkilled_processfloating_dreamsOutputsWithInjectionfloat&#45;&gt;clusterkilled_processOutputsfloating_dreams__float -->\n",
       "<g id=\"edge15\" class=\"edge\">\n",
       "<title>clusterkilled_processfloating_dreamsOutputsWithInjectionfloat&#45;&gt;clusterkilled_processOutputsfloating_dreams__float</title>\n",
       "<path fill=\"none\" stroke=\"#edb22c\" d=\"M1406.95,-448.02C1424.88,-444 1447.34,-437.23 1465.04,-426.25 1482.54,-415.39 1499.32,-401.87 1514.85,-387.4\"/>\n",
       "<path fill=\"none\" stroke=\"#edb22c\" d=\"M1514.85,-387.4C1546.94,-357.5 1573.67,-323.51 1590.34,-300.46\"/>\n",
       "<polygon fill=\"#edb22c\" stroke=\"#edb22c\" points=\"1592.89,-302.9 1595.83,-292.72 1587.19,-298.84 1592.89,-302.9\"/>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<graphviz.graphs.Digraph at 0x145f001d4450>"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "wf.draw(size=(10, 10))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1e29f9d9-85e6-4081-93f7-1bd1ec72c6e2",
   "metadata": {},
   "source": [
    "We can look in the filesystem along the remotely-executed node's semantic path to see when the `run_result.tmp` file is available, this is the cloudpickled result from the remote execution. (In a real case we'd probably be more conveniently looking at a list of jobs running on the queue to see if ours was finished, or leveraging some other syntactic shortcut to see if the job is done, but convenient UI tools around this functionality aren't done yet.)\n",
    "\n",
    "Now we simply reset the failed status and run the workflow again:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "31b54502-fee1-4968-b84b-308529fa4009",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'skipped_my_nap__rmul': 0, 'floating_dreams__float': 2.0}"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "wf.failed = False\n",
    "wf()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "033d01c9-a20a-4c18-a194-1ee5fd9b6532",
   "metadata": {},
   "source": [
    "If we had tried this too early at the results file wasn't there yet, we'd have simply failed again with a message to that effect. As it is, when the `running` `wf.sleep` node re-ran, it saw that its results file was available, unpickled it, processed it, and we continued running.\n",
    "\n",
    "Remember that (as long as caching is being used), all the green-rimmed nodes above have already run and will re-run extremely quickly, so it should always be quite fast to \"catch up\" to these slow, remotely executed parts of the graph.\n",
    "\n",
    "If we look again, we see the entire graph is run:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "fbea58a7-de9c-40c7-8504-4f3704bbfb2e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 12.0.0 (20240803.0821)\n",
       " -->\n",
       "<!-- Title: clusterkilled_process Pages: 1 -->\n",
       "<svg width=\"720pt\" height=\"249pt\"\n",
       " viewBox=\"0.00 0.00 720.00 248.59\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(0.409378 0.409378) rotate(0) translate(4 603.25)\">\n",
       "<title>clusterkilled_process</title>\n",
       "<polygon fill=\"white\" stroke=\"none\" points=\"-4,4 -4,-603.25 1754.77,-603.25 1754.77,4 -4,4\"/>\n",
       "<text text-anchor=\"middle\" x=\"875.38\" y=\"-7.95\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">killed_process: Workflow</text>\n",
       "<g id=\"clust1\" class=\"cluster\">\n",
       "<title>clusterkilled_processInputs</title>\n",
       "<defs>\n",
       "<linearGradient id=\"clust1_l_0\" gradientUnits=\"userSpaceOnUse\" x1=\"8\" y1=\"-152.75\" x2=\"280.04\" y2=\"-152.75\" >\n",
       "<stop offset=\"0\" style=\"stop-color:#a5a4a5;stop-opacity:1.;\"/>\n",
       "<stop offset=\"1\" style=\"stop-color:#e4e3e4;stop-opacity:1.;\"/>\n",
       "</linearGradient>\n",
       "</defs>\n",
       "<polygon fill=\"url(#clust1_l_0)\" stroke=\"#a5a4a5\" points=\"8,-33.25 8,-272.25 280.04,-272.25 280.04,-33.25 8,-33.25\"/>\n",
       "<text text-anchor=\"middle\" x=\"144.02\" y=\"-254.95\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">Inputs</text>\n",
       "</g>\n",
       "<g id=\"clust2\" class=\"cluster\">\n",
       "<title>clusterkilled_processOutputs</title>\n",
       "<defs>\n",
       "<linearGradient id=\"clust2_l_1\" gradientUnits=\"userSpaceOnUse\" x1=\"1742.77\" y1=\"-204.75\" x2=\"1477.04\" y2=\"-204.75\" >\n",
       "<stop offset=\"0\" style=\"stop-color:#a5a4a5;stop-opacity:1.;\"/>\n",
       "<stop offset=\"1\" style=\"stop-color:#e4e3e4;stop-opacity:1.;\"/>\n",
       "</linearGradient>\n",
       "</defs>\n",
       "<polygon fill=\"url(#clust2_l_1)\" stroke=\"#a5a4a5\" points=\"1477.04,-85.25 1477.04,-324.25 1742.77,-324.25 1742.77,-85.25 1477.04,-85.25\"/>\n",
       "<text text-anchor=\"middle\" x=\"1609.9\" y=\"-306.95\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">Outputs</text>\n",
       "</g>\n",
       "<g id=\"clust3\" class=\"cluster\">\n",
       "<title>clusterkilled_processsleep_time</title>\n",
       "<defs>\n",
       "<linearGradient id=\"clust3_l_2\" gradientUnits=\"userSpaceOnUse\" x1=\"292.04\" y1=\"-298.25\" x2=\"675.04\" y2=\"-298.25\" >\n",
       "<stop offset=\"0\" style=\"stop-color:#bfe2bf;stop-opacity:1.;\"/>\n",
       "<stop offset=\"1\" style=\"stop-color:#bfe2bf;stop-opacity:1.;\"/>\n",
       "</linearGradient>\n",
       "</defs>\n",
       "<polygon fill=\"url(#clust3_l_2)\" stroke=\"#2ca02c\" stroke-width=\"2\" points=\"292.04,-185.25 292.04,-411.25 675.04,-411.25 675.04,-185.25 292.04,-185.25\"/>\n",
       "<text text-anchor=\"middle\" x=\"483.54\" y=\"-393.95\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">sleep_time: UserInput</text>\n",
       "</g>\n",
       "<g id=\"clust4\" class=\"cluster\">\n",
       "<title>clusterkilled_processsleep_timeInputs</title>\n",
       "<defs>\n",
       "<linearGradient id=\"clust4_l_3\" gradientUnits=\"userSpaceOnUse\" x1=\"300.04\" y1=\"-285.75\" x2=\"476.04\" y2=\"-285.75\" >\n",
       "<stop offset=\"0\" style=\"stop-color:#a5a4a5;stop-opacity:1.;\"/>\n",
       "<stop offset=\"1\" style=\"stop-color:#e4e3e4;stop-opacity:1.;\"/>\n",
       "</linearGradient>\n",
       "</defs>\n",
       "<polygon fill=\"url(#clust4_l_3)\" stroke=\"#a5a4a5\" points=\"300.04,-193.25 300.04,-378.25 476.04,-378.25 476.04,-193.25 300.04,-193.25\"/>\n",
       "<text text-anchor=\"middle\" x=\"388.04\" y=\"-360.95\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">Inputs</text>\n",
       "</g>\n",
       "<g id=\"clust5\" class=\"cluster\">\n",
       "<title>clusterkilled_processsleep_timeOutputsWithInjection</title>\n",
       "<defs>\n",
       "<linearGradient id=\"clust5_l_4\" gradientUnits=\"userSpaceOnUse\" x1=\"667.04\" y1=\"-285.75\" x2=\"504.04\" y2=\"-285.75\" >\n",
       "<stop offset=\"0\" style=\"stop-color:#a5a4a5;stop-opacity:1.;\"/>\n",
       "<stop offset=\"1\" style=\"stop-color:#e4e3e4;stop-opacity:1.;\"/>\n",
       "</linearGradient>\n",
       "</defs>\n",
       "<polygon fill=\"url(#clust5_l_4)\" stroke=\"#a5a4a5\" points=\"504.04,-193.25 504.04,-378.25 667.04,-378.25 667.04,-193.25 504.04,-193.25\"/>\n",
       "<text text-anchor=\"middle\" x=\"585.54\" y=\"-360.95\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">OutputsWithInjection</text>\n",
       "</g>\n",
       "<g id=\"clust6\" class=\"cluster\">\n",
       "<title>clusterkilled_processskipped_my_nap</title>\n",
       "<defs>\n",
       "<linearGradient id=\"clust6_l_5\" gradientUnits=\"userSpaceOnUse\" x1=\"683.04\" y1=\"-217.25\" x2=\"1074.04\" y2=\"-217.25\" >\n",
       "<stop offset=\"0\" style=\"stop-color:#bfe2bf;stop-opacity:1.;\"/>\n",
       "<stop offset=\"1\" style=\"stop-color:#bfe2bf;stop-opacity:1.;\"/>\n",
       "</linearGradient>\n",
       "</defs>\n",
       "<polygon fill=\"url(#clust6_l_5)\" stroke=\"#2ca02c\" stroke-width=\"2\" points=\"683.04,-77.25 683.04,-357.25 1074.04,-357.25 1074.04,-77.25 683.04,-77.25\"/>\n",
       "<text text-anchor=\"middle\" x=\"878.54\" y=\"-339.95\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">skipped_my_nap: RightMultiply</text>\n",
       "</g>\n",
       "<g id=\"clust7\" class=\"cluster\">\n",
       "<title>clusterkilled_processskipped_my_napInputs</title>\n",
       "<defs>\n",
       "<linearGradient id=\"clust7_l_6\" gradientUnits=\"userSpaceOnUse\" x1=\"691.04\" y1=\"-204.75\" x2=\"867.04\" y2=\"-204.75\" >\n",
       "<stop offset=\"0\" style=\"stop-color:#a5a4a5;stop-opacity:1.;\"/>\n",
       "<stop offset=\"1\" style=\"stop-color:#e4e3e4;stop-opacity:1.;\"/>\n",
       "</linearGradient>\n",
       "</defs>\n",
       "<polygon fill=\"url(#clust7_l_6)\" stroke=\"#a5a4a5\" points=\"691.04,-85.25 691.04,-324.25 867.04,-324.25 867.04,-85.25 691.04,-85.25\"/>\n",
       "<text text-anchor=\"middle\" x=\"779.04\" y=\"-306.95\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">Inputs</text>\n",
       "</g>\n",
       "<g id=\"clust8\" class=\"cluster\">\n",
       "<title>clusterkilled_processskipped_my_napOutputsWithInjection</title>\n",
       "<defs>\n",
       "<linearGradient id=\"clust8_l_7\" gradientUnits=\"userSpaceOnUse\" x1=\"1066.04\" y1=\"-177.75\" x2=\"903.04\" y2=\"-177.75\" >\n",
       "<stop offset=\"0\" style=\"stop-color:#a5a4a5;stop-opacity:1.;\"/>\n",
       "<stop offset=\"1\" style=\"stop-color:#e4e3e4;stop-opacity:1.;\"/>\n",
       "</linearGradient>\n",
       "</defs>\n",
       "<polygon fill=\"url(#clust8_l_7)\" stroke=\"#a5a4a5\" points=\"903.04,-85.25 903.04,-270.25 1066.04,-270.25 1066.04,-85.25 903.04,-85.25\"/>\n",
       "<text text-anchor=\"middle\" x=\"984.54\" y=\"-252.95\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">OutputsWithInjection</text>\n",
       "</g>\n",
       "<g id=\"clust9\" class=\"cluster\">\n",
       "<title>clusterkilled_processsleep</title>\n",
       "<defs>\n",
       "<linearGradient id=\"clust9_l_8\" gradientUnits=\"userSpaceOnUse\" x1=\"683.04\" y1=\"-478.25\" x2=\"1074.04\" y2=\"-478.25\" >\n",
       "<stop offset=\"0\" style=\"stop-color:#bfe2bf;stop-opacity:1.;\"/>\n",
       "<stop offset=\"1\" style=\"stop-color:#bfe2bf;stop-opacity:1.;\"/>\n",
       "</linearGradient>\n",
       "</defs>\n",
       "<polygon fill=\"url(#clust9_l_8)\" stroke=\"#2ca02c\" stroke-width=\"2\" points=\"683.04,-365.25 683.04,-591.25 1074.04,-591.25 1074.04,-365.25 683.04,-365.25\"/>\n",
       "<text text-anchor=\"middle\" x=\"878.54\" y=\"-573.95\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">sleep: Sleep</text>\n",
       "</g>\n",
       "<g id=\"clust10\" class=\"cluster\">\n",
       "<title>clusterkilled_processsleepInputs</title>\n",
       "<defs>\n",
       "<linearGradient id=\"clust10_l_9\" gradientUnits=\"userSpaceOnUse\" x1=\"691.04\" y1=\"-465.75\" x2=\"867.04\" y2=\"-465.75\" >\n",
       "<stop offset=\"0\" style=\"stop-color:#a5a4a5;stop-opacity:1.;\"/>\n",
       "<stop offset=\"1\" style=\"stop-color:#e4e3e4;stop-opacity:1.;\"/>\n",
       "</linearGradient>\n",
       "</defs>\n",
       "<polygon fill=\"url(#clust10_l_9)\" stroke=\"#a5a4a5\" points=\"691.04,-373.25 691.04,-558.25 867.04,-558.25 867.04,-373.25 691.04,-373.25\"/>\n",
       "<text text-anchor=\"middle\" x=\"779.04\" y=\"-540.95\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">Inputs</text>\n",
       "</g>\n",
       "<g id=\"clust11\" class=\"cluster\">\n",
       "<title>clusterkilled_processsleepOutputsWithInjection</title>\n",
       "<defs>\n",
       "<linearGradient id=\"clust11_l_10\" gradientUnits=\"userSpaceOnUse\" x1=\"1066.04\" y1=\"-465.75\" x2=\"903.04\" y2=\"-465.75\" >\n",
       "<stop offset=\"0\" style=\"stop-color:#a5a4a5;stop-opacity:1.;\"/>\n",
       "<stop offset=\"1\" style=\"stop-color:#e4e3e4;stop-opacity:1.;\"/>\n",
       "</linearGradient>\n",
       "</defs>\n",
       "<polygon fill=\"url(#clust11_l_10)\" stroke=\"#a5a4a5\" points=\"903.04,-373.25 903.04,-558.25 1066.04,-558.25 1066.04,-373.25 903.04,-373.25\"/>\n",
       "<text text-anchor=\"middle\" x=\"984.54\" y=\"-540.95\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">OutputsWithInjection</text>\n",
       "</g>\n",
       "<g id=\"clust12\" class=\"cluster\">\n",
       "<title>clusterkilled_processfloating_dreams</title>\n",
       "<defs>\n",
       "<linearGradient id=\"clust12_l_11\" gradientUnits=\"userSpaceOnUse\" x1=\"1082.04\" y1=\"-424.25\" x2=\"1465.04\" y2=\"-424.25\" >\n",
       "<stop offset=\"0\" style=\"stop-color:#bfe2bf;stop-opacity:1.;\"/>\n",
       "<stop offset=\"1\" style=\"stop-color:#bfe2bf;stop-opacity:1.;\"/>\n",
       "</linearGradient>\n",
       "</defs>\n",
       "<polygon fill=\"url(#clust12_l_11)\" stroke=\"#2ca02c\" stroke-width=\"2\" points=\"1082.04,-311.25 1082.04,-537.25 1465.04,-537.25 1465.04,-311.25 1082.04,-311.25\"/>\n",
       "<text text-anchor=\"middle\" x=\"1273.54\" y=\"-519.95\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">floating_dreams: Float</text>\n",
       "</g>\n",
       "<g id=\"clust13\" class=\"cluster\">\n",
       "<title>clusterkilled_processfloating_dreamsInputs</title>\n",
       "<defs>\n",
       "<linearGradient id=\"clust13_l_12\" gradientUnits=\"userSpaceOnUse\" x1=\"1090.04\" y1=\"-411.75\" x2=\"1266.04\" y2=\"-411.75\" >\n",
       "<stop offset=\"0\" style=\"stop-color:#a5a4a5;stop-opacity:1.;\"/>\n",
       "<stop offset=\"1\" style=\"stop-color:#e4e3e4;stop-opacity:1.;\"/>\n",
       "</linearGradient>\n",
       "</defs>\n",
       "<polygon fill=\"url(#clust13_l_12)\" stroke=\"#a5a4a5\" points=\"1090.04,-319.25 1090.04,-504.25 1266.04,-504.25 1266.04,-319.25 1090.04,-319.25\"/>\n",
       "<text text-anchor=\"middle\" x=\"1178.04\" y=\"-486.95\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">Inputs</text>\n",
       "</g>\n",
       "<g id=\"clust14\" class=\"cluster\">\n",
       "<title>clusterkilled_processfloating_dreamsOutputsWithInjection</title>\n",
       "<defs>\n",
       "<linearGradient id=\"clust14_l_13\" gradientUnits=\"userSpaceOnUse\" x1=\"1457.04\" y1=\"-411.75\" x2=\"1294.04\" y2=\"-411.75\" >\n",
       "<stop offset=\"0\" style=\"stop-color:#a5a4a5;stop-opacity:1.;\"/>\n",
       "<stop offset=\"1\" style=\"stop-color:#e4e3e4;stop-opacity:1.;\"/>\n",
       "</linearGradient>\n",
       "</defs>\n",
       "<polygon fill=\"url(#clust14_l_13)\" stroke=\"#a5a4a5\" points=\"1294.04,-319.25 1294.04,-504.25 1457.04,-504.25 1457.04,-319.25 1294.04,-319.25\"/>\n",
       "<text text-anchor=\"middle\" x=\"1375.54\" y=\"-486.95\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">OutputsWithInjection</text>\n",
       "</g>\n",
       "<!-- clusterkilled_processInputsrun -->\n",
       "<g id=\"node1\" class=\"node\">\n",
       "<title>clusterkilled_processInputsrun</title>\n",
       "<polygon fill=\"#21bfd8\" stroke=\"#21bfd8\" points=\"159.02,-71.25 117.02,-71.25 117.02,-47.25 159.02,-47.25 171.02,-59.25 159.02,-71.25\"/>\n",
       "<text text-anchor=\"middle\" x=\"144.02\" y=\"-54.58\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">run</text>\n",
       "</g>\n",
       "<!-- clusterkilled_processOutputsran -->\n",
       "<g id=\"node5\" class=\"node\">\n",
       "<title>clusterkilled_processOutputsran</title>\n",
       "<polygon fill=\"#21bfd8\" stroke=\"#21bfd8\" points=\"1624.9,-123.25 1582.9,-123.25 1582.9,-99.25 1624.9,-99.25 1636.9,-111.25 1624.9,-123.25\"/>\n",
       "<text text-anchor=\"middle\" x=\"1609.9\" y=\"-106.58\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">ran</text>\n",
       "</g>\n",
       "<!-- clusterkilled_processInputsrun&#45;&gt;clusterkilled_processOutputsran -->\n",
       "<!-- clusterkilled_processInputsaccumulate_and_run -->\n",
       "<g id=\"node2\" class=\"node\">\n",
       "<title>clusterkilled_processInputsaccumulate_and_run</title>\n",
       "<polygon fill=\"#21bfd8\" stroke=\"#21bfd8\" points=\"212.02,-125.25 64.02,-125.25 64.02,-101.25 212.02,-101.25 224.02,-113.25 212.02,-125.25\"/>\n",
       "<text text-anchor=\"middle\" x=\"144.02\" y=\"-108.58\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">accumulate_and_run</text>\n",
       "</g>\n",
       "<!-- clusterkilled_processInputssleep_time__user_input -->\n",
       "<g id=\"node3\" class=\"node\">\n",
       "<title>clusterkilled_processInputssleep_time__user_input</title>\n",
       "<ellipse fill=\"#edb22c\" stroke=\"#edb22c\" cx=\"144.02\" cy=\"-221.25\" rx=\"124.34\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"144.02\" y=\"-216.57\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">sleep_time__user_input</text>\n",
       "</g>\n",
       "<!-- clusterkilled_processsleep_timeInputsuser_input -->\n",
       "<g id=\"node11\" class=\"node\">\n",
       "<title>clusterkilled_processsleep_timeInputsuser_input</title>\n",
       "<ellipse fill=\"#edb22c\" stroke=\"#edb22c\" cx=\"388.04\" cy=\"-327.25\" rx=\"62.78\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"388.04\" y=\"-322.57\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">user_input</text>\n",
       "</g>\n",
       "<!-- clusterkilled_processInputssleep_time__user_input&#45;&gt;clusterkilled_processsleep_timeInputsuser_input -->\n",
       "<g id=\"edge12\" class=\"edge\">\n",
       "<title>clusterkilled_processInputssleep_time__user_input&#45;&gt;clusterkilled_processsleep_timeInputsuser_input</title>\n",
       "<path fill=\"none\" stroke=\"#edb22c\" d=\"M173.68,-239.06C192.32,-250.27 218.2,-265.18 245.69,-279.01\"/>\n",
       "<path fill=\"none\" stroke=\"#edb22c\" d=\"M245.69,-279.01C260.97,-286.7 276.74,-294.05 292.04,-300.25 303.48,-304.88 315.99,-309.05 328.07,-312.64\"/>\n",
       "<polygon fill=\"#edb22c\" stroke=\"#edb22c\" points=\"326.97,-315.96 337.55,-315.35 328.9,-309.23 326.97,-315.96\"/>\n",
       "</g>\n",
       "<!-- clusterkilled_processInputsskipped_my_nap__other -->\n",
       "<g id=\"node4\" class=\"node\">\n",
       "<title>clusterkilled_processInputsskipped_my_nap__other</title>\n",
       "<ellipse fill=\"#edb22c\" stroke=\"#edb22c\" cx=\"144.02\" cy=\"-167.25\" rx=\"128.02\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"144.02\" y=\"-162.57\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">skipped_my_nap__other</text>\n",
       "</g>\n",
       "<!-- clusterkilled_processskipped_my_napInputsother -->\n",
       "<g id=\"node18\" class=\"node\">\n",
       "<title>clusterkilled_processskipped_my_napInputsother</title>\n",
       "<ellipse fill=\"#edb22c\" stroke=\"#edb22c\" cx=\"779.04\" cy=\"-165.25\" rx=\"37\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"779.04\" y=\"-160.57\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">other</text>\n",
       "</g>\n",
       "<!-- clusterkilled_processInputsskipped_my_nap__other&#45;&gt;clusterkilled_processskipped_my_napInputsother -->\n",
       "<g id=\"edge13\" class=\"edge\">\n",
       "<title>clusterkilled_processInputsskipped_my_nap__other&#45;&gt;clusterkilled_processskipped_my_napInputsother</title>\n",
       "<path fill=\"none\" stroke=\"#edb22c\" d=\"M272.36,-166.85C342.98,-166.62 433.44,-166.34 518.27,-166.07\"/>\n",
       "<path fill=\"none\" stroke=\"#edb22c\" d=\"M518.27,-166.07C603.11,-165.8 682.33,-165.55 730.45,-165.4\"/>\n",
       "<polygon fill=\"#edb22c\" stroke=\"#edb22c\" points=\"730.33,-168.9 740.32,-165.37 730.3,-161.9 730.33,-168.9\"/>\n",
       "</g>\n",
       "<!-- clusterkilled_processOutputsfailed -->\n",
       "<g id=\"node6\" class=\"node\">\n",
       "<title>clusterkilled_processOutputsfailed</title>\n",
       "<polygon fill=\"#21bfd8\" stroke=\"#21bfd8\" points=\"1625.03,-177.25 1582.78,-177.25 1582.78,-153.25 1625.03,-153.25 1637.03,-165.25 1625.03,-177.25\"/>\n",
       "<text text-anchor=\"middle\" x=\"1609.9\" y=\"-160.57\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">failed</text>\n",
       "</g>\n",
       "<!-- clusterkilled_processOutputsskipped_my_nap__rmul -->\n",
       "<g id=\"node7\" class=\"node\">\n",
       "<title>clusterkilled_processOutputsskipped_my_nap__rmul</title>\n",
       "<ellipse fill=\"#edb22c\" stroke=\"#edb22c\" cx=\"1609.9\" cy=\"-219.25\" rx=\"124.86\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"1609.9\" y=\"-214.57\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">skipped_my_nap__rmul</text>\n",
       "</g>\n",
       "<!-- clusterkilled_processOutputsfloating_dreams__float -->\n",
       "<g id=\"node8\" class=\"node\">\n",
       "<title>clusterkilled_processOutputsfloating_dreams__float</title>\n",
       "<ellipse fill=\"#edb22c\" stroke=\"#edb22c\" cx=\"1609.9\" cy=\"-273.25\" rx=\"120.13\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"1609.9\" y=\"-268.57\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">floating_dreams__float</text>\n",
       "</g>\n",
       "<!-- clusterkilled_processsleep_timeInputsrun -->\n",
       "<g id=\"node9\" class=\"node\">\n",
       "<title>clusterkilled_processsleep_timeInputsrun</title>\n",
       "<polygon fill=\"#21bfd8\" stroke=\"#21bfd8\" points=\"403.04,-231.25 361.04,-231.25 361.04,-207.25 403.04,-207.25 415.04,-219.25 403.04,-231.25\"/>\n",
       "<text text-anchor=\"middle\" x=\"388.04\" y=\"-214.57\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">run</text>\n",
       "</g>\n",
       "<!-- clusterkilled_processsleep_timeOutputsWithInjectionran -->\n",
       "<g id=\"node12\" class=\"node\">\n",
       "<title>clusterkilled_processsleep_timeOutputsWithInjectionran</title>\n",
       "<polygon fill=\"#21bfd8\" stroke=\"#21bfd8\" points=\"600.04,-231.25 558.04,-231.25 558.04,-207.25 600.04,-207.25 612.04,-219.25 600.04,-231.25\"/>\n",
       "<text text-anchor=\"middle\" x=\"585.04\" y=\"-214.57\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">ran</text>\n",
       "</g>\n",
       "<!-- clusterkilled_processsleep_timeInputsrun&#45;&gt;clusterkilled_processsleep_timeOutputsWithInjectionran -->\n",
       "<!-- clusterkilled_processsleep_timeInputsaccumulate_and_run -->\n",
       "<g id=\"node10\" class=\"node\">\n",
       "<title>clusterkilled_processsleep_timeInputsaccumulate_and_run</title>\n",
       "<polygon fill=\"#21bfd8\" stroke=\"#21bfd8\" points=\"456.04,-285.25 308.04,-285.25 308.04,-261.25 456.04,-261.25 468.04,-273.25 456.04,-285.25\"/>\n",
       "<text text-anchor=\"middle\" x=\"388.04\" y=\"-268.57\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">accumulate_and_run</text>\n",
       "</g>\n",
       "<!-- clusterkilled_processskipped_my_napInputsaccumulate_and_run -->\n",
       "<g id=\"node16\" class=\"node\">\n",
       "<title>clusterkilled_processskipped_my_napInputsaccumulate_and_run</title>\n",
       "<polygon fill=\"#21bfd8\" stroke=\"#21bfd8\" points=\"847.04,-231.25 699.04,-231.25 699.04,-207.25 847.04,-207.25 859.04,-219.25 847.04,-231.25\"/>\n",
       "<text text-anchor=\"middle\" x=\"779.04\" y=\"-214.57\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">accumulate_and_run</text>\n",
       "</g>\n",
       "<!-- clusterkilled_processsleep_timeOutputsWithInjectionran&#45;&gt;clusterkilled_processskipped_my_napInputsaccumulate_and_run -->\n",
       "<g id=\"edge6\" class=\"edge\">\n",
       "<title>clusterkilled_processsleep_timeOutputsWithInjectionran&#45;&gt;clusterkilled_processskipped_my_napInputsaccumulate_and_run</title>\n",
       "<path fill=\"none\" stroke=\"#21bfd8\" d=\"M612.33,-219.25C622.13,-219.25 634.02,-219.25 646.89,-219.25\"/>\n",
       "<path fill=\"none\" stroke=\"#21bfd8\" d=\"M646.89,-219.25C659.77,-219.25 673.62,-219.25 687.32,-219.25\"/>\n",
       "<polygon fill=\"#21bfd8\" stroke=\"#21bfd8\" points=\"687.08,-222.75 697.08,-219.25 687.08,-215.75 687.08,-222.75\"/>\n",
       "</g>\n",
       "<!-- clusterkilled_processsleepInputsaccumulate_and_run -->\n",
       "<g id=\"node23\" class=\"node\">\n",
       "<title>clusterkilled_processsleepInputsaccumulate_and_run</title>\n",
       "<polygon fill=\"#21bfd8\" stroke=\"#21bfd8\" points=\"847.04,-465.25 699.04,-465.25 699.04,-441.25 847.04,-441.25 859.04,-453.25 847.04,-465.25\"/>\n",
       "<text text-anchor=\"middle\" x=\"779.04\" y=\"-448.57\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">accumulate_and_run</text>\n",
       "</g>\n",
       "<!-- clusterkilled_processsleep_timeOutputsWithInjectionran&#45;&gt;clusterkilled_processsleepInputsaccumulate_and_run -->\n",
       "<g id=\"edge7\" class=\"edge\">\n",
       "<title>clusterkilled_processsleep_timeOutputsWithInjectionran&#45;&gt;clusterkilled_processsleepInputsaccumulate_and_run</title>\n",
       "<path fill=\"none\" stroke=\"#21bfd8\" d=\"M612.37,-219.53C632.93,-221.3 660.38,-227.38 675.04,-246.25 679.14,-251.53 680.01,-271.63 679.81,-296.64\"/>\n",
       "<path fill=\"none\" stroke=\"#21bfd8\" d=\"M679.81,-296.64C679.41,-346.47 674.77,-415.79 683.04,-426.25 685.08,-428.83 687.36,-431.18 689.81,-433.32\"/>\n",
       "<polygon fill=\"#21bfd8\" stroke=\"#21bfd8\" points=\"687.41,-435.9 697.57,-438.89 691.49,-430.21 687.41,-435.9\"/>\n",
       "</g>\n",
       "<!-- clusterkilled_processsleep_timeOutputsWithInjectionfailed -->\n",
       "<g id=\"node13\" class=\"node\">\n",
       "<title>clusterkilled_processsleep_timeOutputsWithInjectionfailed</title>\n",
       "<polygon fill=\"#21bfd8\" stroke=\"#21bfd8\" points=\"600.17,-285.25 557.92,-285.25 557.92,-261.25 600.17,-261.25 612.17,-273.25 600.17,-285.25\"/>\n",
       "<text text-anchor=\"middle\" x=\"585.04\" y=\"-268.57\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">failed</text>\n",
       "</g>\n",
       "<!-- clusterkilled_processsleep_timeOutputsWithInjectionuser_input -->\n",
       "<g id=\"node14\" class=\"node\">\n",
       "<title>clusterkilled_processsleep_timeOutputsWithInjectionuser_input</title>\n",
       "<ellipse fill=\"#edb22c\" stroke=\"#edb22c\" cx=\"585.04\" cy=\"-327.25\" rx=\"62.78\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"585.04\" y=\"-322.57\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">user_input</text>\n",
       "</g>\n",
       "<!-- clusterkilled_processskipped_my_napInputsobj -->\n",
       "<g id=\"node17\" class=\"node\">\n",
       "<title>clusterkilled_processskipped_my_napInputsobj</title>\n",
       "<ellipse fill=\"#edb22c\" stroke=\"#edb22c\" cx=\"779.04\" cy=\"-273.25\" rx=\"27\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"779.04\" y=\"-268.57\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">obj</text>\n",
       "</g>\n",
       "<!-- clusterkilled_processsleep_timeOutputsWithInjectionuser_input&#45;&gt;clusterkilled_processskipped_my_napInputsobj -->\n",
       "<g id=\"edge8\" class=\"edge\">\n",
       "<title>clusterkilled_processsleep_timeOutputsWithInjectionuser_input&#45;&gt;clusterkilled_processskipped_my_napInputsobj</title>\n",
       "<path fill=\"none\" stroke=\"#edb22c\" d=\"M631.06,-314.59C648.08,-309.8 668.13,-304.16 687.84,-298.62\"/>\n",
       "<path fill=\"none\" stroke=\"#edb22c\" d=\"M687.84,-298.62C707.56,-293.07 726.95,-287.62 742.67,-283.2\"/>\n",
       "<polygon fill=\"#edb22c\" stroke=\"#edb22c\" points=\"743.4,-286.63 752.08,-280.55 741.5,-279.89 743.4,-286.63\"/>\n",
       "</g>\n",
       "<!-- clusterkilled_processsleepInputst -->\n",
       "<g id=\"node24\" class=\"node\">\n",
       "<title>clusterkilled_processsleepInputst</title>\n",
       "<ellipse fill=\"#edb22c\" stroke=\"#edb22c\" cx=\"779.04\" cy=\"-507.25\" rx=\"27\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"779.04\" y=\"-502.57\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">t</text>\n",
       "</g>\n",
       "<!-- clusterkilled_processsleep_timeOutputsWithInjectionuser_input&#45;&gt;clusterkilled_processsleepInputst -->\n",
       "<g id=\"edge9\" class=\"edge\">\n",
       "<title>clusterkilled_processsleep_timeOutputsWithInjectionuser_input&#45;&gt;clusterkilled_processsleepInputst</title>\n",
       "<path fill=\"none\" stroke=\"#edb22c\" d=\"M592.51,-345.39C601.06,-367.53 618.06,-406.19 643.46,-439.79\"/>\n",
       "<path fill=\"none\" stroke=\"#edb22c\" d=\"M643.46,-439.79C654.94,-454.97 668.14,-469.12 683.04,-480.25 699.77,-492.74 722.07,-499.45 740.86,-503.06\"/>\n",
       "<polygon fill=\"#edb22c\" stroke=\"#edb22c\" points=\"740.09,-506.48 750.53,-504.68 741.25,-499.58 740.09,-506.48\"/>\n",
       "</g>\n",
       "<!-- clusterkilled_processskipped_my_napInputsrun -->\n",
       "<g id=\"node15\" class=\"node\">\n",
       "<title>clusterkilled_processskipped_my_napInputsrun</title>\n",
       "<polygon fill=\"#21bfd8\" stroke=\"#21bfd8\" points=\"794.04,-123.25 752.04,-123.25 752.04,-99.25 794.04,-99.25 806.04,-111.25 794.04,-123.25\"/>\n",
       "<text text-anchor=\"middle\" x=\"779.04\" y=\"-106.58\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">run</text>\n",
       "</g>\n",
       "<!-- clusterkilled_processskipped_my_napOutputsWithInjectionran -->\n",
       "<g id=\"node19\" class=\"node\">\n",
       "<title>clusterkilled_processskipped_my_napOutputsWithInjectionran</title>\n",
       "<polygon fill=\"#21bfd8\" stroke=\"#21bfd8\" points=\"999.04,-123.25 957.04,-123.25 957.04,-99.25 999.04,-99.25 1011.04,-111.25 999.04,-123.25\"/>\n",
       "<text text-anchor=\"middle\" x=\"984.04\" y=\"-106.58\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">ran</text>\n",
       "</g>\n",
       "<!-- clusterkilled_processskipped_my_napInputsrun&#45;&gt;clusterkilled_processskipped_my_napOutputsWithInjectionran -->\n",
       "<!-- clusterkilled_processskipped_my_napOutputsWithInjectionfailed -->\n",
       "<g id=\"node20\" class=\"node\">\n",
       "<title>clusterkilled_processskipped_my_napOutputsWithInjectionfailed</title>\n",
       "<polygon fill=\"#21bfd8\" stroke=\"#21bfd8\" points=\"999.17,-177.25 956.92,-177.25 956.92,-153.25 999.17,-153.25 1011.17,-165.25 999.17,-177.25\"/>\n",
       "<text text-anchor=\"middle\" x=\"984.04\" y=\"-160.57\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">failed</text>\n",
       "</g>\n",
       "<!-- clusterkilled_processskipped_my_napOutputsWithInjectionrmul -->\n",
       "<g id=\"node21\" class=\"node\">\n",
       "<title>clusterkilled_processskipped_my_napOutputsWithInjectionrmul</title>\n",
       "<ellipse fill=\"#edb22c\" stroke=\"#edb22c\" cx=\"984.04\" cy=\"-219.25\" rx=\"33.85\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"984.04\" y=\"-214.57\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">rmul</text>\n",
       "</g>\n",
       "<!-- clusterkilled_processskipped_my_napOutputsWithInjectionrmul&#45;&gt;clusterkilled_processOutputsskipped_my_nap__rmul -->\n",
       "<g id=\"edge14\" class=\"edge\">\n",
       "<title>clusterkilled_processskipped_my_napOutputsWithInjectionrmul&#45;&gt;clusterkilled_processOutputsskipped_my_nap__rmul</title>\n",
       "<path fill=\"none\" stroke=\"#edb22c\" d=\"M1018.14,-219.25C1060.06,-219.25 1137.14,-219.25 1222.1,-219.25\"/>\n",
       "<path fill=\"none\" stroke=\"#edb22c\" d=\"M1222.1,-219.25C1307.05,-219.25 1399.88,-219.25 1473.3,-219.25\"/>\n",
       "<polygon fill=\"#edb22c\" stroke=\"#edb22c\" points=\"1473.11,-222.75 1483.11,-219.25 1473.11,-215.75 1473.11,-222.75\"/>\n",
       "</g>\n",
       "<!-- clusterkilled_processsleepInputsrun -->\n",
       "<g id=\"node22\" class=\"node\">\n",
       "<title>clusterkilled_processsleepInputsrun</title>\n",
       "<polygon fill=\"#21bfd8\" stroke=\"#21bfd8\" points=\"794.04,-411.25 752.04,-411.25 752.04,-387.25 794.04,-387.25 806.04,-399.25 794.04,-411.25\"/>\n",
       "<text text-anchor=\"middle\" x=\"779.04\" y=\"-394.57\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">run</text>\n",
       "</g>\n",
       "<!-- clusterkilled_processsleepOutputsWithInjectionran -->\n",
       "<g id=\"node25\" class=\"node\">\n",
       "<title>clusterkilled_processsleepOutputsWithInjectionran</title>\n",
       "<polygon fill=\"#21bfd8\" stroke=\"#21bfd8\" points=\"999.04,-411.25 957.04,-411.25 957.04,-387.25 999.04,-387.25 1011.04,-399.25 999.04,-411.25\"/>\n",
       "<text text-anchor=\"middle\" x=\"984.04\" y=\"-394.57\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">ran</text>\n",
       "</g>\n",
       "<!-- clusterkilled_processsleepInputsrun&#45;&gt;clusterkilled_processsleepOutputsWithInjectionran -->\n",
       "<!-- clusterkilled_processfloating_dreamsInputsaccumulate_and_run -->\n",
       "<g id=\"node29\" class=\"node\">\n",
       "<title>clusterkilled_processfloating_dreamsInputsaccumulate_and_run</title>\n",
       "<polygon fill=\"#21bfd8\" stroke=\"#21bfd8\" points=\"1246.04,-411.25 1098.04,-411.25 1098.04,-387.25 1246.04,-387.25 1258.04,-399.25 1246.04,-411.25\"/>\n",
       "<text text-anchor=\"middle\" x=\"1178.04\" y=\"-394.57\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">accumulate_and_run</text>\n",
       "</g>\n",
       "<!-- clusterkilled_processsleepOutputsWithInjectionran&#45;&gt;clusterkilled_processfloating_dreamsInputsaccumulate_and_run -->\n",
       "<g id=\"edge10\" class=\"edge\">\n",
       "<title>clusterkilled_processsleepOutputsWithInjectionran&#45;&gt;clusterkilled_processfloating_dreamsInputsaccumulate_and_run</title>\n",
       "<path fill=\"none\" stroke=\"#21bfd8\" d=\"M1011.33,-399.25C1021.13,-399.25 1033.02,-399.25 1045.89,-399.25\"/>\n",
       "<path fill=\"none\" stroke=\"#21bfd8\" d=\"M1045.89,-399.25C1058.77,-399.25 1072.62,-399.25 1086.32,-399.25\"/>\n",
       "<polygon fill=\"#21bfd8\" stroke=\"#21bfd8\" points=\"1086.08,-402.75 1096.08,-399.25 1086.08,-395.75 1086.08,-402.75\"/>\n",
       "</g>\n",
       "<!-- clusterkilled_processsleepOutputsWithInjectionfailed -->\n",
       "<g id=\"node26\" class=\"node\">\n",
       "<title>clusterkilled_processsleepOutputsWithInjectionfailed</title>\n",
       "<polygon fill=\"#21bfd8\" stroke=\"#21bfd8\" points=\"999.17,-465.25 956.92,-465.25 956.92,-441.25 999.17,-441.25 1011.17,-453.25 999.17,-465.25\"/>\n",
       "<text text-anchor=\"middle\" x=\"984.04\" y=\"-448.57\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">failed</text>\n",
       "</g>\n",
       "<!-- clusterkilled_processsleepOutputsWithInjectiontime -->\n",
       "<g id=\"node27\" class=\"node\">\n",
       "<title>clusterkilled_processsleepOutputsWithInjectiontime</title>\n",
       "<ellipse fill=\"#edb22c\" stroke=\"#edb22c\" cx=\"984.04\" cy=\"-507.25\" rx=\"32.79\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"984.04\" y=\"-502.57\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">time</text>\n",
       "</g>\n",
       "<!-- clusterkilled_processfloating_dreamsInputsobj -->\n",
       "<g id=\"node30\" class=\"node\">\n",
       "<title>clusterkilled_processfloating_dreamsInputsobj</title>\n",
       "<ellipse fill=\"#edb22c\" stroke=\"#edb22c\" cx=\"1178.04\" cy=\"-453.25\" rx=\"27\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"1178.04\" y=\"-448.57\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">obj</text>\n",
       "</g>\n",
       "<!-- clusterkilled_processsleepOutputsWithInjectiontime&#45;&gt;clusterkilled_processfloating_dreamsInputsobj -->\n",
       "<g id=\"edge11\" class=\"edge\">\n",
       "<title>clusterkilled_processsleepOutputsWithInjectiontime&#45;&gt;clusterkilled_processfloating_dreamsInputsobj</title>\n",
       "<path fill=\"none\" stroke=\"#edb22c\" d=\"M1013.78,-499.17C1030.9,-494.35 1053.74,-487.93 1076.9,-481.41\"/>\n",
       "<path fill=\"none\" stroke=\"#edb22c\" d=\"M1076.9,-481.41C1100.07,-474.9 1123.55,-468.29 1141.95,-463.12\"/>\n",
       "<polygon fill=\"#edb22c\" stroke=\"#edb22c\" points=\"1142.53,-466.59 1151.21,-460.51 1140.64,-459.85 1142.53,-466.59\"/>\n",
       "</g>\n",
       "<!-- clusterkilled_processfloating_dreamsInputsrun -->\n",
       "<g id=\"node28\" class=\"node\">\n",
       "<title>clusterkilled_processfloating_dreamsInputsrun</title>\n",
       "<polygon fill=\"#21bfd8\" stroke=\"#21bfd8\" points=\"1193.04,-357.25 1151.04,-357.25 1151.04,-333.25 1193.04,-333.25 1205.04,-345.25 1193.04,-357.25\"/>\n",
       "<text text-anchor=\"middle\" x=\"1178.04\" y=\"-340.57\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">run</text>\n",
       "</g>\n",
       "<!-- clusterkilled_processfloating_dreamsOutputsWithInjectionran -->\n",
       "<g id=\"node31\" class=\"node\">\n",
       "<title>clusterkilled_processfloating_dreamsOutputsWithInjectionran</title>\n",
       "<polygon fill=\"#21bfd8\" stroke=\"#21bfd8\" points=\"1390.04,-357.25 1348.04,-357.25 1348.04,-333.25 1390.04,-333.25 1402.04,-345.25 1390.04,-357.25\"/>\n",
       "<text text-anchor=\"middle\" x=\"1375.04\" y=\"-340.57\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">ran</text>\n",
       "</g>\n",
       "<!-- clusterkilled_processfloating_dreamsInputsrun&#45;&gt;clusterkilled_processfloating_dreamsOutputsWithInjectionran -->\n",
       "<!-- clusterkilled_processfloating_dreamsOutputsWithInjectionfailed -->\n",
       "<g id=\"node32\" class=\"node\">\n",
       "<title>clusterkilled_processfloating_dreamsOutputsWithInjectionfailed</title>\n",
       "<polygon fill=\"#21bfd8\" stroke=\"#21bfd8\" points=\"1390.17,-411.25 1347.92,-411.25 1347.92,-387.25 1390.17,-387.25 1402.17,-399.25 1390.17,-411.25\"/>\n",
       "<text text-anchor=\"middle\" x=\"1375.04\" y=\"-394.57\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">failed</text>\n",
       "</g>\n",
       "<!-- clusterkilled_processfloating_dreamsOutputsWithInjectionfloat -->\n",
       "<g id=\"node33\" class=\"node\">\n",
       "<title>clusterkilled_processfloating_dreamsOutputsWithInjectionfloat</title>\n",
       "<ellipse fill=\"#edb22c\" stroke=\"#edb22c\" cx=\"1375.04\" cy=\"-453.25\" rx=\"32.79\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"1375.04\" y=\"-448.57\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">float</text>\n",
       "</g>\n",
       "<!-- clusterkilled_processfloating_dreamsOutputsWithInjectionfloat&#45;&gt;clusterkilled_processOutputsfloating_dreams__float -->\n",
       "<g id=\"edge15\" class=\"edge\">\n",
       "<title>clusterkilled_processfloating_dreamsOutputsWithInjectionfloat&#45;&gt;clusterkilled_processOutputsfloating_dreams__float</title>\n",
       "<path fill=\"none\" stroke=\"#edb22c\" d=\"M1406.95,-448.02C1424.88,-444 1447.34,-437.23 1465.04,-426.25 1482.54,-415.39 1499.32,-401.87 1514.85,-387.4\"/>\n",
       "<path fill=\"none\" stroke=\"#edb22c\" d=\"M1514.85,-387.4C1546.94,-357.5 1573.67,-323.51 1590.34,-300.46\"/>\n",
       "<polygon fill=\"#edb22c\" stroke=\"#edb22c\" points=\"1592.89,-302.9 1595.83,-292.72 1587.19,-298.84 1592.89,-302.9\"/>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<graphviz.graphs.Digraph at 0x145efeef2690>"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "wf.draw(size=(10, 10))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "ae65c771-48f4-40be-85a2-2ec1f3767934",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Cleanup\n",
    "wf.delete_storage(filename=wf.as_path().joinpath(\"recovery\"))\n",
    "wf.sleep._clean_graph_directory()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bc8988f2-55ff-41aa-af38-f476097d47a6",
   "metadata": {},
   "source": [
    "Running an entire graph remotely by saving then (loading, running, saving), and running individual nodes remotely with powerful executors _while the parent python process outlives the node run_ are both stable and reliable ways to parallelize your workflow. There's no guarantee you won't discover some unpleasant edge case, but overall you should expect quite smooth sailing.\n",
    "\n",
    "Running an individual node remotely and _killing the parent python process_ is a feature still under development, hence `_serialize_result` is still a private attribute. In particular, there are no guardrails against re-loading a killed graph with one or more remotely run children, and changing its input prior to re-running -- this would result in the loaded result file no longer being in sync with the node input! The power is available, but is still at a stage where it requires responsibility to wield it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "137970f1-03f6-4960-92c1-5fcc6b678cee",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "pyiron/latest (python3.11)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}