Paper (#72)

* paper draft and workflow

* stash commit .. just a rough bullet point version

* more edits

* small updates including aligning

* try commenting

* remove align

* try this

* going crazy

* found it

* now try aligned

* now try cite thing

* more methods

* parameters for optim

* initial scaling draft

* tidying up

* scaling plots

* better parallel, some cleanup, mention background better

* todo = done

* tidying up

* try thicker lines, some details, gpu plots

* incorporate gpu results, tidy up

* i think a first draft is done

* oops

* blumen and held refs

* jax ref

* fix blumen ref

* institutions, per recommendation

* last minute changes

.github/workflows/paper-draft.yaml

@@ -0,0 +1,23 @@
+on: [push]
+
+jobs:
+  paper:
+    runs-on: ubuntu-latest
+    name: Paper Draft
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v4
+      - name: Build draft PDF
+        uses: openjournals/openjournals-draft-action@master
+        with:
+          journal: joss
+          # This should be the path to the paper within your repo.
+          paper-path: paper.md
+      - name: Upload
+        uses: actions/upload-artifact@v1
+        with:
+          name: paper
+          # This is the output path where Pandoc will write the compiled
+          # PDF. Note, this should be the same directory as the input
+          # paper.md
+          path: paper.pdf

codemeta.json

@@ -0,0 +1,17 @@
+{
+  "@context": "https://raw.githubusercontent.com/codemeta/codemeta/master/codemeta.jsonld",
+  "@type": "Code",
+  "author": [
+
+  ],
+  "identifier": "",
+  "codeRepository": "https://github.com/timothyas/xesn",
+  "datePublished": "2023-12-15",
+  "dateModified": "2023-12-15",
+  "dateCreated": "2023-12-15",
+  "description": "Echo State Networks powered by xarray and dask.",
+  "keywords": "echo state networks, machine learning, weather, climate, forecasting",
+  "license": "LGPL v3",
+  "title": "xesn",
+  "version": "v0.1.2"
+}

docs/methods.rst

@@ -23,7 +23,7 @@ is defined as follows:
 
 .. math::
    \mathbf{r}(n + 1) = (1 - \alpha) \mathbf{r}(n) +
-    \alpha \tanh( \mathbf{W}\mathbf{r} + \mathbf{W}_\text{in}\mathbf{u}(n) +
+    \alpha \tanh( \mathbf{W}\mathbf{r}(n) + \mathbf{W}_\text{in}\mathbf{u}(n) +
    \mathbf{b})
 
 .. math::
@@ -38,6 +38,8 @@ The sizes of each of these vectors is specified to :class:`xesn.ESN` via the
 This form of the ESN has a "leaky" reservoir,
 where the ``leak_rate`` parameter :math:`\alpha`
 determines how much of the previous hidden state to propagate forward in time.
+This ESN implementation is *eager*, in the sense that all of the inputs, network
+parameters, targets (during training), and predictions are held in memory.
 
 Also, this form of the ESN assumes a linear readout, i.e., we do not transform
 the hidden state nonlinearly or augment it with the input or output state
@@ -185,8 +187,10 @@ The :class:`xesn.LazyESN` architecture inherits most of its functionality from
 :class:`xesn.ESN`.
 The key difference between the two is how they interact with the underlying data
 they're working with.
-While the standard ESN had a single network, :class:`xesn.LazyESN` distributes
-multiple networks to different subdomains of a single dataset.
+While the standard ESN had a single network that is eagerly operated on in
+memory, :class:`xesn.LazyESN` distributes
+multiple networks to different subdomains of a single dataset and invokes *lazy*
+operations via the `dask.Array` API.
 This process is described with an example below.
 
 Example: SQG Turbulence Dataset
@@ -323,7 +327,7 @@ The distributed ESN equations are
 
 .. math::
    \mathbf{r}_k(n + 1) = (1 - \alpha) \mathbf{r}_k(n) +
-    \alpha \tanh( \mathbf{W}\mathbf{r}_k + \mathbf{W}_\text{in}\mathbf{u}_k(n) +
+    \alpha \tanh( \mathbf{W}\mathbf{r}_k(n) + \mathbf{W}_\text{in}\mathbf{u}_k(n) +
    \mathbf{b})
 
 .. math::

docs/references.bib

@@ -1084,7 +1084,7 @@ @article{tulloch_note_2009
 }
 
 @article{blumen_uniform_1978,
-	title = {{\noopsort{A}{Uniform}} {Potential} {Vorticity} {Flow}: {Part} {I}. {Theory} of {Wave} {Interactions} and {Two}-{Dimensional} {Turbulence}},
+	title = {{Uniform} {Potential} {Vorticity} {Flow}: {Part} {I}. {Theory} of {Wave} {Interactions} and {Two}-{Dimensional} {Turbulence}},
 	volume = {35},
 	issn = {0022-4928, 1520-0469},
 	shorttitle = {Uniform {Potential} {Vorticity} {Flow}},
@@ -1714,3 +1714,149 @@ @article{smith_temporal_2023
 	keywords = {machine learning, recurrent neural networks, numerical weather prediction, geophysical fluid dynamics},
 	pages = {e2023MS003792},
 }
+
+
+@article{hoyer_xarray_2017,
+	title = {xarray: {N}-{D} labeled {Arrays} and {Datasets} in {Python}},
+	volume = {5},
+	issn = {2049-9647},
+	shorttitle = {xarray},
+	url = {https://openresearchsoftware.metajnl.com/articles/10.5334/jors.148},
+	doi = {10.5334/jors.148},
+	abstract = {The Journal of Open Research Software (JORS) features peer reviewed Software Metapapers describing research software with high reuse potential. We are working with a number of specialist and institutional repositories to ensure that the associated software is professionally archived, preserved, and is openly available. Equally importantly, the software and the papers will be citable, and reuse will be tracked.JORS also publishes full-length research papers that cover different aspects of creating, maintaining and evaluating open source research software. The aim of the section is to promote the dissemination of best practice and experience related to the development and maintenance of reusable, sustainable research software.},
+	language = {en-US},
+	number = {1},
+	urldate = {2024-05-06},
+	journal = {Journal of Open Research Software},
+	author = {Hoyer, Stephan and Hamman, Joe},
+	month = apr,
+	year = {2017},
+	file = {Hoyer_Hamman_2017_xarray.pdf:/Users/tsmith/Drive/zotero/Hoyer_Hamman_2017_xarray.pdf:application/pdf},
+}
+
+
+@article{jaeger_harnessing_2004,
+	title = {Harnessing {Nonlinearity}: {Predicting} {Chaotic} {Systems} and {Saving} {Energy} in {Wireless} {Communication}},
+	volume = {304},
+	shorttitle = {Harnessing {Nonlinearity}},
+	url = {https://www.science.org/doi/10.1126/science.1091277},
+	doi = {10.1126/science.1091277},
+	abstract = {We present a method for learning nonlinear systems, echo state networks (ESNs). ESNs employ artificial recurrent neural networks in a way that has recently been proposed independently as a learning mechanism in biological brains. The learning method is computationally efficient and easy to use. On a benchmark task of predicting a chaotic time series, accuracy is improved by a factor of 2400 over previous techniques. The potential for engineering applications is illustrated by equalizing a communication channel, where the signal error rate is improved by two orders of magnitude.},
+	number = {5667},
+	urldate = {2024-05-06},
+	journal = {Science},
+	author = {Jaeger, Herbert and Haas, Harald},
+	month = apr,
+	year = {2004},
+	note = {Publisher: American Association for the Advancement of Science},
+	pages = {78--80},
+	file = {Jaeger_Haas_2004_Harnessing_Nonlinearity.pdf:/Users/tsmith/Drive/zotero/Jaeger_Haas_2004_Harnessing_Nonlinearity.pdf:application/pdf},
+}
+
+
+@article{harris_array_2020,
+	title = {Array programming with {NumPy}},
+	volume = {585},
+	copyright = {2020 The Author(s)},
+	issn = {1476-4687},
+	url = {https://www.nature.com/articles/s41586-020-2649-2},
+	doi = {10.1038/s41586-020-2649-2},
+	abstract = {Array programming provides a powerful, compact and expressive syntax for accessing, manipulating and operating on data in vectors, matrices and higher-dimensional arrays. NumPy is the primary array programming library for the Python language. It has an essential role in research analysis pipelines in fields as diverse as physics, chemistry, astronomy, geoscience, biology, psychology, materials science, engineering, finance and economics. For example, in astronomy, NumPy was an important part of the software stack used in the discovery of gravitational waves1 and in the first imaging of a black hole2. Here we review how a few fundamental array concepts lead to a simple and powerful programming paradigm for organizing, exploring and analysing scientific data. NumPy is the foundation upon which the scientific Python ecosystem is constructed. It is so pervasive that several projects, targeting audiences with specialized needs, have developed their own NumPy-like interfaces and array objects. Owing to its central position in the ecosystem, NumPy increasingly acts as an interoperability layer between such array computation libraries and, together with its application programming interface (API), provides a flexible framework to support the next decade of scientific and industrial analysis.},
+	language = {en},
+	number = {7825},
+	urldate = {2024-05-06},
+	journal = {Nature},
+	author = {Harris, Charles R. and Millman, K. Jarrod and van der Walt, Stéfan J. and Gommers, Ralf and Virtanen, Pauli and Cournapeau, David and Wieser, Eric and Taylor, Julian and Berg, Sebastian and Smith, Nathaniel J. and Kern, Robert and Picus, Matti and Hoyer, Stephan and van Kerkwijk, Marten H. and Brett, Matthew and Haldane, Allan and del Río, Jaime Fernández and Wiebe, Mark and Peterson, Pearu and Gérard-Marchant, Pierre and Sheppard, Kevin and Reddy, Tyler and Weckesser, Warren and Abbasi, Hameer and Gohlke, Christoph and Oliphant, Travis E.},
+	month = sep,
+	year = {2020},
+	note = {Publisher: Nature Publishing Group},
+	keywords = {Computational neuroscience, Computational science, Computer science, Software, Solar physics},
+	pages = {357--362},
+	file = {Harris_et_al_2020_Array_programming_with_NumPy.pdf:/Users/tsmith/Drive/zotero/Harris_et_al_2020_Array_programming_with_NumPy.pdf:application/pdf},
+}
+
+
+@incollection{Trouvain2020,
+  doi = {10.1007/978-3-030-61616-8_40},
+  url = {https://doi.org/10.1007/978-3-030-61616-8_40},
+  year = {2020},
+  publisher = {Springer International Publishing},
+  pages = {494--505},
+  author = {Nathan Trouvain and Luca Pedrelli and Thanh Trung Dinh and Xavier Hinaut},
+  title = {{ReservoirPy}: An Efficient and User-Friendly Library to Design Echo State Networks},
+  booktitle = {Artificial Neural Networks and Machine Learning {\textendash} {ICANN} 2020}
+}
+
+@inproceedings{hyperopt,
+    author={J. Bergstra and D. Yamins and D. D. Cox},
+    title={Making a Science of Model Search: Hyperparameter Optimization in Hundreds of Dimensions for Vision Architectures},
+    year={2013},
+    pages={I-115--I-23},
+    booktitle={TProc. of the 30th International Conference on Machine Learning (ICML 2013)},
+}
+
+
+
+@article{arcomano_hybrid_2022,
+	title = {A {Hybrid} {Approach} to {Atmospheric} {Modeling} {That} {Combines} {Machine} {Learning} {With} a {Physics}-{Based} {Numerical} {Model}},
+	volume = {14},
+	issn = {1942-2466},
+	url = {https://onlinelibrary.wiley.com/doi/abs/10.1029/2021MS002712},
+	doi = {10.1029/2021MS002712},
+	abstract = {This paper describes an implementation of the combined hybrid-parallel prediction (CHyPP) approach of Wikner et al. (2020), https://doi.org/10.1063/5.0005541 on a low-resolution atmospheric global circulation model (AGCM). The CHyPP approach combines a physics-based numerical model of a dynamical system (e.g., the atmosphere) with a computationally efficient type of machine learning (ML) called reservoir computing to construct a hybrid model. This hybrid atmospheric model produces more accurate forecasts of most atmospheric state variables than the host AGCM for the first 7–8 forecast days, and for even longer times for the temperature and humidity near the earth's surface. It also produces more accurate forecasts than a model based only on ML, or a model that combines linear regression, rather than ML, with the AGCM. The potential of the CHyPP approach for climate research is demonstrated by a 10-year long hybrid model simulation of the atmospheric general circulation, which shows that the hybrid model can simulate the general circulation with substantially smaller systematic errors and more realistic variability than the host AGCM.},
+	language = {en},
+	number = {3},
+	urldate = {2022-06-17},
+	journal = {Journal of Advances in Modeling Earth Systems},
+	author = {Arcomano, Troy and Szunyogh, Istvan and Wikner, Alexander and Pathak, Jaideep and Hunt, Brian R. and Ott, Edward},
+	year = {2022},
+	note = {\_eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1029/2021MS002712},
+	pages = {e2021MS002712},
+	file = {Arcomano_et_al_2022_A_Hybrid_Approach_to_Atmospheric_Modeling_That_Combines_Machine_Learning_With_a.pdf:/Users/tsmith/Drive/zotero/Arcomano_et_al_2022_A_Hybrid_Approach_to_Atmospheric_Modeling_That_Combines_Machine_Learning_With_a.pdf:application/pdf;Snapshot:/Users/tsmith/Zotero/storage/4FCCNJQF/2021MS002712.html:text/html},
+}
+
+@article{arcomano_hybrid_2023,
+	title = {A {Hybrid} {Atmospheric} {Model} {Incorporating} {Machine} {Learning} {Can} {Capture} {Dynamical} {Processes} {Not} {Captured} by {Its} {Physics}-{Based} {Component}},
+	volume = {50},
+	issn = {1944-8007},
+	url = {https://onlinelibrary.wiley.com/doi/abs/10.1029/2022GL102649},
+	doi = {10.1029/2022GL102649},
+	abstract = {It is shown that a recently developed hybrid modeling approach that combines machine learning (ML) with an atmospheric global circulation model (AGCM) can serve as a basis for capturing atmospheric processes not captured by the AGCM. This power of the approach is illustrated by three examples from a decades-long climate simulation experiment. The first example demonstrates that the hybrid model can produce sudden stratospheric warming, a dynamical process of nature not resolved by the low resolution AGCM component of the hybrid model. The second and third example show that introducing 6-hr cumulative precipitation and sea surface temperature (SST) as ML-based prognostic variables improves the precipitation climatology and leads to a realistic ENSO signal in the SST and atmospheric surface pressure.},
+	language = {en},
+	number = {8},
+	urldate = {2023-05-02},
+	journal = {Geophysical Research Letters},
+	author = {Arcomano, Troy and Szunyogh, Istvan and Wikner, Alexander and Hunt, Brian R. and Ott, Edward},
+	year = {2023},
+	note = {\_eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1029/2022GL102649},
+	keywords = {machine learning, climate, numerical weather prediction},
+	pages = {e2022GL102649},
+	file = {Arcomano_et_al_2023_A_Hybrid_Atmospheric_Model_Incorporating_Machine_Learning_Can_Capture_Dynamical.pdf:/Users/tsmith/Drive/zotero/Arcomano_et_al_2023_A_Hybrid_Atmospheric_Model_Incorporating_Machine_Learning_Can_Capture_Dynamical.pdf:application/pdf;Snapshot:/Users/tsmith/Zotero/storage/GTX98CFJ/2022GL102649.html:text/html},
+}
+
+
+@misc{arcomano_code,
+	title = {Arcomano1234/{SPEEDY}-{ML}: {V1} - {GRL} {Paper}},
+	url = {https://doi.org/10.5281/zenodo.7508156},
+	publisher = {Zenodo},
+	author = {Arcomano, Troy},
+	month = jan,
+	year = {2023},
+	doi = {10.5281/zenodo.7508156},
+}
+
+@article{liaw2018tune,
+    title={Tune: A Research Platform for Distributed Model Selection and Training},
+    author={Liaw, Richard and Liang, Eric and Nishihara, Robert
+            and Moritz, Philipp and Gonzalez, Joseph E and Stoica, Ion},
+    journal={arXiv preprint arXiv:1807.05118},
+    year={2018}
+}
+
+@software{jax2018github,
+  author = {James Bradbury and Roy Frostig and Peter Hawkins and Matthew James Johnson and Chris Leary and Dougal Maclaurin and George Necula and Adam Paszke and Jake Vander{P}las and Skye Wanderman-{M}ilne and Qiao Zhang},
+  title = {{JAX}: composable transformations of {P}ython+{N}um{P}y programs},
+  url = {http://github.com/google/jax},
+  version = {0.3.13},
+  year = {2018},
+}