Dask CloudFormation improvements (#25)

* enable dask dashboard

* split cloudformation into two for simpler updates

* added additional explanatory steps

* move workers to private subnet and separate security groups

* make deployment optional via the pipeline

* support Fargate spot pricing

* remove LaunchType

* Nested cloudformation stacks

* build container image first

* updated dask environment documentation

* corrected template URL

* clean up working file

* create separate era5 notebook for demo, with cleared outputs

* minor updates to architecture diagram

* allow spot pricing to be enabled in nested stack

* additional demo steps in era5 notebook

* added parameters for notebook and dask sizing

* revert to original era5 notebook

* notify codepipeline even if stack is not being created

* sagemaker notebook lifecycle fixes

* display dask dashboard URL in notebook

Author: stevengillard

examples/dask/README.md

@@ -7,7 +7,7 @@ There are several Jupyter notebooks showing examples of how to work with data di
 
 ### Getting started
 
-[![cloudformation-launch-stack](cloudformation/cloudformation-launch-stack.png)](https://console.aws.amazon.com/cloudformation/home?region=us-east-1#/stacks/new?stackName=dask&templateURL=https://s3.amazonaws.com/docs.opendata.aws/cloudformation/dask-fargate.yaml)
+[![cloudformation-launch-stack](cloudformation/cloudformation-launch-stack.png)](https://console.aws.amazon.com/cloudformation/home?region=us-east-1#/stacks/new?stackName=dask&templateURL=https://s3.amazonaws.com/docs.opendata.aws/cloudformation/dask-fargate-nested.yaml)
 
 1. Launch the stack, by default it will be in the `us-east-1` region (since that is the location of most of the weather & climate data) but you can change it to any region you prefer.
 
@@ -37,4 +37,17 @@ The Jupyter notebook environment will be set up with a kernel called `conda_dask
 
 ### Architecture
 
-![architecture](cloudformation/architecture.png)
+![architecture](cloudformation/dask-architecture.png)
+
+The diagram above shows the architecture at a high level.  The CloudFormation template deploys as two nested stacks, one which deploys a pipeline to build the container image, the second to create the dask environment and associated resources.
+
+The environment includes:
+1. A Virtual Private Cloud (VPC) with security groups to restrict traffic
+1. A public subnet with NAT Gateway for the scheduler and notebook, and a single private subnet for the dask workers
+1. A S3 Gateway endpoint to enable dask workers to access S3 without traversing the NAT Gateway
+1. An Elastic Container Service (ECS) cluster
+1. ECS service definitions for the dask scheduler and dask workers
+1. A SageMaker Notebook instance
+
+When first deployed, the pipeline will create the container image which is deployed into the dask environment.  Future updates to the specified GitHub repository will trigger an automatic rebuild of the container image.  
+

examples/dask/cloudformation/dask-architecture.drawio

@@ -0,0 +1 @@
+<mxfile host="Electron" modified="2021-04-29T11:59:51.663Z" agent="5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) draw.io/14.1.8 Chrome/87.0.4280.88 Electron/11.1.1 Safari/537.36" etag="cdSKm0YymgiD-ka6TRxj" version="14.1.8" type="device"><diagram id="vM1j6jhTlQ7wHuLove94" name="Page-1">7V1bk5s4Fv41rpp5iEuAEPDoa5KtJNs1ndpsnlwY1Dbb2HgBd7vn14/EzSDJbdwWYHc7k5oYAULofOd+JHraaLX7HNqb5ffAxX5PBe6up417qqoCpJJ/aMtL2qIoVtayCD03a9s33Ht/46wRZK1bz8VR5cI4CPzY21QbnWC9xk5cabPDMHiuXvYQ+NWnbuwF5hruHdvnW395brzMXwNZ+xNfsLdY5o9GOkzPrOz86uxVoqXtBs+lJm3S00ZhEMTpr9VuhH06ffnEpPdND5wtRhbidVznhqdf32aTrxp0d3e/R19ms93QfPyU9fJk+9vsjf/CCy9YZ0OOX/KJ2ATeOk4mUx+Sv+RRI9DTyZkRPeqrOtPAHhvVBoU/on1UG9hjo9qgsN0rzPMVdoClBu6o0j1gng9KAyR/tWGwjX1vjUcF7ABpXIS26xFqjAI/CEnbOliT2Rsu45VPjhTy83npxfh+Yzt0Vp8Jz5C2h2AdZ8BX1Pw4m3jaK8HNhv5e7RaUyfr2cwT7izDYbpJHfiXQF56dhSklSQ9xGDzifFQ9VVOgMRkO6LM832dG+4TD2CMMMPC9Be04Duhz7OzIxw8x7ZG8grdefEuOxhrIhi16hGtHS+xm75+hjTwC7w7CWCmYg8gVHKxwHL6QS3KhkvNTJlI0Re1DM2163rOomV+2LHGnDo2+ngkgO5MMi+IJe94hPzL2OYGVkM7x0sgPtu5DEBJZkFBi0FORT+dvHpJfizidnsdPeP3khcF6RaeAZTxCyrWbTCA4AKASvjhKW8jQbMSSQUxjARjS/nLJp2X93NlxjMN10pkKFB4eEsisGQyZc3qWaIyQgMYwv1A+gdU3E3jjbTCVF9dJXRlcq1XJqUOenIqIZTXQFDl5av7HC+OtTW+8C70nOybkAgmFb9rw+rXh08YRsZAKTVOBclVhwY1ADvPorMrThSpPByL+0RtUeRrHQnfbue85lFm28zXmtdmNca6OcSLsbEMvfpntL75PuCjvuBZLkfaJNdUmSJ6JWTxHOr9Bht8sIb9BkfmhaE3pK8gzW6Glbtz2YblN6MtRbkNTdWo26dA1xG26WLuJuU1p0qHjGG5MbHk6qiB8xGFEfv1BxgT6ffK/H3/ypv8pLXS2K8yL/r8N8hOfogRqA2qPq5vd/mTey+DXPTk3tcNFIg/SLslLp73mT5Lqe7iGNQeAx93DA0aOIwcahYOXQ8M0RNBQDQE0cndcOix4N5+bWaGgYbWXpk7pUzkBRM5NDXMCYOnc2AtJR4mLSQRUSF+cnfcx0EeKIaLUQ/KH5dxcLHyz59i/CyIv634exHGwOio3HDIqHFYxckwo2tEmnY4Hb0fHIZaSIY6CbejgVEYOyaFIWj5kWJeDM1ALZzmkWoGZ0TDMMj1XD0XZxRcHoMTAweHkCad2jnIIVARBsU3mKpxJCmggswoZBISQ0ZS+yoOmsRCVeQPNJYPGUC8RNNYNNBcNGnSJoNENDiTYXeB8DjMyVk1MargEi2BNKBlQCiWT+D8cxy8ZKextHFTpVAlsi6PTUWyH8YDmenulyDZpm3r0lUpeUznSjdcucw9p4e4o4XU0QgjsXR76sqcSNMS+HXtP1ftElMluvaOY2wPBhEoVCMoeCHkvqemU3VhOBrN9Gcf7InO4wDHXVwKX4q3ejqA8xX+TO5cpd0xwiXJH4dM2N9RcEmou0i5WrVa0VUUVOb4dRZ7DaKN9LKWqjchb/pc+uK/nh7+zcSQH413l6OWolsI7Ly51SI5+Z0+nv/fd0YO8t7qajcxjomZeY9NswlMd8sqFqBFVabDpZrN99cab1Vn4MHKIsNj6ODwvYhht7HUXUcSiuTqAdoOLVQkoP9IIVWEQWlHajDQqTbvzt1Bj56HGA0BrNdSo3izxi7apkGX1YRU1EPat7q0qvnr4x+AnafhM2OXZfmkDRPpAA0P9/YFobcezRTaNUkBkMSYR1MRpVlFJZWMA0o7YDW+ywVnjWqJZmwvGo2ZtqrrPsGvPm1fYkrtzCVOtdzrTfDgitaInxPuLaSUca76Pcu1ALrrH4ZPn8FXB12zUpUKTFaWFjL0Uow47BApDnxl8MSoJAhcqjA+aH5fNPJE70VjIvi2pUDceX65ktwCwLKFMqZECkCl29KuQ8Lp+o6U8WnarQviCohNUCF2uF+XDuOmQlnVI2LIOQQeiBW2qEZWvi/xK3zgpQL65fue7fl42myX/r0mMGbreRxWUHVhf0q5raLai4IrES3GQpF32WZhXEy9l1XZ+wUAdjWbUtU6sTlUaX7w4WNl/02Xl4N5e4O/2oyj78SOI8TwIHqXrMjgYqtbgNF2mKGisD8/XZVcS5I4IWVYJWeSkgVU2zC0MWLaquDSFA1ajIoXJ5SpHRcpb6pg4dFLbW7bYyUOHjWdzz6NvOyVpJfpqxikETo7ucOiR16V8dqO6FKq3bCiAPlKsCtn7QDlmLtxI3wjp26nqKZFebUiii8jZaHT9OgicexftEdi4EbhVDuYrD/aOgsZ7CH/8e4PpubEd26Xg15/SPQYEBlqi3U/wGFTDIE7Dx/EY9og9c32DXnEVivUOnfkJ/Er7rHKP/LOcB3boCpD55efPOx6HbToYx4tF32JvsKGnbMH466HbHKNhSq7Dodxs3fnP5NXBviFbiF5qGWa3jPV9218ZGj5BEdeZgP4nWaxqNcVqtwEYjc8p/Gu7eSHy4st2fgC89+2gVw4KWWC9DkMGs4dQWEAMcUC8QISlQbbOEMbv/fMLE2iBId10k0b3mk8AFJe/PQGw8lyXjpDDRnGCwVA7FYS+l+DmIOxlqF29qnZVeGBTKV7zQrNvqE3his8uXQmULjKXtI2kpYmYwmVVsIOfLqw/3bfKdxDbCenKWNejaUVDHWNtv35HM9VeaQVPHm06uIrnlTVBoigUvZdtk6jOul4IBGEfEV8BKjqwoJYHC/OshHXGClpQlZ+6oK+GlxhBXjReKvqRbp2C/jrr2pCGynzxKk98ROSrujSsG5rR7xztejvBQGlReQBGI7mIgrAmomC3UVs+fpIH9fhKNhffHdob+eys/9ggJ3snxfDIHzi0RAR9lzE8h8x/sTd1s2VGKrPC0tBg30KcCdluRXQH5qOEKtq6IkW6TtHYxf2lQrFTdUqdvpq2n3hfs8g+UMk03Hq++67E0pUU1lKxNE8mv12ZdAnltfCV2rlkM/pfduwspaNyqkGYOJonoHI4UjT94yS8HDr7z3T2Z6oc/LHl3Z2nvmAH+6TUtrEZNBcWdju2t1nX9oaNqF7VqkLFrK96palLfjuKz16cJJQYzFS4XiiZSmDg2PugxH+j/GBZ/hnPPcLeUX/hxUsyeqE8HE/Hk/FEJEEk8L0CL9AW5tdi32zhj2ML67wtXFpmxvrtmbhUkb2iHLaeR5ueaK8iQlf8YDtxJNdeuYB0YJ6JayWLg3d0RZDtzyL3USAg5X11S4FmnqvJoWhBTi5Bra9bvGjSGxNN17N7m2JWFhJ9An2gHa0LL0Laav292k7M8ki0hG5xbbmSF3UQ134juk0mi1Oj8v0N2chyJe5H27eQKH558DYNtrcu0jYqZ1i4STUlayt4K/ot3hbshPe/71iY1UXP0jltNojGGa+aeMdX4ccdG7IY0KUF9tvPFaKagizlT/nuEfupg3PcI71z9wi1vLaQGo6AWZMOrEIdn7LMTNIW+qUN8/fWw37PfLHtIND+ACAkF+qobsUs6nSfF9T6QsWTVgfcYFL9znNnMOELq5PtgLjSCZqNmBafnn5P+aBryVIyH/9uNVUJQcepIqODHcgOu621xU55jR+qJJCOpJpkL7HOv1l4fJ1Hl9IoHyW3OVn2XXTw3d7wsqnY1xKMvcgJCPDl709mQX08PVFAHdgx6l0LqNkq2W1frNHb3Lusc4mF+C8y0DXGYP8Z5cna3RQexG33srflKrI5jE4HXXJiaDuPi0RtiNhTAi4NoxoTQwaPS0lbnZHDMKCf2Ni7onSivgcuplf8Aw==</diagram></mxfile>