Add docs to clone a new DANDI instance

.github/workflows/publishdocs.yaml

@@ -15,4 +15,4 @@ jobs:
       - name: Deploy docs
         uses: mhausenblas/mkdocs-deploy-gh-pages@master
         env:
-          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

.gitignore

@@ -6,3 +6,4 @@ node_modules
 local_data
 
 site/
+venv/

README.md

@@ -6,7 +6,7 @@ Documentation for interacting with the DANDI Archive.
 Follow the guidelines below when creating and revising text in the DANDI Docs:
 
 *	**dandi-** repositories — hyphenate the names of DANDI GitHub repositories (e.g. **dandi-archive**); "Dandisets" is an exception because it is a
-complete word
+     complete word
 *	**Dandiset**  — use single, unformatted, capitalized word (**not** dandiset or `Dandiset`)
 *	file names — use lower case (e.g. **development.md**)
 *	headings — use Title Capitalization (for 1st and 2nd levels) and follow with an intro sentence
@@ -28,6 +28,6 @@ If you would like to render it locally, you would need to create and configure a
 And your current session would already be using that virtual Python environment, which you could deactivate by executing `deactivate` command.
 If in the future you would need to activate it, just `source venv/bin/activate` again.
 
-After that you can either 
+After that you can either
 - do one time manual build using `mkdocs build` and find built website under `site/` folder.
-- run `mkdocs serve` which would not only build website and start a local webserver for you to visit rendered version at e.g., http://0.0.0.0:8000/, but also it would automatically re-build if you change any source markdown or configuration file.
+- run `mkdocs serve` which would not only build website and start a local webserver for you to visit rendered version at e.g., http://0.0.0.0:8000/, but also it would automatically re-build if you change any source markdown or configuration file.

docs/10_using_dandi.md

@@ -37,7 +37,7 @@ To view a specific public Dandiset and download one of its files:
 
 4. From the right side of the Dandiset landing page, click FILES to see a list of all folders and files for that 
    Dandiset. Click the download icon <img
-src="../img/download_file_icon.png"
+src="../img/download_file_icon.jpg"
 alt="download_file_icon"/> to download a 
    specific file. 
 **Note:** To download an entire Dandiset, you will need to follow the instructions in the 

docs/11_view.md

@@ -7,7 +7,7 @@ on `PUBLIC DANDISET` to access all Dandisets currently available
 in the archive, and you can sort them by name, identifier, size, or date of modification.
 
 <img
-src="../img/web_browse.png"
+src="../img/web_browse.jpg"
 alt="web_browse"
 style="width: 60%; height: auto; display: block; margin-left: auto;  margin-right: auto;"/>
 
@@ -18,7 +18,7 @@ return a subset of all Dandisets, while `"mouse house"` will likely not return a
 word is used as an `OR`.
 
 <img
-src="../img/web_search.png"
+src="../img/web_search.jpg"
 alt="web_search"
 style="width: 60%; height: auto; display: block; margin-left: auto;  margin-right: auto;"/>
 
@@ -27,7 +27,7 @@ When you click on one of the Dandisets, you can see that the searching phrase ca
 appear in the description, keywords, or in the assets summary.
 
 <img
-src="../img/web_search_dandiset.png"
+src="../img/web_search_dandiset.jpg"
 alt="web_search_dandiset"
 style="width: 60%; height: auto; display: block; margin-left: auto;  margin-right: auto;"/>
 
@@ -40,7 +40,7 @@ metadata provided by the owners such as contact information, description, licens
  a summary of the Dandiset including information about species, techniques, and standards.
 
 <img
-src="../img/web_dandiset_lp.png"
+src="../img/web_dandiset_lp.jpg"
 alt="web_dandiset_lp"
 style="width: 60%; height: auto; display: block; margin-left: auto;  margin-right: auto;"/>
 
@@ -54,7 +54,7 @@ found by clicking `Metadata` on the right-side panel. For Dandiset owners, this
 adding relevant metadata to populate the landing page.
 
 <img
-src="../img/web_dandiset_metadata.png"
+src="../img/web_dandiset_metadata.jpg"
 alt="web_dandiset_metadata"
 style="width: 60%; height: auto; display: block; margin-left: auto;  margin-right: auto;"/>
 
@@ -64,7 +64,7 @@ The right side panel allows you also to access a file browser to navigate the li
 in a Dandiset.
 
 <img
-src="../img/web_dandiset_files.png"
+src="../img/web_dandiset_files.jpg"
 alt="web_dandiset_files"
 style="width: 60%; height: auto; display: block; margin-left: auto;  margin-right: auto;"/>
 
@@ -80,7 +80,7 @@ services that can open the file. *Note:* that these services often have size lim
 If you log in as a registered user, you will also see `My Dandisets` tab:
 
 <img
-src="../img/my_dandiset.png"
+src="../img/my_dandiset.jpg"
 alt="my_dandiset"
 style="width: 7
 0%; height: auto; display: block; margin-left: auto;  margin-right: auto;"/>

docs/12_download.md

@@ -10,7 +10,7 @@ On the landing page of each Dandiset, you can find `Download` button on the righ
 button, you will see the specific command you can use with DANDI Python CLI (as well as the information on how to download the CLI).
 
 <img
-src="../img/web_dandiset_rsp_download.png"
+src="../img/web_dandiset_rsp_download.jpg"
 alt="web_dandiset_rsp_download"
 style="width: 30%; height: auto; display: block; margin-left: auto;  margin-right: auto;"/>
 
@@ -20,7 +20,7 @@ style="width: 30%; height: auto; display: block; margin-left: auto;  margin-righ
 The right-side panel of the Dandiset landing page allows you also to access the list of folders and files.
 
 <img
-src="../img/web_dandiset_files.png"
+src="../img/web_dandiset_files.jpg"
 alt="web_dandiset_files"
 style="width: 60%; height: auto; display: block; margin-left: auto;  margin-right: auto;"/>
 

docs/59_overview.md

@@ -0,0 +1,69 @@
+# Overview and Architecture
+
+The series of docs in this directory define how to create your own DANDI ecosystem (i.e. a clone of the entire DANDI ecosystem).
+It is suggested that you briefly read through each of the documents in this guide before starting.
+
+This section provides a high-level view of how DANDI’s core components fit together in a typical “full stack” deployment.
+
+## The Big Picture
+
+The DANDI platform is essentially composed of:
+
+1. **Storage**: S3 buckets (AWS) where data actually resides.
+2. **API**: A Django/Resonant-based backend application (hosted on Heroku) that handles the DANDI data model, user authentication, and orchestrates S3 interactions.
+3. **Frontend**: A Vue-based web application (hosted on Netlify) for users to browse, search, and manage data in the archive.
+4. **Workers**: Celery workers (also on Heroku) for asynchronous tasks such as file checksum calculations, analytics, and housekeeping.
+5. **Observability**: Log aggregation and alerting (Heroku logs, optional additional logs), plus Sentry for error-tracking and notifications. TODO(asmacdo) verify
+6. **Infrastructure-As-Code**: Terraform scripts that glue everything together—AWS S3 resources, Netlify or domain DNS, Heroku apps, etc.
+
+These services interconnect as follows:
+
+<img
+src="../img/client_requests.jpg"
+alt="client_requests"
+style="width: 90%; height: auto; display: block; margin-left: auto;  margin-right: auto;"/>
+
+* The user (or script) interacts with the **Web UI** or the **DANDI CLI**.
+* The **Web UI** calls into the **API** (over HTTPS).
+* The **API** queries or updates metadata in its Postgres DB (hosted on Heroku).
+* The **API** calls AWS S3 to read/write DANDI assets.
+* Certain heavy-lift or background tasks get queued into Celery tasks, handled by the **Workers**.
+* Domain names, certificates, and load-balancing records are handled by AWS Route 53 or Netlify’s DNS, depending on whether it’s the API subdomain or the apex domain for the UI.
+* Large chunks of data can be streamed from S3 directly to the Client via presigned URLs
+
+## Key Components
+
+<img
+src="../img/deployment.jpg"
+alt="dandi_deployment"
+style="width: 90%; height: auto; display: block; margin-left: auto;  margin-right: auto;"/>
+
+
+### 1. AWS S3 Storage
+
+* **Primary Storage**: S3 buckets are the primary storage of the data (Zarr, NWB, etc.).
+* **Configured via terraform**: Bucket creation, IAM policies, route to logs, etc., are specified in `terraform/*.tf`.
+Provides storage buckets, as well as domain management, for resources across the DANDI ecosystem 
+
+### 2. Heroku
+
+Provisions the servers, worker processes, and the database for the API.
+
+1. **API**: Django, extended by [Resonant](https://github.com/kitware-resonant/terraform-heroku-resonant), provides REST endpoints for metadata, asset management, versioning, and authentication.
+2. **Postgres**: Stores user metadata, dandiset metadata, and references to S3 objects.
+3. **Workers (Celery)**: Offload long-running tasks (checksums, analytics, zarr validation, etc.).
+
+### 3. Netlify (UI)
+
+* **Frontend server**: Serves a static build of the DANDI Archive frontend (Vue.js).
+* **Autodeployment**: On each push or merge to `main` (or whichever branch is configured), Netlify automatically builds and deploys.
+* **Configuration**:
+  - **`netlify.toml`**: Describes build commands, environment variables for staging vs. production.
+  - **`.env.production`**: Holds the environment variables for the Vue-based app at runtime (e.g. `VITE_API_URL`, `VITE_SENTRY_DSN`).
+
+### 4. Terraform Infrastructure
+
+The single source of truth for spinning up or tearing down resources such as S3 buckets, IAM users, Route 53 DNS, Heroku pipeline config, Netlify domain config, etc.
+
+* **Repo**: The [`dandi-infrastructure`](https://github.com/dandi/dandi-infrastructure) repo.
+* **Terraform Cloud**: Used to plan or apply changes after you push commits to the infrastructure repo.