RFC: developer docs

core/package.json

@@ -168,7 +168,7 @@
 		"@types/unist": "^3.0.2",
 		"@types/uuid": "^9.0.2",
 		"@vitejs/plugin-react": "^4.2.1",
-		"autoprefixer": "^10.4.14",
+		"autoprefixer": "catalog:",
 		"csv-parse": "^5.5.2",
 		"dotenv": "^16.4.5",
 		"dotenv-cli": "^7.2.1",

docs-nextra/.gitignore

@@ -0,0 +1,43 @@
+# See https://help.github.com/articles/ignoring-files/ for more about ignoring files.
+
+# dependencies
+/node_modules
+/.pnp
+.pnp.*
+.yarn/*
+!.yarn/patches
+!.yarn/plugins
+!.yarn/releases
+!.yarn/versions
+
+# testing
+/coverage
+
+# next.js
+/.next/
+/out/
+
+# production
+/build
+
+# misc
+.DS_Store
+*.pem
+
+# debug
+npm-debug.log*
+yarn-debug.log*
+yarn-error.log*
+.pnpm-debug.log*
+
+# env files (can opt-in for committing if needed)
+.env*
+
+# vercel
+.vercel
+
+# typescript
+*.tsbuildinfo
+next-env.d.ts
+
+_pagefind/

docs-nextra/README.md

@@ -0,0 +1,36 @@
+This is a [Next.js](https://nextjs.org) project bootstrapped with [`create-next-app`](https://nextjs.org/docs/app/api-reference/cli/create-next-app).
+
+## Getting Started
+
+First, run the development server:
+
+```bash
+npm run dev
+# or
+yarn dev
+# or
+pnpm dev
+# or
+bun dev
+```
+
+Open [http://localhost:3000](http://localhost:3000) with your browser to see the result.
+
+You can start editing the page by modifying `app/page.tsx`. The page auto-updates as you edit the file.
+
+This project uses [`next/font`](https://nextjs.org/docs/app/building-your-application/optimizing/fonts) to automatically optimize and load [Geist](https://vercel.com/font), a new font family for Vercel.
+
+## Learn More
+
+To learn more about Next.js, take a look at the following resources:
+
+- [Next.js Documentation](https://nextjs.org/docs) - learn about Next.js features and API.
+- [Learn Next.js](https://nextjs.org/learn) - an interactive Next.js tutorial.
+
+You can check out [the Next.js GitHub repository](https://github.com/vercel/next.js) - your feedback and contributions are welcome!
+
+## Deploy on Vercel
+
+The easiest way to deploy your Next.js app is to use the [Vercel Platform](https://vercel.com/new?utm_medium=default-template&filter=next.js&utm_source=create-next-app&utm_campaign=create-next-app-readme) from the creators of Next.js.
+
+Check out our [Next.js deployment documentation](https://nextjs.org/docs/app/building-your-application/deploying) for more details.

docs-nextra/content/_meta.ts

@@ -0,0 +1,7 @@
+import type { MetaRecord} from 'nextra'
+
+const meta: MetaRecord = {
+	infrastructure: "🏗️ Infrastructure",
+};
+
+export default meta;

docs-nextra/content/index.mdx

@@ -0,0 +1,9 @@
+---
+title: "Introduction"
+---
+
+# Introduction
+
+This is the developer documentation for PubPub Platform.
+
+Internal use.

docs-nextra/content/infrastructure/ecs-cluster.mdx

@@ -0,0 +1,88 @@
+---
+title: Terraform
+---
+
+# ECS Cluster Environment
+
+## Dependencies
+
+### State file bucket/config
+
+You must have some way of storing terraform state files.
+We use and recommend the s3 backend, but you can change
+that configuration. See `./environments` for examples of
+configuring backend.
+
+We store our terraform state in an S3 bucket created by
+the `./environments/global_aws` directory, which has an
+interactive setup, see that readme for more info.
+
+## Change management and workflows
+
+This code is here to make infrastructure declarative rather than imperative.
+It secondarily includes modularization to make it hard for configuration to drift
+between preproduction / production or open source deployments.
+These are two separate concerns.
+
+Declarative code changes are still managed imperatively with `terraform apply`,
+which can be made partially or fully automatic.
+
+In general, production changes are applied manually after we are satisfied with
+preproduction, which may or may not be automatic. Developers should expect a flow like:
+
+1. make a change to a shared module code
+2. make matching change to configuration in ALL environment directories, so they can be reviewed together
+3. apply this new SHA to staging and do validation as desired
+4. apply this same SHA to production.
+
+Now there is no drift between code, staging, and production - we are converged.
+
+## Rollbacks
+
+Generally, rollbacks are done in emergencies and are done first in prod. (If done first in staging, this is
+really no different a process than a roll-forward). Rollbacks are the only situation in which we should expect
+to deploy production from an off-main branch. Changes may be infrastructure or code. In the infrastructure workflow:
+
+1. make changes to terraform code that seems to fix the issue and apply it to production
+2. if it resolves the issue, figure out how it needs to be applied to pre-prod for consistency and open a PR
+3. when this PR is merged, it deploys to pre-prod and we are converged.
+
+In general, code rollbacks can be done without a re-build, by deploying an old SHA, but it is preferable
+if there is time, to do a revert & roll-forward flow,
+because some operations (primarily database migrations) operate on assumptions of monotonic time. Additionally
+this flow makes it easier for rollbacks to include reverts of specific changes in the middle of the commit history
+without reverting everything more recent.
+
+## Adding/updating variables and configuration
+
+Variables are the things that distinguish one environment from another. These include container variables and
+certain extra values such as infrastructure scaling / footprint parameters. There is a tradeoff between ease
+of configuration change and strength of guarantees given by similarity between staging and prod. First decide if
+your change should be applied identically to each environment, or warrants an increase in drift.
+
+To add a variable, modify some terraform resource that depends on it and then thread your way back up. The most
+common case will be to add an environment variable to a container so will use that as example here:
+
+1. modify `modules/deployment/main.tf` to add a variable to the appropriate invocation of `container-generic`.
+1. modify `modules/deployment/variables.tf` to add the variable declaration. (This step is not needed if your new env var can be computed based on changes to the upstream infrastructure, such as a database URL.)
+1. modify each invocation in `environments/*/main.tf` to add this new variable.
+
+Proceed as above. Note that changes to task definitions (which include container configs) are not actually applied until you then trigger a new `deploy` using `act`/`mask` or the Github console.
+
+## Adding secrets
+
+Secrets are a special variety of environment variable, whose process is just like the above but with an extra step after `terraform apply` and before `mask ecs deploy`:
+
+To provide secrets to ECS containers, you should put them in AWS Secrets Manager.
+To do this, replicate the setup in `modules/core-services/main.tf`: create a resource
+that declares the existence of the secret. Since the purpose of this model is to
+avoid having copies of the secrets exist anywhere persistently except the single
+locked-down place, naturally the secret value itself (the "version") can't be passed through terraform.
+
+So you must one-time only, or when changing the secret,
+
+1. go to the AWS Secrets Manager console
+2. and choose your new secret
+3. select "Retrieve secret value" (unintuitive, because there is no value yet)
+4. Console says "Value does not yet exist" and button you just clicked becomes "Set secret value".
+5. Probably, paste your secret in the "plain text" box (you can also do key-value pairs, but then must use the key in the address when retrieving.)

docs-nextra/content/infrastructure/environments/cloudflare.mdx

@@ -0,0 +1,24 @@
+---
+title: "Global Cloudflare configuration"
+---
+
+# Global Cloudflare configuration
+
+This module should generally be created by an admin,
+and assumees the following permissions which are sensitive:
+
+**Cloudflare read-write token** set at `CLOUDFLARE_API_TOKEN`. In general, this
+secret can be used for very nefarious things and should be extra sensitively protected.
+
+**AWS read-write permissions**: in `~/.aws/credentials`. see `../maskfile.md` for more info.
+
+## Relationship to AWS environments
+
+AWS environments assume existence of the Route53 zone and DNS NS records that refer authority
+to that zone. If you are not using Cloudflare this module is not needed for those environments,
+but in general to create a new env it is expected to augment this module with NS records referring
+to this route53 configuration for domains subordinate to that new AWS env.
+
+Therefore updates to this module, which should happen very infrequently, should be applied before
+you attempt to create the new AWS-ECS environment, otherwise that will fail due to the AWS Certificate
+Manager being unsuccessful in validating your ownership of the DNS.

docs-nextra/content/infrastructure/environments/global-aws.mdx

@@ -0,0 +1,21 @@
+---
+title: "Global AWS configurations"
+---
+
+# Global AWS configurations
+
+This module should generally be created by an admin,
+and not applied or updated by a machine user.
+
+## Creation of the terraform state bucket
+
+1. Uncomment the code creating this bucket; comment the backend block
+1. terraform init
+1. Set the bucket name
+1. terraform apply
+1. `terraform state rm aws_s3_bucket.terraform_state`
+1. comment the bucket definition; uncomment the backend block
+1. terraform init ("yes" to copying the state file)
+1. destroy local copies of the state file
+
+This bucket name can now be in your s3.tfbackend files everywhere.

docs-nextra/content/infrastructure/index.mdx

@@ -0,0 +1,10 @@
+---
+title: "Infrastructure"
+description: "Information about the infrastructure that runs PubPub."
+sidebarTitle: "🏗️ Infrastructure"
+asIndexPage: true
+---
+
+# Infrastructure
+
+Information about the infrastructure that runs PubPub