dev: check for env vars at startup

[tefkah]

This PR adds on-startup Zod schema validation of environment variables using @t3-oss/env-next for env vars in core, based on prior work by @kalilsn

It also sets up a common config folder for storing configs. Currently only the tsconfig is stored there, but in follow-up PRs we could store shared eslint, prettier, and tailwind configs there as well. See Notes below for more explanation.

Issue(s) Resolved

#196

Test Plan

1. Pass CD

Locally

1. Run pnpm build
2. Check that build succeeds (regardless of whether the env vars are set correctly)
3. do pnpm --filter core start (make sure you have all the env vars set correctly in .env.local)
4. Check that app starts and works as expected
5. In your shell, override e.g. the DATABASE_URL, e.g. do DATABASE_URL=bla (or set -x DATABASE_URL bla for fish). this just has to be any invalid URL for this to work
6. do pnpm --filter core start
7. Check that app fails to start with error message something like

❌ Invalid environment variables: { DATABASE_URL: [ 'Invalid url' ] }

Screenshots (if applicable)

Optional

Notes/Context/Gotchas

We are using @t3-oss/env-nextjs to validate and parse the environment variables on startup of the app.

I chose to use that package as it works quite well, and will possibly integrate some advanced features like using React.taint to prevent leaking secrets to the frontend in the future.

➡️ Removing .env.template in favor of .env.development

Per @kalilsn 's suggestion, I have removed (most) .env.template files in favor of .env.development files.

This is bc most of the values in .env.template actually did not need to be filled out at all, they were just default values that made the app work in development. To make the app actually work, you needed to do the annoying copying of .env.template to .env.local for every single app.

The .env.development solves this, sort of. You are now able to run the app by only copying the .env.template to .env.local and filling out values for core (there can't be default values for e.g. SUPABASE_PUBLIC_KEY).

For integrations & jobs, the default values will allow you to get a working app up and running.

If you want to test e.g. mailing, you still have to provide e.g. MAILGUN_USERNAME yourself in .env.local tho.

Ⓜ️ Why env.mjs?

@t3-oss/env-* are ESM only packages, so a .(c)js file cannot import it.

If you make it env.ts, you cannot import it into next.config.js (which you need in order to check env vars at startup, not just on page render) without using yet another module.

.mjs is the easiest solution. It still retains type-checking when imported into .ts files.

⏰ When does validation run

Validation runs on app startup, not build. The scope of #196 specifically referred to startup, and given our new docker setup by @ships we would ideally like to change the env vars as we promote the container from testing(?) to staging to prod, and not have to provide them during build.

Because we cannot assure that the app will startup correctly on build (as we don't check the env vars then), it would be good to include in CD a final check to see whether the app can startup correctly, or something similar. A simple ping to check if it returns a 2** would be good enough i think.

🗂️ Config folder

I chose to move the tsconfig folder to config/tsconfig in preparation for a follow up PR that adds ESLint. Otherwise we will end up with /tsconfig, eslint, prettier, etc.

You might ask: why not put these env.mjs thingies in a shared config/env folder?
Very astute, dear reader. Why, I did try that, but due to a combination of factors this did not work out how I wanted.

Roughly:

• If I import an env.mjs file from a package I do not get proper typechecking of the env object, which is a shame. Therefore we must create env.ts files and compile them with preconstruct in a similar way as packages/utils is.
• @t3-oss/env-* are ESM only packages, so naively compiling them and importing them with preconstruct (which is CJS focused) will not work.
• Preconstruct did introduce experimental support for ESM in v2.8.3, but this does not work properly. Specifically, the type: "module" packages need a different structure, but when running preconstruct build from root it complains about that very structure being different. I should probably file a bug report for it.

So because of that I opted to just put a env.mjs file in every place that needs it. I don't think it's that big of a deal.

⚙️ TS Config changes

"Why are you messing with the configs Thomas 😀"

I mostly changed "moduleResolution" to "bundler" for Next.js projects, because this allows us to import ESM only packages, like @t3-oss/env-*.

It is also the recommended setting for when you use, well, a bundler, like Next uses.

I also added incremental: true, as this speeds up typechecking times a little.

🪢 Linting

I decided not to add linting to this PR yet, as it would introduce too many changes all at once.
I also thought @kalilsn initial approach of just adding ESLint to core a bit hard to scale up.

I have prepared a follow up PR that adds a shared ESLint config in config/eslint, that can be easily inherited by any app/library.

We should first discuss what linting we want I think, before just adding it.

Supporting Docs

[kalilsn]

Generally looks great, love the zod validation and it works well! However, .env.development isn't being loaded by my local next server at all. Do you think using and committing the plain .env would make more sense here? That would be automatically loaded and simplify the dotenv cli usage, although we'd probably want to delete that file in the prod builds.

[tefkah]

Hmm @kalilsn do you still have .env.local files hanging around, or have env vars defined in your shell? They "override" .env.development vars (or better said, Next won't check .env.development for those if you it already finds it somewhere higher up according to this list https://nextjs.org/docs/pages/building-your-application/configuring/environment-variables#environment-variable-load-order)

Otherwise, might you be doing pnpm build? .env.development won't get checked then, because NODE_ENV=production (i've also found this somewhat annoying)

We could just set a .env in that case. I don't think it should cause a problem for the "actual"/CI builds atm as all the variables are currently used dynamically, so i think it should be fine to leave them in, although this is definitely something we should have a test for at some point.

[kalilsn]

I had been using pnpm start as @tefkah realized. Works perfectly with pnpm dev as planned!