Proposal: use remixaustin's blog to contribute to the general Remix community

[colbywhite]

I propose that we add a blog to the website, but do it in a way that helps the Remix community at large. There doesn't currently exist (in my opinion) a solid way to quickly add a blog to your remix site. There are several different strategies you can take and thus you need to spend some time thinking through how to accomplish it. A utility library that makes that process easier would help the Remix community in general.

There's a world where we add a blog to the remix-austin site and create a utility library for others to do the same. I propose we aim for that.

This rest of this post will be a recap of my current thought process around implementing a utility lib for this. I've been thinking about it for a while and this is me snapshotting the latest iteration of those thoughts and putting them into text to hopefully spark convo.

But I do suspect there's a valid counter argument of my proposal of making a general utility lib via a meetup group. Feel free to push back if you think that's not a great idea.

Some thoughts on implementation

First, I'll make some assumptions (feel free to challenge these):

• we're using mdx. this seems to be the go-to for most devs writing blog posts, and a least-common denominator for those that don't.
• mdx-bundler is the ideal way to do server-side rendering of mdx files
• a typical remix blog contains the following routes:
  • blog.$slug: to display an individual blog post
  • blog._index: to display a list of all the posts
  • rss[.]xml: to build a rss feed (although this could be a v2 thing)
  • sitemap[.]xml: to buld a sitemap to help search engines (although this could be a v2 thing)

To build each route, these steps need to happen:

1. read mdx source*
2. pipe contents into mdx-bundler
3. render output of mdx-bundler as needed by the blog

* source could be the filesystem, github, a cms, or something else

if we want a general set of utilities, we likely want to focus on the first two steps and give the user what they need to do the third step based on their own style/preferences. if so, that means a reframing of what "a utility library for blogs and remix" would be "a utility library that connects mdx-bundler to different sources"

(and thus it's probably useful to understand the use case for mdx-bundler)

when reframed as that, the question becomes what are the types of sources?

Sources

I think there are three key things an author is going to factor in when choosing where/how to save their content (and thus choosing the source).

1. the authoring experience. As an example, i lean toward mdx files on the filesystem b/c I can edit them quickly in virtually any IDE.
2. preview. As you edit a post, you likely want to preview how the post looks in your blog, before you publish.
3. deploy/publish. What is the process for publishing? git push? clicking a Publish button?

And thus i see three main types of sources.

Option	Authoring	Preview	Deploy/Publish
file system	your fav ide	yarn dev	git push
CMS	the CMS itself	some CMS' provide options for this; some don't	CMS's publish button
database	completely custom	completely custom	completely custom

Observations

• i'd add that a fourth option is to use your git host's api (github/gitlab). this becomes a hybrid of the file system and CMS option. you author as if it's a file on your filesytem, but you pull from the source as if it's a CMS (i.e. via an API).
• i'd also add that a file system struggles in production envs where there is no single file system (i.e. serverless or a distributed server). and thus i've seen the strategy of using the file system during dev, but the github/gitlab api in prod.
• i'd propose ignoring the database option (at least for now) since i think that's an option for folks doing something custom and thus you're likely not going to use a general util lib anyway.
• the CMS option has a near infinite amount of variations due to the amount of CMSs out there.

CMS options

here's a brain dump of CMSs that exist in no particular order

• strapi
• storyblok
• cosmic
• datocms
• sanity
• directus
• prismic
• netlify
• contentful
• agility
• butter
• collected
• obsidian
• wordpress

Potential next steps

based on the above implementation thoughts, here's some next steps that could get this project going

• in the long-term i'm imagining a @remix-cms org in npm with several packages. @remix-cms/strapi, @remix-cms/storyblok, @remix-cms/file etc etc
  • there are likely some cms' that don't really need a lot of help integrating
  • we could start with the filesystem option, then expand
    • obviously a filesystem isn't a CMS per se, but it kind of is
  • you could argue an org is overkill for this
• each pkg probably needs two public functions
  • function getPosts(): Promise<Awaited<ReturnType<typeof mdx-bundler.bundleMDX>>[]>
    • you could argue this one doesn't need to actually bundle things and instead j/ return the frontmatter
    • this likely needs paging logic
  • function getPost(slug: string): Promise<Awaited<ReturnType<typeof mdx-bundler.bundleMDX>>>
• from there, users use getPost/getPosts in their loaders and mdx-bundler's getMDXComponent in their views.
• an additional benefit would be to add typing/linting to the frontmatter so ppl can ensure that they didn't forget to add a post date
• an addon could be logic to help build Cache-Control headers, with config options to control it

Prior art

Other things i've seen in the community at large that influenced all of the above

• remix does support *.mdx routes. but it's barebones and not really meant for a full blog a point remix acknowledges in the docs
• Roger Stringer has a remix-blog stack. it's one commit around 6 months ago, so i'm not sure how actively maintained it is.
• Kent C. Dodds takes the strategy of using the github api as the source
• my own site is an example of me using the filesystem. this is what i came up with a while ago when converting my own site.

Thoughts?

I think that's enough of a vision of what this utility lib would look like in order to discuss whether or not we want to pursue that via this meetup group. This sound like an interesting project?

[brookslybrand]

I think this is awesome Colby! Seems like a super interesting idea and I can definitely see a helpful library or at the very least pattern+example repos+blog post coming out of this.

I think building this within the Remix Austin community is also a great idea, but you're looking to publish an open source package, I definitely don't want it to be tied to the Remix Austin community in any weird way to where the availability of our members are holding you back. Not sure exactly how to express the relationship worries I want to avoid, so mostly thinking out loud.

That being said though, I think lots in the community would be pretty interested in helping. I could even see us using a meetup to hack around ideas or something like that.

Additionally, you really seem to have a strong vision and interest here, so I would definitely be looking for you to take the lead on this, but I'm happy to help you spec out issues, engage with the larger community, give feedback on ideas, and pickup tasks as I have time and bandwidth.

In the meantime, would definitely love @grantnorwood, @sethdavis512, @dmorrison, @tcschiller, and @KarenDouglas's thoughts on the matter (given all of them have contributed to this project so far)

[sethdavis512]

I'll echo some of Brook's thoughts...

the availability of our members are holding you back

Agreed - wouldn't want RA members to be a bottleneck.

I think lots in the community would be pretty interested in helping

Yeup the Remix community is growing more and more. I bet there's a subgroup of Remixers that are thinking this would be helpful.

I could even see us using a meetup to hack around ideas

💯 - would be a great place for new devs to dabble.

I like your thoughts around this but I too would probably not be able to contribute too much.

[sethdavis512]

A good place to bounce around some ideas: https://twitter.com/i/communities/1635733130022912000

[colbywhite]

I definitely don't want it to be tied to the Remix Austin community in any weird way to where the availability of our members are holding you back.

fair point. maybe the best way to structure it is for me to take the role of official "maintainer", but everyone can contribute. and the meetup becomes a seeding ground for ideas and work.

[matt-hernandez]

So I did my best to take this discussion and run with it based upon all the links @colbywhite left behind of blog implementations in my PR #91. I started by copying beat for beat what he did for his personal website, and what Robert Stringer has put out there as the Remix blog stack. I eventually got my head around what it was doing and how to optimize it.

Here's the thing, for really small blogs, I think the level of support for MDX in Remix is actually more than good enough. It offers quite a bit for the person writing the occasional blog or two. And it's as easy to set up as just putting those MDX files in your routes. The only thing that could make me worried is if Remix ever drops support for MDX in the future. If they already advise against compiling MDX at build time today, then maybe that is a possibility.

With that said, if someone is running a more active blog and is planning to use their file system as a CMS, the only way to optimize that experience is to put an accompanying local server next to the Remix dev server on startup. The local server can watch the blog contents and compile the frontmatter into a cached list for the blog page. It can also have a websocket connection that the UI of the Remix app can listen to by way of a conditional script. This would enable auto-refresh on blog change or blog dependency change. Having that second local server is necessary for both of these features because the Remix server is pretty closed off.

And if we want to support full caching of the blog contents without moving files into /public, then we have to get real crazy. We would have to support transforming image imports, or other non .ts or .tsx files, from MDX files into urls that point towards our dev server, or node_modules, or anywhere the user wants an image to live. The only way I can think of doing that sensibly is to make our own remark plugin, which MDX uses under the hood.

Finally, if a user wants to have their blog content rendered locally on dev, but hosted somewhere for prod, we would have to come up with a whole configuration set up for how the MDX files can be stored on an API, but how we will resolve the dependencies of those files. There are a few options that come to mind, but we'd likely have to come to an agreement on the "best" one.

There's definitely room to create something in this space that could be good. And making a recommended set up for a blog that runs on a local filebased CMS that uses mdx-bundler is certainly possible. Beyond that, I think we'd have to really put our heads down and get some discussions going, most likely with people who are more knowledgeable than me.

[matt-hernandez]

I will add one more detail. In the cleanup I did for the PR this morning, I was able to serve up the blog contents in dev mode without moving them to /public which was a pretty big improvement. Honestly, that was the last thing I was hesitant about. I'm planning to take what I've written and use it for my own personal website I plan to make later this year, because it's pretty rad.

[colbywhite]

first off, great work @matt-hernandez on iterating on things via that pr. it's helpful to see it in code to move this convo forward.

here's a couple of thoughts/comments/questions:

• "the level of support for MDX in Remix is actually more than good enough." it depends on what you consider "good enough". after having a better understanding of remix routing, i think you're right in that there's more there than i originally thought, mainly b/c i better understand how to leverage <Outlet /> better in layout routes. my writings layout route is a pretty bland approach and i could be doing more such that i could theoretically use the *.mdx routes out of the Remix box. But the limitation there is that you can't do anything with the frontmatter. for instance the small stuff i do with my title and date in my writings.$slug route isn't possible with Remix's out-of-the-box support of mdx. another example is that your implementation requires certain tags in the frontematter, which i don't think you can reliably do out of the box. am i wrong on those assumptions?
• can you explain the need for the local dev server? it's not clear to me why that's needed when changes to /public will cause a refresh right?
• one grand conclusion i'm seeing after seeing your pr is that the lack of a window into esbuild is really limiting here. what we really want is to plug into the build process so we can continue to use the filesystem as our source. but we simply can't plug into the build process. i'm pretty sure this is a purposeful decision by the remix team, but it's certainly limiting here. and one thought i had to solve some of this would be to j/ add frontmatter support to the existing remix mdx tooling. but that involves really getting into the weeds of remix itself. buuuut, maybe that's a better manifestation of this? any thoughts/appetite for that?
• i do want to call out to anybody stumbling across this thread that these issues aren't unexpected. a blog isn't the ideal use case for Remix. you're likely better suited with something like gatsby for a simple blog. but we're kind of purposefully trying to do this w/ remix for fun and learning.

i'd love to talk through some of this stuff further if you're interested. i do plan on being at the new reactjs austin meetup if you're interested in getting in front of a whiteboard and working through some things. and the next remix meetup obviously. or even discord. j/ let me know. it'd be easier to work through with someone to bounce ideas off of.

[matt-hernandez]

To address your points:

• You're right in that you can't enforce front matter requirements in Remix. But you can access the front matter in the post content and in any files that import the posts through an attributes object that is automatically created. See the docs here. Again, there's quite a lot here for the average blogger, so it kind of seems weird to me that the Remix docs would have that callout right at the top to use mdx-bundler. My best guess is that it's for scalability reasons, which they do mention explicitly as a limitation. Maybe Remix is hoping for big adoption one day by popular blogs, more people like Kent C. Dodds for example.
• There's a few reasons for the second server
  • I was actually not seeing auto-refresh functionality with moving files to /public. I even double checked to make sure I was remembering correctly. If you go to my branch and checkout commit hash 971f180 locally, you'll see the branch using the exact same strategy as your site, with a post-cache.json file that is auto-generated to /public. If you modify any of the mock blogs, you'll see that even though that file is regenerated, the page doesn't auto-refresh and the content is stale.
  • I'm not a fan of moving files into a particular location to trigger a refresh. It can be done, but in making this solution, I wanted to shoot for that feeling we get from running our current build tools in dev mode, where content is served from source files like magic. No copying or moving files around, no compiling to a file on the filesystem (at least that we can see). Just content magically generated from source. I think I got pretty close.
• YUP! I completely agree. This whole process would have been easier if we could just add some esbuild plugins, or make some esbuild plugins, for the Remix compiler. But it's very closed off and it looks to be that way by design. I think this solution does everything except get into the weeds of Remix. So I don't know. Maybe there's a way to persuade Remix to open things up a little more. Or maybe we have to get into the weeds.
• It's been fun making this over-engineered solution 😆 . I somehow ended up making my own little mini build tool at the end of it all, which I've never had to do before. But even with that said, I do think that with a small amount of polish, this could be a reusable thing, maybe even into a custom Remix stack. Maybe call it the Remix Composer Stack (composer being related to both music and writing).

I should be at that meetup too. Let's hang!

[colbywhite]

I was actually not seeing auto-refresh functionality with moving files to /public.

🤔 interesting. i could've sworn that was working for me but ... it's not 😅. ideally that would j/ work.

[matt-hernandez]

So now that we have a PR (#99) that can deploy the blog implementation to staging, we can confirm that the solution we have so far works. Our /public directory hosts the source files for each blog post, the .mdx files, the .tsx/.ts files, and any images, and then every blog post is made on demand by mdx-bundler when the user navigates to a post.

I have two thoughts so far.

• I'd like to make an alias, maybe an environment variable, that can hold the pathway to the /public folder in the app at runtime. The compiled code for the Remix app goes into /build today. Using fs and path we can navigate and .. our way to the right folder so long as we can trust the directory structure. But if experience has taught me anything, it's that as Remix grows and changes, they could change that. I'd really like to have a stable pathway that we define to our public assets, rather than relying on the particular directory structure that Remix has created.
• Our solution only works so long as the app and the static assets are hosted in the same place and file reading is possible. For stuff like serverless applications, the approach wouldn't work. Assuming developers host their /public assets folder somewhere, like a CDN, we could offer a "build AOT" solution. Instead of hosting the blog source files in /public, we build every post into its final JSON form and store each blog post as a separate JSON file in /public. Then a user can host their /public folder wherever they wish, and a serverless application can request the JSON file for the blog post using a simple fetch. It's already compiled, so all that application has to do is either return it or use it outright to render the page. The tradeoff here is that each JSON file for a blog post would be significantly bigger in file size than its individual source files. But I think most devs who really want to go serverless would consider that worth it.

[brookslybrand]

Calling bankruptcy on all discussions and issues today. Going to start revamping the direction of this project, so stay tuned 🚀