feat: parseReadme / better feature sections

[aheissenberger]

This will allow to build better structured documentation and README.md files - #364 .

The new features are:

• structure the READMEs sorted by the categories defined in packages/features/src/features.ts
• simple adding unstructured markdown to any existing feature section
• choose a markdown parser: mdast which allow flexible manipulation of existing markdown documents

Similar to the magicast loader for source files, the existing markdown files are converted to ast (mdast).
This allows the full range of manipulating markdown files or simple creation of markdown bei creating mdast nodes.

To make the transition to the new loader simple, the new loader is using a nearly identical interface - only the wrapper information needs to be added:

import { loadMarkdown, type TransformerProps } from "@batijs/core";

export default async function getReadme(props: TransformerProps) {
  const content = await loadMarkdown(props);
  const about = `
## React
...`;

  content.addMarkdown(about, {
    wrapper: {
      category: "Frontend Framework",
      flag: "react",
    },
  });

  return content.finalize();
}

To compare the functionality of this draft with the existing loader the current commit contains copies of the existing READMEs: boilerplates/shared/files/$TEST.md.ts, boilerplates/sentry/files/$TEST.md.ts, boilerplates/react/files/$TEST.md.ts, boilerplates/aws/files/$TEST.md.ts

use this to test the new Markdown Manipulator to create a test project:
pnpm cli --sentry --hono --aws --react

Concept

All added or predefined section (e.g. intro, features) are wrapped in CommentMarks (e.g. <!--bati:start section="features"--> ...content...<!--bati:end section="features"-->. This allows to place and find any kind of unstructured markdown content.

Based on the mdast tree and the zone callback which based on the filter set by calling content.addMarkdown the system will process all added blocks with content.finalize().

Features supported by the current API:

• find any CommentMark block by providing a filter object which is compared with the attributes on the comment or by providing a custom filter function. Default filter: {filter: {section:"features"}}
• define any kind of attributes which are set on the commentMark wrappers
• define the position of the content block inside the block: before, replace, after (Default) - for features section this is overwritten by the sorting from the features object.
• use a call back function to manipulate the tree directly content.addMarkdown( (start, between, end, _info)=>{},wrapper: { category: "Frontend Framework", flag: "react"}

Example:

content.addMarkdown('# My Markdown Content', {
    filter: {
       category:'Hosting',
       flag:'aws'
     },
    position: "before",
    wrapper: {
      category: "Frontend Framework",
      flag: "react",
    },
  });
`}`

[aheissenberger]

@magne4000 is it possible to access the flag of the current feature the processed $README.md.ts is related too. I would like to have a default wrapper which is based on the flag and category of the feature package. props.meta provide the full list of flags, a posibility to test but no information about the current feature.

[magne4000]

@magne4000 is it possible to access the flag of the current feature the processed $README.md.ts is related too. I would like to have a default wrapper which is based on the flag and category of the feature package. props.meta provide the full list of flags, a posibility to test but no information about the current feature.

The "current feature" is not necessarily defined by a flag, or could be behind multiple flag, so not possible. And currently we have no way to know if a file will be replaced by another later in the process.

[magne4000]

I really like it!
Once this PR is merged, I think that generating a Table of Content could be a good idea.

[aheissenberger]

@magne4000 is it possible to access the flag of the current feature the processed $README.md.ts is related too. I would like to have a default wrapper which is based on the flag and category of the feature package. props.meta provide the full list of flags, a posibility to test but no information about the current feature.

The "current feature" is not necessarily defined by a flag, or could be behind multiple flag, so not possible. And currently we have no way to know if a file will be replaced by another later in the process.

It is less about the "current feature" more about the context a file is processed.
If the $README.md.ts is in the folder boilerplates/react/files, the values for flag="react" and based on a lookup in features.ts the category="Frontend Framework". I was hoping this information is already somewhere available as I did not wanted to look at the file path or parse the package.json. This would allow to make the main use case of adding content even more simple.

[aheissenberger]

I really like it! Once this PR is merged, I think that generating a Table of Content could be a good idea.

I have thought about this, it's not really hard but my Markdown Viewer in VS Code provides this out of the box.

[aheissenberger]

Open Question:

• should I replace/remove the old readme parser and upgrade all READMEs
• is the feature to create markdown elements of the old library needed? - except for creating the link in boilerplates/shared/files/$README.md.ts I coudn't finde any usage.

[magne4000]

It is less about the "current feature" more about the context a file is processed. If the $README.md.ts is in the folder boilerplates/react/files, the values for flag="react" and based on a lookup in features.ts the category="Frontend Framework". I was hoping this information is already somewhere available as I did not wanted to look at the file path or parse the package.json. This would allow to make the main use case of adding content even more simple.

I have an alternative proposal. Why not explicitly give the flag as a parameter to the loader or to the finalize function? in boilerplates/react/files, we indeed know we are in react, so we can give "react" as an argument (I'm not a fan of implicit behaviors in this part of the code anyway).

[magne4000]

should I replace/remove the old readme parser and upgrade all READMEs

Replace the old one yes 👍

is the feature to create markdown elements of the old library needed? - except for creating the link in boilerplates/shared/files/$README.md.ts I coudn't finde any usage.

You can remove everything I think. Creating markdown tags manually is probably better anyway for IDEs that support embedded syntax highlighting.

[aheissenberger]

It is less about the "current feature" more about the context a file is processed. If the $README.md.ts is in the folder boilerplates/react/files, the values for flag="react" and based on a lookup in features.ts the category="Frontend Framework". I was hoping this information is already somewhere available as I did not wanted to look at the file path or parse the package.json. This would allow to make the main use case of adding content even more simple.

I have an alternative proposal. Why not explicitly give the flag as a parameter to the loader or to the finalize function? in boilerplates/react/files, we indeed know we are in react, so we can give "react" as an argument (I'm not a fan of implicit behaviors in this part of the code anyway).

Thats what I would do if the value of flag is part of props.meta.flag:

I would access the flag during loadMarkdown or it could be provide with the content.finalize()call.

export default async function getReadme(props: TransformerProps) {
  const content = await loadMarkdown(props);
  const flags = Array.from(props.meta.BATI)
    .filter((f) => (f as string) !== "force")
    .map((f) => `--${f}`)
    .join(" ");
  const v = getVersion();

  //language=Markdown
  const intro = `#  Markdown`;

  content.addMarkdown(intro);

  return content.finalize(props);
}

[aheissenberger]

should I replace/remove the old readme parser and upgrade all READMEs

Replace the old one yes 👍

is the feature to create markdown elements of the old library needed? - except for creating the link in boilerplates/shared/files/$README.md.ts I coudn't finde any usage.

You can remove everything I think. Creating markdown tags manually is probably better anyway for IDEs that support embedded syntax highlighting.

Why not make it even simpler for the main use case, which is adding markdown without any conditions, as simple as possible by having a specific naming e.g. boilerplate/react/README.inc.md and then include the files in the the section "Frontend framework" with flag="react". This would allow full markdown syntax support.

[magne4000]

Thats what I would do if the value of flag is part of props.meta.flag:

I would access the flag during loadMarkdown or it could be provide with the content.finalize()call.

export default async function getReadme(props: TransformerProps) {
  const content = await loadMarkdown(props);
  const flags = Array.from(props.meta.BATI)
    .filter((f) => (f as string) !== "force")
    .map((f) => `--${f}`)
    .join(" ");
  const v = getVersion();

  //language=Markdown
  const intro = `#  Markdown`;

  content.addMarkdown(intro);

  return content.finalize(props);
}

Sorry I'm still not sure to understand. In this example, where is this file located? Then, let's assume that we have access to this flag under props.meta.flag, how would you use it?

[aheissenberger]

Thats what I would do if the value of flag is part of props.meta.flag:
I would access the flag during loadMarkdown or it could be provide with the content.finalize()call.

export default async function getReadme(props: TransformerProps) {
  const content = await loadMarkdown(props);
  const flags = Array.from(props.meta.BATI)
    .filter((f) => (f as string) !== "force")
    .map((f) => `--${f}`)
    .join(" ");
  const v = getVersion();

  //language=Markdown
  const intro = `#  Markdown`;

  content.addMarkdown(intro);

  return content.finalize(props);
}

Sorry I'm still not sure to understand. In this example, where is this file located? Then, let's assume that we have access to this flag under props.meta.flag, how would you use it?

The code above is a boilerplate/react/files/$README.md.ts file, which is a file as part of the feature flag react. If theoretical props.meta.flag would provide react, I would add this as a second parameter to the function parseMarkdown which creates the instance of the markdown class.
packages/core/src/loaders.ts:

export async function loadMarkdown({ readfile, meta }: TransformerProps) {
  const content = await readfile?.();

  return parseMarkdown(content ?? "", meta.flag);
}

[magne4000]

The code above is a boilerplate/react/files/$README.md.ts file, which is a file as part of the feature flag react. If theoretical props.meta.flag would provide react, I would add this as a second parameter to the function parseMarkdown which creates the instance of the markdown class. packages/core/src/loaders.ts:

export async function loadMarkdown({ readfile, meta }: TransformerProps) {
  const content = await readfile?.();

  return parseMarkdown(content ?? "", meta.flag);
}

Okay, would there be a limitation to write it like this then?

export async function loadMarkdown({ readfile, meta }: TransformerProps) {
  const content = await readfile?.();

  return parseMarkdown(content ?? "", "react");
}

[aheissenberger]

To simplify the discussion 😄 :

1. adapted meta to include a localContext in packages/build/src/index.ts
2. configure the markdown class with loadMarkdown in packages/core/src/loaders.ts
3. add a second function addMarkdownFeature to make it clear that this will add content to the feature section
4. add a table of content - place can be defined in the first readme boilerplates/shared/files/$TEST.md.ts

to test run pnpm cli --react --sentry --aws

This is still a draft and needs some cleanup but should be feature complete.

The only known bugs are headers with special characters which are not correctly converted to internal links e.g.

### `/pages/+config.ts\`

[magne4000]

My remarks about not being able to extract the current flag from the boilerplate name still stands. Its wrong to assume the boilerplate folder name is the flag, it's usually a coincidence for readability, but you cannot assume that this is or will always be the case.

This means that localContext as no conceptual meaning with the current achitecture of Bati.
It should be removed, and with a slight modification, addMarkdownFeature can take an explicit flag as I suggested:

addMarkdownFeature(markdown: string | ZoneHandler, flag: Flags) {
  const category = features.find((f) => f.flag === flag)!.category as CategoryLabels;
  this.contents.push({ markdown, options: { wrapper: { category, flag } } });
}

// usage becomes

content.addMarkdownFeature(todo, "aws");

[aheissenberger]

My remarks about not being able to extract the current flag from the boilerplate name still stands. Its wrong to assume the boilerplate folder name is the flag, it's usually a coincidence for readability, but you cannot assume that this is or will always be the case.

This means that localContext as no conceptual meaning with the current achitecture of Bati. It should be removed, and with a slight modification, addMarkdownFeature can take an explicit flag as I suggested:

addMarkdownFeature(markdown: string | ZoneHandler, flag: Flags) {
  const category = features.find((f) => f.flag === flag)!.category as CategoryLabels;
  this.contents.push({ markdown, options: { wrapper: { category, flag } } });
}

// usage becomes

content.addMarkdownFeature(todo, "aws");

I agree, that reengineering from the path is not the best way, but was a fast solution to show what I need. The right way would be to provide this information with the sources list, but this will be a bigger change in the build source code.

To finish this, I will implement it as suggest by you and wait for the next use case to convince you that the name of the boilerplate value needed in transformed templates ;-)