[docs] Search results targeting FAQ page results in 404 with v6 (and below)

[vitality82]

Steps to reproduce

Guys sorry for dropping in like this, I have no time for all the details of confirming to issue protocol. The fact is this:

Search results only provide correct results with links if using the latest version. We are still on 6 so naturally choose v6 and all the search results are NOOP.

Current behavior

No response

Expected behavior

No response

Context

No response

Your environment

npx @mui/envinfo

  Don't forget to mention which browser you used.
  Output from `npx @mui/envinfo` goes here.

Search keywords: docs 404

[michelengelen]

The only broken links I could find are those targeting the FAQ page which is not backported to v6 apparently. Related PR: #11686

[michelengelen]

cc @MBilalShafi

[MBilalShafi]

Although porting it back to v6 would solve the issue for this particular page and v6, it won't be solved completely.
For example, the same is reproducible in v5.

Even worse, it's true for all new sections being added in the current major, for example, a fairly recently merged feature (v7) is being displayed in v6 documentation and beyond too but takes nowhere relevant.

For a proper fix, and avoiding similar issue to occur in future, we need to investigate more on how to optimize algolia to only index the pages available on currently accessed major version.

This should be a problem for all products under MUI X (at least, not sure if the same holds true for MUI Core too).

[alexfauquette]

@MBilalShafi That's a global issue. We have a unique algolia crawler that runs on https://mui.com and v5.mui.com, v6.mui.com and other versions call the same index.

We can handle version as different instance of the docs and so have on index per version as mention in algolia docs

I'm running the creation of a v5 docs for test purpose

[vitality82]

I hope my issue has not induced noise, for the record we would stay a happy customer even if this was not solved. The ultimate goal is to find time and space to upgrade to latest version (business goals always come first). This is just one of those issues in the moment when you search for docs for a required version.

You guys rock!

[oliviertassinari]

It's the same problem with Material UI v5, the search is broken. It's a docs-infra topic. v4 and v3 searches works because it uses an older index https://dashboard.algolia.com/apps/BH4D9OD16A/explorer/browse/material-ui.

I have tried to fix this a month ago but it was a rabbit hole. I fixed a few obvious problems mui/material-ui#43567, mui/material-ui#43566, and moved on.

We could decide to simply disable the search for none active versions of the docs. This could make sense, especially if we only had open source users to cater for, but with commercial offerings, this doesn't fly. I gave up because it was a high opportunity cost for me, and I wanted to delegate this. So for verisons that are still in LTS, it should work.

About solving this, here is one way we could go about it:

1. Material UI v5. We could either have two crawlers or use the same one with a version filter per major. We went with the latter option for v3 and v4. The advantage of having separate crawlers is that we don't have to maintain logic as we make changes to the docs structure. So one crawler per project major might be best.
2. Material UI v5. We could either have one index or two indexes. I'm note sure it matters much. Whatever is simpler. Two indexes might be more coherent as ranking needs to match with the crawler record structure. Fewer records are also likely faster to search.
3. Material UI v5. During the crawl, we need to make sure all the links are correct. I have fixed a few but there are likely more broken. This will be reported, so it will likely not be too hard.
4. MUI X v6. The challenge now is how do we have a search that works across projects in MUI org, without having to run 4 different requests. We would likely need a centralized index and crawler to get this outcome. Assuming that we deploy docs-infra at the same time between them, it would likely be simpler too. So to reconciled this with 1. it likely means that we want one crawler/index per docs-infra major version. So not per library major versions. We can actually host different libraries major versions under a given crawler/index docs-infra major. When possible, we only have the latest major of the projects under the active index/crawler.
5. MUI X v6. For inside a given docs-infra major version, I guess it's about reapplying [docs] Restore search on v5 material-ui#43566 so each record has the project it relates to and it's major version.

And done?

For MUI X here specially. We could do a few of those steps, with that direction in mind.

[alexfauquette]

Search issue

Duplicating the crawler, and using different indexes is the solution we started to go for.

I considered 1 crawler + index per version such that on index correspond to one sub-domain. For example, having v5.mui.com indexed in v5_material-ui.

Links issues

And done?

For the search yes, for maintaining a coherent docs for previous version not entirely.

The main issue is that we have version missmatch between core and X.

So the core docs were stable (v5) while the X one moved throw v5, v6 (refactoring the pickers), and v7 (introducing charts and tree view)

So when we archive the core docs (v5.mui.com) it matches with the latest version of the X docs (v7.mui.com/x/) which leads to 2 issues:

1. We don't know what users expect. If they are on v5.mui.com, do they want to see the v5, v6 or v7 of the X docs ?
2. We have 404 links because v5.mui.com has links to recent features (ie: https://v5.mui.com/x/react-charts/)
   

A solution could be to replace relative links from v5.mui.com to X docs by absolute links to v7.mui.com/x/ the latest compatible version. Such that we remove 404. And if users prefer an older version, it's up to them to change it. It will stay coherent as long as they stays in the X pages.

That's an argument in favor of having sync major between core and X. Or to have a website per project

[LukasTy]

A solution could be to replace relative links from v5.mui.com to X docs by absolute links to v7.mui.com/x/ the latest compatible version. Such that we remove 404. And if users prefer an older version, it's up to them to change it. It will stay coherent as long as they stays in the X pages.

IMHO, this idea sounds like a nice compromise, at least until we align versions. 🤔
I'm all for a superb docs experience, where everything is proper version aware, but we can also explore such a solution until we are at least version synced and then implement. 🤷

[oliviertassinari]

Duplicating the crawler, and using different indexes is the solution we started to go for.

@alexfauquette If we want to do the least amount of work for this step, we could use the same Algolia crawle/index, docs-indra didn't change between those Material UI and MUI X majors, but it works too to start to have multiple.

I considered 1 crawler + index per version such that on index correspond to one sub-domain. For example, having v5.mui.com indexed in v5_material-ui.

If the crawler/index is a projet specific, it will break the cross projects search experience, so I don't think it works. Unless maybe https://www.algolia.com/doc/guides/building-search-ui/ui-and-ux-patterns/multi-index-search/js/ works, unclear, it sounds like x4 more API requests.

A side note, right now, that cross projet search experience on the latest major blends the projects too much, we should show other hits more separate, so each project feels more standalone.

We don't know what users expect. If they are on v5.mui.com, do they want to see the v5, v6 or v7 of the X docs ?

I think that it should show the results of the most recent major compatible. It's achievable using or/and conditions in https://github.com/mui/material-ui/blob/1719ed4ed17a4eddc3d9ff493e6b8f5dd0bbda51/docs/src/modules/components/AppSearch.js#L452

We have 404 links because v5.mui.com has links to recent features (ie: https://v5.mui.com/x/react-charts/)

Yeah, this was the rabbit hole. I think it's step 3. The UX is broken for developers, even before being broken for the crawler:

• https://v5.mui.com/core/ shouldn't exist or redirect to https://mui.com/core/. It's a [website] page so shouldn't be verisoned.
• Link in https://v5.mui.com/material-ui/react-use-media-query/ to MUI X points to the wrong major version of MUI X
• etc