LLMs.txt #2497
LLMs.txt #2497
Conversation
See https://docs.firecrawl.dev/features/alpha/llmstxt. This work identified a number of issues: - Crawlers cannot crawl all languages, we need better discovery - llms.txt needs to be served from ably.com, not ably.com/docs - There are issues with language selection where if you select Node.js, navigate to a page that does not support Node.js, and then navigate to one that does, it remains on JS. This means we're losing the language selection for crawlers too.
|
Important Review skippedAuto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the You can disable this status message by setting the Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
There was a problem hiding this comment.
I get the intent, this is pretty cool, but in practice this won't work well and the reason is it will always be behind the current PR. We cannot update the text file with the updates from the PR because the PR has not been deployed yet...
That said, I wonder if we cannot simply attempt to create a build step that simply slurps up all the HTML files in public, extracts the elements under a certain selector (so we don't get the navigation elements) and renders it out as a huge text file (or as mutliple text files), and then we could even solve the language problem because we understand the markup
There was a problem hiding this comment.
So to get my head into this - Firecrawl takes an entry point, crawls from there, and spits out a markdown file consisting of crawled links and a one-line synopsis of the contents that's generated through some LLM-ish means.
As part of the nav rewrite we made the data foundations of the pages more accessible. To achieve what you ask here re: languages, we could compile a matrix of pages (pubsub ones as an example here) and languages per product into a page and point firecrawl at that.
To be honest, this would get us most of the way there, by looping in the meta descriptions currently on the textile documents we pretty much have what Firecrawl provides without the need for another service.
Alternatively, lightest approach could be to hide some links on each page pointing to each available language variant, but this would fall apart if Firecrawl assesses the visibility of links in its traversal.
|
Is this mergeable? |
|
@kennethkalmer did you have a chance to look at proxying this if we merge, and/or the scope of work required to generate the files Jamie mentioned? |
See https://llmstxt.site/ for context.
Modern AI friendly tools provide llms.txt files for LLMs.
This script simply calls out to Firecrawl and generates an LLMs.txt file. However, it is not working as expected and some additional work is needed: