Skip to content

Add ability to view randomly selected chunks #27

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 2 commits into from
Closed

Add ability to view randomly selected chunks #27

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
151a8ec
Add ability to view randomly selected chunks
anastasds Jun 10, 2025
44667db
Update instructlab-knowledge.ipynb
anastasds Jun 10, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 11 additions & 5 deletions notebooks/instructlab-knowledge/instructlab-knowledge.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -260,7 +260,9 @@
"source": [
"### View the Chunks\n",
"\n",
"To view the chunks, run through the following cell. As you can see the document is broken into small pieces with metadata about the chunk based on the document's format"
"To view a random sample of the generated chunks, run through the following cell. Re-run the cell to see another random sample. If there are chunking issues, try different conversion and/or chunking configurations.\n",
"\n",
"As you can see, the document is broken into small pieces with metadata about the chunk based on the document's format."
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You mention the metadata here being in the chunk. It would be nice to have an example or two of chunks within the documentation and point out that metadata. Also we know table chunks can be hard to parse upon first glance. I think showing the user an example just to prepare them for seeing a chunk like that might be helpful

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think having an example chunk could either be helpful or confusing here. What if their chunk isn't similar? Is that bad or good?

]
},
{
Expand All @@ -270,8 +272,12 @@
"metadata": {},
"outputs": [],
"source": [
"#print(all_chunks)\n",
"print(chunks[0])"
"import random\n",
"import json\n",
"\n",
"sample = random.sample(all_chunks, min(len(all_chunks), 5))\n",
"for chunk in sample:\n",
" print(f\"{chunk['chunk']}\\n\\n====\\n\")"
]
},
{
Expand Down Expand Up @@ -681,7 +687,7 @@
],
"metadata": {
"kernelspec": {
"display_name": ".venv",
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
Expand All @@ -695,7 +701,7 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.12"
"version": "3.11.9"
}
},
"nbformat": 4,
Expand Down