Skip to content

Commit 838a52d

Browse files
YousefEDmatthewlipskinperez0111
authored
feature: AI (#1674)
This PR adds AI functionality to BlockNote! --------- Co-authored-by: matthewlipski <[email protected]> Co-authored-by: Nick the Sick <[email protected]> Co-authored-by: Matthew Lipski <[email protected]>
1 parent f0946b2 commit 838a52d

File tree

632 files changed

+30538
-3413
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

632 files changed

+30538
-3413
lines changed

‎.github/workflows/build.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -62,7 +62,7 @@ jobs:
6262

6363
- name: Soft release
6464
id: soft-release
65-
run: pnpx pkg-pr-new publish './packages/*' --compact
65+
run: pnpx pkg-pr-new publish './packages/*' # TODO disabled only for AI branch--compact
6666

6767
playwright:
6868
name: "Playwright Tests - ${{ matrix.browser }}"

‎.gitignore

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -33,4 +33,6 @@ release
3333
/test-results/
3434
/playwright-report/
3535
/playwright/.cache/
36-
.nx/
36+
.env
37+
*.pem
38+
.nx/

‎.vscode/settings.json

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,5 +13,10 @@
1313
"typescript.tsdk": "node_modules/typescript/lib",
1414
"[xml]": {
1515
"editor.defaultFormatter": "redhat.vscode-xml"
16+
},
17+
"scm.defaultViewMode": "tree",
18+
"search.defaultViewMode": "tree",
19+
"[typescript]": {
20+
"editor.defaultFormatter": "esbenp.prettier-vscode"
1621
}
1722
}

‎docs/.env.local.example

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -28,3 +28,6 @@ BETTER_AUTH_URL=http://localhost:3000
2828
# # The SENTRY_AUTH_TOKEN variable is picked up by the Sentry Build Plugin.
2929
# # It's used for authentication when uploading source maps.
3030
# SENTRY_AUTH_TOKEN=
31+
32+
NEXT_PUBLIC_BLOCKNOTE_AI_SERVER_API_KEY=
33+
NEXT_PUBLIC_BLOCKNOTE_AI_SERVER_BASE_URL=

‎docs/package.json

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
{
22
"name": "docs",
3-
"version": "0.26.0",
3+
"version": "0.30.0",
44
"private": true,
55
"scripts": {
66
"dev": "next dev",

‎docs/pages/docs/_meta.json

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,7 @@
88
"custom-schemas": "Custom Schemas",
99
"collaboration": "Collaboration",
1010
"advanced": "Advanced",
11+
"ai": "AI",
1112
"discord-link": {
1213
"title": "Community ↗",
1314
"href": "https://discord.gg/Qc2QTTH5dF",

‎docs/pages/docs/ai.mdx

Lines changed: 70 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,70 @@
1+
---
2+
title: AI Rich Text Editing
3+
description: Add AI functionality to your BlockNote rich text editor
4+
imageTitle: BlockNote AI
5+
path: /docs/ai
6+
---
7+
8+
import { Example } from "@/components/example";
9+
import { Callout } from "nextra/components";
10+
11+
# BlockNote AI integration
12+
13+
With BlockNote AI, you can add AI functionality to your rich text editor.
14+
Users can work with an AI agent to edit, write and format their documents.
15+
16+
<video
17+
style={{ marginTop: "2rem" }}
18+
src="/img/screenshots/blocknote-ai.mp4"
19+
aria-label="BlockNote AI Video demo"
20+
controls
21+
/>
22+
23+
<Callout>
24+
BlockNote AI is now in early preview - we'd love to hear your feedback!
25+
We also collaborate with companies to help with the integration or implement
26+
more advanced AI related functionality. [Get in touch](/about).
27+
</Callout>
28+
29+
## Features
30+
31+
BlockNote AI has been designed to be fully customizable.
32+
BlockNote shows exactly what the AI agent is doing - making it easy for users to
33+
work hand in hand with AI agents.
34+
35+
### User Experience
36+
37+
- **Interactive AI Suggestions**: Users can accept or reject AI suggestions with a simple click, maintaining full control over their content
38+
- **Real-time Feedback**: Streaming support provides immediate responses, making the AI interaction feel natural and responsive
39+
- **Transparent Operations**: See exactly what the AI is doing at each step, with clear visual indicators of AI actions
40+
41+
### Technical Capabilities
42+
43+
- **Flexible Command System**: Customize commands to write or update existing content and formatting
44+
- **Multi-step Workflows**: Support for "Human in the Loop" workflows where users can guide the AI
45+
- **Model Agnostic**: Connect any LLM model (from Llama to OpenAI, Mistral or Anthropic)
46+
- **Customizable Prompts**: Fine-tune AI behavior with custom prompts and instructions
47+
- **No Backend Required**: Use your own infrastructure or connect directly to a hosted LLM API
48+
49+
### Integration
50+
51+
- **Built on Vercel AI SDK**: Leverages the power of the [Vercel AI SDK](https://sdk.vercel.ai) for reliable AI integration
52+
- **Easy Setup**: Get started quickly with minimal configuration
53+
- **Completely Customizable**: Fully customizable commands and UI elements
54+
55+
<Callout type={"info"}>
56+
This feature is provided by the `@blocknote/xl-ai` package. `xl-` packages are fully
57+
open source, but released under a copyleft license. A commercial license for
58+
usage in closed source, proprietary products comes as part of the [Business
59+
subscription](/pricing).
60+
</Callout>
61+
62+
### Next Steps
63+
64+
- Try the [basic AI demo](/examples/ai/minimal) to see it in action
65+
- Explore different models in the [AI playground](/examples/ai/playground)
66+
- Check out the [setup documentation](/docs/ai/getting-started)
67+
- Learn about [customizing commands](/docs/ai/custom-commands)
68+
- Review the [API Reference](/docs/ai/reference)
69+
70+
Have a feature request or need help with integration? [Get in touch](/about) with our team.

‎docs/pages/docs/ai/_meta.json

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
{
2+
"getting-started": "Getting Started",
3+
"custom-commands": "Custom Commands",
4+
"reference": "Reference"
5+
}

‎docs/pages/docs/ai/custom-commands.mdx

Lines changed: 126 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,126 @@
1+
---
2+
title: Custom AI Commands
3+
description: Customize the AI menu items (commands) in your BlockNote rich text editor
4+
imageTitle: BlockNote AI
5+
path: /docs/ai/custom-commands
6+
---
7+
8+
import { Example } from "@/components/example";
9+
import { ThemedImage } from "@/components/ThemedImage";
10+
import { Callout } from "nextra/components";
11+
12+
# Custom AI Menu Items (commands)
13+
14+
A central part when users are interacting with the AI agent is the _AI Suggestion Menu_ where users can enter a custom prompt or select a pre-defined command:
15+
16+
<ThemedImage
17+
src="/img/screenshots/ai-menu.png"
18+
darkImage="/img/screenshots/ai-menu-dark.png"
19+
alt="image"
20+
width={518}
21+
height={177}
22+
/>
23+
24+
This menu is easy to customize so you can expose commands fine-tuned to your application.
25+
26+
## Defining your own commands
27+
28+
First, we define a new AI command. Let's create one that makes selected text more informal.
29+
30+
```tsx
31+
import { AIMenuSuggestionItem, getAIExtension } from "@blocknote/xl-ai";
32+
33+
// Custom item to make the text more informal.
34+
export const makeInformal = (
35+
editor: BlockNoteEditor,
36+
): AIMenuSuggestionItem => ({
37+
key: "make_informal",
38+
title: "Make Informal",
39+
aliases: ["informal", "make informal", "casual"],
40+
icon: <RiEmotionHappyFill size={18} />,
41+
onItemClick: async () => {
42+
await getAIExtension(editor).callLLM({
43+
// The prompt to send to the LLM:
44+
userPrompt: "Give the selected text a more informal (casual) tone",
45+
// Tell the LLM to specifically use the selected content as context (instead of the whole document)
46+
useSelection: true,
47+
// We only want the LLM to update selected text, not to add / delete blocks
48+
defaultStreamTools: {
49+
add: false,
50+
delete: false,
51+
update: true,
52+
},
53+
});
54+
},
55+
size: "small",
56+
});
57+
```
58+
59+
Now, we create a customized AI Menu to show this command when the user has selected some text and opened the AI menu:
60+
61+
```tsx
62+
import { AIMenu, getDefaultAIMenuItems } from "@blocknote/xl-ai";
63+
64+
function CustomAIMenu() {
65+
return (
66+
<AIMenu
67+
items={(
68+
editor: BlockNoteEditor<any, any, any>,
69+
aiResponseStatus:
70+
| "user-input"
71+
| "thinking"
72+
| "ai-writing"
73+
| "error"
74+
| "user-reviewing"
75+
| "closed",
76+
) => {
77+
if (aiResponseStatus === "user-input") {
78+
if (editor.getSelection()) {
79+
// When a selection is active (so when the AI Menu is opened via the Formatting Toolbar),
80+
// we add our `makeInformal` command to the default items.
81+
return [
82+
...getDefaultAIMenuItems(editor, aiResponseStatus),
83+
makeInformal(editor),
84+
];
85+
} else {
86+
return getDefaultAIMenuItems(editor, aiResponseStatus);
87+
}
88+
}
89+
90+
// for other states, return the default items
91+
return getDefaultAIMenuItems(editor, aiResponseStatus);
92+
}}
93+
/>
94+
);
95+
}
96+
```
97+
98+
Now, let's use this custom AI Menu in our app:
99+
100+
```tsx
101+
<BlockNoteView
102+
editor={editor}
103+
formattingToolbar={false}
104+
slashMenu={false}
105+
>
106+
{/* Creates a new AIMenu with the default items, as well as our custom
107+
ones. */}
108+
<AIMenuController aiMenu={CustomAIMenu} />
109+
110+
{/* ...other UI Elements... */}
111+
<FormattingToolbarWithAI />
112+
<SuggestionMenuWithAI editor={editor} />
113+
</BlockNoteView>
114+
```
115+
116+
# Full example
117+
118+
Have a look at the full example below, where we also add an AI menu item when no selection is open
119+
(e.g., when the editor is opened by typing `/ai` in the editor).
120+
121+
<Example name="ai/custom-ai-menu-items" />
122+
123+
# Reference
124+
125+
So far, we've added basic commands to the editor, but it's possible to completely customize low level prompts sent to the LLM.
126+
To learn in detail about the `callLLM` method used in this guide, continue to the [AI reference docs](/docs/ai/reference).

‎docs/pages/docs/ai/getting-started.mdx

Lines changed: 112 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,112 @@
1+
---
2+
title: Getting Started with BlockNote AI
3+
description: Add AI functionality to your BlockNote rich text editor
4+
imageTitle: BlockNote AI
5+
path: /docs/ai/getting-started
6+
---
7+
8+
import { Example } from "@/components/example";
9+
import { Callout } from "nextra/components";
10+
11+
# Getting Started with BlockNote AI
12+
13+
This guide walks you through the steps to add AI functionality to your BlockNote rich text editor.
14+
15+
First, install the `@blocknote/xl-ai` package:
16+
17+
```bash
18+
npm install @blocknote/xl-ai
19+
```
20+
21+
## Creating a Model
22+
23+
BlockNote AI uses the the [AI SDK](https://ai-sdk.dev/docs/foundations/overview) to standardize integrating artificial intelligence (AI) models across [supported providers](https://ai-sdk.dev/docs/foundations/providers-and-models).
24+
25+
As a first step, you'll need to register a new model with the AI SDK. For example, for Llama hosted on Groq:
26+
27+
```bash
28+
npm install @ai-sdk/groq
29+
```
30+
31+
```ts
32+
import { createGroq } from "@ai-sdk/groq";
33+
34+
const provider = createGroq({
35+
apiKey: "YOUR_GROQ_API_KEY",
36+
});
37+
38+
const model = provider("llama-3.3-70b-versatile");
39+
```
40+
41+
<Callout type={"warning"}>
42+
Note that this setup directly calls the provider from the client, and exposes
43+
your API keys on the client. For Production scenarios, a more common approach
44+
is to handle authentication on your own server and proxy requests to a
45+
provider. See our [Demo AI
46+
Server](https://github.com/TypeCellOS/BlockNote/tree/main/packages/xl-ai-server)
47+
for a Node.js example or check the AI SDK best practices.
48+
</Callout>
49+
50+
## Setting up the editor
51+
52+
Now, you can create the editor with the AI Extension enabled:
53+
54+
```ts
55+
import { createBlockNoteEditor } from "@blocknote/core";
56+
import { BlockNoteAIExtension } from "@blocknote/xl-ai";
57+
import {
58+
locales as aiLocales,
59+
createAIExtension,
60+
} from "@blocknote/xl-ai";
61+
62+
const editor = createBlockNoteEditor({
63+
dictionary: {
64+
...en,
65+
ai: aiLocales.en, // add default translations for the AI extension
66+
},
67+
extensions: [
68+
createAIExtension({
69+
model,
70+
}),
71+
],
72+
// ... other editor options
73+
});
74+
```
75+
76+
See the [API Reference](/docs/ai/reference) for more information on the `createAIExtension` method.
77+
78+
## Adding AI UI elements
79+
80+
Now, the only thing left to do is to customize the UI elements of your editor.
81+
We want to:
82+
83+
- add an AI button to the formatting toolbar (shown when selecting text)
84+
- add an AI option to the slash menu (shown when typing a `/`)
85+
86+
We do this by disabling the default FormattingToolbar and SuggestionMenu and adding our own. See [Changing UI Elements](/docs/ui-components) for more information.
87+
88+
```tsx
89+
<BlockNoteView
90+
editor={editor}
91+
// We're disabling some default UI elements
92+
formattingToolbar={false}
93+
slashMenu={false}
94+
>
95+
{/* Add the AI Command menu to the editor */}
96+
<AIMenuController />
97+
98+
{/* Create you own Formatting Toolbar with an AI button,
99+
(see the full example code below) */}
100+
<FormattingToolbarWithAI />
101+
102+
{/* Create you own SlashMenu with an AI option,
103+
(see the full example code below) */}
104+
<SuggestionMenuWithAI editor={editor} />
105+
</BlockNoteView>
106+
```
107+
108+
# Full Example
109+
110+
See the full example code and live demo. Select some text and click the AI (stars) button, or type `/ai` anywhere in the editor to access AI functionality.
111+
112+
<Example name="ai/minimal" />

0 commit comments

Comments
 (0)