Skip to content

feat (examples): add example and docs on NIM use via openai-compatible provider #4666

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Feb 3, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
121 changes: 121 additions & 0 deletions content/providers/02-openai-compatible-providers/35-nim.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,121 @@
---
title: NVIDIA NIM
description: Use NVIDIA NIM OpenAI compatible API with the AI SDK.
---

# NVIDIA NIM Provider

[NVIDIA NIM](https://www.nvidia.com/en-us/ai/) provides optimized inference microservices for deploying foundation models. It offers an OpenAI-compatible API that you can use with the AI SDK.

## Setup

The NVIDIA NIM provider is available via the `@ai-sdk/openai-compatible` module as it is compatible with the OpenAI API.
You can install it with:

<Tabs items={['pnpm', 'npm', 'yarn']}>
<Tab>
<Snippet text="pnpm add @ai-sdk/openai-compatible" dark />
</Tab>
<Tab>
<Snippet text="npm install @ai-sdk/openai-compatible" dark />
</Tab>
<Tab>
<Snippet text="yarn add @ai-sdk/openai-compatible" dark />
</Tab>
</Tabs>

## Provider Instance

To use NVIDIA NIM, you can create a custom provider instance with the `createOpenAICompatible` function from `@ai-sdk/openai-compatible`:

```ts
import { createOpenAICompatible } from '@ai-sdk/openai-compatible';

const nim = createOpenAICompatible({
name: 'nim',
baseURL: 'https://integrate.api.nvidia.com/v1',
headers: {
Authorization: `Bearer ${process.env.NIM_API_KEY}`,
},
});
```

<Note>
You can obtain an API key and free credits by registering at [NVIDIA
Build](https://build.nvidia.com/explore/discover). New users receive 1,000
inference credits to get started.
</Note>

## Language Models

You can interact with NIM models using a provider instance. For example, to use [DeepSeek-R1](https://build.nvidia.com/deepseek-ai/deepseek-r1), a powerful open-source language model:

```ts
const model = nim.chatModel('deepseek-ai/deepseek-r1');
```

### Example - Generate Text

You can use NIM language models to generate text with the `generateText` function:

```ts
import { createOpenAICompatible } from '@ai-sdk/openai-compatible';
import { generateText } from 'ai';

const nim = createOpenAICompatible({
name: 'nim',
baseURL: 'https://integrate.api.nvidia.com/v1',
headers: {
Authorization: `Bearer ${process.env.NIM_API_KEY}`,
},
});

const { text, usage, finishReason } = await generateText({
model: nim.chatModel('deepseek-ai/deepseek-r1'),
prompt: 'Tell me the history of the San Francisco Mission-style burrito.',
});

console.log(text);
console.log('Token usage:', usage);
console.log('Finish reason:', finishReason);
```

### Example - Stream Text

NIM language models can also generate text in a streaming fashion with the `streamText` function:

```ts
import { createOpenAICompatible } from '@ai-sdk/openai-compatible';
import { streamText } from 'ai';

const nim = createOpenAICompatible({
name: 'nim',
baseURL: 'https://integrate.api.nvidia.com/v1',
headers: {
Authorization: `Bearer ${process.env.NIM_API_KEY}`,
},
});

const result = streamText({
model: nim.chatModel('deepseek-ai/deepseek-r1'),
prompt: 'Tell me the history of the Northern White Rhino.',
});

for await (const textPart of result.textStream) {
process.stdout.write(textPart);
}

console.log();
console.log('Token usage:', await result.usage);
console.log('Finish reason:', await result.finishReason);
```

NIM language models can also be used with other AI SDK functions like `generateObject` and `streamObject`.

<Note>
Model support for tool calls and structured object generation varies. For
example, the
[`meta/llama-3.3-70b-instruct`](https://build.nvidia.com/meta/llama-3_3-70b-instruct)
model supports object generation capabilities. Check each model's
documentation on NVIDIA Build for specific supported features.
</Note>
1 change: 1 addition & 0 deletions content/providers/02-openai-compatible-providers/index.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ Below we focus on the general setup and provider instance creation. You can also
We provide detailed documentation for the following OpenAI compatible providers:

- [LM Studio](/providers/openai-compatible-providers/lmstudio)
- [NIM](/providers/openai-compatible-providers/nim)
- [Baseten](/providers/openai-compatible-providers/baseten)

The general setup and provider instance creation is the same for all of these providers.
Expand Down
33 changes: 33 additions & 0 deletions examples/ai-core/src/generate-object/nim.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
import { createOpenAICompatible } from '@ai-sdk/openai-compatible';
import { generateObject } from 'ai';
import { z } from 'zod';
import 'dotenv/config';

async function main() {
const nim = createOpenAICompatible({
baseURL: 'https://integrate.api.nvidia.com/v1',
name: 'nim',
headers: {
Authorization: `Bearer ${process.env.NIM_API_KEY}`,
},
});
const model = nim.chatModel('meta/llama-3.3-70b-instruct');
const result = await generateObject({
model,
schema: z.array(
z.object({
name: z.string(),
breed: z.string(),
}),
),
prompt:
'Generate 10 cat names and breeds for a fictional book about a world where cats rule shrimp.',
});

console.log(JSON.stringify(result.object, null, 2));
console.log();
console.log('Token usage:', result.usage);
console.log('Finish reason:', result.finishReason);
}

main().catch(console.error);
25 changes: 25 additions & 0 deletions examples/ai-core/src/generate-text/nim.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
import { createOpenAICompatible } from '@ai-sdk/openai-compatible';
import { generateText } from 'ai';
import 'dotenv/config';

async function main() {
const nim = createOpenAICompatible({
baseURL: 'https://integrate.api.nvidia.com/v1',
name: 'nim',
headers: {
Authorization: `Bearer ${process.env.NIM_API_KEY}`,
},
});
const model = nim.chatModel('deepseek-ai/deepseek-r1');
const result = await generateText({
model,
prompt: 'Tell me the history of the San Francisco Mission-style burrito.',
});

console.log(result.text);
console.log();
console.log('Token usage:', result.usage);
console.log('Finish reason:', result.finishReason);
}

main().catch(console.error);
41 changes: 41 additions & 0 deletions examples/ai-core/src/stream-object/nim.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
import { createOpenAICompatible } from '@ai-sdk/openai-compatible';
import { streamObject } from 'ai';
import { z } from 'zod';
import 'dotenv/config';

async function main() {
const nim = createOpenAICompatible({
baseURL: 'https://integrate.api.nvidia.com/v1',
name: 'nim',
headers: {
Authorization: `Bearer ${process.env.NIM_API_KEY}`,
},
});
const model = nim.chatModel('meta/llama-3.3-70b-instruct');
const result = streamObject({
model,
schema: z.object({
characters: z.array(
z.object({
name: z.string(),
class: z
.string()
.describe('Character class, e.g. warrior, mage, or thief.'),
description: z.string(),
}),
),
}),
prompt:
'Generate 3 character descriptions for a fantasy role playing game.',
});

for await (const partialObject of result.partialObjectStream) {
console.clear();
console.log(partialObject);
}

console.log();
console.log('Token usage:', await result.usage);
}

main().catch(console.error);
28 changes: 28 additions & 0 deletions examples/ai-core/src/stream-text/nim.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
import { createOpenAICompatible } from '@ai-sdk/openai-compatible';
import { streamText } from 'ai';
import 'dotenv/config';

async function main() {
const nim = createOpenAICompatible({
baseURL: 'https://integrate.api.nvidia.com/v1',
name: 'nim',
headers: {
Authorization: `Bearer ${process.env.NIM_API_KEY}`,
},
});
const model = nim.chatModel('deepseek-ai/deepseek-r1');
const result = streamText({
model,
prompt: 'Tell me the history of the Northern White Rhino.',
});

for await (const textPart of result.textStream) {
process.stdout.write(textPart);
}

console.log();
console.log('Token usage:', await result.usage);
console.log('Finish reason:', await result.finishReason);
}

main().catch(console.error);
1 change: 1 addition & 0 deletions turbo.json
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@
"LUMA_API_KEY",
"MISTRAL_API_KEY",
"NEXT_RUNTIME",
"NIM_API_KEY",
"NODE_ENV",
"OPENAI_API_KEY",
"OPENAI_API_BASE",
Expand Down
Loading