Skip to content

Commit

Permalink
[Docs] ggml: add notes about using wasi_nn config
Browse files Browse the repository at this point in the history
Signed-off-by: dm4 <[email protected]>
  • Loading branch information
dm4 committed Oct 28, 2024
1 parent 5c064aa commit 610f4c2
Showing 1 changed file with 9 additions and 2 deletions.
11 changes: 9 additions & 2 deletions wasmedge-ggml/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -207,9 +207,16 @@ Currently, the WASI-NN ggml plugin supports several ways to set the metadata for
When constructing the graph, you can set the metadata by using the `config` method.

```rust
... wasi_nn::GraphBuilder::new(...).config(serde_json::to_string(&options).unwrap()) ...
wasmedge_wasi_nn::GraphBuilder::new(...)
.config(serde_json::to_string(&options).unwrap())
.build_from_cache(...)
.unwrap();
```

> [!NOTE]
> The config will only be set when constructing the graph using `build_from_cache`.
> Due to the file size limitation, you **SHOULD** use `build_from_cache` with `--nn-preload` to load the large model file.
2. From the input tensor

When setting input to the context, specify the index with 1 for the metadata.
Expand All @@ -220,7 +227,7 @@ If you modify the `n-gpu-layers` parameter, the model will be reloaded.
context
.set_input(
1,
wasi_nn::TensorType::U8,
wasmedge_wasi_nn::TensorType::U8,
&[1],
serde_json::to_string(&options).expect("Failed to serialize options").as_bytes().to_vec(),
)
Expand Down

0 comments on commit 610f4c2

Please sign in to comment.