Custom Sampler Stages #961
Open
+318
−36
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…he `LLamaSamplerNative*`, not the `GCHandle` value). - Properly typed as much of it as possible (using delegate pointers) - Replaced pointers with `ref` where possible, making much of the API safe - Created an example for custom sampling (removed test code from `DefaultSamplingPipeline`)
Lyrcaxis
approved these changes
Oct 25, 2024
…pler` - Improved documentation on `Apply` related to setting the `Sorted` flag
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Added support for custom sampler stages, written entirely in safe C#. These stages can be added to a sampler pipeline just like any other. See the example in
LLama.Examples\Examples\CustomSampler.csfor an example of a custom pipeline using a custom stage.The internals here are fairly complex, because we need to keep some manually managed memory (so we can have persistent pointers to it) and also make sure the GC doesn't collect things. The overall flow is as follows:
ICustomSamplerAddCustomCustomSamplerHandleis allocated, to own all of the various bits of memoryCustomSamplerHandleholds aGCHandlewhich keeps itself alive.CustomSamplerHandlealso allocates some memory to hold some native structures (_samplerNativePtrand_samplerNativeInterfacePtr) and passes pointers across to the native side.freeit will destroy theGCHandlekeeping itself alive, free the native memory, and finally callsICustomSampler.Free.