Support of Guardrails in Spring AI

[bntv-config]

Can we implement some guardrails features in built in Spring AI so we no need to fine tune the prompt:

Gurard rails features:

• Should not be baiosed when ansering questions
• Does not answer political questions
• Avoid content related to violence, hate speech, self-harm, or illegal activities.
• Do not share or ask for personal identifiable information (PII).
• Use respectful and professional language.
• Ensure responses do not cause harm or distress to users.

This should also be configurable , like developer can add custom also.

[bntv-config]

@markpollack Could you please let me know if this is in Roadmap and by when it will be available ?

[RRajdev]

@markpollack any update on this. It is crucial for our use case.

[viralpatel]

For those who are using AWS Bedrock and want to integrate Bedrock Guardrails:

Until the Guardrail support is added to the Spring AI project, here's an alternate way of overriding the bedrock client and inject guardrail Id and version.

import software.amazon.awssdk.auth.credentials.DefaultCredentialsProvider;
import software.amazon.awssdk.regions.Region;
import software.amazon.awssdk.services.bedrockruntime.BedrockRuntimeClient;
import software.amazon.awssdk.services.bedrockruntime.model.ConverseRequest;
import software.amazon.awssdk.services.bedrockruntime.model.ConverseResponse;
import software.amazon.awssdk.services.bedrockruntime.model.GuardrailConfiguration;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

@Configuration
public class BedrockGuardrailConfiguration {

    @Bean
    public BedrockRuntimeClient bedrockRuntimeClient() {
        BedrockRuntimeClient bedrockRuntimeClient = BedrockRuntimeClient.builder()
                .region(Region.US_EAST_1)
                .credentialsProvider(DefaultCredentialsProvider.create())
                .build();

        return new GuardrailBedrockClient(bedrockRuntimeClient, "guardrailId", "guardrailVersion");
    }
}

class GuardrailBedrockClient implements BedrockRuntimeClient {
    private final BedrockRuntimeClient instance;
    private final String guardrailId;
    private final String guardrailVersion;

    public GuardrailBedrockClient(BedrockRuntimeClient instance, String guardrailId, String guardrailVersion) {
        this.instance = instance;
        this.guardrailId = guardrailId;
        this.guardrailVersion = guardrailVersion;
    }

    @Override
    public void close() {
        instance.close();
    }

    @Override
    public String serviceName() {
        return instance.serviceName();
    }

    @Override
    public ConverseResponse converse(ConverseRequest converseRequest) {
        ConverseRequest newRequest = converseRequest.toBuilder()
                .guardrailConfig(
                    GuardrailConfiguration.builder()
                        .guardrailIdentifier(guardrailId)
                        .guardrailVersion(guardrailVersion)
                        .build()
                )
                .build();
        
        return instance.converse(newRequest);
    }
}

[bntv-config]

We are more intersted to use some open source capabilities integrated with Spring AI. In my current code where I am using Spring AI M8 version, I have build a SDK for my company use case and using https://github.com/protectai/llm-guard for specific use cases (It is affective and solving my purpose) . I have used advisiors for this.

Just wanted to know if Spring AI community is doing something in this area and can we have some another approach to solve this. Means can we get in built support or user can extend by themself.

[ThomasVitale]

I include here my answer from #1011 (comment):

It should probably be included in the documentation, but the Advisor API is how you can specify guardrails in a ChatClient workflow. Advisors intercept an LLM request before and after its execution, giving you the hooks to define input and output guardrails. Here's some examples of an input guardrail and an output guardrail.

It would be nice to get some specialisation of the Advisor API tailored towards guardrails, providing convenient utils for doing things like deny LLM requests, retry, or re-prompt. You can do all that today with the Advisor API, but less conveniently.

[markpollack]

I am a big fan to add this functionality to Spring AI. We should come up with a plan to divide an conquer. @bntv-config @viralpatel and others - can you please start to create a list of specific guardrails implementations we should put in scope in order or priority. I have some code for integrating with https://github.com/microsoft/presidio

Note, there is also the new Spring AI Community organization, where a guardrails project can also incubate and grow - probably at a much faster pace than we will be making point releases. https://github.com/spring-ai-community

Last but not least, early in my AI learning journey, I started listening to this podcast https://practicalai.fm/ and I believe the host makes this product - https://www.guardrailsai.com/