Expo 52 working in web, but on mobile throws: "AI_APICallError: Failed to process successful response"

[nordwestt]

The problem

I have streaming working on mobile using the new expo/fetch api (without using the vercel/ai library, just calling endpoints directly)

Then, I implemented my streaming using the vercel/ai library - and this works completely fine in the web context.

For mobile, I had to polyfill TransformStream, since this was marked as missing. But after this, it seemed to almost be working, except I get the error "AI_APICallError: Failed to process successful response".

Debug info

Using the docs, I casted the error to the APICallError type, and printed out some info.

url: http://192.168.1.50:11434/api/chat
requestBodyValues: {"format": undefined, "messages": [{"content": "Hello, how are you?", "role": "user"}], "model": "llama3.2:latest", "options": {"temperature": 0}, "tools": undefined}
statusCode: 200
responseBody: undefined
data: undefined

I've been banging my head against the wall on this issue for a few days - does anyone have any input that might help?
Been testing out combinations of various solutions suggested in existing discussions but no luck: #655 (comment)

Would love to use the vercel/ai library in my React Native project - and I'll make the solution available for anyone to clone.

Thank you SO much in advance - I appreciate any help on this topic

[lgrammel]

Can you provide more details about the polyfill you are using and the backend that sends the response? Is the error reproducible on web when using the polyfill only?

[lgrammel]

The issue is an incompatibility between the type of the ReadableStream that is returned by fetch and the polyfill. I tried to fix this in various ways for an hour to no avail.

Either Hermes needs to implement TransformStream, or you can try turning off Hermes: https://reactnative.dev/docs/hermes

[lgrammel]

Also, have you seen our expo guide? https://sdk.vercel.ai/docs/getting-started/expo (you need to customize fetch a bit)

[nordwestt]

Thanks for looking into it - could you point me towards the code section where you found the incompatibility?

Will try to disable hermes and use JavaScriptCore instead and cross my fingers that it's not also missing the necessary functions

[nordwestt]

Yep, I looked at the guide as one of the first things. What's strange is that it reads like it's supposed to work when using Expo 52 with the expo/fetch implementation. I'm curious to get in touch with the Author of the page - do you know who it is?

[lgrammel]

Here's some of the stuff I tried for reference:

import 'web-streams-polyfill/polyfill';

import { fetch as undiciFetch } from 'undici';
import { createReadableStreamWrapper } from '@mattiasbuelens/web-streams-adapter';
import { ReadableStream as PolyfillReadableStream } from 'web-streams-polyfill';
const toPolyfillReadable = createReadableStreamWrapper(
  PolyfillReadableStream as any,
);

import { createOpenAI } from '@ai-sdk/openai';
import { streamText } from 'ai';
import 'dotenv/config';

const openai = createOpenAI({
  fetch: async (url, options) => {
    const response = await undiciFetch(url as any, options as any);
    const polyfillBody = toPolyfillReadable(response.body!);

    console.log(polyfillBody);
    console.log(typeIsObject(polyfillBody));
    console.log(
      Object.prototype.hasOwnProperty.call(
        polyfillBody,
        '_readableStreamController',
      ),
    );
    console.log(isReadableStream2(polyfillBody));

    return new Proxy(response, {
      get(target, prop) {
        if (prop === 'body') {
          return polyfillBody;
        }
        return Reflect.get(target, prop);
      },
    }) as any;
  },
});

async function main() {
  const result = streamText({
    model: openai('gpt-3.5-turbo'),
    maxTokens: 512,
    temperature: 0.3,
    maxRetries: 5,
    prompt: 'Invent a new holiday and describe its traditions.',
  });

  for await (const textPart of result.textStream) {
    process.stdout.write(textPart);
  }

  console.log();
  console.log('Token usage:', await result.usage);
  console.log('Finish reason:', await result.finishReason);
}

main().catch(console.error);

export function typeIsObject(x: any): x is object | Function {
  return (typeof x === 'object' && x !== null) || typeof x === 'function';
}

export function isReadableStream(readable: any) {
  if (!typeIsObject(readable)) {
    return false;
  }
  if (typeof (readable as any).getReader !== 'function') {
    return false;
  }
  return true;
}

export function isReadableStream2(x: unknown): x is ReadableStream {
  if (!typeIsObject(x)) {
    return false;
  }

  if (!Object.prototype.hasOwnProperty.call(x, '_readableStreamController')) {
    return false;
  }

  return x instanceof PolyfillReadableStream;
}

[lgrammel]

Trying to understand this whole issue better. When you say "For mobile, I had to polyfill TransformStream", what do you mean exactly? what are you targeting and what are the library versions? we have seen people do this successful on Expo for mobile so this is confusing to us, we want to learn what's different

[nordwestt]

Mobile in my case means Android (google pixel running Android 15).
I get a runtime error when running called "Property 'TransformStream' doesn't exist", which I found you've already dealt with in this issue: #4083 - it essentially shows up as a big bright error page when loading the app, that won't let you continue.

This happens both when running the APK I build and also when using Expo Go (a development app that can load the project)
And when I polyfill with TransformStream from "web-streams-polyfill" then the app loads normally, and that's as far as I've gotten, where I get the "Failed to process successful response" when sending a message.

Here are some hopefully relevant library versions from my package.json:
"ai": "^4.1.4",
"expo": "^52.0.27",
"text-encoding": "^0.7.0",
"react": "^18.2.0",
"react-native": "^0.76.6",
"web-streams-polyfill": "^4.0.0"

[nordwestt]

Also, since more info is always useful, here is a link to the specific commit so you can see the code in context.
The path to where I'm using vercel/ai library is in: "/src/services/chat/providers/OllamaProvider.ts"

[lgrammel]

for testing purposes, can you try our official openai provider with openai instead of the ollama provider? i want to rule out that it's something ollama provider specific

[nordwestt]

Just tried - and I'm getting the same error

It works via web just fine, but on mobile it says "AI_APICallError: Failed to process successful response"

url: https://api.openai.com/v1/chat/completions
requestBodyValues: {"frequency_penalty": undefined, "function_call": undefined, "functions": undefined, "logit_bias": undefined, "logprobs": undefined, "max_completion_tokens": undefined, "max_tokens": undefined, "messages": [{"content": "Hey, how are you?", "role": "user"}], "metadata": undefined, "model": "gpt-4o", "parallel_tool_calls": undefined, "prediction": undefined, "presence_penalty": undefined, "reasoning_effort": undefined, "response_format": undefined, "seed": undefined, "stop": undefined, "store": undefined, "stream": true, "stream_options": undefined, "temperature": 0, "tool_choice": undefined, "tools": undefined, "top_logprobs": undefined, "top_p": undefined, "user": undefined}
statusCode: 200
responseBody: undefined
data: undefined

[jbreite]

Any updates on this? Not sure my problem is exactly the same, but I'm also using the SDK with an expo-52 project. Everything was working great until I upgraded my version to ^4.1.11 from 4.0.23. I needed to do this so I can add more providers.

This led me to this error:

 (NOBRIDGE) ERROR  [Error: Failed to parse stream string. Invalid code f.] ERROR,  Failed to parse stream string. Invalid code f.

I tried the solution from here. I put that on my useChat on the client, which then led to this error.

 (NOBRIDGE) ERROR  [ReferenceError: Property 'TextDecoderStream' doesn't exist] ERROR,  Property 'TextDecoderStream' doesn't exist

Which makes me think I need to add polyfills to the client to support this. I'm not sure which ones to add though.

Happy to clarify anything that could help

[nordwestt]

Thanks for the message

I tried downgrading to 4.0.23 and removing all polyfills and using the normal fetch. I immediately get the "TransformStream is undefined" error. So I tried setting global.fetch = expoFetch - and noticeably, the app actually loads without the aforementioned error!

But then I bump into errors

Problem 1

However, upon attempting to send a message to OpenAI, I get the error: "Cannot read property 'ReadableStream' of undefined"

Problem 2

I then tried polyfilling ReadableStream with 'web-streams-polyfill', which then leads to another error: "First parameter has member 'readable' that is not a ReadableStream"

This was a dead-end unfortunately - spent a few hours trying out different ideas but to no avail. Here's a discussion related to the issue: #1516

@jbreite are you able to share any polyfills and versions of your project?

[jbreite]

For 4.0.23 I got it working with expoFetch and generateAPIUrl from the Vercel Expo docs without adding any polyfills. I'm using expo router too. So when I use the useChat it looks like this:

  const {
    messages,
    error,
    handleInputChange,
    input,
    handleSubmit,
    append,
    isLoading,
  } = useChat({
    fetch: expoFetch as unknown as typeof globalThis.fetch,
    headers: {
    api: generateAPIUrl("/api/chat"),

Then on the API, I'm using streamText and return result.toDataStreamResponse();

That worked great for me, but I couldn't add Groq or Google to my project without upgrading. Haven't been able to figure out the polyfills yet.

[nordwestt]

Ah alright. The difference then is that I'm trying to run the entire thing in Expo - ie. I want Expo client to call the LLM server directly. Have you tried calling streamText directly within Expo mobile environment?

[jbreite]

I haven't tried that yet, but possibly my issue could help (if I figure out how to solve) it since it seems that the client now specifying streamProtocol to text

[jbreite]

Okay so I was able to get it work with this polyfill. This Reddit user helped me out: https://www.reddit.com/r/expo/comments/1idzgjo/upgraded_the_vercel_ai_sdk_to_the_latest_version/

import { Platform } from "react-native";

if (Platform.OS !== "web") {
  const setupPolyfills = async () => {
    const { polyfillGlobal } = await import(
      "react-native/Libraries/Utilities/PolyfillFunctions"
    );

    const { TextEncoderStream, TextDecoderStream } = await import(
      "@stardazed/streams-text-encoding"
    );

    polyfillGlobal("TextEncoderStream", () => TextEncoderStream);
    polyfillGlobal("TextDecoderStream", () => TextDecoderStream);
  };

  setupPolyfills();
}

export {};

I then import that at the top of my polyfill like this at the top of my _layout.tsx file

import "@/polyfills";

The fact that this works is a plus, but now that I am using streamProtocol: "text", I can't get the stream of tool calls that I am using. Is there a solve for this error a different way?

 (NOBRIDGE) ERROR  [Error: Failed to parse stream string. Invalid code f.] ERROR,  Failed to parse stream string. Invalid code f.