[Embeddings][OpenAI] Support embeddings via engine.embeddings.create()

[CharlieFRuan]

This PR supports embedding model with engine.embeddings.create().

For running example, see examples/embeddings, where we can run with OpenAI API using engine.embeddings.create(), and we can also integrate with Langchain's EmbeddingsInterface and MemoryVectorStore.

  const documents = ["[CLS] The Data Cloud! [SEP]", "[CLS] Mexico City of Course! [SEP]"];
  const engine: webllm.MLCEngineInterface = await webllm.CreateMLCEngine(
    "snowflake-arctic-embed-m-q0f32-MLC-b4",
  );
  const docReply = await engine.embeddings.create({ input: documents });

Currently, only snowflake-arctic-embed-s and snowflake-arctic-embed-m are supported. We add the following models to the prebuilt model list:

• "snowflake-arctic-embed-m-q0f32-MLC-b32"
• "snowflake-arctic-embed-m-q0f32-MLC-b4"
• "snowflake-arctic-embed-s-q0f32-MLC-b32"
• "snowflake-arctic-embed-s-q0f32-MLC-b4"

b32 means the model is compiled to support a maximum batch size of 32. If an input with more than 32 entries are provided, we will call multiple forward() (e.g. if input has 67 entries, we forward 3 times). The larger the maximum batch size, the more memory it takes to load the model. See vram_required_MB in config.ts for specifics.

Besides, we currently do not allow loading multiple models in a single engine, making it a bit inconvenient for usecases like RAG. Engine with multiple models loaded will be supported soon.

Internal code changes

• We implement EmbeddingPipeline in src/embedding.ts, parallel to LLMChatPipeline in llm_chat.ts
• In engine.ts, we determine which pipeline to load based on ModelRecord.model_type
• Implemented embedding() in MLCEngineInterface, hence supporting both MLCEngine and WebWorkerMLCEngine
• Implemented API specifications in src/openai_api_protocols/embedding.ts

Tested

• Input of size 64 with 512 tokens using b32 model, finishes with 2 iterations
• Ensured output in examples/embedding is consistent with transformers in Python
• Tested with WebWorkerMLCEngine