Embeddings

TanStack AI provides embedding generation through dedicated embedding adapters that follow the same tree-shakeable, per-model-typed architecture as every other activity. The embed() function turns text — and, for multimodal models, images — into vectors for semantic search, RAG, clustering, and classification.

Overview

Currently supported:

OpenAI: text-embedding-3-small, text-embedding-3-large (text)
Cohere: embed-v4.0 (text + image, fused multimodal)
Google Gemini: gemini-embedding-001 (text)
Mistral: mistral-embed, codestral-embed (text)
Amazon Bedrock: Titan Text Embeddings V2 (text), Titan Multimodal Embeddings G1 (text + image), Cohere Embed v3 on Bedrock (text)
Ollama: nomic-embed-text, mxbai-embed-large, and any local embedding model (text)

Basic Usage

import { embed } from "@tanstack/ai";
import { openaiEmbedding } from "@tanstack/ai-openai";

const result = await embed({
  adapter: openaiEmbedding("text-embedding-3-small"),
  input: "a red guitar",
});

console.log(result.embeddings[0]?.vector); // number[]

input accepts a single item or an array of items; the result always carries an embeddings array with one vector per input item, in input order:

import { embed } from "@tanstack/ai";
import { openaiEmbedding } from "@tanstack/ai-openai";

const result = await embed({
  adapter: openaiEmbedding("text-embedding-3-large"),
  input: ["a red guitar", "a blue drum kit", "a vintage synthesizer"],
});

for (const embedding of result.embeddings) {
  console.log(embedding.index, embedding.vector.length);
}

Requesting Dimensions

Models with configurable (Matryoshka) dimensions accept a top-level dimensions option:

import { embed } from "@tanstack/ai";
import { openaiEmbedding } from "@tanstack/ai-openai";

const result = await embed({
  adapter: openaiEmbedding("text-embedding-3-large"),
  input: "a red guitar",
  dimensions: 1024,
});

Adapters for fixed-dimension models (for example mistral-embed or Ollama models) throw a clear runtime error when dimensions is set.

Multimodal Embeddings

Multimodal models embed images — alone, or fused with text into a single vector. Image inputs reuse the same content-part shapes as chat messages, and the accepted item types are narrowed per model at compile time: passing an image to a text-only model is a type error.

The top-level input array is always the list of items, and each item produces exactly one vector:

a string or text part embeds that text
an image part embeds that image
a nested array of parts ([textPart, imagePart]) fuses those parts into one vector — the same Array<ContentPart> shape a chat message's content uses

Because a top-level array is the item list, fuse by nesting: [textPart, imagePart] is two vectors, while [[textPart, imagePart]] is one fused vector.

import { embed } from "@tanstack/ai";
import { cohereEmbedding } from "@tanstack/ai-cohere";

const productPhoto = "iVBORw0KGgo..."; // base64 image data

const result = await embed({
  adapter: cohereEmbedding("embed-v4.0"),
  input: [
    "a red guitar",
    {
      type: "image",
      source: {
        type: "data",
        value: productPhoto,
        mimeType: "image/png",
      },
    },
    // A nested array fuses its parts into a single vector.
    [
      { type: "text", content: "Fender Stratocaster, sunburst finish" },
      {
        type: "image",
        source: {
          type: "data",
          value: productPhoto,
          mimeType: "image/png",
        },
      },
    ],
  ],
  modelOptions: { inputType: "search_document" },
});

console.log(result.embeddings.length); // 3 — one vector per input item

Amazon Titan Multimodal works the same way — nest the parts to fuse them into one vector:

import { embed } from "@tanstack/ai";
import { bedrockEmbedding } from "@tanstack/ai-bedrock";

const productPhoto = "iVBORw0KGgo..."; // base64 image data

const result = await embed({
  adapter: bedrockEmbedding("amazon.titan-embed-image-v1"),
  input: [
    [
      { type: "text", content: "a red guitar" },
      {
        type: "image",
        source: {
          type: "data",
          value: productPhoto,
          mimeType: "image/png",
        },
      },
    ],
  ],
  dimensions: 1024,
});

Adapters do not fetch remote image URLs by default — pass base64 data (or a data: URI). The Cohere adapter accepts an allowUrlFetch config option to opt into downloading http(s) image URLs on your behalf.

Search Documents vs. Queries

Retrieval-tuned models embed documents and queries differently. Cohere requires an inputType, which TanStack AI enforces at the type level — modelOptions is required for models with required options:

import { embed } from "@tanstack/ai";
import { cohereEmbedding } from "@tanstack/ai-cohere";

// Index time: embed documents
const docs = await embed({
  adapter: cohereEmbedding("embed-v4.0"),
  input: ["doc one", "doc two"],
  modelOptions: { inputType: "search_document" },
});

// Query time: embed the query
const query = await embed({
  adapter: cohereEmbedding("embed-v4.0"),
  input: "which doc mentions one?",
  modelOptions: { inputType: "search_query" },
});

Gemini expresses the same idea through an optional taskType:

import { embed } from "@tanstack/ai";
import { geminiEmbedding } from "@tanstack/ai-gemini";

const result = await embed({
  adapter: geminiEmbedding("gemini-embedding-001"),
  input: "a red guitar",
  modelOptions: { taskType: "RETRIEVAL_DOCUMENT" },
});

Usage and Observability

Adapters report token usage when the provider does, and embed() supports the same observe-only generation middleware as the media activities (see Generation Hooks):

import { embed } from "@tanstack/ai";
import { openaiEmbedding } from "@tanstack/ai-openai";

const result = await embed({
  adapter: openaiEmbedding("text-embedding-3-small"),
  input: ["a red guitar", "a blue drum kit"],
  middleware: [
    {
      name: "usage-logger",
      onUsage: (ctx, usage) => {
        console.log(`${ctx.model}: ${usage.promptTokens} tokens`);
      },
    },
  ],
});

console.log(result.usage?.promptTokens);

Error Handling

embed() rejects with the provider error; middleware onError hooks run before the rejection propagates:

import { embed } from "@tanstack/ai";
import { openaiEmbedding } from "@tanstack/ai-openai";

try {
  await embed({
    adapter: openaiEmbedding("text-embedding-3-small"),
    input: "a red guitar",
  });
} catch (error) {
  console.error("embedding failed", error);
}