Workers AI and Vectorize

Search for “launching a spacecraft” and find a document called Rockets, though they share no words. That’s semantic search, and it’s two Cloudflare products working together:

• Workers AI turns text into a vector: a list of numbers that captures what the text means. Texts that mean similar things get similar vectors.
• Vectorize keeps vectors in an index, and finds the ones closest to a vector you give it.

In forge-effect, each is a class:

example/search/resources.ts

import { Schema } from 'effect';

import { Ai, Vectorize } from '@forge-effect/infra';

export class Model extends Ai.Make<Model>()('ai') {}

/** BGE base's vectors: 768 dimensions, compared by angle. */
export class Docs extends Vectorize.Index<Docs>()('docs', {
  dimensions: 768,
  metric: 'cosine',
  metadata: Schema.Struct({ title: Schema.String }),
}) {}

This page walks through example/search, a small search service: put documents in, ask questions, get the closest three back.

The model and the index

Ai.Make declares the Worker’s Workers AI binding. There’s nothing to create in your account: the binding is the door to Cloudflare’s models.

Vectorize.Index declares an index, and three things about it:

• dimensions: how long each vector is. It must match the model that makes them. BGE base, the default embedding model here, makes vectors of 768.
• metric: how closeness is measured: 'cosine' (by angle, the usual choice for text), 'euclidean' or 'dot-product'.
• metadata: a schema for what you keep beside each vector. Here, each document’s title, so a search can say what it found.

ship makes the index with its dimensions and metric, named after the app as everything is (search-docs).

Index a document

PUT /docs/<id> with a title and some text embeds the text and keeps the vector:

example/search/search.ts

const Doc = Schema.Struct({ title: Schema.String, text: Schema.String });

const fetch = Effect.gen(function* () {
  const request = yield* Worker.Request;
  const url = new URL(request.url);
  const [, kind, id] = url.pathname.split('/');
  const ai = yield* Model;
  const docs = yield* Docs;
  if (kind === 'docs' && id !== undefined && request.method === 'PUT') {
    const doc = yield* Schema.decodeUnknownEffect(Doc)(yield* Effect.promise(() => request.json()));
    const [values] = yield* ai.embed([doc.text]); // one vector per text
    yield* docs.upsert([{ id, values: values!, metadata: { title: doc.title } }]);
    return new Response(null, { status: 204 });
  }
  // …

• ai.embed(texts) gives one vector for each text, from BGE base unless you name another model as its second argument.
• docs.upsert(vectors) adds each vector, or replaces the one with the same id. Its metadata is typed by the index’s schema: leave out title, or make it a number, and it doesn’t compile. (insert is the same, but keeps a vector already there.)

Search

GET /search?q=… embeds the question the same way, and asks the index for the three closest:

example/search/search.ts

  if (kind === 'search') {
    const [values] = yield* ai.embed([url.searchParams.get('q') ?? '']);
    const matches = yield* docs.query(values!, { topK: 3 });
    return Response.json(matches.map((m) => ({ id: m.id, title: Option.getOrNull(Option.map(m.metadata, (d) => d.title)), score: Math.round(m.score * 1000) / 1000 })));
  }
  return new Response('search: PUT /docs/<id> {title, text}; GET /search?q=…\n');
}).pipe(Effect.orDie);

export default Worker.make('search', { fetch }).bind(Model, Docs);

Each match has its id, its score (for cosine, higher is closer), and its metadata, decoded by the schema, as an Option: a vector written without metadata, or with metadata the schema doesn’t take, comes back as None rather than a surprise. query also takes a namespace, a filter on metadata, and returnValues.

That’s the whole app. Ship it with pnpm infra ship example/search/app.ts, put a few documents in, and wait a minute (more on that below). Live, “launching a spacecraft” found Rockets, “my kitten is sleeping” found Cats, and “fresh baked sourdough” found Bread, with no word in common with any of them.

Test it with a stand-in model

Neither product runs on your machine, so tests use stand-ins. Vectorize.memory(Docs) is an index in memory that compares every vector, and sees writes at once. Ai.memory(Model, run) answers each model call with what your function returns. Here, a toy embedding that counts words, so texts sharing words come out close:

test/search.test.ts

// A stand-in embedding: each word counted in one of 768 places, so texts
// sharing words are close.
const embed = (text: string) => {
  const v = Array<number>(768).fill(0);
  for (const word of text.toLowerCase().match(/[a-z]+/g) ?? []) {
    v[[...word].reduce((h, c) => (h * 31 + c.charCodeAt(0)) % 768, 7)]! += 1;
  }
  return v;
};

test('in memory: documents indexed by their text, and found by meaning, closest first, with their titles', async () => {
  const ai = Ai.memory(Model, (_, inputs) => ({ shape: [1, 768], data: (inputs as { text: string[] }).text.map(embed) }));
  const runtime = ManagedRuntime.make(Layer.mergeAll(ai, Vectorize.memory(Docs)));
  // … `call` runs the Worker with Worker.handle, as in the first app
  assert.equal((await call('PUT', '/docs/1', { title: 'Cats', text: 'cats purr and sleep in the sun' })).status, 204);
  assert.equal((await call('PUT', '/docs/2', { title: 'Rockets', text: 'rockets burn fuel to reach orbit' })).status, 204);
  assert.equal((await call('PUT', '/docs/3', { title: 'Dogs', text: 'dogs bark and sleep at night' })).status, 204);
  const found = JSON.parse((await call('GET', '/search?q=where+do+cats+sleep')).body) as Array<{ id: string; title: string }>;
  assert.deepEqual(found.map((f) => f.title), ['Cats', 'Dogs', 'Rockets']);
});

The stand-in doesn’t understand meaning, of course. It doesn’t need to: the test checks your handler’s plumbing (decode, embed, upsert, query, shape the answer), and the real model does the understanding on Cloudflare.

Other models: run

embed is a shortcut. For any other model, run takes the model’s name and its inputs, as Cloudflare documents them, and gives back its output. Give it a schema, and the output is decoded, typed:

ask.ts

const Answer = Schema.Struct({ response: Schema.String });

const fetch = Effect.gen(function* () {
  const request = yield* Worker.Request;
  const prompt = new URL(request.url).searchParams.get('q') ?? 'Say hello.';
  // The model's output, decoded by Answer: an AiError if it doesn't fit.
  const { response } = yield* (yield* Model).run('@cf/meta/llama-3.1-8b-instruct', { prompt }, Answer);
  return new Response(response);
}).pipe(Effect.orDie);

Without a schema, the output is unknown, and it’s up to you.

Good to know

Writes take about a minute to show up

Vectorize applies a write a little after upsert returns. Measured live, it took about a minute for the first three documents to be found. So a document you’ve just put won’t be in a search right away. The in-memory index sees writes at once, so tests won’t show this.

• infra dev leaves them unwired. Workers AI and Vectorize run only on Cloudflare. infra dev runs the rest of your app, and says so:

  $ pnpm infra dev example/search/app.ts
  example/search/app.ts, locally (state in .infra/dev, beside it):
    search           http://search.localhost:8787/
    and http://127.0.0.1:8787/ goes to search
    ! search: AI is Workers AI, which runs only on Cloudflare, so dev leaves it unset
    ! search: DOCS is the Vectorize index docs, which runs only on Cloudflare, so dev leaves it unset

  A call to either fails there. Test the logic with the stand-ins, and try the real thing in a preview environment (--env).

• An index’s shape is for good. Cloudflare can’t change an index’s dimensions or metric once it’s made. If one with your index’s name exists with other vectors, the ship stops before changing anything (IndexMismatch), saying what it found and what you declared. To change models, declare a new index under a new name.

• Wrong lengths never leave. A vector whose length isn’t the index’s dimensions is refused before it’s sent: “the vector x has 3 values, and the index takes 768”. The same goes for a query.

• Dimensions are a whole number from 1 to 1536, checked where you declare the index.

• Filters on metadata need a metadata index on Cloudflare for that property; the in-memory index matches equal values without one.

Next

• AI Search: the whole search, put together by Cloudflare: upload or crawl, then search and ask, with sources.
• Rate limits, analytics, browsers, images and more: the rest of Cloudflare’s bindings, the same way.
• Environments and state: a preview copy to try search against real models.