← Back
data-quality incident seeded

Embedding model swap leaves old vectors in a silently incompatible index

June 27, 2026

Symptom

The search endpoint stays healthy and returns five citations for every query, but answer quality drops after an embedding-model upgrade. Users report vaguely related passages, missed exact-policy hits, and support macros appearing above canonical docs; dashboards show no 500s because retrieval still returns plausible scored neighbors.

Root Cause

The corpus in Postgres `help_center_embeddings.embedding vector(1536)` was indexed with `text-embedding-ada-002` while the deploy changed query embedding to `text-embedding-3-large` with `dimensions=1536` so it could reuse the existing column and pgvector HNSW index. The dimensionality matched, but the geometry did not: cosine distance via pgvector `<=>` compared old ada points against new 3-large query points from a different embedding manifold. HNSW still returned k rows with normal-looking scores, so the failure surfaced only as recall collapse, not as an exception.

Diagnosis Steps

  1. Read one production trace and record the embedding model name used for the live query vector.
  2. Sample ten stored rows and inspect their `embedding_model`, `embedding_dimensions`, and `indexed_at` metadata beside the vector column.
  3. Run the fixed retrieval eval set with the old query model and then with the new query model against the unchanged index.
  4. Compare cosine-distance distributions for known-good query/document pairs before and after the swap.
  5. Re-embed a small shadow collection with the new model and verify whether Recall@5 returns before touching production aliases.
  6. Check deploy notes for a query-model flag change that lacked a matching index backfill or alias swap.

Fix

Backfilled the whole corpus into a shadow table with the same query model and metric, then atomically moved the read alias after validation: `python scripts/reindex_embeddings.py --source help_center_documents --target help_center_embeddings_v2 --model text-embedding-3-large --dimensions 1536 --metric cosine --batch-size 512`. The fixed eval moved Recall@5 from 31% on the mixed-space index back to 84%, slightly above the 82% baseline measured on the all-ada index.

Prevention

Treat the embedding model name, dimensions, normalization policy, and distance metric as index schema, and block deploys when the query embedder does not exactly match the corpus metadata for the active alias.

Stack

OpenAI text-embedding-ada-002 OpenAI text-embedding-3-large pgvector 0.7.4 Postgres 16 LlamaIndex 0.10.30

Tags

data-quality embeddings vector-search pgvector retrieval

Date

June 27, 2026

Recall@5 fell from 0.82 to 0.31 after the query embedder changed from text-embedding-ada-002 to text-embedding-3-large while the stored corpus vectors stayed untouched. Re-indexing the corpus with the same pinned model brought the eval back to 0.84.

query: "how long can enterprise admins restore a deleted workspace?"
query_model: text-embedding-3-large, dimensions=1536
index_model: text-embedding-ada-002, dimensions=1536
metric: cosine, pgvector operator <=>

before deploy, all ada:      Recall@5 0.82
after deploy, mixed index:   Recall@5 0.31
after full reindex, all v3:  Recall@5 0.84

The bad result was not random noise. It was the wrong kind of coherent. A query about workspace retention pulled audit-log setup pages because those documents lived near the new query vector only by accident inside an old coordinate system. Since text-embedding-3-large was requested at 1536 dimensions, the database shape stayed valid and the HNSW index stayed usable. No dimension mismatch, no failed insert, no noisy crash.

The proof was a two-column eval. Holding the corpus fixed and changing only the query embedder reproduced the drop immediately. Re-embedding just 2,000 documents into a shadow collection with text-embedding-3-large restored the nearest neighbors for the sampled queries, which ruled out prompt changes, reranking, and document freshness.

The production repair was built around a hard model-alignment guarantee: build help_center_embeddings_v2, verify recall and score distributions there, then move the read alias. From that point forward, changing the query embedding model without a matching corpus backfill became a deploy-blocking metadata mismatch, not an unreviewed runtime flag.