← Back
rag blocker seeded

ACL metadata drops the exact table row and RAG answers from near-misses

June 3, 2026

Symptom

Enterprise support agents ask for the contractual credit on a Priority 1 outage. The assistant cites the current SLA matrix and answers with a real percentage, but it is the public-plan credit, not the enterprise-only value agents see in Confluence.

Root Cause

The answer chunk existed in Chroma, but its metadata had `visibility: internal-enterprise` while the query filter used `$in: ["public", "internal"]`. That filter was applied before vector ranking, so the one row containing `Priority 1 | Enterprise Plus | 25% service credit` never entered the candidate set. The retriever still returned adjacent SLA rows from the same HTML export, and Cohere reranking favored the public row because it shared `Priority 1`, `outage`, and `service credit` with the query.

Diagnosis Steps

  1. Run `python scripts/trace_retrieval.py --query "Priority 1 outage enterprise credit" --tenant acme --show-metadata --k 10` and note that every result has `visibility=public` or `visibility=internal`.
  2. Query Chroma directly for the known document ID with no ACL filter and confirm the missing row appears with `visibility=internal-enterprise`.
  3. Search the source export with `rg "Priority 1.*Enterprise Plus.*25%" exports/confluence/sla-matrix.html` to prove the value is present in the indexed corpus.
  4. Inspect the retriever config and find the hard-coded filter `where={"visibility":{"$in":["public","internal"]}}`.
  5. Re-run the failing evaluation with `--acl-profile enterprise_plus` and verify top-5 answer-row recall rises from 58% to 94%.

Fix

Moved ACL construction out of the retriever default and into a tenant-aware policy resolver that maps entitlements to explicit visibility labels, including `internal-enterprise`. Added a startup assertion that refuses to serve an enterprise tenant if its ACL set lacks that label, then rebuilt the Chroma collection so old documents with malformed visibility metadata were rejected instead of silently searchable.

Prevention

Treat permission filters as retrieval logic, not plumbing: test recall under every ACL profile, log filtered candidate counts, and add a canary query whose only correct answer is enterprise-only content.

Stack

LlamaIndex 0.10.30 Chroma 0.4.24 Cohere rerank-english-v3.0 BeautifulSoup 4.12.3 OpenAI text-embedding-3-large

Tags

rag metadata permissions retrieval tables

Date

June 3, 2026

10% and 25% were both in the SLA matrix, but only one applied to an Acme Enterprise Plus Priority 1 outage.

The support agent asked:

What service credit applies to an Enterprise Plus Priority 1 outage?

The assistant answered 10%, cited sla-matrix.html, and linked to the right Confluence export. The number was not fabricated. It was the Priority 1 credit for the public Business plan. The Enterprise Plus row in the same table said 25%.

The retrieval trace was the first useful clue:

filter: {"visibility":{"$in":["public","internal"]}}
result[0] score=0.842 visibility=public text="Priority 1 | Business | 10% service credit | outage under 4h"
result[1] score=0.817 visibility=internal text="Priority 2 | Enterprise Plus | 10% service credit | outage under 8h"
filtered_candidates=37

There was no parsing failure in that trace. The model was simply never shown the answer. Chroma had the correct row, but it carried visibility=internal-enterprise, a label introduced when the Confluence space was split into public, internal, and account-scoped sections. The retriever default still allowed only public and internal, so the exact row was removed before similarity search and before reranking.

That ordering mattered. By the time rerank-english-v3.0 saw candidates, the best it could do was choose among near-misses: a public Priority 1 row with the right outage language, and an Enterprise Plus row for the wrong priority. The final answer combined the strongest visible match with the user’s plan name and sounded authoritative because the citation pointed to the real SLA export.

The fix was not to increase k or change the prompt. The ACL filter had to be treated as part of retrieval correctness. The tenant policy resolver now expands Acme’s profile to ["public", "internal", "internal-enterprise"], and the service logs both pre-filter and post-filter candidate counts. A regression query for Priority 1 Enterprise Plus service credit fails the build unless the 25% row appears in the top five retrieved chunks.