← Back
cost incident seeded

Conversation replay turns a helpful chat app into a monthly token bill shock

June 18, 2026

Symptom

The support chatbot still answers correctly and latency only rises from about 1.8s to 3.4s, so no alert fires during the month. The first visible failure is finance flagging the OpenAI Usage dashboard: input tokens climbed from 180M to 9.6B while daily active users rose only 18%.

Root Cause

The chat service rebuilt each request by serializing the full prior transcript into `input` for every turn. A 30-turn conversation with ~1,900 new tokens per turn therefore sent ~1,900 tokens on turn 1, ~3,800 on turn 2, and ~57,000 by turn 30; total billed prompt input was proportional to 1+2+...+30, not 30. Prompt caching was ineffective because the app prepended a fresh timestamped system diagnostic before the transcript, changing the prefix each turn and causing the OpenAI prompt cache to miss.

Diagnosis Steps

  1. Open the OpenAI Usage dashboard and compare `input_tokens` against `output_tokens` by day for the affected project.
  2. Export one long conversation trace and count prompt tokens per turn with `tiktoken` instead of averaging per conversation.
  3. Plot turn index versus input tokens and verify the line grows roughly linearly inside each thread.
  4. Check whether the request prefix changes before the stable transcript and confirm prompt-cache hit rate is below 10%.
  5. Replay the same 30-turn transcript with summarization after turn 8 and compare total input tokens.

Fix

Changed the message builder to keep only the last 8 verbatim turns plus a rolling, model-generated conversation state capped at 1,200 tokens. Stable system instructions now sit at the very beginning of the request, and volatile diagnostics move to metadata outside the prompt. The service also records `input_tokens_per_turn` and `cache_read_input_tokens` so cost growth is visible before invoice close.

Prevention

Alert on input tokens per conversation turn, not just daily spend, and run a synthetic 40-turn chat in CI to fail any message-builder change that restores quadratic prompt growth.

Stack

OpenAI Responses API gpt-4o-2024-08-06 Node.js 20 Redis 7.2 Grafana 11

Tags

cost context-window prompt-caching token-accounting

Date

June 18, 2026

Cost Impact

At $5.00 per 1M gpt-4o input tokens, 9.6B monthly input tokens produced about $48,000 of input-token spend; the corrected summarizing path projected $3,400 for the same traffic.

invoice lineexpectedactual
conversations over 20 turns11,24012,980
average billed input per 30-turn thread57k tokens883k tokens
prompt-cache hit rate>70%6.4%
monthly input-token spend~$3.4k~$48k

Monthly input_tokens climbed from 180M to 9.6B while p95 latency moved from 1.8s to 3.4s, a change small enough to miss the latency alert. The cost graph was the only real alarm: input_tokens in the OpenAI Usage dashboard had turned into a curve while user traffic stayed almost flat.

One exported trace made the shape obvious. Turn 1 sent the system prompt and the first user message. Turn 2 sent those again, plus the assistant reply and the next user message. By turn 30, the request contained the whole conversation twenty-nine times in aggregate across the thread’s lifetime. A single new 1,900-token exchange was not the unit of billing; the accumulating transcript was.

The cache miss was self-inflicted. The request began with diagnostic_time=2026-06-18T14:03:22Z, then the stable instructions, then the transcript. Because the volatile timestamp came first, the common prefix changed every turn, so cached prompt reads almost never applied. Moving diagnostics out of the prompt helped, but the main fix was to stop replaying everything: keep the recent turns verbatim, summarize older commitments into a bounded state object, and watch token slope per turn as a production metric.