feat(telemetry): backfill historical token-savings economics (#2934)

* feat(telemetry): backfill historical token-savings economics

The 8-month historical_activity backfill rolled up observation counts but
never the token economics, so tokens_saved_vs_naive only existed for the
~6 days since context_injected went live (Jun 9). This adds the historical
counterpart using the SAME formula live uses:

  read_tokens           = sum(ceil(len(observations.text)/CHARS_PER_TOKEN_ESTIMATE))
  tokens_saved_vs_naive = max(0, discovery_tokens - read_tokens)  per UTC day

Both inputs are already persisted in SQLite (session_summaries.discovery_tokens
and observations.text), so the savings series now extends across the full
backfill window instead of starting at Jun 9. Conservative by construction
(once-per-observation read cost, not a replay of real injections) and flagged
backfilled:true.

Generation-side cost (cost_usd/tokens_input/tokens_output) is NOT recoverable
here — it was never written to SQLite — so it is intentionally excluded; see
plans/generation-cost-persistence.md for the forward-only fix.

- backfill.ts: read_tokens rollup + per-day savings derivation, CHARS_PER_TOKEN_ESTIMATE import
- scrub.ts: whitelist read_tokens
- tests: fixture gains observations.text; +2 cases (ceil math + floor-at-0)
- plans/: spec for the generation-cost persistence follow-up

https://claude.ai/code/session_01YWJPckEtd2sLAtng39rasu

* fix(telemetry): version the backfill marker so enriched rollup reaches existing installs

The historical backfill was gated solely on the marker file existing, so
every install that already backfilled under the prior version (#2912) would
hit the one-shot return and never receive the new read_tokens /
tokens_saved_vs_naive economics — the enriched series would only ever reach
fresh installs.

Add BACKFILL_VERSION to the marker. isBackfillComplete() now treats a marker
written by an older version (or a legacy marker with no version field, i.e.
version 1) as incomplete, so already-backfilled installs re-run and pick up
the enriched keys. The re-run is idempotent and does not double count: every
event keeps its deterministic per-(installId, event, day) uuid, so PostHog's
historical-migration dedup replaces each event in place rather than appending
a second row.

- backfill.ts: BACKFILL_VERSION constant, version on BackfillMarker, version-aware
  gate, stamp version at both marker-write sites
- tests: current-version marker skips; legacy (versionless) and older-version
  markers re-run and rewrite at the current version; assert version stamped

https://claude.ai/code/session_01YWJPckEtd2sLAtng39rasu

---------

Co-authored-by: Claude <noreply@anthropic.com>

plans/generation-cost-persistence.md

@@ -0,0 +1,97 @@
+# Spec: persist generation cost (the second telemetry gap)
+
+## Problem
+
+The growth/value story has two data layers with very different histories:
+
+| Layer | Event | Span | Recoverable historically? |
+|---|---|---|---|
+| Observation growth | `historical_activity` | Oct 2025 → now | n/a (already backfilled) |
+| **Injection value** (read-cost avoided) | `context_injected` | live, Jun 9+ | **Yes** — done in this PR via `backfill.ts` (read_tokens from `observations.text`, savings = discovery − read) |
+| **Generation cost** (what it cost to *produce* observations) | `session_compressed` | live, Jun 7+ | **No** — see below |
+
+`session_compressed` carries `tokens_input`, `tokens_output`, `cost_usd`, computed in
+`ResponseProcessor` from the SDK `result` message (Claude path) or
+`usage.cost` (OpenRouter). **These values are emitted to PostHog and then discarded —
+they are never written to SQLite.** `observations` has no token/cost column;
+`sdk_sessions` has none; `session_summaries` only has `discovery_tokens` (read cost,
+not generation cost).
+
+Consequence: there is no way to backfill generation cost for the ~8 months before
+`session_compressed` started firing. The data simply does not exist on disk. The
+cost/economics panel can therefore only ever be a *live, forward-looking* metric
+unless we start persisting it now.
+
+## Goal
+
+Persist per-compression generation cost so that:
+1. future backfills/audits can roll it up per day (same shape as the other rollups), and
+2. a "what it cost to produce vs what it saved" panel can be built from local data.
+
+This does **not** recover the past. It stops the bleed going forward.
+
+## Design
+
+A compression turn is not 1:1 with an observation row (one turn → N observations + 1
+summary), so generation cost belongs on a **per-turn** record, not smeared across
+observation rows (that was the `discovery_tokens` multi-count trap — see backfill.ts
+comments). Two viable homes:
+
+- **Option A (preferred): new `compression_events` table.** One row per
+  `session_compressed`, keyed by `memory_session_id` + turn. Columns:
+  `tokens_input INTEGER, tokens_output INTEGER, cost_usd REAL, model TEXT,
+  provider TEXT, outcome TEXT, created_at_epoch INTEGER`. Clean, append-only,
+  trivially rolled up. Mirrors the event we already emit.
+- **Option B: columns on `session_summaries`.** Cheaper migration, but summaries are
+  per-session-summary not per-turn, and `outcome='invalid_output'` turns have no
+  summary row — those costs would be lost. Rejected for that reason.
+
+Go with Option A.
+
+### Steps
+
+1. **Schema migration (SessionStore.ts, next version after 32):**
+   `ensureCompressionEventsTable()` — `CREATE TABLE IF NOT EXISTS compression_events (...)`
+   with index on `created_at_epoch DESC`. Best-effort, same pattern as the other
+   `ensure*` migrations.
+
+2. **Write path (ResponseProcessor.ts):** at every existing
+   `captureEvent('session_compressed', …)` site, also `INSERT INTO compression_events`
+   the same `tokens_input/tokens_output/cost_usd/model/provider/outcome`. The Claude
+   path stashes the event on `session.pendingCompressionEvent` and fires it from
+   `ClaudeProvider` once the `result` message lands — write to SQLite at that same
+   point so the token/cost fields are populated, not the early-stream placeholders.
+   Guard on a real cost (the abort/kill path ships without token fields — write the
+   row with NULLs rather than dropping it, to keep turn counts honest).
+
+3. **Backfill rollup (backfill.ts → collectDailyRollups):** add a block summing
+   `tokens_input`, `tokens_output`, `cost_usd` from `compression_events` per day into
+   counters `gen_tokens_input`, `gen_tokens_output`, `gen_cost_usd`. Wrapped in the
+   same try/catch (older installs without the table skip the block). This only
+   produces data for days on/after this ships — that is expected and correct.
+
+4. **Whitelist (scrub.ts):** add `gen_tokens_input`, `gen_tokens_output`,
+   `gen_cost_usd` to the backfill section. `cost_usd`/`tokens_input`/`tokens_output`
+   are already whitelisted for the live event; the `gen_*` names disambiguate the
+   per-day rollup from the per-event live values to avoid semantic collisions in
+   PostHog (same reasoning as keeping `read_tokens` distinct).
+
+5. **Docs (docs/public/telemetry.mdx):** document the new table and the three rollup
+   keys. Counts/sums only, no content — consistent with the existing privacy contract.
+
+6. **Tests (tests/telemetry/backfill.test.ts):** fixture row in `compression_events`,
+   assert per-day `gen_cost_usd` / `gen_tokens_*` sums; assert the block no-ops when
+   the table is absent.
+
+## Privacy
+
+`compression_events` stores integers, a float cost, and two closed-enum strings
+(`model`, `provider`) already shipped on the live event. No project names, no text,
+no prompts. Rollups remain counts/sums per UTC day. No new PII surface.
+
+## Out of scope
+
+Recovering pre-instrumentation cost. It was never written down; there is nothing to
+recover. The honest framing for any dashboard is: generation cost is measured from
+the date this ships forward; the observation-growth arc and (as of the sibling PR)
+the read-cost-savings series extend back to Oct 2025.

src/services/telemetry/backfill.ts

@@ -11,6 +11,7 @@ import {
   getOrCreateInstallId,
 } from './consent.js';
 import { scrubProperties } from './scrub.js';
+import { CHARS_PER_TOKEN_ESTIMATE } from '../context/types.js';
 import {
   getTelemetryApiKey,
   getTelemetryHost,
@@ -58,6 +59,23 @@ const BACKFILL_NAMESPACE = '8a9c2f4e-31b7-5d68-9c4a-f02e6d5b8a17';
 
 const BACKFILL_MARKER_FILENAME = 'backfill.json';
 
+/**
+ * Schema version of the backfill payload. Bump this whenever the rollup gains
+ * keys that already-backfilled installs must receive (a marker written by an
+ * older version re-runs so the enriched series reaches the existing base — not
+ * just fresh installs).
+ *
+ *   1 — original anonymized daily rollups (#2912).
+ *   2 — adds read_tokens / tokens_saved_vs_naive economics.
+ *
+ * A re-run is safe and does NOT double count: every event keeps its
+ * deterministic per-(installId, event, day) uuid, so PostHog's
+ * historical-migration dedup replaces each event in place with the enriched
+ * copy rather than appending a second row. Markers predating this field are
+ * treated as version 1.
+ */
+export const BACKFILL_VERSION = 2;
+
 /**
  * Mirror of the private STAT_TYPE_BUCKETS set in
  * src/services/context/ContextBuilder.ts — the closed observation-type
@@ -98,6 +116,8 @@ interface BackfillMarker {
   throughDay: string;
   eventCount: number;
   installId: string;
+  /** Schema version the marker was written at. Absent ⇒ legacy version 1. */
+  version: number;
 }
 
 function getBackfillMarkerPath(): string {
@@ -105,13 +125,24 @@ function getBackfillMarkerPath(): string {
 }
 
 /**
- * True when a completion marker exists. A corrupt marker file still counts as
- * complete: a marker was written at some point, and duplicate sends are worse
- * than a gap (PostHog data cannot be selectively deleted).
+ * True when a completion marker for the CURRENT schema version exists. A marker
+ * written by an older BACKFILL_VERSION counts as incomplete so already-
+ * backfilled installs re-run and pick up the enriched rollup keys — without it,
+ * the one-shot marker would pin them forever to whatever shipped when they
+ * first backfilled (the read_tokens / tokens_saved_vs_naive series would only
+ * ever reach fresh installs).
+ *
+ * A corrupt marker file still counts as complete: a marker was written at some
+ * point, and duplicate sends are worse than a gap (PostHog data cannot be
+ * selectively deleted). A marker missing the `version` field is a legacy
+ * version-1 marker.
  */
 function isBackfillComplete(): boolean {
   try {
-    return readJsonSafe<Partial<BackfillMarker> | null>(getBackfillMarkerPath(), null) !== null;
+    const marker = readJsonSafe<Partial<BackfillMarker> | null>(getBackfillMarkerPath(), null);
+    if (marker === null) return false;
+    const version = typeof marker.version === 'number' ? marker.version : 1;
+    return version >= BACKFILL_VERSION;
   } catch {
     return true;
   }
@@ -286,6 +317,42 @@ export function collectDailyRollups(
     // discovery_tokens arrives via migration — skip.
   }
 
+  // read_tokens / tokens_saved_vs_naive — the historical counterpart to live
+  // context_injected economics. Live telemetry derives savings as
+  // (discovery_tokens - read_tokens) where read_tokens is the size of the
+  // injected observation rendered into context. We cannot replay an actual
+  // injection for a past day, so we approximate per-day READ COST as the cost
+  // of reading every observation that day exactly once, using the SAME formula
+  // live uses (calculateObservationTokens / CHARS_PER_TOKEN_ESTIMATE) so the
+  // historical series is consistent with live numbers rather than a new metric.
+  //
+  // read_tokens         := ceil(len(text) / CHARS_PER_TOKEN_ESTIMATE) summed
+  // tokens_saved_vs_naive := discovery_tokens (rolled up above) - read_tokens,
+  //                          floored at 0 per day (a day can't have negative
+  //                          savings; clamping avoids a handful of summary-less
+  //                          legacy days dragging the series below zero).
+  //
+  // Caveat shipped to PostHog via backfilled:true: this is a once-per-observation
+  // lower bound on read cost, not a replay of real injections, so the historical
+  // savings curve is conservative relative to live (which re-injects context
+  // across many sessions). Generation-side cost (cost_usd / tokens_input /
+  // tokens_output from session_compressed) is NOT recoverable here — it was
+  // never persisted to SQLite — and is intentionally absent from the backfill.
+  try {
+    const f = frag('created_at_epoch');
+    const rows = db
+      .query(
+        `SELECT ${f.day} AS day,
+                COALESCE(SUM(CAST((LENGTH(text) + ${CHARS_PER_TOKEN_ESTIMATE} - 1) / ${CHARS_PER_TOKEN_ESTIMATE} AS INTEGER)), 0) AS read_tokens
+           FROM observations WHERE ${f.where} GROUP BY day`
+      )
+      .all(...params) as Array<{ day: string; read_tokens: number }>;
+    for (const row of rows) add(row.day, 'read_tokens', row.read_tokens);
+  } catch {
+    // observations.text missing on a partially-migrated install — skip; the
+    // savings derivation below simply won't fire for these days.
+  }
+
   // prompt_count — COUNT only; prompt_text is never selected.
   try {
     const f = frag('created_at_epoch');
@@ -319,6 +386,18 @@ export function collectDailyRollups(
     // Either table missing — skip.
   }
 
+  // Derive tokens_saved_vs_naive per day from the two rollups collected above,
+  // mirroring live's savings = discovery_tokens - read_tokens. Floored at 0:
+  // days with read activity but no summary rows (so discovery_tokens absent)
+  // would otherwise emit a negative, which is meaningless for a savings series.
+  // Only emitted when the day actually has a read_tokens figure, so days with
+  // no observations stay clean rather than reporting a spurious 0.
+  for (const counters of byDay.values()) {
+    if (counters.read_tokens === undefined) continue;
+    const discovery = counters.discovery_tokens ?? 0;
+    counters.tokens_saved_vs_naive = Math.max(0, discovery - counters.read_tokens);
+  }
+
   return Array.from(byDay.entries())
     .sort(([a], [b]) => (a < b ? -1 : a > b ? 1 : 0))
     .map(([day, counters]) => ({ day, counters }));
@@ -491,6 +570,7 @@ export async function runHistoricalBackfill(db: Database): Promise<void> {
         throughDay: lastFullDay,
         eventCount: 0,
         installId,
+        version: BACKFILL_VERSION,
       });
       return;
     }
@@ -539,6 +619,7 @@ export async function runHistoricalBackfill(db: Database): Promise<void> {
         throughDay: lastFullDay,
         eventCount: events.length,
         installId,
+        version: BACKFILL_VERSION,
       });
       logger.info('SYSTEM', 'Telemetry historical backfill complete', {
         eventCount: events.length,

src/services/telemetry/scrub.ts

@@ -121,6 +121,11 @@ export const ALLOWED_PROPERTY_KEYS: Set<string> = new Set([
   // session_count, and the obs_type_* family are shared with live
   // context_injected above (different per-event semantics — see PR notes).
   'discovery_tokens',
+  // read_tokens: per-day sum of once-each observation read cost
+  // (ceil(len(text)/CHARS_PER_TOKEN_ESTIMATE)); paired with discovery_tokens to
+  // derive the historical tokens_saved_vs_naive series. tokens_saved_vs_naive
+  // itself is whitelisted above (shared key name with live context_injected).
+  'read_tokens',
   'summary_count',
   'prompt_count',
   'project_count',

tests/telemetry/backfill.test.ts

@@ -16,6 +16,7 @@ import {
   buildBackfillEvents,
   runHistoricalBackfill,
   PROJECT_EPOCH_FLOOR,
+  BACKFILL_VERSION,
 } from '../../src/services/telemetry/backfill';
 import { scrubProperties } from '../../src/services/telemetry/scrub';
 import { __resetTelemetryForTests } from '../../src/services/telemetry/telemetry';
@@ -45,6 +46,7 @@ function makeDb(): Database {
       id INTEGER PRIMARY KEY AUTOINCREMENT,
       memory_session_id TEXT,
       project TEXT NOT NULL,
+      text TEXT NOT NULL DEFAULT '',
       type TEXT NOT NULL DEFAULT 'other',
       agent_type TEXT,
       created_at_epoch INTEGER NOT NULL,
@@ -88,11 +90,12 @@ type ObsRow = {
   type?: string;
   agentType?: string | null;
   tokens?: number;
+  text?: string;
 };
 function insertObs(db: Database, row: ObsRow): void {
   db.prepare(
-    'INSERT INTO observations (memory_session_id, project, type, agent_type, created_at_epoch, discovery_tokens) VALUES (?, ?, ?, ?, ?, ?)'
-  ).run(row.memId ?? null, row.project ?? 'alpha', row.type ?? 'other', row.agentType ?? null, row.epoch, row.tokens ?? 0);
+    'INSERT INTO observations (memory_session_id, project, text, type, agent_type, created_at_epoch, discovery_tokens) VALUES (?, ?, ?, ?, ?, ?, ?)'
+  ).run(row.memId ?? null, row.project ?? 'alpha', row.text ?? '', row.type ?? 'other', row.agentType ?? null, row.epoch, row.tokens ?? 0);
 }
 
 type SummaryRow = { epoch: number; project?: string; memId?: string | null; tokens?: number };
@@ -349,6 +352,36 @@ describe('buildBackfillEvents properties', () => {
     expect(rollups[0].counters.discovery_tokens).toBe(100);
   });
 
+  it('(f2) read_tokens uses ceil(len/4) and tokens_saved_vs_naive = discovery - read', () => {
+    const db = makeDb();
+    const epoch = Date.UTC(2025, 2, 4, 9, 0, 0); // 2025-03-04
+    // read cost mirrors live calculateObservationTokens: ceil(chars / 4).
+    insertObs(db, { epoch, memId: 's-1', text: 'x'.repeat(400) }); // ceil(400/4)=100
+    insertObs(db, { epoch: epoch + 1000, memId: 's-1', text: 'y'.repeat(401) }); // ceil(401/4)=101
+    insertObs(db, { epoch: epoch + 2000, memId: 's-1', text: 'z'.repeat(7) }); // ceil(7/4)=2
+    insertSummary(db, { epoch: epoch + 3000, memId: 's-1', tokens: 5000 });
+
+    const rollups = collectDailyRollups(db, LAST_FULL_DAY, '2024-01-01');
+    expect(rollups.length).toBe(1);
+    expect(rollups[0].counters.read_tokens).toBe(203); // 100 + 101 + 2
+    expect(rollups[0].counters.discovery_tokens).toBe(5000);
+    expect(rollups[0].counters.tokens_saved_vs_naive).toBe(4797); // 5000 - 203
+  });
+
+  it('(f3) tokens_saved_vs_naive floors at 0 when a day has reads but no summary', () => {
+    const db = makeDb();
+    const epoch = Date.UTC(2025, 2, 5, 9, 0, 0); // 2025-03-05
+    // Observations but no session_summaries row -> discovery_tokens absent.
+    // A naive (discovery - read) would go negative; the series must clamp to 0.
+    insertObs(db, { epoch, memId: 's-1', text: 'q'.repeat(4000) }); // 1000 read tokens
+
+    const rollups = collectDailyRollups(db, LAST_FULL_DAY, '2024-01-01');
+    expect(rollups.length).toBe(1);
+    expect(rollups[0].counters.read_tokens).toBe(1000);
+    expect(rollups[0].counters.discovery_tokens).toBeUndefined();
+    expect(rollups[0].counters.tokens_saved_vs_naive).toBe(0);
+  });
+
   it('(g) one session with one observation: no double counting', () => {
     const db = makeDb();
     const epoch = Date.UTC(2025, 3, 4, 14, 0, 0); // 2025-04-04
@@ -406,14 +439,50 @@ describe('runHistoricalBackfill gates', () => {
     expect(existsSync(markerPath())).toBe(false);
   });
 
-  it('(j) existing marker: returns before doing anything', async () => {
+  it('(j) current-version marker: returns before doing anything', async () => {
+    writeFileSync(
+      markerPath(),
+      JSON.stringify({
+        completedAt: 'x',
+        throughDay: '2026-01-01',
+        eventCount: 1,
+        installId: 'i',
+        version: BACKFILL_VERSION,
+      })
+    );
+    await runHistoricalBackfill(seedHistoricalDb());
+    expect(postHogConstructorCalls.length).toBe(0);
+    expect(postHogCaptureCalls.length).toBe(0);
+  });
+
+  it('(j) legacy marker without a version re-runs and rewrites at the current version', async () => {
+    // A version-1 marker (the #2912 rollup, no `version` field) must not pin an
+    // existing install to the old payload — the enriched economics rollup has
+    // to reach the already-backfilled base. Re-send is idempotent via the
+    // deterministic per-(installId, event, day) uuids.
     writeFileSync(
       markerPath(),
       JSON.stringify({ completedAt: 'x', throughDay: '2026-01-01', eventCount: 1, installId: 'i' })
     );
     await runHistoricalBackfill(seedHistoricalDb());
-    expect(postHogConstructorCalls.length).toBe(0);
-    expect(postHogCaptureCalls.length).toBe(0);
+    expect(postHogCaptureCalls.length).toBeGreaterThan(0);
+    expect(readMarker().version).toBe(BACKFILL_VERSION);
+  });
+
+  it('(j) older-version marker (< current) re-runs', async () => {
+    writeFileSync(
+      markerPath(),
+      JSON.stringify({
+        completedAt: 'x',
+        throughDay: '2026-01-01',
+        eventCount: 1,
+        installId: 'i',
+        version: BACKFILL_VERSION - 1,
+      })
+    );
+    await runHistoricalBackfill(seedHistoricalDb());
+    expect(postHogCaptureCalls.length).toBeGreaterThan(0);
+    expect(readMarker().version).toBe(BACKFILL_VERSION);
   });
 
   it('(j) debug mode: stderr dry-run, no send, no marker', async () => {
@@ -445,6 +514,7 @@ describe('runHistoricalBackfill gates', () => {
     expect(postHogCaptureCalls.length).toBe(0);
     expect(existsSync(markerPath())).toBe(true);
     expect(readMarker().eventCount).toBe(0);
+    expect(readMarker().version).toBe(BACKFILL_VERSION);
   });
 
   it('(j) second invocation after success sends nothing', async () => {
@@ -485,6 +555,7 @@ describe('runHistoricalBackfill gates', () => {
     expect([expectedThroughBefore, expectedThroughAfter]).toContain(marker.throughDay as string);
     expect(marker.eventCount).toBe(2);
     expect(marker.installId).toBe(postHogCaptureCalls[0].distinctId as string);
+    expect(marker.version).toBe(BACKFILL_VERSION);
   });
 
   it('(k) an emitted client error prevents the marker (retry on next start)', async () => {