# Backfill & import

Import historical telemetry from Claude Code, Codex, Langfuse, Helicone, and raw OTLP dumps, and check which on-disk sessions are not yet ingested.

---

`tj backfill` imports historical telemetry from external sources into the local DuckDB, where the standard analyzers can read it. Every source maps onto the same internal `NormalizedSpan` schema, so once imported the data looks exactly like natively-captured telemetry. `tj cost`, `tj optimize`, and the Lens web UI treat it no differently.

The posture toward upstream tools is partnership. Keep whatever you use for live tracing, then point `tj backfill <source>` at the same data to run the local cost-optimization analyzers against it.

## Sources

| Source | Command |
|---|---|
| Claude Code (on-disk session JSONL) | `tj backfill claude-code` |
| Codex CLI (on-disk session logs) | `tj backfill codex` |
| Langfuse (live API or JSON dump) | `tj backfill langfuse` |
| Helicone (live API or JSON dump) | `tj backfill helicone` |
| Raw OTLP JSON (file or HTTP dump) | `tj backfill otlp` |

## Two input modes

The `claude-code` and `codex` adapters read a fixed on-disk path (each takes `--root` to override it). Every other adapter takes one of two modes.

**Live API.** Pass `--source-url` and an API key. The adapter calls the source's REST endpoint and follows pagination.

```bash
tj backfill langfuse --source-url https://cloud.langfuse.com --api-key lf_pk_... --since 30d
tj backfill helicone --source-url https://api.helicone.ai --api-key hc_pk_... --since 30d
```

**Local file.** Pass `--source-file` and a JSON or NDJSON dump. This is the right mode for testing, offline analysis, or scripted ingestion from a snapshot.

```bash
tj backfill langfuse --source-file ./langfuse-export.json
tj backfill helicone --source-file ./helicone-export.json
tj backfill otlp     --source-file ./traces.json
```

For Claude Code, no source flag is needed. The adapter parses `~/.claude/projects/*.jsonl` and runs automatically at the end of `tj init --claude-code`.

```bash
tj backfill claude-code
```

Codex is the same shape, reading `~/.codex/sessions/`:

```bash
tj backfill codex
tj backfill codex --since 30d
```

## Checking the gap

Claude Code prunes its own transcripts after roughly 30 days, so a session that never reached the database is on a clock. `tj backfill status` compares the on-disk sessions against what is ingested, by the transcript's internal session id rather than by filename, and lists what is still missing while the source files exist.

```bash
tj backfill status
tj backfill status --since 7d --limit 25
tj backfill status --json
```

## Time windows

`--since` filters ingested records by start time. It accepts the same syntax as `tj cost --since`: a duration like `30d` or `24h`, or an ISO-8601 timestamp.

```bash
tj backfill otlp --source-url https://example.com/traces.json --since 7d
```

## Idempotency

Re-running a backfill is a no-op. Every adapter derives a deterministic `span_id` from the source's own identifiers, and the spans table enforces a `PRIMARY KEY` on `span_id`, so rows that already exist are skipped rather than duplicated.

| Source | Deterministic `span_id` derived from |
|---|---|
| Claude Code | `(session_id, message_uuid)` |
| Langfuse | `(traceId, observation_id)` |
| Helicone | `(request_id)` |
| Raw OTLP | the payload's own `(trace_id, span_id)` |

Because of this you can schedule backfills nightly via cron without duplicating data, combine overlapping `--since` windows, and run a backfill alongside the live daemon. The two share one DB with no conflict as long as their source IDs differ, which they will.

## Plan tier on backfilled sessions

Backfilled sessions land with `plan_tier = 'unknown'` because the source data carries no plan-tier identifier. Run `tj init --reconfigure` to set your plan, and `tj optimize` will render dollar figures correctly for those sessions afterward.

## Notes per source

**Langfuse** reads observations from `/api/public/observations`. Self-hosted instances work by pointing `--source-url` at their base URL. Cost is taken from `calculatedTotalCost` when present, otherwise recomputed from `pricing/models.toml`.

**Helicone** POSTs `/v1/request/query` and maps one request record per span. `billing_account` comes from the record's provider field when present, otherwise it is derived from the model name.

**Raw OTLP** uses the same parser as the live `POST /api/v1/spans` route, so the receive path and the backfill path always agree on attribute extraction. It handles a `{"resourceSpans": [...]}` envelope or NDJSON of envelopes, and reports `spans_seen`, `spans_written`, `spans_skipped`, and `spans_rejected`. Note that this is for JSON dumps only, not live push. For a live collector, send OTLP directly to `tj serve` instead.