ttdbg

The bug that started it

The thing that finally pushed me to build this was a genuinely stupid bug. A dict was showing up with values I never put in it, three function calls away from where I was actually looking, and every time I added a print() to catch it, the bug politely rearranged itself to happen one call earlier. Classic. I must have hit “restart debugger” forty times that night, each time nudging a breakpoint back one more frame, and at some point I stopped debugging the dict and started being angry at the debugger itself. Why am I the one doing the stepping-backwards? The computer already ran the program. It knows exactly what happened. I just want to ask it.

That question is ttdbg: run your program once while it’s being recorded, and afterwards explore that recording like a video. Step backwards as well as forwards. Scrub a timeline. Click a variable and ask “when did you last change?” and get teleported straight to that moment. Nothing re-executes. History is just data now, and you’re querying it.

Why this hasn’t existed already

The annoying part of having this idea is that it’s obviously not a new one. Time-travel debugging is old news for C and C++ — rr and Undo are genuinely loved tools. People have tried to bring it to Python before: birdseye and Cyberbrain both got close, recording execution and letting you backtrace a variable’s history. Both are effectively dormant now. There was even a commercial attempt at this exact pitch around 2020 (PyTrace) that’s just gone.

They all died on the same rock: sys.settrace, the only hook Python gave you for watching a program execute line by line, is catastrophically slow — a 10 to 50x slowdown isn’t unusual. Record a real program with it and you’ve built something nobody can actually use.

What changed is Python 3.12. It shipped sys.monitoring (PEP 669), a set of low-overhead interpreter event hooks built for exactly this kind of tool. So the honest pitch for ttdbg isn’t “nobody’s thought of this” — it’s “everyone’s thought of this, and the rock just moved.” ttdbg is built on sys.monitoring and nothing else. The recorder is pure stdlib, on purpose — no C extension, no Rust, pip-installable everywhere.

What it actually does

Three commands cover the whole workflow:

pip install -e .

ttd run examples/fib.py        # run once, recorded → writes ./trace.ttd
ttd view trace.ttd             # explore it in the browser
ttd dump trace.ttd --limit 40  # or as a terminal timeline

ttd view opens a local web viewer: a source pane with a playhead line sitting on the current line of code, a timeline scrubber along the bottom shaped like a call-depth skyline, and a variables panel showing state at whatever moment you’re looking at. You step back (←) and forward (→) one event at a time, step over a call (Ctrl+←/→), step out (U), or jump straight to the next exception (E). The move that actually sold me on building this in the first place: click any variable, and the playhead jumps to the exact event where it was last written. Click again and it walks further back through that variable’s whole history. That forty-times-restarting-the-debugger night becomes one click.

Recording is scoped in both space and time, because recording literally everything, forever, isn’t the point:

import ttdbg

with ttdbg.record("trace.ttd"):     # scope recording to a region
    suspicious_function()

@ttdbg.record                        # or record every call to one function
def flaky_thing(): ...

ttd run records the whole script; by default it only records files under your own project, never the standard library or anything in site-packages, and you can widen or narrow that with --include / --exclude.

How it’s built

Under the hood there are three deliberately separate layers, and I mean deliberately — I wrote this as a solo project and wanted every layer independently demoable, so a stall in one wouldn’t block the others.

• The recorder subscribes to sys.monitoring events (LINE, PY_START, PY_RETURN, RAISE, and friends) and, at each line, diffs the frame’s locals against the last snapshot for that frame. Only the names that changed get written. Values are stored as safe, truncated reprs with a strict node budget, never a real object graph, because “record everything faithfully” and “stay fast” are directly opposed goals here and I had to keep picking one.
• The trace store is a single SQLite file (.ttd). The event id is the clock — there’s no wall-clock time anywhere in the UI, just a monotonic counter. Every headline feature collapses into one query: “the value of x at event 40,000” is just the latest write to x with event_id <= 40000. “When did x last change?” is the same query, jumped to. I like when a feature that sounds clever turns out to be one indexed SQL query.
• The web viewer is stdlib http.server plus vanilla JS. No build step, no framework. Slower to write some of the UI plumbing by hand, but I didn’t want a debugging tool to also come with its own npm dependency tree.
  

The fight was entirely about overhead

Everything above is the easy version of the story. The real work — the thing that ate most of my time — was that the very first working recorder had a 247x overhead on a mixed workload of JSON parsing, regex, and sorting. That number is a joke. A debugger nobody can afford to run is a debugger nobody runs.

The absolute worst case — a tight, pure-Python arithmetic loop where literally every executed line fires an event — still sits around 180x today, even after everything below. It’s a known, documented limit, not a bug I haven’t found yet.

Getting from 247x down to 48x on the realistic workload took a handful of specific fixes, and each one taught me something I didn’t expect going in:

• A background thread now owns SQLite, so writes release the GIL instead of stalling the interpreter that’s actually running your program.
• I replaced reprlib with a hand-rolled, node-budgeted value renderer. Rendering a nested container went from ~19 microseconds to ~2. reprlib does a lot of work you don’t need when all you actually want is “roughly what this looked like, truncated.”
• The diffing itself got smarter: instead of re-diffing every local on every line, I added per-line bytecode analysis that figures out, statically, which names a given line could even touch, and only diffs those. After any function call, it falls back to a full diff, because a call can mutate things through an alias in ways the line’s own bytecode wouldn’t show — that fallback is deliberately “fail open,” because a partial diff that’s wrong is worse than a slower diff that’s honest.

That alias case is the one that actually worried me. If a mutation happens through a second name pointing at the same object, and my diff only looks at the name the current line touches, I could silently miss a change and show you a lie. So there’s a differential test that runs the partial-diff path and the full-diff path against the same programs and asserts they produce identical per-variable histories. I’d rather ship a slower correct recorder than a fast one I don’t fully trust.

The bug that almost shipped

The one I’m least proud of, and most glad I caught: my first version of “jump to when this variable last changed” technically worked in every test I threw at it by hand, and was still wrong.

The trace store doesn’t literally store every value forever — it stores writes, and reconstructs state by querying backwards. Early on, the query for “the value of x right now” and the query for “when did x last change” were subtly different code paths that happened to agree on every example I tried, until I generated a batch of longer traces and found one where they didn’t. It came down to how I was handling the boundary event — the exact event where a value was written versus the event right after. Off-by-one bugs are humbling in a normal program; in a time-travel debugger, an off-by-one bug means the tool built to show you what actually happened is lying to you about what actually happened. I stopped and wrote both queries against the same underlying index so they’re mechanically incapable of disagreeing, instead of trusting that two hand-written versions would stay in sync as I kept touching the code.

Honest limitations, on purpose

I’d rather list what doesn’t work than have someone find out the hard way:

• CPython 3.12+ only, no fallback path — this project only exists because sys.monitoring exists.
• C extensions are opaque. Something happening inside numpy’s internals is invisible to the hooks; you see values before and after the call, never during.
• History is bounded by design. Values are truncated reprs, not full object graphs, so a mutation buried too deep in a container might not show up on that container’s own recorded value — though it’s still visible on whatever closer local variable passed through it.
• Threads, async, and generators aren’t first-class timelines yet. Events from every thread get recorded and tagged, but the viewer’s mental model is still single-threaded.
  

Where it’s at

MVP is done: the recorder, ttd dump, the full web viewer with timeline scrubbing, backwards stepping, last-changed jumps, jump-to-exception, and multi-file program support. Next up is a pytest plugin — auto-record a failing test and hand someone a time-travel recording of exactly how it failed, which I think is the actual path to anyone besides me using this — then a DAP adapter so the backwards-stepping works inside VS Code’s native debugger UI too.

Chasing that overhead number down was more fun than I expected going in, and there’s clearly more to squeeze. But even at today’s numbers, the thing already does what I wanted at 2am fighting that dict: I can run a program once, then just ask it what happened, instead of guessing where to put the next breakpoint.

Code’s on GitHub if you want to point it at something of your own.