Projects

A benchmark that turns your own commits into tasks for coding agents and grades them with the tests the real commit added, which the agent never sees. Run on 15 fixes merged into Warp's open-source repo, Warp's Oz agent said done in 30 of 30 runs and failed the hidden tests in 10.

Why: Built because an agent saying done is not evidence. I read all 10 failures by hand, then ran a blind Claude judge on the same 30 runs: it passed 5 of the 10 failures, but it caught a stray 1,972-line backup file the tests never looked at. Neither check was enough alone.

Jim Gray's 1985 process pair rebuilt on one laptop and killed 420 times to measure the checkpoint tradeoff he described. Checkpointing every request costs 25% of throughput, lost work is N/2, and the 1974 formula for the best interval lands 20x too high because it assumes lost work gets recomputed.

Why: Built as the first stop in rederiving fault tolerance from the original papers. Two things I did not expect: Gray's own fix, client replay, closes the lost-work hole for free, while a durable log costs 99% of throughput because fsync() on macOS does not wait for the disk. And my first takeover result was wrong: macOS AirPlay listens on port 5000 and was accepting the client's reconnects, which is the gray-failure problem in miniature.

A crash-durable task store for AI agents that guarantees a side effect happens exactly once, even under kill -9 at any point. 500 randomized crashes produced 0 double charges; a naive implementation under the identical harness produced 121.

Why: Built because the MCP Tasks spec defines a store interface and ships only an in-memory implementation, leaving persistence to implementers, and an agent that charges a card twice is a real failure rather than a theoretical one. Peak 5,202 tasks/s in Python and 7,345 in a Go port, both hitting the ceiling at 8 workers, which traced the wall to Postgres's commit path by elimination.

Semantic search over primary-source philosophy that classifies whether a passage argues for or against a claim, not merely whether it is about the topic. On a held-out set written after the fix, 20 of 21 queries surfaced the philosopher who actually holds the position.

Why: Built after finding that embedding search matches vocabulary and imagery rather than the conclusion an argument reaches, so a query about free will returns everyone who mentions it, on either side, undifferentiated. Fixed with HyDE query expansion over a from-scratch HNSW index, and validated on queries written afterwards so the result could not be tuned into existence.

A metrics ingest pipeline that survives the label explosion that OOM-kills a naive one, proven by exit code. Same 20M-sample workload in the same 256 MB container: the naive ingester dies at 8M samples; Peras completes all 20M at 1.67M samples/s with ~3 MB of heap.

Why: Built after measuring, while instrumenting Hapax, what one high-cardinality label costs: adding task_id took an export from 22 series to 30,019 and the payload up 1,488x. Peras is the ingest-side defense: it attributes a breach to the exploding label dimension at runtime and aggregates it away instead of crashing, and CI re-proves the die-vs-survive contrast on every push.