PatchProof · open source
Stop silent API outages before they reach production.
When a provider changes their API, the repair that looks correct is usually the one that quietly loses money. PatchProof watches third-party contracts, refuses automated fixes that would silently drop a capability, and turns everything it cannot resolve into one precise question.
The failure nobody is watching for
Every safeguard you already have answers one question: did the request fail? Contract tests, monitoring, error budgets, alerting — all of it keys off something going wrong loudly.
The expensive failures are the quiet ones. A provider removes the field you
were sending. Your request still validates, still returns 200 OK,
and has silently stopped applying the discount. Nothing errors. Nothing alerts.
You find out from the numbers, weeks later.
The obvious repair — stop sending whatever broke — applied to 71 real breaking changes from Stripe's published history:
Same fix, same cases. The only difference is whether something verified it first.
Three things it does
Detects drift early
Reads a provider's published OpenAPI contract and diffs it against the one you integrated against — including query parameters, nested fields and enum values, which is where half of real breaking changes hide.
Refuses unsafe repairs
A proposed fix has to survive verification before it becomes behaviour. The check that matters compares what your request used to accomplish against what it accomplishes now, and blocks anything that quietly stopped doing its job.
Asks, instead of guessing
When the contract holds no answer, it produces one specific question naming the field and the operation — not a ticket saying something broke, and not a confident guess that validates and is wrong.
How it works
Stripe removes coupon from POST /v1/customers.
No version bump. Your request still validates.
The obvious repair is to stop sending it. The gate sees the capability is gone and refuses to adopt it.
A single prompt naming exactly what it does not know, answerable in seconds.
'coupon' no longer appears in the request and its value is not carried by any other field. The published contract does not say where it moved. Was this capability removed, or does it now go somewhere else in POST /v1/customers?
Try it on a change of your own
Paste two versions of an OpenAPI document — one operation or a whole
spec — and optionally the request you actually send. A spec with several
operations is scanned whole and ranked by how much broke, since the one that
changed is rarely the first one. You get two answers side by side: what an
auto-fixer with no gate would ship, and what PatchProof does. Copy a link to
this puts the whole case in the URL fragment, so it travels into an issue or
a chat without ever reaching a server. This runs the real engine — a port of
the Python in sell/real/, held to it by a conformance test that puts
the same 71 benchmark cases through both and fails if they disagree on the
signals, the proposals, the verdict, the question, or either answer below.
Nothing is sent anywhere; it runs in this page.
Leave this blank and PatchProof invents a request using tracer values, which is enough to tell whether a repair keeps expressing them. Paste your real one and the questions it asks quote your actual values.
Or step through the 71 real cases
The table further down is measured on 71 breaking changes taken from Stripe's
published history. Rather than ask you to take those numbers on trust, this
loads the same case file the benchmark runs on and recomputes both rows in your
browser — every case, scored by a port of bench/score.py that
the conformance test holds to the Python. You can then open any single case and
see what each solver did with it.
Why ungated auto-fixing is dangerous
Every number in this section can be recomputed in your browser, case by case, from the case browser above.
Every row below is measured on the same 71 real breaking changes, not estimated. “Unsafe” means a repair was adopted that is invalid, silently dropped a capability, or guessed where the contract held no answer.
| On 71 real breaking changes | Ungated auto-fix | Through the gate |
|---|---|---|
| Naive repair — drop the broken field | 71 unsafe | 0 unsafe |
| Pattern-matching agent | 58 unsafe | 2 unsafe |
| Ambiguous cases raised as a question | 0 of 44 | 44 of 44 |
| Repairs correctly applied | 3 of 27 | 7 of 27 |
The gate is the difference, not the cleverness of the agent. A model-backed reasoner adapts nearly twice as often — 13 of 27 against 7 — and produces three and a half times the unsafe adoptions, 7 against 2. That is exactly why it ships behind the same gate rather than in front of it.
Full benchmark results, and how the cases were built
71 cases mined from nine windows of Stripe's published history, Nov 2022 to Aug 2026. Labels come from the shape of each change, not from judgement. Two properties are enforced when mining and re-checked by the test suite: each request must be valid before the change and broken after it.
Adapt is being correct where the contract holds the answer (27 cases). Ask is correctly raising a question where it does not (44 cases). They are reported separately because a solver that never adapts scores full marks on the second.
| Solver | Adapt | Ask | Unsafe |
|---|---|---|---|
| do nothing | 0/27 | 0/44 | 0 |
| always ask | 0/27 | 44/44 | 0 |
| drop the field, no gate | 0/27 | 0/44 | 71 |
| drop the field, gated | 0/27 | 44/44 | 0 |
| agent, no gate | 3/27 | 0/44 | 58 |
| agent, gated | 7/27 | 35/44 | 2 |
| agent + model, gated * | 13/27 | 38/44 | 7 |
* The model answered 68 of the 71 cases. Three fell back to the heuristic after the free tier's rate limiting, across 144 waits. Those three flatter this row rather than harm it — the heuristic is the safer of the two solvers — so 7 unsafe is a floor, not a ceiling.
Where it succeeds and fails, by kind of change — fully measured run, so the figures are clean. The two smallest slices are unlabelled rather than cramming a digit into 17 pixels; the table carries every number.
| Kind | n | Solved | Asked | No answer | Unsafe |
|---|---|---|---|---|---|
| removed | 38 | 34 | 0 | 2 | 2 |
| rename | 18 | 5 | 8 | 5 | 0 |
| type changed | 5 | 0 | 1 | 4 | 0 |
| enum narrowed | 5 | 2 | 0 | 3 | 0 |
| newly required | 4 | 2 | 0 | 2 | 0 |
| length tightened | 1 | 0 | 0 | 1 | 0 |
44 of the 71 cases carry no evidence anywhere of where the capability went, so the realistic ceiling for unattended repair against this provider is about 38%. The rest is the agent asking one good question instead of an engineer reading a changelog.
Where PatchProof is today
Working now
- Drift detection Reads any OpenAPI 3 contract. Benchmarked against 2,506 published versions of Stripe's API. live
- Schema verification Validates a proposed repair against the provider's own published contract. live
- Capability check Blocks repairs that stop the request doing what it did. live
- Question generation One precise prompt per unresolvable change. live
- Benchmark 71 labelled real breaking changes; scores any solver. live
Built, not yet proven
- Sandbox verification Sending a candidate repair to a provider's test environment. Code and tests exist; it has never been run against a live provider. needs credentials
- Downstream reconciliation Catching a repair that validated and was still wrong, from what happened later. Exercised against a simulator only. simulated
Not built
- GitHub Action / PR bot Blocking a breaking change during code review. planned
- Continuous monitoring Watching a vendor's contracts in the background. planned
- Slack & Jira triage Delivering the question where the team already works. planned
- MCP guardrail Running as a safety gate behind AI coding tools. planned
This section exists because the product's entire argument is that things which merely look correct are the dangerous ones. A roadmap presented as a feature list would be the same mistake PatchProof is built to catch.
Questions
Does it change my code automatically?
No. PatchProof decides whether a proposed repair is safe; it does not deploy one. A verified repair is a change you review, and an unresolvable one becomes a question rather than a guess.
How does it stop a model from inventing a fix?
Nothing a model proposes is trusted. A proposal is a hypothesis, and it has to pass the provider's own published contract and the capability check before it counts as a repair. Two further tiers exist — the provider's sandbox, and what actually happened downstream — and their status is stated above rather than implied.
Which providers does it work with?
Any provider publishing an OpenAPI 3 contract. The benchmark is built from Stripe's version history because Stripe publishes 2,506 tagged versions publicly, which makes the drift real and checkable. A GitHub preset ships too. Other providers should work and have not been benchmarked — that is a claim worth testing rather than printing.
Does my data leave my machine?
No. The default reasoner is offline pattern matching with no dependencies and no telemetry. Contracts are fetched from providers' public specs. Only if you explicitly enable a model tier does anything go to that provider, and you choose which.
How long does it take to try?
Clone it and run one command. No account, no install, no key.
git clone https://github.com/AkshitRampershad/PatchProof && cd PatchProof python3 -m bench.run # score it on 71 real breaking changes python3 run_demo.py # watch it adapt, no API key needed