Abdul Samad Zeeshan
Email meEmail

Project 04 / 07 / shipped

Tests each new version of an AI agent on thousands of made-up customers, then says ship or hold.

Open the demo Repository

From the write-up

Proving lets you test an AI agent on thousands of made-up customers before it ever meets a real one. The customers have goals, details they only give when asked, and patience that runs out. The agent's tools are recorded or faked, so nothing real is touched. At the end you get one answer, ship or hold, with the numbers behind it. It is tried here on two agents from this portfolio: Warden, which approves changes to live software, and Parley, which books property viewings by phone in English and Gulf Arabic.

Headline numbers / 2

Hostile fake customers got the old Warden to make 1.3 unwanted server calls per conversation. The new one made 0.6. Verdict: ship.
ship
From the Proving README, results table, row Warden, resolver off to Warden as shipped.
Parley's new way of reading requests matched the old one on all 505 ordinary conversations. No gain, so the verdict was hold.
hold
From the Proving README, results table row Parley and the paragraph under the second table.

Decisions I made and things that broke

  1. A test passes only on what actually changed in the agent's systems, never on what the agent says it did.
  2. Judges that read only the transcript scored barely better than a coin, because a booking under the wrong phone number sounds like a success.
  3. Warden's server already refused every bad call, so its ship verdict rests on fewer bad calls sent, not on harm prevented.

What this proves / 5 skills

AI / ML

Python
The fake customers, the recorder between agent and tools, and the report are Python, with automated tests.
Model evaluation
Hostile fake customers pushed the old Warden into 1.3 unwanted server calls per conversation. The new version made 0.6, so it shipped.
Calibration
Three automatic judges were tuned on known answers, then checked against what really happened on 240 sampled calls.
Tool-using agents
It tests an AI agent on made-up customers, with its tools recorded or faked so nothing real is touched.

Infrastructure

Tests and CI
A run recorded with a local model replays in automated tests with no model at all.

The demo, recorded

Simulated customers running against two versions of an agent
Open the demo
Pick an agent and two versions, watch a grid of customers run, then read the report. The last button shows hostile customers and what they got past.

Papers / 3

  • arXiv 2609.00982, Disclosure-gated user simulationFake customers give a detail only when the agent asks for it, and some only when asked twice.
  • arXiv 2609.21841, EnterpriseValHow much the agent acts alone and how costly its mistakes are decide how much evidence a ship verdict needs.
  • arXiv 2609.22512, Agreement overstates evidenceIt showed that judges who agree are not independent, so Proving measures how much the three judges share their mistakes.