We partnered with Parallel to evaluate its search plugin using NeoSigma to test how it performs inside the agents users actually run. On FinSearchComp T2, a set of 119 expert-annotated financial search questions from the public FinSearchComp benchmark, Parallel boosted GPT-6 Sol's accuracy from 41% to 76% on the Pi harness. But the same plugin can produce different results depending on which agent harness uses it. To understand that difference, NeoSigma tested Parallel across Pi, Codex, and Claude Code, measuring both how much the plugin improves accuracy and how well each harness uses it. Parallel delivered gains of 35–41 percentage points across all three, at just $5 per 1,000 calls.
One question from the benchmark shows the difference between harnesses:
“As of the end of December 2024, what was the bank credit of commercial banks in the United States? (seasonally adjusted, in billions of dollars)”
| Harness | Answer | Result |
|---|---|---|
| Codex (GPT-6 Sol) | $17,956 billion | Correct |
| Pi (GPT-6 Sol) | $17,963 billion | Wrong |
| Claude Code (Claude Sonnet 5.5) | $17,975 billion | Wrong |
Each agent made one Parallel search, and the results included several December 2024 figures from Federal Reserve data. Codex took the seasonally adjusted monthly value, which is the correct answer. Pi took a figure from a different table, and Claude Code used a weekly value from late December. Parallel returned the right number in all three harnesses, but only Codex picked it. This proves that better search alone doesn't guarantee better answers. How an agent interprets search results matters just as much as what it retrieves, and evaluations need to capture both.
Why evaluate your plugin on NeoSigma
A plugin's success depends on how agents actually use it. Testing the plugin without an agent shows how it responds to the inputs you chose, not how an agent will use it. In practice, your users' agents decide when to call your plugin, what inputs to send, and what to do with the results, and any of those steps can go wrong.
NeoSigma tests your tools inside live agents on different harnesses, connected through a plugin, to reveal its real impact before your users do. Our customers use the NeoSigma platform to answer questions like these before they ship new features:
Does the plugin help? NeoSigma recreates the same environment for each run, with the same agent harness, model, tasks and sandbox, once with your plugin and once without. The plugin is the only variable that changes, so any difference in results comes from the plugin.
How does it perform across agent harnesses? You can run the same tasks on different agent harnesses, such as Codex, Claude Code, and Pi and compare how well each one uses your plugin.
Does a new model still use it well? A harness tuned for an older model can perform worse with a newer one, which we call the scaffolding tax. Rerun your evals when a new model comes out to see whether performance drops, so you can update your plugin before your users notice.
Where does it break? Every tool call is traced and every answer is graded against a reference, so you can open the failed tasks and see exactly what went wrong.
Is it getting better? NeoSigma turns failures into new evals and automatically reruns them, so each fix to your plugin is checked and a regression is caught before your users find it.
What to test your plugin on
The best test cases come from how your users actually use your plugin. NeoSigma turns production traces into eval tasks, so you can test against real agent behavior. You can also import open-source benchmarks or task sets in one click through our Eval Hub, bringing real-world usage and standardized benchmarks together in one place.
How Parallel tested its plugin on NeoSigma
Parallel builds web search APIs for AI agents. We tested its Search tool, the same API behind Parallel's new finance solution.
Setup
Tasks: the 119 global questions of FinSearchComp's T2 task set, Simple Historical Lookup. Each asks for an exact past figure, such as a company's reported value at the end of a fiscal year, with answers taken from company filings, government sources and financial databases.
Agents: we ran Parallel through its MCP server on three harnesses: Pi and Codex with GPT-6 Sol, and Claude Code with Claude Sonnet 5.5.
Configurations: two experiments on each harness, with the same agent, model and tasks.
No plugin: the agent answers from its own knowledge.
Parallel Search: the agent makes one Parallel search with 10 results and answers from the results, adding its own knowledge when the results are incomplete. A trial that made more than one search call counts as wrong.
Grading: each answer was scored against the reference with FinSearchComp’s own grading prompt. Each question ran once per configuration.
Pricing: Parallel’s published list price of $5 per 1,000 search calls. Each question used at most one Parallel call.
Experiment Results
Running each harness with and without Parallel reveals two things: how much value the plugin adds, and how effectively each harness uses it.
Parallel delivered consistent gains across all three harnesses: +35 percentage points on Pi, +41 on Codex, and +40 on Claude Code. But their final accuracy differed. Codex led at 78%, followed by Pi at 76% and Claude Code at 71%. Claude Code also had a lower baseline without the plugin, suggesting the gap reflects differences in the underlying harness and model rather than Parallel itself.
Measuring accuracy alone doesn't show whether strong results come from the plugin or the environment running it. For Parallel, the takeaway is clear: the plugin delivers substantial gains across all three harnesses, while Codex offers users the highest overall accuracy on the FinSearchComp T2 benchmark.
Create a Leaderboard for Your Plugin Comparing Against Your Competitors
Your users compare your plugin with every tool that does the same job, inside the agents they already use. NeoSigma lets you run that comparison first. Connect your plugin and compare the performance with your competitors' plugins, and NeoSigma runs them all on the same tasks, rules, harnesses and models, so the plugin is the only difference. You see each plugin's accuracy and cost side by side, with every answer traced.
For example, we compared Parallel with two other search plugins, Exa and You.com Search, on the same 119 FinSearchComp T2 questions across Pi, Codex and Claude Code.
Parallel had the highest accuracy across all three harnesses, 75%, and led or tied for first in each one, at $5 per 1,000 calls. Every plugin added 28 to 41 points over no plugin, but the ranking shifted by harness. On Pi and Claude Code, Parallel and Exa tied. On Codex, Parallel was 11 points ahead of Exa and 10 points ahead of You.com. A single benchmark run on Pi or Claude would have called Parallel and Exa equal. Which plugin is best depends on the agent your users run.
Try it on your plugin
Your users will find out how your plugin performs inside their agents. With NeoSigma, you find out first. Running these experiments takes only a few steps: connect your plugin's MCP server, use public benchmarks or let NeoSigma create evals from your production traces, and run the same tasks with and without your plugin on the harnesses and models your users run. Request a demo to run your first plugin eval.