Benchmarking Jev for our OSINT agent
Benchmarking Jev (typesafe/jev-1.13), a fast typed classifier, as the tool router in a multi-step OSINT investigation agent and why whole-workflow benchmarks beat component benchmarks.
Jev is a new typed classifier, available as typesafe/jev-1.13 on OpenRouter. It does one pass and returns a structured label with a confidence score instead of generating text token by token, which suits routing decisions. We tested it as the tool classifier at the front of FindMore's OSINT investigation agent, where every question is routed before any tool runs. On the classify step in isolation, Jev is about ten times faster and, depending on the model it replaces, two to fifty times cheaper. Across ten full investigations that speedup was invisible in the total. This post covers what we measured, why the workflow benchmark is the one that matters, and what actually moves investigation quality.
The classify step
Every question the agent receives starts with one decision: what the question is about, and whether there is enough to start or the agent needs to ask for more before running anything. It is a classification, it runs on every turn, and it is cheap. Jev is built for exactly this shape of decision.
We benchmarked it against the general models we would otherwise use for the call: DeepSeek V4 Flash, GPT-5.6 Luna, and Claude Haiku 4.5. Jev was about ten times faster on the decision and two to fifty times cheaper, depending on which model it stood in for. It also held accuracy: it matched the ask-or-skip decision on every routing question and got the topic label right almost every time.
In isolation this is a clear win: cheaper, faster, essentially as accurate on the call. The isolation is the problem.
Benchmarking the whole workflow
A component benchmark measures a step, not what the user waits for. So we ran full investigations end to end, identical questions and evidence gathering and reports, changing only whether Jev made the first routing call.
The classify step is a rounding error in a full investigation. The run is dominated by gathering evidence and writing a cited report; the routing decision Jev accelerates is a fraction of a percent of it. Making that step ten times faster moves the total by less than run-to-run variance: we ran the same investigation on the same model twice, and the difference between runs was larger than the entire saving from the swap.
A component benchmark alone would have shipped a change that looked like a large win and moved nothing measurable.
Model quality diverges only on hard cases
With the classifier ruled out, the question was what does move the workflow. We changed the model running the investigation, GPT-5.6 Luna against Claude Haiku 4.5, and raised task difficulty.
On easy questions the two are indistinguishable: named subject, clear records, clean report. Easy questions do not separate models. The hard cases do: a person with a thin footprint and an easily confused name, or an organization behind a common word where the work is identifying which domain is even theirs.
On these, Luna sprawls. It wrote roughly twice as many claims as Haiku and cited fewer than half of them, and in one case it cited a step that never ran. The citation guard dropped the unsupported claims before they reached the report, but the model was generating unbacked material and the guard was the only thing keeping it honest.
Haiku is the opposite: fewer claims, every one cited, nothing invented. It costs several times more per investigation and is the model you would trust on a hard subject. The trade is not free. Haiku is conservative enough that it occasionally omits a real fact that Luna surfaces. That trade, precision against recall, is the one that matters for this product, and it is only visible on hard cases.
What we shipped
Jev stays, at the front door. Not because it makes an investigation faster, it does not, but because it makes the routing call for a negligible cost and returns a calibrated confidence the agent can act on.
The method is the takeaway. Benchmark the whole workflow, because the component you optimized is usually not the bottleneck, and measure quality on the cases that break, because easy cases make every model look equal right up until they do not.