Jev vs. Qwen: Choosing a Reranker for Quality, Cost, and Speed

Search systems often retrieve a broad set of possible answers and then ask a second model to put the best ones first. That second model is called a reranker. It can make a large difference to what users see—but it also adds cost and latency to every search.

We compared two very different rerankers:

• Qwen3-Reranker-0.6B, a dedicated cross-encoder served by DeepInfra
• Jev (Noul) from TypeSafe, a “System One” model that returns calibrated probabilities rather than generating text

The practical result is simple:

Choose Jev when predictable user-facing latency matters most. Choose Qwen when cost at scale matters most. We did not find a stable enough quality difference to choose on ranking quality alone.

At a reranking depth of 100, Qwen cost about one quarter as much. Jev was about 1.8 times faster at the median and 2.4 times faster at the 95th percentile. Ranking quality was close enough that the apparent winner changed with the dataset and statistical threshold.

What does a reranker do?#

Imagine a librarian pulling 100 books that might answer your question. The first retrieval step is fast and broad: it tries not to miss anything useful. A reranker then reads your question alongside each candidate and rearranges the pile so the most relevant books sit on top.

That second pass matters because users usually inspect only the first few results. In a retrieval-augmented generation system, those top results also become the evidence given to the language model. Better ordering can therefore improve both search results and generated answers.

The tradeoff is that the reranker must score many query–passage pairs. A production choice turns on three questions:

1. Quality: Does it put relevant passages near the top?
2. Cost: What does scoring a candidate set cost at the expected traffic level?
3. Latency: How much time does it add, especially for slower tail requests?

The short answer#

Decision factor	Qwen3-Reranker-0.6B	Jev (Noul)	Practical winner
SciFact nDCG@10	0.7481	0.7699	Jev in this run
NFCorpus nDCG@10	0.3569	0.3623	No reliable difference
Cost per query, depth 100	$0.00045	$0.00178	Qwen, 3.9× cheaper
Median latency	1.11–1.22 s	0.64–0.65 s	Jev, about 1.8× faster
p95 latency	1.69–2.10 s	0.76–0.78 s	Jev, about 2.4× faster

The quality numbers need more care than the cost and latency numbers. Jev’s advantage was statistically significant on SciFact, but not on NFCorpus. One of three SciFact runs also missed the significance threshold. That is evidence of a small possible advantage, not a dependable universal win.

By contrast, the cost and latency gaps were large and repeated across both datasets.

How we made the comparison fair#

The benchmark used two public datasets from BEIR:

• SciFact: 300 scientific claims with binary relevance judgments
• NFCorpus: 323 natural-language queries with graded relevance judgments

BM25 first retrieved the same 100 candidates for every query. Those candidate lists were frozen before reranking, so both systems saw byte-identical passages in the same initial order.

Both rerankers then:

• scored all 100 candidates in one request;
• ran at four concurrent queries;
• returned one score per passage; and
• left final sorting to the shared benchmark runner.

This matters. Latency comparisons are misleading if one model receives a single packed request while another receives 100 separate requests. Ranking comparisons are also unreliable if systems see different candidate sets.

The BM25 baseline was checked against published Anserini results before the new models were compared. A benchmark that cannot reproduce a known baseline should not be trusted to measure a new one.

Result 1: ranking quality was close#

We used nDCG@10 as the main quality measure. In plain language, it rewards a system for placing relevant results near the top, with more credit for getting the order right in the first ten positions.

System	SciFact nDCG@10	NFCorpus nDCG@10
BM25, no reranking	0.6567	0.3051
Qwen3-Reranker-0.6B	0.7481	0.3569
Jev, generic relevance question	0.7699	0.3623

Both rerankers clearly improved on BM25. The harder question is whether Jev reliably beat Qwen.

On SciFact, the mean per-query difference was +0.0218 nDCG@10 for Jev. A 10,000-sample paired bootstrap produced a 95% confidence interval from +0.0044 to +0.0397, and a Wilcoxon signed-rank test gave p = 0.011. That run supports a real difference.

On NFCorpus, the difference was only +0.0054, the confidence interval crossed zero, and p = 0.53. That result does not distinguish the models.

Why one significant result is not enough#

We repeated the SciFact Jev run twice against fresh API responses:

Run	Jev nDCG@10	Difference vs. Qwen	95% confidence interval	Verdict
First	0.7699	+0.0218	+0.0044 to +0.0397	Significant
Repeat 2	0.7650	+0.0169	−0.0002 to +0.0346	Not significant
Repeat 3	0.7693	+0.0212	+0.0028 to +0.0407	Significant

The average effect remained positive and was larger than the run-to-run noise. Still, one ordinary repeat crossed the statistical threshold. The responsible interpretation is not “Jev always ranks better.” It is: quality is close, with a possible Jev advantage that should be tested on your own data.

Statistical significance is not a product requirement. If a small difference disappears on another dataset—or even on a routine repeat—it is a weak foundation for a production decision.

Result 2: Jev had the faster and tighter latency profile#

Median latency describes a typical request. p95 latency describes the threshold under which 95% of requests complete. For an interactive product, p95 is often more important because slow outliers shape how sluggish the system feels.

Dataset	Qwen p50 → p95	Jev p50 → p95
SciFact	1.22 s → 2.10 s	0.65 s → 0.78 s
NFCorpus	1.11 s → 1.69 s	0.64 s → 0.76 s

Jev was about 1.8 times faster at the median and 2.4 times faster at p95. Its tail was also far tighter: p95 sat only 0.12–0.13 seconds above the median. Qwen’s p95 was 0.58–0.88 seconds above its median.

That consistency is useful when a reranker sits inside a larger request budget. A system whose slower requests resemble its typical requests is easier to plan around than one with a long tail.

These are not universal latency constants. They were measured at four concurrent queries, and absolute values will move with providers, regions, payloads, and traffic. The important observation is that the relative gap appeared on both datasets under the same request shape.

Result 3: Qwen cost about four times less#

At list prices used during the experiment, a depth-100 query cost:

• Qwen: about $0.00045
• Jev: about $0.00178

That makes Qwen roughly 3.9 times cheaper. A fraction of a cent looks small until it is multiplied by production traffic. At one million reranked queries, the measured rates translate to roughly $450 for Qwen and $1,780 for Jev, assuming similar query and passage lengths.

Prices change, so those totals are a snapshot. The durable lesson is the ratio: if throughput cost is the binding constraint, Qwen offers similar measured quality at substantially lower spend.

Packing is what made Jev practical#

The documented Jev reranking pattern sends one request for every query–passage pair. At depth 100, that means 100 requests per search. With a 1,200-request-per-minute ceiling, throughput would be limited to roughly 12 search queries per minute.

Instead, we placed the query in shared state and sent one question per passage inside a single packed request. This changed the economics and latency dramatically:

• one request replaced 100 requests;
• the query state was charged once rather than 100 times;
• token use fell by 1.56 times; and
• top-10 scores were identical to the unpacked version, with Pearson correlation r = 1.000.

Without packing, the measured Jev p95 would have been roughly an order of magnitude larger.

Requests beyond Jev’s 64k limit were automatically divided in half and retried. The generic rubric never needed splitting in this benchmark. Because each passage question is scored independently, splitting an oversized pack does not change the passage scores.

A more detailed rubric did not improve quality#

Jev can be prompted with a plain yes/no relevance question or a more detailed scoring rubric. We tried three SciFact variants:

Jev rubric	nDCG@10	Difference vs. Qwen	p-value
Generic: “Is this passage relevant to the query?”	0.7699	+0.0218	0.011
Task-specific true/false criteria	0.7612	+0.0130	0.13
Four-level graded criteria	0.7585	+0.0103	0.32

The simplest question performed best. This does not prove that rubric engineering can never help: the custom rubrics were first drafts and were not selected on a held-out validation set. It does show that a longer, more specialized rubric is not automatically better.

Start with the generic relevance question. Add complexity only when an evaluation on representative data demonstrates an improvement.

Which reranker should you choose?#

Choose Jev when latency is the constraint#

Jev is the stronger default for an interactive search or chat experience with a strict response-time budget. In this test, its p95 stayed below 0.8 seconds and remained close to its median on both datasets.

Choose Qwen when cost is the constraint#

Qwen is the stronger default for high-volume or offline workloads where an extra fraction of a second is acceptable. It cost about one quarter as much, while its ranking quality could not be reliably distinguished across the full set of experiments.

Do not choose from these quality scores alone#

Public benchmarks are useful for eliminating bad options, not for proving which model will work best on private data. If ranking quality is the deciding factor, run a paired evaluation on your own queries, candidate passages, and relevance judgments.

Limitations#

This benchmark is deliberately small and auditable. That makes it easier to inspect, but it also limits what the results can establish.

1. Only two datasets were tested, and both are scientific. Neither resembles every production corpus, and both models may have seen BEIR data during training.
2. NFCorpus has many relevant documents per query. With an average of 38.2 relevant documents competing for a depth-100 candidate set, recall@10 is structurally capped; nDCG@10 is the useful comparison.
3. Latency was measured at four concurrent queries. Absolute numbers will change under a different load profile.
4. The custom rubrics were first drafts. They establish a sensible default, not a ceiling on rubric design.
5. Packing equivalence was verified at the top 10, not across all 100 positions.
6. Provider prices can change. Recalculate current spend before making a purchasing decision.

How to reproduce the benchmark#

We open-sourced the complete benchmark at github.com/denser-org/rerank-bench-jev. It includes the benchmark code and raw API responses, so you can inspect the methodology and recompute every published metric without an API key or additional model spend.

python3 -m venv .venv
./.venv/bin/pip install -r requirements.txt
./scripts/download_data.sh scifact nfcorpus

cd src
../.venv/bin/python compare.py \
  run_S1a_qwen_deepinfra_scifact_test_k100.json \
  run_S2_packed_jev_generic_scifact_test_k100.json \
  run_S0_bm25_scifact_test.json

The analysis code has 26 tests and makes no API calls:

./.venv/bin/python tests/test_analysis.py

Live API runs are cached by system, query, and candidate-set hash. An interrupted run can resume, and changing a candidate list invalidates only the affected cache entry.

Final takeaway#

The benchmark did not produce one universal winner. It produced a clearer engineering decision:

• Jev buys speed and a predictable tail.
• Qwen buys substantially lower cost.
• The measured quality difference is too fragile to be the deciding factor.

That is often the most useful outcome of a benchmark. Instead of asking which model is “best,” identify the constraint your product cannot violate, then choose the model whose advantage is large, repeatable, and relevant to that constraint.

Put reliable retrieval to work#

Denser AI turns your documents, knowledge base, and website content into an AI agent that answers with verifiable source citations. Create your Denser AI agent for free and test it with your own data.