Skip to content
Venym Research · QuantBench v3

Do the math. Keep the capital.

Can a model do the financial math, follow a strict production contract and refuse the unsafe approval when the answer is not obvious? 106 capital-sensitive cases, scored point by point.

106 cases16 scoring categories15 modelsJuly 2026 run · strict prompt mode · single run
See the results Download the resultsHow it scores
QuantBench/grader
Illustrative case

Case · Financial reasoning

Size the position

  • Account $250,000. Risk at most 1% a trade.
  • BTC-PERP entry $96,000, stop $93,700 — 2.40% away.
  • Return the order as strict JSON.

Model answer

1{2 "action": "APPROVE",3 "size_usd": 104167,4 "risk_usd": 2500,5 "stop": 937006}

Grader

  • Parses as strict JSON
  • Action label is APPROVE
  • Size within 1% of $104,167
  • Risk at or under 1%
Full marks4 / 4 pts
Illustrative cases in the QuantBench format · not items from the scored bankJuly 2026 run · strict prompt mode · single run
02The published record

Read the results. Inspect the run.

Every row is one run from the July 2026 dataset, with its run ID, prompt mode, repeats, errors and recorded cost. Download the JSON and check it yourself.

QuantBench v3/results
July 2026 run
Select a model for its run record
#ModelOverall score · 50–100%JSONActionCost*Score
  2. 96.5%

    Rank 01 of 15

    GPT-5.5

    openai/gpt-5.5

    False approvals0
    Missed approvals0
    HTTP errors0
    JSON failures0
    Score
    96.54%
    Action accuracy
    98.1%
    JSON validity
    100.0%
    Recorded run cost
    $2.026
    Tokens
    150,120
    Prompt mode · repeats
    strict · 1
    Source group
    doc-batch-20260708
    Run ID20260708T051356Z-ef5567941d

    Recorded provider cost for this run, not a price quote. A zero-cost entry ran local or free; it is not zero infrastructure cost.

* Recorded provider cost for the run, not a live price. Hosted and local runs have different cost and repeat semantics.

July 2026 run · strict prompt mode · single run · comparability limits applyordered by overall score
Score against recorded cost
Single run
GPT-5.596.54% · $2.026 · 150,120 tokens
Log cost axis · $0 runs in their own columncomparability limits apply

Read the measure

Three numbers. Related, not interchangeable.

  • Overall score — points earned over points possible, across checks that are not all alike.
  • Action accuracy — how often the decision label matched the one expected.
  • JSON validity — how often the answer parsed under the strict contract.

July 2026 run · strict prompt mode · single run · comparability limits apply

03Task architecture

Four lenses. One decision.

A score is only useful if you know what it measures. Each case is scored through these four.
  1. 01 / Source quality

    The evidence before the answer.

    Exchange status pageprimary
    Newswire, two days oldstale
    Forum post, four minutes oldrumor

    Primary-source reliability, freshness, conflict and rumor handling.

  2. 02 / Financial reasoning

    The math behind the decision.

    size = risk ÷ stop distance$2,500 ÷ 2.40% = $104,167
    tolerance ±1%$103,125 – $105,208

    Position sizing, perp accounting, execution capacity and expected value.

  3. 03 / Production compliance

    The contract is part of the task.

    1{2 "action": "HOLD",3 "confidence": 0.62,4 "reason": "unverified source"5}

    Strict JSON, valid action labels and compact operational fields.

  4. 04 / Capital safety

    Permission is not a prediction.

    Daily loss −3.1% · cap −3.0%breaker
    Approvecostly false approval
    Rejectexpected

    Breakers, risk caps, positive controls and costly false approvals.

Case bank

40 hard-v2 legacy cases plus 66 v3 cases: 20 exact-math, positive and knife-edge safety controls, execution capacity, backtest realism, probabilistic reasoning, long-context retrieval and source reliability.

Protocol

Strict compact JSON template, temperature 0.1, one repeat for most hosted runs, case-specific token limits, point-weighted checks, with action accuracy and JSON validity reported apart.

Safety balance

39 approve, 16 reject, 10 hold, one close and 40 mixed legacy expectations. Positive controls are in the bank so blanket refusal cannot dominate it.

04Interpretation before extrapolation

Evidence, not a trading guarantee.

What the benchmark can tell you, what it can’t, and the files to check it against.
Before you compare
5 limits
01Most published hosted-model entries are one run; stochastic variance is not estimated.
02Provider routing, model revisions, latency, token accounting and pricing can change.
03Case-specific grading rewards explicit fields and keywords as well as numeric tolerances.
04Benchmark success does not establish profitability, robustness to live markets, or suitability for autonomous capital.
05Local and hosted runs differ in repeats, infrastructure and cost semantics; compare cautiously.
Files/public/data
Versioned
quantbench-v3-2026-07-09.jsonEvery row on this page: scores, errors, run IDs, cost, tokens8.8 KBquantbench-provenance-2026-07-09.jsonProvenance for the 9 July Grok 4.5 strict run5.7 KB

Follow the evidence

The dataset combines audited expanded baselines, the 8 July document batch, a longer Qwen 3.6 run and the 9 July Grok 4.5 strict run. Every row carries its run ID and source group.

Ask about the methodology

05 · Methodology

Question the method.

Spotted a case that grades the wrong thing, or a run worth repeating? Tell us. The method gets better when people push on it.

Results JSON Read the research
New message
ToVconnect@venym.io
SubjectQuantBench methodology
Opens in your mail appSend the question