4 min read

Same model. A third fewer steps.

The same open-weight model on 30 LAB firm-knowledge tasks used about a third fewer steps with Infona. Accuracy showed no detectable change.

Infona builds one knowledge graph from all of a firm's documents, ahead of time, and gives it to any AI agent as a small set of read-only tools. The bet is that a map of the firm cuts the wandering: fewer file reads, fewer turns, less time, without asking the model to get smarter. We tested that bet on Harvey's open-source Legal Agent Bench, a public set of firm-knowledge tasks in a synthetic firm of 266 matters, 9,288 files and about 108 million tokens of text. Infona is not affiliated with Harvey. LAB and Harvey Graph are Harvey's names for its benchmark and product.

Same model. About a third fewer steps. No detectable change in accuracy.

What we found

We ran the same agent, on the same open-weight model, on 30 firm-knowledge tasks. The baseline worked from the firm's files alone. The Infona arm got a graph built from those same files, four read-only tools, a search index over a newer document parse, and a short prompt paragraph that named the tools. So the comparison is the graph plus search plus that paragraph, against none of them. It is not the graph alone. The sample was drawn at random and the analysis plan was written down before any task ran. Each task ran once per arm.

The Infona arm was leaner. Totals across all 30 tasks: turns 34% lower, input tokens 33% lower, and agent inference cost 29% lower. Wall time was 33% lower on the 29 tasks that finished. Infona used fewer turns on 26 of 30 tasks.

34%

fewer turns

Totals, all 30 tasks

33%

fewer input tokens

Totals; 44% on a typical task

33%

less wall time

Totals, 29 finished tasks

26 / 30

tasks with fewer turns

Input tokens and cost, same split

Efficiency, totals (baseline = 100)

Turns10066 · 34% fewerInput tokens10067 · 33% fewerWall time10067 · 33% fewerBaselineInfona
Totals across the run, indexed to the baseline. Wall time covers the 29 tasks that finished. Input tokens were 33% lower in totals and 44% lower on a typical task.

Those percentages are totals: what the whole run cost. On a typical task the cuts were similar or larger. Turns were 35% lower, input tokens 44% lower, cost 35% lower, and wall time 32% lower. Totals and per-task averages differ because the size of the cut varies across tasks, and the largest tasks dominate the totals.

Accuracy showed no detectable change. The primary figure counts every task, with the one crashed run scored as zero. The mean-of-judges criterion pass rate was 84.8% without the graph and 82.6% with it. That is 2.1 points lower, with a 95% interval of -14.3 to +5.7. The interval includes zero and is wide, because a single crashed run on a large task dominates it. The study was too small to show accuracy is unchanged.

Accuracy difference (Infona minus baseline)

-15-10-50+5-2.1 pts
Primary accuracy difference, all 30 tasks, with the crashed run scored as zero. The interval includes zero. The study was too small to show accuracy is unchanged.

Harvey has published its own graph results on LAB. See Harvey, "Improving Agent Accuracy With Harvey Graph". Models, metrics and task sets differ, so these numbers are not a comparison.

Caveats

  • Accuracy showed no detectable difference. The interval is wide and includes zero.
  • One open-weight model, one provider, one run per task, and 30 of 250 tasks in one synthetic firm.
  • An earlier pilot, on a different sample and an older graph, scored lower on accuracy.
  • The graph cost about $7 to build once for the whole firm. At this model's prices the agent saving pays that back after about 300 tasks.

What's next

  • A search-only control, so we can separate the graph from a better document search.
  • Repeat each task several times, to measure run-to-run variance.
  • LAB's unmodified harness plus one frontier model, to explain why our baseline scores as high as it does.

Discussion

Sign in with GitHub