Nanonets
Leaderboard
Nanonets
Leaderboard
Task-Relevant Applied Constraint Execution
Real enterprise work runs on complex, multi-step processes. The TRACE benchmark, published by Nanonets, measures whether AI can actually execute them, start to finish.
TRACE checks whether an AI can finish a real task while following the rules that govern it. Each sample is a long enterprise conversation, the kind of history that piles up across documents, spreadsheets, and chat, with rules dropped into it at random. At the end the AI gets a task, and a grader it never sees.
One enterprise conversation with the rules mentioned once and buried inside it. No index, no "here are the rules" list.
Close the entry, write the audit note. The rules that apply to it were never restated in the ask.
Scored on the facts and every buried rule that applies, against a hidden rubric the solver never sees.
In large enterprises, the rules that govern how work gets done are scattered and unstructured. Resolving that means a clear operational pipeline: first gathering disparate documentation from SharePoint, Jira, and emails, then structuring it into repositories for retrieval.
Explore how guidelines are collected from disparate sources, structured into repositories, and where standard search APIs fail downstream.
“GL transfers involving holding partners must be fully capitalized in standard legal casing.”
All GL partner accounts (e.g. SALES REVENUE) must be fully UPPERCASED for ERP parsing.
Vector search filters on keyword and semantic similarity. But a downstream task prompt never says which rules to fetch. The right rule has to be inferred from a deeper reading of the whole context.
(Import) tag.Memory frameworks automate ingestion and auto-injection, but the agent has no freedom to navigate the underlying context. It can only ingest and query passively.
Dumping raw Markdown into the prompt builds a massive context wall. Forcing the agent to read hundreds of lines of prose scatters its focus.
To build reliable task systems, your knowledge base has to be organized so the agent can navigate it fully. Blind keyword fetches and locked memory black boxes fail silently. Agents need the freedom to query, cross-reference, and inspect their rule corpus during execution.
On the benchmarks, memory looks like a solved problem. Then you hand the AI a real task, like closing the month or writing an audit note. The facts it needs are spread across PDFs, spreadsheets, and Slack or Teams messages, and it finds them. But the rules for how the work has to be done were never phrased as a question: how accounts get named, which number format to use, the threshold your team agreed on once and never mentioned again. So it breaks them, quietly.
Full-context baselines, ten purpose-built memory systems, and agentic solvers. Every memory backend answers with the same Gemini 3.1 Pro, so the only variable is retrieval. Task pass rate is binary, all-or-nothing: the agent has to satisfy every rule at once.
Joint pass (blue) is all-or-nothing per task. Rubric pass (gray) is the fraction of individual checks passed. The gap between them is the TRACE difficulty, and it widens with context length. Memory systems weren't tuned to their ideal config, and the 1M board omits backends whose runs were too costly.
What makes TRACE more than a harder retrieval test is the structure of the grade. For every task, each constraint in the conversation is classified relative to that specific task. Dump every rule you found and precision tanks; retrieve narrowly and recall tanks. You need both at once.
The prompt names none of the rules below. Each was stated once, mid-conversation, weeks earlier. A solver that's great at lookup can still return a fluent, arithmetically correct draft that trips three of them and scores zero.
45.000,00.(Import), but only if the supplier is Chinese.LoCoMo, LongMemEval, and BEAM all score recall: the question names what to fetch, and the system fetches it. But an enterprise task never arrives as a question. So a system can top those leaderboards and still get your month-end close wrong, because none of them test whether it applies the rules a real deliverable has to follow.
| Benchmark | Shape | Query names target? | Grades one deliverable? | Distractor rules? |
|---|---|---|---|---|
| LoCoMo | Conversational recall | Yes | No | No |
| LongMemEval | Recall, 5 abilities | Yes | No | No |
| BEAM | Scale + 10 abilities | Mostly | Per-probe | No |
| TRACE | Task completion under buried rules | No | Yes | Yes |
TRACE is built on BEAM's generation pipeline and its 100k / 1M scales, so the numbers are comparable. What it adds is the two things real work demands and recall benchmarks skip. First, combination: one deliverable, graded on getting the facts right and following every applicable rule at once. Second, distractors: near-miss rules that govern a neighboring artifact and must not leak into this one. That's why the score reflects how an agent will actually perform, not just how well it remembers.
The current dataset is built around R2R finance workflows: journal entries, reconciliations, month-end close, intercompany eliminations, FX. Finance teams accumulate exactly the kind of standing conventions the benchmark needs. Two sizes, one already-unified transcript per sample.
In a demo, the model finds the answer and looks brilliant. In production, it has to follow the rules your team never wrote down as a question, and that's where trust breaks. The gap is real, it's measurable, and today it's unsolved. It's the problem Nanonets is built to close.