Nanonets Research
What is the TRACE benchmark?
It turns real enterprise work into a test.
The TRACE benchmark (stands for Task-Relevant Applied Constraint Execution) checks whether an agent harness can finish real enterprise tasks by:
- Correctly retrieving rules that govern it
- Correctly applying those rules to solve the task
Each sample is a large enterprise context history, the kind of history that piles up across documents, spreadsheets, chat, databases, apps. It has relevant rules dropped into the vast history at random. The agent harnesses get a task and a grader they never see.
Here is an example from the dataset -
The solver (agent harness) reads one long finance context. The rules are inside it, scattered far apart and buried mid-thought. In this example, five of them are relevant to the task. In the flow of the context, you can see they read like small talk.
The agent harness is asked to do a task. Read the task below. You can see it does not name any of those five rules explicitly. The solver has to know they exist, deduce which ones fire here, and then fire them.
Summarize the process we finalized and write out the explanation for the journal entry involving the Sales Revenue and COGS accounts, assuming an intercompany balance of exactly 45000 EUR.
No word about uppercase, number format, or entry order. But the three rules apply anyway. The variance note stays out since the task sets no threshold. The import tag never fires since no supplier here is Chinese.
The solver solves against a hidden rubric it never sees. An LLM-as-a-judge reads the answer and scores each line by meaning. The task passes only if the facts are right and every rule that applies passes with them.
45.000,00.(Import) tag, only for a Chinese supplier. Not here, so it drops out.Say the entry is arithmetically perfect but the accounts come back lowercase. Five rules apply, four pass, so rubric pass is 80%. Task pass is 0. To the analyst, that entry is wrong.
Leaderboards and demos show this is a solved problem, but every deployment says not yet
A lot of benchmarks paint a false rosy picture. This happens because these benchmarks treat memory retrieval as the final step, and conclude memory is a solved problem.
Then you hand the agent harness a real task, like closing the month or writing an accurate audit note. The facts it needs are spread across PDFs, spreadsheets, Slack / Teams messages, etc., and it finds them.
But the rules for how the work has to be done are never phrased as a question (how accounts get named, which number format to use, the threshold the finance team agreed on once and never mentioned again). So it breaks them quietly.
Here is that failure, drawn to scale. This is the strongest system we tested, on the easier of the two context history sizes.
It gets worse at larger context history sizes. Only 10.5% of tasks are fully correct at 1M tokens. And interestingly, 79% of every failure is a rule the agent harness didn't follow, not a fact it got wrong. The problem is that most benchmarks measure the latter.
A task is either right or it is not. It passes only if the answer is correct and every applicable rule holds at once. That is what the red bar counts. The gray bar counts partial credit, the fraction of individual checks it got right.
If we want to have a discussion on deploying autonomous agents, we first need to admit that real work does not pay partial credit. A journal entry with perfect arithmetic and the wrong number format is still wrong.
Rules that don't apply serve as traps. Even if you solve recall, precision fails.
What makes TRACE (and enterprise tasks) more than a harder retrieval test is the structure of the grade. For every task, each rule in the context history is classified relative to that specific task.
Dump every rule you found and precision tanks. Retrieve narrowly and recall tanks. You need both at once.
And the 305 distractors are not random noise, but instead built to look relevant. Task pass rate is 1 only if the answer is correct and every active check passes and every distractor is avoided.
A journal entry that is arithmetically perfect but uses the wrong number format is, to the analyst, wrong.
The complete leaderboard
We tested with the following agent harnesses:
- Full-context baselines
- Ten purpose-built memory systems
- Coding agents
Every memory backend answers using Gemini 3.1 Pro. Task pass is all-or-nothing per task. Rubric pass is fraction of individual checks it got right. The gap between them is TRACE.
Memory systems were not tuned to their ideal config, and the 1M board drops backends whose runs got too expensive. Even so, the ranking barely moves: nothing built for recall closes the gap.
Most systems get individual checks right. Over 80% rubric pass is common. Task pass falls because passing the whole task means honoring every rule together, on the same draft, at the same time. One missed format rule sinks an otherwise perfect entry.
What went wrong
The rules already sit somewhere. A SharePoint runbook, a closed Jira ticket, an email from the compliance lead, a vendor contract.
45.000,00 number format.(Import).Each memory method we tried in our harnesses breaks rules. Check the examples outlined below.
Vector search filters on keyword and semantic similarity. But a downstream task prompt never says which rules to fetch. The right rule has to be inferred from a deeper reading of the whole context.
(Import) tag.No "Shenzhen" keyword in the query.The deficit is keyword filtering. Standard RAG misses Rule B because the query lacks the contextual parameter, so the vital compliance constraint is never fetched.
Memory frameworks automate ingestion and auto-injection, but the agent has no freedom to navigate the underlying context. It can only ingest and query passively.
The deficit is lack of exploration. Rigid pipelines deny the agent the freedom to search, scan and audit rule relationships, so wrong retrieval becomes work failure.
Dumping raw Markdown into the prompt builds a massive context wall. Forcing the agent to read hundreds of lines of prose scatters its focus.
## Section 1: Introduction
Lorem ipsum dolor sit amet...
BURIED RULE: (Import) tag in Shenzhen ledgers
## Section 2: Audit Guidelines
All numbers must be justified...
The deficit is the context wall itself. Flat files mean token bloat and attention drift. Critical rule exceptions get lost inside the prose walls.
What ultimately worked better was letting the agent navigate.
Give the agent the run of the rules. Let it search, cross-check, and re-read while it works. Blind fetches and locked stores fail without a sound, and that quiet failure is the whole problem.
TRACE vs. other benchmarks
LoCoMo, LongMemEval, BEAM measure retrieval recall, which is only one part of the actual deployment problem. On LoCoMo and LongMemEval, the best systems already clear 90%, which basically tells us that problem is solved.
Thye ask a model to fetch a fact when the question names it. But a real task never arrives as a question.
Top memory systems clear 90% on this conversational-recall test.
Mem0 92.5% · MemCog 92.98%
Over 98% on single-session recall, the rest close behind.
Mem0 94.4% overall
Hard at 1M tokens. But it still grades scale and recall, not rule-following.
Top result ~62% at 1M · ~49% at 10M
Scores as reported by Mem0's 2026 benchmark survey and the BEAM paper (ICLR 2026). Numbers move with backbone and config.
BEAM is the exception on scores. It is genuinely hard, about 62% at one million tokens, and it does not saturate. But BEAM still grades recall.
TRACE is built on BEAM's generation pipeline and its 100k and 1M scales, so the numbers line up. Then it adds two aspects of real work:
- Combination: One deliverable graded on facts and every applicable rule.
- Distractors: Near-miss rules that must not leak in.
That is why a TRACE score measures whether an agent survives real work when deployed, instead of just measuring how well it remembers.
| Benchmark | Shape | Query names target? | One deliverable? | Distractors? |
|---|---|---|---|---|
| LoCoMo | Conversational recall | Yes | No | No |
| LongMemEval | Recall, 5 abilities | Yes | No | No |
| BEAM | Scale + 10 abilities | Mostly | Per-probe | No |
| TRACE | Task under buried rules | No | Yes | Yes |
About the dataset
The current dataset is built around R2R finance workflows: journal entries, reconciliations, month-end close, intercompany eliminations, FX. Finance teams accumulate exactly the kind of standing conventions the benchmark needs.
There are two sizes (context history), at 100k and 1M tokens. Each sample is one long context history, already stitched into a single transcript. Rules can sit mid-paragraph, mid-sentence, in routine, back-and-forth. There is no inherent index or "here are the rules" sections.
Note that the TRACE dataset does not test whether you can merge scattered systems into one context. It provides context and asks a harder question, which is "can you apply the right buried rules and drop the wrong ones?".1M is still small. To show why architectures like agentic search earn their keep, and why context graphs improve upon those results by saving multi-hop jumps and latency, we need 10M tokens and beyond. Below that scale, building the graph costs more than it returns. The blocker is data quality. A clean 10M-token transcript (with rules buried and no contradictions) is slow to build. This is what's pacing the next release now.
Choose benchmarks based on goals
Every benchmark leads to solutions to the exact problems they score. Score recall, and systems get better at recall. TRACE scores the aspects that stall agentic AI deployments today. Which is following the rules teams never wrote down as a question.
And the gap still feels significant. No agent harness we tested crosses 40% at 100k tokens, or 11% at 1M. Note that this is the state-of-the-art on work that looks, to a person, entirely routine.
The harness and data are open. For folks close to agentic development, it is very easy to submit your own solvers in github and get them added to the leaderboard.
PS: We would absolutely love to talk to you about the workflows you are trying to automate. Feel free to schedule a call.
In a demo, the agent finds the answer and looks brilliant. In production, it has to follow the rules your team never wrote down as a question, and that's where things fail and trust breaks. This gap is real, measurable, currently unsolved. It's the problem Nanonets is built to close.