The evaluation layer: how buy-side firms are benchmarking agentic research outputs against human analysts

The buy-side procurement conversation around agentic research has shifted. In 2024, the question was which model, which content library, which UI. In 2026, with Rogo deployed across more than 25 investment firms, AlphaSense reporting that AI features drive roughly half of its new-business conversations, and Magnetar's AI-agent equity fund on the 2026 launch calendar, the question is narrower and harder: how do you measure whether an agent's research output is trustworthy enough to size a position on?

Our read is that evaluation methodology, not model choice and not content coverage, is becoming the layer where AI research vendors win or lose institutional contracts. It is also spawning a new internal role (the AI research QA lead) and a small but growing third-party audit market. The firms that treat evaluation as a procurement checkbox are the ones getting burned in year two.

The procurement question the 2024 RFP didn't ask

The first wave of agentic research RFPs, roughly 2023 through mid-2024, was structured around capability demos. Vendors would run a query against a client's document set, produce a synthesized answer with citations, and the evaluation committee would look at the output and vote. The failure mode was obvious in hindsight: a demo is a curated best case, and a buy-side research team makes hundreds of queries a week under conditions no vendor gets to curate.

The production data changed the conversation. Firms that deployed agents across analyst pods in 2025 started tracking two numbers that had not appeared in the pilot memos. The first is citation faithfulness, the rate at which a specific factual claim in the agent's output actually appears, verbatim or in unambiguous paraphrase, in the source the agent cites for it. The second is disagreement rate, the fraction of agent outputs that a senior human analyst on the same question would materially revise before committing capital.

Neither of these is a model benchmark. Both are workflow benchmarks. And both require a golden dataset, meaning a fixed set of questions with adjudicated correct answers, to measure at all. That is the piece the 2024 RFP was missing, and the piece that 2026 procurement teams are now building before they issue one.

What the evaluation stack actually looks like

The stack that has emerged across serious buy-side deployments has four layers, and the order matters. A vendor that scores well on layer one and poorly on layer four is a vendor that will pass a pilot and fail a fund.

Layer one is the golden question set. Typically 200 to 2,000 questions covering the firm's actual research workflow: earnings-call comprehension, 10-K comparisons across a peer set, thesis-testing questions with a known contrarian answer, and edge cases the firm has seen an agent get wrong before. The questions are adjudicated by senior analysts and locked. Vendors are scored blind.

Layer two is citation faithfulness. For every factual claim in an agent output, a rater checks whether the cited source contains that claim. This is where the wheels come off many vendors. An agent that cites the correct document but paraphrases beyond what the document supports is a hallucination with a footnote, and buy-side compliance teams have started treating it as strictly worse than an uncited claim because it launders confidence.

Layer three is hallucination rate and calibration. Beyond citation, does the agent invent tickers, misattribute quotes to speakers on an earnings call, or confuse two similarly named subsidiaries? Calibration is the companion metric: when the agent expresses confidence, is it right at that rate? An agent that is 90% confident and 60% correct is a specific kind of dangerous.

Layer four is the human-analyst comparison panel. A panel of three to five senior analysts, blind to which output is human and which is agent, ranks outputs on a fixed rubric: factual accuracy, analytical depth, source quality, and would-I-use-this-in-an-IC-memo. This is the layer that ties evaluation back to the procurement decision, because it produces the only number a CIO actually cares about, which is how the agent stacks against the firm's own bench.

The public benchmarks and where they help

The public evaluation frameworks that have shaped the vocabulary here are not finance-specific, and vendors that lean too heavily on them in RFPs are getting called on it. Stanford's HELM established the discipline of scoring language models across many tasks and many metrics rather than a single leaderboard number, which is the correct instinct for buy-side use. Anthropic's published guidance on evaluating AI systems is a useful starting frame for teams building their first golden set. OpenAI's evals repository is a starter kit for the mechanics.

The finance-specific work matters more in procurement conversations. FinanceBench from Patronus AI is a public benchmark of open-book financial questions grounded in 10-K, 10-Q, and 8-K filings, and it has become a shared reference point in RFP conversations because it produces numbers a procurement team can quote without having to build the whole thing from scratch. Its limits are the same as any public benchmark: once vendors know the questions, the questions start leaking into training and evaluation loops, and the signal decays. Serious buy-side teams treat FinanceBench as a floor (a vendor that scores badly here fails, but a vendor that scores well here has not yet won) and build a private golden set on top.

Our view is that the public benchmarks are doing their job when they get a procurement team to the point of asking better questions. They are being misused when a vendor's RFP response leans on a FinanceBench number as a proxy for the firm-specific evaluation the firm has not yet run.

Regulation is pulling the same direction

The evaluation stack is not just a procurement preference. Three regulatory threads are pulling the buy-side toward formal evaluation whether it wants to move or not.

The SEC's July 2023 proposed rule on predictive data analytics for broker-dealers and investment advisers directed at conflicts of interest arising from the use of predictive data analytics and similar technologies. The proposal remains proposed, and the final shape will differ, but the direction of travel is that firms using AI in investor-facing workflows will need to identify and address conflicts, which in practice requires documentation of what the systems do and how they were evaluated.

MiFID II best-execution logic, originally scoped to trade execution, has been extended in practice by European compliance teams to cover research inputs, on the reasoning that a decision based on flawed research is a decision that cannot meet a best-execution standard. That extension is not universally accepted, but where it is applied, it converts evaluation documentation from a nice-to-have into a defensibility requirement.

ISO/IEC 42001, the AI management system standard published in December 2023, is the standards-body equivalent of ISO 27001 for information security. It does not prescribe specific evaluation metrics, but it does require organizations to establish policies, objectives, and processes for AI systems, including evaluation and monitoring. Buy-side firms whose institutional LP base includes European pensions and sovereign wealth funds have begun to see 42001 alignment in due diligence questionnaires.

The common thread across all three is that a firm that cannot produce evaluation evidence for its AI research stack is a firm that cannot defend the stack when a regulator, LP, or IC asks. The days when a screenshot of a good agent answer would satisfy a compliance sign-off are ending.

The new internal role: AI research QA lead

A year ago, most buy-side firms did not have anyone whose job description included evaluating AI research outputs. The work was distributed across whoever ran the pilot, usually a data-science hire or a technology-forward analyst.

That is not tenable at production scale. A firm with 40 analysts running an agent across earnings season is generating tens of thousands of agent outputs a month, and no distributed process will catch the citation drift, the calibration slip, or the occasional confident hallucination that gets embedded in a memo. The role that has emerged in response is what several firms are now calling an AI research QA lead, sometimes reporting into research operations, sometimes into risk, occasionally into compliance.

The job is a hybrid one. The QA lead maintains the golden question set, runs quarterly re-evaluations against every deployed agent, sits on the human-analyst comparison panel, and owns the internal reporting line to the CIO on where the agents are getting better and where they are getting worse. It is not a data-science role and it is not a compliance role. The closest analog is a sell-side model risk management function, extended from valuation models to language-model outputs. We expect this role to be a standing headcount at every buy-side firm above a certain size within 24 months, in the same way that ESG analyst became a standing headcount in the mid-2010s.

A parallel category is emerging on the vendor side of the wall: third-party evaluation auditors. The value proposition is straightforward. A vendor cannot credibly grade its own homework in an RFP, and a buy-side firm building its first evaluation program benefits from an outside methodology check. Whether this settles into a Big Four extension, a specialist boutique category, or a function absorbed back into the vendors themselves is not yet clear.

Three scenarios for how this plays out

Base case: evaluation becomes procurement table stakes, and vendors converge. By late 2026, no serious buy-side RFP is issued without a required evaluation section, and the top three or four agentic research vendors all publish standardized citation-faithfulness and calibration numbers against a public benchmark like FinanceBench. Private golden sets are the differentiator on top. The market segments not by capability (which converges) but by which firms have built the internal QA muscle to use the tools well. In this scenario, evaluation infrastructure providers do well, and vendors compete on the margin of how well their platforms support customer-built evaluation loops.

Bull case for the incumbents: evaluation becomes a moat. The firms that invested earliest in evaluation methodology (both vendors and buy-side deployers) build compounding advantages. Their golden sets get sharper, their calibration data gets richer, and the switching cost for a firm that has spent a year tuning workflows to a specific vendor's evaluation regime is high. New entrants have to match not just capability but the evaluation apparatus around it, which is harder to reproduce than a model wrapper. This scenario favors AlphaSense, Hebbia, Rogo, and Bridgetown to the extent they can convert first-mover deployment data into durable evaluation lock-in. It also favors internal QA leads who become the institutional memory of what works.

Bear case: evaluation fatigue and a retreat to human-only workflows in some pods. A subset of firms concludes that the evaluation overhead is not paying for itself. Running quarterly panels, maintaining golden sets, and defending calibration numbers to compliance is expensive, and the productivity gains from agentic research, while real, are not uniformly distributed across analyst styles. Fundamental long-only pods with concentrated portfolios and long holding periods may find the evaluation cost exceeds the analyst-hour savings, and quietly retreat to a human-primary workflow with agents relegated to first-pass summarization. This scenario does not kill the category, but it splits the buy-side into evaluation-heavy adopters (multi-strat, systematic-adjacent, high-turnover) and evaluation-light skeptics (concentrated fundamental, long-duration).

Our read is the base case is the most likely, with the bull case as the tail outcome for two or three vendors. The bear case is real but sector-specific rather than industry-wide.

Who is affected downstream

The evaluation layer thesis has second-order effects that reach past the buy-side procurement office.

Expert networks sit in an interesting spot. If citation faithfulness becomes the gating metric, the value of a source that can be cited unambiguously (an earnings call transcript, an investor day recording, a regulatory filing) increases relative to a source that requires paraphrase (a background conversation, a color call). Networks that invest in verifiable, timestamped, and transcript-anchored deliverables benefit. Networks that lean heavily on background color without a citable artifact face a harder sell into agent-driven research workflows.

Data vendors and transcript providers benefit from the same dynamic. An agent that can cite a specific timestamp in a specific transcript passes a citation-faithfulness check more cleanly than an agent citing a summarized secondary source. The provenance chain matters more than it did in a pre-agent world, and the providers with the cleanest chains have an easier time getting embedded.

Compliance and risk software vendors are seeing pull-through demand for AI-specific monitoring. The buy-side firm that has built an evaluation stack still needs to log, retain, and audit the outputs, and the tooling for that at scale is not yet a solved category. We expect at least one significant funding round in the AI observability space aimed specifically at financial services workflows within the next 12 months.

Sell-side research finds itself in a curious position. To the extent buy-side agents can synthesize primary sources directly, the marginal value of a sell-side note as a research input decreases. To the extent citation faithfulness rewards named, credentialed, and durable sources, a sell-side note with a named author and a datable timestamp is a citation-friendly artifact. The net effect is not clear yet, and it likely differs by sector and analyst.

Questions a research analyst should be asking now

For a buy-side team standing up or hardening its evaluation program, the questions worth putting to the market (and to the internal team) are specific.

• What is our citation-faithfulness rate on the last 90 days of production agent outputs, and how has it moved quarter over quarter?
• When our agent expresses high confidence on a factual claim, at what rate is it correct? Is that calibration curve flatter or steeper than it was six months ago?
• On our internal golden set, how do the top three vendor agents compare against a blind panel of our own senior analysts on IC-memo readiness?
• Which analyst workflows show the biggest gap between agent output and human output, and is that gap closing, holding, or widening?
• If a regulator asked tomorrow how we evaluate the AI systems in our research workflow, what document do we hand them, and who signed it?

The firms that can answer these crisply are the firms that will still be running agents at scale in 2028. The firms that cannot are the firms whose 2026 pilots will quietly disappear from the next annual letter.