A reviewer's attention is a fixed budget

A reviewer can only pay attention to so much code. The PR size grew significantly this year. For the past months we have been interviewing the tech leads who review AI-generated code for a living, and tinkering with our own review surface around what they told us. These are our conclusions so far.

The human attention budget didn't grow. The PR size did

Every quote below is verbatim from the people we interviewed for our last post: four months, more than 100 tech leads, architects, and senior engineers, on how their teams actually use AI.

"Now that you remove the bottleneck of writing the code, we've just moved the bottleneck to reviews."— an engineering manager

Everyone sees the amount of PRs growing. Another problem is that each PR got larger than one person can hold in their head. It is impossible for a human to properly review the 100-file PRs AI is sending. Human attention is limited.

Three problems with the review surface at this volume

1. It gives every file the same weight, in alphabetical order, which nobody chose as the order to spend attention in.
2. It shows what changed and not why, so the reviewer reconstructs the reasoning from the code, one file at a time.
3. It is one tool out of three: for any sizeable PR, the reviewer comments through GitHub, checks out the branch locally to navigate the code in their IDE, and opens a local agent for the questions neither one can answer.

The rest of this post is what we built against those, including the parts that did not work.

1. Triage by criticality

In the last post we said that industry-standard code, a boilerplate integration or reading a CSV in Python, is easy for a model, and that as models commoditize, that kind of code commoditizes with them. This is the first point towards deciding what we should review and what we should not. The seniors we sat with already split their reviews in two, although they do not do it consciously.

Risky code gets read first, every time. Depending on the profile of the reviewer, their definition of risky code changes. For a backend, the data model, the APIs, the contracts between services, auth, payments, anything that touches money or identity. This risky code is read first, every time. One CTO wrote their order down for us in a Slack message.

"1. Check we haven't dumped [junk] into the data models. 2. Check we haven't modified contracts between micro-services 3. Then, once those parts are validated, I look at the coherence and readability of the code."

Boilerplate is skimmed over. This type of code gets the last part of the review, when the reviewer is already not really paying attention anymore, e.g., the helpers, the tests, the connectors to a well-documented API. If it is not business critical, it is not your expertise, and it does not need to be non-standard; the model has seen the pattern ten thousand times and it will probably produce code that is good enough.

What we built

The first thing we built is a change map. Every changed file gets a criticality tier, and the tiers are set per viewer rather than per repo: a backend reviewer marks the data model and the APIs critical and the frontend tests low; a frontend reviewer flips the same PR the other way. This is done by matching file paths with rules. Before reading a line, you see the shape of what you are about to spend attention on, and you discard what you will not open.

"Where I'm losing time is I need to skim through the changes to see what is touchy in the PR, now I can just see in a minute. This PR has critical files touched, or it's just low impact."— a lead architect

Result: we shipped the criticality ordering and triage first: you can discard files you don't want to read, then the diff is displayed with critical files first, noise last. Users do like it, although they feel a bit uncomfortable explicitly saying "I'm not reading this" at first. Our users asked for the alphabetical file tree to navigate as well, because that is what they are used to, so we provide both and they choose. Either you navigate via alphabetical order, or you navigate by criticality.

This is not perfect yet. A one-line change in a utils file can be the whole blast radius, and a path rule will not flag it. Our belief is that those changes are being treated as the last part deserving attention and thus not read carefully anyway. The tiers do not make it worse.

A way to improve would be to make the rules smarter and flag tiers based on the semantic meaning of the changes. Another addition we're looking at is a clean way to mark "this we did not really check, and that is fine according to our policy." We have not shipped that yet.

2. Show the why behind the changes

The reasoning that used to live in a dev's head, or on a whiteboard when the feature needed some more design, the edge cases, the alternatives that were rejected, the constraints, now gets typed into a prompt before the agent writes a line. Recovering what the human thought about and determining if the logic is correct has always been the slow part of the review. Now the issue is that there are many more decisions in a single PR, and you do not know whether a change was backed by actual thinking from the developer or was an arbitrary choice by the agent.

"I spend my time on PRs writing why, why, why [...] I'm forcing the devs to own their decisions."— a tech lead

What we built

Hervé captures the developer's thinking (the prompts) and puts it next to the diff in each PR. That way, we can expose important decisions that were made during implementation, e.g., we cache X but not Y. Our first version surfaced all of it at the top of the page next to the PR description. That was too much, and a user told us so within a week:

"The info at the top is just too much and I just skip it."— a CTO

It felt like reading the PR twice. The second version keeps only what deserves the top of the page: the intent shifts, where the agent added or dropped scope against the spec it was given, and the undiscussed decisions, where the agent chose something on its own without asking the developer. Everything else is available on demand. Hervé can read the AI sessions behind the PR, so you can ask it whether the developer prompted for a given edge case, and it finds the answer in the session, or tells you it is not there.

One thing that surprised us: the PR author finds it useful to read the undiscussed list, to see what the agent decided on their behalf.

3. The IDE feeling in the review surface

Reviewing one PR today takes three tools. GitHub or GitLab, to read the diff and leave comments. An IDE on a local checkout, to navigate the code: go to definition, find usages, search for a symbol. And a local agent, to ask the higher-level questions about the change. Three applications for a single review.

"I checkout locally on the PR to get the code in the IDE, asks questions via Claude-code, and make comments on Github. The worst is when I review multiple PRs in parallel, I start mixing the windows and do not know which is which anymore."— a lead dev at a startup

What we built

We built the IDE features the reviewer needs into the review surface: go to definition, find usages, symbol search, on the review page. That folds two of the three tools into Hervé. The third, the local agent/harness, we have not folded in yet. It comes with a subscription most developers already pay for, so one more question about a PR costs them nothing, and we have not found a way to do better or cheaper than that yet. For now, we have decided not to compete with it. Developers love their harness anyway. In fact, they're asking us to expose Hervé's features to their harness rather than trying to replace it for review. This is probably where we will be going.

Conclusion

We are asking people to digest an amount of information per PR that is far more than a head was built to hold. The tooling has not caught up to the AI revolution. We have been coming at it from a few angles: criticality and ordering, so the attention goes where it counts; the captured intent, so less of it goes to reconstructing the why; the IDE features, so it stops requiring three windows for a single review. Some of it worked, some of it did not. It is not humanly possible to review everything properly at this volume, and we are building to maximize review quality.