Same model, better review
General AI assistants are remarkable. Asked to review a long technical document, they still miss things, invent quotes and hand back one long list in document order. Here is what we measured.
Why not just use my AI agent? →
The straight answer, with screenshots and a side-by-side table.The test
A public draft standard with known problems: the second public draft of NIST SP 800-63B (about 36,000 words). Its public comment period produced reviews from 28 experts. We kept the twelve objective defects that the experts raised and the final version fixed: a reference to a table that does not exist, a term spelled two ways, a clause repeated in one sentence, and so on.
Each tool reviewed the same Word file with the same AI model and no internet access. A person scored every result by hand against the twelve defects.
| Setup | Defects found (of 12) | What you get | Time |
|---|---|---|---|
| General AI agent, private model on our own hardware | 0 | 20 findings; 6 of its 8 quotes do not exist in the document | 6 min |
| DocReview, same private model | 4 | 193 ranked comments inside the Word file | 51 min |
| General AI agent, cloud model | 4 | 196 findings in one report, in document order | 10 min |
| DocReview, same cloud model | 4 (+2 harder issues) | 225 ranked comments inside the Word file | 29 min |
| Best single human reviewer of 28 | 6 | about 19 comments | — |
What the numbers say
- On a private model, DocReview is the difference between a review and a guess. Organisations that cannot send documents to the cloud run smaller models; there, a general agent loses track of a long document and fabricates quotes. DocReview reads every section, checks every reference and anchors every comment to real text.
- On a strong cloud model, the finding is about equal; the result is not. DocReview delivers ranked comments in the Word file where the team already works, plus deterministic checks that a chat answer misses, instead of a report someone has to reconcile by hand.
- Neither replaces the expert. The best human found six. DocReview does the exhaustive checking so the expert's time goes to judgement.
Then version 2 arrives
We also checked DocReview's review of the draft against the published final version. The four defects NIST fixed came back as resolved, each with the passage that shows it; the rest were reported open or partly addressed. A general agent has no memory of the previous round.
Method notes: single runs on one public document, hand-scored; we repeat the benchmark on every release and with every new model generation, and we will publish the documents, scoring sheets and prompts used for the general agent.