How to Compare Two Document Versions with AI Without Hallucinations

To compare two document versions with AI while minimizing the risk of fabricated differences, first generate a mechanical diff, then provide the model with individual diff items rather than the full documents. Each item should include a stable identifier and exact source quotations. The AI should classify changes that have already been recorded instead of searching for them again. Finish the process with a register in which every diff item has one of three statuses: “accounted for,” “noise,” or “manual review required.”

What You Need

• two immutable copies of the document: the old and new versions;
• a comparison tool that records insertions, deletions, and replacements;
• verified extraction of text and structure from the source files;
• a register of diff items with unique identifiers;
• a model that receives only prepared diff items;
• manual verification of quotations and conclusions against the source documents.

This method is suitable for contracts, technical specifications, regulations, policies, instructions, product specifications, and other documents where it is important to distinguish an actual edit from an interpretation. It reduces the risk of omissions and hallucinations but does not guarantee their complete absence.

Do not use the model’s output as the sole source of truth for legally significant, financial, or regulatory documents. A mechanical diff confirms that a change occurred, but assessing its consequences may require review by a lawyer, accountant, engineer, or process owner.

Step 1. Fix the Versions and Comparison Rules

Store the source files separately and label them unambiguously, for example: “Regulation_2026-05-10_old_version” and “Regulation_2026-07-01_new_version.” Record identifying file properties such as the name, size, date received, and, when necessary, a checksum. Do not edit these copies during the comparison.

Before generating the diff, define which types of changes must be included in the report:

• substantive changes: obligations, rights, prohibitions, conditions, and exceptions;
• numeric changes: amounts, percentages, dates, deadlines, and units of measurement;
• structural changes: additions, deletions, and movements of sections;
• editorial changes: spelling, punctuation, and style;
• formatting changes: numbering, emphasis, lists, tables, and the placement of document details.

Formatting must not automatically be treated as noise. In an ordinary instruction, a font change is often insignificant, but in a contract form, table, regulatory template, or document with mandatory emphasis, a change in numbering, signature placement, or table structure may have independent significance.

Step 2. Verify Text and Structure Extraction

You should compare not merely two sets of lines but two verified textual projections of the source documents. When extracting content from DOCX and PDF files, the following may be lost or distorted:

• comments and replies to comments;
• tracked changes, deleted text, and inserted text;
• footnotes and endnotes;
• headers, footers, page numbers, and watermarks;
• figure captions and text inside diagrams;
• merged cells and multi-level table headers;
• list numbering and cross-references;
• the reading order of columns and text blocks in a PDF;
• characters incorrectly recognized from a scanned document.

Before comparison, check the extracted text against the source file selectively and in critical areas. Verify the beginning and end of the document, several tables, footnotes, headers and footers, and clauses containing numbers or negations. When OCR is used, separately check similar-looking characters such as “0” and “O,” “1” and “I,” decimal separators, minus signs, and negation particles.

When comments, tracked changes, or footnotes are within the scope of comparison, export them as separate entities. The comparison cannot be considered complete if the tool extracted only the main body text.

Step 3. Generate a Mechanical Diff

Use the built-in comparison feature of a text editor, a version control system, or a specialized diff tool. At this stage, you do not need to explain the meaning of the changes. The tool’s task is to record exact insertions, deletions, replacements, and suspected movements.

Normalize whitespace before semantic classification and according to predefined rules. For example, a change in the number of spaces or line breaks may be marked separately as formatting noise. However, do not discard such differences unconditionally: a line break inside a table, numbered requirement, address, or set of document details may alter the document’s structure.

Diff Item Format

Assign a unique identifier to every change. Store not only the changed text but also its location, operation type, and context:

ID: DIFF-0042
Old version: Regulation_2026-05-10
New version: Regulation_2026-07-01
Old-version section: 4.2
New-version section: 4.2
Mechanical operation type: replacement
Old fragment: within 10 business days
New fragment: within 5 business days
Context before: The Contractor submits the report
Context after: after the end of the reporting period
Source: page 7, paragraph 3
Tool note: none

For a pure insertion, the old-fragment field must contain the value “absent from the old version.” For a pure deletion, use “absent from the new version.” This is not a quotation but an explicit indication that one side is missing:

ID: DIFF-0043
Mechanical operation type: insertion
Old fragment: absent from the old version
New fragment: The Customer may request an interim report.
New-version section: 4.3
Context before: 4.2. Report submission deadline
Context after: 5. Liability of the Parties

The context should help locate the fragment without replacing the change itself. Usually, the section heading, clause number, and one adjacent sentence on each side are sufficient. For a table, also preserve the table name, column headers, and row key.

How to Mark Movements

A movement must not automatically be treated as a deletion and an insertion. Create a suspected link between two items only when the text matches exactly or in a verifiable way:

ID: DIFF-0051
Mechanical operation type: suspected movement
Old location: section 3.4
New location: section 6.2
Old fragment: [full clause text]
New fragment: [full clause text]
Match: exact
Numbering and references require review: yes

Even when the text itself is unchanged, moving it may affect the clause’s scope or references from other sections, so it should be classified separately.

Step 4. Remove Data That Must Not Be Sent to the Model

Before sending data to an external service, review your organization’s confidentiality requirements. When necessary, replace personal data, account details, access secrets, and internal identifiers with stable labels:

[КЛИЕНТ_1]
[СОТРУДНИК_3]
[СУММА_A]
[АДРЕС_ОБЪЕКТА]
[НОМЕР_ДОГОВОРА]

The same value must be replaced with the same label in both versions and across all diff items. Store the mapping table separately. After anonymization, check the diff again: the replacement process itself must not create new false differences.

Step 5. Send Diff Items to the Model, Not Full Documents

The primary workflow is to classify a predefined list of changes. The model must not search for differences independently in the full text. Send items in small batches while preserving their identifiers.

You classify items from a mechanical diff between two document versions.
Rules:


Consider only the provided diff items.


Do not search for additional changes or use external knowledge.


Do not correct or supplement the source quotations.


Preserve the identifier of every item.


For an insertion, the following value is allowed:
“absent from the old version.”


For a deletion, the following value is allowed:
“absent from the new version.”


If the old and new fragments cannot be matched reliably,
set the status to “manual review required.”


Do not discard a formatting change when the supplied data does not prove
that it has no structural or legal significance.


State a practical consequence only when it follows directly
from the quotations. Otherwise write:
“cannot be determined without additional context.”


Return a table with the following columns:


diff ID;


section or clause;


mechanical operation;


classification;


old fragment;


new fragment;


practical consequence;


review status;


reason for manual review.


Allowed classifications:


insertion;


deletion;


replacement;


numeric change;


deadline change;


obligation change;


right change;


prohibition change;


exception change;


editorial change;


formatting change;


movement without text changes;


ambiguous match.


DIFF ITEMS:
[вставьте подготовленные элементы]

Full document versions should be provided to the model only as a fallback when generating a mechanical diff is impossible. Such a comparison is less reliable because the model must search for, match, and interpret changes at the same time. This increases the risk of omissions, false matches, and the merging of separate edits. In this mode, the result cannot be used as evidence of completeness.

Step 6. Verify Every Row by Identifier

A row is considered accounted for not because the model produced a confident answer, but because it can be linked to a specific mechanical diff item. For each row, verify that:

1. the ID exists in the source diff register;
2. the old and new fragments are reproduced without changes;
3. the missing side is correctly indicated for an insertion or deletion;
4. the section and context make it possible to locate the fragment in the source document;
5. the classification matches the mechanical operation;
6. the stated consequence actually follows from the quotations;
7. a diff item has not been lost inside a merged row.

The model may merge several adjacent changes into one entry. This may be acceptable in an analytical report, but the verification register must retain all source IDs. For example, a merged row should reference “DIFF-0042, DIFF-0043” rather than hiding the second item.

Step 7. Review High-Risk Changes Separately

Filter the register for fragments containing numbers, dates, units of measurement, negations, obligations, rights, prohibitions, and exceptions. For this review, the model should again receive only the mechanical diff:

Review the provided diff items for high-risk changes.
Flag:


dates and deadlines;


amounts, percentages, and units of measurement;


negations;


words such as “must,” “may,” “prohibited,” and “permitted”;


conditions under which a rule takes effect;


liability, penalties, and restrictions;


references to appendices and other sections.


Do not search for new changes outside the list.
Do not alter the quotations.
For every conclusion, provide the diff ID.
If a value cannot be interpreted without an adjacent section,
set the status to “manual review required.”

Pay particular attention to a numeric change together with its unit of measurement. Replacing “10 days” with “10 business days” is not a simple editorial edit. Currencies, time zones, decimal separators, and negation particles also require separate review.

Step 8. Maintain a Final Coverage Register

The comparison cannot be considered complete merely because the model states that “no unaccounted items were found.” Evidence of coverage is a register containing every mechanical diff identifier.

Use only three final statuses:

• accounted for — the item is reflected in the report and verified against the source;
• noise — the item was reviewed and excluded with justification;
• manual review required — the match or significance is ambiguous.

The review is complete when every ID from the mechanical diff appears in the register, every item has a status and a reference to the source fragment, and all items marked “manual review required” have been assigned to an appropriate specialist. You cannot rely only on comparing the number of diff blocks with the number of rows in the final table: one block may contain several edits, while one report row may combine several items.

Example of Verifiable Classification

Mechanical diff item:

ID: DIFF-0067
Mechanical operation type: replacement
Section: 5.1
Old fragment: within 10 business days
New fragment: within 5 business days
Context: The Contractor submits the report after the end of the period.

Correct classification: “deadline change; the deadline was reduced from 10 to 5 business days.” Both values and the unit of time are present in the source fragments.

Incorrect interpretation: “the customer tightened control.” The customer’s motive is not stated in the diff, so this conclusion should be replaced with “cannot be determined without additional context.”

Common Mistakes

• Sending two full documents to the model instead of a diff. The model searches for changes again and may miss some edits.
• Missing IDs. It becomes impossible to prove that every mechanical comparison item was accounted for.
• Requiring two quotations for a pure insertion. This encourages the model to invent a nonexistent old fragment.
• Automatically discarding formatting changes. A structurally or legally significant edit may be lost.
• Using unverified extraction from PDF or DOCX. The comparison is performed against an incomplete representation of the document.
• Matching by row count. The number of report rows does not confirm diff coverage.
• Trusting confidence scores. The model’s self-assessment does not replace verification against the source files.
• Omitting the “manual review required” status. Ambiguous items are forced into false classifications.

Final Checklist

• The old and new versions are unambiguously labeled and preserved without changes.
• Extraction of the main text, tables, footnotes, comments, and tracked changes has been verified.
• A mechanical diff with unique identifiers has been generated.
• The operation, quotations, section, and context are stored for every item.
• Insertions and deletions explicitly indicate that one side is absent.
• Movements are separated from deletions and insertions.
• Whitespace normalization is separated from the assessment of semantic significance.
• The model receives diff items rather than full documents.
• Every conclusion includes the source change ID.
• Numbers, deadlines, negations, obligations, and exceptions are reviewed separately.
• Every ID is entered in the register with a status and a source reference.
• All ambiguous items are sent for manual review.

The workflow consists of four roles: the comparison tool records changes, the prepared register preserves their identifiers and context, the AI classifies only the supplied items, and a human verifies the quotations and consequences. This process does not eliminate errors completely, but it makes them more visible, reduces the risk of hallucinations, and leaves an auditable trail for every edit.