Summarization should be checked not “by eye,” but by comparing every substantive claim with the source. A reliable process has four stages: define the requirements for the summary, break the result into verifiable claims, find support for each claim in the source text, and separately check numbers, negations, causal relationships, and levels of certainty. The verification outcome should be reproducible: another person using the same source and comparison table should reach the same conclusions.
What Counts as Distortion
An AI model has distorted the data if the summary contains at least one of the following types of errors:
- a fact was added that does not appear in the source;
- a number, unit of measurement, date, percentage, or range was changed;
- the actor was misidentified: who performed, reported, or confirmed an action;
- an assumption was presented as an established fact;
- an important condition, limitation, or exception was omitted;
- a negation was changed: “did not confirm” became “confirmed”;
- correlation or a sequence of events was presented as a causal relationship;
- information from different sections, periods, or documents was combined;
- a cautious statement in the source was replaced with a stronger claim;
- a fact was omitted in a way that makes the overall meaning incorrect.
Semantic similarity is not the same as factual accuracy. A statement may sound convincing while still changing a condition, scope, source, or level of certainty from the original text.
Step 1. Define the Summarization Task
Before verification, determine exactly what the summary was supposed to preserve. Without this, it is impossible to distinguish acceptable compression from a critical omission.
Record at least four parameters:
- Purpose: what the summary is for — rapid review, an executive decision, legal review, technical analysis, or metric extraction.
- Required data: which facts must not be lost — amounts, deadlines, risks, conclusions, limitations, or responsible parties.
- Allowed length: for example, five bullet points, one paragraph, or a table.
- Restriction on interpretation: whether the model should only shorten the text or may combine conclusions and explain consequences.
For documents where errors may have serious consequences, add a separate instruction: “Do not add conclusions that are absent from the source. Preserve negations, conditions, ranges, and levels of certainty.”
Step 2. Break the Summary into Atomic Claims
Do not verify a long paragraph as a single unit. Divide it into the smallest claims that can each be confirmed or disproved independently.
Example summary sentence:
“The company completed the migration in June, reduced costs by 18%, and plans to decommission the legacy system by the end of the quarter.”
This sentence contains three claims:
- the migration was completed in June;
- costs decreased by 18%;
- the legacy system is scheduled to be decommissioned by the end of the quarter.
When the sentence is checked as a whole, it is easy to miss that two claims are correct while one is fabricated or distorted.
Step 3. Create a Comparison Table
Use a table that links every claim to a specific passage in the source.
| No. | Claim from the Summary | Source Passage | Status | Comment |
|---|---|---|---|---|
| 1 | The migration was completed in June | Exact quotation or link to the paragraph | Confirmed | The event and date match |
| 2 | Costs decreased by 18% | The source reports a projected 18% decrease | Distorted | A forecast was presented as an achieved result |
| 3 | The system will be decommissioned by the end of the quarter | No supporting passage found | Unconfirmed | Possibly added by the model |
Use only three primary statuses:
- Confirmed — the meaning fully matches the source;
- Distorted — the source contains a similar fact, but the conditions, scope, time, actor, or level of certainty were changed;
- Unconfirmed — the corresponding fact does not appear in the source.
You may also use the status “Omitted” for important information missing from the summary.
Step 4. Check Numbers and Structured Data
Check numbers separately from the surrounding text. Even when the overall meaning is correct, the model may alter a value, sign, unit, or reporting period.
What to Compare
- integers, percentages, amounts, and ratios;
- dates, deadlines, and intervals;
- units of measurement;
- direction of change: increase, decrease, deficit, or surplus;
- comparison basis: month over month, year over year, or plan versus actual;
- ranges and approximate values;
- the number of objects, participants, errors, or incidents.
The phrases “approximately 20%,” “no more than 20%,” and “at least 20%” are not equivalent. Likewise, “by 20%” and “up to 20%” have different meanings.
Practical Method
- Extract every number from the source into a separate list.
- For each number, record the context: what is being measured, for which period, and relative to what baseline.
- Extract the numbers from the summary.
- Compare not only the values but also their labels and context.
- Check whether words such as “approximately,” “up to,” “from,” “at least,” or “forecast” were omitted.
Step 5. Check Negations, Conditions, and Limitations
The most dangerous distortions often involve a missing limitation rather than an incorrect standalone fact.
Compare:
- “The feature is available” and “The feature is available only to administrators”;
- “The bug has been fixed” and “The bug cannot be reproduced in the test environment”;
- “The supplier guarantees the deadline” and “The supplier expects the deadline to be met”;
- “The method is safe” and “The method showed no critical risks in the scenario examined.”
During verification, look for markers such as:
- “if,” “provided that,” “only,” “except,” “with the exception of”;
- “not,” “not always,” “not confirmed”;
- “possibly,” “probably,” “presumably”;
- “according to the author,” “according to a participant,” “according to the report”;
- “in the test environment,” “in the sample,” “for the specified version.”
Step 6. Check Causal Relationships
An AI model may combine two adjacent events and present one as the cause of the other. Therefore, every construction using words such as “because of,” “therefore,” “led to,” or “as a result” should be checked separately.
If the source says, “The number of support requests increased after the update,” this does not necessarily mean, “The update caused the increase in support requests.” A causal conclusion requires an explicit statement or supporting evidence in the source, not merely a sequence of events.
When direct confirmation is absent, a safe formulation is: “The number of support requests increased after the update; the source does not establish a causal relationship.”
Step 7. Check the Level of Certainty
The summary must preserve the epistemic status of each claim — that is, the distinction between a fact, assessment, forecast, hypothesis, and someone else’s opinion.
| Source Wording | Acceptable Summary | Unacceptable Summary |
|---|---|---|
| The author assumes that the configuration is the cause | The author links the issue to the configuration | The configuration caused the issue |
| An increase in load is expected | An increase in load is forecast | The load increased |
| The result has not been confirmed by an independent review | The result has not yet been confirmed | The result has been proven |
Step 8. Check Completeness Against the Purpose
A factually correct summary may still be useless if it omits a key risk or a condition required for decision-making.
Create a list of required elements and mark whether each is present:
- the main conclusion;
- critical figures;
- limitations and exceptions;
- risks and uncertainties;
- deadlines;
- responsible parties;
- unresolved decisions;
- next actions.
An omission is critical if it could cause the reader to make a different decision than they would after reading the full source.
Step 9. Use a Second AI Pass Only as an Auxiliary Check
You can ask an AI model to perform the comparison, but its conclusion should not be considered final without human verification. The prompt should require references to specific source passages and explicitly prohibit unsupported inference.
Compare the summary with the source text.
For each claim in the summary:
Write the claim separately.
Find the exact supporting passage in the source.
Assign a status:
confirmed;
distorted;
unconfirmed.
Explain the discrepancy without adding new facts.
Check separately:
numbers and dates;
negations;
conditions and exceptions;
causal relationships;
forecasts and assumptions;
omitted facts that change the overall conclusion.
If no supporting passage exists, write explicitly:
“No supporting passage was found in the provided text.”For a long document, it is useful to provide numbered paragraphs or sections together with the summary. You can then require the response to cite a passage number instead of paraphrasing “from memory.”
Step 10. Assess the Severity of Errors
Not all discrepancies are equally dangerous. Divide them into three levels.
| Level | Example | Action |
|---|---|---|
| Critical | A number, negation, deadline, actor, or legally significant condition was changed | Do not use the summary until it is corrected |
| Major | A forecast was presented as fact, a limitation was omitted, or a causal relationship was added | Correct the error and recheck the entire related section |
| Editorial | Poor compression without a change in meaning | Revise the wording |
How to Formally Accept or Reject a Summary
Define the acceptance criteria in advance. For example:
- there are no unconfirmed claims;
- there are no critical distortions;
- all numbers have been checked manually;
- all required elements are present;
- each claim is linked to a source passage;
- assumptions and forecasts are not presented as facts.
A match percentage alone is insufficient. One error in a contract amount, a vulnerability remediation deadline, or a medical contraindication matters more than ten correctly shortened sentences.
Minimum Working Process
- Save the source text without modifications.
- Define the purpose and the list of required information.
- Break the summary into separate claims.
- Find the source for each claim.
- Check numbers, dates, negations, conditions, and levels of certainty.
- Mark unconfirmed and distorted claims.
- Check whether any information that could change the decision was omitted.
- Correct the summary.
- Repeat the verification using only the corrected version.
- Store the comparison table together with the final summary.
Final Checklist
- Every claim is supported by a specific source passage.
- The summary contains no facts absent from the source.
- All numbers, dates, units, and periods match.
- Negations, conditions, exceptions, and ranges are preserved.
- Forecasts, assessments, and hypotheses are not presented as facts.
- Causal relationships appear only where they are explicitly stated.
- Authors, participants, and information sources are not confused.
- Critical risks and limitations have not been omitted.
- Errors are classified by severity.
- The corrected version has undergone another comparison.
Limitations
There is no universal automated check that can guarantee the absence of semantic distortions. A second review by another AI model may detect some errors, but it does not replace comparison with the primary text. For legal, financial, medical, engineering, and information security documents, the final decision should be made by a specialist capable of verifying the source data and context.
This guide does not rely on version-specific features of any particular model or service. No external sources were used, so no references section has been included.