Tesseract joined two words
At 600 DPI, Tesseract returned onthe as one high-confidence token even though the page visibly contains on the.
Case 1 · Detailed incident report
Last modified: (dates/navigation repair). Content current as of: (last retained content commit; content verification date unavailable). Original content last edited: . Current claims have not been freshly reverified.
Why did “on the” become “on onthe”?
Tesseract made a bad word-boundary proposal. PDFMD’s own replacement adapter then turned that proposal into a duplicated word. The immediate product defect is ours.
On this page
Executive verdict
Two errors combined; only one created the duplicate
Our adapter broke token conservation
PDFMD applied onthe only to the base token the. It failed to consume the already-existing on, producing on onthe.
Accountability. It would be misleading to blame this output solely on “bad OCR.” A provider may return a flawed candidate; our code must reject any replacement whose geometry and token cardinality cannot be reconciled safely.
Exact retained trace · page 8
The baseline was right; the “improvement” made it wrong
carrying on the IES. The green box identifies the target base token the; the 600-DPI candidate extended left across neighboring on.216-DPI base pass
Tesseract PSM 3 emitted separate tokens on and the. The target the had confidence 84.784927%. The phrase was correct.
600-DPI verification pass
Tesseract PSM 3 emitted one token, onthe, at confidence 94.499130%. Its box overlapped both base tokens.
PDFMD candidate gate
Our detector version 1.0 accepted the proposal because overlap with the was 0.9628, confidence exceeded the 87.784927% floor, and edit distance 2 met the allowed limit.
PDFMD materialization
The adapter replaced only the with onthe. The separate base token on survived. Canonical output became carrying on onthe IES.
Source and correct baseline
carrying on the IES
Output after refinement
carryingon ontheIES
The census found the same mechanism three times: on onthe, a aresult, and in inthe. These are word-boundary association failures, not paragraph-joining failures.
Resolution investigation
600 DPI was manufactured from a 200-PPI page image
A PDF does not necessarily have one “original DPI.” PDF pages use physical coordinates—72 points per inch. Vector text is resolution-independent; embedded images have an effective pixel density determined by their pixel dimensions and displayed size. The exact 20-page OCR input in this run is image-only: each A4 page contains one 1-bit grayscale JBIG2 image and no OCR text layer.
StageRetained page 8 facts
Embedded source image1654 × 2339 pixels at 200 × 200 PPI; this is the available optical information.
PDF page595.44 × 842.04 points (A4); one full-page raster image.
216-DPI base render1787 × 2527 RGB pixels; 1.08× the source image in each dimension.
600-DPI refinement render4963 × 7017 RGB pixels; 3× the source image in each dimension and about 9× its pixel count.
1. PDFium creates the 216-DPI base bitmap
The retained OCR configuration requested 216 DPI. PDFMD opened the image-only PDF with pypdfium2/PDFium and used render scale = 216 / 72 = 3 pixels per PDF point. PDFium resampled page 8 into a 1787 × 2527 PNG. Tesseract then received that already-rendered image:
tesseract page-8.png output -l eng --dpi 216 --psm 3 tsv makeboxPSM 3 asks Tesseract to discover page regions, lines, and word boundaries automatically. It correctly emitted separate base words on and the, both with the retained score 84.784927.
2. A broad page-level trigger starts refinement
PDFMD starts high-resolution refinement when any token on a page has a score below 0.90, contains non-ASCII text, begins with |, or resembles a suspicious ordinal. A low-scoring token therefore triggers another render and another full-page OCR pass. In this retained VNAT run, all 20 pages produced full-page 600-DPI TSV output.
3. PDFium creates a new 600-DPI bitmap
The refinement resolution is a fixed code constant, _REFINEMENT_DPI = 600. It is not read from the PDF, derived from the embedded image’s 200 PPI, or configured separately. PDFMD used render scale = 600 / 72 = 8.3333 pixels per PDF point, producing a 4963 × 7017 PNG.
Upscaling is not rescanning. The 600-DPI bitmap contains many more pixels but no new optical detail. It is a 3×-per-dimension resampling of the original 200-PPI page image. Resampling can change edge smoothing, apparent stroke widths, interword gaps, and the connected components seen by OCR.
4. Tesseract segments the full page again
tesseract page-8-text-refinement.png output -l eng --dpi 600 --psm 3 tsvThe --dpi 600 option does not create the additional pixels; PDFium has already done that. It tells Tesseract the intended density of the supplied bitmap. PSM 3 then performs a new, independent word-segmentation decision. This time Tesseract emitted onthe as one word.
5. Pixel boxes are normalized back into PDF coordinates
The 216-DPI base words occupied on: x=383…415 and the: x=422…463 pixels. The 600-DPI candidate occupied x=1065…1284. PDFMD converts 600-DPI coordinates back to PDF points by multiplying by 72 / 600 = 0.12:
ObservationHorizontal range in normalized PDF points
216-DPI
onapproximately 127.67…138.33216-DPI
theapproximately 140.67…154.33600-DPI
ontheapproximately 127.80…154.08—spanning both base words.6. The candidate gate accepts the wider word for only one token
For the, the wider candidate had 0.9628 geometric overlap, exceeded the required score floor by 6.714203 points, and met the edit-distance limit. For on, changing to onthe exceeded the edit-distance limit, so on survived. The adapter consequently materialized on + onthe.
Why was 600 selected?
The documented design intent was to give weak words, tiny superscripts, and missing gutter numbers a second high-resolution observation. Separate 600-DPI crops can also be read under Tesseract PSM 10 and PSM 13 for special isolated-token cases; this particular the change came from the full-page PSM 3 pass. The retained trace contains no benchmark or rationale establishing that 600 is optimal. The value entered the module as a hard-coded “high-resolution verification” constant.
For this 200-PPI input, 600 DPI could not recover detail that was never captured. It changed the representation and Tesseract’s segmentation. The base Tesseract call took 1.514 seconds; the full-page 600-DPI call took 3.309 seconds. The larger, slower observation then overrode a correct base phrase.
Resolution conclusion. A second-resolution check is a plausible OCR technique, especially for genuinely small glyphs. In this case it was neither source-resolution-aware nor independently proven superior. The product error was allowing its wider candidate to replace one member of a two-word region.
Components and ownership
Who did the job?
ComponentOwnershipRole and finding
pypdfium2 5.9.0
PDFium 150.0.7869.0Third-partyOpened and rendered the PDF. No retained evidence attributes this word duplication to PDFium.
PDFium 150.0.7869.0Third-partyOpened and rendered the PDF. No retained evidence attributes this word duplication to PDFium.
Tesseract CLI 5.5.2
English · PSM 3Third-partyRead the 216-DPI base words correctly, then proposed the incorrect merged token
English · PSM 3Third-partyRead the 216-DPI base words correctly, then proposed the incorrect merged token
onthe at 600 DPI.High-resolution candidate gate
detector 1.0Our code
detector 1.0Our code
pdfmd/recognition/raster_text_refinement.py selected the candidate using overlap, confidence gain, and edit-distance rules.Replacement materializationOur codeReplaced one base token without conserving the second overlapped token. This is the direct cause of
on onthe.Source oracle and censusOur evaluationCompared all 27 retained refinements against independently frozen source expectations and exposed the three duplicated-neighbor cases.
Not involved in this failure path. Docling is a project dependency, but it was not the OCR provider or replacement decision-maker for this trace. No language model reviewed or changed this token.
Why this happens in 2026
Advanced OCR is not an end-to-end correctness guarantee
The error looks ridiculous because a human reads the phrase instantly. The system does not receive “words” from the page. It receives pixels, competing character hypotheses, rectangular regions, confidence scores, and then a sequence of software decisions that must agree on boundaries.
OCR confidence is not source truth
The wrong onthe proposal scored 94.50%, higher than the correct the at 84.78%. The score measures the engine’s preference among its own hypotheses; it does not certify that the source was read correctly.
More pixels can change segmentation
A 600-DPI render enlarges the available pixels, but it also changes connected components, spacing, and the word boxes chosen by page segmentation. With a 200-PPI source it adds no optical detail and can merge a gap that the 216-DPI pass treated correctly.
Libraries stop at different boundaries
Tesseract supplied candidate text and geometry. It did not know PDFMD intended to replace exactly one earlier token. That cross-pass reconciliation is application logic, and in this case it was incomplete.
An ensemble can amplify a mistake
A second OCR pass helps only when the adoption rule is safer than either pass alone. Here, a higher-confidence but wider candidate overrode a correct baseline and made the final text worse.
Generic OCR optimizes averages
Leading vendors and open-source engines are trained for broad recognition quality. They cannot encode every downstream invariant of legal-document reconstruction: exact word conservation, printed labels, note links, style spans, and source provenance.
The last mile belongs to us
Using a mature OCR library is sensible; blindly trusting it is not. PDFMD must preserve alternatives, compare complete output with the source, and abstain when geometry does not support a safe replacement.
Why not just switch OCR libraries? Another engine may avoid this particular merge and introduce different substitutions, splits, omissions, or layout errors. The product defect here is an unsafe integration invariant; replacing the provider would not repair that invariant.
Current repair status
Preserve the correct raw reading
The earlier association proposal produced no valid source-scored OCR result. The historical gate report (outdated · September 2026) broadens the prerequisite: correct raw text must survive every downstream stage before dictionary, style, or structure work can advance.
Bind the complete page evidence
Retain the exact source, raw OCR, expected page text, settings, and runtime before examining another candidate.
Preserve correct raw words
Trace every selected replacement through refinement and canonical output. A wider candidate must not duplicate or discard source words.
Test a real correction and a confounder
The control must prove both that a legitimate correction survives and that nearby correct words remain unchanged.
Review before a broader run
Independent review must confirm the exact phrase, complete output, and audit lineage. No new recognition trial has run under this gate yet.
Limit of the earlier proposal. It targeted three neighbor duplications. It has not passed an actual-source trial and would not establish reliable English recognition or resolve the eleven direct lexical losses, two still-wrong readings, or existing paragraph, note, and style failures.
Evidence record
Reproducible facts
ItemValue
Original 22-page source PDF SHA-256
ee83eb5bbb2b05fd762402c8ad8214a54a3b46637925c0dcd8dfe8abc85ecf84Exact image-only OCR input SHA-256
18356059d277b4988000adbae0868d904aaff91fac9c8893f01d3ce75c2565a1RegionVNAT page 8 · region 17 · paragraph 47
Base evidence
ev-ocr-token-ee371c2e994f9959 · theDerived evidence
ev-ocr-token-ca12b01a8c3f2d4f · ontheExperiment result27 refinements: 10 gains, 14 losses, exact targets 15 → 11; no complete affected unit passed source fidelity.