44 KiB

Raw Permalink Blame History

Article-Cropping Strategy & Learnings (self-improving loop)

Living doc for the geometric article-cropping in smart_extractor.py (cluster_all_article_blocks → crop_all_article_blocks). Updated each loop iteration.

Validation rules (NEVER violate)

Never run smart_extractor.py — it calls the paid Claude API. Validate ONLY with the geometric replay scripts: _replay_all.py (AJ), _replay_namaste.py (NT Siddipet), _replay_jangaon.py (NT Jangaon). These run the full cluster chain with NO API.
Measure read-only BEFORE editing (blast-radius of any new rule), then re-validate after.
Baselines (block counts) — keep stable unless an improvement is intentional & documented:
- AJ: 12 / 18 / 14 / 21 (pg4 20→21: the ';'-delimiter fix recovered a real గజ్వేల్ dateline R12)
- NT Siddipet: 11 / 23 / 20 / 25 (pg2 22→23: అంగన్‌వాడీ R127 split off the ప్రజావాణి section R110; pg3 19→20: పర్యావరణ R60 split off; both are the Type-2 paragraph_title-anchor split)
- NT Jangaon: pg1 → 6, pg2 → 18 (pg1 7→6: orphan photo R25 bound into its article; pg2 Type-2 splits: R65→3 కలెక్టర్/సుందరయ్య/పోస్టుమార్టం, R64→2 జాతీయరహదారి/ఉపసర్పంచుల, R97→2 గుండెచికిత్స/మృతుడిపరామర్శ; + L89 court notice, F44 furniture)
- Sakshi Jangaon: pg1 → 22, pg2 → 20 (new paper, no line separators; all orphan PHOTOS bound to articles — 0 image orphans remain)

Layout model (from the line-pattern study)

NT (Namaste) = boxed grid. ~139 horizontal rules/4pp, many ~2-column wide; long vertical column dividers. ~50% of articles are ruled on 3–4 sides. Boundaries are largely drawn.
AJ (Andhra Jyothi) = whitespace layout. Few rules (~69/4pp, mostly ≤0.6-col caption underlines). Most articles ruled on 0–2 sides. Boundaries come from column gutters + grow-right-to-dead-end + headline/dateline structure. Lines are only confirmation in AJ.
Rule width classifies horizontal lines: ≤0.6 col = caption underline (NEVER a boundary); ~1 col = single-col article edge / box edge; ~2 col = article boundary (NT workhorse); ≥2.5 col = section/banner divider.
Vertical rules: long (≥0.5 page H) = column divider / article wall; short = quote-box edge.

Why earlier runs failed (root causes found)

Over-merge from clustering + containment-merge. Multi-column articles whose datelines sit only in the left column get over-extended upward; a separate dated article then falls inside the box and is absorbed → 2 articles in one crop (JG R28, R64). Fixed by header-split.
Banner over multiple sub-stories not cropped as one rectangle, or its sub-stories double-cropped → section-banner rectangle + containment-merge.
L-shaped articles (one column longer): rectangular crop's empty corner = a different article masked white → looks like white space (JG H28). Generic ragged-bottom fix REJECTED — it would also move AJ R48/R53 (currently fine). Still open; needs a tighter signal.
Top-of-column banner with no anchor falls into the article below → page-tall crop (JG R2). Fixed by ceiling-trim.
Body-less header strips (doc_title or quote-box paragraph_title) cropped alone, split from body → body-less header merge (H and now P).

Strategies implemented (all gated + measured to touch only defective blocks)

Banner rectangle (H & P, ≥1.6-col header, ≥2 cols, sure bottom boundary): crop whole rect.
Section-banner rectangle (doc_title ≥2.5 col with ≥2 headed sub-stories): crop banner-top → next spanning full-width separator; absorb sub-blocks. (Only JG R67 across all runs.)
Containment-merge (block ≥90% inside a larger one → fold in). Fixes split columns, roundup sub-stories, orphan tails.
Header-split (H block with a dateline >20px ABOVE its headline → split at headline). Only JG R28, R64.
Ceiling-trim (DL block, leading furniture >300px above the dateline → drop). Only JG R2.
Body-less header merge (H or P, no text, sits ≤90px above a text block, ≥60% overlap → fold). H: NT R65→R73. P (iter-1): NT R133→R132, JG R52→R4.
Right-edge → page (rightmost-column block → extend right edge to page-content edge). Touches only rightmost blocks (AJ 4/6/3/4, NT 2/3/3/8), ≤15px; interior untouched.
Drop classifieds (drop_classifieds_regions, runs right after drop_masthead_regions). Classifieds = the "Classified Ads / phone-number listing" columns = page FURNITURE, not news. Left in, a neighbour article's column-fold grows down into them (JG R65 swallowed the whole left column to page-bottom). A region is classifieds when (only TALL regions ≥400px are OCR-scanned) its OCR text carries a classifieds keyword (classified/contact) OR ≥5 phone-like \d{6,} runs (a news article rarely has >3). OCR is half-res eng-only — the phone-runs + English keyword are the real signals, so this is ~2× faster than full-res tel+eng and gives the IDENTICAL drop set on the corpus. Court/legal notices are intentionally NOT dropped (user: classifieds ONLY) — they are carved into their own section instead (next).
Carve court/legal notices (tag_legal_notices + a carve pass in cluster_all_article_blocks). Court summons / name-change & matrimonial petitions / "by order of the court" notices are page furniture too — but, per the brief, KEPT and cropped as their OWN section (kind L), not dropped and not absorbed into a neighbour (JG R65 swallowed a court summons in its left column). tag_legal_notices half-res-eng-OCRs single-column text/paragraph_title regions and tags r['_legal']='core'|'support' on a legal-boilerplate keyword (CORE: hon'ble, in the court of, by order of, plaintiff/defendant, petitioner/respondent, summons, o.s.no, civil judge … ; SUPPORT: advocate, whereas, vakalat). The clusterer then groups adjacent tagged regions and carves a cluster into its own LOCKED L block only when (1) ≥2 tagged regions incl. ≥1 CORE, (2) they all sit in ONE host block, and (3) that host is WIDE (>1100px → masked). Gate 3 is the safety rule: a wide masked host whites the carved area out (no duplication), whereas a NARROW host is cropped verbatim, so carving out of it would duplicate the notice — those are left alone (e.g. NT pg3's petition notice straddling the narrow locked #75 + open #62 is NOT carved, so the NT baseline is untouched). The carved L block shows in BOTH the per-article crops and the debug boundary map (the carve lives inside cluster_all_article_blocks, which every consumer re-runs).
Type-2 split (split pass in cluster_all_article_blocks, after the legal carve). TWO layouts look like one big box but need OPPOSITE treatment (AJ/Namaste = ruled papers):
- Type 1 — sub-stories under ONE header with a COMMON BOTTOM rule → the rule + wide banner already made it a _raw_rect section (banner/section pass) → crop as ONE rectangle. NEVER split.
- Type 2 — independent articles stacked one below another, EACH with its own doc_title + its own dateline, NO common bottom rule (JG R65: కలెక్టర్/సుందరయ్య/పోస్టుమార్టం) → over-merge → SPLIT into one article per headline (members assigned to the nearest headline at/above them). Discriminator = the _raw_rect flag (set ⇒ sure bottom rule ⇒ Type 1). The split anchors are headlines that OWN a dateline (each dateline assigned to the nearest headline above it) — a single article with a headline + a sub-headline sharing ONE dateline has only ONE owning headline, so it is NOT split (this fixed the AJ R50/R57 & R35/R79 false splits). Anchors must be vertically stacked ≥120px apart (not two columns headed side-by-side). NOTE for line-less papers (Sakshi): the _raw_rect bottom-rule signal won't exist — a different bottom cue would be needed.

Iteration log

iter-1: body-less header merge extended to paragraph_title → NT R133→R132, JG R52→R4 (title-only fragments fixed). AJ untouched; NT pg4 26→25, JG pg1→9.
iter-2: drop graphic-label orphans (pure-text O block ≥80% inside a FOREIGN image) → JG R34 dropped (label inside banner image). AJ/NT 0 drops; JG pg1 9→8.
iter-3: orphan-TEXT absorb (fold pure-text orphan UP into clear above-article, gap≤80, no down-folds) — TRIED & REVERTED. Only match was NT pg4 R75→R132, a CLASSIFIEDS "change of name" notice; could not verify R75 belongs to that ad vs a separate notice → merging distinct classifieds is a regression risk. Deferred. (AJ/JG had 0 matches anyway.)

Assessment after iter-3 — safe heuristics largely exhausted

The remaining backlog items are risk-bearing or need human judgment, not safe heuristics:

Orphan PHOTOS (JG R25, AJ R12/R26): binding a standalone photo to "the right" article is ambiguous (far gaps, multiple candidates).
Orphan TEXT in classifieds/notices (NT R75, etc.): can't tell same-ad vs separate-ad.
JG front-page over-merges (R64/R65): dense multi-headline roundups; further splitting is complex and could fragment legit roundups.
JG H28 L-shape white space: a generic ragged-bottom split also moves AJ R48/R53 (fine now). Recommendation: pause autonomous loop; these need a human call per case. Resume with a specific directive if desired.

Open defect backlog (prioritized for next iterations)

Orphan photo/text blocks (kind O) — REMAINING: AJ pg1 R12 (photo+cap+text), pg1 R76 (text), pg3 R32 (text), pg3 R26 (img+text); NT pg3 R7 (text tail, 36px below H98), pg4 R75 (text, 14px below P132); JG pg1 R25 (img-only photo). [R34 done in iter-2.] Next safest: fold orphan TEXT into the article directly above/below within a small gap (NT R75→P132 gap14, R7→H98 gap36) — measure first (risk: wrong-article merge). Harder: bind orphan IMAGE (R25) to caption+nearest article.
JG front-page over-merges R65 (4DL), R64 (residual 2DL below headline) — dense section roundups; may need below-headline multi-headline split.
JG H28 L-shape white space — needs a signal that distinguishes it from AJ R48/R53 (which are fine). Candidate: only when the empty corner is masked-out foreign article AND the block's tail is a single-column sidebar with its own sub-head/byline → split the sidebar.

Hard limitation: undetected headlines (OCR gap, not a cropping bug)

On some pages PaddleOCR detects no doc_title for the top articles (JG pg1: only 3 doc_titles total, ALL in the bottom half; top-half headlines భానుడి భగభగ / Indiramma / కరీమాబాద్ were missed entirely). A geometric crop cannot include a headline that has no region. Symptoms: DL-anchored crops with the heading missing (JG a01_DL3, a03_DL4) and lead photos orphaned from their headless article (JG a02_O25). The REAL pipeline recovers these via the paid Claude "catch by eye" pass (pipeline log: "N article(s) Claude must catch by EYE — no headline box"). The geometric all_article_crops (real run AND replay) will lack them. The only geometric fix is a NEW pixel-level headline-band detector (find the large-type band above a headless dateline) — substantial + risky for AJ/NT baselines; deferred unless explicitly requested.

Replay validation cheatsheet

python3 _replay_all.py        # AJ → all_article_crops_final
python3 _replay_namaste.py    # NT Siddipet → all_article_crops_final
python3 _replay_jangaon.py    # NT Jangaon → all_article_crops_review

Read-only block inspection: replicate the chain (dedup → merge_stacked → drop_masthead → promote_paragraph_titles → mark_bullet_titles → find_article_starts_by_dateline → [probe cluster → recover_orphan_text_headlines → re-detect] → cluster_all_article_blocks).

Partition passes added (post iter-2) — all gated + validated

Seed/lock confident cells — a block is _locked (confident) when it's a single complete article (exactly 1 dateline + a headline) OR fully boxed (≥3 ruled sides). Over-merges stay open.
Step 4 — grow OPEN cells into gaps (bounded by walls/cells/frame; locked never move; strict guard: never increase overlap with a locked cell). AJ/NT: 0 open∩locked overlaps.
Step 5 — containing-assignment — every unassigned region inside a cell's box joins it (no box growth). Cleared NT pg4's 11 section-absorbed regions; counts unchanged.
No-overlap reconciliation — trims a 0-member over-reaching box back to its members.
Lead-photo rebind (caption-anchored) — a PHOTO→CAPTION→HEADLINE stack = lead photo; move the photo+caption down into the headline's block. Fixes JG a06_P37/maize R29 + AJ pg4/NT lead photos. The caption requirement excludes the H93 false positive (photo at an article's bottom).
Horizontal-line LENGTH FLOOR (_lines.separator_barriers) — a horizontal rule counts as an article separator only if width ≥ 0.6× column (median text width). Drops short caption/decorative underlines. NOTE: ≥1.0× is TOO aggressive (a single-column article's own boundary is <1 col → ≥1.0 merged NT pg4 R121); 0.6× merges ZERO articles. Caption-vs-separator is still decided primarily by "what's directly above the line" (photo/caption) — length is a secondary floor. A rule below a caption ≈ caption width = caption underline (dropped); only a rule much WIDER than the caption above it (≥1.5×) is kept as a boundary.

Tried & REVERTED (kept for the record)

Orphan-TEXT absorb — only match was a NT classifieds box (R75/R132); couldn't verify same-ad vs separate-ad → regression risk. Deferred.
Ragged-bottom sidebar split — landed for JG H28 (split the రాత్రికి రాత్రికే sidebar so the L-shape white space cleared); later the lead-photo/grow cascade made H28 tile both quote boxes with no white space anyway. (Gate: main part must stay multi-column → AJ R53 never split.)
Below-divider cut (cut content sitting below a vertical divider's END out of the right-column article above — JG DL2 holding the maize wide photo R30). Detection is SAFE (straddle gate ⇒ only JG R2; AJ/NT untouched), but the repair destabilised JG (merged into wrong neighbour R49; emptied DL2 regrew). Deferred — needs correct merge-target (the dateline governing the band) + a "don't regrow a just-cut block" guard.

RUN-WIDE VALIDATION (all 7 output runs, geometric, no API) — 2026-06-09

Ran the full cluster chain on every run in output/ and checked counts + 2+dateline over-merges.

AJ Siddipet ×3 (incl. 2 never-loaded runs): 12 / 18 / 14 / 20 on all three, 0 over-merges.
NT Siddipet ×3 (incl. 2 never-loaded runs): 11 / 22 / 19 / 25 on all three.
NT Jangaon: 7 / 12.
Every 2+dateline block is an intended ROUNDUP (verified): NT pg2 [110,127], pg3 [54,60,62], pg4 [67,121] (R121 = a raw_rect section: headline + 2 dated sub-stories + photos, same accepted pattern as R67), JG [64,65] (front-page section roundups).
New runs match the baselines exactly ⇒ rules generalise (not overfit). No crashes, no new over-merges, no merged articles from the length floor. Conclusion: the horizontal-line + caption handling and the partition/lead-photo passes are validated across the whole corpus.

Classifieds-drop validation (end-to-end replay, no API) — 2026-06-09

Wired drop_classifieds_regions into the pipeline (after drop_masthead_regions) + all 3 replay scripts, then re-ran every replay:

AJ: 12 / 18 / 14 / 20 — baseline EXACT, 0 classifieds dropped (no false positives).
NT Siddipet: 11 / 22 / 19 / 25 — baseline EXACT; roundups [110,127] / [54,60,62] / [67,121] all intact; 0 classifieds dropped.
NT Jangaon 130635: pg1 = 7, pg2 = 13 (was 12). The only intended change: R65 shrank from [113,1958,1977,6279] (page bottom, 4 datelines, swallowing the phone-listing classifieds) to [113,1958,1977,5113] (3-DL front-page roundup). Classifieds R87 (y5143–6267, phones=8) dropped; the freed left-column content below becomes its own block (12→13). a06_H65 renders as a bounded roundup, no longer page-tall over the classifieds.
OCR cost addressed: detector now OCRs each tall (≥400px) region at half-res eng-only (was full-res tel+eng) — ~2× faster per call, verified to produce the IDENTICAL drop set (R87 still phones=8/kw=True; 6 sampled news bodies still 0/False).
NOTE: R65 still includes a court/legal notice stacked in its left column. Per the user's "classifieds ONLY" directive this is intentionally left; separating court notices is deferred.

Court/legal-notice carve validation (end-to-end replay, no API) — 2026-06-09

Follow-up directive: "include court/legal notices as box/section in populating the crops + debug img." Added tag_legal_notices (pipeline + 3 replays, after classifieds) + the carve pass in cluster_all_article_blocks, then re-ran every replay:

JG 130635 pg2: 13 → 14 crops. The court summons (R51/R89/R101/R108/R93) was carved out of R65 into a new a08_L89 block [118,2887,595,4080] (5 regs) — a clean, complete notice ("IN THE COURT OF THE HONBLE JUNIOR CIVIL JUDGE … BY ORDER OF THE HON'BLE COURT … ADVOCATE"). R65 dropped 30→25 members and whites the carved area out (verified in a06_H65 — no duplication). The debug boundary map shows it as LOCK #89 [L] (5), nested in #65. pg1 = 7.
AJ: 12 / 18 / 14 / 20 — baseline EXACT, 0 legal carves.
NT Siddipet: 11 / 22 / 19 / 25 — baseline EXACT, 0 legal carves; roundups intact. NT pg3 has a real petition notice (R47/R79/R89) but it straddles the NARROW locked #75 + open #62, so Gate 3 (wide-host-only) correctly SKIPS it → baseline untouched.
Detection sweep (read-only) across the corpus: legal keywords flagged ONLY the JG court notice and the NT pg3 petition; 0 false positives on any AJ/NT news region.
COST: tag_legal_notices OCRs ~40–70 single-column text regions/page (the expensive part). Acceptable for the paid pipeline (Claude vision dominates), but a candidate for a cheaper pre-gate (e.g. skip dateline-anchored regions) if it becomes a bottleneck.
DEFERRED (harder, needs the seed-before-clustering model): a legal notice that straddles a NARROW straight-cropped article (NT pg3) — to carve it cleanly the host article's box must be shrunk away from the notice, which can't be done by masking alone.

Type-2 split validation (end-to-end replay, no API) — 2026-06-09

Directive: Type-1 (shared header + common bottom rule) = ONE rectangle (fine); Type-2 (stacked articles, own headline+dateline each, no common rule) = SEPARATE articles. Added the split pass.

JG 130635 pg2: 14 → 16 crops. R65 (the კალెక్టర్ over-merge) split into 3 clean articles: R65 కలెక్టర్ [113,1958,1975,2859], R86 సుందరయ్య [123,2894,1977,4292], R112 పోస్టుమార్టం [133,4325,1975,5113]. Court notice R89 [L] still its own section. pg1 = 7.
AJ: 12 / 18 / 14 / 20 — baseline EXACT (after the dateline-OWNERSHIP fix; the first cut wrongly split AJ pg1 R50/R57 and pg3 R35/R79 — headline+subhead sharing one dateline — 13/15, now fixed).
NT Siddipet: 11 / 22 / 19 / 25 — baseline EXACT. The 2+dl roundups [110,127]/[54,60,62] are single-header-multi-dateline (Type 1) so they have <2 dateline-owning headlines → NOT split; [67,121] are _raw_rect → excluded. All correctly preserved.
KNOWN minor issue: R65's left column held a furniture stack (court notice + classifieds logo + a small job ad + a గమనిక notice). The court notice is carved (L89) and the main classifieds (R87) dropped, but the leftover logo (R78 image) bleeds into R86's crop and the job ad + గమనిక bleed into R112's crop (they're <400px / images, below the classifieds-drop gate). Secondary cleanup — extend furniture detection if needed. [RESOLVED below: furniture section.]

Furniture section (Type-2 split) — column-aware member assignment — 2026-06-09

Directive: when splitting a Type-2 stack, LEAVE classifieds / notices / unrelated content OUT of the article crops, but KEEP that section visible on the debug empty-canvas map. Implemented geometrically (no OCR) inside the split: each member is assigned to the nearest headline at/above it ONLY IF it also sits within that headline's COLUMN x-span (±0.3 col). A member that lands in a headline's y-band but a DIFFERENT column is unrelated content sharing the page — it falls out as LEFTOVER and is grouped into a _furniture section block. Furniture blocks are: (1) EXCLUDED from article crops — crop_all_article_blocks skips saving them, and because each article sub-block's bbox is rebuilt from its in-column members only, the furniture column isn't even inside the article rectangle; (2) DRAWN on the debug boundary map as a hatched GREY "FURNITURE #id" box; (3) NEVER saved as a crop. The split fires only on JG R65, so this touches ONLY that page — AJ/NT untouched.

JG pg2: R65 → R65 కలెక్టర్ + R86 సుందరయ్య (x682, was x123 w/ the classifieds logo) + R112 పోస్టుమార్టం (x682, clean) + R89 [L] court notice + F44 furniture [123,4139,626,5113] (classifieds logo + job ad + గమనిక — debug-only, not cropped). 16 saved crops; boundary map shows the FURNITURE box. a09_H86 & a13_H112 verified clean (no furniture bleed).
AJ 12/18/14/20 and NT 11/22/19/25 — baselines EXACT.

Type-2 split — paragraph_title anchors + ';' dateline + section discriminator — 2026-06-09

Two more JG pg2 over-merges were reported (R97 = గుండె చికిత్స + మృతుడి పరామర్శ; R64 = జాతీయ రహదారి + ఉపసర్పంచుల చెక్ పవర్). Three root causes, three fixes:

2nd headline detected as paragraph_title, not doc_title (R58, R84) → my split's anchors were doc_title-only. FIX: anchors = doc_title OR paragraph_title, with COLUMN-AWARE dateline ownership (each dateline → nearest headline above WHOSE x-span covers it; a caption in another column can't steal it). Captions/sub-heads still own no dateline ⇒ never an anchor.
2nd dateline's colon OCR'd as a semicolon (R97: 'చిల్పూరు, మే 19 ;') → _dateline_in_text required ':'. FIX: accept ';' too (guards: digit + short prefix + no sentence punctuation).
R64 was _raw_rect (banner-rectangle extended the wide headline to the next banner, NO spanning rule) so the split skipped it. FIX: the split's Type-1 exclusion is now the _section flag (set ONLY by the section-banner pass, which requires a real spanning bottom rule), not the coarse _raw_rect flag. R64 (no rule) → splits; R67/R121 (real section) → stay one rectangle.

JG pg2 16→18: R64→R64+R84, R97→R97+R58 (verified: each crop is one complete article).
Side effects, all VERIFIED correct (same rule, genuinely distinct articles — not fragmentation): NT pg2 22→23 (అంగన్‌వాడీ R127 off the ప్రజావాణి section), NT pg3 19→20 (పర్యావరణ R60 off), AJ pg4 20→21 (గజ్వేల్ R12 — a real dateline the ';' fix recovered; an existing pass then split it). R110 (ప్రజావాణి) stayed whole as a _section ⇒ the discriminator protects real sections.

Orphan-PHOTO binding — an image may never stand alone — 2026-06-09

RULE (user): an article may be orphan, but an IMAGE may not — every photo must belong to some article. A final pass in cluster_all_article_blocks binds each ORPHAN-PHOTO block (no doc_title/paragraph_title headline AND no dateline; image-only, OR image + caption only, OR image-area ≥50%) to the best NEWS article (furniture/legal blocks are excluded as targets): the article whose top is just BELOW the photo in the same column (lead photo), else just above, else the nearest column-overlapping article. A TEXT/body orphan is LEFT ALONE.

Sakshi pg1 23→22, pg2 27→20: every orphan photo bound (R28→R78 lead photo verified — the 'collector inspecting BL-1' photo + caption now sit in R78; pg2 R38/R42/R48/R52/R60/R63 all bound). 0 image orphans remain on either page. The text orphan R18 (pg1) correctly stays.
JG pg1 7→6: orphan photo R25 bound into its article.
AJ 12/18/14/21 and NT 11/23/20/25 — UNCHANGED (no qualifying image orphans on those).
NOTE: Sakshi itself has rough article structure (headlines/datelines under-detected, no rules) — some bound photos land on a column-overlapping article that may need per-case review; but the rule (no floating image) now holds. Deeper Sakshi clustering is a separate task.
Respect horizontal separators in the photo bind (refinement): a photo binds to an ADJACENT article (gap ≤200px) even if a rule sits between them — that rule is the article's own headline border and the photo is its lead photo (JG R28→R78, 7px gap). But a FAR fallback bind is REJECTED if a horizontal rule LONGER than a caption (>0.6 col) lies between the photo and the article — the user's cross-paper line rule. This kept the Sakshi pg2 "student photo" R52 (top of page) from binding down through two FULL-WIDTH rules + a page of white space into R118 (the భక్తులకు/ Kaleshwaram story); R52 now binds to R127 in its own band, and a08_H118 is clean. Verified: Sakshi pg1 22 / pg2 20 (0 image orphans); AJ 12/18/14/21, NT 11/23/20/25, JG 6/18 all unchanged.

Sakshi page-1 deep fix round — 2026-06-10 (visual review of all 22 pg1 crops → 14 BAD)

A crop-by-crop visual audit of Sakshi JG pg1 found 14/22 BAD crops. Root causes & fixes, ALL Sakshi-gated (AJ 12/18/14/21, NT 11/23/20/25, JG 6/18 re-validated EXACT after every change):

Masthead drop for Sakshi (drop_masthead_regions, paper=="sakshi" branch, every page): drop every region that ends ABOVE the topmost headline-type region (sanity: that line must be in the top 30%). pg1: weather box R64, photo-brief R62/R59, nameplate R63, and the page-date strip R71 — which OCR'd as a DATELINE and seeded a fake DL71 article. pg2: R68/R71/R84/R104.
detect_sakshi_headline_bands (called after promote_paragraph_titles, pipeline + replay) — the big one. Sakshi's display headlines are often NOT boxed by PaddleOCR at all. Two passes:
1. WIDEN: a doc_title grows sideways over adjacent UNCOVERED large-ink columns — its un-boxed kicker half (R52 'అనూహ్య స్పందన' + red kicker 'రిబేట్ ఆఫర్‌కు'). Widest titles first; extensions join the coverage map so a narrow neighbour can't steal a wide headline's kicker. Ink within 25px of a foreign region's edge is the neighbour's own glyph overflow — ignored.
2. SYNTHESIZE: an uncovered big-type band (55–280px tall, ≥0.75 col wide, ink 5–60%, text following below, never above the first real headline) becomes a synthetic doc_title (_synth_band) anchor. Found BOTH missing pg1 banners (lift-1 'జూలై 31...' R99 and SIR 'మ్యాపింగ్‌లో జిల్లాకు అగ్రస్థానం' R100) and pg2's గమనిక box R139. CRITICAL detail: two headlines can share pixel rows with only a ~50px gutter (same size as a word gap) — a gap splits a band run ONLY if it contains a known text-column LEFT edge (≥35px wide); plain gap-size segmentation merges both banners into one anchor (measured: pg1 12 crops, 45-member monster).
Box-edge rule disarm (_is_box_edge in _barrier_between, Sakshi only): a horizontal rule with a PARALLEL partner 80–400px away (≥70% overlap) is a quote/bullet-box border, never a story separator. Without it the lift-1 bullet-box borders fenced the article's own columns off.
Photo-bottom border disarm (_in_photo, Sakshi only): a rule within 14px BELOW an image, horizontally inside the photo span ±25px, is the photo's frame/caption rule (PaddleOCR's box ends a few px above the printed border, so the interior test missed it). Without it R61/R46/R23 (lift-1 caption + col-3 body) fell into the story below.
Caption never a headline line (headline-stack recovery, all papers — baselines unaffected): a figure_title directly under an image (≤90px) is a CAPTION; stop the stack there. Without it the synth SIR headline pulled the Indiramma photo's caption R69 down as a "wrapped line".
No lead-photo rebind into a _synth_band headline: a synth band's lead photo is BELOW it by construction; a photo+caption ABOVE it belongs to the story above (R36/R69 stay with R77).
Continuation-column fold (Sakshi only, end of clusterer): a TEXT-only orphan block directly beside an article block with the SAME vertical band (top ±90px, ≥80% overlap, gap ≤80px, no vertical rule between) is the other column of the same box → merge (pg1 R18 → R79; the text even continues mid-word 'విద్యా సంస్థ|లకు' across the two).
Stray-image re-home (Sakshi only): an image member vertically DETACHED from the rest of its block but fully inside ANOTHER article's box moves there. A 3-column synth banner column-overlaps everything below it, so R50's tiny సాక్షి-ఎఫెక్ట్ clipping R44 (whose own narrow headline does NOT column-overlap it) had fallen to the banner 1100px above.
Narrow-crop masking (crop_all_article_blocks, all papers): narrow (≤1100px) crops now white out regions OWNED BY OTHER blocks (gpad=2) before the ink-trim — kills the sliced-glyph slivers at crop edges. Un-owned ink is left alone (it may be the crop's own un-boxed headline). Members are repasted, so nothing of the article itself can be erased. Counts unchanged on all papers.
Review overlays: crop_all_article_blocks(..., overlay=True) (used by _replay_sakshi.py) draws member boxes on the review crops — blue=anchor, magenta=synth band, green=member.
RESULT (Sakshi JG pg1): 22 → 16 crops, every reduction a genuine reunification. Visual re-audit of all 16: 14 GOOD / 2 MINOR / 0 BAD (was 8 GOOD / 14 BAD). The rebate feature package (was shredded into P94+H52+DL71+DL11 with sliced letters) is ONE crop; lift-1 and SIR-mapping (whose headlines have NO region) are complete articles anchored by synth headlines; H77 is exactly the Indiramma story; H54/H78/H55 no longer hold neighbours' content. pg2: 20 → 17 (top photo-gallery orphans folded into their stories; not yet visually audited).
NEW Sakshi baselines: pg1 → 16, pg2 → 17.
Known remaining (MINOR): a01_H55 loses the unboxed last line of its మ్యూజియాలజీ sub-brief (no region covers it; the next block's white-out grazes it); the EAPCET ad image R33 sits mid-article in H55 (it IS physically inside the article band; ad detection deferred).
Known remaining (pg2, PRE-EXISTING): the Type-2 split's column-aware leftover pass marks the విద్యార్థిని ఆత్మహత్య article as FURNITURE (R4/R10 F) → it is NOT cropped at all. Present in the pre-fix baseline too. Needs its own round. [RESOLVED in the pg2 round below.]

Sakshi page-2 round — 2026-06-10 (visual audit of all pg2 crops; user-reported defects)

Fixes, validated after each change (AJ 12/18/14/21, NT 11/23/20/25 incl. 2+dl=[110]-only on pg2, JG 6/18 — all EXACT):

a04_H118 body truncated at the column boundary (col5 ends mid-word 'నిర్మా-', col6 holds 'ణం…'): the col6 body R20 was stolen by the headless dateline R14's GROW-UP, which climbed PAST the R14 story's own subhead R121 because _is_caption lets a 197px two-paragraph text through (≤200px + a photo 82px above). FIX (all papers, baselines unchanged): grow-up STOPS after claiming a paragraph_title — the heading IS the article's top; nothing above it is ours.
విద్యార్థిని ఆత్మహత్య never cropped (whole story = furniture): its heading R88 IS detected (paragraph_title) but the R41 Type-2 split's anchors don't span col4 → all 8 regions fell out as LEFTOVER → furniture. FIX (all papers; gates keep AJ/NT/JG exact): a leftover group that LEADS with its own COLUMN-WIDE heading (≥0.6 of the group width — a narrow WANTED label stays furniture) and has body text → emit as an ARTICLE (kind P), not furniture.
a13_H93/'విషాదయాత్ర' cut mid-sentence, its tail + tributes in the girl's crop: the col4 leftover zone holds TWO stacked items (girl's story y3658-4643, శ్రీనివాస్ tail y4723-5938) with a separator rule at y4679 between them. FIX (Sakshi only): (1) leftover clustering never merges across a separator rule; (2) a heading-less leftover group whose band MATCHES an adjacent-column split article (top ±90px, ≥80% overlap, gap ≤80px) is that article's column CONTINUATION → its members merge there (the tail's text literally resumes H93's "...అయితే వైద్యులు" mid-sentence). ORDER MATTERS: the wide-heading→ARTICLE check runs FIRST, else the girl's story (gap-close to H99's band) band-attaches into H99. Both passes are Sakshi-gated: un-gated they re-shuffled an NT pg2 leftover into R105 (2+dl-blocks [110]→[110,105]) — caught by the 2+dl indicator and re-gated.
a03_P129/a05_H91: dead farmer's portrait R47 + caption R115 stuck at the briefs column's bottom, white hole in his own story's crop: the stray-image re-home pass couldn't see R47 as detached because its caption R115 sat below it inside the same block. FIX: re-home treats an image + its caption (figure_title ≤90px below, ≥40% overlap) as ONE UNIT — excluded from the envelope test and moved together (→ H91, inside whose box they sit).
గమనిక notice box synthesized as a headline (R139, its own bogus crop): a filled red header measures ink 0.51; real Sakshi display headlines measure 0.30-0.35. FIX: synth band ink ceiling 0.60 → 0.45; the notice text returns to the classifieds-column furniture (not cropped).
RESULT (Sakshi JG pg2): 20 → 17 crops. Visual audit of all 17: 11 GOOD / 4 MINOR / 2 BAD → after the H93/P88 boundary fix the BAD pair re-audited GOOD. The H118 two-column body, the విద్యార్థిని article (new crop P88), H91's portrait, and the P129 briefs column all verified.
Known remaining (pg2, magazine-style kids/summer feature band only): H82/H127 share the gallery collage and H82's story conclusion ('–దుగ్గొండి' sign-off + 'నీతి' moral box) renders inside H127's crop — a collage-page attribution problem, low priority for the news pipeline. The classifieds crops (P136/P111) bundle neighbouring ads and clip one ref number (MINOR).

POLITICAL crops now BLOCK-SNAP to the geometric blocks — 2026-06-10

crop_political_articles used to re-derive each Claude-selected article's rectangle from the raw member_region_ids union plus ~400 lines of parallel ad-hoc geometry (banner walls, foreign-headline clips, floors, grow-right) — none of which carried the geometric-crop fixes. Now it BLOCK-SNAPS:

It clusters the page once (cluster_all_article_blocks) and renders every block crop via crop_all_article_blocks into a temp dir — the exact validated pipeline (furniture drop, un-boxed headline recovery, neighbour masking, box-edge rules, continuation folds).
Each political article is mapped to the geometric block owning the MAJORITY (≥50%) of its member regions; that block's crop is shipped as article.png (method: "smart_extractor_blocksnap", info.json carries the block's full member list).
A block already claimed by an earlier article on the page is not re-used (no duplicate crops); any article that fails to map falls back to the legacy path unchanged.
The pipeline call passes dateline_starts + sep_lines through so the clustering matches the all_article_crops output exactly.
VALIDATION (no API — _replay_political.py replays saved page_NNN.political_result.json member lists through the new path): Sakshi JG pg1 2/2 snapped (the rebate package + DL5, both verified visually as the block-quality crops); AJ Siddipet pg2/3/4 4/4 snapped (10-13 of 12-13 member votes, incl. the R48/R33 banner blocks); NT Jangaon pg1 2/2 snapped (13/13, 4/4). ZERO legacy fallbacks needed on the corpus.
NOTE: replayed member lists come from OLD paid runs (stale region understanding) — e.g. the Sakshi 'Transfers delayed!' entry maps to the rebate block because its old members [66,29] sit there now. On the next paid run Claude selects members against the CURRENT regions (incl. the synthesized Sakshi headlines, which are in regions.json and the prompt's region list), so the member→block vote will be consistent end-to-end.

New-run defects round (Sakshi_Jangaon District_20260513, first PAID run on new code) — 2026-06-10

User-reported bad crops on the fresh paper; three more Sakshi-gated passes (validated: old-run Sakshi pg1 16 / pg2 17 UNCHANGED; AJ 12/18/14/21, NT 11/23/20/25 [110], JG 6/18 all EXACT):

pg2 a02_H106 was a 695×5055 column strip (statue story + 4000px of masked white): TWO causes. (a) The dance-photo caption R104 (page top) fell down-column into H106 — its photo R44 lived in another block. FIX: stray caption re-home — a figure_title member moves to the block that owns the image it abuts (±90px above/below, with -30px overlap tolerance: PaddleOCR's photo box overlaps the caption strip by a few px, R104 vs R44 = -11px, which defeated a 0-gap test on the first attempt). Donor block shrinks back to its members. (b) The aviation story's heading is a paragraph_title 1200px below the statue story's doc_title in-column → never promoted → whole stack merged. FIX: stacked-story heading promotion — a pt ≥0.55 col wide with content directly beneath (≤150px) and the nearest in-column base anchor >900px above becomes an anchor. The gate was first 600px and broke the old-run rebate package (its sub-head R94 sits 614px under the widened R52 headline and re-split, whiting R88's box over the stats infographic — caught by re-validating the old run); 900px keeps feature sub-heads folded and still promotes genuinely stacked stories.
pg2 a08_H100 stretched ~900px up with a foreign fragment above its own headline (the NPDCL story's tail folded down-column into H100). FIX: stray text re-home — a plain-text member of an H block lying entirely ABOVE the anchor headline's top is never that article's content (kickers/decks are title-typed); move it to the smallest other article box containing it.
Verified visually: new-run a10_H106 (639×1326, exactly the statue story), a09_H100 (complete, no stray top), old-run a02_H52 (infographic numbers 15,941/5.79/11,376/3.72/1,00,20,087 intact).
New-run pages have NOT had a full visual audit (only the reported crops + block maps checked); pg1 = 14 crops / pg2 = 16 on replay.

NT Adilabad flood-special front page — NT rule split + late structural cut — 2026-06-10

NamastheTelangana_Adilabad_20260528 pg1 produced 5 page-scale OVERLAPPING mega-blocks. This page is DETECTION-LIMITED at the source: 65 regions only, 4 doc_titles for ~8 stories, and BOTH big feature headlines ('రెక్కల కష్టం.. నీళ్లపాలు', 'వడ్ల తరలింపు ఎక్కడిదాకా?') are INSIDE PaddleOCR image boxes — invisible to every geometric pass incl. the synth-band detector (coverage test). With anchors missing, the image-must-belong bind + grow/merge built page-tall blocks. Fixes:

NT horizontal-rule split enabled (was Sakshi-only): same machinery, but only at rules ≥1.8 col (per the line study a ≥2-col rule is an article boundary on NT; the boxed grid's ~1-col caption/box rules must not cut). _replay_adilabad.py added (NTA_RUN env overridable).
NT LATE structural cut (end of clusterer, after grow/merge/photo binds): the mid-pipeline split runs BEFORE the passes that built the mega-block, so DL5 [668,901,4084,6302] was never re-checked against the 4.4-col rule at y=4764 (and at mid-split time the bottom band's R30 headline wasn't yet a member, so the new-start gate failed). Final blocks (not locked/raw_rect/ section/furniture) are re-cut at ≥1.8-col rules with the underline guard; a rule ≥2.2 col cuts even with no detected start below. DL5 → DL56 (the whole flood feature, verified coherent) + H30 ('తడిసిన వడ్లు కొనాలి' bottom band, verified clean). pg1 5 → 6 crops.
VALIDATION: NT Siddipet 11/23/20/25 EXACT (2+dl [110],[54,62],[67,121]); NT Jangaon 6/18 EXACT; Adilabad pg2 31 crops unchanged (no shredding). AJ/Sakshi untouched (gate).
HONEST REMAINDER: H44 still spans the page top (left story + photo row + nameplate share the band; the 60%-block-span guard correctly refuses partial rules on a page-wide block) and the H59/DL9 boxes still overlap neighbours. Only a better layout model (headline bands inside photo boxes) or the paid pipeline's Claude-vision pass can fully fix this page. NT masthead drop never fired here (nameplate is INSIDE image R46, no header region) — benign, the strip rides atop the photo-row crop.

⚠ REPLAY-TOOLING BUG (found 2026-06-10, fixed): _ACTIVE_PAPER was never set for NT

_replay_jangaon.py / _replay_namaste.py never set se._ACTIVE_PAPER, so every prior "NT validated EXACT" run measured with ALL paper-gated clustering passes OFF while the real pipeline runs them ON. (AJ has no gated passes, Sakshi's replay always set it — only NT was affected.) Both scripts now set it; _replay_jangaon.py takes JG_RUN env (default = the 130635 run; the user-facing run of the same paper is ..._20260608_124720). FIRST true gated-NT measurement: NT Siddipet 11/23/20/25 + 2+dl [110],[54,62],[67,121] EXACT (gated passes are no-ops there ✓), JG both runs 6/18, Adilabad 6/31.

NT Jangaon pg1 (user-reported: a01_DL3 white holes, a04_H28 over-merge, a05_DL2 foreign photo)

Root cause is the same family as Sakshi/Adilabad: THREE un-detected display headlines (brush-font 'భానుడి భగభగ', script 'నమ్మిండు..', DL2's heading). The heat feature's lower content (vendor + ice-cream photos, captions, continuation column) folded DOWN to the 'ఆలస్యంపై రైతుల ఆగ్రహం' headline 2200px below (nearest-anchor metric); the white holes in a01_DL3 were those stolen regions masked out. Fixes (gated in ("sakshi","namaste_telangana") — AJ untouched):

Re-home family enabled for NT (stray image+caption unit, stray caption, above-own-headline members). The above-headline re-home is GENERALISED from text-only to text/image/figure_title (title-typed members stay — kickers/decks are legitimate above a headline).
Synth headline bands enabled for NT — FRONT PAGE ONLY (n == 1): inner NT pages are dense and well-detected; un-gated synth split the protected Siddipet pg4 roundups (25→28, caught). Band height cap 280→400 (tall display/script faces; ink-fraction still rejects photos).
RESULT (JG 124720 pg1, counts unchanged at 6): DL3 = the COMPLETE heat feature (mem 8→13, no white holes, verified visually); H28 starts AT its own headline [y3089, was 838] = exactly the ఆగ్రహం article (verified); DL2 = clean single-column committee article, its foreign tractor photo+caption re-homed to R29's procurement story (verified).
Residual: DL2 remains heading-less (its display heading has no region and the synth band gates didn't admit it) — hard-limitation class; the heading pixels sit above the crop.

44 KiB Raw Permalink Blame History Unescape Escape