sundeep-news-scan/CROP_STRATEGY.md

# Article-Cropping Strategy & Learnings (self-improving loop)

Living doc for the geometric article-cropping in `smart_extractor.py`
(`cluster_all_article_blocks` → `crop_all_article_blocks`). Updated each loop iteration.

## Validation rules (NEVER violate)
- **Never run `smart_extractor.py`** — it calls the paid Claude API. Validate ONLY with the
  geometric replay scripts: `_replay_all.py` (AJ), `_replay_namaste.py` (NT Siddipet),
  `_replay_jangaon.py` (NT Jangaon). These run the full cluster chain with NO API.
- **Measure read-only BEFORE editing** (blast-radius of any new rule), then **re-validate after**.
- **Baselines** (block counts) — keep stable unless an improvement is intentional & documented:
  - AJ: 12 / 18 / 14 / **21** (pg4 20→21: the ';'-delimiter fix recovered a real గజ్వేల్ dateline R12)
  - NT Siddipet: **11 / 23 / 20 / 25** (pg2 22→23: అంగన్‌వాడీ R127 split off the ప్రజావాణి section R110;
    pg3 19→20: పర్యావరణ R60 split off; both are the Type-2 paragraph_title-anchor split)
  - NT Jangaon: pg1 → **6**, pg2 → **18** (pg1 7→6: orphan photo R25 bound into its article;
    pg2 Type-2 splits: R65→3 కలెక్టర్/సుందరయ్య/పోస్టుమార్టం, R64→2 జాతీయరహదారి/ఉపసర్పంచుల, R97→2
    గుండెచికిత్స/మృతుడిపరామర్శ; + L89 court notice, F44 furniture)
  - Sakshi Jangaon: pg1 → **22**, pg2 → **20** (new paper, no line separators; all orphan PHOTOS
    bound to articles — 0 image orphans remain)

## Layout model (from the line-pattern study)
- **NT (Namaste) = boxed grid.** ~139 horizontal rules/4pp, many ~2-column wide; long vertical
  column dividers. ~50% of articles are ruled on 3–4 sides. Boundaries are largely *drawn*.
- **AJ (Andhra Jyothi) = whitespace layout.** Few rules (~69/4pp, mostly ≤0.6-col caption
  underlines). Most articles ruled on 0–2 sides. Boundaries come from column gutters +
  grow-right-to-dead-end + headline/dateline structure. Lines are only *confirmation* in AJ.
- **Rule width classifies horizontal lines:** ≤0.6 col = caption underline (NEVER a boundary);
  ~1 col = single-col article edge / box edge; ~2 col = article boundary (NT workhorse);
  ≥2.5 col = section/banner divider.
- **Vertical rules:** long (≥0.5 page H) = column divider / article wall; short = quote-box edge.

## Why earlier runs failed (root causes found)
1. **Over-merge from clustering + containment-merge.** Multi-column articles whose datelines sit
   only in the left column get over-extended upward; a separate dated article then falls *inside*
   the box and is absorbed → 2 articles in one crop (JG R28, R64). Fixed by **header-split**.
2. **Banner over multiple sub-stories** not cropped as one rectangle, or its sub-stories
   double-cropped → **section-banner rectangle** + **containment-merge**.
3. **L-shaped articles** (one column longer): rectangular crop's empty corner = a *different*
   article masked white → looks like white space (JG H28). Generic ragged-bottom fix REJECTED —
   it would also move AJ R48/R53 (currently fine). Still open; needs a tighter signal.
4. **Top-of-column banner with no anchor** falls into the article below → page-tall crop (JG R2).
   Fixed by **ceiling-trim**.
5. **Body-less header strips** (doc_title or quote-box paragraph_title) cropped alone, split from
   body → **body-less header merge** (H and now P).

## Strategies implemented (all gated + measured to touch only defective blocks)
- **Banner rectangle** (H & P, ≥1.6-col header, ≥2 cols, sure bottom boundary): crop whole rect.
- **Section-banner rectangle** (doc_title ≥2.5 col with ≥2 headed sub-stories): crop banner-top →
  next spanning full-width separator; absorb sub-blocks. (Only JG R67 across all runs.)
- **Containment-merge** (block ≥90% inside a larger one → fold in). Fixes split columns,
  roundup sub-stories, orphan tails.
- **Header-split** (H block with a dateline >20px ABOVE its headline → split at headline).
  Only JG R28, R64.
- **Ceiling-trim** (DL block, leading furniture >300px above the dateline → drop). Only JG R2.
- **Body-less header merge** (H or P, no text, sits ≤90px above a text block, ≥60% overlap → fold).
  H: NT R65→R73. P (iter-1): NT R133→R132, JG R52→R4.
- **Right-edge → page** (rightmost-column block → extend right edge to page-content edge).
  Touches only rightmost blocks (AJ 4/6/3/4, NT 2/3/3/8), ≤15px; interior untouched.
- **Drop classifieds** (`drop_classifieds_regions`, runs right after `drop_masthead_regions`).
  Classifieds = the "Classified Ads / phone-number listing" columns = page FURNITURE, not news.
  Left in, a neighbour article's column-fold grows down into them (JG R65 swallowed the whole
  left column to page-bottom). A region is classifieds when (only TALL regions ≥400px are
  OCR-scanned) its OCR text carries a classifieds keyword (`classified`/`contact`) **OR** ≥5
  phone-like `\d{6,}` runs (a news article rarely has >3). OCR is **half-res `eng`-only** — the
  phone-runs + English keyword are the real signals, so this is ~2× faster than full-res tel+eng
  and gives the IDENTICAL drop set on the corpus. Court/legal notices are intentionally NOT
  dropped (user: classifieds ONLY) — they are carved into their own section instead (next).
- **Carve court/legal notices** (`tag_legal_notices` + a carve pass in `cluster_all_article_blocks`).
  Court summons / name-change & matrimonial petitions / "by order of the court" notices are page
  furniture too — but, per the brief, KEPT and cropped as their OWN section (kind **`L`**), not
  dropped and not absorbed into a neighbour (JG R65 swallowed a court summons in its left column).
  `tag_legal_notices` half-res-`eng`-OCRs single-column text/paragraph_title regions and tags
  `r['_legal']='core'|'support'` on a legal-boilerplate keyword (CORE: hon'ble, in the court of,
  by order of, plaintiff/defendant, petitioner/respondent, summons, o.s.no, civil judge … ;
  SUPPORT: advocate, whereas, vakalat). The clusterer then groups adjacent tagged regions and
  carves a cluster into its own LOCKED `L` block **only when** (1) ≥2 tagged regions incl. ≥1
  CORE, (2) they all sit in ONE host block, and (3) that host is WIDE (>1100px → masked). Gate 3
  is the safety rule: a wide masked host whites the carved area out (no duplication), whereas a
  NARROW host is cropped verbatim, so carving out of it would duplicate the notice — those are
  left alone (e.g. NT pg3's petition notice straddling the narrow locked #75 + open #62 is NOT
  carved, so the NT baseline is untouched). The carved `L` block shows in BOTH the per-article
  crops and the debug boundary map (the carve lives inside `cluster_all_article_blocks`, which
  every consumer re-runs).
- **Type-2 split** (split pass in `cluster_all_article_blocks`, after the legal carve). TWO
  layouts look like one big box but need OPPOSITE treatment (AJ/Namaste = ruled papers):
  - **Type 1** — sub-stories under ONE header with a COMMON BOTTOM rule → the rule + wide banner
    already made it a `_raw_rect` section (banner/section pass) → crop as ONE rectangle. NEVER split.
  - **Type 2** — independent articles stacked one below another, EACH with its own doc_title +
    its own dateline, NO common bottom rule (JG R65: కలెక్టర్/సుందరయ్య/పోస్టుమార్టం) → over-merge →
    SPLIT into one article per headline (members assigned to the nearest headline at/above them).
  Discriminator = the `_raw_rect` flag (set ⇒ sure bottom rule ⇒ Type 1). The split anchors are
  headlines that **OWN** a dateline (each dateline assigned to the nearest headline above it) — a
  single article with a headline + a sub-headline sharing ONE dateline has only ONE owning
  headline, so it is NOT split (this fixed the AJ R50/R57 & R35/R79 false splits). Anchors must be
  vertically stacked ≥120px apart (not two columns headed side-by-side). NOTE for line-less papers
  (Sakshi): the `_raw_rect` bottom-rule signal won't exist — a different bottom cue would be needed.

## Iteration log
- **iter-1:** body-less header merge extended to `paragraph_title` → NT R133→R132, JG R52→R4
  (title-only fragments fixed). AJ untouched; NT pg4 26→25, JG pg1→9.
- **iter-2:** drop graphic-label orphans (pure-text O block ≥80% inside a FOREIGN image) →
  JG R34 dropped (label inside banner image). AJ/NT 0 drops; JG pg1 9→8.
- **iter-3:** orphan-TEXT absorb (fold pure-text orphan UP into clear above-article, gap≤80,
  no down-folds) — TRIED & **REVERTED**. Only match was NT pg4 R75→R132, a CLASSIFIEDS
  "change of name" notice; could not verify R75 belongs to that ad vs a separate notice →
  merging distinct classifieds is a regression risk. Deferred. (AJ/JG had 0 matches anyway.)

## Assessment after iter-3 — safe heuristics largely exhausted
The remaining backlog items are **risk-bearing or need human judgment**, not safe heuristics:
- Orphan PHOTOS (JG R25, AJ R12/R26): binding a standalone photo to "the right" article is
  ambiguous (far gaps, multiple candidates).
- Orphan TEXT in classifieds/notices (NT R75, etc.): can't tell same-ad vs separate-ad.
- JG front-page over-merges (R64/R65): dense multi-headline roundups; further splitting is
  complex and could fragment legit roundups.
- JG H28 L-shape white space: a generic ragged-bottom split also moves AJ R48/R53 (fine now).
Recommendation: pause autonomous loop; these need a human call per case. Resume with a
specific directive if desired.

## Open defect backlog (prioritized for next iterations)
1. **Orphan photo/text blocks (kind O)** — REMAINING: AJ pg1 R12 (photo+cap+text), pg1 R76
   (text), pg3 R32 (text), pg3 R26 (img+text); NT pg3 R7 (text tail, 36px below H98),
   pg4 R75 (text, 14px below P132); JG pg1 R25 (img-only photo). [R34 done in iter-2.]
   Next safest: fold orphan TEXT into the article directly above/below within a small gap
   (NT R75→P132 gap14, R7→H98 gap36) — measure first (risk: wrong-article merge).
   Harder: bind orphan IMAGE (R25) to caption+nearest article.
2. **JG front-page over-merges** R65 (4DL), R64 (residual 2DL below headline) — dense section
   roundups; may need below-headline multi-headline split.
3. **JG H28 L-shape white space** — needs a signal that distinguishes it from AJ R48/R53 (which
   are fine). Candidate: only when the empty corner is masked-out *foreign* article AND the
   block's tail is a single-column sidebar with its own sub-head/byline → split the sidebar.

## Hard limitation: undetected headlines (OCR gap, not a cropping bug)
On some pages PaddleOCR detects **no doc_title** for the top articles (JG pg1: only 3 doc_titles
total, ALL in the bottom half; top-half headlines `భానుడి భగభగ` / Indiramma / కరీమాబాద్ were
missed entirely). A geometric crop cannot include a headline that has no region. Symptoms:
DL-anchored crops with the heading missing (JG a01_DL3, a03_DL4) and lead photos orphaned from
their headless article (JG a02_O25). The REAL pipeline recovers these via the paid Claude
"catch by eye" pass (pipeline log: "N article(s) Claude must catch by EYE — no headline box").
The geometric `all_article_crops` (real run AND replay) will lack them. The only geometric fix
is a NEW pixel-level headline-band detector (find the large-type band above a headless dateline)
— substantial + risky for AJ/NT baselines; deferred unless explicitly requested.

## Replay validation cheatsheet
```
python3 _replay_all.py        # AJ → all_article_crops_final
python3 _replay_namaste.py    # NT Siddipet → all_article_crops_final
python3 _replay_jangaon.py    # NT Jangaon → all_article_crops_review
```
Read-only block inspection: replicate the chain (dedup → merge_stacked → drop_masthead →
promote_paragraph_titles → mark_bullet_titles → find_article_starts_by_dateline →
[probe cluster → recover_orphan_text_headlines → re-detect] → cluster_all_article_blocks).

## Partition passes added (post iter-2) — all gated + validated
- **Seed/lock confident cells** — a block is `_locked` (confident) when it's a single complete
  article (exactly 1 dateline + a headline) OR fully boxed (≥3 ruled sides). Over-merges stay open.
- **Step 4 — grow OPEN cells into gaps** (bounded by walls/cells/frame; locked never move;
  strict guard: never increase overlap with a locked cell). AJ/NT: 0 open∩locked overlaps.
- **Step 5 — containing-assignment** — every unassigned region inside a cell's box joins it
  (no box growth). Cleared NT pg4's 11 section-absorbed regions; counts unchanged.
- **No-overlap reconciliation** — trims a 0-member over-reaching box back to its members.
- **Lead-photo rebind (caption-anchored)** — a PHOTO→CAPTION→HEADLINE stack = lead photo; move
  the photo+caption down into the headline's block. Fixes JG a06_P37/maize R29 + AJ pg4/NT lead
  photos. The caption requirement excludes the H93 false positive (photo at an article's bottom).
- **Horizontal-line LENGTH FLOOR (`_lines.separator_barriers`)** — a horizontal rule counts as an
  article separator only if width ≥ **0.6× column** (median text width). Drops short
  caption/decorative underlines. NOTE: ≥1.0× is TOO aggressive (a single-column article's own
  boundary is <1 col → ≥1.0 merged NT pg4 R121); 0.6× merges ZERO articles. Caption-vs-separator
  is still decided primarily by "what's directly above the line" (photo/caption) — length is a
  secondary floor. A rule below a caption ≈ caption width = caption underline (dropped); only a
  rule much WIDER than the caption above it (≥1.5×) is kept as a boundary.

## Tried & REVERTED (kept for the record)
- **Orphan-TEXT absorb** — only match was a NT classifieds box (R75/R132); couldn't verify
  same-ad vs separate-ad → regression risk. Deferred.
- **Ragged-bottom sidebar split** — landed for JG H28 (split the `రాత్రికి రాత్రికే` sidebar so the
  L-shape white space cleared); later the lead-photo/grow cascade made H28 tile both quote boxes
  with no white space anyway. (Gate: main part must stay multi-column → AJ R53 never split.)
- **Below-divider cut** (cut content sitting below a vertical divider's END out of the right-column
  article above — JG DL2 holding the maize wide photo R30). Detection is SAFE (straddle gate ⇒
  only JG R2; AJ/NT untouched), but the repair destabilised JG (merged into wrong neighbour R49;
  emptied DL2 regrew). Deferred — needs correct merge-target (the dateline governing the band) +
  a "don't regrow a just-cut block" guard.

## RUN-WIDE VALIDATION (all 7 output runs, geometric, no API) — 2026-06-09
Ran the full cluster chain on every run in `output/` and checked counts + 2+dateline over-merges.
- **AJ Siddipet ×3** (incl. 2 never-loaded runs): **12 / 18 / 14 / 20 on all three, 0 over-merges.**
- **NT Siddipet ×3** (incl. 2 never-loaded runs): **11 / 22 / 19 / 25 on all three.**
- **NT Jangaon**: 7 / 12.
- **Every `2+dateline` block is an intended ROUNDUP** (verified): NT pg2 [110,127], pg3 [54,60,62],
  pg4 [67,121] (R121 = a `raw_rect` section: headline + 2 dated sub-stories + photos, same accepted
  pattern as R67), JG [64,65] (front-page section roundups).
- **New runs match the baselines exactly** ⇒ rules generalise (not overfit). No crashes, no new
  over-merges, no merged articles from the length floor.
Conclusion: the horizontal-line + caption handling and the partition/lead-photo passes are
validated across the whole corpus.

## Classifieds-drop validation (end-to-end replay, no API) — 2026-06-09
Wired `drop_classifieds_regions` into the pipeline (after `drop_masthead_regions`) + all 3 replay
scripts, then re-ran every replay:
- **AJ: 12 / 18 / 14 / 20** — baseline EXACT, **0 classifieds dropped** (no false positives).
- **NT Siddipet: 11 / 22 / 19 / 25** — baseline EXACT; roundups [110,127] / [54,60,62] /
  [67,121] all intact; 0 classifieds dropped.
- **NT Jangaon 130635: pg1 = 7, pg2 = 13** (was 12). The only intended change: R65 shrank from
  `[113,1958,1977,6279]` (page bottom, 4 datelines, swallowing the phone-listing classifieds) to
  `[113,1958,1977,5113]` (3-DL front-page roundup). Classifieds R87 (`y5143–6267`, phones=8)
  dropped; the freed left-column content below becomes its own block (12→13). a06_H65 renders as
  a bounded roundup, no longer page-tall over the classifieds.
- **OCR cost addressed:** detector now OCRs each tall (≥400px) region at **half-res `eng`-only**
  (was full-res `tel+eng`) — ~2× faster per call, verified to produce the IDENTICAL drop set
  (R87 still phones=8/kw=True; 6 sampled news bodies still 0/False).
- NOTE: R65 still includes a *court/legal* notice stacked in its left column. Per the user's
  "classifieds ONLY" directive this is intentionally left; separating court notices is deferred.

## Court/legal-notice carve validation (end-to-end replay, no API) — 2026-06-09
Follow-up directive: "include court/legal notices as box/section in populating the crops + debug
img." Added `tag_legal_notices` (pipeline + 3 replays, after classifieds) + the carve pass in
`cluster_all_article_blocks`, then re-ran every replay:
- **JG 130635 pg2: 13 → 14 crops.** The court summons (R51/R89/R101/R108/R93) was carved out of
  R65 into a new **`a08_L89`** block `[118,2887,595,4080]` (5 regs) — a clean, complete notice
  ("IN THE COURT OF THE HONBLE JUNIOR CIVIL JUDGE … BY ORDER OF THE HON'BLE COURT … ADVOCATE").
  R65 dropped 30→25 members and **whites the carved area out** (verified in `a06_H65` — no
  duplication). The debug boundary map shows it as **LOCK #89 [L] (5)**, nested in #65. pg1 = 7.
- **AJ: 12 / 18 / 14 / 20** — baseline EXACT, 0 legal carves.
- **NT Siddipet: 11 / 22 / 19 / 25** — baseline EXACT, 0 legal carves; roundups intact. NT pg3 has
  a real petition notice (R47/R79/R89) but it straddles the NARROW locked #75 + open #62, so Gate 3
  (wide-host-only) correctly SKIPS it → baseline untouched.
- Detection sweep (read-only) across the corpus: legal keywords flagged ONLY the JG court notice
  and the NT pg3 petition; **0 false positives on any AJ/NT news region.**
- COST: `tag_legal_notices` OCRs ~40–70 single-column text regions/page (the expensive part).
  Acceptable for the paid pipeline (Claude vision dominates), but a candidate for a cheaper
  pre-gate (e.g. skip dateline-anchored regions) if it becomes a bottleneck.
- DEFERRED (harder, needs the seed-before-clustering model): a legal notice that straddles a
  NARROW straight-cropped article (NT pg3) — to carve it cleanly the host article's box must be
  shrunk away from the notice, which can't be done by masking alone.

## Type-2 split validation (end-to-end replay, no API) — 2026-06-09
Directive: Type-1 (shared header + common bottom rule) = ONE rectangle (fine); Type-2 (stacked
articles, own headline+dateline each, no common rule) = SEPARATE articles. Added the split pass.
- **JG 130635 pg2: 14 → 16 crops.** R65 (the კალెక్టర్ over-merge) split into 3 clean articles:
  R65 కలెక్టర్ `[113,1958,1975,2859]`, R86 సుందరయ్య `[123,2894,1977,4292]`, R112 పోస్టుమార్టం
  `[133,4325,1975,5113]`. Court notice R89 [L] still its own section. pg1 = 7.
- **AJ: 12 / 18 / 14 / 20** — baseline EXACT (after the dateline-OWNERSHIP fix; the first cut wrongly
  split AJ pg1 R50/R57 and pg3 R35/R79 — headline+subhead sharing one dateline — 13/15, now fixed).
- **NT Siddipet: 11 / 22 / 19 / 25** — baseline EXACT. The 2+dl roundups [110,127]/[54,60,62] are
  single-header-multi-dateline (Type 1) so they have <2 dateline-owning headlines → NOT split;
  [67,121] are `_raw_rect` → excluded. All correctly preserved.
- KNOWN minor issue: R65's left column held a furniture stack (court notice + classifieds logo +
  a small job ad + a గమనిక notice). The court notice is carved (L89) and the main classifieds
  (R87) dropped, but the leftover logo (R78 image) bleeds into R86's crop and the job ad + గమనిక
  bleed into R112's crop (they're <400px / images, below the classifieds-drop gate). Secondary
  cleanup — extend furniture detection if needed.  [RESOLVED below: furniture section.]

## Furniture section (Type-2 split) — column-aware member assignment — 2026-06-09
Directive: when splitting a Type-2 stack, LEAVE classifieds / notices / unrelated content OUT of
the article crops, but KEEP that section visible on the debug empty-canvas map. Implemented
**geometrically (no OCR)** inside the split: each member is assigned to the nearest headline
at/above it ONLY IF it also sits within that headline's COLUMN x-span (±0.3 col). A member that
lands in a headline's y-band but a DIFFERENT column is unrelated content sharing the page — it
falls out as LEFTOVER and is grouped into a `_furniture` section block. Furniture blocks are:
(1) EXCLUDED from article crops — `crop_all_article_blocks` skips saving them, and because each
article sub-block's bbox is rebuilt from its in-column members only, the furniture column isn't
even inside the article rectangle; (2) DRAWN on the debug boundary map as a hatched GREY
"FURNITURE #id" box; (3) NEVER saved as a crop. The split fires only on JG R65, so this touches
ONLY that page — AJ/NT untouched.
- **JG pg2:** R65 → R65 కలెక్టర్ + R86 సుందరయ్య (x682, was x123 w/ the classifieds logo) + R112
  పోస్టుమార్టం (x682, clean) + R89 [L] court notice + **F44 furniture** [123,4139,626,5113]
  (classifieds logo + job ad + గమనిక — debug-only, not cropped). 16 saved crops; boundary map shows
  the FURNITURE box. a09_H86 & a13_H112 verified clean (no furniture bleed).
- **AJ 12/18/14/20** and **NT 11/22/19/25** — baselines EXACT.

## Type-2 split — paragraph_title anchors + ';' dateline + section discriminator — 2026-06-09
Two more JG pg2 over-merges were reported (R97 = గుండె చికిత్స + మృతుడి పరామర్శ; R64 = జాతీయ రహదారి +
ఉపసర్పంచుల చెక్ పవర్). Three root causes, three fixes:
1. **2nd headline detected as `paragraph_title`, not `doc_title`** (R58, R84) → my split's anchors
   were doc_title-only. FIX: anchors = `doc_title` OR `paragraph_title`, with COLUMN-AWARE dateline
   ownership (each dateline → nearest headline above WHOSE x-span covers it; a caption in another
   column can't steal it). Captions/sub-heads still own no dateline ⇒ never an anchor.
2. **2nd dateline's colon OCR'd as a semicolon** (R97: 'చిల్పూరు, మే 19 ;') → `_dateline_in_text`
   required ':'. FIX: accept ';' too (guards: digit + short prefix + no sentence punctuation).
3. **R64 was `_raw_rect`** (banner-rectangle extended the wide headline to the next banner, NO
   spanning rule) so the split skipped it. FIX: the split's Type-1 exclusion is now the `_section`
   flag (set ONLY by the section-banner pass, which requires a real spanning bottom rule), not the
   coarse `_raw_rect` flag. R64 (no rule) → splits; R67/R121 (real section) → stay one rectangle.
- **JG pg2 16→18:** R64→R64+R84, R97→R97+R58 (verified: each crop is one complete article).
- Side effects, all VERIFIED correct (same rule, genuinely distinct articles — not fragmentation):
  NT pg2 22→23 (అంగన్‌వాడీ R127 off the ప్రజావాణి section), NT pg3 19→20 (పర్యావరణ R60 off), AJ pg4
  20→21 (గజ్వేల్ R12 — a real dateline the ';' fix recovered; an existing pass then split it).
  R110 (ప్రజావాణి) stayed whole as a `_section` ⇒ the discriminator protects real sections.

## Orphan-PHOTO binding — an image may never stand alone — 2026-06-09
RULE (user): an article may be orphan, but an IMAGE may not — every photo must belong to some
article. A final pass in `cluster_all_article_blocks` binds each ORPHAN-PHOTO block (no
doc_title/paragraph_title headline AND no dateline; image-only, OR image + caption only, OR
image-area ≥50%) to the best NEWS article (furniture/legal blocks are excluded as targets): the
article whose top is just BELOW the photo in the same column (lead photo), else just above, else
the nearest column-overlapping article. A TEXT/body orphan is LEFT ALONE.
- **Sakshi pg1 23→22, pg2 27→20:** every orphan photo bound (R28→R78 lead photo verified — the
  'collector inspecting BL-1' photo + caption now sit in R78; pg2 R38/R42/R48/R52/R60/R63 all
  bound). **0 image orphans remain** on either page. The text orphan R18 (pg1) correctly stays.
- **JG pg1 7→6:** orphan photo R25 bound into its article.
- **AJ 12/18/14/21** and **NT 11/23/20/25** — UNCHANGED (no qualifying image orphans on those).
- NOTE: Sakshi itself has rough article structure (headlines/datelines under-detected, no rules) —
  some bound photos land on a column-overlapping article that may need per-case review; but the
  rule (no floating image) now holds. Deeper Sakshi clustering is a separate task.
- **Respect horizontal separators in the photo bind (refinement):** a photo binds to an ADJACENT
  article (gap ≤200px) even if a rule sits between them — that rule is the article's own headline
  border and the photo is its lead photo (JG R28→R78, 7px gap). But a FAR fallback bind is REJECTED
  if a horizontal rule LONGER than a caption (>0.6 col) lies between the photo and the article —
  the user's cross-paper line rule. This kept the Sakshi pg2 "student photo" R52 (top of page) from
  binding down through two FULL-WIDTH rules + a page of white space into R118 (the భక్తులకు/
  Kaleshwaram story); R52 now binds to R127 in its own band, and `a08_H118` is clean. Verified:
  Sakshi pg1 22 / pg2 20 (0 image orphans); AJ 12/18/14/21, NT 11/23/20/25, JG 6/18 all unchanged.

## Sakshi page-1 deep fix round — 2026-06-10 (visual review of all 22 pg1 crops → 14 BAD)
A crop-by-crop visual audit of Sakshi JG pg1 found 14/22 BAD crops. Root causes & fixes, ALL
Sakshi-gated (AJ 12/18/14/21, NT 11/23/20/25, JG 6/18 re-validated EXACT after every change):
- **Masthead drop for Sakshi** (`drop_masthead_regions`, `paper=="sakshi"` branch, every page):
  drop every region that ends ABOVE the topmost headline-type region (sanity: that line must be
  in the top 30%). pg1: weather box R64, photo-brief R62/R59, nameplate R63, and the page-date
  strip R71 — which OCR'd as a DATELINE and seeded a fake DL71 article. pg2: R68/R71/R84/R104.
- **`detect_sakshi_headline_bands`** (called after `promote_paragraph_titles`, pipeline + replay)
  — the big one. Sakshi's display headlines are often NOT boxed by PaddleOCR at all. Two passes:
  1. **WIDEN**: a doc_title grows sideways over adjacent UNCOVERED large-ink columns — its
     un-boxed kicker half (R52 'అనూహ్య స్పందన' + red kicker 'రిబేట్ ఆఫర్‌కు'). Widest titles first;
     extensions join the coverage map so a narrow neighbour can't steal a wide headline's kicker.
     Ink within 25px of a foreign region's edge is the neighbour's own glyph overflow — ignored.
  2. **SYNTHESIZE**: an uncovered big-type band (55–280px tall, ≥0.75 col wide, ink 5–60%, text
     following below, never above the first real headline) becomes a synthetic `doc_title`
     (`_synth_band`) anchor. Found BOTH missing pg1 banners (lift-1 'జూలై 31...' R99 and SIR
     'మ్యాపింగ్‌లో జిల్లాకు అగ్రస్థానం' R100) and pg2's గమనిక box R139. CRITICAL detail: two
     headlines can share pixel rows with only a ~50px gutter (same size as a word gap) — a gap
     splits a band run ONLY if it contains a known text-column LEFT edge (≥35px wide); plain
     gap-size segmentation merges both banners into one anchor (measured: pg1 12 crops, 45-member
     monster).
- **Box-edge rule disarm** (`_is_box_edge` in `_barrier_between`, Sakshi only): a horizontal rule
  with a PARALLEL partner 80–400px away (≥70% overlap) is a quote/bullet-box border, never a story
  separator. Without it the lift-1 bullet-box borders fenced the article's own columns off.
- **Photo-bottom border disarm** (`_in_photo`, Sakshi only): a rule within 14px BELOW an image,
  horizontally inside the photo span ±25px, is the photo's frame/caption rule (PaddleOCR's box
  ends a few px above the printed border, so the interior test missed it). Without it R61/R46/R23
  (lift-1 caption + col-3 body) fell into the story below.
- **Caption never a headline line** (headline-stack recovery, all papers — baselines unaffected):
  a figure_title directly under an image (≤90px) is a CAPTION; stop the stack there. Without it
  the synth SIR headline pulled the Indiramma photo's caption R69 down as a "wrapped line".
- **No lead-photo rebind into a `_synth_band` headline**: a synth band's lead photo is BELOW it by
  construction; a photo+caption ABOVE it belongs to the story above (R36/R69 stay with R77).
- **Continuation-column fold** (Sakshi only, end of clusterer): a TEXT-only orphan block directly
  beside an article block with the SAME vertical band (top ±90px, ≥80% overlap, gap ≤80px, no
  vertical rule between) is the other column of the same box → merge (pg1 R18 → R79; the text even
  continues mid-word 'విద్యా సంస్థ|లకు' across the two).
- **Stray-image re-home** (Sakshi only): an image member vertically DETACHED from the rest of its
  block but fully inside ANOTHER article's box moves there. A 3-column synth banner column-overlaps
  everything below it, so R50's tiny సాక్షి-ఎఫెక్ట్ clipping R44 (whose own narrow headline does
  NOT column-overlap it) had fallen to the banner 1100px above.
- **Narrow-crop masking** (`crop_all_article_blocks`, all papers): narrow (≤1100px) crops now white
  out regions OWNED BY OTHER blocks (gpad=2) before the ink-trim — kills the sliced-glyph slivers
  at crop edges. Un-owned ink is left alone (it may be the crop's own un-boxed headline). Members
  are repasted, so nothing of the article itself can be erased. Counts unchanged on all papers.
- **Review overlays**: `crop_all_article_blocks(..., overlay=True)` (used by `_replay_sakshi.py`)
  draws member boxes on the review crops — blue=anchor, magenta=synth band, green=member.
- **RESULT (Sakshi JG pg1): 22 → 16 crops, every reduction a genuine reunification.** Visual
  re-audit of all 16: 14 GOOD / 2 MINOR / 0 BAD (was 8 GOOD / 14 BAD). The rebate feature package
  (was shredded into P94+H52+DL71+DL11 with sliced letters) is ONE crop; lift-1 and SIR-mapping
  (whose headlines have NO region) are complete articles anchored by synth headlines; H77 is
  exactly the Indiramma story; H54/H78/H55 no longer hold neighbours' content. pg2: 20 → 17
  (top photo-gallery orphans folded into their stories; not yet visually audited).
- **NEW Sakshi baselines: pg1 → 16, pg2 → 17.**
- Known remaining (MINOR): a01_H55 loses the unboxed last line of its మ్యూజియాలజీ sub-brief
  (no region covers it; the next block's white-out grazes it); the EAPCET ad image R33 sits
  mid-article in H55 (it IS physically inside the article band; ad detection deferred).
- Known remaining (pg2, PRE-EXISTING): the Type-2 split's column-aware leftover pass marks the
  విద్యార్థిని ఆత్మహత్య article as FURNITURE (R4/R10 F) → it is NOT cropped at all. Present in the
  pre-fix baseline too. Needs its own round.  [RESOLVED in the pg2 round below.]

## Sakshi page-2 round — 2026-06-10 (visual audit of all pg2 crops; user-reported defects)
Fixes, validated after each change (AJ 12/18/14/21, NT 11/23/20/25 incl. 2+dl=[110]-only on pg2,
JG 6/18 — all EXACT):
- **a04_H118 body truncated at the column boundary** (col5 ends mid-word 'నిర్మా-', col6 holds
  'ణం…'): the col6 body R20 was stolen by the headless dateline R14's GROW-UP, which climbed PAST
  the R14 story's own subhead R121 because `_is_caption` lets a 197px two-paragraph text through
  (≤200px + a photo 82px above). FIX (all papers, baselines unchanged): grow-up STOPS after
  claiming a `paragraph_title` — the heading IS the article's top; nothing above it is ours.
- **విద్యార్థిని ఆత్మహత్య never cropped** (whole story = furniture): its heading R88 IS detected
  (paragraph_title) but the R41 Type-2 split's anchors don't span col4 → all 8 regions fell out
  as LEFTOVER → furniture. FIX (all papers; gates keep AJ/NT/JG exact): a leftover group that
  LEADS with its own COLUMN-WIDE heading (≥0.6 of the group width — a narrow WANTED label stays
  furniture) and has body text → emit as an ARTICLE (kind P), not furniture.
- **a13_H93/'విషాదయాత్ర' cut mid-sentence, its tail + tributes in the girl's crop**: the col4
  leftover zone holds TWO stacked items (girl's story y3658-4643, శ్రీనివాస్ tail y4723-5938) with
  a separator rule at y4679 between them. FIX (Sakshi only): (1) leftover clustering never merges
  across a separator rule; (2) a heading-less leftover group whose band MATCHES an adjacent-column
  split article (top ±90px, ≥80% overlap, gap ≤80px) is that article's column CONTINUATION → its
  members merge there (the tail's text literally resumes H93's "...అయితే వైద్యులు" mid-sentence).
  ORDER MATTERS: the wide-heading→ARTICLE check runs FIRST, else the girl's story (gap-close to
  H99's band) band-attaches into H99. Both passes are Sakshi-gated: un-gated they re-shuffled an
  NT pg2 leftover into R105 (2+dl-blocks [110]→[110,105]) — caught by the 2+dl indicator and
  re-gated.
- **a03_P129/a05_H91: dead farmer's portrait R47 + caption R115 stuck at the briefs column's
  bottom, white hole in his own story's crop**: the stray-image re-home pass couldn't see R47 as
  detached because its caption R115 sat below it inside the same block. FIX: re-home treats an
  image + its caption (figure_title ≤90px below, ≥40% overlap) as ONE UNIT — excluded from the
  envelope test and moved together (→ H91, inside whose box they sit).
- **గమనిక notice box synthesized as a headline** (R139, its own bogus crop): a filled red header
  measures ink 0.51; real Sakshi display headlines measure 0.30-0.35. FIX: synth band ink ceiling
  0.60 → 0.45; the notice text returns to the classifieds-column furniture (not cropped).
- **RESULT (Sakshi JG pg2): 20 → 17 crops.** Visual audit of all 17: 11 GOOD / 4 MINOR / 2 BAD →
  after the H93/P88 boundary fix the BAD pair re-audited GOOD. The H118 two-column body, the
  విద్యార్థిని article (new crop P88), H91's portrait, and the P129 briefs column all verified.
- Known remaining (pg2, magazine-style kids/summer feature band only): H82/H127 share the gallery
  collage and H82's story conclusion ('–దుగ్గొండి' sign-off + 'నీతి' moral box) renders inside
  H127's crop — a collage-page attribution problem, low priority for the news pipeline. The
  classifieds crops (P136/P111) bundle neighbouring ads and clip one ref number (MINOR).

## POLITICAL crops now BLOCK-SNAP to the geometric blocks — 2026-06-10
`crop_political_articles` used to re-derive each Claude-selected article's rectangle from the raw
member_region_ids union plus ~400 lines of parallel ad-hoc geometry (banner walls, foreign-headline
clips, floors, grow-right) — none of which carried the geometric-crop fixes. Now it BLOCK-SNAPS:
- It clusters the page once (`cluster_all_article_blocks`) and renders every block crop via
  `crop_all_article_blocks` into a temp dir — the exact validated pipeline (furniture drop,
  un-boxed headline recovery, neighbour masking, box-edge rules, continuation folds).
- Each political article is mapped to the geometric block owning the MAJORITY (≥50%) of its
  member regions; that block's crop is shipped as `article.png` (`method:
  "smart_extractor_blocksnap"`, info.json carries the block's full member list).
- A block already claimed by an earlier article on the page is not re-used (no duplicate crops);
  any article that fails to map falls back to the legacy path unchanged.
- The pipeline call passes `dateline_starts` + `sep_lines` through so the clustering matches the
  all_article_crops output exactly.
- VALIDATION (no API — `_replay_political.py` replays saved `page_NNN.political_result.json`
  member lists through the new path): Sakshi JG pg1 2/2 snapped (the rebate package + DL5, both
  verified visually as the block-quality crops); AJ Siddipet pg2/3/4 4/4 snapped (10-13 of 12-13
  member votes, incl. the R48/R33 banner blocks); NT Jangaon pg1 2/2 snapped (13/13, 4/4). ZERO
  legacy fallbacks needed on the corpus.
- NOTE: replayed member lists come from OLD paid runs (stale region understanding) — e.g. the
  Sakshi 'Transfers delayed!' entry maps to the rebate block because its old members [66,29] sit
  there now. On the next paid run Claude selects members against the CURRENT regions (incl. the
  synthesized Sakshi headlines, which are in regions.json and the prompt's region list), so the
  member→block vote will be consistent end-to-end.

## New-run defects round (Sakshi_Jangaon District_20260513, first PAID run on new code) — 2026-06-10
User-reported bad crops on the fresh paper; three more Sakshi-gated passes (validated: old-run
Sakshi pg1 16 / pg2 17 UNCHANGED; AJ 12/18/14/21, NT 11/23/20/25 [110], JG 6/18 all EXACT):
- **pg2 a02_H106 was a 695×5055 column strip** (statue story + 4000px of masked white):
  TWO causes. (a) The dance-photo caption R104 (page top) fell down-column into H106 —
  its photo R44 lived in another block. FIX: **stray caption re-home** — a figure_title member
  moves to the block that owns the image it abuts (±90px above/below, with **-30px overlap
  tolerance**: PaddleOCR's photo box overlaps the caption strip by a few px, R104 vs R44 = -11px,
  which defeated a 0-gap test on the first attempt). Donor block shrinks back to its members.
  (b) The aviation story's heading is a `paragraph_title` 1200px below the statue story's
  doc_title in-column → never promoted → whole stack merged. FIX: **stacked-story heading
  promotion** — a pt ≥0.55 col wide with content directly beneath (≤150px) and the nearest
  in-column base anchor **>900px** above becomes an anchor. The gate was first 600px and broke
  the old-run rebate package (its sub-head R94 sits 614px under the widened R52 headline and
  re-split, whiting R88's box over the stats infographic — caught by re-validating the old run);
  900px keeps feature sub-heads folded and still promotes genuinely stacked stories.
- **pg2 a08_H100 stretched ~900px up with a foreign fragment above its own headline** (the NPDCL
  story's tail folded down-column into H100). FIX: **stray text re-home** — a plain-text member
  of an H block lying entirely ABOVE the anchor headline's top is never that article's content
  (kickers/decks are title-typed); move it to the smallest other article box containing it.
- Verified visually: new-run a10_H106 (639×1326, exactly the statue story), a09_H100 (complete,
  no stray top), old-run a02_H52 (infographic numbers 15,941/5.79/11,376/3.72/1,00,20,087 intact).
- New-run pages have NOT had a full visual audit (only the reported crops + block maps checked);
  pg1 = 14 crops / pg2 = 16 on replay.

## NT Adilabad flood-special front page — NT rule split + late structural cut — 2026-06-10
NamastheTelangana_Adilabad_20260528 pg1 produced 5 page-scale OVERLAPPING mega-blocks. This page
is DETECTION-LIMITED at the source: 65 regions only, 4 doc_titles for ~8 stories, and BOTH big
feature headlines ('రెక్కల కష్టం.. నీళ్లపాలు', 'వడ్ల తరలింపు ఎక్కడిదాకా?') are INSIDE PaddleOCR
image boxes — invisible to every geometric pass incl. the synth-band detector (coverage test).
With anchors missing, the image-must-belong bind + grow/merge built page-tall blocks. Fixes:
- **NT horizontal-rule split enabled** (was Sakshi-only): same machinery, but only at rules
  ≥1.8 col (per the line study a ≥2-col rule is an article boundary on NT; the boxed grid's
  ~1-col caption/box rules must not cut). _replay_adilabad.py added (NTA_RUN env overridable).
- **NT LATE structural cut** (end of clusterer, after grow/merge/photo binds): the mid-pipeline
  split runs BEFORE the passes that built the mega-block, so DL5 [668,901,4084,6302] was never
  re-checked against the 4.4-col rule at y=4764 (and at mid-split time the bottom band's R30
  headline wasn't yet a member, so the new-start gate failed). Final blocks (not locked/raw_rect/
  section/furniture) are re-cut at ≥1.8-col rules with the underline guard; a rule ≥2.2 col cuts
  even with no detected start below. DL5 → DL56 (the whole flood feature, verified coherent) +
  H30 ('తడిసిన వడ్లు కొనాలి' bottom band, verified clean). pg1 5 → 6 crops.
- VALIDATION: NT Siddipet 11/23/20/25 EXACT (2+dl [110],[54,62],[67,121]); NT Jangaon 6/18 EXACT;
  Adilabad pg2 31 crops unchanged (no shredding). AJ/Sakshi untouched (gate).
- HONEST REMAINDER: H44 still spans the page top (left story + photo row + nameplate share the
  band; the 60%-block-span guard correctly refuses partial rules on a page-wide block) and the
  H59/DL9 boxes still overlap neighbours. Only a better layout model (headline bands inside
  photo boxes) or the paid pipeline's Claude-vision pass can fully fix this page. NT masthead
  drop never fired here (nameplate is INSIDE image R46, no header region) — benign, the strip
  rides atop the photo-row crop.

## ⚠ REPLAY-TOOLING BUG (found 2026-06-10, fixed): _ACTIVE_PAPER was never set for NT
`_replay_jangaon.py` / `_replay_namaste.py` never set `se._ACTIVE_PAPER`, so every prior "NT
validated EXACT" run measured with ALL paper-gated clustering passes OFF while the real pipeline
runs them ON. (AJ has no gated passes, Sakshi's replay always set it — only NT was affected.)
Both scripts now set it; `_replay_jangaon.py` takes JG_RUN env (default = the 130635 run; the
user-facing run of the same paper is ..._20260608_124720). FIRST true gated-NT measurement:
NT Siddipet 11/23/20/25 + 2+dl [110],[54,62],[67,121] EXACT (gated passes are no-ops there ✓),
JG both runs 6/18, Adilabad 6/31.

## NT Jangaon pg1 (user-reported: a01_DL3 white holes, a04_H28 over-merge, a05_DL2 foreign photo)
Root cause is the same family as Sakshi/Adilabad: THREE un-detected display headlines (brush-font
'భానుడి భగభగ', script 'నమ్మిండు..', DL2's heading). The heat feature's lower content (vendor +
ice-cream photos, captions, continuation column) folded DOWN to the 'ఆలస్యంపై రైతుల ఆగ్రహం'
headline 2200px below (nearest-anchor metric); the white holes in a01_DL3 were those stolen
regions masked out. Fixes (gated `in ("sakshi","namaste_telangana")` — AJ untouched):
- **Re-home family enabled for NT** (stray image+caption unit, stray caption, above-own-headline
  members). The above-headline re-home is GENERALISED from text-only to text/image/figure_title
  (title-typed members stay — kickers/decks are legitimate above a headline).
- **Synth headline bands enabled for NT — FRONT PAGE ONLY** (`n == 1`): inner NT pages are dense
  and well-detected; un-gated synth split the protected Siddipet pg4 roundups (25→28, caught).
  Band height cap 280→400 (tall display/script faces; ink-fraction still rejects photos).
- RESULT (JG 124720 pg1, counts unchanged at 6): DL3 = the COMPLETE heat feature (mem 8→13, no
  white holes, verified visually); H28 starts AT its own headline [y3089, was 838] = exactly the
  ఆగ్రహం article (verified); DL2 = clean single-column committee article, its foreign tractor
  photo+caption re-homed to R29's procurement story (verified).
- Residual: DL2 remains heading-less (its display heading has no region and the synth band gates
  didn't admit it) — hard-limitation class; the heading pixels sit above the crop.