commit 900bdcc302b8f863f573d55fcc44ea5d0b69cd1a Author: Deep Koluguri Date: Fri Jun 12 09:21:14 2026 -0400 Initial commit diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..d9f7fe4 --- /dev/null +++ b/.gitignore @@ -0,0 +1,38 @@ +# Virtual environment +venv/ +.venv/ + +# Python cache +__pycache__/ +*.py[cod] +*.egg-info/ + +# Pipeline outputs (large, regenerated each run) +output/ +uploads/ + +# Input newspaper PDFs (large binary data) +*.pdf + +# Scratch / diagnostic preview images +_*.png +pg-*.jpg +q-*.jpg +gp-*.jpg +gq-*.jpg + +# Generated HTML guides +Extractor_Guide.html +HOW_IT_WORKS.html + +# Secrets / API keys (never commit) +.keyenv +.env +.env.* + +# OS / editor cruft +~$* +.DS_Store +*.swp +.idea/ +.vscode/ diff --git a/CROP_STRATEGY.md b/CROP_STRATEGY.md new file mode 100644 index 0000000..a754baa --- /dev/null +++ b/CROP_STRATEGY.md @@ -0,0 +1,515 @@ +# Article-Cropping Strategy & Learnings (self-improving loop) + +Living doc for the geometric article-cropping in `smart_extractor.py` +(`cluster_all_article_blocks` → `crop_all_article_blocks`). Updated each loop iteration. + +## Validation rules (NEVER violate) +- **Never run `smart_extractor.py`** — it calls the paid Claude API. Validate ONLY with the + geometric replay scripts: `_replay_all.py` (AJ), `_replay_namaste.py` (NT Siddipet), + `_replay_jangaon.py` (NT Jangaon). These run the full cluster chain with NO API. +- **Measure read-only BEFORE editing** (blast-radius of any new rule), then **re-validate after**. +- **Baselines** (block counts) — keep stable unless an improvement is intentional & documented: + - AJ: 12 / 18 / 14 / **21** (pg4 20→21: the ';'-delimiter fix recovered a real గజ్వేల్ dateline R12) + - NT Siddipet: **11 / 23 / 20 / 25** (pg2 22→23: అంగన్‌వాడీ R127 split off the ప్రజావాణి section R110; + pg3 19→20: పర్యావరణ R60 split off; both are the Type-2 paragraph_title-anchor split) + - NT Jangaon: pg1 → **6**, pg2 → **18** (pg1 7→6: orphan photo R25 bound into its article; + pg2 Type-2 splits: R65→3 కలెక్టర్/సుందరయ్య/పోస్టుమార్టం, R64→2 జాతీయరహదారి/ఉపసర్పంచుల, R97→2 + గుండెచికిత్స/మృతుడిపరామర్శ; + L89 court notice, F44 furniture) + - Sakshi Jangaon: pg1 → **22**, pg2 → **20** (new paper, no line separators; all orphan PHOTOS + bound to articles — 0 image orphans remain) + +## Layout model (from the line-pattern study) +- **NT (Namaste) = boxed grid.** ~139 horizontal rules/4pp, many ~2-column wide; long vertical + column dividers. ~50% of articles are ruled on 3–4 sides. Boundaries are largely *drawn*. +- **AJ (Andhra Jyothi) = whitespace layout.** Few rules (~69/4pp, mostly ≤0.6-col caption + underlines). Most articles ruled on 0–2 sides. Boundaries come from column gutters + + grow-right-to-dead-end + headline/dateline structure. Lines are only *confirmation* in AJ. +- **Rule width classifies horizontal lines:** ≤0.6 col = caption underline (NEVER a boundary); + ~1 col = single-col article edge / box edge; ~2 col = article boundary (NT workhorse); + ≥2.5 col = section/banner divider. +- **Vertical rules:** long (≥0.5 page H) = column divider / article wall; short = quote-box edge. + +## Why earlier runs failed (root causes found) +1. **Over-merge from clustering + containment-merge.** Multi-column articles whose datelines sit + only in the left column get over-extended upward; a separate dated article then falls *inside* + the box and is absorbed → 2 articles in one crop (JG R28, R64). Fixed by **header-split**. +2. **Banner over multiple sub-stories** not cropped as one rectangle, or its sub-stories + double-cropped → **section-banner rectangle** + **containment-merge**. +3. **L-shaped articles** (one column longer): rectangular crop's empty corner = a *different* + article masked white → looks like white space (JG H28). Generic ragged-bottom fix REJECTED — + it would also move AJ R48/R53 (currently fine). Still open; needs a tighter signal. +4. **Top-of-column banner with no anchor** falls into the article below → page-tall crop (JG R2). + Fixed by **ceiling-trim**. +5. **Body-less header strips** (doc_title or quote-box paragraph_title) cropped alone, split from + body → **body-less header merge** (H and now P). + +## Strategies implemented (all gated + measured to touch only defective blocks) +- **Banner rectangle** (H & P, ≥1.6-col header, ≥2 cols, sure bottom boundary): crop whole rect. +- **Section-banner rectangle** (doc_title ≥2.5 col with ≥2 headed sub-stories): crop banner-top → + next spanning full-width separator; absorb sub-blocks. (Only JG R67 across all runs.) +- **Containment-merge** (block ≥90% inside a larger one → fold in). Fixes split columns, + roundup sub-stories, orphan tails. +- **Header-split** (H block with a dateline >20px ABOVE its headline → split at headline). + Only JG R28, R64. +- **Ceiling-trim** (DL block, leading furniture >300px above the dateline → drop). Only JG R2. +- **Body-less header merge** (H or P, no text, sits ≤90px above a text block, ≥60% overlap → fold). + H: NT R65→R73. P (iter-1): NT R133→R132, JG R52→R4. +- **Right-edge → page** (rightmost-column block → extend right edge to page-content edge). + Touches only rightmost blocks (AJ 4/6/3/4, NT 2/3/3/8), ≤15px; interior untouched. +- **Drop classifieds** (`drop_classifieds_regions`, runs right after `drop_masthead_regions`). + Classifieds = the "Classified Ads / phone-number listing" columns = page FURNITURE, not news. + Left in, a neighbour article's column-fold grows down into them (JG R65 swallowed the whole + left column to page-bottom). A region is classifieds when (only TALL regions ≥400px are + OCR-scanned) its OCR text carries a classifieds keyword (`classified`/`contact`) **OR** ≥5 + phone-like `\d{6,}` runs (a news article rarely has >3). OCR is **half-res `eng`-only** — the + phone-runs + English keyword are the real signals, so this is ~2× faster than full-res tel+eng + and gives the IDENTICAL drop set on the corpus. Court/legal notices are intentionally NOT + dropped (user: classifieds ONLY) — they are carved into their own section instead (next). +- **Carve court/legal notices** (`tag_legal_notices` + a carve pass in `cluster_all_article_blocks`). + Court summons / name-change & matrimonial petitions / "by order of the court" notices are page + furniture too — but, per the brief, KEPT and cropped as their OWN section (kind **`L`**), not + dropped and not absorbed into a neighbour (JG R65 swallowed a court summons in its left column). + `tag_legal_notices` half-res-`eng`-OCRs single-column text/paragraph_title regions and tags + `r['_legal']='core'|'support'` on a legal-boilerplate keyword (CORE: hon'ble, in the court of, + by order of, plaintiff/defendant, petitioner/respondent, summons, o.s.no, civil judge … ; + SUPPORT: advocate, whereas, vakalat). The clusterer then groups adjacent tagged regions and + carves a cluster into its own LOCKED `L` block **only when** (1) ≥2 tagged regions incl. ≥1 + CORE, (2) they all sit in ONE host block, and (3) that host is WIDE (>1100px → masked). Gate 3 + is the safety rule: a wide masked host whites the carved area out (no duplication), whereas a + NARROW host is cropped verbatim, so carving out of it would duplicate the notice — those are + left alone (e.g. NT pg3's petition notice straddling the narrow locked #75 + open #62 is NOT + carved, so the NT baseline is untouched). The carved `L` block shows in BOTH the per-article + crops and the debug boundary map (the carve lives inside `cluster_all_article_blocks`, which + every consumer re-runs). +- **Type-2 split** (split pass in `cluster_all_article_blocks`, after the legal carve). TWO + layouts look like one big box but need OPPOSITE treatment (AJ/Namaste = ruled papers): + - **Type 1** — sub-stories under ONE header with a COMMON BOTTOM rule → the rule + wide banner + already made it a `_raw_rect` section (banner/section pass) → crop as ONE rectangle. NEVER split. + - **Type 2** — independent articles stacked one below another, EACH with its own doc_title + + its own dateline, NO common bottom rule (JG R65: కలెక్టర్/సుందరయ్య/పోస్టుమార్టం) → over-merge → + SPLIT into one article per headline (members assigned to the nearest headline at/above them). + Discriminator = the `_raw_rect` flag (set ⇒ sure bottom rule ⇒ Type 1). The split anchors are + headlines that **OWN** a dateline (each dateline assigned to the nearest headline above it) — a + single article with a headline + a sub-headline sharing ONE dateline has only ONE owning + headline, so it is NOT split (this fixed the AJ R50/R57 & R35/R79 false splits). Anchors must be + vertically stacked ≥120px apart (not two columns headed side-by-side). NOTE for line-less papers + (Sakshi): the `_raw_rect` bottom-rule signal won't exist — a different bottom cue would be needed. + +## Iteration log +- **iter-1:** body-less header merge extended to `paragraph_title` → NT R133→R132, JG R52→R4 + (title-only fragments fixed). AJ untouched; NT pg4 26→25, JG pg1→9. +- **iter-2:** drop graphic-label orphans (pure-text O block ≥80% inside a FOREIGN image) → + JG R34 dropped (label inside banner image). AJ/NT 0 drops; JG pg1 9→8. +- **iter-3:** orphan-TEXT absorb (fold pure-text orphan UP into clear above-article, gap≤80, + no down-folds) — TRIED & **REVERTED**. Only match was NT pg4 R75→R132, a CLASSIFIEDS + "change of name" notice; could not verify R75 belongs to that ad vs a separate notice → + merging distinct classifieds is a regression risk. Deferred. (AJ/JG had 0 matches anyway.) + +## Assessment after iter-3 — safe heuristics largely exhausted +The remaining backlog items are **risk-bearing or need human judgment**, not safe heuristics: +- Orphan PHOTOS (JG R25, AJ R12/R26): binding a standalone photo to "the right" article is + ambiguous (far gaps, multiple candidates). +- Orphan TEXT in classifieds/notices (NT R75, etc.): can't tell same-ad vs separate-ad. +- JG front-page over-merges (R64/R65): dense multi-headline roundups; further splitting is + complex and could fragment legit roundups. +- JG H28 L-shape white space: a generic ragged-bottom split also moves AJ R48/R53 (fine now). +Recommendation: pause autonomous loop; these need a human call per case. Resume with a +specific directive if desired. + +## Open defect backlog (prioritized for next iterations) +1. **Orphan photo/text blocks (kind O)** — REMAINING: AJ pg1 R12 (photo+cap+text), pg1 R76 + (text), pg3 R32 (text), pg3 R26 (img+text); NT pg3 R7 (text tail, 36px below H98), + pg4 R75 (text, 14px below P132); JG pg1 R25 (img-only photo). [R34 done in iter-2.] + Next safest: fold orphan TEXT into the article directly above/below within a small gap + (NT R75→P132 gap14, R7→H98 gap36) — measure first (risk: wrong-article merge). + Harder: bind orphan IMAGE (R25) to caption+nearest article. +2. **JG front-page over-merges** R65 (4DL), R64 (residual 2DL below headline) — dense section + roundups; may need below-headline multi-headline split. +3. **JG H28 L-shape white space** — needs a signal that distinguishes it from AJ R48/R53 (which + are fine). Candidate: only when the empty corner is masked-out *foreign* article AND the + block's tail is a single-column sidebar with its own sub-head/byline → split the sidebar. + +## Hard limitation: undetected headlines (OCR gap, not a cropping bug) +On some pages PaddleOCR detects **no doc_title** for the top articles (JG pg1: only 3 doc_titles +total, ALL in the bottom half; top-half headlines `భానుడి భగభగ` / Indiramma / కరీమాబాద్ were +missed entirely). A geometric crop cannot include a headline that has no region. Symptoms: +DL-anchored crops with the heading missing (JG a01_DL3, a03_DL4) and lead photos orphaned from +their headless article (JG a02_O25). The REAL pipeline recovers these via the paid Claude +"catch by eye" pass (pipeline log: "N article(s) Claude must catch by EYE — no headline box"). +The geometric `all_article_crops` (real run AND replay) will lack them. The only geometric fix +is a NEW pixel-level headline-band detector (find the large-type band above a headless dateline) +— substantial + risky for AJ/NT baselines; deferred unless explicitly requested. + +## Replay validation cheatsheet +``` +python3 _replay_all.py # AJ → all_article_crops_final +python3 _replay_namaste.py # NT Siddipet → all_article_crops_final +python3 _replay_jangaon.py # NT Jangaon → all_article_crops_review +``` +Read-only block inspection: replicate the chain (dedup → merge_stacked → drop_masthead → +promote_paragraph_titles → mark_bullet_titles → find_article_starts_by_dateline → +[probe cluster → recover_orphan_text_headlines → re-detect] → cluster_all_article_blocks). + +## Partition passes added (post iter-2) — all gated + validated +- **Seed/lock confident cells** — a block is `_locked` (confident) when it's a single complete + article (exactly 1 dateline + a headline) OR fully boxed (≥3 ruled sides). Over-merges stay open. +- **Step 4 — grow OPEN cells into gaps** (bounded by walls/cells/frame; locked never move; + strict guard: never increase overlap with a locked cell). AJ/NT: 0 open∩locked overlaps. +- **Step 5 — containing-assignment** — every unassigned region inside a cell's box joins it + (no box growth). Cleared NT pg4's 11 section-absorbed regions; counts unchanged. +- **No-overlap reconciliation** — trims a 0-member over-reaching box back to its members. +- **Lead-photo rebind (caption-anchored)** — a PHOTO→CAPTION→HEADLINE stack = lead photo; move + the photo+caption down into the headline's block. Fixes JG a06_P37/maize R29 + AJ pg4/NT lead + photos. The caption requirement excludes the H93 false positive (photo at an article's bottom). +- **Horizontal-line LENGTH FLOOR (`_lines.separator_barriers`)** — a horizontal rule counts as an + article separator only if width ≥ **0.6× column** (median text width). Drops short + caption/decorative underlines. NOTE: ≥1.0× is TOO aggressive (a single-column article's own + boundary is <1 col → ≥1.0 merged NT pg4 R121); 0.6× merges ZERO articles. Caption-vs-separator + is still decided primarily by "what's directly above the line" (photo/caption) — length is a + secondary floor. A rule below a caption ≈ caption width = caption underline (dropped); only a + rule much WIDER than the caption above it (≥1.5×) is kept as a boundary. + +## Tried & REVERTED (kept for the record) +- **Orphan-TEXT absorb** — only match was a NT classifieds box (R75/R132); couldn't verify + same-ad vs separate-ad → regression risk. Deferred. +- **Ragged-bottom sidebar split** — landed for JG H28 (split the `రాత్రికి రాత్రికే` sidebar so the + L-shape white space cleared); later the lead-photo/grow cascade made H28 tile both quote boxes + with no white space anyway. (Gate: main part must stay multi-column → AJ R53 never split.) +- **Below-divider cut** (cut content sitting below a vertical divider's END out of the right-column + article above — JG DL2 holding the maize wide photo R30). Detection is SAFE (straddle gate ⇒ + only JG R2; AJ/NT untouched), but the repair destabilised JG (merged into wrong neighbour R49; + emptied DL2 regrew). Deferred — needs correct merge-target (the dateline governing the band) + + a "don't regrow a just-cut block" guard. + +## RUN-WIDE VALIDATION (all 7 output runs, geometric, no API) — 2026-06-09 +Ran the full cluster chain on every run in `output/` and checked counts + 2+dateline over-merges. +- **AJ Siddipet ×3** (incl. 2 never-loaded runs): **12 / 18 / 14 / 20 on all three, 0 over-merges.** +- **NT Siddipet ×3** (incl. 2 never-loaded runs): **11 / 22 / 19 / 25 on all three.** +- **NT Jangaon**: 7 / 12. +- **Every `2+dateline` block is an intended ROUNDUP** (verified): NT pg2 [110,127], pg3 [54,60,62], + pg4 [67,121] (R121 = a `raw_rect` section: headline + 2 dated sub-stories + photos, same accepted + pattern as R67), JG [64,65] (front-page section roundups). +- **New runs match the baselines exactly** ⇒ rules generalise (not overfit). No crashes, no new + over-merges, no merged articles from the length floor. +Conclusion: the horizontal-line + caption handling and the partition/lead-photo passes are +validated across the whole corpus. + +## Classifieds-drop validation (end-to-end replay, no API) — 2026-06-09 +Wired `drop_classifieds_regions` into the pipeline (after `drop_masthead_regions`) + all 3 replay +scripts, then re-ran every replay: +- **AJ: 12 / 18 / 14 / 20** — baseline EXACT, **0 classifieds dropped** (no false positives). +- **NT Siddipet: 11 / 22 / 19 / 25** — baseline EXACT; roundups [110,127] / [54,60,62] / + [67,121] all intact; 0 classifieds dropped. +- **NT Jangaon 130635: pg1 = 7, pg2 = 13** (was 12). The only intended change: R65 shrank from + `[113,1958,1977,6279]` (page bottom, 4 datelines, swallowing the phone-listing classifieds) to + `[113,1958,1977,5113]` (3-DL front-page roundup). Classifieds R87 (`y5143–6267`, phones=8) + dropped; the freed left-column content below becomes its own block (12→13). a06_H65 renders as + a bounded roundup, no longer page-tall over the classifieds. +- **OCR cost addressed:** detector now OCRs each tall (≥400px) region at **half-res `eng`-only** + (was full-res `tel+eng`) — ~2× faster per call, verified to produce the IDENTICAL drop set + (R87 still phones=8/kw=True; 6 sampled news bodies still 0/False). +- NOTE: R65 still includes a *court/legal* notice stacked in its left column. Per the user's + "classifieds ONLY" directive this is intentionally left; separating court notices is deferred. + +## Court/legal-notice carve validation (end-to-end replay, no API) — 2026-06-09 +Follow-up directive: "include court/legal notices as box/section in populating the crops + debug +img." Added `tag_legal_notices` (pipeline + 3 replays, after classifieds) + the carve pass in +`cluster_all_article_blocks`, then re-ran every replay: +- **JG 130635 pg2: 13 → 14 crops.** The court summons (R51/R89/R101/R108/R93) was carved out of + R65 into a new **`a08_L89`** block `[118,2887,595,4080]` (5 regs) — a clean, complete notice + ("IN THE COURT OF THE HONBLE JUNIOR CIVIL JUDGE … BY ORDER OF THE HON'BLE COURT … ADVOCATE"). + R65 dropped 30→25 members and **whites the carved area out** (verified in `a06_H65` — no + duplication). The debug boundary map shows it as **LOCK #89 [L] (5)**, nested in #65. pg1 = 7. +- **AJ: 12 / 18 / 14 / 20** — baseline EXACT, 0 legal carves. +- **NT Siddipet: 11 / 22 / 19 / 25** — baseline EXACT, 0 legal carves; roundups intact. NT pg3 has + a real petition notice (R47/R79/R89) but it straddles the NARROW locked #75 + open #62, so Gate 3 + (wide-host-only) correctly SKIPS it → baseline untouched. +- Detection sweep (read-only) across the corpus: legal keywords flagged ONLY the JG court notice + and the NT pg3 petition; **0 false positives on any AJ/NT news region.** +- COST: `tag_legal_notices` OCRs ~40–70 single-column text regions/page (the expensive part). + Acceptable for the paid pipeline (Claude vision dominates), but a candidate for a cheaper + pre-gate (e.g. skip dateline-anchored regions) if it becomes a bottleneck. +- DEFERRED (harder, needs the seed-before-clustering model): a legal notice that straddles a + NARROW straight-cropped article (NT pg3) — to carve it cleanly the host article's box must be + shrunk away from the notice, which can't be done by masking alone. + +## Type-2 split validation (end-to-end replay, no API) — 2026-06-09 +Directive: Type-1 (shared header + common bottom rule) = ONE rectangle (fine); Type-2 (stacked +articles, own headline+dateline each, no common rule) = SEPARATE articles. Added the split pass. +- **JG 130635 pg2: 14 → 16 crops.** R65 (the კალెక్టర్ over-merge) split into 3 clean articles: + R65 కలెక్టర్ `[113,1958,1975,2859]`, R86 సుందరయ్య `[123,2894,1977,4292]`, R112 పోస్టుమార్టం + `[133,4325,1975,5113]`. Court notice R89 [L] still its own section. pg1 = 7. +- **AJ: 12 / 18 / 14 / 20** — baseline EXACT (after the dateline-OWNERSHIP fix; the first cut wrongly + split AJ pg1 R50/R57 and pg3 R35/R79 — headline+subhead sharing one dateline — 13/15, now fixed). +- **NT Siddipet: 11 / 22 / 19 / 25** — baseline EXACT. The 2+dl roundups [110,127]/[54,60,62] are + single-header-multi-dateline (Type 1) so they have <2 dateline-owning headlines → NOT split; + [67,121] are `_raw_rect` → excluded. All correctly preserved. +- KNOWN minor issue: R65's left column held a furniture stack (court notice + classifieds logo + + a small job ad + a గమనిక notice). The court notice is carved (L89) and the main classifieds + (R87) dropped, but the leftover logo (R78 image) bleeds into R86's crop and the job ad + గమనిక + bleed into R112's crop (they're <400px / images, below the classifieds-drop gate). Secondary + cleanup — extend furniture detection if needed. [RESOLVED below: furniture section.] + +## Furniture section (Type-2 split) — column-aware member assignment — 2026-06-09 +Directive: when splitting a Type-2 stack, LEAVE classifieds / notices / unrelated content OUT of +the article crops, but KEEP that section visible on the debug empty-canvas map. Implemented +**geometrically (no OCR)** inside the split: each member is assigned to the nearest headline +at/above it ONLY IF it also sits within that headline's COLUMN x-span (±0.3 col). A member that +lands in a headline's y-band but a DIFFERENT column is unrelated content sharing the page — it +falls out as LEFTOVER and is grouped into a `_furniture` section block. Furniture blocks are: +(1) EXCLUDED from article crops — `crop_all_article_blocks` skips saving them, and because each +article sub-block's bbox is rebuilt from its in-column members only, the furniture column isn't +even inside the article rectangle; (2) DRAWN on the debug boundary map as a hatched GREY +"FURNITURE #id" box; (3) NEVER saved as a crop. The split fires only on JG R65, so this touches +ONLY that page — AJ/NT untouched. +- **JG pg2:** R65 → R65 కలెక్టర్ + R86 సుందరయ్య (x682, was x123 w/ the classifieds logo) + R112 + పోస్టుమార్టం (x682, clean) + R89 [L] court notice + **F44 furniture** [123,4139,626,5113] + (classifieds logo + job ad + గమనిక — debug-only, not cropped). 16 saved crops; boundary map shows + the FURNITURE box. a09_H86 & a13_H112 verified clean (no furniture bleed). +- **AJ 12/18/14/20** and **NT 11/22/19/25** — baselines EXACT. + +## Type-2 split — paragraph_title anchors + ';' dateline + section discriminator — 2026-06-09 +Two more JG pg2 over-merges were reported (R97 = గుండె చికిత్స + మృతుడి పరామర్శ; R64 = జాతీయ రహదారి + +ఉపసర్పంచుల చెక్ పవర్). Three root causes, three fixes: +1. **2nd headline detected as `paragraph_title`, not `doc_title`** (R58, R84) → my split's anchors + were doc_title-only. FIX: anchors = `doc_title` OR `paragraph_title`, with COLUMN-AWARE dateline + ownership (each dateline → nearest headline above WHOSE x-span covers it; a caption in another + column can't steal it). Captions/sub-heads still own no dateline ⇒ never an anchor. +2. **2nd dateline's colon OCR'd as a semicolon** (R97: 'చిల్పూరు, మే 19 ;') → `_dateline_in_text` + required ':'. FIX: accept ';' too (guards: digit + short prefix + no sentence punctuation). +3. **R64 was `_raw_rect`** (banner-rectangle extended the wide headline to the next banner, NO + spanning rule) so the split skipped it. FIX: the split's Type-1 exclusion is now the `_section` + flag (set ONLY by the section-banner pass, which requires a real spanning bottom rule), not the + coarse `_raw_rect` flag. R64 (no rule) → splits; R67/R121 (real section) → stay one rectangle. +- **JG pg2 16→18:** R64→R64+R84, R97→R97+R58 (verified: each crop is one complete article). +- Side effects, all VERIFIED correct (same rule, genuinely distinct articles — not fragmentation): + NT pg2 22→23 (అంగన్‌వాడీ R127 off the ప్రజావాణి section), NT pg3 19→20 (పర్యావరణ R60 off), AJ pg4 + 20→21 (గజ్వేల్ R12 — a real dateline the ';' fix recovered; an existing pass then split it). + R110 (ప్రజావాణి) stayed whole as a `_section` ⇒ the discriminator protects real sections. + +## Orphan-PHOTO binding — an image may never stand alone — 2026-06-09 +RULE (user): an article may be orphan, but an IMAGE may not — every photo must belong to some +article. A final pass in `cluster_all_article_blocks` binds each ORPHAN-PHOTO block (no +doc_title/paragraph_title headline AND no dateline; image-only, OR image + caption only, OR +image-area ≥50%) to the best NEWS article (furniture/legal blocks are excluded as targets): the +article whose top is just BELOW the photo in the same column (lead photo), else just above, else +the nearest column-overlapping article. A TEXT/body orphan is LEFT ALONE. +- **Sakshi pg1 23→22, pg2 27→20:** every orphan photo bound (R28→R78 lead photo verified — the + 'collector inspecting BL-1' photo + caption now sit in R78; pg2 R38/R42/R48/R52/R60/R63 all + bound). **0 image orphans remain** on either page. The text orphan R18 (pg1) correctly stays. +- **JG pg1 7→6:** orphan photo R25 bound into its article. +- **AJ 12/18/14/21** and **NT 11/23/20/25** — UNCHANGED (no qualifying image orphans on those). +- NOTE: Sakshi itself has rough article structure (headlines/datelines under-detected, no rules) — + some bound photos land on a column-overlapping article that may need per-case review; but the + rule (no floating image) now holds. Deeper Sakshi clustering is a separate task. +- **Respect horizontal separators in the photo bind (refinement):** a photo binds to an ADJACENT + article (gap ≤200px) even if a rule sits between them — that rule is the article's own headline + border and the photo is its lead photo (JG R28→R78, 7px gap). But a FAR fallback bind is REJECTED + if a horizontal rule LONGER than a caption (>0.6 col) lies between the photo and the article — + the user's cross-paper line rule. This kept the Sakshi pg2 "student photo" R52 (top of page) from + binding down through two FULL-WIDTH rules + a page of white space into R118 (the భక్తులకు/ + Kaleshwaram story); R52 now binds to R127 in its own band, and `a08_H118` is clean. Verified: + Sakshi pg1 22 / pg2 20 (0 image orphans); AJ 12/18/14/21, NT 11/23/20/25, JG 6/18 all unchanged. + +## Sakshi page-1 deep fix round — 2026-06-10 (visual review of all 22 pg1 crops → 14 BAD) +A crop-by-crop visual audit of Sakshi JG pg1 found 14/22 BAD crops. Root causes & fixes, ALL +Sakshi-gated (AJ 12/18/14/21, NT 11/23/20/25, JG 6/18 re-validated EXACT after every change): +- **Masthead drop for Sakshi** (`drop_masthead_regions`, `paper=="sakshi"` branch, every page): + drop every region that ends ABOVE the topmost headline-type region (sanity: that line must be + in the top 30%). pg1: weather box R64, photo-brief R62/R59, nameplate R63, and the page-date + strip R71 — which OCR'd as a DATELINE and seeded a fake DL71 article. pg2: R68/R71/R84/R104. +- **`detect_sakshi_headline_bands`** (called after `promote_paragraph_titles`, pipeline + replay) + — the big one. Sakshi's display headlines are often NOT boxed by PaddleOCR at all. Two passes: + 1. **WIDEN**: a doc_title grows sideways over adjacent UNCOVERED large-ink columns — its + un-boxed kicker half (R52 'అనూహ్య స్పందన' + red kicker 'రిబేట్ ఆఫర్‌కు'). Widest titles first; + extensions join the coverage map so a narrow neighbour can't steal a wide headline's kicker. + Ink within 25px of a foreign region's edge is the neighbour's own glyph overflow — ignored. + 2. **SYNTHESIZE**: an uncovered big-type band (55–280px tall, ≥0.75 col wide, ink 5–60%, text + following below, never above the first real headline) becomes a synthetic `doc_title` + (`_synth_band`) anchor. Found BOTH missing pg1 banners (lift-1 'జూలై 31...' R99 and SIR + 'మ్యాపింగ్‌లో జిల్లాకు అగ్రస్థానం' R100) and pg2's గమనిక box R139. CRITICAL detail: two + headlines can share pixel rows with only a ~50px gutter (same size as a word gap) — a gap + splits a band run ONLY if it contains a known text-column LEFT edge (≥35px wide); plain + gap-size segmentation merges both banners into one anchor (measured: pg1 12 crops, 45-member + monster). +- **Box-edge rule disarm** (`_is_box_edge` in `_barrier_between`, Sakshi only): a horizontal rule + with a PARALLEL partner 80–400px away (≥70% overlap) is a quote/bullet-box border, never a story + separator. Without it the lift-1 bullet-box borders fenced the article's own columns off. +- **Photo-bottom border disarm** (`_in_photo`, Sakshi only): a rule within 14px BELOW an image, + horizontally inside the photo span ±25px, is the photo's frame/caption rule (PaddleOCR's box + ends a few px above the printed border, so the interior test missed it). Without it R61/R46/R23 + (lift-1 caption + col-3 body) fell into the story below. +- **Caption never a headline line** (headline-stack recovery, all papers — baselines unaffected): + a figure_title directly under an image (≤90px) is a CAPTION; stop the stack there. Without it + the synth SIR headline pulled the Indiramma photo's caption R69 down as a "wrapped line". +- **No lead-photo rebind into a `_synth_band` headline**: a synth band's lead photo is BELOW it by + construction; a photo+caption ABOVE it belongs to the story above (R36/R69 stay with R77). +- **Continuation-column fold** (Sakshi only, end of clusterer): a TEXT-only orphan block directly + beside an article block with the SAME vertical band (top ±90px, ≥80% overlap, gap ≤80px, no + vertical rule between) is the other column of the same box → merge (pg1 R18 → R79; the text even + continues mid-word 'విద్యా సంస్థ|లకు' across the two). +- **Stray-image re-home** (Sakshi only): an image member vertically DETACHED from the rest of its + block but fully inside ANOTHER article's box moves there. A 3-column synth banner column-overlaps + everything below it, so R50's tiny సాక్షి-ఎఫెక్ట్ clipping R44 (whose own narrow headline does + NOT column-overlap it) had fallen to the banner 1100px above. +- **Narrow-crop masking** (`crop_all_article_blocks`, all papers): narrow (≤1100px) crops now white + out regions OWNED BY OTHER blocks (gpad=2) before the ink-trim — kills the sliced-glyph slivers + at crop edges. Un-owned ink is left alone (it may be the crop's own un-boxed headline). Members + are repasted, so nothing of the article itself can be erased. Counts unchanged on all papers. +- **Review overlays**: `crop_all_article_blocks(..., overlay=True)` (used by `_replay_sakshi.py`) + draws member boxes on the review crops — blue=anchor, magenta=synth band, green=member. +- **RESULT (Sakshi JG pg1): 22 → 16 crops, every reduction a genuine reunification.** Visual + re-audit of all 16: 14 GOOD / 2 MINOR / 0 BAD (was 8 GOOD / 14 BAD). The rebate feature package + (was shredded into P94+H52+DL71+DL11 with sliced letters) is ONE crop; lift-1 and SIR-mapping + (whose headlines have NO region) are complete articles anchored by synth headlines; H77 is + exactly the Indiramma story; H54/H78/H55 no longer hold neighbours' content. pg2: 20 → 17 + (top photo-gallery orphans folded into their stories; not yet visually audited). +- **NEW Sakshi baselines: pg1 → 16, pg2 → 17.** +- Known remaining (MINOR): a01_H55 loses the unboxed last line of its మ్యూజియాలజీ sub-brief + (no region covers it; the next block's white-out grazes it); the EAPCET ad image R33 sits + mid-article in H55 (it IS physically inside the article band; ad detection deferred). +- Known remaining (pg2, PRE-EXISTING): the Type-2 split's column-aware leftover pass marks the + విద్యార్థిని ఆత్మహత్య article as FURNITURE (R4/R10 F) → it is NOT cropped at all. Present in the + pre-fix baseline too. Needs its own round. [RESOLVED in the pg2 round below.] + +## Sakshi page-2 round — 2026-06-10 (visual audit of all pg2 crops; user-reported defects) +Fixes, validated after each change (AJ 12/18/14/21, NT 11/23/20/25 incl. 2+dl=[110]-only on pg2, +JG 6/18 — all EXACT): +- **a04_H118 body truncated at the column boundary** (col5 ends mid-word 'నిర్మా-', col6 holds + 'ణం…'): the col6 body R20 was stolen by the headless dateline R14's GROW-UP, which climbed PAST + the R14 story's own subhead R121 because `_is_caption` lets a 197px two-paragraph text through + (≤200px + a photo 82px above). FIX (all papers, baselines unchanged): grow-up STOPS after + claiming a `paragraph_title` — the heading IS the article's top; nothing above it is ours. +- **విద్యార్థిని ఆత్మహత్య never cropped** (whole story = furniture): its heading R88 IS detected + (paragraph_title) but the R41 Type-2 split's anchors don't span col4 → all 8 regions fell out + as LEFTOVER → furniture. FIX (all papers; gates keep AJ/NT/JG exact): a leftover group that + LEADS with its own COLUMN-WIDE heading (≥0.6 of the group width — a narrow WANTED label stays + furniture) and has body text → emit as an ARTICLE (kind P), not furniture. +- **a13_H93/'విషాదయాత్ర' cut mid-sentence, its tail + tributes in the girl's crop**: the col4 + leftover zone holds TWO stacked items (girl's story y3658-4643, శ్రీనివాస్ tail y4723-5938) with + a separator rule at y4679 between them. FIX (Sakshi only): (1) leftover clustering never merges + across a separator rule; (2) a heading-less leftover group whose band MATCHES an adjacent-column + split article (top ±90px, ≥80% overlap, gap ≤80px) is that article's column CONTINUATION → its + members merge there (the tail's text literally resumes H93's "...అయితే వైద్యులు" mid-sentence). + ORDER MATTERS: the wide-heading→ARTICLE check runs FIRST, else the girl's story (gap-close to + H99's band) band-attaches into H99. Both passes are Sakshi-gated: un-gated they re-shuffled an + NT pg2 leftover into R105 (2+dl-blocks [110]→[110,105]) — caught by the 2+dl indicator and + re-gated. +- **a03_P129/a05_H91: dead farmer's portrait R47 + caption R115 stuck at the briefs column's + bottom, white hole in his own story's crop**: the stray-image re-home pass couldn't see R47 as + detached because its caption R115 sat below it inside the same block. FIX: re-home treats an + image + its caption (figure_title ≤90px below, ≥40% overlap) as ONE UNIT — excluded from the + envelope test and moved together (→ H91, inside whose box they sit). +- **గమనిక notice box synthesized as a headline** (R139, its own bogus crop): a filled red header + measures ink 0.51; real Sakshi display headlines measure 0.30-0.35. FIX: synth band ink ceiling + 0.60 → 0.45; the notice text returns to the classifieds-column furniture (not cropped). +- **RESULT (Sakshi JG pg2): 20 → 17 crops.** Visual audit of all 17: 11 GOOD / 4 MINOR / 2 BAD → + after the H93/P88 boundary fix the BAD pair re-audited GOOD. The H118 two-column body, the + విద్యార్థిని article (new crop P88), H91's portrait, and the P129 briefs column all verified. +- Known remaining (pg2, magazine-style kids/summer feature band only): H82/H127 share the gallery + collage and H82's story conclusion ('–దుగ్గొండి' sign-off + 'నీతి' moral box) renders inside + H127's crop — a collage-page attribution problem, low priority for the news pipeline. The + classifieds crops (P136/P111) bundle neighbouring ads and clip one ref number (MINOR). + +## POLITICAL crops now BLOCK-SNAP to the geometric blocks — 2026-06-10 +`crop_political_articles` used to re-derive each Claude-selected article's rectangle from the raw +member_region_ids union plus ~400 lines of parallel ad-hoc geometry (banner walls, foreign-headline +clips, floors, grow-right) — none of which carried the geometric-crop fixes. Now it BLOCK-SNAPS: +- It clusters the page once (`cluster_all_article_blocks`) and renders every block crop via + `crop_all_article_blocks` into a temp dir — the exact validated pipeline (furniture drop, + un-boxed headline recovery, neighbour masking, box-edge rules, continuation folds). +- Each political article is mapped to the geometric block owning the MAJORITY (≥50%) of its + member regions; that block's crop is shipped as `article.png` (`method: + "smart_extractor_blocksnap"`, info.json carries the block's full member list). +- A block already claimed by an earlier article on the page is not re-used (no duplicate crops); + any article that fails to map falls back to the legacy path unchanged. +- The pipeline call passes `dateline_starts` + `sep_lines` through so the clustering matches the + all_article_crops output exactly. +- VALIDATION (no API — `_replay_political.py` replays saved `page_NNN.political_result.json` + member lists through the new path): Sakshi JG pg1 2/2 snapped (the rebate package + DL5, both + verified visually as the block-quality crops); AJ Siddipet pg2/3/4 4/4 snapped (10-13 of 12-13 + member votes, incl. the R48/R33 banner blocks); NT Jangaon pg1 2/2 snapped (13/13, 4/4). ZERO + legacy fallbacks needed on the corpus. +- NOTE: replayed member lists come from OLD paid runs (stale region understanding) — e.g. the + Sakshi 'Transfers delayed!' entry maps to the rebate block because its old members [66,29] sit + there now. On the next paid run Claude selects members against the CURRENT regions (incl. the + synthesized Sakshi headlines, which are in regions.json and the prompt's region list), so the + member→block vote will be consistent end-to-end. + +## New-run defects round (Sakshi_Jangaon District_20260513, first PAID run on new code) — 2026-06-10 +User-reported bad crops on the fresh paper; three more Sakshi-gated passes (validated: old-run +Sakshi pg1 16 / pg2 17 UNCHANGED; AJ 12/18/14/21, NT 11/23/20/25 [110], JG 6/18 all EXACT): +- **pg2 a02_H106 was a 695×5055 column strip** (statue story + 4000px of masked white): + TWO causes. (a) The dance-photo caption R104 (page top) fell down-column into H106 — + its photo R44 lived in another block. FIX: **stray caption re-home** — a figure_title member + moves to the block that owns the image it abuts (±90px above/below, with **-30px overlap + tolerance**: PaddleOCR's photo box overlaps the caption strip by a few px, R104 vs R44 = -11px, + which defeated a 0-gap test on the first attempt). Donor block shrinks back to its members. + (b) The aviation story's heading is a `paragraph_title` 1200px below the statue story's + doc_title in-column → never promoted → whole stack merged. FIX: **stacked-story heading + promotion** — a pt ≥0.55 col wide with content directly beneath (≤150px) and the nearest + in-column base anchor **>900px** above becomes an anchor. The gate was first 600px and broke + the old-run rebate package (its sub-head R94 sits 614px under the widened R52 headline and + re-split, whiting R88's box over the stats infographic — caught by re-validating the old run); + 900px keeps feature sub-heads folded and still promotes genuinely stacked stories. +- **pg2 a08_H100 stretched ~900px up with a foreign fragment above its own headline** (the NPDCL + story's tail folded down-column into H100). FIX: **stray text re-home** — a plain-text member + of an H block lying entirely ABOVE the anchor headline's top is never that article's content + (kickers/decks are title-typed); move it to the smallest other article box containing it. +- Verified visually: new-run a10_H106 (639×1326, exactly the statue story), a09_H100 (complete, + no stray top), old-run a02_H52 (infographic numbers 15,941/5.79/11,376/3.72/1,00,20,087 intact). +- New-run pages have NOT had a full visual audit (only the reported crops + block maps checked); + pg1 = 14 crops / pg2 = 16 on replay. + +## NT Adilabad flood-special front page — NT rule split + late structural cut — 2026-06-10 +NamastheTelangana_Adilabad_20260528 pg1 produced 5 page-scale OVERLAPPING mega-blocks. This page +is DETECTION-LIMITED at the source: 65 regions only, 4 doc_titles for ~8 stories, and BOTH big +feature headlines ('రెక్కల కష్టం.. నీళ్లపాలు', 'వడ్ల తరలింపు ఎక్కడిదాకా?') are INSIDE PaddleOCR +image boxes — invisible to every geometric pass incl. the synth-band detector (coverage test). +With anchors missing, the image-must-belong bind + grow/merge built page-tall blocks. Fixes: +- **NT horizontal-rule split enabled** (was Sakshi-only): same machinery, but only at rules + ≥1.8 col (per the line study a ≥2-col rule is an article boundary on NT; the boxed grid's + ~1-col caption/box rules must not cut). _replay_adilabad.py added (NTA_RUN env overridable). +- **NT LATE structural cut** (end of clusterer, after grow/merge/photo binds): the mid-pipeline + split runs BEFORE the passes that built the mega-block, so DL5 [668,901,4084,6302] was never + re-checked against the 4.4-col rule at y=4764 (and at mid-split time the bottom band's R30 + headline wasn't yet a member, so the new-start gate failed). Final blocks (not locked/raw_rect/ + section/furniture) are re-cut at ≥1.8-col rules with the underline guard; a rule ≥2.2 col cuts + even with no detected start below. DL5 → DL56 (the whole flood feature, verified coherent) + + H30 ('తడిసిన వడ్లు కొనాలి' bottom band, verified clean). pg1 5 → 6 crops. +- VALIDATION: NT Siddipet 11/23/20/25 EXACT (2+dl [110],[54,62],[67,121]); NT Jangaon 6/18 EXACT; + Adilabad pg2 31 crops unchanged (no shredding). AJ/Sakshi untouched (gate). +- HONEST REMAINDER: H44 still spans the page top (left story + photo row + nameplate share the + band; the 60%-block-span guard correctly refuses partial rules on a page-wide block) and the + H59/DL9 boxes still overlap neighbours. Only a better layout model (headline bands inside + photo boxes) or the paid pipeline's Claude-vision pass can fully fix this page. NT masthead + drop never fired here (nameplate is INSIDE image R46, no header region) — benign, the strip + rides atop the photo-row crop. + +## ⚠ REPLAY-TOOLING BUG (found 2026-06-10, fixed): _ACTIVE_PAPER was never set for NT +`_replay_jangaon.py` / `_replay_namaste.py` never set `se._ACTIVE_PAPER`, so every prior "NT +validated EXACT" run measured with ALL paper-gated clustering passes OFF while the real pipeline +runs them ON. (AJ has no gated passes, Sakshi's replay always set it — only NT was affected.) +Both scripts now set it; `_replay_jangaon.py` takes JG_RUN env (default = the 130635 run; the +user-facing run of the same paper is ..._20260608_124720). FIRST true gated-NT measurement: +NT Siddipet 11/23/20/25 + 2+dl [110],[54,62],[67,121] EXACT (gated passes are no-ops there ✓), +JG both runs 6/18, Adilabad 6/31. + +## NT Jangaon pg1 (user-reported: a01_DL3 white holes, a04_H28 over-merge, a05_DL2 foreign photo) +Root cause is the same family as Sakshi/Adilabad: THREE un-detected display headlines (brush-font +'భానుడి భగభగ', script 'నమ్మిండు..', DL2's heading). The heat feature's lower content (vendor + +ice-cream photos, captions, continuation column) folded DOWN to the 'ఆలస్యంపై రైతుల ఆగ్రహం' +headline 2200px below (nearest-anchor metric); the white holes in a01_DL3 were those stolen +regions masked out. Fixes (gated `in ("sakshi","namaste_telangana")` — AJ untouched): +- **Re-home family enabled for NT** (stray image+caption unit, stray caption, above-own-headline + members). The above-headline re-home is GENERALISED from text-only to text/image/figure_title + (title-typed members stay — kickers/decks are legitimate above a headline). +- **Synth headline bands enabled for NT — FRONT PAGE ONLY** (`n == 1`): inner NT pages are dense + and well-detected; un-gated synth split the protected Siddipet pg4 roundups (25→28, caught). + Band height cap 280→400 (tall display/script faces; ink-fraction still rejects photos). +- RESULT (JG 124720 pg1, counts unchanged at 6): DL3 = the COMPLETE heat feature (mem 8→13, no + white holes, verified visually); H28 starts AT its own headline [y3089, was 838] = exactly the + ఆగ్రహం article (verified); DL2 = clean single-column committee article, its foreign tractor + photo+caption re-homed to R29's procurement story (verified). +- Residual: DL2 remains heading-less (its display heading has no region and the synth band gates + didn't admit it) — hard-limitation class; the heading pixels sit above the crop. diff --git a/EDGE_CASES.txt b/EDGE_CASES.txt new file mode 100644 index 0000000..cdcada5 --- /dev/null +++ b/EDGE_CASES.txt @@ -0,0 +1,266 @@ +================================================================================ +TELUGU POLITICAL-ARTICLE EXTRACTOR - EDGE CASES, BUGS & RULES +================================================================================ +Pipeline: PDF -> PaddleOCR layout regions -> Claude grouping + (identify_and_group_political_articles) -> deterministic geometric + safeguards in crop_political_articles() -> per-page crops -> + political_articles.json. Separator/line + page-bottom detection + lives in _lines.py (classical CV, no ML, no API). + +This document lists every edge case / bug / rule handled so far, plus the ones +still in progress. Each point: PROBLEM -> RULE/FIX -> STATUS -> VERIFICATION. + +Papers seen (identified by page-1 pixel height): + 6422 = namaste_telangana (no printer calibration strip) + 6549 / 6833 = andhra_jyothi (CMYK calibration strip at bottom) + 7040-7060 = sakshi (CMYK calibration strip at bottom) + +================================================================================ +1. GROW-RIGHT TO THE BOUNDING VERTICAL WALL [FIXED] +================================================================================ +PROBLEM: + An under-grouped article was cropped too NARROW - its body clearly continued + into the next column to the right, but Claude grouped only the left column, so + the crop stopped mid-article. + Example: run 20260602_093820 p003_a003 "రైతన్న లోడెత్తేదెలా?". + +RULE / FIX (in crop_political_articles, after the Rule-B floor block): + - Detect a long vertical separator line (len >= 600 px) that sits to the RIGHT + of the current crop (x > rx2 + 10) and overlaps the article vertically + (> 50% y-overlap). That line is the article's right "wall". + - Absorb every text / image / figure_title / paragraph_title region whose + center-x falls between the current right edge (rx2) and the wall. + - Push the right edge out to the wall: rx2 = wall_x, right_cap = wall_x. + - The padding step clamps x2 = min(x2, right_cap) so it never overshoots. + +WIDTH-CAP GUARD (prevents over-grab): + too_far = (wall_x - rx2) > max(art_w, 1) + If the wall is farther away than the article's own width, it is a page-margin + rule swallowing a NEIGHBOUR, not the article's wall -> do NOT grow. + Also bail if a FOREIGN doc_title (a different story's headline) lies inside the + strip we are about to absorb. + Regression that drove this guard: run 20260601_234709 p007_a001 grew from + [77,634,1035,1609] to [77,634,4127,2581], eating neighbour "భూముల విలువ పెంపు". + +STATUS: Fixed & committed (HEAD 2345453). Prints "➡️ Grew ...". + +================================================================================ +2. ROBUST JSON PARSING OF CLAUDE'S GROUPING RESPONSE [FIXED] +================================================================================ +PROBLEM: + One page's grouping response was empty / truncated / fence-wrapped, and the + fragile parse of content[0].text crashed the WHOLE run: + json.JSONDecodeError: Expecting value: line 1 column 1 (char 0) + Example: a run crashed on page 10. + +RULE / FIX (smart_extractor.py): + - New helper _extract_json_obj(text): strips ```json fences / prose, extracts + the outermost {...} object, raises ValueError on empty/garbage. + - Gather text from ALL response content blocks (not just block 0). + - max_tokens raised to 8192 to avoid truncated JSON. + - On (ValueError, json.JSONDecodeError): save page_NNN.badresponse.txt for + debugging, then RETRY once in strict mode ("Return ONLY the raw JSON ..."). + - If the retry also fails: return {"political_articles": [], "rejected": []} + so that ONE bad page degrades gracefully instead of killing the run. + (json.JSONDecodeError is a subclass of ValueError - both are caught.) + +STATUS: Fixed. + +================================================================================ +3. RULE-B OVER-FIRE: in-article subhead wrongly treated as a new headline [FIXED] +================================================================================ +PROBLEM: + Rule B floors a crop at the next headline that starts a DIFFERENT story in the + same column. But a paragraph_title that is really an IN-ARTICLE byline / + subhead (narrow, with body text continuing below it) was being treated as a + new-article headline, so the crop was floored far too high and dropped 9 of the + article's member regions. + Example: run 20260603_112014 p003_a001 "తీరని కొనుగోళ్ల వెతలు" - paragraph_title + R79 [1193,5226,2035,5271] (an internal subhead) floored the crop at y5226. + +RULE / FIX (Rule B block in crop_political_articles): + - A doc_title floors the crop UNCONDITIONALLY (a real new story headline). + - A paragraph_title floors ONLY if there is a HORIZONTAL SEPARATOR RULE + directly above it (a printed line above a heading => genuine new section). + Implemented via _rule_above(): a detected h-line whose y is between the + primary headline bottom and the candidate top, overlapping > 40% of the + candidate's width. + - R79 has NO rule above -> no longer floors (correct). + +VERIFICATION (offline, no API): + - 20260603_112014 p003_a001 now keeps all members; bbox grew correctly down. + - 20260602_093820 R81 [1162,1879,2063,1926] HAS a rule above (y1707, + overlap 787) -> STILL floors correctly (legitimate case preserved). + - 20260601_234709 known-good crops unaffected. + +STATUS: Fixed. + +NOTE / FOLLOW-UP: the cross-run downward-growth sweep (>250 px) flagged a few +crops that grew DOWN after this change (they were previously floored by a +paragraph_title without a rule above). Largest legit one is the 112014 fix +itself. Others to re-confirm are NOT over-merges (swallowing a separate article +below): + 20260602_081720 p004_a001 DOWN+1951 + 20260531_231056 p002_a004/a005/a006, p002_a001 + 20260601_005917 p002_a006 + 20260531_205325 p002_a005 +[STILL TO VALIDATE] + +================================================================================ +4. PAGE-BOTTOM PRINTER CALIBRATION STRIP DETECTION [FIXED - needs wiring] +================================================================================ +PROBLEM: + Andhra Jyothi & Sakshi print a CMYK colour-calibration strip (registration + dots + grey bars) just above the bottom paper margin. Article crops whose + member regions reach the page bottom (esp. footer-type regions) bleed this + strip of coloured dots into the article image. + +RULE / FIX (_lines.py: detect_content_bottom): + Returns the y at the TOP of the calibration band (clamp crops to it), or the + full height H when there is no strip (nothing to clamp). + Algorithm: + a. Search the bottom search_frac (12%) of the page. + b. Per-row colour fraction col_frac (saturated CMYK: sat>55 & max>80) and ink + fraction; the strip = the bottom-most run of rows whose col_frac >= 0.06. + c. Require a WHITE PAPER MARGIN below the band down to the page edge + (>=90% blank) - a real strip is followed by margin. + d. Walk UP from the band bottom while still coloured, tolerating small white + gaps between dot rows (<=12 px). + e. Guard max_band (160 px): if the band is too tall it is fused with a photo + -> bail (return H). + f. PEAK-COLOUR DISCRIMINATOR (key false-positive fix): require the band's + peak col_frac >= 0.15. Real registration strips are dense multi-hue CMYK + dots peaking 0.22-0.45; a single-hue coloured ad/notice box peaks ~0.11. + +FALSE POSITIVE that drove the peak-colour gate: + namaste 20260603_113847 p004 has a GREEN-bordered notice box near the bottom + (col peak 0.113) embedded in article text. An earlier "white gutter above" + gate rejected it but WRONGLY rejected real AJ strips on busy pages (p3/p6 had a + photo extending into the gutter). The peak-colour gate (0.15) cleanly + separates: namaste box 0.11 -> reject; AJ 0.25 / Sakshi 0.22 / AJ-6549 0.45 -> + accept. + +VERIFICATION (offline): + AJ-6833 -> clamps ~6674-6687 (drop ~150) + AJ-6549 -> clamps ~6377-6401 (drop ~150-172) (was a false NEGATIVE before) + Sakshi -> clamps ~6720-6721 (drop ~332) + namaste -> H (no clamp) (correct - no strip; green box not clamped) + AJ pages with a full-bleed bottom photo -> H (acceptable: no clean strip). + +STATUS: Detector fixed & verified across all 4 papers. +PENDING: wire into crop_political_articles as a per-page y2 clamp + (y2 = min(y2, content_bottom), computed once per page); then re-verify + 112014 p003_a001 no longer includes calibration dots + regression-check. + +================================================================================ +5. MULTI-COLUMN BANNER HEADLINE = ONE ARTICLE SPANNING ALL ITS COLUMNS + [FIXED] +================================================================================ +PROBLEM: + When a single wide headline (banner) spans MULTIPLE columns, everything beneath + it (down to its bottom rule) is ONE article - even though it contains several + internal sub-headlines / sub-sections and photos. The foreign-headline clip + (Rule A) and Rule B wrongly treat those internal sub-headlines as new stories + and truncate the crop, so the banner article comes out severely under-cropped. + Example: namaste 20260603_113847 p004 article 1 + "రైతులను వీడని అన్‌లోడింగ్ సమస్య" + Banner headline id=64 [1264,423,3500,628] (width 2236 = 0.53 of page, + crosses the internal vertical rule at x=2388 -> multi-column banner). + Bounding vertical walls: x=1232 (left), x=3545 (right). + Bottom rule: full-banner-width h-line at y=2756 (x[1268..3512], len 2244). + Correct crop ~= [1232, 423, 3545, 2756]. + Current (buggy) crop = [1264,423,3500,2183] - cut off ~570 px too short and + not extended to the side walls. + +RULE (to implement): + - BANNER TEST: an article's top headline (doc_title) is a banner if it CROSSES + at least one internal vertical separator line (i.e. it spans >1 column). + (Equivalently width >= ~0.4 of page width.) + - For a banner article: + * Horizontal extent = bounded by the vertical separator lines just LEFT and + just RIGHT of the headline (left_wall .. right_wall). + * Bottom = the LOWEST/own bottom horizontal rule that spans most of the + banner width (x-overlap > ~0.6 of banner width) and lies below the + headline -> the first such full-width rule below the grouped content. + * Crop = [left_wall, headline_top-pad, right_wall, bottom_rule]. + * SKIP the internal foreign-headline clip and Rule B floor for this article + (its sub-headlines are part of it). + - "If the multi-column header HAS a bottom line, crop to it; if it does NOT, + fall back to the previous (member-union + Rule A/B) logic." (user's words) + +IMPLEMENTED (crop_political_articles, banner_box branch): + - BANNER TEST: topmost member doc_title width >= 0.40 * page_width. + - left_wall / right_wall = vertical separator lines reaching alongside the + headline, just left of / right of it (fallback to member edge if none). + - bottom_rule = the first horizontal line BELOW the grouped content whose + x-span overlaps > 60% of the banner width (the >60% width test rejects short + caption/photo underlines - see point 6). Require bottom_rule > headline_top+200. + - Only when BOTH walls-ish extent AND a bottom_rule exist do we set + banner_box = [left_wall, headline_top, right_wall, bottom_rule]; this OVERRIDES + the per-column clip / Rule-B floor / grow-right results and restores the full + member set. No bottom rule -> banner_box stays None -> normal logic (fallback). + +VERIFICATION (offline, namaste 20260603_113847 p4 art1): + - Banner fired: x[1232..3545] y[423..2756] -> padded crop [1217,408,3560,2771]. + - Visual check: crop now contains the full headline + every column beneath it + down to the bottom rule (previously cut at y~2183). + - Regression: banner does NOT fire on normal single-column-headline articles + (threshold 0.40*pw); other pages' crops unchanged. + +STATUS: Fixed & verified offline. + +================================================================================ +6. HORIZONTAL RULE WITH TEXT/IMAGE DIRECTLY ABOVE IT = UNDERLINE, NOT SEPARATOR + [PARTIALLY APPLIED - new rule, affects points 3, 4, 5] +================================================================================ +PROBLEM: + Not every horizontal printed line separates two articles. A line that has text + or an image DIRECTLY ABOVE it is an UNDERLINE - a caption underline beneath a + photo, or a decorative underline under a heading. Using such a line as an + article boundary (a Rule-B "rule above", or a banner BOTTOM rule, or a page + separator) is wrong. + Examples (user-supplied): photo captions like "జక్కల్ శివారులో నిర్మించిన ..." + and "నర్సాపూర్‌లో మాట్లాడుతున్న మాజీ మంత్రి ..." each sit ABOVE a thin rule - + those rules underline the caption, they do not divide articles. + +RULE (to implement): + When deciding whether a horizontal line is an ARTICLE SEPARATOR (for Rule B's + "rule above" check, for the banner bottom rule in point 5, and anywhere a + h-line is used as a boundary): + - Reject the line if the tight band immediately ABOVE it is mostly INK + (text) or is an image/figure region -> it is an underline, not a separator. + - A genuine article separator sits in a whitespace gutter (paper above AND + below), OR has a NEW headline starting just below it. + (This is analogous to the existing _isolated() whitespace-gutter test in + _lines.py used for separator-line detection.) + +APPLIED SO FAR: + - Banner bottom rule (point 5): only horizontal lines spanning > 60% of the + banner width qualify, which already excludes short caption/photo underlines + (those are < ~0.5 column wide). + - Note: _lines.detect_separator_lines() already runs the _isolated() gutter + test, so the h-lines fed to Rule B / banner are whitespace-gutter lines, not + arbitrary text strokes. +STILL TO DO: + - Add an explicit "ink/image directly above" rejection to Rule-B's + _rule_above() so a caption underline beneath a photo can never corroborate a + paragraph_title floor, even if a future detector change lets one through. + +STATUS: Partially applied (via width test + existing gutter detection); + explicit ink-above guard for Rule B still to add. + +================================================================================ +CROSS-CUTTING NOTES / CONVENTIONS +================================================================================ +- BILLING/SECURITY CONSTRAINT: do NOT call the Claude APIs while iterating. All + verification is done OFFLINE via fixture replay of saved page PNGs + + regions.json + cached per-page political_result.json. +- Calibration strip is NOT an OCR region; crops only reach it via member-region + bboxes (especially footer/header/number page-furniture regions) or padding. + Consider dropping footer/header/number members from crops as well. +- Test harnesses (in /tmp): replay_crop.py, replay_run.py, replay_regress.py + (new-vs-stored bbox diff), sweep_floor.py (flags downward growth >250 px), + test_bottom.py / test_all.py (detect_content_bottom across papers). +================================================================================ + +can u creat img with boundaris of ech articl in th  empty whit pg wit diffrnt colr ech articl so tht i can see if thy r overlppng r ? diff --git a/Extractor_Guide.docx b/Extractor_Guide.docx new file mode 100644 index 0000000..b9e428d Binary files /dev/null and b/Extractor_Guide.docx differ diff --git a/HOW_IT_WORKS.docx b/HOW_IT_WORKS.docx new file mode 100644 index 0000000..e88f0e5 Binary files /dev/null and b/HOW_IT_WORKS.docx differ diff --git a/PROCESS_AND_PROMPTS.md b/PROCESS_AND_PROMPTS.md new file mode 100644 index 0000000..78aadec --- /dev/null +++ b/PROCESS_AND_PROMPTS.md @@ -0,0 +1,296 @@ +# Telugu Newspaper Political Article Extractor +## Complete Process & Prompts Reference + +--- + +## STEP-BY-STEP PROCESS + +### Step 1: Upload PDF +- User uploads Telugu newspaper PDF on localhost:5000 +- File: `app.py` (Flask web server) +- Cost: Free + +### Step 2: PDF to Page Images +- PyMuPDF converts each PDF page to high-res PNG (4200 x 7400 pixels) +- File: `extractor.py` → `render_pdf_pages()` +- Output: `output//pages/page_001.png`, `page_002.png`, etc. +- Cost: Free (local) + +### Step 3: Layout Detection +- PaddleOCR LayoutDetection scans each page image +- Finds pixel-accurate bounding boxes for every text block, headline, image, table +- Each region gets: ID, type (text/title/image/header/etc.), bbox [x1,y1,x2,y2], confidence score +- File: `extractor.py` → `detect_regions()` +- Output: `output//pages/page_001.regions.json` +- Debug: `output//pages/page_001_regions_debug.png` (visual bounding boxes) +- Cost: Free (local) +- Typically finds 70-130 regions per page + +### Step 4: Article Grouping (Claude API) +- Sends page thumbnail image + all region IDs/coordinates to Claude +- Claude groups regions into complete articles (which text blocks belong to the same news story) +- File: `extractor.py` → `_group_with_claude()` +- Model: claude-opus-4-7 +- Cost: 1 API call per page +- Output: `output//pages/page_001.articles.json` + +### Step 5: Crop ALL Articles +- Each grouped article is cropped into its own PNG using PaddleOCR's precise bounding boxes +- Every article gets a unique ID: p001_a001, p001_a002, etc. +- File: `extractor.py` → `crop_articles()` +- Cost: Free (local) + +### Step 6: Political Filter (Claude API) +- Sends page image + list of article IDs (with headlines and positions) to Claude +- Claude evaluates each article with Q1/Q2 questions +- Returns only the politically significant IDs +- File: `extractor.py` → `_filter_articles_with_claude()` +- Model: claude-opus-4-7 +- Cost: 1 API call per page +- Output: `output//filter_reasoning_page1.json`, `political_articles.json` + +### Step 7: Keep or Delete +- Articles Claude flagged as significant: kept, updated with political analysis +- All other articles: deleted from output folder +- Cost: Free (local) + +### Step 8: Generate Outputs +- `opposition_daily_brief.txt` — readable brief ranked by priority +- `political_articles.json` — structured data +- `all_political_images/` — all kept images in one folder +- `political_articles.pdf` — all images in single PDF +- Cost: Free (local) + +### Step 9 (Optional): Text Extraction +- Run `python claude_ocr.py` separately +- Sends each kept article image to Claude Vision API +- Extracts accurate Telugu text in proper reading order +- File: `claude_ocr.py` +- Cost: 1 API call per article + +--- + +## ALTERNATIVE: Smart Extractor (ONE Claude call per page) + +File: `smart_extractor.py` + +### Steps: +1. PDF → page images (same as above) +2. PaddleOCR → all regions with precise boxes (same as above) +3. ONE Claude call: sends page image + all region IDs → Claude identifies political articles AND groups their region IDs → returns only political groups +4. Crop using PaddleOCR's precise boxes for only the political articles +5. Generate outputs + +### Difference from app.py: +- app.py: 2 Claude calls per page (grouping + filtering) +- smart_extractor.py: 1 Claude call per page (combined) + +--- + +## GROUPING PROMPT (Step 4) + +Model: claude-opus-4-7 + +``` +You group newspaper page regions into INDIVIDUAL articles. Each news story is a SEPARATE article. + +You receive: +1. A thumbnail of the full page (for spatial context). +2. A JSON list of regions on the page. Each has: + - id (int) + - type ("title", "text", "figure", "table", etc.) + - bbox [left, top, right, bottom] in pixels (top-left origin) + - top_text: a Tesseract OCR snippet from the top of text regions + - dateline: if non-null, the region starts an article body (strong signal) + +CRITICAL RULES — BE CONSERVATIVE, SPLIT MORE: +1. A `title` or `doc_title` region ALWAYS starts a NEW article. Every title = separate article. +2. A region with a non-null `dateline` ALWAYS starts a new article body. +3. Text regions attach to the NEAREST title above them IN THE SAME COLUMN ONLY. +4. Newspapers have COLUMNS. A region in column 1 NEVER belongs to an article in column 2, even if vertically close. +5. When in doubt, SPLIT into separate articles rather than merge. Too many small articles is better than one giant article containing multiple stories. +6. An article should typically have 1-5 regions. If an article has more than 6 regions, you are probably merging multiple stories — split them. +7. A banner title spanning multiple columns owns ONLY the text directly below it, not the entire page. +8. Types "header", "footer", "page_number" — make these their own separate article. +9. Figures/images attach to the nearest title above them in THEIR column only. +10. NEVER merge two articles that have different headlines or topics into one. + +Return ONLY valid JSON, no prose, no markdown fences: +{ + "articles": [ + { + "article_id": 1, + "title_region_id": , + "member_region_ids": [, ...], + "headline_hint": "", + "dateline": "" + } + ] +} + +Order articles top-to-bottom, then left-to-right by topmost member region. +Every region (except headers/footers) appears in exactly one article. +AIM FOR 15-25 articles per page for a typical Telugu newspaper front page. +``` + +--- + +## POLITICAL FILTER PROMPT (Step 6) + +Model: claude-opus-4-7 + +``` +You are a senior opposition party strategist in Telangana state, India. + +I have already cropped the following articles from this newspaper page: + +[article IDs with headlines and positions] + +For EACH article, ask yourself these TWO questions: + +QUESTION 1: "Does this article describe actual DAMAGE, LOSS, DEATH, CORRUPTION, or SUFFERING happening to people in Telangana? And can the government (state OR central) be BLAMED for it — either for causing it, failing to prevent it, or failing to respond to it?" +- The article must describe a NEGATIVE EVENT that has already occurred — not a future plan, not an announcement, not a statistic. +- There must be a VICTIM — someone who died, lost money, was cheated, is suffering, or was harmed. +- The government (Telangana state OR central/BJP) can be blamed — for causing it, failing to prevent it, or failing to respond to it. +- Central government failures also qualify IF the article describes actual harm to Telangana citizens (e.g., NEET paper leak harming Telangana students, central policy causing job losses in Telangana). +- If the article describes something POSITIVE (new scheme, development, achievement) → answer is NO. +- If the article describes a PLAN or ANNOUNCEMENT (not something bad that happened) → answer is NO. +- If there is NO victim and NO damage described → answer is NO. +- RESPONSIBILITY CHECK: Ask "WHO caused this damage?" If the damage was caused by nature (fire, flood, lightning), individual negligence (reckless driving, personal accident), criminals (theft, murder), or private parties — and the government had NO role in causing or preventing it — answer is NO. Only answer YES if the government is CLEARLY responsible through its policy, negligence, corruption, or failure to act. +- If YES → proceed to Question 2 +- If NO → REJECT this article immediately + +QUESTION 2: "Can the opposition party DIRECTLY USE this article to attack the Telangana government in a press conference, assembly session, or public rally?" +- If YES → SELECT this article +- If NO → REJECT this article + +BOTH answers must be YES to select. If either answer is NO, reject. + +DEDUPLICATION: If two or more articles cover the SAME issue or event (e.g., same farmer protest, same accident, same scam), only SELECT the ONE with the most complete coverage or the larger article. REJECT the duplicates. Do not keep two articles about the same news story. + +NOW go through EACH article ID I gave you. Ask Q1 and Q2 for each. Only select if BOTH are YES. + +IMPORTANT: This application is for TELANGANA STATE politics only. Farmer issues are the MOST IMPORTANT — always select if farmers are suffering. + +DO NOT SELECT — automatically REJECT: +- International news (wars, foreign affairs, immigration) +- Government achievements, new schemes, welfare announcements, development projects +- Newspaper masthead, headers, advertisements, classifieds +- Sports, entertainment, cinema, horoscopes +- General statistics without actual damage (birth rate, population, GDP) +- News about other states +- Resignations — ALWAYS REJECT unless the article explicitly contains words like corruption, scam, fired, forced out, or political pressure. A person resigning for personal, health, family, or workload reasons is NOT a government failure. +- PM Modi speeches or party meetings (unless specific policy failure hurting Telangana people) +- Individual accidents (road accidents, electric shocks, drownings, fire accidents) — these are NOT government failures, they can happen due to personal negligence or bad luck. Only select if there is a MASS disaster (10+ deaths) or CLEAR systemic government negligence (e.g., bridge collapse, building collapse, factory fire due to no safety inspections). + +Return a JSON object with TWO lists: + +1. "reasoning": for EVERY article (selected or not), explain your thinking: + - "id": the article ID + - "headline_english": English translation + - "q1": "YES" or "NO" + - "q1_reason": why (1 sentence) + - "q2": "YES" or "NO" (skip if q1 is NO) + - "q2_reason": why (1 sentence, skip if q1 is NO) + - "decision": "SELECT" or "REJECT" + +2. "selected": only the articles where both Q1 and Q2 are YES: + - "id": the article ID + - "headline_english": English translation of the headline + - "q1_answer": your answer to Question 1 (1 sentence) + - "q2_answer": your answer to Question 2 (1 sentence) + - "attack_angle": what opposition should say in a press conference (1 sentence) + - "priority": "high" or "medium" + - "category": one of corruption, governance_failure, public_grievance, law_order, health, education, infrastructure, ruling_party_crisis, farmer_issues + +Return ONLY valid JSON. No markdown. If no articles pass both questions, return {"reasoning": [...], "selected": []} +``` + +--- + +## SMART EXTRACTOR PROMPT (Combined — smart_extractor.py Step 3) + +Model: claude-opus-4-7 + +``` +You are a senior opposition party strategist in Telangana state, India. + +I am showing you a Telugu newspaper page image. I have also detected {num_regions} text/image regions on this page using OCR layout detection. Each region has an ID and precise pixel coordinates. + +REGIONS: +[list of all region IDs with types and bbox coordinates] + +YOUR TASK: Look at the newspaper page and identify ONLY the politically significant articles. For each political article, tell me which region IDs belong to it so I can crop them together precisely. + +For EACH article on the page, ask: + +QUESTION 1: "Does this article describe actual DAMAGE, LOSS, DEATH, CORRUPTION, or SUFFERING happening to people in Telangana? And can the government (state OR central) be BLAMED for it?" +- Must be a NEGATIVE EVENT that already occurred — not a plan, announcement, or statistic. +- Must have a VICTIM — someone who died, lost money, was cheated, is suffering. +- Government (Telangana state OR central/BJP) can be blamed — for causing, failing to prevent, or failing to respond. +- Central government failures qualify IF actual harm to Telangana citizens. +- RESPONSIBILITY CHECK: If damage caused by nature, individual negligence, criminals, or private parties with NO government role → NO. +- If POSITIVE (new scheme, development, achievement) → NO. +- If PLAN or ANNOUNCEMENT → NO. +- If NO victim and NO damage → NO. + +QUESTION 2: "Can the opposition party DIRECTLY USE this article to attack the government?" + +Only include articles where BOTH answers are YES. + +DEDUPLICATION: Same issue appearing in multiple places → group ALL related regions into ONE article. + +TELANGANA STATE politics only. Farmer issues are MOST IMPORTANT. + +DO NOT SELECT: +- International news, other states news +- Government achievements, new schemes, welfare announcements, development projects +- Newspaper masthead, headers, ads, classifieds +- Sports, entertainment, cinema, horoscopes +- General statistics without actual damage +- Resignations (unless corruption/scam/fired/forced out) +- PM Modi speeches (unless specific policy failure hurting Telangana) +- Individual accidents (road, electric shock, drowning, fire) — only if MASS disaster (10+ deaths) or CLEAR systemic government negligence + +For each selected political article, return: +- "headline_telugu": exact Telugu headline from the page +- "headline_english": English translation +- "member_region_ids": list of region IDs that belong to this article (for precise cropping) +- "q1_reason": why this is government failure (1 sentence) +- "q2_reason": how opposition can use this (1 sentence) +- "attack_angle": what to say in press conference (1 sentence) +- "priority": "high" or "medium" +- "category": one of corruption, governance_failure, public_grievance, law_order, health, education, infrastructure, ruling_party_crisis, farmer_issues + +Also return "rejected" list with brief reasoning for EVERY rejected article. + +Return ONLY valid JSON: +{ + "political_articles": [...], + "rejected": [ + {"headline": "...", "reason": "..."} + ] +} + +If no political articles found, return {"political_articles": [], "rejected": [...]} +``` + +--- + +## FILES REFERENCE + +| File | Purpose | Runs | +|------|---------|------| +| `app.py` | Flask web server, upload UI | `python app.py` | +| `extractor.py` | All processing logic (Steps 2-8) | Called by app.py | +| `smart_extractor.py` | Standalone CLI, combined approach | `python smart_extractor.py newspaper.pdf` | +| `claude_ocr.py` | Extract Telugu text from kept images | `python claude_ocr.py` | +| `political_filter.py` | Standalone political filter (backup) | `python political_filter.py` | + +## API COSTS PER PAGE + +| Approach | Claude Calls | What | +|----------|-------------|------| +| app.py | 2 per page | 1 grouping + 1 filtering | +| smart_extractor.py | 1 per page | Combined grouping + filtering | +| claude_ocr.py | 1 per article | Text extraction (optional) | diff --git a/README.md b/README.md new file mode 100644 index 0000000..5d25941 --- /dev/null +++ b/README.md @@ -0,0 +1,179 @@ +# Telugu Article Extractor + +A local web app that takes a Telugu newspaper PDF and produces one PNG + +one TXT file per article. + +## Pipeline + +``` +PDF + │ Stage 1 PyMuPDF → page_001.png (target: 4200×7400 px, fits aspect) + ▼ +Pages + │ Stage 2 PaddleOCR PP-Structure → regions.json + │ (title / text / figure rectangles with bboxes) + ▼ +Regions + │ Stage 3 Article grouping → articles.json + │ Primary: Claude API (spatial reasoning over regions, + │ sees page thumbnail + region JSON) + │ Fallback: column-aware + dateline-anchored Python rules + │ • title region starts a new article + │ • ", మే (ఆంధ్రజ్యోతి):" dateline + │ also starts a new article + │ • regions partitioned by column first to avoid + │ cross-column contamination + ▼ +Articles (bboxes) + │ Stage 4 PIL crop → article.png per article + ▼ +Article PNGs + │ Stage 5 Tesseract (lang=tel) → article.txt per article + ▼ +Done. Browse in the UI, download all as zip. +``` + +## Setup + +### 1. System dependencies + +```bash +# Ubuntu / Debian +sudo apt update +sudo apt install -y tesseract-ocr tesseract-ocr-tel libgl1 +``` + +`tesseract-ocr-tel` is the Telugu language data — required for the per-article +OCR step. + +`libgl1` is needed by paddlepaddle on headless Linux. + +### 2. Python environment + +```bash +cd telugu_extractor +python3 -m venv .venv +source .venv/bin/activate +pip install -r requirements.txt +``` + +First run of paddlepaddle/paddleocr will download model weights (~400 MB) +into `~/.paddleocr/`. This happens once. + +### 3. (Optional) Enable Claude grouping + +Stage 3 (article grouping) works two ways: + +- **Without an API key**: falls back to column-aware + dateline Python + rules. Works fine for tidy pages; struggles on banner headlines and + cross-column layouts. +- **With an API key**: Claude looks at a thumbnail of the page plus + the regions JSON and does the spatial reasoning. Much better on + complex layouts. Costs ~$0.01–0.05 per page. + +To enable, set the environment variable before running: + +```bash +export ANTHROPIC_API_KEY="your-api-key-here" +``` + +Get a key at https://console.anthropic.com/. If the variable isn't set +or the API call fails, the app silently falls back to the Python rules. + +### 4. Run + +```bash +python app.py +``` + +Then open . + +## How to use + +1. On the homepage, choose a PDF and pick a target render size. + Defaults: 4200 × 7400 px (≈ 600 DPI for a tabloid newspaper page). + Each page is scaled to fit *inside* this box preserving aspect. + +2. Click "Start extraction". A background job kicks off and the page + redirects to a job view that polls for status. + +3. When the job finishes, every article is shown as a card with: + - The cropped image (`article.png`) + - A link to its OCR'd text (`article.txt`) + - The detected dateline (e.g. `చిన్నగూడూరు, మే 15 (ఆంధ్రజ్యోతి):`) + if the grouper found one + - The first non-empty line of OCR text as a headline preview + +4. Click "Download all (zip)" to grab everything in one file. + +## What's in each job folder + +``` +output// +├── input.pdf the uploaded PDF +├── meta.json job metadata +├── pages/ +│ ├── page_001.png rendered page +│ ├── page_001.regions.json raw PaddleOCR layout output +│ └── page_001.articles.json grouping result +└── articles/ + └── p001_a001/ + ├── article.png per-article crop + ├── article.txt per-article OCR + └── info.json bbox + dateline + member regions +``` + +Everything is on disk and inspectable. If a crop looks wrong, you can +look at `pages/page_NNN.articles.json` to see exactly which regions +were grouped together. + +## Tuning the grouping + +Stage 3 has two implementations in `extractor.py`: + +### Path A: Claude API (`_group_with_claude`) + +The system prompt is in `CLAUDE_GROUPING_PROMPT`. It tells Claude: +- Use `title` regions as article starts +- Use the `dateline` field as a strong "this is an article body start" signal +- Respect column boundaries +- Handle banner headlines that span columns +- Pull quotes inside another article's bbox belong to that article + +To tune, edit the prompt. To debug, look at `pages/page_NNN.articles.json` +and check the `grouped_by` field — `"claude"` means the API call worked, +`"rules"` means it fell back. + +### Path B: Python rules (`_group_with_rules`) + +Used when no API key is set or the API call fails. + +1. **Column partition** — regions clustered by `left` coordinate. +2. **Within each column**: `title` region OR dateline match starts a + new article; everything else attaches to the current one. + +To tune the rules: +- Adjust `DATELINE_RE` for different paper conventions. +- Adjust the column-clustering threshold in `_assign_columns`. + +## Known limitations + +- **L-shaped articles**: an article whose body wraps around a photo into + a non-rectangular shape can't be captured by one bbox. The current + pipeline produces the smallest rect that covers everything, which may + include a sliver of a neighboring article. You'd need a polygon-based + crop to fix that — out of scope here. + +- **Cross-page continuations** (`మిగతా 3వ పేజీలో`): not stitched. Each + page is processed independently. + +- **PaddleOCR layout model is trained mostly on English/Chinese + documents.** Expect 80–95% of articles to be bounded correctly out of + the box. Edge cases are why the per-job folder keeps the raw + `regions.json` — so you can audit and tune. + +## License & data + +All processing is local. The app never sends your PDF or anything +extracted from it to a remote service. The PaddleOCR model weights are +downloaded once from PaddlePaddle's CDN on first run. diff --git a/_aj_base.log b/_aj_base.log new file mode 100644 index 0000000..fbc898a --- /dev/null +++ b/_aj_base.log @@ -0,0 +1,5 @@ +page 1: 12 crops, 9 datelines | Headline-strip:2, Floor-trim:2, orphan overlap:2 +page 2: 18 crops, 18 datelines | bullet titles flagged:1, Headline-strip:2 +page 3: 14 crops, 7 datelines | bullet titles flagged:1 +page 4: 21 crops, 18 datelines | bullet titles flagged:1, Headline-strip:2, Block dateline split:2, Multi-column body:2 +DONE -> output/AndhraJyothi_Siddipet District_20260602_20260603_144919/all_article_crops_final diff --git a/_aj_val.log b/_aj_val.log new file mode 100644 index 0000000..fbc898a --- /dev/null +++ b/_aj_val.log @@ -0,0 +1,5 @@ +page 1: 12 crops, 9 datelines | Headline-strip:2, Floor-trim:2, orphan overlap:2 +page 2: 18 crops, 18 datelines | bullet titles flagged:1, Headline-strip:2 +page 3: 14 crops, 7 datelines | bullet titles flagged:1 +page 4: 21 crops, 18 datelines | bullet titles flagged:1, Headline-strip:2, Block dateline split:2, Multi-column body:2 +DONE -> output/AndhraJyothi_Siddipet District_20260602_20260603_144919/all_article_crops_final diff --git a/_aj_val2.log b/_aj_val2.log new file mode 100644 index 0000000..fbc898a --- /dev/null +++ b/_aj_val2.log @@ -0,0 +1,5 @@ +page 1: 12 crops, 9 datelines | Headline-strip:2, Floor-trim:2, orphan overlap:2 +page 2: 18 crops, 18 datelines | bullet titles flagged:1, Headline-strip:2 +page 3: 14 crops, 7 datelines | bullet titles flagged:1 +page 4: 21 crops, 18 datelines | bullet titles flagged:1, Headline-strip:2, Block dateline split:2, Multi-column body:2 +DONE -> output/AndhraJyothi_Siddipet District_20260602_20260603_144919/all_article_crops_final diff --git a/_aj_val3.log b/_aj_val3.log new file mode 100644 index 0000000..fbc898a --- /dev/null +++ b/_aj_val3.log @@ -0,0 +1,5 @@ +page 1: 12 crops, 9 datelines | Headline-strip:2, Floor-trim:2, orphan overlap:2 +page 2: 18 crops, 18 datelines | bullet titles flagged:1, Headline-strip:2 +page 3: 14 crops, 7 datelines | bullet titles flagged:1 +page 4: 21 crops, 18 datelines | bullet titles flagged:1, Headline-strip:2, Block dateline split:2, Multi-column body:2 +DONE -> output/AndhraJyothi_Siddipet District_20260602_20260603_144919/all_article_crops_final diff --git a/_aj_val4.log b/_aj_val4.log new file mode 100644 index 0000000..fbc898a --- /dev/null +++ b/_aj_val4.log @@ -0,0 +1,5 @@ +page 1: 12 crops, 9 datelines | Headline-strip:2, Floor-trim:2, orphan overlap:2 +page 2: 18 crops, 18 datelines | bullet titles flagged:1, Headline-strip:2 +page 3: 14 crops, 7 datelines | bullet titles flagged:1 +page 4: 21 crops, 18 datelines | bullet titles flagged:1, Headline-strip:2, Block dateline split:2, Multi-column body:2 +DONE -> output/AndhraJyothi_Siddipet District_20260602_20260603_144919/all_article_crops_final diff --git a/_aj_val5.log b/_aj_val5.log new file mode 100644 index 0000000..fbc898a --- /dev/null +++ b/_aj_val5.log @@ -0,0 +1,5 @@ +page 1: 12 crops, 9 datelines | Headline-strip:2, Floor-trim:2, orphan overlap:2 +page 2: 18 crops, 18 datelines | bullet titles flagged:1, Headline-strip:2 +page 3: 14 crops, 7 datelines | bullet titles flagged:1 +page 4: 21 crops, 18 datelines | bullet titles flagged:1, Headline-strip:2, Block dateline split:2, Multi-column body:2 +DONE -> output/AndhraJyothi_Siddipet District_20260602_20260603_144919/all_article_crops_final diff --git a/_cls.txt b/_cls.txt new file mode 100644 index 0000000..527b674 --- /dev/null +++ b/_cls.txt @@ -0,0 +1,4 @@ +R16 text bbox=[119, 616, 612, 1899] phones=3 kw=[] :: జనగామ, మే 19 (నమస్తే తెలంగాణ) : ఆల్‌ ఇండియా కెమిస్ +R89 paragraph_title bbox=[121, 2887, 595, 2978] phones=0 kw=['in the court'] :: IN THE COURT OF THE HONBLE JUNIOR CIVIL JUDGE AT G +R93 text bbox=[174, 4010, 545, 4080] phones=1 kw=['advocate'] :: ADVOCATE - JANGAON Cell: 9848259371 +R87 text bbox=[150, 5143, 604, 6267] phones=8 kw=['classified', 'contact'] :: For Classified Ads Please Contact... 9182777552 HA diff --git a/_cmp_mc.py b/_cmp_mc.py new file mode 100644 index 0000000..c14c410 --- /dev/null +++ b/_cmp_mc.py @@ -0,0 +1,34 @@ +import sys, json, contextlib, io +sys.path.insert(0, ".") +from pathlib import Path +import smart_extractor as se +from _lines import separator_barriers + +D = Path("output/AndhraJyothi_Siddipet District_20260602_20260603_144919/pages") + + +def blocks_for(pg, strip_mc): + png = D / f"page_{pg:03d}.png" + regs = json.loads((D / f"page_{pg:03d}.regions.json").read_text())["regions"] + regs = se.drop_masthead_regions(regs, "andhra_jyothi", pg, png) + ds = se.find_article_starts_by_dateline(regs, str(png), "andhra_jyothi") + bars = separator_barriers(str(png), regs) + if strip_mc: + bars = [{k: v for k, v in b.items() if k not in ("mc", "anchor")} for b in bars] + bl = se.cluster_all_article_blocks(regs, dateline_starts=ds, sep_lines=bars) + return {b["anchor_id"]: tuple(sorted(b["members"])) for b in bl} + + +for pg in (1, 2, 3, 4): + with contextlib.redirect_stdout(io.StringIO()): + cur = blocks_for(pg, False) + base = blocks_for(pg, True) + print(f"==== PAGE {pg} ====") + changed = False + for k in sorted(set(cur) | set(base)): + if cur.get(k) != base.get(k): + changed = True + print(f" anchor {k}: base={base.get(k)}") + print(f" curr={cur.get(k)}") + if not changed: + print(" (no change from mc clamp)") diff --git a/_dump_all.py b/_dump_all.py new file mode 100644 index 0000000..463403b --- /dev/null +++ b/_dump_all.py @@ -0,0 +1,18 @@ +import sys, json +sys.path.insert(0, ".") +from pathlib import Path +import smart_extractor as se +from _lines import separator_barriers +D = Path("output/AndhraJyothi_Siddipet District_20260602_20260603_144919/pages") +png = D / "page_001.png" +regs0 = json.loads((D / "page_001.regions.json").read_text())["regions"] +regs = se.drop_masthead_regions(regs0, "andhra_jyothi", 1, png) +ds = se.find_article_starts_by_dateline(regs, str(png), "andhra_jyothi") +bars = separator_barriers(str(png), regs) +bl = se.cluster_all_article_blocks(regs, dateline_starts=ds, sep_lines=bars) +allmem = set() +for b in sorted(bl, key=lambda b: (b["bbox"][1], b["bbox"][0])): + allmem |= set(b["members"]) + print(f" {b['kind']:>1} anchor={b['anchor_id']:>3} bbox={b['bbox']} members={sorted(b['members'])}") +present = {r["id"] for r in regs if r.get("type") not in ("header","number")} +print("MISSING (not in any block):", sorted(present - allmem)) diff --git a/_gap_probe.py b/_gap_probe.py new file mode 100644 index 0000000..e75f420 --- /dev/null +++ b/_gap_probe.py @@ -0,0 +1,40 @@ +import sys, json +sys.path.insert(0, "/Users/kranthikumar/Documents/telugu_extractor") +from pathlib import Path +import smart_extractor as se + +D = Path("output/AndhraJyothi_Siddipet District_20260602_20260603_144919/pages") + +def headline_above(regions, start_bboxes, body, max_gap): + cx = (body[0] + body[2]) / 2 + cands = [r for r in regions + if (r.get("type") == "doc_title" or r.get("_promoted_from")) + and r["bbox"][0] <= cx <= r["bbox"][2] + and r["bbox"][3] <= body[1] + 40 + and (body[1] - r["bbox"][3]) <= max_gap] + if not cands: + return None, None + h = max(cands, key=lambda r: r["bbox"][3]) + gap = body[1] - h["bbox"][3] + for ob in start_bboxes: + if ob is body: + continue + ocx = (ob[0] + ob[2]) / 2 + if h["bbox"][3] < ob[1] < body[1] and cx - 120 <= ocx <= cx + 120: + return None, gap + return h["id"], gap + +for pg in (1, 2, 3, 4): + png = D / f"page_{pg:03d}.png" + regions = json.loads((D / f"page_{pg:03d}.regions.json").read_text())["regions"] + regions = se.drop_masthead_regions(regions, "andhra_jyothi", pg, png) + ds = se.find_article_starts_by_dateline(regions, str(png), "andhra_jyothi") + sb = [s["bbox"] for s in ds] + print(f"\n==== PAGE {pg} ====") + for s in ds: + rid = s["region_id"] + row = [] + for g in (900, 1000, 1100, 1300): + hid, gap = headline_above(regions, sb, s["bbox"], g) + row.append(f"g{g}:R{hid}" ) + print(f" DL R{rid:>3} gap_to_nearest_doc_title above -> " + " ".join(row)) diff --git a/_jg_base.log b/_jg_base.log new file mode 100644 index 0000000..0865293 --- /dev/null +++ b/_jg_base.log @@ -0,0 +1,36 @@ + +=== page 1: 6 crops, 6 datelines === + R 3 DL bbox=[1483,838,4073,3081] 2590x2243 mem=13 {'text': 8, 'image': 3, 'figure_title': 2} + R 4 DL bbox=[105,846,1467,4295] 1362x3449 mem=15 {'text': 11, 'image': 2, 'paragraph_title': 2} + R 28 H bbox=[1498,3089,3031,4921] 1533x1832 mem=13 {'text': 8, 'image': 1, 'doc_title': 1, 'paragraph_title': 2, 'figure_title': 1} + R 2 DL bbox=[3039,3957,4081,6269] 1042x2312 mem=4 {'text': 2, 'image': 1, 'figure_title': 1} + R 49 H RAW bbox=[106,4377,1429,6301] 1323x1924 mem=10 {'doc_title': 1, 'text': 5, 'image': 2, 'figure_title': 2} + R 29 H RAW bbox=[1500,5012,3211,6283] 1711x1271 mem=5 {'doc_title': 1, 'text': 2, 'figure_title': 1, 'image': 1} + [Containment-merge] x4 + [Headline-merge] x2 + [Banner rectangle] x4 + +=== page 2: 18 crops, 17 datelines === + R109 H bbox=[119,393,612,1899] 493x1506 mem=2 {'doc_title': 1, 'text': 1} + R 79 H bbox=[2693,405,4087,1833] 1394x1428 mem=12 {'doc_title': 1, 'text': 7, 'image': 2, 'figure_title': 1, 'paragraph_title': 1} + R 55 H RAW bbox=[679,415,2630,1919] 1951x1504 mem=14 {'doc_title': 1, 'text': 9, 'image': 1, 'paragraph_title': 2, 'figure_title': 1} + R 77 H bbox=[2692,1594,4087,1903] 1395x309 mem=2 {'doc_title': 1, 'text': 1} + R 14 DL bbox=[2045,1950,3265,3097] 1220x1147 mem=6 {'text': 3, 'image': 1, 'figure_title': 1, 'paragraph_title': 1} + R 65 H bbox=[113,1958,1975,2859] 1862x901 mem=8 {'text': 5, 'image': 1, 'doc_title': 1, 'figure_title': 1} + R 96 H bbox=[3326,1964,4087,3638] 761x1674 mem=4 {'doc_title': 1, 'text': 2, 'image': 1} + R 89 L bbox=[118,2887,595,4080] 477x1193 mem=5 {'text': 4, 'paragraph_title': 1} + R 86 H bbox=[682,2894,1977,4292] 1295x1398 mem=8 {'text': 4, 'image': 1, 'paragraph_title': 2, 'doc_title': 1} + R 64 H bbox=[2046,3116,3263,3907] 1217x791 mem=6 {'text': 3, 'image': 1, 'figure_title': 1, 'doc_title': 1} + R 97 H bbox=[3324,3756,4087,4627] 763x871 mem=2 {'doc_title': 1, 'text': 1} + R 84 H bbox=[2046,3920,3264,4820] 1218x900 mem=6 {'text': 3, 'image': 1, 'figure_title': 1, 'paragraph_title': 1} + R 44 F bbox=[123,4139,626,5113] 503x974 mem=4 {'text': 2, 'image': 1, 'paragraph_title': 1} + R112 H bbox=[682,4325,1975,5056] 1293x731 mem=5 {'text': 2, 'image': 1, 'figure_title': 1, 'doc_title': 1} + R 58 H bbox=[3322,4658,4087,5831] 765x1173 mem=2 {'doc_title': 1, 'text': 1} + R110 H bbox=[2044,4858,3265,5685] 1221x827 mem=7 {'doc_title': 1, 'text': 4, 'image': 1, 'paragraph_title': 1} + R 75 H RAW bbox=[680,5078,1975,5692] 1295x614 mem=5 {'doc_title': 1, 'image': 1, 'text': 2, 'figure_title': 1} + R 59 H RAW bbox=[680,5731,3267,6318] 2587x587 mem=7 {'doc_title': 2, 'text': 5} + R 7 DL bbox=[3308,5839,4087,6305] 779x466 mem=1 {'text': 1} SINGLE-region + [Containment-merge] x10 + [Banner rectangle] x8 + +DONE -> output/NamastheTelangana_Jangaon_20260520_20260609_130635/all_article_crops_review diff --git a/_jg_val.log b/_jg_val.log new file mode 100644 index 0000000..7b02133 --- /dev/null +++ b/_jg_val.log @@ -0,0 +1,36 @@ + +=== page 1: 6 crops, 6 datelines === + R 3 DL bbox=[1497,137,4081,3023] 2584x2886 mem=8 {'text': 7, 'image': 1} + R 4 DL bbox=[105,145,1467,4295] 1362x4150 mem=15 {'text': 11, 'image': 2, 'paragraph_title': 2} + R 28 H bbox=[1498,838,3031,4921] 1533x4083 mem=18 {'doc_title': 1, 'text': 9, 'image': 3, 'paragraph_title': 2, 'figure_title': 3} + R 2 DL bbox=[3039,3957,4081,5748] 1042x1791 mem=3 {'text': 1, 'image': 1, 'figure_title': 1} + R 49 H RAW bbox=[106,4377,1429,6301] 1323x1924 mem=10 {'doc_title': 1, 'text': 5, 'image': 2, 'figure_title': 2} + R 29 H RAW bbox=[1500,5012,4081,6283] 2581x1271 mem=6 {'doc_title': 1, 'text': 3, 'image': 1, 'figure_title': 1} + [Containment-merge] x2 + [Headline-merge] x2 + [Banner rectangle] x4 + +=== page 2: 18 crops, 17 datelines === + R109 H bbox=[119,393,612,1899] 493x1506 mem=2 {'doc_title': 1, 'text': 1} + R 79 H bbox=[2693,405,4087,1833] 1394x1428 mem=12 {'doc_title': 1, 'text': 7, 'image': 2, 'figure_title': 1, 'paragraph_title': 1} + R 55 H RAW bbox=[679,415,2630,1919] 1951x1504 mem=14 {'doc_title': 1, 'text': 9, 'image': 1, 'paragraph_title': 2, 'figure_title': 1} + R 77 H bbox=[2692,1594,4087,1903] 1395x309 mem=2 {'doc_title': 1, 'text': 1} + R 14 DL bbox=[2045,1950,3265,3097] 1220x1147 mem=6 {'text': 3, 'image': 1, 'figure_title': 1, 'paragraph_title': 1} + R 65 H bbox=[113,1958,1975,2859] 1862x901 mem=8 {'text': 5, 'image': 1, 'doc_title': 1, 'figure_title': 1} + R 96 H bbox=[3326,1964,4087,3638] 761x1674 mem=4 {'doc_title': 1, 'text': 2, 'image': 1} + R 89 L bbox=[118,2887,595,4080] 477x1193 mem=5 {'text': 4, 'paragraph_title': 1} + R 86 H bbox=[682,2894,1977,4292] 1295x1398 mem=8 {'text': 4, 'image': 1, 'paragraph_title': 2, 'doc_title': 1} + R 64 H bbox=[2046,3116,3263,3907] 1217x791 mem=6 {'text': 3, 'image': 1, 'figure_title': 1, 'doc_title': 1} + R 97 H bbox=[3324,3756,4087,4627] 763x871 mem=2 {'doc_title': 1, 'text': 1} + R 84 H bbox=[2046,3920,3264,4820] 1218x900 mem=6 {'text': 3, 'image': 1, 'figure_title': 1, 'paragraph_title': 1} + R 44 F bbox=[123,4139,626,5113] 503x974 mem=4 {'text': 2, 'image': 1, 'paragraph_title': 1} + R112 H bbox=[682,4325,1975,5056] 1293x731 mem=5 {'text': 2, 'image': 1, 'figure_title': 1, 'doc_title': 1} + R 58 H bbox=[3322,4658,4087,5831] 765x1173 mem=2 {'doc_title': 1, 'text': 1} + R110 H bbox=[2044,4858,3265,5685] 1221x827 mem=7 {'doc_title': 1, 'text': 4, 'image': 1, 'paragraph_title': 1} + R 75 H RAW bbox=[680,5078,1975,5692] 1295x614 mem=5 {'doc_title': 1, 'image': 1, 'text': 2, 'figure_title': 1} + R 59 H RAW bbox=[680,5731,3267,6318] 2587x587 mem=7 {'doc_title': 2, 'text': 5} + R 7 DL bbox=[3308,5839,4087,6305] 779x466 mem=1 {'text': 1} SINGLE-region + [Containment-merge] x10 + [Banner rectangle] x8 + +DONE -> output/NamastheTelangana_Jangaon_20260520_20260609_130635/all_article_crops_review diff --git a/_jg_val2.log b/_jg_val2.log new file mode 100644 index 0000000..7b02133 --- /dev/null +++ b/_jg_val2.log @@ -0,0 +1,36 @@ + +=== page 1: 6 crops, 6 datelines === + R 3 DL bbox=[1497,137,4081,3023] 2584x2886 mem=8 {'text': 7, 'image': 1} + R 4 DL bbox=[105,145,1467,4295] 1362x4150 mem=15 {'text': 11, 'image': 2, 'paragraph_title': 2} + R 28 H bbox=[1498,838,3031,4921] 1533x4083 mem=18 {'doc_title': 1, 'text': 9, 'image': 3, 'paragraph_title': 2, 'figure_title': 3} + R 2 DL bbox=[3039,3957,4081,5748] 1042x1791 mem=3 {'text': 1, 'image': 1, 'figure_title': 1} + R 49 H RAW bbox=[106,4377,1429,6301] 1323x1924 mem=10 {'doc_title': 1, 'text': 5, 'image': 2, 'figure_title': 2} + R 29 H RAW bbox=[1500,5012,4081,6283] 2581x1271 mem=6 {'doc_title': 1, 'text': 3, 'image': 1, 'figure_title': 1} + [Containment-merge] x2 + [Headline-merge] x2 + [Banner rectangle] x4 + +=== page 2: 18 crops, 17 datelines === + R109 H bbox=[119,393,612,1899] 493x1506 mem=2 {'doc_title': 1, 'text': 1} + R 79 H bbox=[2693,405,4087,1833] 1394x1428 mem=12 {'doc_title': 1, 'text': 7, 'image': 2, 'figure_title': 1, 'paragraph_title': 1} + R 55 H RAW bbox=[679,415,2630,1919] 1951x1504 mem=14 {'doc_title': 1, 'text': 9, 'image': 1, 'paragraph_title': 2, 'figure_title': 1} + R 77 H bbox=[2692,1594,4087,1903] 1395x309 mem=2 {'doc_title': 1, 'text': 1} + R 14 DL bbox=[2045,1950,3265,3097] 1220x1147 mem=6 {'text': 3, 'image': 1, 'figure_title': 1, 'paragraph_title': 1} + R 65 H bbox=[113,1958,1975,2859] 1862x901 mem=8 {'text': 5, 'image': 1, 'doc_title': 1, 'figure_title': 1} + R 96 H bbox=[3326,1964,4087,3638] 761x1674 mem=4 {'doc_title': 1, 'text': 2, 'image': 1} + R 89 L bbox=[118,2887,595,4080] 477x1193 mem=5 {'text': 4, 'paragraph_title': 1} + R 86 H bbox=[682,2894,1977,4292] 1295x1398 mem=8 {'text': 4, 'image': 1, 'paragraph_title': 2, 'doc_title': 1} + R 64 H bbox=[2046,3116,3263,3907] 1217x791 mem=6 {'text': 3, 'image': 1, 'figure_title': 1, 'doc_title': 1} + R 97 H bbox=[3324,3756,4087,4627] 763x871 mem=2 {'doc_title': 1, 'text': 1} + R 84 H bbox=[2046,3920,3264,4820] 1218x900 mem=6 {'text': 3, 'image': 1, 'figure_title': 1, 'paragraph_title': 1} + R 44 F bbox=[123,4139,626,5113] 503x974 mem=4 {'text': 2, 'image': 1, 'paragraph_title': 1} + R112 H bbox=[682,4325,1975,5056] 1293x731 mem=5 {'text': 2, 'image': 1, 'figure_title': 1, 'doc_title': 1} + R 58 H bbox=[3322,4658,4087,5831] 765x1173 mem=2 {'doc_title': 1, 'text': 1} + R110 H bbox=[2044,4858,3265,5685] 1221x827 mem=7 {'doc_title': 1, 'text': 4, 'image': 1, 'paragraph_title': 1} + R 75 H RAW bbox=[680,5078,1975,5692] 1295x614 mem=5 {'doc_title': 1, 'image': 1, 'text': 2, 'figure_title': 1} + R 59 H RAW bbox=[680,5731,3267,6318] 2587x587 mem=7 {'doc_title': 2, 'text': 5} + R 7 DL bbox=[3308,5839,4087,6305] 779x466 mem=1 {'text': 1} SINGLE-region + [Containment-merge] x10 + [Banner rectangle] x8 + +DONE -> output/NamastheTelangana_Jangaon_20260520_20260609_130635/all_article_crops_review diff --git a/_jg_val3.log b/_jg_val3.log new file mode 100644 index 0000000..eb07770 --- /dev/null +++ b/_jg_val3.log @@ -0,0 +1,38 @@ + +=== page 1: 6 crops, 6 datelines === + R 3 DL bbox=[1497,137,4081,3023] 2584x2886 mem=8 {'text': 7, 'image': 1} + R 4 DL bbox=[105,145,1467,4295] 1362x4150 mem=15 {'text': 11, 'image': 2, 'paragraph_title': 2} + R 28 H bbox=[1498,838,3031,4921] 1533x4083 mem=18 {'doc_title': 1, 'text': 9, 'image': 3, 'paragraph_title': 2, 'figure_title': 3} + R 2 DL bbox=[3039,3957,4081,5748] 1042x1791 mem=3 {'text': 1, 'image': 1, 'figure_title': 1} + R 49 H RAW bbox=[106,4377,1429,6301] 1323x1924 mem=10 {'doc_title': 1, 'text': 5, 'image': 2, 'figure_title': 2} + R 29 H RAW bbox=[1500,5012,4081,6283] 2581x1271 mem=6 {'doc_title': 1, 'text': 3, 'image': 1, 'figure_title': 1} + [Containment-merge] x2 + [Headline-merge] x2 + [Banner rectangle] x4 + +=== page 2: 18 crops, 17 datelines === + R109 H bbox=[119,393,612,1899] 493x1506 mem=2 {'doc_title': 1, 'text': 1} + R 79 H bbox=[2693,405,4087,1833] 1394x1428 mem=12 {'doc_title': 1, 'text': 7, 'image': 2, 'figure_title': 1, 'paragraph_title': 1} + R 55 H RAW bbox=[679,415,2630,1919] 1951x1504 mem=14 {'doc_title': 1, 'text': 9, 'image': 1, 'paragraph_title': 2, 'figure_title': 1} + R 77 H bbox=[2692,1594,4087,1903] 1395x309 mem=2 {'doc_title': 1, 'text': 1} + R 14 DL bbox=[2045,1950,3265,3097] 1220x1147 mem=6 {'text': 3, 'image': 1, 'figure_title': 1, 'paragraph_title': 1} + R 65 H bbox=[113,1958,1975,2859] 1862x901 mem=8 {'text': 5, 'image': 1, 'doc_title': 1, 'figure_title': 1} + R 96 H bbox=[3326,1964,4087,3638] 761x1674 mem=4 {'doc_title': 1, 'text': 2, 'image': 1} + R 89 L bbox=[118,2887,595,4080] 477x1193 mem=5 {'text': 4, 'paragraph_title': 1} + R 86 H bbox=[682,2894,1977,4292] 1295x1398 mem=8 {'text': 4, 'image': 1, 'paragraph_title': 2, 'doc_title': 1} + R 64 H bbox=[2046,3116,3263,3907] 1217x791 mem=6 {'text': 3, 'image': 1, 'figure_title': 1, 'doc_title': 1} + R 97 H bbox=[3324,3756,4087,4627] 763x871 mem=2 {'doc_title': 1, 'text': 1} + R 84 H bbox=[2046,3920,3264,4820] 1218x900 mem=6 {'text': 3, 'image': 1, 'figure_title': 1, 'paragraph_title': 1} + R 78 F bbox=[123,4139,626,4285] 503x146 mem=1 {'image': 1} IMG-ONLY(no text) SINGLE-region + R112 H bbox=[682,4325,1975,5056] 1293x731 mem=5 {'text': 2, 'image': 1, 'figure_title': 1, 'doc_title': 1} + R 82 F bbox=[137,4423,618,4782] 481x359 mem=2 {'text': 1, 'paragraph_title': 1} + R 58 H bbox=[3322,4658,4087,5831] 765x1173 mem=2 {'doc_title': 1, 'text': 1} + R 44 F bbox=[133,4811,616,5113] 483x302 mem=1 {'text': 1} SINGLE-region + R110 H bbox=[2044,4858,3265,5685] 1221x827 mem=7 {'doc_title': 1, 'text': 4, 'image': 1, 'paragraph_title': 1} + R 75 H RAW bbox=[680,5078,1975,5692] 1295x614 mem=5 {'doc_title': 1, 'image': 1, 'text': 2, 'figure_title': 1} + R 59 H RAW bbox=[680,5731,3267,6318] 2587x587 mem=7 {'doc_title': 2, 'text': 5} + R 7 DL bbox=[3308,5839,4087,6305] 779x466 mem=1 {'text': 1} SINGLE-region + [Containment-merge] x10 + [Banner rectangle] x8 + +DONE -> output/NamastheTelangana_Jangaon_20260520_20260609_130635/all_article_crops_review diff --git a/_jg_val4.log b/_jg_val4.log new file mode 100644 index 0000000..7b02133 --- /dev/null +++ b/_jg_val4.log @@ -0,0 +1,36 @@ + +=== page 1: 6 crops, 6 datelines === + R 3 DL bbox=[1497,137,4081,3023] 2584x2886 mem=8 {'text': 7, 'image': 1} + R 4 DL bbox=[105,145,1467,4295] 1362x4150 mem=15 {'text': 11, 'image': 2, 'paragraph_title': 2} + R 28 H bbox=[1498,838,3031,4921] 1533x4083 mem=18 {'doc_title': 1, 'text': 9, 'image': 3, 'paragraph_title': 2, 'figure_title': 3} + R 2 DL bbox=[3039,3957,4081,5748] 1042x1791 mem=3 {'text': 1, 'image': 1, 'figure_title': 1} + R 49 H RAW bbox=[106,4377,1429,6301] 1323x1924 mem=10 {'doc_title': 1, 'text': 5, 'image': 2, 'figure_title': 2} + R 29 H RAW bbox=[1500,5012,4081,6283] 2581x1271 mem=6 {'doc_title': 1, 'text': 3, 'image': 1, 'figure_title': 1} + [Containment-merge] x2 + [Headline-merge] x2 + [Banner rectangle] x4 + +=== page 2: 18 crops, 17 datelines === + R109 H bbox=[119,393,612,1899] 493x1506 mem=2 {'doc_title': 1, 'text': 1} + R 79 H bbox=[2693,405,4087,1833] 1394x1428 mem=12 {'doc_title': 1, 'text': 7, 'image': 2, 'figure_title': 1, 'paragraph_title': 1} + R 55 H RAW bbox=[679,415,2630,1919] 1951x1504 mem=14 {'doc_title': 1, 'text': 9, 'image': 1, 'paragraph_title': 2, 'figure_title': 1} + R 77 H bbox=[2692,1594,4087,1903] 1395x309 mem=2 {'doc_title': 1, 'text': 1} + R 14 DL bbox=[2045,1950,3265,3097] 1220x1147 mem=6 {'text': 3, 'image': 1, 'figure_title': 1, 'paragraph_title': 1} + R 65 H bbox=[113,1958,1975,2859] 1862x901 mem=8 {'text': 5, 'image': 1, 'doc_title': 1, 'figure_title': 1} + R 96 H bbox=[3326,1964,4087,3638] 761x1674 mem=4 {'doc_title': 1, 'text': 2, 'image': 1} + R 89 L bbox=[118,2887,595,4080] 477x1193 mem=5 {'text': 4, 'paragraph_title': 1} + R 86 H bbox=[682,2894,1977,4292] 1295x1398 mem=8 {'text': 4, 'image': 1, 'paragraph_title': 2, 'doc_title': 1} + R 64 H bbox=[2046,3116,3263,3907] 1217x791 mem=6 {'text': 3, 'image': 1, 'figure_title': 1, 'doc_title': 1} + R 97 H bbox=[3324,3756,4087,4627] 763x871 mem=2 {'doc_title': 1, 'text': 1} + R 84 H bbox=[2046,3920,3264,4820] 1218x900 mem=6 {'text': 3, 'image': 1, 'figure_title': 1, 'paragraph_title': 1} + R 44 F bbox=[123,4139,626,5113] 503x974 mem=4 {'text': 2, 'image': 1, 'paragraph_title': 1} + R112 H bbox=[682,4325,1975,5056] 1293x731 mem=5 {'text': 2, 'image': 1, 'figure_title': 1, 'doc_title': 1} + R 58 H bbox=[3322,4658,4087,5831] 765x1173 mem=2 {'doc_title': 1, 'text': 1} + R110 H bbox=[2044,4858,3265,5685] 1221x827 mem=7 {'doc_title': 1, 'text': 4, 'image': 1, 'paragraph_title': 1} + R 75 H RAW bbox=[680,5078,1975,5692] 1295x614 mem=5 {'doc_title': 1, 'image': 1, 'text': 2, 'figure_title': 1} + R 59 H RAW bbox=[680,5731,3267,6318] 2587x587 mem=7 {'doc_title': 2, 'text': 5} + R 7 DL bbox=[3308,5839,4087,6305] 779x466 mem=1 {'text': 1} SINGLE-region + [Containment-merge] x10 + [Banner rectangle] x8 + +DONE -> output/NamastheTelangana_Jangaon_20260520_20260609_130635/all_article_crops_review diff --git a/_jg_val5.log b/_jg_val5.log new file mode 100644 index 0000000..7b02133 --- /dev/null +++ b/_jg_val5.log @@ -0,0 +1,36 @@ + +=== page 1: 6 crops, 6 datelines === + R 3 DL bbox=[1497,137,4081,3023] 2584x2886 mem=8 {'text': 7, 'image': 1} + R 4 DL bbox=[105,145,1467,4295] 1362x4150 mem=15 {'text': 11, 'image': 2, 'paragraph_title': 2} + R 28 H bbox=[1498,838,3031,4921] 1533x4083 mem=18 {'doc_title': 1, 'text': 9, 'image': 3, 'paragraph_title': 2, 'figure_title': 3} + R 2 DL bbox=[3039,3957,4081,5748] 1042x1791 mem=3 {'text': 1, 'image': 1, 'figure_title': 1} + R 49 H RAW bbox=[106,4377,1429,6301] 1323x1924 mem=10 {'doc_title': 1, 'text': 5, 'image': 2, 'figure_title': 2} + R 29 H RAW bbox=[1500,5012,4081,6283] 2581x1271 mem=6 {'doc_title': 1, 'text': 3, 'image': 1, 'figure_title': 1} + [Containment-merge] x2 + [Headline-merge] x2 + [Banner rectangle] x4 + +=== page 2: 18 crops, 17 datelines === + R109 H bbox=[119,393,612,1899] 493x1506 mem=2 {'doc_title': 1, 'text': 1} + R 79 H bbox=[2693,405,4087,1833] 1394x1428 mem=12 {'doc_title': 1, 'text': 7, 'image': 2, 'figure_title': 1, 'paragraph_title': 1} + R 55 H RAW bbox=[679,415,2630,1919] 1951x1504 mem=14 {'doc_title': 1, 'text': 9, 'image': 1, 'paragraph_title': 2, 'figure_title': 1} + R 77 H bbox=[2692,1594,4087,1903] 1395x309 mem=2 {'doc_title': 1, 'text': 1} + R 14 DL bbox=[2045,1950,3265,3097] 1220x1147 mem=6 {'text': 3, 'image': 1, 'figure_title': 1, 'paragraph_title': 1} + R 65 H bbox=[113,1958,1975,2859] 1862x901 mem=8 {'text': 5, 'image': 1, 'doc_title': 1, 'figure_title': 1} + R 96 H bbox=[3326,1964,4087,3638] 761x1674 mem=4 {'doc_title': 1, 'text': 2, 'image': 1} + R 89 L bbox=[118,2887,595,4080] 477x1193 mem=5 {'text': 4, 'paragraph_title': 1} + R 86 H bbox=[682,2894,1977,4292] 1295x1398 mem=8 {'text': 4, 'image': 1, 'paragraph_title': 2, 'doc_title': 1} + R 64 H bbox=[2046,3116,3263,3907] 1217x791 mem=6 {'text': 3, 'image': 1, 'figure_title': 1, 'doc_title': 1} + R 97 H bbox=[3324,3756,4087,4627] 763x871 mem=2 {'doc_title': 1, 'text': 1} + R 84 H bbox=[2046,3920,3264,4820] 1218x900 mem=6 {'text': 3, 'image': 1, 'figure_title': 1, 'paragraph_title': 1} + R 44 F bbox=[123,4139,626,5113] 503x974 mem=4 {'text': 2, 'image': 1, 'paragraph_title': 1} + R112 H bbox=[682,4325,1975,5056] 1293x731 mem=5 {'text': 2, 'image': 1, 'figure_title': 1, 'doc_title': 1} + R 58 H bbox=[3322,4658,4087,5831] 765x1173 mem=2 {'doc_title': 1, 'text': 1} + R110 H bbox=[2044,4858,3265,5685] 1221x827 mem=7 {'doc_title': 1, 'text': 4, 'image': 1, 'paragraph_title': 1} + R 75 H RAW bbox=[680,5078,1975,5692] 1295x614 mem=5 {'doc_title': 1, 'image': 1, 'text': 2, 'figure_title': 1} + R 59 H RAW bbox=[680,5731,3267,6318] 2587x587 mem=7 {'doc_title': 2, 'text': 5} + R 7 DL bbox=[3308,5839,4087,6305] 779x466 mem=1 {'text': 1} SINGLE-region + [Containment-merge] x10 + [Banner rectangle] x8 + +DONE -> output/NamastheTelangana_Jangaon_20260520_20260609_130635/all_article_crops_review diff --git a/_jg_val6.log b/_jg_val6.log new file mode 100644 index 0000000..7b02133 --- /dev/null +++ b/_jg_val6.log @@ -0,0 +1,36 @@ + +=== page 1: 6 crops, 6 datelines === + R 3 DL bbox=[1497,137,4081,3023] 2584x2886 mem=8 {'text': 7, 'image': 1} + R 4 DL bbox=[105,145,1467,4295] 1362x4150 mem=15 {'text': 11, 'image': 2, 'paragraph_title': 2} + R 28 H bbox=[1498,838,3031,4921] 1533x4083 mem=18 {'doc_title': 1, 'text': 9, 'image': 3, 'paragraph_title': 2, 'figure_title': 3} + R 2 DL bbox=[3039,3957,4081,5748] 1042x1791 mem=3 {'text': 1, 'image': 1, 'figure_title': 1} + R 49 H RAW bbox=[106,4377,1429,6301] 1323x1924 mem=10 {'doc_title': 1, 'text': 5, 'image': 2, 'figure_title': 2} + R 29 H RAW bbox=[1500,5012,4081,6283] 2581x1271 mem=6 {'doc_title': 1, 'text': 3, 'image': 1, 'figure_title': 1} + [Containment-merge] x2 + [Headline-merge] x2 + [Banner rectangle] x4 + +=== page 2: 18 crops, 17 datelines === + R109 H bbox=[119,393,612,1899] 493x1506 mem=2 {'doc_title': 1, 'text': 1} + R 79 H bbox=[2693,405,4087,1833] 1394x1428 mem=12 {'doc_title': 1, 'text': 7, 'image': 2, 'figure_title': 1, 'paragraph_title': 1} + R 55 H RAW bbox=[679,415,2630,1919] 1951x1504 mem=14 {'doc_title': 1, 'text': 9, 'image': 1, 'paragraph_title': 2, 'figure_title': 1} + R 77 H bbox=[2692,1594,4087,1903] 1395x309 mem=2 {'doc_title': 1, 'text': 1} + R 14 DL bbox=[2045,1950,3265,3097] 1220x1147 mem=6 {'text': 3, 'image': 1, 'figure_title': 1, 'paragraph_title': 1} + R 65 H bbox=[113,1958,1975,2859] 1862x901 mem=8 {'text': 5, 'image': 1, 'doc_title': 1, 'figure_title': 1} + R 96 H bbox=[3326,1964,4087,3638] 761x1674 mem=4 {'doc_title': 1, 'text': 2, 'image': 1} + R 89 L bbox=[118,2887,595,4080] 477x1193 mem=5 {'text': 4, 'paragraph_title': 1} + R 86 H bbox=[682,2894,1977,4292] 1295x1398 mem=8 {'text': 4, 'image': 1, 'paragraph_title': 2, 'doc_title': 1} + R 64 H bbox=[2046,3116,3263,3907] 1217x791 mem=6 {'text': 3, 'image': 1, 'figure_title': 1, 'doc_title': 1} + R 97 H bbox=[3324,3756,4087,4627] 763x871 mem=2 {'doc_title': 1, 'text': 1} + R 84 H bbox=[2046,3920,3264,4820] 1218x900 mem=6 {'text': 3, 'image': 1, 'figure_title': 1, 'paragraph_title': 1} + R 44 F bbox=[123,4139,626,5113] 503x974 mem=4 {'text': 2, 'image': 1, 'paragraph_title': 1} + R112 H bbox=[682,4325,1975,5056] 1293x731 mem=5 {'text': 2, 'image': 1, 'figure_title': 1, 'doc_title': 1} + R 58 H bbox=[3322,4658,4087,5831] 765x1173 mem=2 {'doc_title': 1, 'text': 1} + R110 H bbox=[2044,4858,3265,5685] 1221x827 mem=7 {'doc_title': 1, 'text': 4, 'image': 1, 'paragraph_title': 1} + R 75 H RAW bbox=[680,5078,1975,5692] 1295x614 mem=5 {'doc_title': 1, 'image': 1, 'text': 2, 'figure_title': 1} + R 59 H RAW bbox=[680,5731,3267,6318] 2587x587 mem=7 {'doc_title': 2, 'text': 5} + R 7 DL bbox=[3308,5839,4087,6305] 779x466 mem=1 {'text': 1} SINGLE-region + [Containment-merge] x10 + [Banner rectangle] x8 + +DONE -> output/NamastheTelangana_Jangaon_20260520_20260609_130635/all_article_crops_review diff --git a/_jg_val7.log b/_jg_val7.log new file mode 100644 index 0000000..4f92a23 --- /dev/null +++ b/_jg_val7.log @@ -0,0 +1,36 @@ + +=== page 1: 6 crops, 6 datelines === + R 3 DL bbox=[1497,137,4081,3023] 2584x2886 mem=8 {'text': 7, 'image': 1} + R 4 DL bbox=[105,145,1467,4295] 1362x4150 mem=15 {'text': 11, 'image': 2, 'paragraph_title': 2} + R 28 H bbox=[1498,838,3031,4921] 1533x4083 mem=18 {'doc_title': 1, 'text': 9, 'image': 3, 'paragraph_title': 2, 'figure_title': 3} + R 2 DL bbox=[3039,3957,4081,5748] 1042x1791 mem=3 {'text': 1, 'image': 1, 'figure_title': 1} + R 49 H RAW bbox=[106,4377,1429,6301] 1323x1924 mem=10 {'doc_title': 1, 'text': 5, 'image': 2, 'figure_title': 2} + R 29 H RAW bbox=[1500,5012,4081,6283] 2581x1271 mem=6 {'doc_title': 1, 'text': 3, 'image': 1, 'figure_title': 1} + [Containment-merge] x2 + [Headline-merge] x2 + [Banner rectangle] x4 + +=== page 2: 18 crops, 17 datelines === + R109 H bbox=[119,393,612,1899] 493x1506 mem=2 {'doc_title': 1, 'text': 1} + R 79 H bbox=[2693,405,4087,1833] 1394x1428 mem=12 {'doc_title': 1, 'text': 7, 'image': 2, 'figure_title': 1, 'paragraph_title': 1} + R 55 H RAW bbox=[679,415,2630,1919] 1951x1504 mem=14 {'doc_title': 1, 'text': 9, 'image': 1, 'paragraph_title': 2, 'figure_title': 1} + R 77 H bbox=[2692,1594,4087,1903] 1395x309 mem=2 {'doc_title': 1, 'text': 1} + R 14 DL bbox=[2045,1950,3265,3097] 1220x1147 mem=6 {'text': 3, 'image': 1, 'figure_title': 1, 'paragraph_title': 1} + R 65 H bbox=[113,1958,2023,2859] 1910x901 mem=8 {'text': 5, 'image': 1, 'doc_title': 1, 'figure_title': 1} + R 96 H bbox=[3326,1964,4087,3638] 761x1674 mem=4 {'doc_title': 1, 'text': 2, 'image': 1} + R 89 L bbox=[118,2887,595,4080] 477x1193 mem=5 {'text': 4, 'paragraph_title': 1} + R 86 H bbox=[682,2894,1977,4292] 1295x1398 mem=8 {'text': 4, 'image': 1, 'paragraph_title': 2, 'doc_title': 1} + R 64 H bbox=[2046,3116,3263,3907] 1217x791 mem=6 {'text': 3, 'image': 1, 'figure_title': 1, 'doc_title': 1} + R 97 H bbox=[3324,3756,4087,4627] 763x871 mem=2 {'doc_title': 1, 'text': 1} + R 84 H bbox=[2046,3920,3264,4820] 1218x900 mem=6 {'text': 3, 'image': 1, 'figure_title': 1, 'paragraph_title': 1} + R 44 F bbox=[123,4139,626,5113] 503x974 mem=4 {'text': 2, 'image': 1, 'paragraph_title': 1} + R112 H bbox=[682,4325,1975,5056] 1293x731 mem=5 {'text': 2, 'image': 1, 'figure_title': 1, 'doc_title': 1} + R 58 H bbox=[3322,4658,4087,5831] 765x1173 mem=2 {'doc_title': 1, 'text': 1} + R110 H bbox=[2044,4858,3265,5685] 1221x827 mem=7 {'doc_title': 1, 'text': 4, 'image': 1, 'paragraph_title': 1} + R 75 H RAW bbox=[680,5078,1975,5692] 1295x614 mem=5 {'doc_title': 1, 'image': 1, 'text': 2, 'figure_title': 1} + R 59 H RAW bbox=[113,5731,3267,6318] 3154x587 mem=7 {'doc_title': 2, 'text': 5} + R 7 DL bbox=[3308,5839,4087,6305] 779x466 mem=1 {'text': 1} SINGLE-region + [Containment-merge] x10 + [Banner rectangle] x8 + +DONE -> output/NamastheTelangana_Jangaon_20260520_20260609_130635/all_article_crops_review diff --git a/_jg_val8.log b/_jg_val8.log new file mode 100644 index 0000000..6baa4dc --- /dev/null +++ b/_jg_val8.log @@ -0,0 +1,36 @@ + +=== page 1: 6 crops, 6 datelines === + R 3 DL bbox=[1497,137,4081,3023] 2584x2886 mem=13 {'text': 8, 'image': 3, 'figure_title': 2} + R 4 DL bbox=[105,145,1467,4295] 1362x4150 mem=15 {'text': 11, 'image': 2, 'paragraph_title': 2} + R 28 H bbox=[1498,3089,3031,4921] 1533x1832 mem=13 {'doc_title': 1, 'text': 8, 'image': 1, 'paragraph_title': 2, 'figure_title': 1} + R 2 DL bbox=[3092,3957,4079,4960] 987x1003 mem=1 {'text': 1} SINGLE-region + R 49 H RAW bbox=[106,4377,1429,6301] 1323x1924 mem=10 {'doc_title': 1, 'text': 5, 'image': 2, 'figure_title': 2} + R 29 H RAW bbox=[1500,5012,4081,6283] 2581x1271 mem=8 {'text': 3, 'image': 2, 'doc_title': 1, 'figure_title': 2} + [Containment-merge] x2 + [Headline-merge] x2 + [Banner rectangle] x4 + +=== page 2: 18 crops, 17 datelines === + R109 H bbox=[119,393,612,1899] 493x1506 mem=2 {'doc_title': 1, 'text': 1} + R 79 H bbox=[2693,405,4078,1567] 1385x1162 mem=11 {'doc_title': 1, 'text': 7, 'image': 1, 'figure_title': 1, 'paragraph_title': 1} + R 55 H RAW bbox=[679,415,2630,1919] 1951x1504 mem=14 {'doc_title': 1, 'text': 9, 'image': 1, 'paragraph_title': 2, 'figure_title': 1} + R 77 H bbox=[2692,1594,4087,1903] 1395x309 mem=3 {'text': 1, 'image': 1, 'doc_title': 1} + R 14 DL bbox=[2045,1950,3265,3097] 1220x1147 mem=6 {'text': 3, 'image': 1, 'figure_title': 1, 'paragraph_title': 1} + R 65 H bbox=[113,1958,1975,2859] 1862x901 mem=8 {'text': 5, 'image': 1, 'doc_title': 1, 'figure_title': 1} + R 96 H bbox=[3326,1964,4087,3638] 761x1674 mem=4 {'doc_title': 1, 'text': 2, 'image': 1} + R 89 L bbox=[118,2887,595,4080] 477x1193 mem=5 {'text': 4, 'paragraph_title': 1} + R 86 H bbox=[682,2894,1977,4292] 1295x1398 mem=8 {'text': 4, 'image': 1, 'paragraph_title': 2, 'doc_title': 1} + R 64 H bbox=[2046,3116,3263,3907] 1217x791 mem=6 {'text': 3, 'image': 1, 'figure_title': 1, 'doc_title': 1} + R 97 H bbox=[3324,3756,4087,4627] 763x871 mem=2 {'doc_title': 1, 'text': 1} + R 84 H bbox=[2046,3920,3264,4820] 1218x900 mem=6 {'text': 3, 'image': 1, 'figure_title': 1, 'paragraph_title': 1} + R 44 F bbox=[123,4139,626,5113] 503x974 mem=4 {'text': 2, 'image': 1, 'paragraph_title': 1} + R112 H bbox=[682,4325,1975,5056] 1293x731 mem=5 {'text': 2, 'image': 1, 'figure_title': 1, 'doc_title': 1} + R 58 H bbox=[3322,4658,4087,5831] 765x1173 mem=2 {'doc_title': 1, 'text': 1} + R110 H bbox=[2044,4858,3265,5685] 1221x827 mem=7 {'doc_title': 1, 'text': 4, 'image': 1, 'paragraph_title': 1} + R 75 H RAW bbox=[680,5078,1975,5692] 1295x614 mem=5 {'doc_title': 1, 'image': 1, 'text': 2, 'figure_title': 1} + R 59 H RAW bbox=[680,5731,3267,6318] 2587x587 mem=7 {'doc_title': 2, 'text': 5} + R 7 DL bbox=[3308,5839,4087,6305] 779x466 mem=1 {'text': 1} SINGLE-region + [Containment-merge] x10 + [Banner rectangle] x8 + +DONE -> output/NamastheTelangana_Jangaon_20260520_20260609_130635/all_article_crops_review diff --git a/_jg_val8b.log b/_jg_val8b.log new file mode 100644 index 0000000..d04f320 --- /dev/null +++ b/_jg_val8b.log @@ -0,0 +1,36 @@ + +=== page 1: 6 crops, 6 datelines === + R 3 DL bbox=[1497,137,4081,3023] 2584x2886 mem=13 {'text': 8, 'image': 3, 'figure_title': 2} + R 4 DL bbox=[105,145,1467,4295] 1362x4150 mem=15 {'text': 11, 'image': 2, 'paragraph_title': 2} + R 28 H bbox=[1498,3089,3031,4921] 1533x1832 mem=13 {'doc_title': 1, 'text': 8, 'image': 1, 'paragraph_title': 2, 'figure_title': 1} + R 2 DL bbox=[3092,3957,4079,4960] 987x1003 mem=1 {'text': 1} SINGLE-region + R 49 H RAW bbox=[106,4377,1429,6301] 1323x1924 mem=10 {'doc_title': 1, 'text': 5, 'image': 2, 'figure_title': 2} + R 29 H RAW bbox=[1500,5012,4081,6283] 2581x1271 mem=8 {'text': 3, 'image': 2, 'doc_title': 1, 'figure_title': 2} + [Containment-merge] x2 + [Headline-merge] x2 + [Banner rectangle] x4 + +=== page 2: 18 crops, 17 datelines === + R109 H bbox=[119,393,612,1899] 493x1506 mem=2 {'doc_title': 1, 'text': 1} + R 79 H bbox=[2693,405,4078,1567] 1385x1162 mem=11 {'doc_title': 1, 'text': 7, 'image': 1, 'figure_title': 1, 'paragraph_title': 1} + R 55 H RAW bbox=[679,415,2630,1919] 1951x1504 mem=14 {'doc_title': 1, 'text': 9, 'image': 1, 'paragraph_title': 2, 'figure_title': 1} + R 77 H bbox=[2692,1594,4087,1903] 1395x309 mem=3 {'text': 1, 'image': 1, 'doc_title': 1} + R 14 DL bbox=[2045,1950,3265,3097] 1220x1147 mem=6 {'text': 3, 'image': 1, 'figure_title': 1, 'paragraph_title': 1} + R 65 H bbox=[113,1958,1975,2859] 1862x901 mem=8 {'text': 5, 'image': 1, 'doc_title': 1, 'figure_title': 1} + R 96 H bbox=[3326,1964,4087,3638] 761x1674 mem=4 {'doc_title': 1, 'text': 2, 'image': 1} + R 89 L bbox=[118,2887,595,4080] 477x1193 mem=5 {'text': 4, 'paragraph_title': 1} + R 86 H bbox=[682,2894,1977,4292] 1295x1398 mem=8 {'text': 4, 'image': 1, 'paragraph_title': 2, 'doc_title': 1} + R 64 H bbox=[2046,3116,3263,3907] 1217x791 mem=6 {'text': 3, 'image': 1, 'figure_title': 1, 'doc_title': 1} + R 97 H bbox=[3324,3756,4087,4627] 763x871 mem=2 {'doc_title': 1, 'text': 1} + R 84 H bbox=[2046,3920,3264,4820] 1218x900 mem=6 {'text': 3, 'image': 1, 'figure_title': 1, 'paragraph_title': 1} + R 44 F bbox=[123,4139,626,5113] 503x974 mem=4 {'text': 2, 'image': 1, 'paragraph_title': 1} + R112 H bbox=[682,4325,1975,5056] 1293x731 mem=5 {'text': 2, 'image': 1, 'figure_title': 1, 'doc_title': 1} + R 58 H bbox=[3322,4658,4087,5831] 765x1173 mem=2 {'doc_title': 1, 'text': 1} + R110 H bbox=[2044,4858,3265,5685] 1221x827 mem=7 {'doc_title': 1, 'text': 4, 'image': 1, 'paragraph_title': 1} + R 75 H RAW bbox=[680,5078,1975,5692] 1295x614 mem=5 {'doc_title': 1, 'image': 1, 'text': 2, 'figure_title': 1} + R 59 H RAW bbox=[680,5731,3267,6318] 2587x587 mem=7 {'doc_title': 2, 'text': 5} + R 7 DL bbox=[3308,5839,4087,6305] 779x466 mem=1 {'text': 1} SINGLE-region + [Containment-merge] x10 + [Banner rectangle] x8 + +DONE -> output/NamastheTelangana_Jangaon_20260520_20260608_124720/all_article_crops_review diff --git a/_lc3.txt b/_lc3.txt new file mode 100644 index 0000000..9544705 --- /dev/null +++ b/_lc3.txt @@ -0,0 +1,10 @@ +AJ pg1: 12 blocks (baseline AJ 12/18/14/20 NT 11/22/19/25 JG 7/12) +AJ pg2: 18 blocks (baseline AJ 12/18/14/20 NT 11/22/19/25 JG 7/12) +AJ pg3: 14 blocks (baseline AJ 12/18/14/20 NT 11/22/19/25 JG 7/12) +AJ pg4: 20 blocks (baseline AJ 12/18/14/20 NT 11/22/19/25 JG 7/12) +NT pg1: 11 blocks (baseline AJ 12/18/14/20 NT 11/22/19/25 JG 7/12) +NT pg2: 22 blocks (baseline AJ 12/18/14/20 NT 11/22/19/25 JG 7/12) +NT pg3: 19 blocks (baseline AJ 12/18/14/20 NT 11/22/19/25 JG 7/12) +NT pg4: 24 blocks (baseline AJ 12/18/14/20 NT 11/22/19/25 JG 7/12) +JG pg1: 7 blocks (baseline AJ 12/18/14/20 NT 11/22/19/25 JG 7/12) +JG pg2: 12 blocks (baseline AJ 12/18/14/20 NT 11/22/19/25 JG 7/12) diff --git a/_lencheck.txt b/_lencheck.txt new file mode 100644 index 0000000..fe769c9 --- /dev/null +++ b/_lencheck.txt @@ -0,0 +1,10 @@ +AJ pg1 pitch755 hbars10 base12 | thr0.6: 12 (drop6) | thr0.8: 12 (drop7) | thr1.0: 12 (drop7) +AJ pg2 pitch719 hbars19 base18 | thr0.6: 18 (drop5) | thr0.8: 18 (drop8) | thr1.0: 18 (drop9) +AJ pg3 pitch728 hbars15 base14 | thr0.6: 14 (drop7) | thr0.8: 13 (drop9) | thr1.0: 13 (drop9) +AJ pg4 pitch728 hbars25 base20 | thr0.6: 20 (drop11) | thr0.8: 20 (drop13) | thr1.0: 20 (drop13) +NT pg1 pitch536 hbars24 base11 | thr0.6: 11 (drop5) | thr0.8: 11 (drop8) | thr1.0: 11 (drop9) +NT pg2 pitch527 hbars47 base22 | thr0.6: 22 (drop7) | thr0.8: 22 (drop10) | thr1.0: 22 (drop15) +NT pg3 pitch525 hbars25 base19 | thr0.6: 19 (drop3) | thr0.8: 19 (drop4) | thr1.0: 19 (drop7) +NT pg4 pitch523 hbars43 base25 | thr0.6: 25 (drop5) | thr0.8: 25 (drop8) | thr1.0: 21 (drop25) +JG pg1 pitch601 hbars15 base7 | thr0.6: 7 (drop5) | thr0.8: 7 (drop6) | thr1.0: 7 (drop9) +JG pg2 pitch594 hbars17 base12 | thr0.6: 12 (drop0) | thr0.8: 12 (drop0) | thr1.0: 12 (drop9) diff --git a/_lencheck2.txt b/_lencheck2.txt new file mode 100644 index 0000000..e69de29 diff --git a/_lines.py b/_lines.py new file mode 100644 index 0000000..ea4ad7d --- /dev/null +++ b/_lines.py @@ -0,0 +1,416 @@ +"""Separator-rule detection for newspaper pages (classical CV, no ML / no API). + +Detects the thin printed grey/black straight lines the paper uses to divide +stories, with a TEXT-ISOLATION filter to reject false positives from dense +Telugu text (a real rule sits in a whitespace gutter; a text stroke has ink +immediately above/below it). + +Public: + detect_separator_lines(png_path) -> {"h": [...], "v": [...], "size": (W,H)} + each line is a dict: {x1,y1,x2,y2,len,gray} (gray = mean intensity). +""" +import cv2 +import numpy as np + + +def _candidates(bw, horizontal, min_len, max_thick=55): + if horizontal: + k = cv2.getStructuringElement(cv2.MORPH_RECT, (min_len // 2, 1)) + else: + k = cv2.getStructuringElement(cv2.MORPH_RECT, (1, min_len // 2)) + opened = cv2.morphologyEx(bw, cv2.MORPH_OPEN, k, iterations=1) + # bridge small collinear gaps along the line direction + if horizontal: + opened = cv2.dilate(opened, cv2.getStructuringElement(cv2.MORPH_RECT, (25, 3))) + else: + opened = cv2.dilate(opened, cv2.getStructuringElement(cv2.MORPH_RECT, (3, 25))) + cnts, _ = cv2.findContours(opened, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE) + out = [] + for c in cnts: + x, y, w, h = cv2.boundingRect(c) + if horizontal and w >= min_len and h <= max_thick: + out.append((x, y + h // 2, x + w, y + h // 2, w)) + if (not horizontal) and h >= min_len and w <= max_thick: + out.append((x + w // 2, y, x + w // 2, y + h, h)) + return out + + +def _isolated(gray, line, horizontal, ink_thresh=110, gap=(5, 16), + max_adjacent_ink=0.12): + """A true separator sits in a whitespace gutter: the tight band on BOTH sides + (parallel to the line) is mostly paper. Returns (is_isolated, mean_gray).""" + H, W = gray.shape + x1, y1, x2, y2, ln = line + if horizontal: + seg = gray[y1, x1:x2] + lo, hi = gap + above = gray[max(0, y1 - hi):max(0, y1 - lo), x1:x2] + below = gray[min(H, y1 + lo):min(H, y1 + hi), x1:x2] + else: + seg = gray[y1:y2, x1] + lo, hi = gap + above = gray[y1:y2, max(0, x1 - hi):max(0, x1 - lo)] + below = gray[y1:y2, min(W, x1 + lo):min(W, x1 + hi)] + if seg.size == 0 or above.size == 0 or below.size == 0: + return False, 255 + ink_a = float((above < ink_thresh).mean()) + ink_b = float((below < ink_thresh).mean()) + isolated = (ink_a <= max_adjacent_ink) and (ink_b <= max_adjacent_ink) + return isolated, float(seg.mean()) + + +def detect_separator_lines(png_path, h_frac=0.045, v_frac=0.06): + img = cv2.imread(str(png_path)) + gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY) + H, W = gray.shape + blur = cv2.GaussianBlur(gray, (3, 3), 0) + bw = cv2.threshold(blur, 0, 255, cv2.THRESH_BINARY_INV + cv2.THRESH_OTSU)[1] + + h_min = int(h_frac * W) + v_min = int(v_frac * H) + h_out, v_out = [], [] + for ln in _candidates(bw, True, h_min): + iso, g = _isolated(gray, ln, True) + if iso: + h_out.append({"x1": ln[0], "y1": ln[1], "x2": ln[2], "y2": ln[3], + "len": ln[4], "gray": round(g, 1)}) + for ln in _candidates(bw, False, v_min): + iso, g = _isolated(gray, ln, False) + if iso: + v_out.append({"x1": ln[0], "y1": ln[1], "x2": ln[2], "y2": ln[3], + "len": ln[4], "gray": round(g, 1)}) + return {"h": h_out, "v": v_out, "size": (W, H)} + + +def detect_content_bottom(png_path, search_frac=0.12, color_thr=0.06, + blank_color=0.02, blank_ink=0.035, peak_thr=0.15, + max_band=160, min_gap=25, min_margin=18): + """Find the y below which a page holds only the printer's COLOUR-CALIBRATION + strip (the row of CMYK registration dots + grey bars) and the bottom paper + margin — i.e. the bottom of real article content. + + The calibration strip is a thin band of strongly-saturated colour dots near + the page bottom, isolated by a white paper margin BELOW it (down to the page + edge) and a white gap ABOVE it (separating it from article content). Crops + should be clamped to the returned y so the dots never bleed into an article. + + Returns the y at the TOP of the calibration band, or the full page height H + when no such strip is found (nothing to clamp).""" + img = cv2.imread(str(png_path)) + if img is None: + return None + H, W = img.shape[:2] + b, g, r = img[..., 0].astype(np.int16), img[..., 1].astype(np.int16), img[..., 2].astype(np.int16) + mx = np.maximum(np.maximum(r, g), b) + mn = np.minimum(np.minimum(r, g), b) + sat = mx - mn + colored = (sat > 55) & (mx > 80) # saturated colour (registration dots) + dark = mx < 110 # ink / dark + col_frac = colored.mean(axis=1) # per-row colour fraction + ink_frac = dark.mean(axis=1) # per-row ink fraction + blank = (col_frac < blank_color) & (ink_frac < blank_ink) + + y0 = int(H * (1 - search_frac)) + # The calibration strip is a band of strongly-coloured rows; identify it as the + # bottom-most run of rows whose colour fraction clears `color_thr`. + strip_rows = [y for y in range(y0, H) if col_frac[y] >= color_thr] + if not strip_rows: + return H # no coloured strip near the bottom + band_bottom = strip_rows[-1] + # A real calibration strip is followed by white paper margin to the page edge. + below = blank[band_bottom + 1:H] + if below.size < min_margin or float(below.mean()) < 0.9: + return H + # Walk up from band_bottom while still in the coloured band (tolerate the small + # white gaps between dot rows). + band_top = band_bottom + gap = 0 + y = band_bottom - 1 + while y > y0: + if col_frac[y] >= color_thr: + band_top = y + gap = 0 + else: + gap += 1 + if gap > 12: + break + y -= 1 + if (band_bottom - band_top + 1) > max_band: + return H # too tall -> fused with a photo, bail + # A genuine printer calibration strip is a dense row of saturated CMYK + # registration dots: its peak per-row colour fraction is high (~0.22-0.45 across + # Andhra Jyothi / Sakshi). A single-hue coloured ad/notice box embedded in + # article text peaks far lower (~0.11) -> reject it as a false positive. + peak = float(col_frac[band_top:band_bottom + 1].max()) + if peak < peak_thr: + return H + return band_top + + +def _hspan_overlap(ax1, ax2, bx1, bx2): + ix = max(0, min(ax2, bx2) - max(ax1, bx1)) + return ix / max(1, min(ax2 - ax1, bx2 - bx1)) + + +def _faint_rule_above(gray, hbbox, max_gap=170, min_gap=6, gray_lo=140, gray_hi=215, + cover=0.6, max_dark=0.06): + """Find a faint GREY horizontal rule sitting in the whitespace directly above a + (multi-column) header. Such a light rule reads as background to the ink-based + Otsu detector in `detect_separator_lines`, so we look for a thin row that is + mostly uniform mid-grey across the header's width and is isolated by white paper + above and below. Returns the rule's y, or None. (Used only above multi-column + headers, where it marks the article's TOP boundary.)""" + H, W = gray.shape + hx1, hy, hx2 = hbbox[0], hbbox[1], hbbox[2] + y_hi = max(0, hy - min_gap) + y_lo = max(0, hy - max_gap) + for y in range(y_hi, y_lo, -1): + row = gray[y, hx1:hx2].astype(np.int16) + if row.size == 0: + continue + mid = float(((row >= gray_lo) & (row <= gray_hi)).mean()) + dark = float((row < gray_lo).mean()) + if mid >= cover and dark <= max_dark: + above = gray[max(0, y - 6):max(0, y - 2), hx1:hx2] + below = gray[min(H, y + 2):min(H, y + 6), hx1:hx2] + if (above.size and below.size + and float((above > gray_hi).mean()) > 0.85 + and float((below > gray_hi).mean()) > 0.85): + return y + return None + + +def _faint_grey_band_above(gray, hbbox, max_gap=185, min_gap=6, max_thick=12, + gray_lo=135, gray_hi=232, cover=0.8, max_dark=0.08, + white_thr=235, white_cover=0.9): + """Find a faint GREY horizontal rule (1-`max_thick` px thick) sitting in the + whitespace directly above a SINGLE-column header. Per the layout convention a + grey rule whose length matches the article/header width marks the boundary + between two stacked stories, so the next story's headline must not fold into + the article above the rule. Such light rules read as background to the ink-based + Otsu detector in `detect_separator_lines`, and `_faint_rule_above` only accepts + a 1-px isolated rule, so a thicker grey band is missed; here we group the band + and require it to span the header's width and be isolated by white paper above + and below. Returns the band's centre y, or None.""" + H, W = gray.shape + hx1, hy, hx2 = int(hbbox[0]), int(hbbox[1]), int(hbbox[2]) + y_hi = max(0, hy - min_gap) + y_lo = max(0, hy - max_gap) + cols = slice(hx1, hx2) + grey_rows = [] + for y in range(y_lo, y_hi): + row = gray[y, cols].astype(np.int16) + if row.size == 0: + continue + mid = float(((row >= gray_lo) & (row <= gray_hi)).mean()) + dark = float((row < gray_lo).mean()) + if mid >= cover and dark <= max_dark: + grey_rows.append(y) + if not grey_rows: + return None + bands = [] + s = p = grey_rows[0] + for y in grey_rows[1:]: + if y == p + 1: + p = y + else: + bands.append((s, p)); s = p = y + bands.append((s, p)) + for top, bot in sorted(bands, key=lambda b: -b[1]): # nearest the header first + if bot - top + 1 > max_thick: + continue + above = gray[max(0, top - 4):top, cols] + below = gray[bot + 1:min(H, bot + 5), cols] + if (above.size and below.size + and float((above > white_thr).mean()) >= white_cover + and float((below > white_thr).mean()) >= white_cover): + return (top + bot) // 2 + return None + + +def _faint_grey_band_below(gray, bbox, max_gap=150, min_gap=4, max_thick=12, + gray_lo=135, gray_hi=232, cover=0.8, max_dark=0.08, + white_thr=235, white_cover=0.9): + """Mirror of `_faint_grey_band_above`, searching DOWNWARD: find the faint grey + rule that sits in the whitespace just below an article (within `max_gap` px of + its bottom edge) and spans its column width. That rule is the article's BOTTOM + boundary, so the crop can be cut precisely at it. Returns the band's top y (the + cut line), or None when no clean full-width grey rule is found below.""" + H, W = gray.shape + bx1, by, bx2 = int(bbox[0]), int(bbox[3]), int(bbox[2]) + y_lo = min(H, by + min_gap) + y_hi = min(H, by + max_gap) + cols = slice(bx1, bx2) + grey_rows = [] + for y in range(y_lo, y_hi): + row = gray[y, cols].astype(np.int16) + if row.size == 0: + continue + mid = float(((row >= gray_lo) & (row <= gray_hi)).mean()) + dark = float((row < gray_lo).mean()) + if mid >= cover and dark <= max_dark: + grey_rows.append(y) + if not grey_rows: + return None + bands = [] + s = p = grey_rows[0] + for y in grey_rows[1:]: + if y == p + 1: + p = y + else: + bands.append((s, p)); s = p = y + bands.append((s, p)) + for top, bot in sorted(bands, key=lambda b: b[0]): # nearest below first + if bot - top + 1 > max_thick: + continue + # Skip the 1-px anti-aliased fade row hugging the rule before checking for + # the white paper gutter above and below it. + above = gray[max(0, top - 7):max(0, top - 2), cols] + below = gray[bot + 2:min(H, bot + 7), cols] + if (above.size and below.size + and float((above > white_thr).mean()) >= white_cover + and float((below > white_thr).mean()) >= white_cover): + return top + return None + + +def separator_barriers(png_path, regions, near=80, min_overlap=0.3, + title_types=("doc_title", "paragraph_title", "figure_title"), + multicol_frac=0.22): + """Article-boundary rules: the horizontal printed lines that mark the END of a + story. Per the layout convention, a rule with NORMAL text (or whitespace) above + it separates two stacked articles, while a rule sitting right under a BOLD + headline/title is a decorative flourish and must NOT split anything. + + We approximate 'bold above' by a title-type region (doc_title / paragraph_title + / figure_title) whose bottom edge is within `near` px above the rule and that + horizontally overlaps it. Such rules are dropped; the rest are returned as + barriers {y, x1, x2} for the clusterer to forbid cross-line folding. + + Additionally, for every MULTI-COLUMN header (a doc_title at least `multicol_frac` + of the page width), we look for a faint grey rule directly above it and add it as + a barrier spanning the header's width: that rule is the article's TOP boundary, so + nothing above it folds into the multi-column article below.""" + res = detect_separator_lines(png_path) + + # A real article separator sits in a WHITESPACE GUTTER between two stacked + # boxes; it never lands deep inside a body-text region. PaddleOCR's morphology + # occasionally mistakes a dense Telugu text stroke for a thin rule, producing a + # phantom line straight through a paragraph. Reject any detected rule whose + # midpoint falls well inside (margin-shrunk) a 'text' box it column-overlaps. + _text_boxes = [r["bbox"] for r in regions if r.get("type") == "text"] + + def _in_text(y, lx1, lx2, margin=28): + mx = (lx1 + lx2) / 2 + for tx1, ty1, tx2, ty2 in _text_boxes: + if (tx1 <= mx <= tx2 + and ty1 + margin <= y <= ty2 - margin + and _hspan_overlap(lx1, lx2, tx1, tx2) >= min_overlap): + return True + return False + + # Column width (median text-region width). A horizontal line counts as an article + # separator only when it spans most of a column. NOTE: a strict ≥1.0× column floor is + # too aggressive — a SINGLE-column article's own boundary rule is itself <1 column wide, + # so ≥1.0× drops real separators and merges articles (measured: NT pg4 R121 → 2 datelines). + # 0.6× column is the safe floor: it removes only the short caption / decorative underlines + # and merges NO articles on AJ / NT / JG. + _twid = sorted((r["bbox"][2] - r["bbox"][0]) for r in regions if r.get("type") == "text") + _pitch = _twid[len(_twid) // 2] if _twid else 0 + _len_floor = 0.6 * _pitch + + bars = [] + for L in res["h"]: + y, lx1, lx2 = L["y1"], L["x1"], L["x2"] + if _in_text(y, lx1, lx2): + continue + if _pitch and (lx2 - lx1) < _len_floor: + continue # too short to be an article separator (caption/underline) + bold_above = False + title_w_above = 0 + for r in regions: + b = r["bbox"] + if (b[3] <= y and (y - b[3]) <= near + and r.get("type") in title_types + and _hspan_overlap(lx1, lx2, b[0], b[2]) >= min_overlap): + bold_above = True + title_w_above = max(title_w_above, b[2] - b[0]) + # A rule under a bold title is a decorative flourish ONLY when it merely + # underlines that title (rule width ≈ title width). A rule spanning much + # WIDER than the title above it runs across several columns — e.g. a photo + # caption (figure_title) sits above ONE column while the rule runs the FULL + # article width — so it is a genuine article boundary and must be kept. + if (not bold_above) or ((lx2 - lx1) > 1.5 * title_w_above): + bars.append({"y": y, "x1": lx1, "x2": lx2}) + + gray = cv2.cvtColor(cv2.imread(str(png_path)), cv2.COLOR_BGR2GRAY) + W = gray.shape[1] + _img_boxes = [r["bbox"] for r in regions if r.get("type") == "image"] + + def _lead_photo_above(yb, hx1, hx2, gap=80): + """A faint rule directly under the article's OWN lead photo is that photo's + caption underline, not the article's top boundary. If an image sits flush + (within `gap`px) above the rule and overlaps the header's width, the rule must + NOT bound the article — otherwise the lead photo gets cut off.""" + for ix1, iy1, ix2, iy2 in _img_boxes: + if (0 <= yb - iy2 <= gap + and _hspan_overlap(hx1, hx2, ix1, ix2) >= min_overlap): + return True + return False + + for r in regions: + if r.get("type") != "doc_title": + continue + b = r["bbox"] + if (b[2] - b[0]) < multicol_frac * W: + continue # single-column header → skip + yb = _faint_rule_above(gray, b) + if yb is not None and not _lead_photo_above(yb, b[0], b[2]): + bars.append({"y": yb, "x1": b[0], "x2": b[2], + "mc": True, "anchor": r["id"]}) + + # Same rule for SINGLE-column headers: a faint grey rule spanning the header's + # width directly above it bounds the article ABOVE it, so the header (and its + # story) cannot fold upward across the rule. Plain barrier (no mc clamp). + for r in regions: + if r.get("type") not in ("doc_title", "paragraph_title"): + continue + b = r["bbox"] + if (b[2] - b[0]) >= multicol_frac * W: + continue # multi-column → handled above + yb = _faint_grey_band_above(gray, b) + if yb is not None and not _lead_photo_above(yb, b[0], b[2]): + bars.append({"y": yb, "x1": b[0], "x2": b[2]}) + + # Vertical column-divider rules. detect_separator_lines already requires each + # line to sit in a whitespace gutter (isolated on both sides), so photo/text + # internal strokes are filtered out. These fence rightward column-growth: an + # article must not grow across a printed vertical rule into the next column. + for L in res["v"]: + bars.append({"vert": True, "x": (L["x1"] + L["x2"]) // 2, + "y1": min(L["y1"], L["y2"]), "y2": max(L["y1"], L["y2"])}) + return bars + + +def overlay(png_path, lines, out_path): + img = cv2.imread(str(png_path)) + for L in lines["h"]: + cv2.line(img, (L["x1"], L["y1"]), (L["x2"], L["y2"]), (0, 0, 255), 6) + for L in lines["v"]: + cv2.line(img, (L["x1"], L["y1"]), (L["x2"], L["y2"]), (255, 0, 0), 6) + cv2.imwrite(str(out_path), img) + + +if __name__ == "__main__": + import sys + from pathlib import Path + p = Path(sys.argv[1]) + res = detect_separator_lines(p) + print(f"{p.name} {res['size']} H={len(res['h'])} V={len(res['v'])}") + for L in sorted(res["h"], key=lambda d: d["y1"]): + print(f" H y={L['y1']:>5} x[{L['x1']:>4}..{L['x2']:>4}] len={L['len']:>4} gray={L['gray']}") + for L in sorted(res["v"], key=lambda d: d["x1"]): + print(f" V x={L['x1']:>5} y[{L['y1']:>4}..{L['y2']:>4}] len={L['len']:>4} gray={L['gray']}") + out = Path(sys.argv[2]) if len(sys.argv) > 2 else Path(f"/tmp/lines_{p.stem}.png") + overlay(p, res, out) + print("overlay:", out) diff --git a/_nt4.txt b/_nt4.txt new file mode 100644 index 0000000..e336e83 --- /dev/null +++ b/_nt4.txt @@ -0,0 +1,4 @@ +WITH ≥1col floor: 24 blocks + R67 H 4DL members=[5, 7, 28, 34, 42, 48, 50, 67, 68, 73, 74, 80, 100, 102, 107, 127, 135] + R121 H 2DL members=[12, 14, 21, 39, 59, 111, 116, 121] +anchors: R23,R67,R81,R85,R87,R89,R90,R91,R93,R94,R95,R96,R97,R98,R101,R103,R108,R118,R119,R121,R122,R128,R131,R132 diff --git a/_nt_base.log b/_nt_base.log new file mode 100644 index 0000000..b5cf83d --- /dev/null +++ b/_nt_base.log @@ -0,0 +1,5 @@ +page 1: 11 crops, 11 datelines, 2+dl-blocks=none | Caption bound:4, Headline-strip:8, Multi-column body:4 +page 2: 23 crops, 22 datelines, 2+dl-blocks=[110] | +page 3: 20 crops, 21 datelines, 2+dl-blocks=[54, 62] | +page 4: 25 crops, 25 datelines, 2+dl-blocks=[67, 121] | +DONE -> output/NamastheTelangana_Siddipet_20260602_20260604_214937/all_article_crops_final diff --git a/_nt_val.log b/_nt_val.log new file mode 100644 index 0000000..b5cf83d --- /dev/null +++ b/_nt_val.log @@ -0,0 +1,5 @@ +page 1: 11 crops, 11 datelines, 2+dl-blocks=none | Caption bound:4, Headline-strip:8, Multi-column body:4 +page 2: 23 crops, 22 datelines, 2+dl-blocks=[110] | +page 3: 20 crops, 21 datelines, 2+dl-blocks=[54, 62] | +page 4: 25 crops, 25 datelines, 2+dl-blocks=[67, 121] | +DONE -> output/NamastheTelangana_Siddipet_20260602_20260604_214937/all_article_crops_final diff --git a/_nt_val2.log b/_nt_val2.log new file mode 100644 index 0000000..b5cf83d --- /dev/null +++ b/_nt_val2.log @@ -0,0 +1,5 @@ +page 1: 11 crops, 11 datelines, 2+dl-blocks=none | Caption bound:4, Headline-strip:8, Multi-column body:4 +page 2: 23 crops, 22 datelines, 2+dl-blocks=[110] | +page 3: 20 crops, 21 datelines, 2+dl-blocks=[54, 62] | +page 4: 25 crops, 25 datelines, 2+dl-blocks=[67, 121] | +DONE -> output/NamastheTelangana_Siddipet_20260602_20260604_214937/all_article_crops_final diff --git a/_nt_val3.log b/_nt_val3.log new file mode 100644 index 0000000..e20286d --- /dev/null +++ b/_nt_val3.log @@ -0,0 +1,5 @@ +page 1: 11 crops, 11 datelines, 2+dl-blocks=none | Caption bound:4, Headline-strip:8, Multi-column body:4 +page 2: 23 crops, 22 datelines, 2+dl-blocks=[110, 105] | +page 3: 20 crops, 21 datelines, 2+dl-blocks=[54, 62] | +page 4: 25 crops, 25 datelines, 2+dl-blocks=[67, 121] | +DONE -> output/NamastheTelangana_Siddipet_20260602_20260604_214937/all_article_crops_final diff --git a/_nt_val4.log b/_nt_val4.log new file mode 100644 index 0000000..b5cf83d --- /dev/null +++ b/_nt_val4.log @@ -0,0 +1,5 @@ +page 1: 11 crops, 11 datelines, 2+dl-blocks=none | Caption bound:4, Headline-strip:8, Multi-column body:4 +page 2: 23 crops, 22 datelines, 2+dl-blocks=[110] | +page 3: 20 crops, 21 datelines, 2+dl-blocks=[54, 62] | +page 4: 25 crops, 25 datelines, 2+dl-blocks=[67, 121] | +DONE -> output/NamastheTelangana_Siddipet_20260602_20260604_214937/all_article_crops_final diff --git a/_nt_val5.log b/_nt_val5.log new file mode 100644 index 0000000..b5cf83d --- /dev/null +++ b/_nt_val5.log @@ -0,0 +1,5 @@ +page 1: 11 crops, 11 datelines, 2+dl-blocks=none | Caption bound:4, Headline-strip:8, Multi-column body:4 +page 2: 23 crops, 22 datelines, 2+dl-blocks=[110] | +page 3: 20 crops, 21 datelines, 2+dl-blocks=[54, 62] | +page 4: 25 crops, 25 datelines, 2+dl-blocks=[67, 121] | +DONE -> output/NamastheTelangana_Siddipet_20260602_20260604_214937/all_article_crops_final diff --git a/_nt_val6.log b/_nt_val6.log new file mode 100644 index 0000000..b5cf83d --- /dev/null +++ b/_nt_val6.log @@ -0,0 +1,5 @@ +page 1: 11 crops, 11 datelines, 2+dl-blocks=none | Caption bound:4, Headline-strip:8, Multi-column body:4 +page 2: 23 crops, 22 datelines, 2+dl-blocks=[110] | +page 3: 20 crops, 21 datelines, 2+dl-blocks=[54, 62] | +page 4: 25 crops, 25 datelines, 2+dl-blocks=[67, 121] | +DONE -> output/NamastheTelangana_Siddipet_20260602_20260604_214937/all_article_crops_final diff --git a/_nt_val7.log b/_nt_val7.log new file mode 100644 index 0000000..bfffd7c --- /dev/null +++ b/_nt_val7.log @@ -0,0 +1,5 @@ +page 1: 11 crops, 11 datelines, 2+dl-blocks=none | Caption bound:4, Headline-strip:8, Multi-column body:4 +page 2: 23 crops, 22 datelines, 2+dl-blocks=[110] | +page 3: 20 crops, 21 datelines, 2+dl-blocks=[54, 62] | +page 4: 28 crops, 25 datelines, 2+dl-blocks=[80] | +DONE -> output/NamastheTelangana_Siddipet_20260602_20260604_214937/all_article_crops_final diff --git a/_nt_val8.log b/_nt_val8.log new file mode 100644 index 0000000..b5cf83d --- /dev/null +++ b/_nt_val8.log @@ -0,0 +1,5 @@ +page 1: 11 crops, 11 datelines, 2+dl-blocks=none | Caption bound:4, Headline-strip:8, Multi-column body:4 +page 2: 23 crops, 22 datelines, 2+dl-blocks=[110] | +page 3: 20 crops, 21 datelines, 2+dl-blocks=[54, 62] | +page 4: 25 crops, 25 datelines, 2+dl-blocks=[67, 121] | +DONE -> output/NamastheTelangana_Siddipet_20260602_20260604_214937/all_article_crops_final diff --git a/_nta.log b/_nta.log new file mode 100644 index 0000000..6e6f3f5 --- /dev/null +++ b/_nta.log @@ -0,0 +1,179 @@ +=== page 1: 5 crops, 6 datelines === + H44 [106,111,4084,3144] 3978x3033 mem=21 + DL5 [668,901,4084,6302] 3416x5401 mem=22 + DL9 [105,2334,1197,6302] 1092x3968 mem=5 + H59 [112,2762,3229,4860] 3117x2098 mem=13 + H41 [2577,4793,4084,6309] 1507x1516 mem=4 + 🖼 Orphan photo: R38 → article R5 (an image must belong to an article) +=== page 1: 5 crops, 6 datelines === + H44 [106,111,4084,3144] 3978x3033 mem=21 + DL5 [668,901,4084,6302] 3416x5401 mem=22 + DL9 [105,2334,1197,6302] 1092x3968 mem=5 + H59 [112,2762,3229,4860] 3117x2098 mem=13 + H41 [2577,4793,4084,6309] 1507x1516 mem=4 + 🖼 Orphan photo: R38 → article R5 (an image must belong to an article) +=== page 1: 5 crops, 6 datelines === + H44 [106,111,4084,3144] 3978x3033 mem=21 + DL5 [668,901,4084,6302] 3416x5401 mem=22 + DL9 [105,2334,1197,6302] 1092x3968 mem=5 + H59 [112,2762,3229,4860] 3117x2098 mem=13 + H41 [2577,4793,4084,6309] 1507x1516 mem=4 + 🖼 Orphan photo: R38 → article R5 (an image must belong to an article) +=== page 1: 5 crops, 6 datelines === + H44 [106,111,4084,3144] 3978x3033 mem=21 + DL5 [668,901,4084,6302] 3416x5401 mem=22 + DL9 [105,2334,1197,6302] 1092x3968 mem=5 + H59 [112,2762,3229,4860] 3117x2098 mem=13 + H41 [2577,4793,4084,6309] 1507x1516 mem=4 + 🖼 Orphan photo: R38 → article R5 (an image must belong to an article) +=== page 1: 5 crops, 6 datelines === + H44 [106,111,4084,3144] 3978x3033 mem=21 + DL5 [668,901,4084,6302] 3416x5401 mem=22 + DL9 [105,2334,1197,6302] 1092x3968 mem=5 + H59 [112,2762,3229,4860] 3117x2098 mem=13 + H41 [2577,4793,4084,6309] 1507x1516 mem=4 + 🖼 Orphan photo: R38 → article R5 (an image must belong to an article) +=== page 1: 5 crops, 6 datelines === + H44 [106,111,4084,3144] 3978x3033 mem=21 + DL5 [668,901,4084,6302] 3416x5401 mem=22 + DL9 [105,2334,1197,6302] 1092x3968 mem=5 + H59 [112,2762,3229,4860] 3117x2098 mem=13 + H41 [2577,4793,4084,6309] 1507x1516 mem=4 + 🖼 Orphan photo: R38 → article R5 (an image must belong to an article) +=== page 1: 5 crops, 6 datelines === + H44 [106,111,4084,3144] 3978x3033 mem=21 + DL5 [668,901,4084,6302] 3416x5401 mem=22 + DL9 [105,2334,1197,6302] 1092x3968 mem=5 + H59 [112,2762,3229,4860] 3117x2098 mem=13 + H41 [2577,4793,4084,6309] 1507x1516 mem=4 + 🖼 Orphan photo: R38 → article R5 (an image must belong to an article) +=== page 2: 31 crops, 30 datelines === + H80 [1833,465,3228,1814] 1395x1349 mem=6 + H65 [3293,465,4079,987] 786x522 mem=2 + H84 [106,501,1776,1820] 1670x1319 mem=6 + H43 [3292,1014,4079,1562] 787x548 mem=2 + DL9 [3292,1590,4079,2136] 787x546 mem=2 + H78 [1838,1678,2517,1805] 679x127 mem=2 + DL7 [104,1851,923,2566] 819x715 mem=3 + H74 [981,1883,2350,3060] 1369x1177 mem=4 + H72 [2408,1884,3227,2713] 819x829 mem=2 + DL5 [3292,2167,4079,2759] 787x592 mem=2 + DL23 [104,2596,922,2998] 818x402 mem=2 + H58 [2408,2772,3226,3479] 818x707 mem=2 + DL13 [3292,2787,4079,3333] 787x546 mem=2 + DL20 [106,3050,923,3478] 817x428 mem=2 + H60 [981,3117,2351,3479] 1370x362 mem=3 + DL1 [3293,3382,4079,4190] 786x808 mem=2 + P67 [131,3486,599,5211] 468x1725 mem=4 + DL10 [1559,3528,3226,4291] 1667x763 mem=4 + P63 [677,3558,1499,4144] 822x586 mem=2 + H50 [678,4170,1497,4966] 819x796 mem=2 + DL12 [3292,4217,4079,4815] 787x598 mem=2 + H71 [1560,4365,3226,4958] 1666x593 mem=3 + H51 [3292,4842,4079,5479] 787x637 mem=2 + DL41 [1561,4960,3225,5110] 1664x150 mem=2 + H54 [678,4987,1498,5443] 820x456 mem=2 + H76 [1559,5171,3228,5881] 1669x710 mem=5 + H53 [677,5467,1498,5964] 821x497 mem=2 + H83 [3291,5509,4079,6303] 788x794 mem=3 + H56 [1559,5944,3228,6302] 1669x358 mem=3 + H46 [680,5993,1498,6302] 818x309 mem=2 + O79 [152,6141,590,6257] 438x116 mem=1 + 🖼 Orphan photo: R90 → article R67 (an image must belong to an article) +=== page 2: 31 crops, 30 datelines === + H80 [1833,465,3228,1814] 1395x1349 mem=6 + H65 [3293,465,4079,987] 786x522 mem=2 + H84 [106,501,1776,1820] 1670x1319 mem=6 + H43 [3292,1014,4079,1562] 787x548 mem=2 + DL9 [3292,1590,4079,2136] 787x546 mem=2 + H78 [1838,1678,2517,1805] 679x127 mem=2 + DL7 [104,1851,923,2566] 819x715 mem=3 + H74 [981,1883,2350,3060] 1369x1177 mem=4 + H72 [2408,1884,3227,2713] 819x829 mem=2 + DL5 [3292,2167,4079,2759] 787x592 mem=2 + DL23 [104,2596,922,2998] 818x402 mem=2 + H58 [2408,2772,3226,3479] 818x707 mem=2 + DL13 [3292,2787,4079,3333] 787x546 mem=2 + DL20 [106,3050,923,3478] 817x428 mem=2 + H60 [981,3117,2351,3479] 1370x362 mem=3 + DL1 [3293,3382,4079,4190] 786x808 mem=2 + P67 [131,3486,599,5211] 468x1725 mem=4 + DL10 [1559,3528,3226,4291] 1667x763 mem=4 + P63 [677,3558,1499,4144] 822x586 mem=2 + H50 [678,4170,1497,4966] 819x796 mem=2 + DL12 [3292,4217,4079,4815] 787x598 mem=2 + H71 [1560,4365,3226,4958] 1666x593 mem=3 + H51 [3292,4842,4079,5479] 787x637 mem=2 + DL41 [1561,4960,3225,5110] 1664x150 mem=2 + H54 [678,4987,1498,5443] 820x456 mem=2 + H76 [1559,5171,3228,5881] 1669x710 mem=5 + H53 [677,5467,1498,5964] 821x497 mem=2 + H83 [3291,5509,4079,6303] 788x794 mem=3 + H56 [1559,5944,3228,6302] 1669x358 mem=3 + H46 [680,5993,1498,6302] 818x309 mem=2 + O79 [152,6141,590,6257] 438x116 mem=1 + 🖼 Orphan photo: R90 → article R67 (an image must belong to an article) +=== page 2: 31 crops, 30 datelines === + H80 [1833,465,3228,1814] 1395x1349 mem=6 + H65 [3293,465,4079,987] 786x522 mem=2 + H84 [106,501,1776,1820] 1670x1319 mem=6 + H43 [3292,1014,4079,1562] 787x548 mem=2 + DL9 [3292,1590,4079,2136] 787x546 mem=2 + H78 [1838,1678,2517,1805] 679x127 mem=2 + DL7 [104,1851,923,2566] 819x715 mem=3 + H74 [981,1883,2350,3060] 1369x1177 mem=4 + H72 [2408,1884,3227,2713] 819x829 mem=2 + DL5 [3292,2167,4079,2759] 787x592 mem=2 + DL23 [104,2596,922,2998] 818x402 mem=2 + H58 [2408,2772,3226,3479] 818x707 mem=2 + DL13 [3292,2787,4079,3333] 787x546 mem=2 + DL20 [106,3050,923,3478] 817x428 mem=2 + H60 [981,3117,2351,3479] 1370x362 mem=3 + DL1 [3293,3382,4079,4190] 786x808 mem=2 + P67 [131,3486,599,5211] 468x1725 mem=4 + DL10 [1559,3528,3226,4291] 1667x763 mem=4 + P63 [677,3558,1499,4144] 822x586 mem=2 + H50 [678,4170,1497,4966] 819x796 mem=2 + DL12 [3292,4217,4079,4815] 787x598 mem=2 + H71 [1560,4365,3226,4958] 1666x593 mem=3 + H51 [3292,4842,4079,5479] 787x637 mem=2 + DL41 [1561,4960,3225,5110] 1664x150 mem=2 + H54 [678,4987,1498,5443] 820x456 mem=2 + H76 [1559,5171,3228,5881] 1669x710 mem=5 + H53 [677,5467,1498,5964] 821x497 mem=2 + H83 [3291,5509,4079,6303] 788x794 mem=3 + H56 [1559,5944,3228,6302] 1669x358 mem=3 + H46 [680,5993,1498,6302] 818x309 mem=2 + O79 [152,6141,590,6257] 438x116 mem=1 + 🖼 Orphan photo: R90 → article R67 (an image must belong to an article) +=== page 2: 31 crops, 30 datelines === + H80 [1833,465,3228,1814] 1395x1349 mem=6 + H65 [3293,465,4079,987] 786x522 mem=2 + H84 [106,501,1776,1820] 1670x1319 mem=6 + H43 [3292,1014,4079,1562] 787x548 mem=2 + DL9 [3292,1590,4079,2136] 787x546 mem=2 + H78 [1838,1678,2517,1805] 679x127 mem=2 + DL7 [104,1851,923,2566] 819x715 mem=3 + H74 [981,1883,2350,3060] 1369x1177 mem=4 + H72 [2408,1884,3227,2713] 819x829 mem=2 + DL5 [3292,2167,4079,2759] 787x592 mem=2 + DL23 [104,2596,922,2998] 818x402 mem=2 + H58 [2408,2772,3226,3479] 818x707 mem=2 + DL13 [3292,2787,4079,3333] 787x546 mem=2 + DL20 [106,3050,923,3478] 817x428 mem=2 + H60 [981,3117,2351,3479] 1370x362 mem=3 + DL1 [3293,3382,4079,4190] 786x808 mem=2 + P67 [131,3486,599,5211] 468x1725 mem=4 + DL10 [1559,3528,3226,4291] 1667x763 mem=4 + P63 [677,3558,1499,4144] 822x586 mem=2 + H50 [678,4170,1497,4966] 819x796 mem=2 + DL12 [3292,4217,4079,4815] 787x598 mem=2 + H71 [1560,4365,3226,4958] 1666x593 mem=3 + H51 [3292,4842,4079,5479] 787x637 mem=2 + DL41 [1561,4960,3225,5110] 1664x150 mem=2 + H54 [678,4987,1498,5443] 820x456 mem=2 + H76 [1559,5171,3228,5881] 1669x710 mem=5 + H53 [677,5467,1498,5964] 821x497 mem=2 + H83 [3291,5509,4079,6303] 788x794 mem=3 + H56 [1559,5944,3228,6302] 1669x358 mem=3 + H46 [680,5993,1498,6302] 818x309 mem=2 \ No newline at end of file diff --git a/_nta2.log b/_nta2.log new file mode 100644 index 0000000..6c11505 --- /dev/null +++ b/_nta2.log @@ -0,0 +1,295 @@ +=== page 1: 6 crops, 6 datelines === + H44 [106,111,4084,3144] 3978x3033 mem=21 + DL56 [1265,901,4079,4724] 2814x3823 mem=11 + DL9 [105,2334,1197,6302] 1092x3968 mem=5 + H59 [112,2762,3229,4860] 3117x2098 mem=13 + H41 [2577,4793,4084,6309] 1507x1516 mem=4 + H30 [668,4829,3208,6302] 2540x1473 mem=11 + ─ NT late rule-split: R5 → 2 bands at structural rules [4764] + 🖼 Orphan photo: R38 → article R5 (an image must belong to an article) +=== page 1: 6 crops, 6 datelines === + H44 [106,111,4084,3144] 3978x3033 mem=21 + DL56 [1265,901,4079,4724] 2814x3823 mem=11 + DL9 [105,2334,1197,6302] 1092x3968 mem=5 + H59 [112,2762,3229,4860] 3117x2098 mem=13 + H41 [2577,4793,4084,6309] 1507x1516 mem=4 + H30 [668,4829,3208,6302] 2540x1473 mem=11 + ─ NT late rule-split: R5 → 2 bands at structural rules [4764] + 🖼 Orphan photo: R38 → article R5 (an image must belong to an article) +=== page 1: 6 crops, 6 datelines === + H44 [106,111,4084,3144] 3978x3033 mem=21 + DL56 [1265,901,4079,4724] 2814x3823 mem=11 + DL9 [105,2334,1197,6302] 1092x3968 mem=5 + H59 [112,2762,3229,4860] 3117x2098 mem=13 + H41 [2577,4793,4084,6309] 1507x1516 mem=4 + H30 [668,4829,3208,6302] 2540x1473 mem=11 + ─ NT late rule-split: R5 → 2 bands at structural rules [4764] + 🖼 Orphan photo: R38 → article R5 (an image must belong to an article) +=== page 1: 6 crops, 6 datelines === + H44 [106,111,4084,3144] 3978x3033 mem=21 + DL56 [1265,901,4079,4724] 2814x3823 mem=11 + DL9 [105,2334,1197,6302] 1092x3968 mem=5 + H59 [112,2762,3229,4860] 3117x2098 mem=13 + H41 [2577,4793,4084,6309] 1507x1516 mem=4 + H30 [668,4829,3208,6302] 2540x1473 mem=11 + ─ NT late rule-split: R5 → 2 bands at structural rules [4764] + 🖼 Orphan photo: R38 → article R5 (an image must belong to an article) +=== page 1: 6 crops, 6 datelines === + H44 [106,111,4084,3144] 3978x3033 mem=21 + DL56 [1265,901,4079,4724] 2814x3823 mem=11 + DL9 [105,2334,1197,6302] 1092x3968 mem=5 + H59 [112,2762,3229,4860] 3117x2098 mem=13 + H41 [2577,4793,4084,6309] 1507x1516 mem=4 + H30 [668,4829,3208,6302] 2540x1473 mem=11 + ─ NT late rule-split: R5 → 2 bands at structural rules [4764] + 🖼 Orphan photo: R38 → article R5 (an image must belong to an article) +=== page 1: 6 crops, 6 datelines === + H44 [106,111,4084,3144] 3978x3033 mem=21 + DL56 [1265,901,4079,4724] 2814x3823 mem=11 + DL9 [105,2334,1197,6302] 1092x3968 mem=5 + H59 [112,2762,3229,4860] 3117x2098 mem=13 + H41 [2577,4793,4084,6309] 1507x1516 mem=4 + H30 [668,4829,3208,6302] 2540x1473 mem=11 + ─ NT late rule-split: R5 → 2 bands at structural rules [4764] + 🖼 Orphan photo: R38 → article R5 (an image must belong to an article) +=== page 1: 6 crops, 6 datelines === + H44 [106,111,4084,3144] 3978x3033 mem=21 + DL56 [1265,901,4079,4724] 2814x3823 mem=11 + DL9 [105,2334,1197,6302] 1092x3968 mem=5 + H59 [112,2762,3229,4860] 3117x2098 mem=13 + H41 [2577,4793,4084,6309] 1507x1516 mem=4 + H30 [668,4829,3208,6302] 2540x1473 mem=11 + ─ NT late rule-split: R5 → 2 bands at structural rules [4764] + 🖼 Orphan photo: R38 → article R5 (an image must belong to an article) +=== page 2: 31 crops, 30 datelines === + H80 [1833,465,3228,1814] 1395x1349 mem=6 + H65 [3293,465,4079,987] 786x522 mem=2 + H84 [106,501,1776,1820] 1670x1319 mem=6 + H43 [3292,1014,4079,1562] 787x548 mem=2 + DL9 [3292,1590,4079,2136] 787x546 mem=2 + H78 [1838,1678,2517,1805] 679x127 mem=2 + DL7 [104,1851,923,2566] 819x715 mem=3 + H74 [981,1883,2350,3060] 1369x1177 mem=4 + H72 [2408,1884,3227,2713] 819x829 mem=2 + DL5 [3292,2167,4079,2759] 787x592 mem=2 + DL23 [104,2596,922,2998] 818x402 mem=2 + H58 [2408,2772,3226,3479] 818x707 mem=2 + DL13 [3292,2787,4079,3333] 787x546 mem=2 + DL20 [106,3050,923,3478] 817x428 mem=2 + H60 [981,3117,2351,3479] 1370x362 mem=3 + DL1 [3293,3382,4079,4190] 786x808 mem=2 + P67 [131,3486,599,5211] 468x1725 mem=4 + DL10 [1559,3528,3226,4291] 1667x763 mem=4 + P63 [677,3558,1499,4144] 822x586 mem=2 + H50 [678,4170,1497,4966] 819x796 mem=2 + DL12 [3292,4217,4079,4815] 787x598 mem=2 + H71 [1560,4365,3226,4958] 1666x593 mem=3 + H51 [3292,4842,4079,5479] 787x637 mem=2 + DL41 [1561,4960,3225,5110] 1664x150 mem=2 + H54 [678,4987,1498,5443] 820x456 mem=2 + H76 [1559,5171,3228,5881] 1669x710 mem=5 + H53 [677,5467,1498,5964] 821x497 mem=2 + H83 [3291,5509,4079,6303] 788x794 mem=3 + H56 [1559,5944,3228,6302] 1669x358 mem=3 + H46 [680,5993,1498,6302] 818x309 mem=2 + O79 [152,6141,590,6257] 438x116 mem=1 + 🖼 Orphan photo: R90 → article R67 (an image must belong to an article) +=== page 2: 31 crops, 30 datelines === + H80 [1833,465,3228,1814] 1395x1349 mem=6 + H65 [3293,465,4079,987] 786x522 mem=2 + H84 [106,501,1776,1820] 1670x1319 mem=6 + H43 [3292,1014,4079,1562] 787x548 mem=2 + DL9 [3292,1590,4079,2136] 787x546 mem=2 + H78 [1838,1678,2517,1805] 679x127 mem=2 + DL7 [104,1851,923,2566] 819x715 mem=3 + H74 [981,1883,2350,3060] 1369x1177 mem=4 + H72 [2408,1884,3227,2713] 819x829 mem=2 + DL5 [3292,2167,4079,2759] 787x592 mem=2 + DL23 [104,2596,922,2998] 818x402 mem=2 + H58 [2408,2772,3226,3479] 818x707 mem=2 + DL13 [3292,2787,4079,3333] 787x546 mem=2 + DL20 [106,3050,923,3478] 817x428 mem=2 + H60 [981,3117,2351,3479] 1370x362 mem=3 + DL1 [3293,3382,4079,4190] 786x808 mem=2 + P67 [131,3486,599,5211] 468x1725 mem=4 + DL10 [1559,3528,3226,4291] 1667x763 mem=4 + P63 [677,3558,1499,4144] 822x586 mem=2 + H50 [678,4170,1497,4966] 819x796 mem=2 + DL12 [3292,4217,4079,4815] 787x598 mem=2 + H71 [1560,4365,3226,4958] 1666x593 mem=3 + H51 [3292,4842,4079,5479] 787x637 mem=2 + DL41 [1561,4960,3225,5110] 1664x150 mem=2 + H54 [678,4987,1498,5443] 820x456 mem=2 + H76 [1559,5171,3228,5881] 1669x710 mem=5 + H53 [677,5467,1498,5964] 821x497 mem=2 + H83 [3291,5509,4079,6303] 788x794 mem=3 + H56 [1559,5944,3228,6302] 1669x358 mem=3 + H46 [680,5993,1498,6302] 818x309 mem=2 + O79 [152,6141,590,6257] 438x116 mem=1 + 🖼 Orphan photo: R90 → article R67 (an image must belong to an article) +=== page 2: 31 crops, 30 datelines === + H80 [1833,465,3228,1814] 1395x1349 mem=6 + H65 [3293,465,4079,987] 786x522 mem=2 + H84 [106,501,1776,1820] 1670x1319 mem=6 + H43 [3292,1014,4079,1562] 787x548 mem=2 + DL9 [3292,1590,4079,2136] 787x546 mem=2 + H78 [1838,1678,2517,1805] 679x127 mem=2 + DL7 [104,1851,923,2566] 819x715 mem=3 + H74 [981,1883,2350,3060] 1369x1177 mem=4 + H72 [2408,1884,3227,2713] 819x829 mem=2 + DL5 [3292,2167,4079,2759] 787x592 mem=2 + DL23 [104,2596,922,2998] 818x402 mem=2 + H58 [2408,2772,3226,3479] 818x707 mem=2 + DL13 [3292,2787,4079,3333] 787x546 mem=2 + DL20 [106,3050,923,3478] 817x428 mem=2 + H60 [981,3117,2351,3479] 1370x362 mem=3 + DL1 [3293,3382,4079,4190] 786x808 mem=2 + P67 [131,3486,599,5211] 468x1725 mem=4 + DL10 [1559,3528,3226,4291] 1667x763 mem=4 + P63 [677,3558,1499,4144] 822x586 mem=2 + H50 [678,4170,1497,4966] 819x796 mem=2 + DL12 [3292,4217,4079,4815] 787x598 mem=2 + H71 [1560,4365,3226,4958] 1666x593 mem=3 + H51 [3292,4842,4079,5479] 787x637 mem=2 + DL41 [1561,4960,3225,5110] 1664x150 mem=2 + H54 [678,4987,1498,5443] 820x456 mem=2 + H76 [1559,5171,3228,5881] 1669x710 mem=5 + H53 [677,5467,1498,5964] 821x497 mem=2 + H83 [3291,5509,4079,6303] 788x794 mem=3 + H56 [1559,5944,3228,6302] 1669x358 mem=3 + H46 [680,5993,1498,6302] 818x309 mem=2 + O79 [152,6141,590,6257] 438x116 mem=1 + 🖼 Orphan photo: R90 → article R67 (an image must belong to an article) +=== page 2: 31 crops, 30 datelines === + H80 [1833,465,3228,1814] 1395x1349 mem=6 + H65 [3293,465,4079,987] 786x522 mem=2 + H84 [106,501,1776,1820] 1670x1319 mem=6 + H43 [3292,1014,4079,1562] 787x548 mem=2 + DL9 [3292,1590,4079,2136] 787x546 mem=2 + H78 [1838,1678,2517,1805] 679x127 mem=2 + DL7 [104,1851,923,2566] 819x715 mem=3 + H74 [981,1883,2350,3060] 1369x1177 mem=4 + H72 [2408,1884,3227,2713] 819x829 mem=2 + DL5 [3292,2167,4079,2759] 787x592 mem=2 + DL23 [104,2596,922,2998] 818x402 mem=2 + H58 [2408,2772,3226,3479] 818x707 mem=2 + DL13 [3292,2787,4079,3333] 787x546 mem=2 + DL20 [106,3050,923,3478] 817x428 mem=2 + H60 [981,3117,2351,3479] 1370x362 mem=3 + DL1 [3293,3382,4079,4190] 786x808 mem=2 + P67 [131,3486,599,5211] 468x1725 mem=4 + DL10 [1559,3528,3226,4291] 1667x763 mem=4 + P63 [677,3558,1499,4144] 822x586 mem=2 + H50 [678,4170,1497,4966] 819x796 mem=2 + DL12 [3292,4217,4079,4815] 787x598 mem=2 + H71 [1560,4365,3226,4958] 1666x593 mem=3 + H51 [3292,4842,4079,5479] 787x637 mem=2 + DL41 [1561,4960,3225,5110] 1664x150 mem=2 + H54 [678,4987,1498,5443] 820x456 mem=2 + H76 [1559,5171,3228,5881] 1669x710 mem=5 + H53 [677,5467,1498,5964] 821x497 mem=2 + H83 [3291,5509,4079,6303] 788x794 mem=3 + H56 [1559,5944,3228,6302] 1669x358 mem=3 + H46 [680,5993,1498,6302] 818x309 mem=2 + O79 [152,6141,590,6257] 438x116 mem=1 + 🖼 Orphan photo: R90 → article R67 (an image must belong to an article) +=== page 2: 31 crops, 30 datelines === + H80 [1833,465,3228,1814] 1395x1349 mem=6 + H65 [3293,465,4079,987] 786x522 mem=2 + H84 [106,501,1776,1820] 1670x1319 mem=6 + H43 [3292,1014,4079,1562] 787x548 mem=2 + DL9 [3292,1590,4079,2136] 787x546 mem=2 + H78 [1838,1678,2517,1805] 679x127 mem=2 + DL7 [104,1851,923,2566] 819x715 mem=3 + H74 [981,1883,2350,3060] 1369x1177 mem=4 + H72 [2408,1884,3227,2713] 819x829 mem=2 + DL5 [3292,2167,4079,2759] 787x592 mem=2 + DL23 [104,2596,922,2998] 818x402 mem=2 + H58 [2408,2772,3226,3479] 818x707 mem=2 + DL13 [3292,2787,4079,3333] 787x546 mem=2 + DL20 [106,3050,923,3478] 817x428 mem=2 + H60 [981,3117,2351,3479] 1370x362 mem=3 + DL1 [3293,3382,4079,4190] 786x808 mem=2 + P67 [131,3486,599,5211] 468x1725 mem=4 + DL10 [1559,3528,3226,4291] 1667x763 mem=4 + P63 [677,3558,1499,4144] 822x586 mem=2 + H50 [678,4170,1497,4966] 819x796 mem=2 + DL12 [3292,4217,4079,4815] 787x598 mem=2 + H71 [1560,4365,3226,4958] 1666x593 mem=3 + H51 [3292,4842,4079,5479] 787x637 mem=2 + DL41 [1561,4960,3225,5110] 1664x150 mem=2 + H54 [678,4987,1498,5443] 820x456 mem=2 + H76 [1559,5171,3228,5881] 1669x710 mem=5 + H53 [677,5467,1498,5964] 821x497 mem=2 + H83 [3291,5509,4079,6303] 788x794 mem=3 + H56 [1559,5944,3228,6302] 1669x358 mem=3 + H46 [680,5993,1498,6302] 818x309 mem=2 + O79 [152,6141,590,6257] 438x116 mem=1 + 🖼 Orphan photo: R90 → article R67 (an image must belong to an article) +=== page 2: 31 crops, 30 datelines === + H80 [1833,465,3228,1814] 1395x1349 mem=6 + H65 [3293,465,4079,987] 786x522 mem=2 + H84 [106,501,1776,1820] 1670x1319 mem=6 + H43 [3292,1014,4079,1562] 787x548 mem=2 + DL9 [3292,1590,4079,2136] 787x546 mem=2 + H78 [1838,1678,2517,1805] 679x127 mem=2 + DL7 [104,1851,923,2566] 819x715 mem=3 + H74 [981,1883,2350,3060] 1369x1177 mem=4 + H72 [2408,1884,3227,2713] 819x829 mem=2 + DL5 [3292,2167,4079,2759] 787x592 mem=2 + DL23 [104,2596,922,2998] 818x402 mem=2 + H58 [2408,2772,3226,3479] 818x707 mem=2 + DL13 [3292,2787,4079,3333] 787x546 mem=2 + DL20 [106,3050,923,3478] 817x428 mem=2 + H60 [981,3117,2351,3479] 1370x362 mem=3 + DL1 [3293,3382,4079,4190] 786x808 mem=2 + P67 [131,3486,599,5211] 468x1725 mem=4 + DL10 [1559,3528,3226,4291] 1667x763 mem=4 + P63 [677,3558,1499,4144] 822x586 mem=2 + H50 [678,4170,1497,4966] 819x796 mem=2 + DL12 [3292,4217,4079,4815] 787x598 mem=2 + H71 [1560,4365,3226,4958] 1666x593 mem=3 + H51 [3292,4842,4079,5479] 787x637 mem=2 + DL41 [1561,4960,3225,5110] 1664x150 mem=2 + H54 [678,4987,1498,5443] 820x456 mem=2 + H76 [1559,5171,3228,5881] 1669x710 mem=5 + H53 [677,5467,1498,5964] 821x497 mem=2 + H83 [3291,5509,4079,6303] 788x794 mem=3 + H56 [1559,5944,3228,6302] 1669x358 mem=3 + H46 [680,5993,1498,6302] 818x309 mem=2 + O79 [152,6141,590,6257] 438x116 mem=1 + 🖼 Orphan photo: R90 → article R67 (an image must belong to an article) +=== page 2: 31 crops, 30 datelines === + H80 [1833,465,3228,1814] 1395x1349 mem=6 + H65 [3293,465,4079,987] 786x522 mem=2 + H84 [106,501,1776,1820] 1670x1319 mem=6 + H43 [3292,1014,4079,1562] 787x548 mem=2 + DL9 [3292,1590,4079,2136] 787x546 mem=2 + H78 [1838,1678,2517,1805] 679x127 mem=2 + DL7 [104,1851,923,2566] 819x715 mem=3 + H74 [981,1883,2350,3060] 1369x1177 mem=4 + H72 [2408,1884,3227,2713] 819x829 mem=2 + DL5 [3292,2167,4079,2759] 787x592 mem=2 + DL23 [104,2596,922,2998] 818x402 mem=2 + H58 [2408,2772,3226,3479] 818x707 mem=2 + DL13 [3292,2787,4079,3333] 787x546 mem=2 + DL20 [106,3050,923,3478] 817x428 mem=2 + H60 [981,3117,2351,3479] 1370x362 mem=3 + DL1 [3293,3382,4079,4190] 786x808 mem=2 + P67 [131,3486,599,5211] 468x1725 mem=4 + DL10 [1559,3528,3226,4291] 1667x763 mem=4 + P63 [677,3558,1499,4144] 822x586 mem=2 + H50 [678,4170,1497,4966] 819x796 mem=2 + DL12 [3292,4217,4079,4815] 787x598 mem=2 + H71 [1560,4365,3226,4958] 1666x593 mem=3 + H51 [3292,4842,4079,5479] 787x637 mem=2 + DL41 [1561,4960,3225,5110] 1664x150 mem=2 + H54 [678,4987,1498,5443] 820x456 mem=2 + H76 [1559,5171,3228,5881] 1669x710 mem=5 + H53 [677,5467,1498,5964] 821x497 mem=2 + H83 [3291,5509,4079,6303] 788x794 mem=3 + H56 [1559,5944,3228,6302] 1669x358 mem=3 + H46 [680,5993,1498,6302] 818x309 mem=2 + O79 [152,6141,590,6257] 438x116 mem=1 + 🖼 Orphan photo: R90 → article R67 (an image must belong to an article) +DONE -> output/NamastheTelangana_Adilabad_20260528_20260610_160056/all_article_crops_review diff --git a/_nta3.log b/_nta3.log new file mode 100644 index 0000000..c69bbe9 --- /dev/null +++ b/_nta3.log @@ -0,0 +1,281 @@ +=== page 1: 6 crops, 6 datelines === + H44 [106,111,4084,3144] 3978x3033 mem=23 + DL9 [105,2334,1200,6302] 1095x3968 mem=6 + DL13 [1320,2761,4079,4724] 2759x1963 mem=14 + H59 [115,3752,3228,4860] 3113x1108 mem=7 + H41 [2577,4793,4084,6309] 1507x1516 mem=4 + H30 [668,4829,3208,6302] 2540x1473 mem=11 + ─ NT late rule-split: R5 → 2 bands at structural rules [4764] + 🖼 Orphan photo: R38 → article R5 (an image must belong to an article) +=== page 1: 6 crops, 6 datelines === + H44 [106,111,4084,3144] 3978x3033 mem=23 + DL9 [105,2334,1200,6302] 1095x3968 mem=6 + DL13 [1320,2761,4079,4724] 2759x1963 mem=14 + H59 [115,3752,3228,4860] 3113x1108 mem=7 + H41 [2577,4793,4084,6309] 1507x1516 mem=4 + H30 [668,4829,3208,6302] 2540x1473 mem=11 + ─ NT late rule-split: R5 → 2 bands at structural rules [4764] + 🖼 Orphan photo: R38 → article R5 (an image must belong to an article) +=== page 1: 6 crops, 6 datelines === + H44 [106,111,4084,3144] 3978x3033 mem=23 + DL9 [105,2334,1200,6302] 1095x3968 mem=6 + DL13 [1320,2761,4079,4724] 2759x1963 mem=14 + H59 [115,3752,3228,4860] 3113x1108 mem=7 + H41 [2577,4793,4084,6309] 1507x1516 mem=4 + H30 [668,4829,3208,6302] 2540x1473 mem=11 + ─ NT late rule-split: R5 → 2 bands at structural rules [4764] + 🖼 Orphan photo: R38 → article R5 (an image must belong to an article) +=== page 1: 6 crops, 6 datelines === + H44 [106,111,4084,3144] 3978x3033 mem=23 + DL9 [105,2334,1200,6302] 1095x3968 mem=6 + DL13 [1320,2761,4079,4724] 2759x1963 mem=14 + H59 [115,3752,3228,4860] 3113x1108 mem=7 + H41 [2577,4793,4084,6309] 1507x1516 mem=4 + H30 [668,4829,3208,6302] 2540x1473 mem=11 + ─ NT late rule-split: R5 → 2 bands at structural rules [4764] + 🖼 Orphan photo: R38 → article R5 (an image must belong to an article) +=== page 1: 6 crops, 6 datelines === + H44 [106,111,4084,3144] 3978x3033 mem=23 + DL9 [105,2334,1200,6302] 1095x3968 mem=6 + DL13 [1320,2761,4079,4724] 2759x1963 mem=14 + H59 [115,3752,3228,4860] 3113x1108 mem=7 + H41 [2577,4793,4084,6309] 1507x1516 mem=4 + H30 [668,4829,3208,6302] 2540x1473 mem=11 + ─ NT late rule-split: R5 → 2 bands at structural rules [4764] + 🖼 Orphan photo: R38 → article R5 (an image must belong to an article) +=== page 1: 6 crops, 6 datelines === + H44 [106,111,4084,3144] 3978x3033 mem=23 + DL9 [105,2334,1200,6302] 1095x3968 mem=6 + DL13 [1320,2761,4079,4724] 2759x1963 mem=14 + H59 [115,3752,3228,4860] 3113x1108 mem=7 + H41 [2577,4793,4084,6309] 1507x1516 mem=4 + H30 [668,4829,3208,6302] 2540x1473 mem=11 + ─ NT late rule-split: R5 → 2 bands at structural rules [4764] + 🖼 Orphan photo: R38 → article R5 (an image must belong to an article) +=== page 1: 6 crops, 6 datelines === + H44 [106,111,4084,3144] 3978x3033 mem=23 + DL9 [105,2334,1200,6302] 1095x3968 mem=6 + DL13 [1320,2761,4079,4724] 2759x1963 mem=14 + H59 [115,3752,3228,4860] 3113x1108 mem=7 + H41 [2577,4793,4084,6309] 1507x1516 mem=4 + H30 [668,4829,3208,6302] 2540x1473 mem=11 + ─ NT late rule-split: R5 → 2 bands at structural rules [4764] + 🖼 Orphan photo: R38 → article R5 (an image must belong to an article) +=== page 2: 30 crops, 30 datelines === + H80 [1833,465,3228,1814] 1395x1349 mem=6 + H65 [3293,465,4079,987] 786x522 mem=2 + H84 [106,501,1776,1820] 1670x1319 mem=6 + H43 [3292,1014,4079,1562] 787x548 mem=2 + DL9 [3292,1590,4079,2136] 787x546 mem=2 + H78 [1838,1678,2517,1805] 679x127 mem=2 + DL7 [104,1851,923,2566] 819x715 mem=3 + H74 [981,1883,2350,3060] 1369x1177 mem=4 + H72 [2408,1884,3227,2713] 819x829 mem=2 + DL5 [3292,2167,4079,2759] 787x592 mem=2 + DL23 [104,2596,922,2998] 818x402 mem=2 + H58 [2408,2772,3226,3479] 818x707 mem=2 + DL13 [3292,2787,4079,3333] 787x546 mem=2 + DL20 [106,3050,923,3478] 817x428 mem=2 + H60 [981,3117,2351,3479] 1370x362 mem=3 + DL1 [3293,3382,4079,4190] 786x808 mem=2 + P67 [131,3486,599,4979] 468x1493 mem=3 + DL10 [1559,3528,3226,4291] 1667x763 mem=4 + P63 [677,3558,1499,4144] 822x586 mem=2 + H50 [610,4170,1497,4966] 887x796 mem=2 + DL12 [3292,4217,4079,4815] 787x598 mem=2 + H71 [1560,4365,3226,4958] 1666x593 mem=3 + H51 [3292,4842,4079,5479] 787x637 mem=2 + DL41 [1561,4960,3225,5110] 1664x150 mem=2 + DL26 [154,4987,1498,5443] 1344x456 mem=3 + H76 [1559,5171,3228,5881] 1669x710 mem=5 + H53 [114,5467,1498,6269] 1384x802 mem=3 + H83 [3291,5509,4079,6303] 788x794 mem=3 + H56 [1559,5944,3228,6302] 1669x358 mem=3 + H46 [590,5993,1498,6302] 908x309 mem=2 +=== page 2: 30 crops, 30 datelines === + H80 [1833,465,3228,1814] 1395x1349 mem=6 + H65 [3293,465,4079,987] 786x522 mem=2 + H84 [106,501,1776,1820] 1670x1319 mem=6 + H43 [3292,1014,4079,1562] 787x548 mem=2 + DL9 [3292,1590,4079,2136] 787x546 mem=2 + H78 [1838,1678,2517,1805] 679x127 mem=2 + DL7 [104,1851,923,2566] 819x715 mem=3 + H74 [981,1883,2350,3060] 1369x1177 mem=4 + H72 [2408,1884,3227,2713] 819x829 mem=2 + DL5 [3292,2167,4079,2759] 787x592 mem=2 + DL23 [104,2596,922,2998] 818x402 mem=2 + H58 [2408,2772,3226,3479] 818x707 mem=2 + DL13 [3292,2787,4079,3333] 787x546 mem=2 + DL20 [106,3050,923,3478] 817x428 mem=2 + H60 [981,3117,2351,3479] 1370x362 mem=3 + DL1 [3293,3382,4079,4190] 786x808 mem=2 + P67 [131,3486,599,4979] 468x1493 mem=3 + DL10 [1559,3528,3226,4291] 1667x763 mem=4 + P63 [677,3558,1499,4144] 822x586 mem=2 + H50 [610,4170,1497,4966] 887x796 mem=2 + DL12 [3292,4217,4079,4815] 787x598 mem=2 + H71 [1560,4365,3226,4958] 1666x593 mem=3 + H51 [3292,4842,4079,5479] 787x637 mem=2 + DL41 [1561,4960,3225,5110] 1664x150 mem=2 + DL26 [154,4987,1498,5443] 1344x456 mem=3 + H76 [1559,5171,3228,5881] 1669x710 mem=5 + H53 [114,5467,1498,6269] 1384x802 mem=3 + H83 [3291,5509,4079,6303] 788x794 mem=3 + H56 [1559,5944,3228,6302] 1669x358 mem=3 + H46 [590,5993,1498,6302] 908x309 mem=2 +=== page 2: 30 crops, 30 datelines === + H80 [1833,465,3228,1814] 1395x1349 mem=6 + H65 [3293,465,4079,987] 786x522 mem=2 + H84 [106,501,1776,1820] 1670x1319 mem=6 + H43 [3292,1014,4079,1562] 787x548 mem=2 + DL9 [3292,1590,4079,2136] 787x546 mem=2 + H78 [1838,1678,2517,1805] 679x127 mem=2 + DL7 [104,1851,923,2566] 819x715 mem=3 + H74 [981,1883,2350,3060] 1369x1177 mem=4 + H72 [2408,1884,3227,2713] 819x829 mem=2 + DL5 [3292,2167,4079,2759] 787x592 mem=2 + DL23 [104,2596,922,2998] 818x402 mem=2 + H58 [2408,2772,3226,3479] 818x707 mem=2 + DL13 [3292,2787,4079,3333] 787x546 mem=2 + DL20 [106,3050,923,3478] 817x428 mem=2 + H60 [981,3117,2351,3479] 1370x362 mem=3 + DL1 [3293,3382,4079,4190] 786x808 mem=2 + P67 [131,3486,599,4979] 468x1493 mem=3 + DL10 [1559,3528,3226,4291] 1667x763 mem=4 + P63 [677,3558,1499,4144] 822x586 mem=2 + H50 [610,4170,1497,4966] 887x796 mem=2 + DL12 [3292,4217,4079,4815] 787x598 mem=2 + H71 [1560,4365,3226,4958] 1666x593 mem=3 + H51 [3292,4842,4079,5479] 787x637 mem=2 + DL41 [1561,4960,3225,5110] 1664x150 mem=2 + DL26 [154,4987,1498,5443] 1344x456 mem=3 + H76 [1559,5171,3228,5881] 1669x710 mem=5 + H53 [114,5467,1498,6269] 1384x802 mem=3 + H83 [3291,5509,4079,6303] 788x794 mem=3 + H56 [1559,5944,3228,6302] 1669x358 mem=3 + H46 [590,5993,1498,6302] 908x309 mem=2 +=== page 2: 30 crops, 30 datelines === + H80 [1833,465,3228,1814] 1395x1349 mem=6 + H65 [3293,465,4079,987] 786x522 mem=2 + H84 [106,501,1776,1820] 1670x1319 mem=6 + H43 [3292,1014,4079,1562] 787x548 mem=2 + DL9 [3292,1590,4079,2136] 787x546 mem=2 + H78 [1838,1678,2517,1805] 679x127 mem=2 + DL7 [104,1851,923,2566] 819x715 mem=3 + H74 [981,1883,2350,3060] 1369x1177 mem=4 + H72 [2408,1884,3227,2713] 819x829 mem=2 + DL5 [3292,2167,4079,2759] 787x592 mem=2 + DL23 [104,2596,922,2998] 818x402 mem=2 + H58 [2408,2772,3226,3479] 818x707 mem=2 + DL13 [3292,2787,4079,3333] 787x546 mem=2 + DL20 [106,3050,923,3478] 817x428 mem=2 + H60 [981,3117,2351,3479] 1370x362 mem=3 + DL1 [3293,3382,4079,4190] 786x808 mem=2 + P67 [131,3486,599,4979] 468x1493 mem=3 + DL10 [1559,3528,3226,4291] 1667x763 mem=4 + P63 [677,3558,1499,4144] 822x586 mem=2 + H50 [610,4170,1497,4966] 887x796 mem=2 + DL12 [3292,4217,4079,4815] 787x598 mem=2 + H71 [1560,4365,3226,4958] 1666x593 mem=3 + H51 [3292,4842,4079,5479] 787x637 mem=2 + DL41 [1561,4960,3225,5110] 1664x150 mem=2 + DL26 [154,4987,1498,5443] 1344x456 mem=3 + H76 [1559,5171,3228,5881] 1669x710 mem=5 + H53 [114,5467,1498,6269] 1384x802 mem=3 + H83 [3291,5509,4079,6303] 788x794 mem=3 + H56 [1559,5944,3228,6302] 1669x358 mem=3 + H46 [590,5993,1498,6302] 908x309 mem=2 +=== page 2: 30 crops, 30 datelines === + H80 [1833,465,3228,1814] 1395x1349 mem=6 + H65 [3293,465,4079,987] 786x522 mem=2 + H84 [106,501,1776,1820] 1670x1319 mem=6 + H43 [3292,1014,4079,1562] 787x548 mem=2 + DL9 [3292,1590,4079,2136] 787x546 mem=2 + H78 [1838,1678,2517,1805] 679x127 mem=2 + DL7 [104,1851,923,2566] 819x715 mem=3 + H74 [981,1883,2350,3060] 1369x1177 mem=4 + H72 [2408,1884,3227,2713] 819x829 mem=2 + DL5 [3292,2167,4079,2759] 787x592 mem=2 + DL23 [104,2596,922,2998] 818x402 mem=2 + H58 [2408,2772,3226,3479] 818x707 mem=2 + DL13 [3292,2787,4079,3333] 787x546 mem=2 + DL20 [106,3050,923,3478] 817x428 mem=2 + H60 [981,3117,2351,3479] 1370x362 mem=3 + DL1 [3293,3382,4079,4190] 786x808 mem=2 + P67 [131,3486,599,4979] 468x1493 mem=3 + DL10 [1559,3528,3226,4291] 1667x763 mem=4 + P63 [677,3558,1499,4144] 822x586 mem=2 + H50 [610,4170,1497,4966] 887x796 mem=2 + DL12 [3292,4217,4079,4815] 787x598 mem=2 + H71 [1560,4365,3226,4958] 1666x593 mem=3 + H51 [3292,4842,4079,5479] 787x637 mem=2 + DL41 [1561,4960,3225,5110] 1664x150 mem=2 + DL26 [154,4987,1498,5443] 1344x456 mem=3 + H76 [1559,5171,3228,5881] 1669x710 mem=5 + H53 [114,5467,1498,6269] 1384x802 mem=3 + H83 [3291,5509,4079,6303] 788x794 mem=3 + H56 [1559,5944,3228,6302] 1669x358 mem=3 + H46 [590,5993,1498,6302] 908x309 mem=2 +=== page 2: 30 crops, 30 datelines === + H80 [1833,465,3228,1814] 1395x1349 mem=6 + H65 [3293,465,4079,987] 786x522 mem=2 + H84 [106,501,1776,1820] 1670x1319 mem=6 + H43 [3292,1014,4079,1562] 787x548 mem=2 + DL9 [3292,1590,4079,2136] 787x546 mem=2 + H78 [1838,1678,2517,1805] 679x127 mem=2 + DL7 [104,1851,923,2566] 819x715 mem=3 + H74 [981,1883,2350,3060] 1369x1177 mem=4 + H72 [2408,1884,3227,2713] 819x829 mem=2 + DL5 [3292,2167,4079,2759] 787x592 mem=2 + DL23 [104,2596,922,2998] 818x402 mem=2 + H58 [2408,2772,3226,3479] 818x707 mem=2 + DL13 [3292,2787,4079,3333] 787x546 mem=2 + DL20 [106,3050,923,3478] 817x428 mem=2 + H60 [981,3117,2351,3479] 1370x362 mem=3 + DL1 [3293,3382,4079,4190] 786x808 mem=2 + P67 [131,3486,599,4979] 468x1493 mem=3 + DL10 [1559,3528,3226,4291] 1667x763 mem=4 + P63 [677,3558,1499,4144] 822x586 mem=2 + H50 [610,4170,1497,4966] 887x796 mem=2 + DL12 [3292,4217,4079,4815] 787x598 mem=2 + H71 [1560,4365,3226,4958] 1666x593 mem=3 + H51 [3292,4842,4079,5479] 787x637 mem=2 + DL41 [1561,4960,3225,5110] 1664x150 mem=2 + DL26 [154,4987,1498,5443] 1344x456 mem=3 + H76 [1559,5171,3228,5881] 1669x710 mem=5 + H53 [114,5467,1498,6269] 1384x802 mem=3 + H83 [3291,5509,4079,6303] 788x794 mem=3 + H56 [1559,5944,3228,6302] 1669x358 mem=3 + H46 [590,5993,1498,6302] 908x309 mem=2 +=== page 2: 30 crops, 30 datelines === + H80 [1833,465,3228,1814] 1395x1349 mem=6 + H65 [3293,465,4079,987] 786x522 mem=2 + H84 [106,501,1776,1820] 1670x1319 mem=6 + H43 [3292,1014,4079,1562] 787x548 mem=2 + DL9 [3292,1590,4079,2136] 787x546 mem=2 + H78 [1838,1678,2517,1805] 679x127 mem=2 + DL7 [104,1851,923,2566] 819x715 mem=3 + H74 [981,1883,2350,3060] 1369x1177 mem=4 + H72 [2408,1884,3227,2713] 819x829 mem=2 + DL5 [3292,2167,4079,2759] 787x592 mem=2 + DL23 [104,2596,922,2998] 818x402 mem=2 + H58 [2408,2772,3226,3479] 818x707 mem=2 + DL13 [3292,2787,4079,3333] 787x546 mem=2 + DL20 [106,3050,923,3478] 817x428 mem=2 + H60 [981,3117,2351,3479] 1370x362 mem=3 + DL1 [3293,3382,4079,4190] 786x808 mem=2 + P67 [131,3486,599,4979] 468x1493 mem=3 + DL10 [1559,3528,3226,4291] 1667x763 mem=4 + P63 [677,3558,1499,4144] 822x586 mem=2 + H50 [610,4170,1497,4966] 887x796 mem=2 + DL12 [3292,4217,4079,4815] 787x598 mem=2 + H71 [1560,4365,3226,4958] 1666x593 mem=3 + H51 [3292,4842,4079,5479] 787x637 mem=2 + DL41 [1561,4960,3225,5110] 1664x150 mem=2 + DL26 [154,4987,1498,5443] 1344x456 mem=3 + H76 [1559,5171,3228,5881] 1669x710 mem=5 + H53 [114,5467,1498,6269] 1384x802 mem=3 + H83 [3291,5509,4079,6303] 788x794 mem=3 + H56 [1559,5944,3228,6302] 1669x358 mem=3 + H46 [590,5993,1498,6302] 908x309 mem=2 +DONE -> output/NamastheTelangana_Adilabad_20260528_20260610_160056/all_article_crops_review diff --git a/_nta4.log b/_nta4.log new file mode 100644 index 0000000..2fb4f59 --- /dev/null +++ b/_nta4.log @@ -0,0 +1,295 @@ +=== page 1: 6 crops, 6 datelines === + H44 [106,111,4084,3144] 3978x3033 mem=23 + DL9 [105,2334,1200,6302] 1095x3968 mem=6 + DL13 [1320,2761,4079,4724] 2759x1963 mem=14 + H59 [115,3752,3228,4860] 3113x1108 mem=7 + H41 [2577,4793,4084,6309] 1507x1516 mem=4 + H30 [668,4829,3208,6302] 2540x1473 mem=11 + ─ NT late rule-split: R5 → 2 bands at structural rules [4764] + 🖼 Orphan photo: R38 → article R5 (an image must belong to an article) +=== page 1: 6 crops, 6 datelines === + H44 [106,111,4084,3144] 3978x3033 mem=23 + DL9 [105,2334,1200,6302] 1095x3968 mem=6 + DL13 [1320,2761,4079,4724] 2759x1963 mem=14 + H59 [115,3752,3228,4860] 3113x1108 mem=7 + H41 [2577,4793,4084,6309] 1507x1516 mem=4 + H30 [668,4829,3208,6302] 2540x1473 mem=11 + ─ NT late rule-split: R5 → 2 bands at structural rules [4764] + 🖼 Orphan photo: R38 → article R5 (an image must belong to an article) +=== page 1: 6 crops, 6 datelines === + H44 [106,111,4084,3144] 3978x3033 mem=23 + DL9 [105,2334,1200,6302] 1095x3968 mem=6 + DL13 [1320,2761,4079,4724] 2759x1963 mem=14 + H59 [115,3752,3228,4860] 3113x1108 mem=7 + H41 [2577,4793,4084,6309] 1507x1516 mem=4 + H30 [668,4829,3208,6302] 2540x1473 mem=11 + ─ NT late rule-split: R5 → 2 bands at structural rules [4764] + 🖼 Orphan photo: R38 → article R5 (an image must belong to an article) +=== page 1: 6 crops, 6 datelines === + H44 [106,111,4084,3144] 3978x3033 mem=23 + DL9 [105,2334,1200,6302] 1095x3968 mem=6 + DL13 [1320,2761,4079,4724] 2759x1963 mem=14 + H59 [115,3752,3228,4860] 3113x1108 mem=7 + H41 [2577,4793,4084,6309] 1507x1516 mem=4 + H30 [668,4829,3208,6302] 2540x1473 mem=11 + ─ NT late rule-split: R5 → 2 bands at structural rules [4764] + 🖼 Orphan photo: R38 → article R5 (an image must belong to an article) +=== page 1: 6 crops, 6 datelines === + H44 [106,111,4084,3144] 3978x3033 mem=23 + DL9 [105,2334,1200,6302] 1095x3968 mem=6 + DL13 [1320,2761,4079,4724] 2759x1963 mem=14 + H59 [115,3752,3228,4860] 3113x1108 mem=7 + H41 [2577,4793,4084,6309] 1507x1516 mem=4 + H30 [668,4829,3208,6302] 2540x1473 mem=11 + ─ NT late rule-split: R5 → 2 bands at structural rules [4764] + 🖼 Orphan photo: R38 → article R5 (an image must belong to an article) +=== page 1: 6 crops, 6 datelines === + H44 [106,111,4084,3144] 3978x3033 mem=23 + DL9 [105,2334,1200,6302] 1095x3968 mem=6 + DL13 [1320,2761,4079,4724] 2759x1963 mem=14 + H59 [115,3752,3228,4860] 3113x1108 mem=7 + H41 [2577,4793,4084,6309] 1507x1516 mem=4 + H30 [668,4829,3208,6302] 2540x1473 mem=11 + ─ NT late rule-split: R5 → 2 bands at structural rules [4764] + 🖼 Orphan photo: R38 → article R5 (an image must belong to an article) +=== page 1: 6 crops, 6 datelines === + H44 [106,111,4084,3144] 3978x3033 mem=23 + DL9 [105,2334,1200,6302] 1095x3968 mem=6 + DL13 [1320,2761,4079,4724] 2759x1963 mem=14 + H59 [115,3752,3228,4860] 3113x1108 mem=7 + H41 [2577,4793,4084,6309] 1507x1516 mem=4 + H30 [668,4829,3208,6302] 2540x1473 mem=11 + ─ NT late rule-split: R5 → 2 bands at structural rules [4764] + 🖼 Orphan photo: R38 → article R5 (an image must belong to an article) +=== page 2: 31 crops, 30 datelines === + H80 [1833,465,3228,1814] 1395x1349 mem=6 + H65 [3293,465,4079,987] 786x522 mem=2 + H84 [106,501,1776,1820] 1670x1319 mem=6 + H43 [3292,1014,4079,1562] 787x548 mem=2 + DL9 [3292,1590,4079,2136] 787x546 mem=2 + H78 [1838,1678,2517,1805] 679x127 mem=2 + DL7 [104,1851,923,2566] 819x715 mem=3 + H74 [981,1883,2350,3060] 1369x1177 mem=4 + H72 [2408,1884,3227,2713] 819x829 mem=2 + DL5 [3292,2167,4079,2759] 787x592 mem=2 + DL23 [104,2596,922,2998] 818x402 mem=2 + H58 [2408,2772,3226,3479] 818x707 mem=2 + DL13 [3292,2787,4079,3333] 787x546 mem=2 + DL20 [106,3050,923,3478] 817x428 mem=2 + H60 [981,3117,2351,3479] 1370x362 mem=3 + DL1 [3293,3382,4079,4190] 786x808 mem=2 + P67 [131,3486,599,5211] 468x1725 mem=4 + DL10 [1559,3528,3226,4291] 1667x763 mem=4 + P63 [677,3558,1499,4144] 822x586 mem=2 + H50 [678,4170,1497,4966] 819x796 mem=2 + DL12 [3292,4217,4079,4815] 787x598 mem=2 + H71 [1560,4365,3226,4958] 1666x593 mem=3 + H51 [3292,4842,4079,5479] 787x637 mem=2 + DL41 [1561,4960,3225,5110] 1664x150 mem=2 + H54 [678,4987,1498,5443] 820x456 mem=2 + H76 [1559,5171,3228,5881] 1669x710 mem=5 + H53 [677,5467,1498,5964] 821x497 mem=2 + H83 [3291,5509,4079,6303] 788x794 mem=3 + H56 [1559,5944,3228,6302] 1669x358 mem=3 + H46 [680,5993,1498,6302] 818x309 mem=2 + O79 [152,6141,590,6257] 438x116 mem=1 + 🖼 Orphan photo: R90 → article R67 (an image must belong to an article) +=== page 2: 31 crops, 30 datelines === + H80 [1833,465,3228,1814] 1395x1349 mem=6 + H65 [3293,465,4079,987] 786x522 mem=2 + H84 [106,501,1776,1820] 1670x1319 mem=6 + H43 [3292,1014,4079,1562] 787x548 mem=2 + DL9 [3292,1590,4079,2136] 787x546 mem=2 + H78 [1838,1678,2517,1805] 679x127 mem=2 + DL7 [104,1851,923,2566] 819x715 mem=3 + H74 [981,1883,2350,3060] 1369x1177 mem=4 + H72 [2408,1884,3227,2713] 819x829 mem=2 + DL5 [3292,2167,4079,2759] 787x592 mem=2 + DL23 [104,2596,922,2998] 818x402 mem=2 + H58 [2408,2772,3226,3479] 818x707 mem=2 + DL13 [3292,2787,4079,3333] 787x546 mem=2 + DL20 [106,3050,923,3478] 817x428 mem=2 + H60 [981,3117,2351,3479] 1370x362 mem=3 + DL1 [3293,3382,4079,4190] 786x808 mem=2 + P67 [131,3486,599,5211] 468x1725 mem=4 + DL10 [1559,3528,3226,4291] 1667x763 mem=4 + P63 [677,3558,1499,4144] 822x586 mem=2 + H50 [678,4170,1497,4966] 819x796 mem=2 + DL12 [3292,4217,4079,4815] 787x598 mem=2 + H71 [1560,4365,3226,4958] 1666x593 mem=3 + H51 [3292,4842,4079,5479] 787x637 mem=2 + DL41 [1561,4960,3225,5110] 1664x150 mem=2 + H54 [678,4987,1498,5443] 820x456 mem=2 + H76 [1559,5171,3228,5881] 1669x710 mem=5 + H53 [677,5467,1498,5964] 821x497 mem=2 + H83 [3291,5509,4079,6303] 788x794 mem=3 + H56 [1559,5944,3228,6302] 1669x358 mem=3 + H46 [680,5993,1498,6302] 818x309 mem=2 + O79 [152,6141,590,6257] 438x116 mem=1 + 🖼 Orphan photo: R90 → article R67 (an image must belong to an article) +=== page 2: 31 crops, 30 datelines === + H80 [1833,465,3228,1814] 1395x1349 mem=6 + H65 [3293,465,4079,987] 786x522 mem=2 + H84 [106,501,1776,1820] 1670x1319 mem=6 + H43 [3292,1014,4079,1562] 787x548 mem=2 + DL9 [3292,1590,4079,2136] 787x546 mem=2 + H78 [1838,1678,2517,1805] 679x127 mem=2 + DL7 [104,1851,923,2566] 819x715 mem=3 + H74 [981,1883,2350,3060] 1369x1177 mem=4 + H72 [2408,1884,3227,2713] 819x829 mem=2 + DL5 [3292,2167,4079,2759] 787x592 mem=2 + DL23 [104,2596,922,2998] 818x402 mem=2 + H58 [2408,2772,3226,3479] 818x707 mem=2 + DL13 [3292,2787,4079,3333] 787x546 mem=2 + DL20 [106,3050,923,3478] 817x428 mem=2 + H60 [981,3117,2351,3479] 1370x362 mem=3 + DL1 [3293,3382,4079,4190] 786x808 mem=2 + P67 [131,3486,599,5211] 468x1725 mem=4 + DL10 [1559,3528,3226,4291] 1667x763 mem=4 + P63 [677,3558,1499,4144] 822x586 mem=2 + H50 [678,4170,1497,4966] 819x796 mem=2 + DL12 [3292,4217,4079,4815] 787x598 mem=2 + H71 [1560,4365,3226,4958] 1666x593 mem=3 + H51 [3292,4842,4079,5479] 787x637 mem=2 + DL41 [1561,4960,3225,5110] 1664x150 mem=2 + H54 [678,4987,1498,5443] 820x456 mem=2 + H76 [1559,5171,3228,5881] 1669x710 mem=5 + H53 [677,5467,1498,5964] 821x497 mem=2 + H83 [3291,5509,4079,6303] 788x794 mem=3 + H56 [1559,5944,3228,6302] 1669x358 mem=3 + H46 [680,5993,1498,6302] 818x309 mem=2 + O79 [152,6141,590,6257] 438x116 mem=1 + 🖼 Orphan photo: R90 → article R67 (an image must belong to an article) +=== page 2: 31 crops, 30 datelines === + H80 [1833,465,3228,1814] 1395x1349 mem=6 + H65 [3293,465,4079,987] 786x522 mem=2 + H84 [106,501,1776,1820] 1670x1319 mem=6 + H43 [3292,1014,4079,1562] 787x548 mem=2 + DL9 [3292,1590,4079,2136] 787x546 mem=2 + H78 [1838,1678,2517,1805] 679x127 mem=2 + DL7 [104,1851,923,2566] 819x715 mem=3 + H74 [981,1883,2350,3060] 1369x1177 mem=4 + H72 [2408,1884,3227,2713] 819x829 mem=2 + DL5 [3292,2167,4079,2759] 787x592 mem=2 + DL23 [104,2596,922,2998] 818x402 mem=2 + H58 [2408,2772,3226,3479] 818x707 mem=2 + DL13 [3292,2787,4079,3333] 787x546 mem=2 + DL20 [106,3050,923,3478] 817x428 mem=2 + H60 [981,3117,2351,3479] 1370x362 mem=3 + DL1 [3293,3382,4079,4190] 786x808 mem=2 + P67 [131,3486,599,5211] 468x1725 mem=4 + DL10 [1559,3528,3226,4291] 1667x763 mem=4 + P63 [677,3558,1499,4144] 822x586 mem=2 + H50 [678,4170,1497,4966] 819x796 mem=2 + DL12 [3292,4217,4079,4815] 787x598 mem=2 + H71 [1560,4365,3226,4958] 1666x593 mem=3 + H51 [3292,4842,4079,5479] 787x637 mem=2 + DL41 [1561,4960,3225,5110] 1664x150 mem=2 + H54 [678,4987,1498,5443] 820x456 mem=2 + H76 [1559,5171,3228,5881] 1669x710 mem=5 + H53 [677,5467,1498,5964] 821x497 mem=2 + H83 [3291,5509,4079,6303] 788x794 mem=3 + H56 [1559,5944,3228,6302] 1669x358 mem=3 + H46 [680,5993,1498,6302] 818x309 mem=2 + O79 [152,6141,590,6257] 438x116 mem=1 + 🖼 Orphan photo: R90 → article R67 (an image must belong to an article) +=== page 2: 31 crops, 30 datelines === + H80 [1833,465,3228,1814] 1395x1349 mem=6 + H65 [3293,465,4079,987] 786x522 mem=2 + H84 [106,501,1776,1820] 1670x1319 mem=6 + H43 [3292,1014,4079,1562] 787x548 mem=2 + DL9 [3292,1590,4079,2136] 787x546 mem=2 + H78 [1838,1678,2517,1805] 679x127 mem=2 + DL7 [104,1851,923,2566] 819x715 mem=3 + H74 [981,1883,2350,3060] 1369x1177 mem=4 + H72 [2408,1884,3227,2713] 819x829 mem=2 + DL5 [3292,2167,4079,2759] 787x592 mem=2 + DL23 [104,2596,922,2998] 818x402 mem=2 + H58 [2408,2772,3226,3479] 818x707 mem=2 + DL13 [3292,2787,4079,3333] 787x546 mem=2 + DL20 [106,3050,923,3478] 817x428 mem=2 + H60 [981,3117,2351,3479] 1370x362 mem=3 + DL1 [3293,3382,4079,4190] 786x808 mem=2 + P67 [131,3486,599,5211] 468x1725 mem=4 + DL10 [1559,3528,3226,4291] 1667x763 mem=4 + P63 [677,3558,1499,4144] 822x586 mem=2 + H50 [678,4170,1497,4966] 819x796 mem=2 + DL12 [3292,4217,4079,4815] 787x598 mem=2 + H71 [1560,4365,3226,4958] 1666x593 mem=3 + H51 [3292,4842,4079,5479] 787x637 mem=2 + DL41 [1561,4960,3225,5110] 1664x150 mem=2 + H54 [678,4987,1498,5443] 820x456 mem=2 + H76 [1559,5171,3228,5881] 1669x710 mem=5 + H53 [677,5467,1498,5964] 821x497 mem=2 + H83 [3291,5509,4079,6303] 788x794 mem=3 + H56 [1559,5944,3228,6302] 1669x358 mem=3 + H46 [680,5993,1498,6302] 818x309 mem=2 + O79 [152,6141,590,6257] 438x116 mem=1 + 🖼 Orphan photo: R90 → article R67 (an image must belong to an article) +=== page 2: 31 crops, 30 datelines === + H80 [1833,465,3228,1814] 1395x1349 mem=6 + H65 [3293,465,4079,987] 786x522 mem=2 + H84 [106,501,1776,1820] 1670x1319 mem=6 + H43 [3292,1014,4079,1562] 787x548 mem=2 + DL9 [3292,1590,4079,2136] 787x546 mem=2 + H78 [1838,1678,2517,1805] 679x127 mem=2 + DL7 [104,1851,923,2566] 819x715 mem=3 + H74 [981,1883,2350,3060] 1369x1177 mem=4 + H72 [2408,1884,3227,2713] 819x829 mem=2 + DL5 [3292,2167,4079,2759] 787x592 mem=2 + DL23 [104,2596,922,2998] 818x402 mem=2 + H58 [2408,2772,3226,3479] 818x707 mem=2 + DL13 [3292,2787,4079,3333] 787x546 mem=2 + DL20 [106,3050,923,3478] 817x428 mem=2 + H60 [981,3117,2351,3479] 1370x362 mem=3 + DL1 [3293,3382,4079,4190] 786x808 mem=2 + P67 [131,3486,599,5211] 468x1725 mem=4 + DL10 [1559,3528,3226,4291] 1667x763 mem=4 + P63 [677,3558,1499,4144] 822x586 mem=2 + H50 [678,4170,1497,4966] 819x796 mem=2 + DL12 [3292,4217,4079,4815] 787x598 mem=2 + H71 [1560,4365,3226,4958] 1666x593 mem=3 + H51 [3292,4842,4079,5479] 787x637 mem=2 + DL41 [1561,4960,3225,5110] 1664x150 mem=2 + H54 [678,4987,1498,5443] 820x456 mem=2 + H76 [1559,5171,3228,5881] 1669x710 mem=5 + H53 [677,5467,1498,5964] 821x497 mem=2 + H83 [3291,5509,4079,6303] 788x794 mem=3 + H56 [1559,5944,3228,6302] 1669x358 mem=3 + H46 [680,5993,1498,6302] 818x309 mem=2 + O79 [152,6141,590,6257] 438x116 mem=1 + 🖼 Orphan photo: R90 → article R67 (an image must belong to an article) +=== page 2: 31 crops, 30 datelines === + H80 [1833,465,3228,1814] 1395x1349 mem=6 + H65 [3293,465,4079,987] 786x522 mem=2 + H84 [106,501,1776,1820] 1670x1319 mem=6 + H43 [3292,1014,4079,1562] 787x548 mem=2 + DL9 [3292,1590,4079,2136] 787x546 mem=2 + H78 [1838,1678,2517,1805] 679x127 mem=2 + DL7 [104,1851,923,2566] 819x715 mem=3 + H74 [981,1883,2350,3060] 1369x1177 mem=4 + H72 [2408,1884,3227,2713] 819x829 mem=2 + DL5 [3292,2167,4079,2759] 787x592 mem=2 + DL23 [104,2596,922,2998] 818x402 mem=2 + H58 [2408,2772,3226,3479] 818x707 mem=2 + DL13 [3292,2787,4079,3333] 787x546 mem=2 + DL20 [106,3050,923,3478] 817x428 mem=2 + H60 [981,3117,2351,3479] 1370x362 mem=3 + DL1 [3293,3382,4079,4190] 786x808 mem=2 + P67 [131,3486,599,5211] 468x1725 mem=4 + DL10 [1559,3528,3226,4291] 1667x763 mem=4 + P63 [677,3558,1499,4144] 822x586 mem=2 + H50 [678,4170,1497,4966] 819x796 mem=2 + DL12 [3292,4217,4079,4815] 787x598 mem=2 + H71 [1560,4365,3226,4958] 1666x593 mem=3 + H51 [3292,4842,4079,5479] 787x637 mem=2 + DL41 [1561,4960,3225,5110] 1664x150 mem=2 + H54 [678,4987,1498,5443] 820x456 mem=2 + H76 [1559,5171,3228,5881] 1669x710 mem=5 + H53 [677,5467,1498,5964] 821x497 mem=2 + H83 [3291,5509,4079,6303] 788x794 mem=3 + H56 [1559,5944,3228,6302] 1669x358 mem=3 + H46 [680,5993,1498,6302] 818x309 mem=2 + O79 [152,6141,590,6257] 438x116 mem=1 + 🖼 Orphan photo: R90 → article R67 (an image must belong to an article) +DONE -> output/NamastheTelangana_Adilabad_20260528_20260610_160056/all_article_crops_review diff --git a/_orphan_check.py b/_orphan_check.py new file mode 100644 index 0000000..e1a3647 --- /dev/null +++ b/_orphan_check.py @@ -0,0 +1,21 @@ +import sys, json, contextlib, io +sys.path.insert(0, ".") +from pathlib import Path +import smart_extractor as se +from _lines import separator_barriers + +D = Path("output/AndhraJyothi_Siddipet District_20260602_20260603_144919/pages") +for pg in (2, 3, 4): + png = D / f"page_{pg:03d}.png" + regs = json.loads((D / f"page_{pg:03d}.regions.json").read_text())["regions"] + with contextlib.redirect_stdout(io.StringIO()): + regs = se.drop_masthead_regions(regs, "andhra_jyothi", pg, png) + ds = se.find_article_starts_by_dateline(regs, str(png), "andhra_jyothi") + bars = separator_barriers(str(png), regs) + bl = se.cluster_all_article_blocks(regs, dateline_starts=ds, sep_lines=bars) + dl_ids = {s["region_id"] for s in ds} + n_dl = sum(1 for b in bl if any(m in dl_ids for m in b["members"])) + orph = [b for b in bl if b["kind"] == "O"] + print(f"==== PAGE {pg}: {len(bl)} blocks, {n_dl} dateline, {len(orph)} orphan ====") + for b in orph: + print(f" O anchor={b['anchor_id']} bbox={b['bbox']} members={sorted(b['members'])}") diff --git a/_partition_debug.py b/_partition_debug.py new file mode 100644 index 0000000..e36aee5 --- /dev/null +++ b/_partition_debug.py @@ -0,0 +1,86 @@ +"""TEST (read-only): finalize article boundaries using the RIGHTMOST dead-end (nearest big +vertical wall to the right, else page edge) and the TOPMOST dead-end (masthead bottom, for +top-row articles). Draw both the original block boundary (thin gray) and the dead-end-snapped +boundary (thick colour) on a blank canvas, plus the big vertical walls (red) and the masthead +band. No smart_extractor change, no API. Test on 1-2 pages.""" +import os, tempfile, json, io, contextlib, statistics +tempfile.tempdir = os.path.abspath("_tess_tmp"); os.environ["TMPDIR"] = tempfile.tempdir +os.makedirs("_tess_tmp", exist_ok=True) +from pathlib import Path +from PIL import Image, ImageDraw, ImageFont +import smart_extractor as se +from _lines import separator_barriers, detect_separator_lines + +BIG_FRAC = 0.45 + +def big_walls(png): + res = detect_separator_lines(png); W, H = res["size"] + out = [] + for L in res["v"]: + y1, y2 = min(L["y1"], L["y2"]), max(L["y1"], L["y2"]) + if (y2 - y1) >= BIG_FRAC * H: + out.append({"x": (L["x1"]+L["x2"])//2, "y1": y1, "y2": y2}) + return out, (W, H) + +def masthead_bottom(png, raw, paper, pg, W, H): + # compute from RAW regions (before masthead removal) so the dropped band's bottom is visible + with contextlib.redirect_stdout(io.StringIO()): + dd = se.dedup_overlapping_regions([dict(r) for r in raw]) + kept = se.drop_masthead_regions([dict(r) for r in dd], paper, pg, png) + drop_bottom = max((r["bbox"][3] for r in dd + if r["id"] not in {k["id"] for k in kept}), default=0) + res = detect_separator_lines(png) + wide = sorted([L for L in res["h"] if (L["x2"]-L["x1"]) >= 0.30*W + and L["y1"] < 0.20*H], key=lambda L: L["y1"]) + top_rule = wide[0]["y1"] if wide else 0 + return max(drop_bottom, top_rule) + +def run(pdir, pg, paper, tag): + png = f"{pdir}/page_{pg:03d}.png" + raw = json.loads(Path(f"{pdir}/page_{pg:03d}.regions.json").read_text())["regions"] + with contextlib.redirect_stdout(io.StringIO()): + regs = se.dedup_overlapping_regions(raw); regs = se.merge_stacked_title_lines(regs, png) + regs = se.drop_masthead_regions(regs, paper, pg, png); regs = se.promote_paragraph_titles(regs, png, paper); regs = se.mark_bullet_titles(regs, png) + ds = se.find_article_starts_by_dateline(regs, png, paper) + _p = se.cluster_all_article_blocks(regs, ds, sep_lines=separator_barriers(png, regs)) + if se.recover_orphan_text_headlines(_p, regs, png): ds = se.find_article_starts_by_dateline(regs, png, paper) + bars = separator_barriers(png, regs) + blocks = se.cluster_all_article_blocks(regs, ds, sep_lines=bars) + walls, (W, H) = big_walls(png) + cright = max((r["bbox"][2] for r in regs), default=W) + mh = masthead_bottom(png, raw, paper, pg, W, H) + top_row_cut = mh + 320 # an article whose top is within this band is "top row" + + def deadends(b): + x1, y1, x2, y2 = b["bbox"]; cx = (x1 + x2) / 2; bh = y2 - y1 + # rightmost dead-end: nearest big wall right of the block centre that spans it; else page edge + rcand = [w["x"] for w in walls if w["x"] > cx + and max(0, min(y2, w["y2"]) - max(y1, w["y1"])) >= 0.4 * bh] + right = min(rcand) if rcand else cright + right = max(right, x2) # never shrink + # topmost dead-end: top-row article → up to masthead bottom + top = mh if (y1 <= top_row_cut and mh > 0) else y1 + top = min(top, y1) # never shrink downward + return [x1, top, right, y2] + + img = Image.new("RGB", (W, H), "white"); d = ImageDraw.Draw(img) + try: font = ImageFont.truetype("DejaVuSans-Bold.ttf", 30) + except Exception: font = ImageFont.load_default() + if mh > 4: + d.rectangle([0, 0, W, mh], fill=(232,232,232)); d.line([0,mh,W,mh], fill=(140,140,140), width=5) + d.text((14, max(2, mh//2-16)), "MASTHEAD (top dead-end)", fill=(110,110,110), font=font) + for w in walls: + d.line([w["x"], w["y1"], w["x"], w["y2"]], fill=(235,70,70), width=8) + pal = se._ARTICLE_COLORS + for i, b in enumerate(blocks): + ox1, oy1, ox2, oy2 = b["bbox"]; nx1, ny1, nx2, ny2 = deadends(b); c = pal[i % len(pal)] + d.rectangle([ox1, oy1, ox2, oy2], outline=(180,180,180), width=3) # original (gray) + d.rectangle([nx1, ny1, nx2, ny2], outline=c, width=8) # dead-end (color) + lab = f"#{b['anchor_id']} [{b['kind']}]" + d.rectangle([nx1+5, ny1+6, nx1+16+len(lab)*15, ny1+42], fill=c) + d.text((nx1+11, ny1+9), lab, fill="white", font=font) + out = f"{pdir}/page_{pg:03d}_partition_debug.png"; img.save(out) + print(f"{tag} pg{pg}: {len(blocks)} blocks, {len(walls)} walls, masthead_bottom={int(mh)} -> {out}") + +run("output/NamastheTelangana_Siddipet_20260602_20260604_214937/pages", 2, "namaste_telangana", "NT") +run("output/AndhraJyothi_Siddipet District_20260602_20260603_144919/pages", 1, "andhra_jyothi", "AJ") diff --git a/_probe71_tmp.py b/_probe71_tmp.py new file mode 100644 index 0000000..fc9652f --- /dev/null +++ b/_probe71_tmp.py @@ -0,0 +1,26 @@ +import sys, types; sys.modules.setdefault("fitz", types.ModuleType("fitz")) +import os, tempfile, json +tempfile.tempdir = os.path.abspath("_tess_tmp"); os.environ["TMPDIR"]=tempfile.tempdir +os.makedirs("_tess_tmp", exist_ok=True) +from pathlib import Path +import smart_extractor as se +from _lines import separator_barriers +RUN = Path("output/Sakshi_Jangaon_District_20260520_20260609_210104") +png = str(RUN/"pages/page_001.png") +regions = json.loads((RUN/"pages/page_001.regions.json").read_text())["regions"] +regions = se.dedup_overlapping_regions(regions) +regions = se.merge_stacked_title_lines(regions, png) +regions = se.drop_masthead_regions(regions, "sakshi", 1, png) +regions = se.promote_paragraph_titles(regions, png, "sakshi") +dls = se.find_article_starts_by_dateline(regions, png, "sakshi") +seps = separator_barriers(png, regions) +blocks = se.cluster_all_article_blocks(regions, dls, sep_lines=seps) +rmap = {r['id']:r for r in regions} +for b in blocks: + if b['anchor_id']==71: + print("BLOCK:", {k:v for k,v in b.items() if k!='members'}) + print("members:") + for m in sorted(b['members']+[b['anchor_id']], key=lambda m: rmap[m]['bbox'][1] if m in rmap else 0): + r = rmap.get(m) + print(" ", m, r['type'] if r else '?', r['bbox'] if r else '?') +json.dump({'blocks':[{'anchor':b['anchor_id'],'kind':b['kind'],'bbox':b['bbox'],'members':b['members']} for b in blocks]}, open('/tmp/blocks71.json','w')) diff --git a/_probe_above.py b/_probe_above.py new file mode 100644 index 0000000..3e19e45 --- /dev/null +++ b/_probe_above.py @@ -0,0 +1,29 @@ +import sys, json +sys.path.insert(0, "/Users/kranthikumar/Documents/telugu_extractor") +from pathlib import Path +import smart_extractor as se +from _lines import separator_barriers + +D = Path("output/AndhraJyothi_Siddipet District_20260602_20260603_144919/pages") +for pg in (1, 2, 3, 4): + png = D / f"page_{pg:03d}.png" + regions = json.loads((D / f"page_{pg:03d}.regions.json").read_text())["regions"] + regions = se.drop_masthead_regions(regions, "andhra_jyothi", pg, png) + ds = se.find_article_starts_by_dateline(regions, str(png), "andhra_jyothi") + bars = separator_barriers(str(png), regions) + blocks = se.cluster_all_article_blocks(regions, dateline_starts=ds, sep_lines=bars) + rmap = {r["id"]: r for r in regions} + print(f"\n==== PAGE {pg} ====") + for b in blocks: + anchor = rmap.get(b["anchor_id"]) + if not anchor: + continue + atop = anchor["bbox"][1] + # members whose ENTIRE box is above the anchor headline top + above_mems = [m for m in b["members"] + if m != b["anchor_id"] and rmap[m]["bbox"][3] <= atop + 20] + if above_mems and b["kind"] == "H": + print(f" block H anchor={b['anchor_id']} (top y={atop}) has {len(above_mems)} member(s) ABOVE it:") + for m in above_mems: + rb = rmap[m]["bbox"] + print(f" R{m} {rmap[m]['type']:<14} y[{rb[1]}..{rb[3]}] x[{rb[0]}..{rb[2]}]") diff --git a/_probe_bars.py b/_probe_bars.py new file mode 100644 index 0000000..6b19295 --- /dev/null +++ b/_probe_bars.py @@ -0,0 +1,21 @@ +import sys, json, contextlib, io +sys.path.insert(0, ".") +from pathlib import Path +import smart_extractor as se +from _lines import separator_barriers, detect_separator_lines + +D = Path("output/AndhraJyothi_Siddipet District_20260602_20260603_144919/pages") +png = D / "page_001.png" +regs0 = json.loads((D / "page_001.regions.json").read_text())["regions"] +with contextlib.redirect_stdout(io.StringIO()): + regs = se.drop_masthead_regions(regs0, "andhra_jyothi", 1, png) + +res = detect_separator_lines(png) +print(f"=== {len(res['h'])} raw horizontal lines ===") +for L in sorted(res["h"], key=lambda d: d["y1"]): + print(f" H y={L['y1']:>5} x[{L['x1']:>4}..{L['x2']:>4}] len={L['len']:>4} gray={L['gray']}") + +print("\n=== separator_barriers (after filters) ===") +bars = separator_barriers(str(png), regs) +for L in sorted(bars, key=lambda d: d["y"]): + print(f" bar y={L['y']:>5} x[{L['x1']:>4}..{L['x2']:>4}] mc={L.get('mc')}") diff --git a/_probe_dl.py b/_probe_dl.py new file mode 100644 index 0000000..a91f98b --- /dev/null +++ b/_probe_dl.py @@ -0,0 +1,22 @@ +import sys, json, contextlib, io +sys.path.insert(0, ".") +from pathlib import Path +import smart_extractor as se +from _lines import separator_barriers, detect_separator_lines + +D = Path("output/AndhraJyothi_Siddipet District_20260602_20260603_144919/pages") +png = D / "page_001.png" +regs0 = json.loads((D / "page_001.regions.json").read_text())["regions"] +with contextlib.redirect_stdout(io.StringIO()): + regs = se.drop_masthead_regions(regs0, "andhra_jyothi", 1, png) + ds = se.find_article_starts_by_dateline(regs, str(png), "andhra_jyothi") + +print(f"=== {len(ds)} dateline starts ===") +for s in sorted(ds, key=lambda s: s["bbox"][1]): + print(f" DL region={s['region_id']:>3} bbox={s['bbox']} headline={s.get('headline_region')}") + +byid = {r["id"]: r for r in regs} +print("\n=== all regions sorted by y ===") +for r in sorted(regs, key=lambda r: (r["bbox"][1], r["bbox"][0])): + b = r["bbox"] + print(f" R{r['id']:>3} {r['type']:<16} bbox={b}") diff --git a/_probe_fake.py b/_probe_fake.py new file mode 100644 index 0000000..8ceee91 --- /dev/null +++ b/_probe_fake.py @@ -0,0 +1,38 @@ +import sys, json, contextlib, io +sys.path.insert(0, ".") +from pathlib import Path +import smart_extractor as se +from _lines import separator_barriers, detect_separator_lines + +D = Path("output/AndhraJyothi_Siddipet District_20260602_20260603_144919/pages") +png = D / "page_001.png" +regs0 = json.loads((D / "page_001.regions.json").read_text())["regions"] +with contextlib.redirect_stdout(io.StringIO()): + regs = se.drop_masthead_regions(regs0, "andhra_jyothi", 1, png) + +byid = {r["id"]: r for r in regs} +# focus region 35 and 55 +for rid in (35, 55): + if rid in byid: + r = byid[rid] + print(f"R{rid} type={r['type']} bbox={r['bbox']}") + else: + print(f"R{rid} NOT in regs (dropped?)") + +print("\n-- all regions near x 150..750, y 1100..1600 --") +for r in sorted(regs, key=lambda r: r["bbox"][1]): + b = r["bbox"] + if b[1] < 1700 and b[3] > 1050 and b[0] < 760: + print(f" R{r['id']:>3} {r['type']:<16} bbox={b}") + +print("\n-- raw horizontal lines near y=1375 --") +res = detect_separator_lines(png) +for L in sorted(res["h"], key=lambda d: d["y1"]): + if 1300 < L["y1"] < 1450: + print(f" H y={L['y1']} x[{L['x1']}..{L['x2']}] len={L['len']} gray={L['gray']}") + +print("\n-- separator_barriers near y=1375 --") +bars = separator_barriers(str(png), regs) +for L in sorted(bars, key=lambda d: d["y"]): + if 1300 < L["y"] < 1450: + print(f" bar y={L['y']} x[{L['x1']}..{L['x2']}] mc={L.get('mc')}") diff --git a/_probe_fake2.py b/_probe_fake2.py new file mode 100644 index 0000000..0f1686a --- /dev/null +++ b/_probe_fake2.py @@ -0,0 +1,21 @@ +import sys, json, contextlib, io +sys.path.insert(0, ".") +from pathlib import Path +import smart_extractor as se +from _lines import separator_barriers + +D = Path("output/AndhraJyothi_Siddipet District_20260602_20260603_144919/pages") +png = D / "page_001.png" +regs0 = json.loads((D / "page_001.regions.json").read_text())["regions"] +with contextlib.redirect_stdout(io.StringIO()): + regs = se.drop_masthead_regions(regs0, "andhra_jyothi", 1, png) + ds = se.find_article_starts_by_dateline(regs, str(png), "andhra_jyothi") + bars = separator_barriers(str(png), regs) + bl = se.cluster_all_article_blocks(regs, dateline_starts=ds, sep_lines=bars) + +print("bars near y=1375:", [b for b in bars if 1300 < b["y"] < 1450]) +print() +dl_ids = {s["region_id"] for s in ds} +for b in sorted(bl, key=lambda b: (b["bbox"][1], b["bbox"][0])): + kind = b["kind"] + print(f" {kind} anchor={b['anchor_id']:>3} bbox={b['bbox']} members={sorted(b['members'])}") diff --git a/_probe_fix.py b/_probe_fix.py new file mode 100644 index 0000000..197045a --- /dev/null +++ b/_probe_fix.py @@ -0,0 +1,41 @@ +import os, tempfile, contextlib, io, json +tempfile.tempdir = os.path.abspath("_tess_tmp"); os.environ["TMPDIR"] = tempfile.tempdir +os.makedirs("_tess_tmp", exist_ok=True) +from pathlib import Path +import smart_extractor as se +from _lines import separator_barriers +se._ACTIVE_PAPER = "sakshi" +RUN = Path("output/Sakshi_Jangaon_District_20260520_20260609_210104"); D = RUN / "pages" +png = str(D / "page_001.png") +regs = json.loads((D / "page_001.regions.json").read_text())["regions"] +buf = io.StringIO() +with contextlib.redirect_stdout(buf): + regs = se.dedup_overlapping_regions(regs); regs = se.merge_stacked_title_lines(regs, png) + regs = se.drop_masthead_regions(regs, "sakshi", 1, png) + regs = se.drop_classifieds_regions(regs, png); regs = se.tag_legal_notices(regs, png) + regs = se.promote_paragraph_titles(regs, png, "sakshi") + regs = se.detect_sakshi_headline_bands(regs, png) + regs = se.mark_bullet_titles(regs, png) + ds = se.find_article_starts_by_dateline(regs, png, "sakshi") + bars = separator_barriers(png, regs) + _p = se.cluster_all_article_blocks(regs, ds, sep_lines=bars) + if se.recover_orphan_text_headlines(_p, regs, png): + ds = se.find_article_starts_by_dateline(regs, png, "sakshi") + blocks = se.cluster_all_article_blocks(regs, ds, sep_lines=bars) +WATCH = [44, 16, 36, 69, 61, 46, 23, 4, 74, 49, 17, 88, 33, 28, 3, 53, 56] +own = {} +for b in blocks: + for m in set(b["members"]) | {b["anchor_id"]}: + own.setdefault(m, []).append(f"{b['kind']}{b['anchor_id']}") +print("region owners:") +for w in WATCH: + print(f" R{w:>3} -> {own.get(w, ['(NONE)'])}") +print("\nhorizontal bars near lift-1 (y3300-4100) and R50 zone (y4350-4900):") +for L in bars: + if not L.get("vert") and "y" in L and (3300 <= L["y"] <= 4100 or 4350 <= L["y"] <= 4900): + print(f" y={L['y']} x={L['x1']}-{L['x2']} w={L['x2']-L['x1']}") +print("\ndateline regions:", sorted(s["region_id"] for s in ds)) +print("\npipeline lines mentioning watched regions / passes:") +for l in buf.getvalue().splitlines(): + if any(k in l for k in ("R44", "R69", "R61", "R46", "R23", "rebind", "Sakshi", "Orphan photo", "Masthead", "📐")): + print(" ", l.strip()) diff --git a/_probe_fix2.py b/_probe_fix2.py new file mode 100644 index 0000000..47d41f4 --- /dev/null +++ b/_probe_fix2.py @@ -0,0 +1,46 @@ +import os, tempfile, contextlib, io, json +tempfile.tempdir = os.path.abspath("_tess_tmp"); os.environ["TMPDIR"] = tempfile.tempdir +os.makedirs("_tess_tmp", exist_ok=True) +from pathlib import Path +import smart_extractor as se +from _lines import separator_barriers +se._ACTIVE_PAPER = "sakshi" +RUN = Path("output/Sakshi_Jangaon_District_20260520_20260609_210104"); D = RUN / "pages" +png = str(D / "page_002.png") +regs = json.loads((D / "page_002.regions.json").read_text())["regions"] +buf = io.StringIO() +with contextlib.redirect_stdout(buf): + regs = se.dedup_overlapping_regions(regs); regs = se.merge_stacked_title_lines(regs, png) + regs = se.drop_masthead_regions(regs, "sakshi", 2, png) + regs = se.drop_classifieds_regions(regs, png); regs = se.tag_legal_notices(regs, png) + regs = se.promote_paragraph_titles(regs, png, "sakshi") + regs = se.detect_sakshi_headline_bands(regs, png) + regs = se.mark_bullet_titles(regs, png) + ds = se.find_article_starts_by_dateline(regs, png, "sakshi") + bars = separator_barriers(png, regs) + _p = se.cluster_all_article_blocks(regs, ds, sep_lines=bars) + if se.recover_orphan_text_headlines(_p, regs, png): + ds = se.find_article_starts_by_dateline(regs, png, "sakshi") + blocks = se.cluster_all_article_blocks(regs, ds, sep_lines=bars) +rm = {r["id"]: r for r in regs} +print("BLOCKS:") +for b in sorted(blocks, key=lambda b: (b["bbox"][1], b["bbox"][0])): + x1, y1, x2, y2 = b["bbox"] + f = " FURN" if b.get("_furniture") else "" + print(f" {b['kind']}{b['anchor_id']:<4} [{x1},{y1},{x2},{y2}] mem={sorted(set(b['members']))}{f}") +WATCH = [20, 29, 34, 51, 3, 14, 31, 65, 87, 105, 121, 76, 8, 10, 36, 46, 81] +own = {} +for b in blocks: + tag = ("F" if b.get("_furniture") else b["kind"]) + str(b["anchor_id"]) + for m in set(b["members"]) | {b["anchor_id"]}: + own.setdefault(m, []).append(tag) +print("\nWATCH owners:", {f"R{w}": own.get(w, "NONE") for w in WATCH}) +print("\nbars y2380-2820 / y3400-3700 / y4640-4760:") +for L in bars: + if not L.get("vert") and "y" in L and (2380 <= L["y"] <= 2820 or 3400 <= L["y"] <= 3700 or 4640 <= L["y"] <= 4760): + print(f" y={L['y']} x={L['x1']}-{L['x2']} w={L['x2']-L['x1']}") +print("\ndatelines:", sorted(s["region_id"] for s in ds)) +print("\npipeline lines:") +for l in buf.getvalue().splitlines(): + if any(k in l for k in ("Type-2", "rule-split", "📐", "Masthead", "⇤", "re-home", "📸", "Orphan photo", "carve", "furniture")): + print(" ", l.strip()) diff --git a/_r121.txt b/_r121.txt new file mode 100644 index 0000000..406bd44 --- /dev/null +++ b/_r121.txt @@ -0,0 +1,9 @@ +R121 bbox=[2113, 2214, 3502, 3634] kind=H raw=True + R12 text y[1996,2719] x[2979,3480] DATELINE + R39 image y[2214,2622] x[2113,2939] + R111 figure_title y[2628,2725] x[2196,2856] + R121 doc_title y[2799,2898] x[2422,3493] + R59 image y[2902,3115] x[2981,3502] + R21 text y[2907,3621] x[2419,2945] DATELINE + R116 figure_title y[3123,3163] x[3113,3376] + R14 text y[3195,3622] x[2977,3502] diff --git a/_r65.txt b/_r65.txt new file mode 100644 index 0000000..65212aa --- /dev/null +++ b/_r65.txt @@ -0,0 +1,20 @@ +page WxH=4200x6422; 12 blocks + +R65 bbox=[113, 1958, 1977, 6279] kind=H raw=True locked=False #members=36 +R65 datelines: [43, 47, 26, 21] +vertical walls: [(616, 4683, 6292), (141, 4683, 6292), (123, 4123, 6309), (637, 4122, 6309), (2015, 1949, 5689), (3300, 1929, 6308), (2669, 404, 1905)] +wide H-seps (>0.5W): [(1936, 114, 4091), (5709, 686, 3276)] + +all block anchors+bbox: + R109 H bbox=[119, 393, 612, 1899] ndl=1 + R79 H bbox=[2693, 405, 4087, 1833] ndl=1 + R55 H bbox=[679, 415, 2630, 1919] ndl=1 + R77 H bbox=[2692, 1594, 4087, 1903] ndl=1 + R14 DL bbox=[2045, 1950, 3265, 3097] ndl=1 + R65 H bbox=[113, 1958, 1977, 6279] ndl=4 + R96 H bbox=[3326, 1964, 4087, 3638] ndl=1 + R64 H bbox=[2046, 3116, 3264, 4832] ndl=2 + R97 H bbox=[3322, 3756, 4087, 5831] ndl=1 + R110 H bbox=[2044, 4858, 3265, 5685] ndl=1 + R59 H bbox=[680, 5731, 3267, 6318] ndl=1 + R7 DL bbox=[3308, 5839, 4087, 6305] ndl=1 diff --git a/_rSAK.txt b/_rSAK.txt new file mode 100644 index 0000000..e69de29 diff --git a/_replay_adilabad.py b/_replay_adilabad.py new file mode 100644 index 0000000..52b443a --- /dev/null +++ b/_replay_adilabad.py @@ -0,0 +1,39 @@ +import os, tempfile, contextlib, io, json, shutil +tempfile.tempdir = os.path.abspath("_tess_tmp"); os.environ["TMPDIR"] = tempfile.tempdir +os.makedirs("_tess_tmp", exist_ok=True) +from pathlib import Path +import smart_extractor as se +from _lines import separator_barriers +se._ACTIVE_PAPER = "namaste_telangana" +RUN = Path(os.environ.get("NTA_RUN", "output/NamastheTelangana_Adilabad_20260528_20260610_160056")) +D = RUN / "pages"; P = "namaste_telangana" +OUT = RUN / "all_article_crops_review" +if OUT.exists(): shutil.rmtree(OUT) +pages = sorted(int(p.stem.split("_")[1]) for p in D.glob("page_*.png")) +for pg in pages: + png = str(D / f"page_{pg:03d}.png") + regs = json.loads((D / f"page_{pg:03d}.regions.json").read_text())["regions"] + buf = io.StringIO() + with contextlib.redirect_stdout(buf): + regs = se.dedup_overlapping_regions(regs); regs = se.merge_stacked_title_lines(regs, png) + regs = se.drop_masthead_regions(regs, P, pg, png); regs = se.drop_classifieds_regions(regs, png) + regs = se.tag_legal_notices(regs, png); regs = se.promote_paragraph_titles(regs, png, P) + if pg == 1: regs = se.detect_sakshi_headline_bands(regs, png) # NT: front-page only + regs = se.mark_bullet_titles(regs, png) + ds = se.find_article_starts_by_dateline(regs, png, P) + bars = separator_barriers(png, regs) + _p = se.cluster_all_article_blocks(regs, ds, sep_lines=bars) + if se.recover_orphan_text_headlines(_p, regs, png): + ds = se.find_article_starts_by_dateline(regs, png, P) + blocks = se.cluster_all_article_blocks(regs, ds, sep_lines=bars) + n = se.crop_all_article_blocks(png, regs, str(OUT), pg, dateline_starts=ds, + sep_lines=bars, overlay=True) + print(f"=== page {pg}: {n} crops, {len(ds)} datelines ===") + for b in sorted(blocks, key=lambda b: (b["bbox"][1], b["bbox"][0])): + x1, y1, x2, y2 = b["bbox"] + f = " FURN" if b.get("_furniture") else "" + print(f" {b['kind']}{b['anchor_id']:<4} [{x1},{y1},{x2},{y2}] {x2-x1}x{y2-y1} mem={len(b['members'])}{f}") + for l in sorted({l.strip() for l in buf.getvalue().splitlines() + if "rule-split" in l or "Masthead" in l or "Orphan photo" in l}): + print(" ", l) +print("DONE ->", OUT) diff --git a/_replay_all.py b/_replay_all.py new file mode 100644 index 0000000..ab14e2f --- /dev/null +++ b/_replay_all.py @@ -0,0 +1,35 @@ +import os,tempfile,contextlib,io,json,shutil +tempfile.tempdir=os.path.abspath("_tess_tmp"); os.environ["TMPDIR"]=tempfile.tempdir +os.makedirs("_tess_tmp",exist_ok=True) +from pathlib import Path +import smart_extractor as se +from _lines import separator_barriers +RUN=Path("output/AndhraJyothi_Siddipet District_20260602_20260603_144919"); D=RUN/"pages" +OUT=RUN/"all_article_crops_final"; PAPER="andhra_jyothi" +if OUT.exists(): shutil.rmtree(OUT) +for pg in (1,2,3,4): + png=str(D/f"page_{pg:03d}.png") + regs=json.loads((D/f"page_{pg:03d}.regions.json").read_text())["regions"] # 0.5 saved + buf=io.StringIO() + with contextlib.redirect_stdout(buf): + regs=se.dedup_overlapping_regions(regs) + regs=se.merge_stacked_title_lines(regs,png) + regs=se.drop_masthead_regions(regs,PAPER,pg,png) + regs=se.drop_classifieds_regions(regs,png) + regs=se.tag_legal_notices(regs,png) + regs=se.promote_paragraph_titles(regs,png,PAPER) + regs=se.mark_bullet_titles(regs,png) + bars=separator_barriers(png,regs) + ds=se.find_article_starts_by_dateline(regs,png,PAPER) + _probe=se.cluster_all_article_blocks(regs,ds,sep_lines=bars) + if se.recover_orphan_text_headlines(_probe,regs,png): ds=se.find_article_starts_by_dateline(regs,png,PAPER) + blocks=se.cluster_all_article_blocks(regs,ds,sep_lines=bars) + se.draw_boundary_map_debug(png,regs,D,pg,dateline_starts=ds,sep_lines=bars) + n=se.crop_all_article_blocks(png,regs,str(OUT),pg,dateline_starts=ds,sep_lines=bars) + log=buf.getvalue() + tags={k:sum(1 for l in log.splitlines() if k in l) for k in + ["bullet titles flagged","Headline-strip","Floor-trim","Block dateline split", + "Multi-column body","orphan overlap"]} + tags={k:v//2 if k in ("Headline-strip","Floor-trim","Multi-column body","Block dateline split","orphan overlap") else v for k,v in tags.items()} + print(f"page {pg}: {n} crops, {len(ds)} datelines | "+", ".join(f"{k}:{v}" for k,v in tags.items() if v)) +print("DONE ->",OUT) diff --git a/_replay_crop_all.py b/_replay_crop_all.py new file mode 100644 index 0000000..297c331 --- /dev/null +++ b/_replay_crop_all.py @@ -0,0 +1,36 @@ +"""Re-crop EVERY article on all 4 pages using the EXISTING smart_extractor code +(no Claude / no paid API). Runs the FULL pre-processing chain the real pipeline uses +(dedup -> merge stacked titles -> drop masthead -> promote paragraph-titles -> dateline +scan -> separators -> cluster/crop), so the replay crops match real output. Tesseract +temp files are redirected off /tmp so OCR can read them. Output goes to a fresh dir.""" +import os, tempfile, json +tempfile.tempdir = os.path.abspath("_tess_tmp") # keep Tesseract off /tmp +os.environ["TMPDIR"] = tempfile.tempdir +os.makedirs("_tess_tmp", exist_ok=True) + +from pathlib import Path +import smart_extractor as se +from _lines import separator_barriers + +RUN = Path("output/AndhraJyothi_Siddipet District_20260602_20260603_144919") +PAGES = RUN / "pages" +OUT = RUN / "all_article_crops_new" # fresh output dir +PAPER = "andhra_jyothi" + +for pg in (1, 2, 3, 4): + png = str(PAGES / f"page_{pg:03d}.png") + regions = json.loads((PAGES / f"page_{pg:03d}.regions.json").read_text())["regions"] + + # Full pre-processing chain (matches the real pipeline order): + regions = se.dedup_overlapping_regions(regions) + regions = se.merge_stacked_title_lines(regions, png) + regions = se.drop_masthead_regions(regions, PAPER, pg, png) + regions = se.promote_paragraph_titles(regions, png, PAPER) # paragraph_title -> doc_title + dateline_starts = se.find_article_starts_by_dateline(regions, png, PAPER) + sep_lines = separator_barriers(png, regions) + + n = se.crop_all_article_blocks(png, regions, str(OUT), pg, + dateline_starts=dateline_starts, sep_lines=sep_lines) + print(f"page {pg}: {len(dateline_starts)} datelines -> {n} article crop(s) written") + +print(f"\nCrops written under: {OUT}") diff --git a/_replay_dl9.py b/_replay_dl9.py new file mode 100644 index 0000000..004ad4b --- /dev/null +++ b/_replay_dl9.py @@ -0,0 +1,31 @@ +import sys, json, shutil, os +sys.path.insert(0, "/Users/kranthikumar/Documents/telugu_extractor") +from pathlib import Path +import smart_extractor as se +from _lines import separator_barriers + +D = Path("output/AndhraJyothi_Siddipet District_20260602_20260603_144919/pages") +png = D / "page_001.png" +regions = json.loads((D / "page_001.regions.json").read_text())["regions"] +regions2 = se.drop_masthead_regions(regions, "andhra_jyothi", 1, png) +ds = se.find_article_starts_by_dateline(regions2, str(png), "andhra_jyothi") +bars = separator_barriers(str(png), regions2) +blocks = se.cluster_all_article_blocks(regions2, dateline_starts=ds, sep_lines=bars) + +dl_ids = {s["region_id"] for s in ds} +dl_anchors = {b["anchor_id"] for b in blocks if any(m in dl_ids for m in b["members"])} +print(f"{len(ds)} datelines -> {len(dl_anchors)} dateline blocks: {sorted(dl_anchors)}") +for b in sorted(blocks, key=lambda d: (d["bbox"][1], d["bbox"][0])): + if b["anchor_id"] in dl_anchors: + print(f" {b['kind']} anchor={b['anchor_id']:>3} bbox={b['bbox']} members={sorted(b['members'])}") + +out = Path("output/AndhraJyothi_Siddipet District_20260602_20260603_144919/dateline_crops") +if out.exists(): + shutil.rmtree(out) +out.mkdir(parents=True) +n = se.crop_all_article_blocks(str(png), regions2, str(out), 1, + dateline_starts=ds, sep_lines=bars, + only_anchor_ids=dl_anchors) +print(f"\ndateline crops written: {n}") +for f in sorted(os.listdir(out / "page_001")): + print(" ", f) diff --git a/_replay_hstrip.py b/_replay_hstrip.py new file mode 100644 index 0000000..b4113e5 --- /dev/null +++ b/_replay_hstrip.py @@ -0,0 +1,24 @@ +import os,tempfile,contextlib,io,json,shutil +tempfile.tempdir=os.path.abspath("_tess_tmp"); os.environ["TMPDIR"]=tempfile.tempdir +os.makedirs("_tess_tmp",exist_ok=True) +from pathlib import Path +import smart_extractor as se +from _lines import separator_barriers +RUN=Path("output/AndhraJyothi_Siddipet District_20260602_20260603_144919"); D=RUN/"pages" +OUT=RUN/"all_article_crops_hstrip"; PAPER="andhra_jyothi" +if OUT.exists(): shutil.rmtree(OUT) +for pg in (1,2,3,4): + png=str(D/f"page_{pg:03d}.png") + regs=json.loads((D/f"page_{pg:03d}.regions.json").read_text())["regions"] + buf=io.StringIO() + with contextlib.redirect_stdout(buf): + regs=se.dedup_overlapping_regions(regs); regs=se.merge_stacked_title_lines(regs,png) + regs=se.drop_masthead_regions(regs,PAPER,pg,png); regs=se.promote_paragraph_titles(regs,png,PAPER) + bars=separator_barriers(png,regs) + ds=se.find_article_starts_by_dateline(regs,png,PAPER) + blocks=se.cluster_all_article_blocks(regs,ds,sep_lines=bars) + n=se.crop_all_article_blocks(png,regs,str(OUT),pg,dateline_starts=ds,sep_lines=bars) + ext=[l for l in buf.getvalue().splitlines() if "Headline-strip" in l] + print(f"page {pg}: {n} crops | {len(ext)} headline-strip extension(s)") + for l in ext: print(" ",l.strip()) +print("DONE") diff --git a/_replay_jangaon.py b/_replay_jangaon.py new file mode 100644 index 0000000..c26b469 --- /dev/null +++ b/_replay_jangaon.py @@ -0,0 +1,48 @@ +import os,tempfile,contextlib,io,json,shutil +tempfile.tempdir=os.path.abspath("_tess_tmp"); os.environ["TMPDIR"]=tempfile.tempdir +os.makedirs("_tess_tmp",exist_ok=True) +from pathlib import Path +import smart_extractor as se +from _lines import separator_barriers +se._ACTIVE_PAPER = "namaste_telangana" # gates paper-specific clustering passes (pipeline sets this too) +RUN=Path(os.environ.get("JG_RUN","output/NamastheTelangana_Jangaon_20260520_20260609_130635")); D=RUN/"pages" +OUT=RUN/"all_article_crops_review"; P="namaste_telangana" +if OUT.exists(): shutil.rmtree(OUT) +for pg in (1,2): + png=str(D/f"page_{pg:03d}.png") + regs=json.loads((D/f"page_{pg:03d}.regions.json").read_text())["regions"] + buf=io.StringIO() + with contextlib.redirect_stdout(buf): + regs=se.dedup_overlapping_regions(regs); regs=se.merge_stacked_title_lines(regs,png) + regs=se.drop_masthead_regions(regs,P,pg,png); regs=se.drop_classifieds_regions(regs,png); regs=se.tag_legal_notices(regs,png); regs=se.promote_paragraph_titles(regs,png,P) + if pg==1: regs=se.detect_sakshi_headline_bands(regs,png) # NT: front-page display headlines only + regs=se.mark_bullet_titles(regs,png) + ds=se.find_article_starts_by_dateline(regs,png,P) + bars=separator_barriers(png,regs) + _probe=se.cluster_all_article_blocks(regs,ds,sep_lines=bars) + if se.recover_orphan_text_headlines(_probe,regs,png): ds=se.find_article_starts_by_dateline(regs,png,P) + blocks=se.cluster_all_article_blocks(regs,ds,sep_lines=bars) + se.draw_boundary_map_debug(png,regs,D,pg,dateline_starts=ds,sep_lines=bars) + n=se.crop_all_article_blocks(png,regs,str(OUT),pg,dateline_starts=ds,sep_lines=bars) + log=buf.getvalue() + rm={r['id']:r for r in regs}; dlids={s['region_id'] for s in ds} + print(f"\n=== page {pg}: {n} crops, {len(ds)} datelines ===") + for b in sorted(blocks,key=lambda b:(b['bbox'][1],b['bbox'][0])): + x1,y1,x2,y2=b['bbox']; w,h=x2-x1,y2-y1 + ndl=sum(1 for m in b['members'] if m in dlids) + types={} + for m in b['members']: + t=rm.get(m,{}).get('type','?'); types[t]=types.get(t,0)+1 + ntext=types.get('text',0); nimg=types.get('image',0) + flags=[] + if ndl>=2: flags.append(f"{ndl}DL(roundup?)") + if ntext==0 and nimg>0: flags.append("IMG-ONLY(no text)") + if len(b['members'])==1: flags.append("SINGLE-region") + if b['kind']=='O': flags.append("ORPHAN") + print(f" R{b['anchor_id']:>3} {b['kind']} {'RAW' if b.get('_raw_rect') else ' '} " + f"bbox=[{x1},{y1},{x2},{y2}] {w}x{h} mem={len(b['members'])} {types} " + f"{' '.join(flags)}") + for key in ("Section banner","Containment-merge","Headline-merge","Banner rectangle"): + c=[l.strip() for l in log.splitlines() if key in l] + if c: print(f" [{key}] x{len(c)//2 if key in ('Headline-merge','Containment-merge','Section banner','Banner rectangle') else len(c)}") +print("\nDONE ->",OUT) diff --git a/_replay_namaste.py b/_replay_namaste.py new file mode 100644 index 0000000..4e8bfca --- /dev/null +++ b/_replay_namaste.py @@ -0,0 +1,34 @@ +import os,tempfile,contextlib,io,json,shutil +tempfile.tempdir=os.path.abspath("_tess_tmp"); os.environ["TMPDIR"]=tempfile.tempdir +os.makedirs("_tess_tmp",exist_ok=True) +from pathlib import Path +import smart_extractor as se +from _lines import separator_barriers +se._ACTIVE_PAPER = "namaste_telangana" # gates paper-specific clustering passes (pipeline sets this too) +RUN=Path("output/NamastheTelangana_Siddipet_20260602_20260604_214937"); D=RUN/"pages" +OUT=RUN/"all_article_crops_final"; P="namaste_telangana" +if OUT.exists(): shutil.rmtree(OUT) +for pg in (1,2,3,4): + png=str(D/f"page_{pg:03d}.png") + regs=json.loads((D/f"page_{pg:03d}.regions.json").read_text())["regions"] + buf=io.StringIO() + with contextlib.redirect_stdout(buf): + regs=se.dedup_overlapping_regions(regs); regs=se.merge_stacked_title_lines(regs,png) + regs=se.drop_masthead_regions(regs,P,pg,png); regs=se.drop_classifieds_regions(regs,png); regs=se.tag_legal_notices(regs,png); regs=se.promote_paragraph_titles(regs,png,P) + if pg==1: regs=se.detect_sakshi_headline_bands(regs,png) # NT: front-page display headlines only + regs=se.mark_bullet_titles(regs,png) + ds=se.find_article_starts_by_dateline(regs,png,P) + bars=separator_barriers(png,regs) + _probe=se.cluster_all_article_blocks(regs,ds,sep_lines=bars) + if se.recover_orphan_text_headlines(_probe,regs,png): ds=se.find_article_starts_by_dateline(regs,png,P) + blocks=se.cluster_all_article_blocks(regs,ds,sep_lines=bars) + se.draw_boundary_map_debug(png,regs,D,pg,dateline_starts=ds,sep_lines=bars) + n=se.crop_all_article_blocks(png,regs,str(OUT),pg,dateline_starts=ds,sep_lines=bars) + log=buf.getvalue() + tags={k:sum(1 for l in log.splitlines() if k in l) for k in + ["Caption bound","Caption orphan dropped","bullet titles flagged","Headline-strip", + "Floor-trim","Block dateline split","Multi-column body","orphan overlap"]} + tags={k:(v//2 if k in ("Headline-strip","Floor-trim","Multi-column body","Block dateline split","orphan overlap") else v) for k,v in tags.items()} + over=[b['anchor_id'] for b in blocks if len([m for m in b['members'] if m in {s['region_id'] for s in ds}])>=2] + print(f"page {pg}: {n} crops, {len(ds)} datelines, 2+dl-blocks={over or 'none'} | "+", ".join(f"{k}:{v}" for k,v in tags.items() if v)) +print("DONE ->",OUT) diff --git a/_replay_pg1.py b/_replay_pg1.py new file mode 100644 index 0000000..7159692 --- /dev/null +++ b/_replay_pg1.py @@ -0,0 +1,47 @@ +import sys, json +sys.path.insert(0, "/Users/kranthikumar/Documents/telugu_extractor") +from pathlib import Path +import smart_extractor as se +from _lines import separator_barriers + +D = Path("output/AndhraJyothi_Siddipet District_20260602_20260603_144919/pages") +png = D / "page_001.png" +regions = json.loads((D / "page_001.regions.json").read_text())["regions"] +print(f"raw regions: {len(regions)}") + +# 1) masthead drop +regions2 = se.drop_masthead_regions(regions, "andhra_jyothi", 1, png) +dropped = {r['id'] for r in regions} - {r['id'] for r in regions2} +print(f"masthead dropped ids: {sorted(dropped)} -> {len(regions2)} regions") + +# 2) dateline starts +ds = se.find_article_starts_by_dateline(regions2, str(png), "andhra_jyothi") +print(f"dateline starts: {len(ds)}") +for s in ds: + print(f" DL region_id={s['region_id']} headline_id={s.get('headline_id')} bbox={s['bbox']}") + +# 3) separator barriers +bars = separator_barriers(str(png), regions2) +print(f"separator barriers: {len(bars)}") + +# 4) cluster +blocks = se.cluster_all_article_blocks(regions2, dateline_starts=ds, sep_lines=bars) +print(f"\nblocks: {len(blocks)}") +dl_ids = {s['region_id'] for s in ds} +for b in sorted(blocks, key=lambda d: (d['bbox'][1], d['bbox'][0])): + mem = b['members'] + has_dl = any(m in dl_ids for m in mem) + print(f" block anchor={b['anchor_id']} kind={b['kind']} dl={has_dl} bbox={b['bbox']} members={sorted(mem)}") + +# 5) render crops to /tmp for visual check +import shutil +out = Path("output/AndhraJyothi_Siddipet District_20260602_20260603_144919/all_article_crops_test") +if out.exists(): + shutil.rmtree(out) +out.mkdir(parents=True) +n = se.crop_all_article_blocks(str(png), regions2, str(out), 1, + dateline_starts=ds, sep_lines=bars) +print(f"\ncrops written: {n} -> {out}/page_001") +import os +for f in sorted(os.listdir(out / "page_001")): + print(" ", f) diff --git a/_replay_pg23.py b/_replay_pg23.py new file mode 100644 index 0000000..5f1720c --- /dev/null +++ b/_replay_pg23.py @@ -0,0 +1,33 @@ +import sys, json +sys.path.insert(0, "/Users/kranthikumar/Documents/telugu_extractor") +from pathlib import Path +import smart_extractor as se +from _lines import separator_barriers + +D = Path("output/AndhraJyothi_Siddipet District_20260602_20260603_144919/pages") +for pg in (2, 3): + png = D / f"page_{pg:03d}.png" + regions = json.loads((D / f"page_{pg:03d}.regions.json").read_text())["regions"] + regions = se.drop_masthead_regions(regions, "andhra_jyothi", pg, png) + ds = se.find_article_starts_by_dateline(regions, str(png), "andhra_jyothi") + bars = separator_barriers(str(png), regions) + blocks = se.cluster_all_article_blocks(regions, dateline_starts=ds, sep_lines=bars) + dl_ids = {s['region_id'] for s in ds} + print(f"\n==== PAGE {pg} datelines={len(ds)} bars={len(bars)} blocks={len(blocks)} ====") + # each dateline must sit in a block that ALSO has a headline/title anchor + for s in ds: + rid = s['region_id'] + owner = None + for b in blocks: + if rid in b['members'] or rid == b['anchor_id']: + owner = b; break + kind = owner['kind'] if owner else '??' + ndl = sum(1 for m in owner['members'] if m in dl_ids) if owner else 0 + flag = '' + if owner is None: + flag = 'LOST' + elif kind == 'DL': + flag = 'headless(DL)' + elif ndl > 1: + flag = f'MERGED({ndl} datelines)' + print(f" DL R{rid} -> block anchor={owner['anchor_id'] if owner else None} kind={kind} {flag}") diff --git a/_replay_political.py b/_replay_political.py new file mode 100644 index 0000000..4720e03 --- /dev/null +++ b/_replay_political.py @@ -0,0 +1,53 @@ +import os, tempfile, contextlib, io, json, sys, shutil +tempfile.tempdir = os.path.abspath("_tess_tmp"); os.environ["TMPDIR"] = tempfile.tempdir +os.makedirs("_tess_tmp", exist_ok=True) +from pathlib import Path +import smart_extractor as se +from _lines import separator_barriers + +# Replays the POLITICAL crop step (no API): saved political_result.json member ids +# + freshly recomputed region prep chain -> crop_political_articles (block-snap). +RUNS = [ + ("sakshi", Path("output/Sakshi_Jangaon_District_20260520_20260609_210104"), [1]), + ("andhra_jyothi", Path("output/AndhraJyothi_Siddipet District_20260602_20260603_144919"), [2, 3, 4]), + ("namaste_telangana", Path("output/NamastheTelangana_Jangaon_20260520_20260608_124720"), [1]), +] +for paper, RUN, pages in RUNS: + se._ACTIVE_PAPER = paper + D = RUN / "pages" + OUT = RUN / "articles_review" + if OUT.exists(): + shutil.rmtree(OUT) + for pg in pages: + png = str(D / f"page_{pg:03d}.png") + prf = D / f"page_{pg:03d}.political_result.json" + if not prf.exists() or not os.path.exists(png): + continue + political = json.load(open(prf)).get("political_articles", []) + if not political: + continue + regs = json.loads((D / f"page_{pg:03d}.regions.json").read_text())["regions"] + buf = io.StringIO() + with contextlib.redirect_stdout(buf): + regs = se.dedup_overlapping_regions(regs) + regs = se.merge_stacked_title_lines(regs, png) + regs = se.drop_masthead_regions(regs, paper, pg, png) + regs = se.drop_classifieds_regions(regs, png) + regs = se.tag_legal_notices(regs, png) + regs = se.promote_paragraph_titles(regs, png, paper) + if paper == "sakshi": + regs = se.detect_sakshi_headline_bands(regs, png) + regs = se.mark_bullet_titles(regs, png) + ds = se.find_article_starts_by_dateline(regs, png, paper) + bars = separator_barriers(png, regs) + _p = se.cluster_all_article_blocks(regs, ds, sep_lines=bars) + if se.recover_orphan_text_headlines(_p, regs, png): + ds = se.find_article_starts_by_dateline(regs, png, paper) + recs = se.crop_political_articles(png, political, regs, OUT, pg, + dateline_starts=ds, sep_lines=bars) + print(f"=== {RUN.name} page {pg}: {len(political)} political → {len(recs)} crops") + for l in buf.getvalue().splitlines(): + if any(k in l for k in ("Block-snap", "Cropped:", "Clipped", "Floored", "Grew", + "Banner", "block-snap unavailable")): + print(" ", l.strip()) +print("DONE") diff --git a/_replay_rescue.py b/_replay_rescue.py new file mode 100644 index 0000000..b536288 --- /dev/null +++ b/_replay_rescue.py @@ -0,0 +1,25 @@ +import os,tempfile,contextlib,io,json,shutil +tempfile.tempdir=os.path.abspath("_tess_tmp"); os.environ["TMPDIR"]=tempfile.tempdir +os.makedirs("_tess_tmp",exist_ok=True) +from pathlib import Path +import smart_extractor as se +from _lines import separator_barriers +RUN=Path("output/AndhraJyothi_Siddipet District_20260602_20260603_144919"); D=RUN/"pages" +OUT=RUN/"all_article_crops_rescue"; PAPER="andhra_jyothi" +if OUT.exists(): shutil.rmtree(OUT) +for pg in (1,2,3,4): + png=str(D/f"page_{pg:03d}.png") + regs=json.loads((D/f"page_{pg:03d}.regions.json").read_text())["regions"] # 0.5 saved + buf=io.StringIO() + with contextlib.redirect_stdout(buf): + regs=se.dedup_overlapping_regions(regs); regs=se.merge_stacked_title_lines(regs,png) + regs=se.drop_masthead_regions(regs,PAPER,pg,png); regs=se.promote_paragraph_titles(regs,png,PAPER) + bars=separator_barriers(png,regs) + regs=se.rescue_headlines_above_photos(regs,png,PAPER,bars) # NEW + ds=se.find_article_starts_by_dateline(regs,png,PAPER) + blocks=se.cluster_all_article_blocks(regs,ds,sep_lines=bars) + n=se.crop_all_article_blocks(png,regs,str(OUT),pg,dateline_starts=ds,sep_lines=bars) + resc=[l for l in buf.getvalue().splitlines() if "HEADLINE RESCUE" in l] + print(f"page {pg}: {n} crops | {len(resc)} headline(s) rescued") + for l in resc: print(" ",l.strip()) +print("DONE") diff --git a/_replay_sakshi.py b/_replay_sakshi.py new file mode 100644 index 0000000..d47d665 --- /dev/null +++ b/_replay_sakshi.py @@ -0,0 +1,47 @@ +import os,tempfile,contextlib,io,json,shutil +tempfile.tempdir=os.path.abspath("_tess_tmp"); os.environ["TMPDIR"]=tempfile.tempdir +os.makedirs("_tess_tmp",exist_ok=True) +from pathlib import Path +import smart_extractor as se +from _lines import separator_barriers +se._ACTIVE_PAPER = "sakshi" # gate Sakshi-specific clustering passes +RUN=Path(os.environ.get("SAK_RUN","output/Sakshi_Jangaon_District_20260520_20260609_210104")); D=RUN/"pages" +OUT=RUN/"all_article_crops_review"; P="sakshi" +if OUT.exists(): shutil.rmtree(OUT) +for pg in (1,2): + png=str(D/f"page_{pg:03d}.png") + regs=json.loads((D/f"page_{pg:03d}.regions.json").read_text())["regions"] + buf=io.StringIO() + with contextlib.redirect_stdout(buf): + regs=se.dedup_overlapping_regions(regs); regs=se.merge_stacked_title_lines(regs,png) + regs=se.drop_masthead_regions(regs,P,pg,png); regs=se.drop_classifieds_regions(regs,png); regs=se.tag_legal_notices(regs,png); regs=se.promote_paragraph_titles(regs,png,P) + regs=se.detect_sakshi_headline_bands(regs,png) + regs=se.mark_bullet_titles(regs,png) + ds=se.find_article_starts_by_dateline(regs,png,P) + bars=separator_barriers(png,regs) + _probe=se.cluster_all_article_blocks(regs,ds,sep_lines=bars) + if se.recover_orphan_text_headlines(_probe,regs,png): ds=se.find_article_starts_by_dateline(regs,png,P) + blocks=se.cluster_all_article_blocks(regs,ds,sep_lines=bars) + se.draw_boundary_map_debug(png,regs,D,pg,dateline_starts=ds,sep_lines=bars) + n=se.crop_all_article_blocks(png,regs,str(OUT),pg,dateline_starts=ds,sep_lines=bars,overlay=True) + log=buf.getvalue() + rm={r['id']:r for r in regs}; dlids={s['region_id'] for s in ds} + print(f"\n=== page {pg}: {n} crops, {len(ds)} datelines ===") + img_orphans=[] + for b in sorted(blocks,key=lambda b:(b['bbox'][1],b['bbox'][0])): + x1,y1,x2,y2=b['bbox']; w,h=x2-x1,y2-y1 + types={} + for m in b['members']: + t=rm.get(m,{}).get('type','?'); types[t]=types.get(t,0)+1 + ntext=types.get('text',0); nimg=types.get('image',0) + is_art=any(rm.get(m,{}).get('type') in ('doc_title','paragraph_title') for m in b['members']) or any(m in dlids for m in b['members']) + flags=[] + if nimg>0 and not is_art: flags.append("IMG-ORPHAN?") + if b['kind']=='O': flags.append("ORPHAN") + if nimg>0 and not is_art and ntext==0: img_orphans.append(b['anchor_id']) + print(f" R{b['anchor_id']:>3} {b['kind']} bbox=[{x1},{y1},{x2},{y2}] {w}x{h} mem={len(b['members'])} {types} {' '.join(flags)}") + binds=sorted({l.strip() for l in log.splitlines() + if "Orphan photo" in l or "Masthead" in l or "📐" in l or "rule-split" in l}) + print(f" >>> image-only orphans REMAINING: {img_orphans}") + for l in binds: print(f" {l}") +print("\nDONE ->",OUT) diff --git a/_replay_thresh045.py b/_replay_thresh045.py new file mode 100644 index 0000000..01652a4 --- /dev/null +++ b/_replay_thresh045.py @@ -0,0 +1,24 @@ +import os,tempfile,contextlib,io +tempfile.tempdir=os.path.abspath("_tess_tmp"); os.environ["TMPDIR"]=tempfile.tempdir +os.makedirs("_tess_tmp",exist_ok=True) +from pathlib import Path +import smart_extractor as se +from _lines import separator_barriers +RUN=Path("output/AndhraJyothi_Siddipet District_20260602_20260603_144919"); D=RUN/"pages" +OUT=RUN/"all_article_crops_t045"; PAPER="andhra_jyothi" +import shutil +if OUT.exists(): shutil.rmtree(OUT) +for pg in (1,2,3,4): + png=str(D/f"page_{pg:03d}.png") + with contextlib.redirect_stdout(io.StringIO()): + regs=se.detect_regions(png, threshold=0.45) # NEW cutoff + regs=se.dedup_overlapping_regions(regs) + regs=se.merge_stacked_title_lines(regs,png) + regs=se.drop_masthead_regions(regs,PAPER,pg,png) + regs=se.promote_paragraph_titles(regs,png,PAPER) + ds=se.find_article_starts_by_dateline(regs,png,PAPER) + bars=separator_barriers(png,regs) + n=se.crop_all_article_blocks(png,regs,str(OUT),pg,dateline_starts=ds,sep_lines=bars) + nreg=len(regs) + print(f"page {pg}: {nreg} regions, {len(ds)} datelines, {n} crops") +print("DONE ->",OUT) diff --git a/_sak_base.log b/_sak_base.log new file mode 100644 index 0000000..5048a17 --- /dev/null +++ b/_sak_base.log @@ -0,0 +1,78 @@ + +=== page 1: 22 crops, 17 datelines === + R 55 H bbox=[2796,291,4114,2002] 1318x1711 mem=8 {'image': 2, 'text': 5, 'doc_title': 1} + R 94 P bbox=[2113,299,2785,3703] 672x3404 mem=5 {'paragraph_title': 1, 'text': 4} + R 52 H bbox=[1080,371,2766,2350] 1686x1979 mem=6 {'doc_title': 1, 'image': 3, 'paragraph_title': 1, 'text': 1} + R 71 DL bbox=[769,913,1375,1781] 606x868 mem=3 {'text': 2, 'paragraph_title': 1} + R 83 H bbox=[92,1079,686,2507] 594x1428 mem=4 {'doc_title': 1, 'text': 1, 'image': 1, 'paragraph_title': 1} + R 11 DL bbox=[694,1789,1440,2678] 746x889 mem=1 {'text': 1} + R 77 H bbox=[2793,1973,4114,3303] 1321x1330 mem=10 {'text': 6, 'image': 1, 'figure_title': 1, 'doc_title': 1, 'paragraph_title': 1} + R 72 H bbox=[92,2536,685,3404] 593x868 mem=2 {'doc_title': 1, 'text': 1} + R 66 P bbox=[694,2686,1391,3745] 697x1059 mem=4 {'paragraph_title': 1, 'text': 3} + R 51 P bbox=[1399,2686,2105,3458] 706x772 mem=3 {'paragraph_title': 1, 'text': 2} + R 67 H bbox=[92,3432,686,4743] 594x1311 mem=3 {'doc_title': 1, 'text': 1, 'image': 1} + R 78 H bbox=[749,3466,2070,6049] 1321x2583 mem=11 {'text': 8, 'image': 1, 'figure_title': 1, 'doc_title': 1} + R 54 H bbox=[2793,3509,4114,6094] 1321x2585 mem=13 {'doc_title': 1, 'text': 9, 'image': 2, 'paragraph_title': 1} + R 4 DL bbox=[2113,3707,2752,4876] 639x1169 mem=2 {'text': 1, 'image': 1} + R 3 DL bbox=[750,3753,1388,4588] 638x835 mem=1 {'text': 1} + R 50 H bbox=[2112,4357,2752,5581] 640x1224 mem=4 {'doc_title': 1, 'text': 3} + R 9 DL bbox=[92,4771,685,5982] 593x1211 mem=2 {'text': 1, 'paragraph_title': 1} + R 5 DL bbox=[2112,5609,2753,6686] 641x1077 mem=2 {'text': 1, 'paragraph_title': 1} + R 75 H bbox=[91,6012,688,6640] 597x628 mem=2 {'doc_title': 1, 'text': 1} + R 79 P bbox=[772,6082,1391,6637] 619x555 mem=2 {'paragraph_title': 1, 'text': 1} + R 18 O bbox=[1430,6086,2049,6635] 619x549 mem=1 {'text': 1} ORPHAN + R 98 H bbox=[2793,6114,4114,6696] 1321x582 mem=3 {'doc_title': 1, 'text': 2} + >>> image-only orphans REMAINING: [] + 🖼 Orphan photo: R28 → article R78 (an image must belong to an article) + 🖼 Orphan photo: R28 → article R78 (an image must belong to an article) + 🖼 Orphan photo: R28 → article R78 (an image must belong to an article) + 🖼 Orphan photo: R28 → article R78 (an image must belong to an article) + +=== page 2: 20 crops, 11 datelines === + R 61 O bbox=[777,474,1506,2308] 729x1834 mem=8 {'image': 3, 'text': 3, 'figure_title': 2} IMG-ORPHAN? ORPHAN + R 82 H bbox=[695,478,2747,2308] 2052x1830 mem=10 {'text': 3, 'image': 4, 'figure_title': 2, 'doc_title': 1} + R 55 O bbox=[2549,479,2801,1028] 252x549 mem=2 {'image': 1, 'text': 1} IMG-ORPHAN? ORPHAN + R129 P bbox=[80,482,769,1053] 689x571 mem=2 {'paragraph_title': 1, 'text': 1} + R 72 H bbox=[2755,482,3519,2232] 764x1750 mem=8 {'image': 3, 'text': 3, 'doc_title': 1, 'figure_title': 1} + R127 H bbox=[3353,482,4115,2215] 762x1733 mem=9 {'text': 4, 'image': 3, 'figure_title': 1, 'doc_title': 1} + R 24 O bbox=[80,1061,687,2305] 607x1244 mem=4 {'image': 1, 'figure_title': 1, 'text': 2} IMG-ORPHAN? ORPHAN + R118 H bbox=[2795,2298,4115,2954] 1320x656 mem=4 {'doc_title': 1, 'text': 1, 'image': 2} + R 91 H bbox=[72,2313,707,3101] 635x788 mem=3 {'doc_title': 1, 'text': 2} + R 41 H bbox=[753,2316,2753,3600] 2000x1284 mem=11 {'text': 9, 'doc_title': 1, 'image': 1} + R113 H bbox=[72,2417,709,4641] 637x2224 mem=7 {'doc_title': 1, 'text': 3, 'image': 2, 'figure_title': 1} + R 14 DL bbox=[3475,2685,4115,3999] 640x1314 mem=5 {'text': 3, 'image': 1, 'paragraph_title': 1} + R 3 DL bbox=[2761,2958,3467,4049] 706x1091 mem=3 {'text': 1, 'image': 1, 'figure_title': 1} + R 99 H bbox=[752,3656,2071,4649] 1319x993 mem=5 {'image': 1, 'text': 3, 'doc_title': 1} + R 4 F bbox=[2002,3658,2752,5938] 750x2280 mem=8 {'text': 4, 'image': 1, 'paragraph_title': 3} + R 70 H bbox=[2760,4057,4115,5702] 1355x1645 mem=11 {'doc_title': 1, 'text': 9, 'image': 1} + R 93 H bbox=[1216,4710,1963,5939] 747x1229 mem=6 {'text': 3, 'image': 1, 'doc_title': 1, 'figure_title': 1} + R136 P bbox=[95,4735,1161,5529] 1066x794 mem=3 {'paragraph_title': 1, 'text': 1, 'image': 1} + R 44 F bbox=[650,4979,1128,6617] 478x1638 mem=6 {'text': 5, 'paragraph_title': 1} + R111 P bbox=[120,5618,603,6628] 483x1010 mem=3 {'paragraph_title': 1, 'text': 2} + R 98 H bbox=[2798,5710,4115,6696] 1317x986 mem=6 {'doc_title': 1, 'text': 4, 'image': 1} + R 77 H bbox=[1217,5969,2752,6682] 1535x713 mem=4 {'text': 3, 'doc_title': 1} + >>> image-only orphans REMAINING: [] + 🖼 Orphan photo: R42 → article R82 (an image must belong to an article) + 🖼 Orphan photo: R52 → article R127 (an image must belong to an article) + 🖼 Orphan photo: R63 → article R82 (an image must belong to an article) + 🖼 Orphan photo: R60 → article R82 (an image must belong to an article) + 🖼 Orphan photo: R38 → article R72 (an image must belong to an article) + 🖼 Orphan photo: R48 → article R127 (an image must belong to an article) + 🖼 Orphan photo: R42 → article R82 (an image must belong to an article) + 🖼 Orphan photo: R52 → article R127 (an image must belong to an article) + 🖼 Orphan photo: R63 → article R82 (an image must belong to an article) + 🖼 Orphan photo: R60 → article R82 (an image must belong to an article) + 🖼 Orphan photo: R38 → article R72 (an image must belong to an article) + 🖼 Orphan photo: R48 → article R127 (an image must belong to an article) + 🖼 Orphan photo: R42 → article R82 (an image must belong to an article) + 🖼 Orphan photo: R52 → article R127 (an image must belong to an article) + 🖼 Orphan photo: R63 → article R82 (an image must belong to an article) + 🖼 Orphan photo: R60 → article R82 (an image must belong to an article) + 🖼 Orphan photo: R38 → article R72 (an image must belong to an article) + 🖼 Orphan photo: R48 → article R127 (an image must belong to an article) + 🖼 Orphan photo: R42 → article R82 (an image must belong to an article) + 🖼 Orphan photo: R52 → article R127 (an image must belong to an article) + 🖼 Orphan photo: R63 → article R82 (an image must belong to an article) + 🖼 Orphan photo: R60 → article R82 (an image must belong to an article) + +DONE -> output/Sakshi_Jangaon_District_20260520_20260609_210104/all_article_crops_review diff --git a/_sak_fix2.log b/_sak_fix2.log new file mode 100644 index 0000000..4a9833e --- /dev/null +++ b/_sak_fix2.log @@ -0,0 +1,63 @@ + +=== page 1: 19 crops, 16 datelines === + R 55 H bbox=[2796,1049,4114,2002] 1318x953 mem=6 {'doc_title': 1, 'text': 4, 'image': 1} + R 52 H bbox=[664,1067,2749,2657] 2085x1590 mem=8 {'doc_title': 1, 'text': 3, 'image': 2, 'paragraph_title': 2} + R 83 H bbox=[92,1079,686,2507] 594x1428 mem=4 {'doc_title': 1, 'text': 1, 'image': 1, 'paragraph_title': 1} + R 94 P bbox=[2113,1941,2755,3257] 642x1316 mem=4 {'paragraph_title': 1, 'text': 3} + R 77 H bbox=[2793,1973,4114,3315] 1321x1342 mem=8 {'text': 6, 'doc_title': 1, 'paragraph_title': 1} + R 72 H bbox=[92,2536,685,3404] 593x868 mem=2 {'doc_title': 1, 'text': 1} + R 66 P bbox=[693,2665,1391,3349] 698x684 mem=2 {'paragraph_title': 1, 'text': 1} + R 51 P bbox=[1399,2665,2105,3349] 706x684 mem=3 {'paragraph_title': 1, 'text': 2} + R100 H bbox=[2113,2860,4114,4876] 2001x2016 mem=10 {'doc_title': 1, 'text': 5, 'image': 3, 'figure_title': 1} + R 99 H bbox=[750,3357,2068,4588] 1318x1231 mem=5 {'doc_title': 1, 'text': 3, 'image': 1} + R 67 H bbox=[92,3432,686,4743] 594x1311 mem=3 {'doc_title': 1, 'text': 1, 'image': 1} + R 78 H bbox=[749,3891,2070,6049] 1321x2158 mem=10 {'doc_title': 1, 'text': 8, 'figure_title': 1} + R 50 H bbox=[2112,4357,2752,5581] 640x1224 mem=4 {'doc_title': 1, 'text': 3} + R 54 H bbox=[2793,4368,4114,6094] 1321x1726 mem=9 {'doc_title': 1, 'text': 6, 'image': 1, 'paragraph_title': 1} + R 9 DL bbox=[92,4771,685,5982] 593x1211 mem=2 {'text': 1, 'paragraph_title': 1} + R 5 DL bbox=[2112,5609,2753,6686] 641x1077 mem=2 {'text': 1, 'paragraph_title': 1} + R 75 H bbox=[91,6012,688,6640] 597x628 mem=2 {'doc_title': 1, 'text': 1} + R 79 P bbox=[772,6082,2049,6637] 1277x555 mem=3 {'text': 2, 'paragraph_title': 1} + R 98 H bbox=[2793,6114,4114,6696] 1321x582 mem=3 {'doc_title': 1, 'text': 2} + >>> image-only orphans REMAINING: [] + 📐 Sakshi headline SYNTH: R100 doc_title [2126, 3353, 4097, 3452] (un-boxed display headline band, ink=0.35) + 📐 Sakshi headline SYNTH: R99 doc_title [763, 3357, 2068, 3452] (un-boxed display headline band, ink=0.30) + 📐 Sakshi headline WIDEN: R52 [1182, 1067, 2738, 1327] → [664, 1067, 2738, 1327] (un-boxed kicker ink alongside) + 🗞️ Masthead: dropped 5 Sakshi top-band region(s) [59, 62, 63, 64, 71] (above first headline y=1049) + +=== page 2: 21 crops, 11 datelines === + R 61 O bbox=[777,474,1506,2308] 729x1834 mem=8 {'image': 3, 'text': 3, 'figure_title': 2} IMG-ORPHAN? ORPHAN + R 82 H bbox=[695,478,2747,2308] 2052x1830 mem=10 {'text': 3, 'image': 4, 'figure_title': 2, 'doc_title': 1} + R 55 O bbox=[2549,479,2801,1028] 252x549 mem=2 {'image': 1, 'text': 1} IMG-ORPHAN? ORPHAN + R129 P bbox=[80,482,769,1053] 689x571 mem=2 {'paragraph_title': 1, 'text': 1} + R 72 H bbox=[2755,482,3519,2232] 764x1750 mem=8 {'image': 3, 'text': 3, 'doc_title': 1, 'figure_title': 1} + R127 H bbox=[3353,482,4115,2215] 762x1733 mem=9 {'text': 4, 'image': 3, 'figure_title': 1, 'doc_title': 1} + R 24 O bbox=[80,1061,687,2305] 607x1244 mem=4 {'image': 1, 'figure_title': 1, 'text': 2} IMG-ORPHAN? ORPHAN + R118 H bbox=[2795,2298,4115,2954] 1320x656 mem=4 {'doc_title': 1, 'text': 1, 'image': 2} + R 91 H bbox=[72,2313,707,3101] 635x788 mem=3 {'doc_title': 1, 'text': 2} + R 41 H bbox=[753,2316,2753,3600] 2000x1284 mem=11 {'text': 9, 'doc_title': 1, 'image': 1} + R113 H bbox=[72,2417,709,4641] 637x2224 mem=7 {'doc_title': 1, 'text': 3, 'image': 2, 'figure_title': 1} + R 14 DL bbox=[3475,2685,4115,3999] 640x1314 mem=5 {'text': 3, 'image': 1, 'paragraph_title': 1} + R 3 DL bbox=[2761,2958,3467,4049] 706x1091 mem=3 {'text': 1, 'image': 1, 'figure_title': 1} + R 99 H bbox=[650,3656,2071,6145] 1421x2489 mem=10 {'image': 1, 'text': 7, 'doc_title': 1, 'paragraph_title': 1} + R 10 F bbox=[2113,3658,2752,4643] 639x985 mem=4 {'text': 2, 'image': 1, 'paragraph_title': 1} + R 70 H bbox=[2760,4057,4115,5702] 1355x1645 mem=11 {'doc_title': 1, 'text': 9, 'image': 1} + R 93 H bbox=[1216,4710,1963,5939] 747x1229 mem=6 {'text': 3, 'image': 1, 'doc_title': 1, 'figure_title': 1} + R 4 F bbox=[2002,4723,2752,5938] 750x1215 mem=4 {'text': 2, 'paragraph_title': 2} + R136 P bbox=[95,4735,1161,5529] 1066x794 mem=3 {'paragraph_title': 1, 'text': 1, 'image': 1} + R111 P bbox=[120,5618,603,6628] 483x1010 mem=3 {'paragraph_title': 1, 'text': 2} + R 98 H bbox=[2798,5710,4115,6696] 1317x986 mem=6 {'doc_title': 1, 'text': 4, 'image': 1} + R 77 H bbox=[1104,5969,2752,6682] 1648x713 mem=4 {'text': 3, 'doc_title': 1} + R139 H bbox=[614,6155,1168,6617] 554x462 mem=2 {'doc_title': 1, 'text': 1} + >>> image-only orphans REMAINING: [] + 📐 Sakshi headline SYNTH: R139 doc_title [614, 6155, 1168, 6236] (un-boxed display headline band, ink=0.51) + 📐 Sakshi headline WIDEN: R77 [1217, 5969, 2690, 6109] → [1104, 5969, 2690, 6109] (un-boxed kicker ink alongside) + 🖼 Orphan photo: R38 → article R72 (an image must belong to an article) + 🖼 Orphan photo: R42 → article R82 (an image must belong to an article) + 🖼 Orphan photo: R48 → article R127 (an image must belong to an article) + 🖼 Orphan photo: R52 → article R127 (an image must belong to an article) + 🖼 Orphan photo: R60 → article R82 (an image must belong to an article) + 🖼 Orphan photo: R63 → article R82 (an image must belong to an article) + 🗞️ Masthead: dropped 4 Sakshi top-band region(s) [68, 71, 84, 104] (above first headline y=499) + +DONE -> output/Sakshi_Jangaon_District_20260520_20260609_210104/all_article_crops_review diff --git a/_sak_fix3.log b/_sak_fix3.log new file mode 100644 index 0000000..6681937 --- /dev/null +++ b/_sak_fix3.log @@ -0,0 +1,54 @@ + +=== page 1: 16 crops, 16 datelines === + R 55 H bbox=[2763,1049,4114,2732] 1351x1683 mem=12 {'doc_title': 1, 'text': 9, 'image': 1, 'paragraph_title': 1} + R 52 H bbox=[664,1067,2755,3314] 2091x2247 mem=17 {'doc_title': 1, 'text': 9, 'image': 2, 'paragraph_title': 5} + R 83 H bbox=[92,1079,686,2507] 594x1428 mem=4 {'doc_title': 1, 'text': 1, 'image': 1, 'paragraph_title': 1} + R 72 H bbox=[92,2536,685,3404] 593x868 mem=2 {'doc_title': 1, 'text': 1} + R 77 H bbox=[2793,2740,4114,3303] 1321x563 mem=4 {'doc_title': 1, 'text': 1, 'image': 1, 'figure_title': 1} + R100 H bbox=[2113,3353,4113,4297] 2000x944 mem=7 {'doc_title': 1, 'text': 5, 'image': 1} + R 99 H bbox=[750,3357,2069,4605] 1319x1248 mem=8 {'doc_title': 1, 'text': 5, 'image': 1, 'figure_title': 1} + R 67 H bbox=[92,3432,686,4743] 594x1311 mem=3 {'doc_title': 1, 'text': 1, 'image': 1} + R 50 H bbox=[2112,4357,2752,5581] 640x1224 mem=5 {'text': 3, 'image': 1, 'doc_title': 1} + R 54 H bbox=[2793,4368,4114,6094] 1321x1726 mem=9 {'doc_title': 1, 'text': 6, 'image': 1, 'paragraph_title': 1} + R 78 H bbox=[749,4622,2070,6049] 1321x1427 mem=7 {'doc_title': 1, 'text': 6} + R 9 DL bbox=[92,4771,685,5982] 593x1211 mem=2 {'text': 1, 'paragraph_title': 1} + R 5 DL bbox=[2112,5609,2753,6686] 641x1077 mem=2 {'text': 1, 'paragraph_title': 1} + R 75 H bbox=[91,6012,688,6640] 597x628 mem=2 {'doc_title': 1, 'text': 1} + R 79 P bbox=[772,6082,2049,6637] 1277x555 mem=3 {'text': 2, 'paragraph_title': 1} + R 98 H bbox=[2793,6114,4114,6696] 1321x582 mem=3 {'doc_title': 1, 'text': 2} + >>> image-only orphans REMAINING: [] + 📐 Sakshi headline SYNTH: R100 doc_title [2126, 3353, 4097, 3452] (un-boxed display headline band, ink=0.35) + 📐 Sakshi headline SYNTH: R99 doc_title [763, 3357, 2068, 3452] (un-boxed display headline band, ink=0.30) + 📐 Sakshi headline WIDEN: R52 [1182, 1067, 2738, 1327] → [664, 1067, 2738, 1327] (un-boxed kicker ink alongside) + 🗞️ Masthead: dropped 5 Sakshi top-band region(s) [59, 62, 63, 64, 71] (above first headline y=1049) + +=== page 2: 17 crops, 11 datelines === + R 82 H bbox=[777,474,2801,2212] 2024x1738 mem=20 {'text': 7, 'image': 8, 'figure_title': 4, 'doc_title': 1} + R127 H bbox=[2755,485,4086,2232] 1331x1747 mem=17 {'text': 8, 'image': 6, 'figure_title': 2, 'doc_title': 1} + R129 P bbox=[95,499,741,2736] 646x2237 mem=8 {'paragraph_title': 1, 'image': 2, 'text': 3, 'figure_title': 2} + R118 H bbox=[2795,2298,4115,2954] 1320x656 mem=4 {'text': 1, 'image': 2, 'doc_title': 1} + R 91 H bbox=[72,2313,707,3101] 635x788 mem=3 {'doc_title': 1, 'text': 2} + R 41 H bbox=[753,2316,2753,3600] 2000x1284 mem=11 {'text': 9, 'doc_title': 1, 'image': 1} + R 14 DL bbox=[3475,2685,4115,3999] 640x1314 mem=5 {'text': 3, 'image': 1, 'paragraph_title': 1} + R 3 DL bbox=[2761,2958,3467,4049] 706x1091 mem=3 {'text': 1, 'image': 1, 'figure_title': 1} + R113 H bbox=[72,3129,709,4641] 637x1512 mem=5 {'doc_title': 1, 'text': 3, 'image': 1} + R 99 H bbox=[650,3656,2071,6145] 1421x2489 mem=10 {'image': 1, 'text': 7, 'doc_title': 1, 'paragraph_title': 1} + R 10 F bbox=[2113,3658,2752,4643] 639x985 mem=4 {'text': 2, 'image': 1, 'paragraph_title': 1} + R 70 H bbox=[2760,4057,4115,5702] 1355x1645 mem=11 {'doc_title': 1, 'text': 9, 'image': 1} + R 93 H bbox=[1216,4710,1963,5939] 747x1229 mem=6 {'text': 3, 'image': 1, 'doc_title': 1, 'figure_title': 1} + R 4 F bbox=[2002,4723,2752,5938] 750x1215 mem=4 {'text': 2, 'paragraph_title': 2} + R136 P bbox=[95,4735,1161,5529] 1066x794 mem=3 {'paragraph_title': 1, 'text': 1, 'image': 1} + R111 P bbox=[120,5618,603,6628] 483x1010 mem=3 {'paragraph_title': 1, 'text': 2} + R 98 H bbox=[2798,5710,4115,6696] 1317x986 mem=6 {'doc_title': 1, 'text': 4, 'image': 1} + R 77 H bbox=[1104,5969,2752,6682] 1648x713 mem=4 {'text': 3, 'doc_title': 1} + R139 H bbox=[614,6155,1168,6617] 554x462 mem=2 {'doc_title': 1, 'text': 1} + >>> image-only orphans REMAINING: [] + ─ Sakshi rule-split: R41 → 3 bands at rules [1066, 2252] + ─ Sakshi rule-split: R74 → 2 bands at rules [1066] + 📐 Sakshi headline SYNTH: R139 doc_title [614, 6155, 1168, 6236] (un-boxed display headline band, ink=0.51) + 📐 Sakshi headline WIDEN: R77 [1217, 5969, 2690, 6109] → [1104, 5969, 2690, 6109] (un-boxed kicker ink alongside) + 🖼 Orphan photo: R38 → article R127 (an image must belong to an article) + 🖼 Orphan photo: R61 → article R82 (an image must belong to an article) + 🗞️ Masthead: dropped 4 Sakshi top-band region(s) [68, 71, 84, 104] (above first headline y=499) + +DONE -> output/Sakshi_Jangaon_District_20260520_20260609_210104/all_article_crops_review diff --git a/_sak_fix4.log b/_sak_fix4.log new file mode 100644 index 0000000..9032b7b --- /dev/null +++ b/_sak_fix4.log @@ -0,0 +1,54 @@ + +=== page 1: 16 crops, 16 datelines === + R 55 H bbox=[2763,1049,4114,2732] 1351x1683 mem=12 {'doc_title': 1, 'text': 9, 'image': 1, 'paragraph_title': 1} + R 52 H bbox=[688,1067,2755,3314] 2067x2247 mem=17 {'doc_title': 1, 'text': 9, 'image': 2, 'paragraph_title': 5} + R 83 H bbox=[92,1079,686,2507] 594x1428 mem=4 {'doc_title': 1, 'text': 1, 'image': 1, 'paragraph_title': 1} + R 72 H bbox=[92,2536,685,3404] 593x868 mem=2 {'doc_title': 1, 'text': 1} + R 77 H bbox=[2793,2740,4114,3303] 1321x563 mem=4 {'doc_title': 1, 'text': 1, 'image': 1, 'figure_title': 1} + R100 H bbox=[2113,3353,4113,4297] 2000x944 mem=7 {'doc_title': 1, 'text': 5, 'image': 1} + R 99 H bbox=[750,3357,2069,4605] 1319x1248 mem=8 {'doc_title': 1, 'text': 5, 'image': 1, 'figure_title': 1} + R 67 H bbox=[92,3432,686,4743] 594x1311 mem=3 {'doc_title': 1, 'text': 1, 'image': 1} + R 50 H bbox=[2112,4357,2752,5581] 640x1224 mem=5 {'text': 3, 'image': 1, 'doc_title': 1} + R 54 H bbox=[2793,4368,4114,6094] 1321x1726 mem=9 {'doc_title': 1, 'text': 6, 'image': 1, 'paragraph_title': 1} + R 78 H bbox=[749,4622,2070,6049] 1321x1427 mem=7 {'doc_title': 1, 'text': 6} + R 9 DL bbox=[92,4771,685,5982] 593x1211 mem=2 {'text': 1, 'paragraph_title': 1} + R 5 DL bbox=[2112,5609,2753,6686] 641x1077 mem=2 {'text': 1, 'paragraph_title': 1} + R 75 H bbox=[91,6012,688,6640] 597x628 mem=2 {'doc_title': 1, 'text': 1} + R 79 P bbox=[772,6082,2049,6637] 1277x555 mem=3 {'text': 2, 'paragraph_title': 1} + R 98 H bbox=[2793,6114,4114,6696] 1321x582 mem=3 {'doc_title': 1, 'text': 2} + >>> image-only orphans REMAINING: [] + 📐 Sakshi headline SYNTH: R100 doc_title [2126, 3353, 4097, 3452] (un-boxed display headline band, ink=0.35) + 📐 Sakshi headline SYNTH: R99 doc_title [763, 3357, 2068, 3452] (un-boxed display headline band, ink=0.30) + 📐 Sakshi headline WIDEN: R52 [1182, 1067, 2738, 1327] → [688, 1067, 2738, 1327] (un-boxed kicker ink alongside) + 🗞️ Masthead: dropped 5 Sakshi top-band region(s) [59, 62, 63, 64, 71] (above first headline y=1049) + +=== page 2: 17 crops, 11 datelines === + R 82 H bbox=[777,474,2801,2212] 2024x1738 mem=20 {'text': 7, 'image': 8, 'figure_title': 4, 'doc_title': 1} + R127 H bbox=[2755,485,4086,2232] 1331x1747 mem=17 {'text': 8, 'image': 6, 'figure_title': 2, 'doc_title': 1} + R129 P bbox=[95,499,741,2736] 646x2237 mem=8 {'paragraph_title': 1, 'image': 2, 'text': 3, 'figure_title': 2} + R118 H bbox=[2795,2298,4115,2954] 1320x656 mem=4 {'text': 1, 'image': 2, 'doc_title': 1} + R 91 H bbox=[72,2313,707,3101] 635x788 mem=3 {'doc_title': 1, 'text': 2} + R 41 H bbox=[753,2316,2753,3600] 2000x1284 mem=11 {'text': 9, 'doc_title': 1, 'image': 1} + R 14 DL bbox=[3475,2685,4115,3999] 640x1314 mem=5 {'text': 3, 'image': 1, 'paragraph_title': 1} + R 3 DL bbox=[2761,2958,3467,4049] 706x1091 mem=3 {'text': 1, 'image': 1, 'figure_title': 1} + R113 H bbox=[72,3129,709,4641] 637x1512 mem=5 {'doc_title': 1, 'text': 3, 'image': 1} + R 99 H bbox=[752,3656,2071,4649] 1319x993 mem=5 {'image': 1, 'text': 3, 'doc_title': 1} + R 4 F bbox=[2002,3658,2752,5938] 750x2280 mem=8 {'text': 4, 'image': 1, 'paragraph_title': 3} + R 70 H bbox=[2760,4057,4115,5702] 1355x1645 mem=11 {'doc_title': 1, 'text': 9, 'image': 1} + R 93 H bbox=[1216,4710,1963,5939] 747x1229 mem=4 {'text': 3, 'doc_title': 1} + R136 P bbox=[95,4735,1161,5529] 1066x794 mem=3 {'paragraph_title': 1, 'text': 1, 'image': 1} + R 44 F bbox=[650,4979,1128,6145] 478x1166 mem=5 {'text': 4, 'paragraph_title': 1} + R 77 H bbox=[1147,5193,2752,6694] 1605x1501 mem=6 {'doc_title': 1, 'text': 3, 'image': 1, 'figure_title': 1} + R111 P bbox=[120,5618,603,6628] 483x1010 mem=3 {'paragraph_title': 1, 'text': 2} + R 98 H bbox=[2798,5710,4115,6696] 1317x986 mem=6 {'doc_title': 1, 'text': 4, 'image': 1} + R139 H bbox=[614,6155,1168,6617] 554x462 mem=2 {'doc_title': 1, 'text': 1} + >>> image-only orphans REMAINING: [] + ─ Sakshi rule-split: R41 → 3 bands at rules [1066, 2252] + ─ Sakshi rule-split: R74 → 2 bands at rules [1066] + 📐 Sakshi headline SYNTH: R139 doc_title [614, 6155, 1168, 6236] (un-boxed display headline band, ink=0.51) + 📐 Sakshi headline WIDEN: R77 [1217, 5969, 2690, 6109] → [1147, 5969, 2690, 6109] (un-boxed kicker ink alongside) + 🖼 Orphan photo: R38 → article R127 (an image must belong to an article) + 🖼 Orphan photo: R61 → article R82 (an image must belong to an article) + 🗞️ Masthead: dropped 4 Sakshi top-band region(s) [68, 71, 84, 104] (above first headline y=499) + +DONE -> output/Sakshi_Jangaon_District_20260520_20260609_210104/all_article_crops_review diff --git a/_sak_fix5.log b/_sak_fix5.log new file mode 100644 index 0000000..b1068d9 --- /dev/null +++ b/_sak_fix5.log @@ -0,0 +1,52 @@ + +=== page 1: 16 crops, 16 datelines === + R 55 H bbox=[2763,1049,4114,2732] 1351x1683 mem=12 {'doc_title': 1, 'text': 9, 'image': 1, 'paragraph_title': 1} + R 52 H bbox=[688,1067,2755,3314] 2067x2247 mem=17 {'doc_title': 1, 'text': 9, 'image': 2, 'paragraph_title': 5} + R 83 H bbox=[92,1079,686,2507] 594x1428 mem=4 {'doc_title': 1, 'text': 1, 'image': 1, 'paragraph_title': 1} + R 72 H bbox=[92,2536,685,3404] 593x868 mem=2 {'doc_title': 1, 'text': 1} + R 77 H bbox=[2793,2740,4114,3303] 1321x563 mem=4 {'doc_title': 1, 'text': 1, 'image': 1, 'figure_title': 1} + R100 H bbox=[2113,3353,4113,4297] 2000x944 mem=7 {'doc_title': 1, 'text': 5, 'image': 1} + R 99 H bbox=[750,3357,2069,4605] 1319x1248 mem=8 {'doc_title': 1, 'text': 5, 'image': 1, 'figure_title': 1} + R 67 H bbox=[92,3432,686,4743] 594x1311 mem=3 {'doc_title': 1, 'text': 1, 'image': 1} + R 50 H bbox=[2112,4357,2752,5581] 640x1224 mem=5 {'text': 3, 'image': 1, 'doc_title': 1} + R 54 H bbox=[2793,4368,4114,6094] 1321x1726 mem=9 {'doc_title': 1, 'text': 6, 'image': 1, 'paragraph_title': 1} + R 78 H bbox=[749,4622,2070,6049] 1321x1427 mem=7 {'doc_title': 1, 'text': 6} + R 9 DL bbox=[92,4771,685,5982] 593x1211 mem=2 {'text': 1, 'paragraph_title': 1} + R 5 DL bbox=[2112,5609,2753,6686] 641x1077 mem=2 {'text': 1, 'paragraph_title': 1} + R 75 H bbox=[91,6012,688,6640] 597x628 mem=2 {'doc_title': 1, 'text': 1} + R 79 P bbox=[772,6082,2049,6637] 1277x555 mem=3 {'text': 2, 'paragraph_title': 1} + R 98 H bbox=[2793,6114,4114,6696] 1321x582 mem=3 {'doc_title': 1, 'text': 2} + >>> image-only orphans REMAINING: [] + 📐 Sakshi headline SYNTH: R100 doc_title [2126, 3353, 4097, 3452] (un-boxed display headline band, ink=0.35) + 📐 Sakshi headline SYNTH: R99 doc_title [763, 3357, 2068, 3452] (un-boxed display headline band, ink=0.30) + 📐 Sakshi headline WIDEN: R52 [1182, 1067, 2738, 1327] → [688, 1067, 2738, 1327] (un-boxed kicker ink alongside) + 🗞️ Masthead: dropped 5 Sakshi top-band region(s) [59, 62, 63, 64, 71] (above first headline y=1049) + +=== page 2: 17 crops, 11 datelines === + R 82 H bbox=[777,474,2801,2212] 2024x1738 mem=20 {'text': 7, 'image': 8, 'figure_title': 4, 'doc_title': 1} + R127 H bbox=[2755,485,4086,2232] 1331x1747 mem=17 {'text': 8, 'image': 6, 'figure_title': 2, 'doc_title': 1} + R129 P bbox=[95,499,741,2736] 646x2237 mem=8 {'paragraph_title': 1, 'image': 2, 'text': 3, 'figure_title': 2} + R118 H bbox=[2795,2298,4115,2957] 1320x659 mem=5 {'text': 2, 'image': 2, 'doc_title': 1} + R 91 H bbox=[72,2313,707,3101] 635x788 mem=3 {'doc_title': 1, 'text': 2} + R 41 H bbox=[753,2316,2753,3600] 2000x1284 mem=11 {'text': 9, 'doc_title': 1, 'image': 1} + R 3 DL bbox=[2761,2958,3467,4049] 706x1091 mem=3 {'text': 1, 'image': 1, 'figure_title': 1} + R 14 DL bbox=[3475,3022,4115,3999] 640x977 mem=4 {'text': 2, 'image': 1, 'paragraph_title': 1} + R113 H bbox=[72,3129,709,4641] 637x1512 mem=5 {'doc_title': 1, 'text': 3, 'image': 1} + R 99 H bbox=[752,3656,2071,4649] 1319x993 mem=5 {'image': 1, 'text': 3, 'doc_title': 1} + R 88 P bbox=[2002,3658,2752,5938] 750x2280 mem=8 {'text': 4, 'image': 1, 'paragraph_title': 3} + R 70 H bbox=[2760,4057,4115,5702] 1355x1645 mem=11 {'doc_title': 1, 'text': 9, 'image': 1} + R 93 H bbox=[1216,4710,1963,5939] 747x1229 mem=6 {'text': 3, 'image': 1, 'doc_title': 1, 'figure_title': 1} + R136 P bbox=[95,4735,1161,5529] 1066x794 mem=3 {'paragraph_title': 1, 'text': 1, 'image': 1} + R 44 F bbox=[650,4979,1128,6617] 478x1638 mem=6 {'text': 5, 'paragraph_title': 1} + R111 P bbox=[120,5618,603,6628] 483x1010 mem=3 {'paragraph_title': 1, 'text': 2} + R 98 H bbox=[2798,5710,4115,6696] 1317x986 mem=6 {'doc_title': 1, 'text': 4, 'image': 1} + R 77 H bbox=[1147,5969,2752,6682] 1605x713 mem=4 {'text': 3, 'doc_title': 1} + >>> image-only orphans REMAINING: [] + ─ Sakshi rule-split: R41 → 3 bands at rules [1066, 2252] + ─ Sakshi rule-split: R74 → 2 bands at rules [1066] + 📐 Sakshi headline WIDEN: R77 [1217, 5969, 2690, 6109] → [1147, 5969, 2690, 6109] (un-boxed kicker ink alongside) + 🖼 Orphan photo: R38 → article R127 (an image must belong to an article) + 🖼 Orphan photo: R61 → article R82 (an image must belong to an article) + 🗞️ Masthead: dropped 4 Sakshi top-band region(s) [68, 71, 84, 104] (above first headline y=499) + +DONE -> output/Sakshi_Jangaon_District_20260520_20260609_210104/all_article_crops_review diff --git a/_sak_fix6.log b/_sak_fix6.log new file mode 100644 index 0000000..178f746 --- /dev/null +++ b/_sak_fix6.log @@ -0,0 +1,52 @@ + +=== page 1: 16 crops, 16 datelines === + R 55 H bbox=[2763,1049,4114,2732] 1351x1683 mem=12 {'doc_title': 1, 'text': 9, 'image': 1, 'paragraph_title': 1} + R 52 H bbox=[688,1067,2755,3314] 2067x2247 mem=17 {'doc_title': 1, 'text': 9, 'image': 2, 'paragraph_title': 5} + R 83 H bbox=[92,1079,686,2507] 594x1428 mem=4 {'doc_title': 1, 'text': 1, 'image': 1, 'paragraph_title': 1} + R 72 H bbox=[92,2536,685,3404] 593x868 mem=2 {'doc_title': 1, 'text': 1} + R 77 H bbox=[2793,2740,4114,3303] 1321x563 mem=4 {'doc_title': 1, 'text': 1, 'image': 1, 'figure_title': 1} + R100 H bbox=[2113,3353,4113,4297] 2000x944 mem=7 {'doc_title': 1, 'text': 5, 'image': 1} + R 99 H bbox=[750,3357,2069,4605] 1319x1248 mem=8 {'doc_title': 1, 'text': 5, 'image': 1, 'figure_title': 1} + R 67 H bbox=[92,3432,686,4743] 594x1311 mem=3 {'doc_title': 1, 'text': 1, 'image': 1} + R 50 H bbox=[2112,4357,2752,5581] 640x1224 mem=5 {'text': 3, 'image': 1, 'doc_title': 1} + R 54 H bbox=[2793,4368,4114,6094] 1321x1726 mem=9 {'doc_title': 1, 'text': 6, 'image': 1, 'paragraph_title': 1} + R 78 H bbox=[749,4622,2070,6049] 1321x1427 mem=7 {'doc_title': 1, 'text': 6} + R 9 DL bbox=[92,4771,685,5982] 593x1211 mem=2 {'text': 1, 'paragraph_title': 1} + R 5 DL bbox=[2112,5609,2753,6686] 641x1077 mem=2 {'text': 1, 'paragraph_title': 1} + R 75 H bbox=[91,6012,688,6640] 597x628 mem=2 {'doc_title': 1, 'text': 1} + R 79 P bbox=[772,6082,2049,6637] 1277x555 mem=3 {'text': 2, 'paragraph_title': 1} + R 98 H bbox=[2793,6114,4114,6696] 1321x582 mem=3 {'doc_title': 1, 'text': 2} + >>> image-only orphans REMAINING: [] + 📐 Sakshi headline SYNTH: R100 doc_title [2126, 3353, 4097, 3452] (un-boxed display headline band, ink=0.35) + 📐 Sakshi headline SYNTH: R99 doc_title [763, 3357, 2068, 3452] (un-boxed display headline band, ink=0.30) + 📐 Sakshi headline WIDEN: R52 [1182, 1067, 2738, 1327] → [688, 1067, 2738, 1327] (un-boxed kicker ink alongside) + 🗞️ Masthead: dropped 5 Sakshi top-band region(s) [59, 62, 63, 64, 71] (above first headline y=1049) + +=== page 2: 17 crops, 11 datelines === + R 82 H bbox=[777,474,2801,2212] 2024x1738 mem=20 {'text': 7, 'image': 8, 'figure_title': 4, 'doc_title': 1} + R127 H bbox=[2755,485,4086,2232] 1331x1747 mem=17 {'text': 8, 'image': 6, 'figure_title': 2, 'doc_title': 1} + R129 P bbox=[95,499,741,2209] 646x1710 mem=6 {'paragraph_title': 1, 'image': 1, 'text': 3, 'figure_title': 1} + R118 H bbox=[2795,2298,4115,2957] 1320x659 mem=5 {'text': 2, 'image': 2, 'doc_title': 1} + R 91 H bbox=[72,2313,707,3101] 635x788 mem=5 {'text': 2, 'image': 1, 'doc_title': 1, 'figure_title': 1} + R 41 H bbox=[753,2316,2753,3600] 2000x1284 mem=11 {'text': 9, 'doc_title': 1, 'image': 1} + R 3 DL bbox=[2761,2958,3467,4049] 706x1091 mem=3 {'text': 1, 'image': 1, 'figure_title': 1} + R 14 DL bbox=[3475,3022,4115,3999] 640x977 mem=4 {'text': 2, 'image': 1, 'paragraph_title': 1} + R113 H bbox=[72,3129,709,4641] 637x1512 mem=5 {'doc_title': 1, 'text': 3, 'image': 1} + R 99 H bbox=[752,3656,2071,4649] 1319x993 mem=5 {'image': 1, 'text': 3, 'doc_title': 1} + R 88 P bbox=[2002,3658,2752,5938] 750x2280 mem=8 {'text': 4, 'image': 1, 'paragraph_title': 3} + R 70 H bbox=[2760,4057,4115,5702] 1355x1645 mem=11 {'doc_title': 1, 'text': 9, 'image': 1} + R 93 H bbox=[1216,4710,1963,5939] 747x1229 mem=6 {'text': 3, 'image': 1, 'doc_title': 1, 'figure_title': 1} + R136 P bbox=[95,4735,1161,5529] 1066x794 mem=3 {'paragraph_title': 1, 'text': 1, 'image': 1} + R 44 F bbox=[650,4979,1128,6617] 478x1638 mem=6 {'text': 5, 'paragraph_title': 1} + R111 P bbox=[120,5618,603,6628] 483x1010 mem=3 {'paragraph_title': 1, 'text': 2} + R 98 H bbox=[2798,5710,4115,6696] 1317x986 mem=6 {'doc_title': 1, 'text': 4, 'image': 1} + R 77 H bbox=[1147,5969,2752,6682] 1605x713 mem=4 {'text': 3, 'doc_title': 1} + >>> image-only orphans REMAINING: [] + ─ Sakshi rule-split: R41 → 3 bands at rules [1066, 2252] + ─ Sakshi rule-split: R74 → 2 bands at rules [1066] + 📐 Sakshi headline WIDEN: R77 [1217, 5969, 2690, 6109] → [1147, 5969, 2690, 6109] (un-boxed kicker ink alongside) + 🖼 Orphan photo: R38 → article R127 (an image must belong to an article) + 🖼 Orphan photo: R61 → article R82 (an image must belong to an article) + 🗞️ Masthead: dropped 4 Sakshi top-band region(s) [68, 71, 84, 104] (above first headline y=499) + +DONE -> output/Sakshi_Jangaon_District_20260520_20260609_210104/all_article_crops_review diff --git a/_sak_fix7.log b/_sak_fix7.log new file mode 100644 index 0000000..9317ffa --- /dev/null +++ b/_sak_fix7.log @@ -0,0 +1,55 @@ + +=== page 1: 16 crops, 16 datelines === + R 55 H bbox=[2763,1049,4114,2732] 1351x1683 mem=12 {'doc_title': 1, 'text': 9, 'image': 1, 'paragraph_title': 1} + R 52 H bbox=[688,1067,2755,3314] 2067x2247 mem=17 {'doc_title': 1, 'text': 9, 'image': 2, 'paragraph_title': 5} + R 83 H bbox=[92,1079,686,2507] 594x1428 mem=4 {'doc_title': 1, 'text': 1, 'image': 1, 'paragraph_title': 1} + R 72 H bbox=[92,2536,685,3404] 593x868 mem=2 {'doc_title': 1, 'text': 1} + R 77 H bbox=[2793,2740,4114,3303] 1321x563 mem=4 {'doc_title': 1, 'text': 1, 'image': 1, 'figure_title': 1} + R100 H bbox=[2113,3353,4113,4297] 2000x944 mem=7 {'doc_title': 1, 'text': 5, 'image': 1} + R 99 H bbox=[750,3357,2069,4605] 1319x1248 mem=8 {'doc_title': 1, 'text': 5, 'image': 1, 'figure_title': 1} + R 67 H bbox=[92,3432,686,4743] 594x1311 mem=3 {'doc_title': 1, 'text': 1, 'image': 1} + R 50 H bbox=[2112,4357,2752,5581] 640x1224 mem=5 {'text': 3, 'image': 1, 'doc_title': 1} + R 54 H bbox=[2793,4368,4114,6094] 1321x1726 mem=9 {'doc_title': 1, 'text': 6, 'image': 1, 'paragraph_title': 1} + R 78 H bbox=[749,4622,2070,6049] 1321x1427 mem=7 {'doc_title': 1, 'text': 6} + R 9 DL bbox=[92,4771,685,5982] 593x1211 mem=2 {'text': 1, 'paragraph_title': 1} + R 5 DL bbox=[2112,5609,2753,6686] 641x1077 mem=2 {'text': 1, 'paragraph_title': 1} + R 75 H bbox=[91,6012,688,6640] 597x628 mem=2 {'doc_title': 1, 'text': 1} + R 79 P bbox=[772,6082,2049,6637] 1277x555 mem=3 {'text': 2, 'paragraph_title': 1} + R 98 H bbox=[2793,6114,4114,6696] 1321x582 mem=3 {'doc_title': 1, 'text': 2} + >>> image-only orphans REMAINING: [] + 📐 Sakshi headline SYNTH: R100 doc_title [2126, 3353, 4097, 3452] (un-boxed display headline band, ink=0.35) + 📐 Sakshi headline SYNTH: R99 doc_title [763, 3357, 2068, 3452] (un-boxed display headline band, ink=0.30) + 📐 Sakshi headline WIDEN: R52 [1182, 1067, 2738, 1327] → [688, 1067, 2738, 1327] (un-boxed kicker ink alongside) + 🗞️ Masthead: dropped 5 Sakshi top-band region(s) [59, 62, 63, 64, 71] (above first headline y=1049) + +=== page 2: 17 crops, 11 datelines === + R 82 H bbox=[777,474,2801,2212] 2024x1738 mem=20 {'text': 7, 'image': 8, 'figure_title': 4, 'doc_title': 1} + R127 H bbox=[2755,485,4086,2232] 1331x1747 mem=17 {'text': 8, 'image': 6, 'figure_title': 2, 'doc_title': 1} + R129 P bbox=[95,499,741,2209] 646x1710 mem=6 {'paragraph_title': 1, 'image': 1, 'text': 3, 'figure_title': 1} + R118 H bbox=[2795,2298,4115,2957] 1320x659 mem=5 {'text': 2, 'image': 2, 'doc_title': 1} + R 91 H bbox=[72,2313,707,3101] 635x788 mem=5 {'text': 2, 'image': 1, 'doc_title': 1, 'figure_title': 1} + R 41 H bbox=[753,2316,2753,3600] 2000x1284 mem=11 {'text': 9, 'doc_title': 1, 'image': 1} + R 3 DL bbox=[2761,2958,3467,4049] 706x1091 mem=3 {'text': 1, 'image': 1, 'figure_title': 1} + R 14 DL bbox=[3475,3022,4115,3999] 640x977 mem=4 {'text': 2, 'image': 1, 'paragraph_title': 1} + R113 H bbox=[72,3129,709,4641] 637x1512 mem=5 {'doc_title': 1, 'text': 3, 'image': 1} + R 99 H bbox=[752,3656,2071,4649] 1319x993 mem=5 {'image': 1, 'text': 3, 'doc_title': 1} + R 88 P bbox=[2113,3658,2752,4643] 639x985 mem=4 {'text': 2, 'image': 1, 'paragraph_title': 1} + R 70 H bbox=[2760,4057,4115,5702] 1355x1645 mem=11 {'doc_title': 1, 'text': 9, 'image': 1} + R 93 H bbox=[1216,4710,2752,5939] 1536x1229 mem=10 {'text': 5, 'image': 1, 'doc_title': 1, 'paragraph_title': 2, 'figure_title': 1} + R136 P bbox=[95,4735,1161,5529] 1066x794 mem=3 {'paragraph_title': 1, 'text': 1, 'image': 1} + R 44 F bbox=[650,4979,1128,5185] 478x206 mem=1 {'text': 1} + R 92 F bbox=[653,5252,1127,5744] 474x492 mem=3 {'text': 2, 'paragraph_title': 1} + R111 P bbox=[120,5618,603,6628] 483x1010 mem=3 {'paragraph_title': 1, 'text': 2} + R 98 H bbox=[2798,5710,4115,6696] 1317x986 mem=6 {'doc_title': 1, 'text': 4, 'image': 1} + R 90 F bbox=[685,5924,1099,6145] 414x221 mem=1 {'text': 1} + R 77 H bbox=[1147,5969,2752,6682] 1605x713 mem=4 {'text': 3, 'doc_title': 1} + R110 F bbox=[668,6258,1106,6617] 438x359 mem=1 {'text': 1} + >>> image-only orphans REMAINING: [] + ─ Sakshi rule-split: R41 → 3 bands at rules [1066, 2252] + ─ Sakshi rule-split: R74 → 2 bands at rules [1066] + 📐 Sakshi headline WIDEN: R77 [1217, 5969, 2690, 6109] → [1147, 5969, 2690, 6109] (un-boxed kicker ink alongside) + 🖼 Orphan photo: R38 → article R127 (an image must belong to an article) + 🖼 Orphan photo: R61 → article R82 (an image must belong to an article) + 🗞️ Masthead: dropped 4 Sakshi top-band region(s) [68, 71, 84, 104] (above first headline y=499) + +DONE -> output/Sakshi_Jangaon_District_20260520_20260609_210104/all_article_crops_review diff --git a/_sak_fixABC.log b/_sak_fixABC.log new file mode 100644 index 0000000..2be5047 --- /dev/null +++ b/_sak_fixABC.log @@ -0,0 +1,55 @@ + +=== page 1: 12 crops, 16 datelines === + R 55 H bbox=[2796,1049,4114,2002] 1318x953 mem=6 {'doc_title': 1, 'text': 4, 'image': 1} + R 52 H bbox=[664,1067,2749,2657] 2085x1590 mem=8 {'doc_title': 1, 'text': 3, 'image': 2, 'paragraph_title': 2} + R 83 H bbox=[92,1079,686,2507] 594x1428 mem=4 {'doc_title': 1, 'text': 1, 'image': 1, 'paragraph_title': 1} + R 94 P bbox=[2113,1941,2755,3257] 642x1316 mem=4 {'paragraph_title': 1, 'text': 3} + R 77 H bbox=[2793,1973,4114,3315] 1321x1342 mem=8 {'text': 6, 'doc_title': 1, 'paragraph_title': 1} + R 72 H bbox=[92,2536,685,3404] 593x868 mem=2 {'doc_title': 1, 'text': 1} + R 66 P bbox=[693,2665,1391,6678] 698x4013 mem=2 {'paragraph_title': 1, 'text': 1} + R 51 P bbox=[1399,2665,2105,6678] 706x4013 mem=3 {'paragraph_title': 1, 'text': 2} + R 99 H bbox=[749,2860,4114,6698] 3365x3838 mem=45 {'text': 30, 'image': 5, 'figure_title': 2, 'paragraph_title': 3, 'doc_title': 5} + R 67 H bbox=[92,3432,686,4743] 594x1311 mem=3 {'doc_title': 1, 'text': 1, 'image': 1} + R 9 DL bbox=[92,4771,685,5982] 593x1211 mem=2 {'text': 1, 'paragraph_title': 1} + R 75 H bbox=[91,6012,688,6640] 597x628 mem=2 {'doc_title': 1, 'text': 1} + >>> image-only orphans REMAINING: [] + 📐 Sakshi headline SYNTH: R99 doc_title [763, 3353, 4097, 3452] (un-boxed display headline band, ink=0.32) + 📐 Sakshi headline WIDEN: R52 [1182, 1067, 2738, 1327] → [664, 1067, 2738, 1327] (un-boxed kicker ink alongside) + 🗞️ Masthead: dropped 5 Sakshi top-band region(s) [59, 62, 63, 64, 71] (above first headline y=1049) + +=== page 2: 21 crops, 11 datelines === + R 61 O bbox=[777,474,1506,2308] 729x1834 mem=8 {'image': 3, 'text': 3, 'figure_title': 2} IMG-ORPHAN? ORPHAN + R 82 H bbox=[695,478,2747,2308] 2052x1830 mem=10 {'text': 3, 'image': 4, 'figure_title': 2, 'doc_title': 1} + R 55 O bbox=[2549,479,2801,1028] 252x549 mem=2 {'image': 1, 'text': 1} IMG-ORPHAN? ORPHAN + R129 P bbox=[80,482,769,1053] 689x571 mem=2 {'paragraph_title': 1, 'text': 1} + R 72 H bbox=[2755,482,3519,2232] 764x1750 mem=8 {'image': 3, 'text': 3, 'doc_title': 1, 'figure_title': 1} + R127 H bbox=[3353,482,4115,2215] 762x1733 mem=9 {'text': 4, 'image': 3, 'figure_title': 1, 'doc_title': 1} + R 24 O bbox=[80,1061,687,2305] 607x1244 mem=4 {'image': 1, 'figure_title': 1, 'text': 2} IMG-ORPHAN? ORPHAN + R118 H bbox=[2795,2298,4115,2954] 1320x656 mem=4 {'doc_title': 1, 'text': 1, 'image': 2} + R 91 H bbox=[72,2313,707,3101] 635x788 mem=3 {'doc_title': 1, 'text': 2} + R 41 H bbox=[753,2316,2753,3600] 2000x1284 mem=11 {'text': 9, 'doc_title': 1, 'image': 1} + R113 H bbox=[72,2417,709,4641] 637x2224 mem=7 {'doc_title': 1, 'text': 3, 'image': 2, 'figure_title': 1} + R 14 DL bbox=[3475,2685,4115,3999] 640x1314 mem=5 {'text': 3, 'image': 1, 'paragraph_title': 1} + R 3 DL bbox=[2761,2958,3467,4049] 706x1091 mem=3 {'text': 1, 'image': 1, 'figure_title': 1} + R 99 H bbox=[650,3656,2071,6145] 1421x2489 mem=10 {'image': 1, 'text': 7, 'doc_title': 1, 'paragraph_title': 1} + R 10 F bbox=[2113,3658,2752,4643] 639x985 mem=4 {'text': 2, 'image': 1, 'paragraph_title': 1} + R 70 H bbox=[2760,4057,4115,5702] 1355x1645 mem=11 {'doc_title': 1, 'text': 9, 'image': 1} + R 93 H bbox=[1216,4710,1963,5939] 747x1229 mem=6 {'text': 3, 'image': 1, 'doc_title': 1, 'figure_title': 1} + R 4 F bbox=[2002,4723,2752,5938] 750x1215 mem=4 {'text': 2, 'paragraph_title': 2} + R136 P bbox=[95,4735,1161,5529] 1066x794 mem=3 {'paragraph_title': 1, 'text': 1, 'image': 1} + R111 P bbox=[120,5618,603,6628] 483x1010 mem=3 {'paragraph_title': 1, 'text': 2} + R 98 H bbox=[2798,5710,4115,6696] 1317x986 mem=6 {'doc_title': 1, 'text': 4, 'image': 1} + R 77 H bbox=[1104,5969,2752,6682] 1648x713 mem=4 {'text': 3, 'doc_title': 1} + R139 H bbox=[614,6155,1168,6617] 554x462 mem=2 {'doc_title': 1, 'text': 1} + >>> image-only orphans REMAINING: [] + 📐 Sakshi headline SYNTH: R139 doc_title [614, 6155, 1168, 6236] (un-boxed display headline band, ink=0.51) + 📐 Sakshi headline WIDEN: R77 [1217, 5969, 2690, 6109] → [1104, 5969, 2690, 6109] (un-boxed kicker ink alongside) + 🖼 Orphan photo: R38 → article R72 (an image must belong to an article) + 🖼 Orphan photo: R42 → article R82 (an image must belong to an article) + 🖼 Orphan photo: R48 → article R127 (an image must belong to an article) + 🖼 Orphan photo: R52 → article R127 (an image must belong to an article) + 🖼 Orphan photo: R60 → article R82 (an image must belong to an article) + 🖼 Orphan photo: R63 → article R82 (an image must belong to an article) + 🗞️ Masthead: dropped 4 Sakshi top-band region(s) [68, 71, 84, 104] (above first headline y=499) + +DONE -> output/Sakshi_Jangaon_District_20260520_20260609_210104/all_article_crops_review diff --git a/_sak_newrun.log b/_sak_newrun.log new file mode 100644 index 0000000..2fc667b --- /dev/null +++ b/_sak_newrun.log @@ -0,0 +1,41 @@ + +=== page 1: 13 crops, 14 datelines === + R 53 H bbox=[2792,1069,4115,3285] 1323x2216 mem=14 {'doc_title': 1, 'text': 9, 'image': 1, 'figure_title': 1, 'paragraph_title': 2} + R 46 H bbox=[684,1077,2784,3285] 2100x2208 mem=21 {'doc_title': 2, 'text': 11, 'image': 2, 'paragraph_title': 6} + R 66 H bbox=[99,1078,681,2654] 582x1576 mem=3 {'doc_title': 1, 'text': 1, 'paragraph_title': 1} + R 65 H bbox=[99,2678,680,4455] 581x1777 mem=6 {'text': 3, 'image': 1, 'doc_title': 1, 'paragraph_title': 1} + R 82 H bbox=[2114,3293,4115,5007] 2001x1714 mem=11 {'doc_title': 1, 'text': 7, 'image': 1, 'figure_title': 1, 'paragraph_title': 1} + R 75 H bbox=[749,3323,2069,4792] 1320x1469 mem=7 {'doc_title': 1, 'text': 5, 'image': 1} + R 3 DL bbox=[98,4463,741,5703] 643x1240 mem=2 {'text': 1, 'image': 1} + R 47 H bbox=[749,4850,2368,5588] 1619x738 mem=6 {'doc_title': 1, 'text': 2, 'image': 1, 'figure_title': 1, 'paragraph_title': 1} + R 93 H bbox=[2791,5058,4115,6247] 1324x1189 mem=6 {'doc_title': 1, 'text': 3, 'image': 1, 'figure_title': 1} + R 55 H bbox=[2137,5113,2736,6130] 599x1017 mem=2 {'doc_title': 1, 'text': 1} + R 44 H bbox=[98,5711,681,6639] 583x928 mem=2 {'doc_title': 1, 'text': 1} + R100 H bbox=[750,5782,2072,6697] 1322x915 mem=5 {'doc_title': 1, 'text': 2, 'image': 2} + R 56 H bbox=[2564,6284,4115,6691] 1551x407 mem=3 {'doc_title': 1, 'text': 2} + >>> image-only orphans REMAINING: [] + 📐 Sakshi headline WIDEN: R100 [765, 6130, 1350, 6228] → [765, 6130, 1949, 6228] (un-boxed kicker ink alongside) + +=== page 2: 16 crops, 15 datelines === + R 61 H bbox=[931,490,4115,2536] 3184x2046 mem=32 {'text': 15, 'image': 8, 'doc_title': 2, 'paragraph_title': 5, 'figure_title': 2} + R106 H bbox=[2081,490,2753,5535] 672x5045 mem=6 {'doc_title': 1, 'text': 1, 'image': 1, 'figure_title': 2, 'paragraph_title': 1} + R105 P bbox=[78,498,898,1656] 820x1158 mem=5 {'paragraph_title': 1, 'image': 1, 'text': 2, 'figure_title': 1} + R128 P bbox=[106,1664,860,2507] 754x843 mem=4 {'paragraph_title': 1, 'text': 1, 'image': 1, 'figure_title': 1} + R 57 H bbox=[751,2579,2754,4161] 2003x1582 mem=10 {'doc_title': 1, 'text': 7, 'image': 1, 'paragraph_title': 1} + R 69 H bbox=[2794,2589,4115,3863] 1321x1274 mem=6 {'doc_title': 1, 'text': 4, 'image': 1} + R121 DL bbox=[70,2805,709,3948] 639x1143 mem=6 {'text': 3, 'image': 1, 'paragraph_title': 1, 'figure_title': 1} + R100 H bbox=[752,3306,2070,5044] 1318x1738 mem=5 {'text': 4, 'doc_title': 1} + R 95 H bbox=[2795,3914,4115,4990] 1320x1076 mem=6 {'doc_title': 2, 'image': 1, 'text': 2, 'figure_title': 1} + R 87 DL bbox=[73,3974,709,5049] 636x1075 mem=5 {'text': 2, 'image': 2, 'figure_title': 1} + R 80 H bbox=[2814,5054,4115,6172] 1301x1118 mem=7 {'text': 4, 'image': 2, 'doc_title': 1} + R115 DL bbox=[94,5097,1161,6624] 1067x1527 mem=5 {'text': 4, 'image': 1} + R107 H bbox=[1199,5102,2071,6685] 872x1583 mem=5 {'text': 3, 'image': 1, 'doc_title': 1} + R 74 F bbox=[658,5350,1139,6624] 481x1274 mem=4 {'text': 4} + R 7 DL bbox=[2114,5559,2754,6684] 640x1125 mem=3 {'text': 1, 'image': 1, 'paragraph_title': 1} + R 90 H bbox=[2814,5760,3431,6635] 617x875 mem=4 {'doc_title': 1, 'text': 3} + R 76 H bbox=[3474,5970,4115,6634] 641x664 mem=5 {'doc_title': 1, 'text': 3, 'image': 1} + >>> image-only orphans REMAINING: [] + ─ Sakshi rule-split: R12 → 2 bands at rules [4170] + 🖼 Orphan photo: R36 → article R87 (an image must belong to an article) + +DONE -> output/Sakshi_Jangaon District_20260513_20260610_140735/all_article_crops_review diff --git a/_sak_newrun2.log b/_sak_newrun2.log new file mode 100644 index 0000000..1ffcc65 --- /dev/null +++ b/_sak_newrun2.log @@ -0,0 +1,50 @@ + +=== page 1: 19 crops, 14 datelines === + R 53 H bbox=[2792,1069,4115,3285] 1323x2216 mem=14 {'text': 9, 'image': 1, 'doc_title': 1, 'figure_title': 1, 'paragraph_title': 2} + R 46 H bbox=[684,1077,2784,2187] 2100x1110 mem=8 {'doc_title': 2, 'text': 4, 'image': 1, 'paragraph_title': 1} + R 84 P bbox=[1433,1077,2107,2755] 674x1678 mem=2 {'paragraph_title': 1, 'text': 1} + R 66 H bbox=[99,1078,681,2654] 582x1576 mem=3 {'doc_title': 1, 'text': 1, 'paragraph_title': 1} + R 45 P bbox=[689,2195,1425,3315] 736x1120 mem=2 {'paragraph_title': 1, 'text': 1} + R 72 P bbox=[2115,2195,2784,2915] 669x720 mem=5 {'paragraph_title': 1, 'text': 3, 'image': 1} + R 65 H bbox=[99,2678,680,4291] 581x1613 mem=4 {'text': 2, 'image': 1, 'doc_title': 1} + R 48 P bbox=[1433,2763,2107,3315] 674x552 mem=2 {'paragraph_title': 1, 'text': 1} + R 60 P bbox=[2115,2923,2784,3285] 669x362 mem=2 {'paragraph_title': 1, 'text': 1} + R 82 H bbox=[2114,3293,4115,5007] 2001x1714 mem=11 {'text': 7, 'image': 1, 'figure_title': 1, 'paragraph_title': 1, 'doc_title': 1} + R 75 H bbox=[749,3323,2069,4792] 1320x1469 mem=7 {'doc_title': 1, 'text': 5, 'image': 1} + R 86 P bbox=[102,4299,741,4455] 639x156 mem=2 {'paragraph_title': 1, 'text': 1} + R 3 DL bbox=[98,4463,741,5703] 643x1240 mem=2 {'text': 1, 'image': 1} + R 47 H bbox=[749,4850,2368,5588] 1619x738 mem=6 {'doc_title': 1, 'text': 2, 'image': 1, 'figure_title': 1, 'paragraph_title': 1} + R 93 H bbox=[2791,5058,4115,6247] 1324x1189 mem=6 {'doc_title': 1, 'text': 3, 'image': 1, 'figure_title': 1} + R 55 H bbox=[2137,5113,2736,6130] 599x1017 mem=2 {'doc_title': 1, 'text': 1} + R 44 H bbox=[98,5711,681,6639] 583x928 mem=2 {'doc_title': 1, 'text': 1} + R100 H bbox=[750,5782,2072,6697] 1322x915 mem=5 {'doc_title': 1, 'text': 2, 'image': 2} + R 56 H bbox=[2564,6284,4115,6691] 1551x407 mem=3 {'doc_title': 1, 'text': 2} + >>> image-only orphans REMAINING: [] + 📐 Sakshi headline WIDEN: R100 [765, 6130, 1350, 6228] → [765, 6130, 1949, 6228] (un-boxed kicker ink alongside) + +=== page 2: 19 crops, 15 datelines === + R 61 H bbox=[931,490,4115,2536] 3184x2046 mem=32 {'text': 15, 'image': 8, 'doc_title': 2, 'paragraph_title': 5, 'figure_title': 2} + R127 P bbox=[2081,490,2754,4147] 673x3657 mem=3 {'paragraph_title': 1, 'text': 1, 'figure_title': 1} + R105 P bbox=[78,498,898,1518] 820x1020 mem=5 {'paragraph_title': 1, 'image': 1, 'text': 2, 'figure_title': 1} + R121 DL bbox=[70,1526,743,3817] 673x2291 mem=4 {'text': 2, 'image': 1, 'figure_title': 1} + R128 P bbox=[106,1664,860,2507] 754x843 mem=4 {'paragraph_title': 1, 'text': 1, 'image': 1, 'figure_title': 1} + R 57 H bbox=[754,2579,2751,3639] 1997x1060 mem=6 {'doc_title': 1, 'text': 4, 'image': 1} + R 69 H bbox=[2794,2589,4115,3863] 1321x1274 mem=6 {'doc_title': 1, 'text': 4, 'image': 1} + R 4 DL bbox=[751,3306,2071,4149] 1320x843 mem=3 {'text': 3} + R 81 P bbox=[75,3825,743,3966] 668x141 mem=2 {'paragraph_title': 1, 'text': 1} + R 95 H bbox=[2795,3914,4115,4990] 1320x1076 mem=6 {'doc_title': 2, 'image': 1, 'text': 2, 'figure_title': 1} + R 87 DL bbox=[73,3974,709,5049] 636x1075 mem=5 {'text': 2, 'image': 2, 'figure_title': 1} + R100 H bbox=[752,4203,2070,5044] 1318x841 mem=4 {'text': 3, 'doc_title': 1} + R106 H bbox=[2114,4209,2753,5535] 639x1326 mem=5 {'doc_title': 1, 'text': 1, 'image': 1, 'figure_title': 1, 'paragraph_title': 1} + R 80 H bbox=[2814,5054,4115,6172] 1301x1118 mem=7 {'text': 4, 'image': 2, 'doc_title': 1} + R115 DL bbox=[94,5097,1161,6624] 1067x1527 mem=5 {'text': 4, 'image': 1} + R107 H bbox=[1199,5102,2071,6685] 872x1583 mem=5 {'text': 3, 'image': 1, 'doc_title': 1} + R 74 F bbox=[658,5350,1139,6624] 481x1274 mem=4 {'text': 4} + R 7 DL bbox=[2114,5559,2754,6684] 640x1125 mem=3 {'text': 1, 'image': 1, 'paragraph_title': 1} + R 90 H bbox=[2814,5760,3431,6635] 617x875 mem=4 {'doc_title': 1, 'text': 3} + R 76 H bbox=[3474,5970,4115,6634] 641x664 mem=5 {'doc_title': 1, 'text': 3, 'image': 1} + >>> image-only orphans REMAINING: [] + ─ Sakshi rule-split: R12 → 2 bands at rules [4170] + 🖼 Orphan photo: R36 → article R87 (an image must belong to an article) + +DONE -> output/Sakshi_Jangaon District_20260513_20260610_140735/all_article_crops_review diff --git a/_sak_newrun3.log b/_sak_newrun3.log new file mode 100644 index 0000000..3949b33 --- /dev/null +++ b/_sak_newrun3.log @@ -0,0 +1,42 @@ + +=== page 1: 14 crops, 14 datelines === + R 53 H bbox=[2792,1069,4115,3285] 1323x2216 mem=14 {'text': 9, 'image': 1, 'doc_title': 1, 'figure_title': 1, 'paragraph_title': 2} + R 46 H bbox=[684,1077,2784,3285] 2100x2208 mem=21 {'text': 11, 'image': 2, 'paragraph_title': 6, 'doc_title': 2} + R 66 H bbox=[99,1078,681,2654] 582x1576 mem=3 {'doc_title': 1, 'text': 1, 'paragraph_title': 1} + R 65 H bbox=[99,2678,680,4291] 581x1613 mem=4 {'text': 2, 'image': 1, 'doc_title': 1} + R 82 H bbox=[2114,3293,4115,5007] 2001x1714 mem=11 {'text': 7, 'image': 1, 'figure_title': 1, 'paragraph_title': 1, 'doc_title': 1} + R 75 H bbox=[749,3323,2069,4792] 1320x1469 mem=7 {'doc_title': 1, 'text': 5, 'image': 1} + R 86 P bbox=[102,4299,741,4455] 639x156 mem=2 {'paragraph_title': 1, 'text': 1} + R 3 DL bbox=[98,4463,741,5703] 643x1240 mem=2 {'text': 1, 'image': 1} + R 47 H bbox=[749,4850,2368,5588] 1619x738 mem=6 {'doc_title': 1, 'text': 2, 'image': 1, 'figure_title': 1, 'paragraph_title': 1} + R 93 H bbox=[2791,5058,4115,6247] 1324x1189 mem=6 {'doc_title': 1, 'text': 3, 'image': 1, 'figure_title': 1} + R 55 H bbox=[2137,5113,2736,6130] 599x1017 mem=2 {'doc_title': 1, 'text': 1} + R 44 H bbox=[98,5711,681,6639] 583x928 mem=2 {'doc_title': 1, 'text': 1} + R100 H bbox=[750,5782,2072,6697] 1322x915 mem=5 {'doc_title': 1, 'text': 2, 'image': 2} + R 56 H bbox=[2564,6284,4115,6691] 1551x407 mem=3 {'doc_title': 1, 'text': 2} + >>> image-only orphans REMAINING: [] + 📐 Sakshi headline WIDEN: R100 [765, 6130, 1350, 6228] → [765, 6130, 1949, 6228] (un-boxed kicker ink alongside) + +=== page 2: 16 crops, 15 datelines === + R 61 H bbox=[931,490,4115,2536] 3184x2046 mem=32 {'text': 15, 'image': 8, 'doc_title': 2, 'paragraph_title': 5, 'figure_title': 2} + R106 H bbox=[2081,490,2753,5535] 672x5045 mem=6 {'doc_title': 1, 'text': 1, 'image': 1, 'figure_title': 2, 'paragraph_title': 1} + R105 P bbox=[78,498,898,1656] 820x1158 mem=5 {'paragraph_title': 1, 'image': 1, 'text': 2, 'figure_title': 1} + R128 P bbox=[106,1664,860,2507] 754x843 mem=4 {'paragraph_title': 1, 'text': 1, 'image': 1, 'figure_title': 1} + R 57 H bbox=[751,2579,2754,4161] 2003x1582 mem=11 {'text': 8, 'image': 1, 'doc_title': 1, 'paragraph_title': 1} + R 69 H bbox=[2794,2589,4115,3863] 1321x1274 mem=6 {'doc_title': 1, 'text': 4, 'image': 1} + R121 DL bbox=[70,2805,709,3948] 639x1143 mem=6 {'text': 3, 'image': 1, 'paragraph_title': 1, 'figure_title': 1} + R 95 H bbox=[2795,3914,4115,4990] 1320x1076 mem=6 {'doc_title': 2, 'image': 1, 'text': 2, 'figure_title': 1} + R 87 DL bbox=[73,3974,709,5049] 636x1075 mem=5 {'text': 2, 'image': 2, 'figure_title': 1} + R100 H bbox=[752,4203,2070,5044] 1318x841 mem=4 {'text': 3, 'doc_title': 1} + R 80 H bbox=[2814,5054,4115,6172] 1301x1118 mem=7 {'text': 4, 'image': 2, 'doc_title': 1} + R115 DL bbox=[94,5097,1161,6624] 1067x1527 mem=5 {'text': 4, 'image': 1} + R107 H bbox=[1199,5102,2071,6685] 872x1583 mem=5 {'text': 3, 'image': 1, 'doc_title': 1} + R 74 F bbox=[658,5350,1139,6624] 481x1274 mem=4 {'text': 4} + R 7 DL bbox=[2114,5559,2754,6684] 640x1125 mem=3 {'text': 1, 'image': 1, 'paragraph_title': 1} + R 90 H bbox=[2814,5760,3431,6635] 617x875 mem=4 {'doc_title': 1, 'text': 3} + R 76 H bbox=[3474,5970,4115,6634] 641x664 mem=5 {'doc_title': 1, 'text': 3, 'image': 1} + >>> image-only orphans REMAINING: [] + ─ Sakshi rule-split: R12 → 2 bands at rules [4170] + 🖼 Orphan photo: R36 → article R87 (an image must belong to an article) + +DONE -> output/Sakshi_Jangaon District_20260513_20260610_140735/all_article_crops_review diff --git a/_sak_newrun4.log b/_sak_newrun4.log new file mode 100644 index 0000000..b046ab8 --- /dev/null +++ b/_sak_newrun4.log @@ -0,0 +1,42 @@ + +=== page 1: 14 crops, 14 datelines === + R 53 H bbox=[2792,1069,4115,3285] 1323x2216 mem=14 {'text': 9, 'image': 1, 'doc_title': 1, 'figure_title': 1, 'paragraph_title': 2} + R 46 H bbox=[684,1077,2784,3285] 2100x2208 mem=21 {'text': 11, 'image': 2, 'paragraph_title': 6, 'doc_title': 2} + R 66 H bbox=[99,1078,681,2654] 582x1576 mem=3 {'doc_title': 1, 'text': 1, 'paragraph_title': 1} + R 65 H bbox=[99,2678,680,4291] 581x1613 mem=4 {'text': 2, 'image': 1, 'doc_title': 1} + R 82 H bbox=[2114,3293,4115,5007] 2001x1714 mem=11 {'text': 7, 'image': 1, 'figure_title': 1, 'paragraph_title': 1, 'doc_title': 1} + R 75 H bbox=[749,3323,2069,4792] 1320x1469 mem=7 {'doc_title': 1, 'text': 5, 'image': 1} + R 86 P bbox=[102,4299,741,4455] 639x156 mem=2 {'paragraph_title': 1, 'text': 1} + R 3 DL bbox=[98,4463,741,5703] 643x1240 mem=2 {'text': 1, 'image': 1} + R 47 H bbox=[749,4850,2368,5588] 1619x738 mem=6 {'doc_title': 1, 'text': 2, 'image': 1, 'figure_title': 1, 'paragraph_title': 1} + R 93 H bbox=[2791,5058,4115,6247] 1324x1189 mem=6 {'doc_title': 1, 'text': 3, 'image': 1, 'figure_title': 1} + R 55 H bbox=[2137,5113,2736,6130] 599x1017 mem=2 {'doc_title': 1, 'text': 1} + R 44 H bbox=[98,5711,681,6639] 583x928 mem=2 {'doc_title': 1, 'text': 1} + R100 H bbox=[750,5782,2072,6697] 1322x915 mem=5 {'doc_title': 1, 'text': 2, 'image': 2} + R 56 H bbox=[2564,6284,4115,6691] 1551x407 mem=3 {'doc_title': 1, 'text': 2} + >>> image-only orphans REMAINING: [] + 📐 Sakshi headline WIDEN: R100 [765, 6130, 1350, 6228] → [765, 6130, 1949, 6228] (un-boxed kicker ink alongside) + +=== page 2: 16 crops, 15 datelines === + R 61 H bbox=[931,490,4115,2536] 3184x2046 mem=33 {'text': 15, 'image': 8, 'doc_title': 2, 'paragraph_title': 5, 'figure_title': 3} + R105 P bbox=[78,498,898,1656] 820x1158 mem=5 {'paragraph_title': 1, 'image': 1, 'text': 2, 'figure_title': 1} + R128 P bbox=[106,1664,860,2507] 754x843 mem=4 {'paragraph_title': 1, 'text': 1, 'image': 1, 'figure_title': 1} + R 57 H bbox=[751,2579,2754,4161] 2003x1582 mem=11 {'text': 8, 'image': 1, 'doc_title': 1, 'paragraph_title': 1} + R 69 H bbox=[2794,2589,4115,3863] 1321x1274 mem=6 {'doc_title': 1, 'text': 4, 'image': 1} + R121 DL bbox=[70,2805,709,3948] 639x1143 mem=6 {'text': 3, 'image': 1, 'paragraph_title': 1, 'figure_title': 1} + R 95 H bbox=[2795,3914,4115,4990] 1320x1076 mem=6 {'doc_title': 2, 'image': 1, 'text': 2, 'figure_title': 1} + R 87 DL bbox=[73,3974,709,5049] 636x1075 mem=5 {'text': 2, 'image': 2, 'figure_title': 1} + R100 H bbox=[752,4203,2070,5044] 1318x841 mem=4 {'text': 3, 'doc_title': 1} + R106 H bbox=[2114,4209,2753,5535] 639x1326 mem=5 {'doc_title': 1, 'text': 1, 'image': 1, 'figure_title': 1, 'paragraph_title': 1} + R 80 H bbox=[2814,5054,4115,6172] 1301x1118 mem=7 {'text': 4, 'image': 2, 'doc_title': 1} + R115 DL bbox=[94,5097,1161,6624] 1067x1527 mem=5 {'text': 4, 'image': 1} + R107 H bbox=[1199,5102,2071,6685] 872x1583 mem=5 {'text': 3, 'image': 1, 'doc_title': 1} + R 74 F bbox=[658,5350,1139,6624] 481x1274 mem=4 {'text': 4} + R 7 DL bbox=[2114,5559,2754,6684] 640x1125 mem=3 {'text': 1, 'image': 1, 'paragraph_title': 1} + R 90 H bbox=[2814,5760,3431,6635] 617x875 mem=4 {'doc_title': 1, 'text': 3} + R 76 H bbox=[3474,5970,4115,6634] 641x664 mem=5 {'doc_title': 1, 'text': 3, 'image': 1} + >>> image-only orphans REMAINING: [] + ─ Sakshi rule-split: R12 → 2 bands at rules [4170] + 🖼 Orphan photo: R36 → article R87 (an image must belong to an article) + +DONE -> output/Sakshi_Jangaon District_20260513_20260610_140735/all_article_crops_review diff --git a/_sak_oldrun2.log b/_sak_oldrun2.log new file mode 100644 index 0000000..d901e23 --- /dev/null +++ b/_sak_oldrun2.log @@ -0,0 +1,58 @@ + +=== page 1: 19 crops, 16 datelines === + R 55 H bbox=[2763,1049,4114,2732] 1351x1683 mem=12 {'text': 9, 'image': 1, 'doc_title': 1, 'paragraph_title': 1} + R 52 H bbox=[688,1067,2749,2854] 2061x1787 mem=9 {'doc_title': 1, 'text': 4, 'image': 2, 'paragraph_title': 2} + R 83 H bbox=[92,1079,686,2507] 594x1428 mem=4 {'doc_title': 1, 'text': 1, 'image': 1, 'paragraph_title': 1} + R 94 P bbox=[2113,1941,2755,3257] 642x1316 mem=4 {'paragraph_title': 1, 'text': 3} + R 72 H bbox=[92,2536,685,3404] 593x868 mem=2 {'doc_title': 1, 'text': 1} + R 66 P bbox=[749,2686,1391,3302] 642x616 mem=2 {'paragraph_title': 1, 'text': 1} + R 77 H bbox=[2793,2740,4114,3303] 1321x563 mem=4 {'doc_title': 1, 'text': 1, 'image': 1, 'figure_title': 1} + R 51 P bbox=[1399,2862,2105,3349] 706x487 mem=2 {'paragraph_title': 1, 'text': 1} + R100 H bbox=[2113,3353,4113,4297] 2000x944 mem=7 {'doc_title': 1, 'text': 5, 'image': 1} + R 99 H bbox=[750,3357,2069,4605] 1319x1248 mem=8 {'doc_title': 1, 'text': 5, 'image': 1, 'figure_title': 1} + R 67 H bbox=[92,3432,686,4743] 594x1311 mem=3 {'doc_title': 1, 'text': 1, 'image': 1} + R 50 H bbox=[2112,4357,2752,5581] 640x1224 mem=5 {'text': 3, 'image': 1, 'doc_title': 1} + R 54 H bbox=[2793,4368,4114,6094] 1321x1726 mem=9 {'doc_title': 1, 'text': 6, 'image': 1, 'paragraph_title': 1} + R 78 H bbox=[749,4622,2070,6049] 1321x1427 mem=7 {'doc_title': 1, 'text': 6} + R 9 DL bbox=[92,4771,685,5982] 593x1211 mem=2 {'text': 1, 'paragraph_title': 1} + R 5 DL bbox=[2112,5609,2753,6686] 641x1077 mem=2 {'text': 1, 'paragraph_title': 1} + R 75 H bbox=[91,6012,688,6640] 597x628 mem=2 {'doc_title': 1, 'text': 1} + R 79 P bbox=[772,6082,2049,6637] 1277x555 mem=3 {'text': 2, 'paragraph_title': 1} + R 98 H bbox=[2793,6114,4114,6696] 1321x582 mem=3 {'doc_title': 1, 'text': 2} + >>> image-only orphans REMAINING: [] + 📐 Sakshi headline SYNTH: R100 doc_title [2126, 3353, 4097, 3452] (un-boxed display headline band, ink=0.35) + 📐 Sakshi headline SYNTH: R99 doc_title [763, 3357, 2068, 3452] (un-boxed display headline band, ink=0.30) + 📐 Sakshi headline WIDEN: R52 [1182, 1067, 2738, 1327] → [688, 1067, 2738, 1327] (un-boxed kicker ink alongside) + 🗞️ Masthead: dropped 5 Sakshi top-band region(s) [59, 62, 63, 64, 71] (above first headline y=1049) + +=== page 2: 17 crops, 11 datelines === + R 82 H bbox=[777,474,2801,2212] 2024x1738 mem=20 {'text': 7, 'image': 8, 'figure_title': 4, 'doc_title': 1} + R127 H bbox=[2755,485,4086,2232] 1331x1747 mem=17 {'text': 8, 'image': 6, 'figure_title': 2, 'doc_title': 1} + R129 P bbox=[95,499,741,2209] 646x1710 mem=6 {'paragraph_title': 1, 'image': 1, 'text': 3, 'figure_title': 1} + R118 H bbox=[2795,2298,4115,2957] 1320x659 mem=5 {'text': 2, 'image': 2, 'doc_title': 1} + R 91 H bbox=[72,2313,707,3101] 635x788 mem=5 {'text': 2, 'image': 1, 'doc_title': 1, 'figure_title': 1} + R 41 H bbox=[753,2316,2753,3600] 2000x1284 mem=11 {'text': 9, 'doc_title': 1, 'image': 1} + R 3 DL bbox=[2761,2958,3467,4049] 706x1091 mem=3 {'text': 1, 'image': 1, 'figure_title': 1} + R 14 DL bbox=[3475,3022,4115,3999] 640x977 mem=4 {'text': 2, 'image': 1, 'paragraph_title': 1} + R113 H bbox=[72,3129,709,4641] 637x1512 mem=5 {'doc_title': 1, 'text': 3, 'image': 1} + R 99 H bbox=[752,3656,2071,4649] 1319x993 mem=5 {'image': 1, 'text': 3, 'doc_title': 1} + R 88 P bbox=[2113,3658,2752,4643] 639x985 mem=4 {'text': 2, 'image': 1, 'paragraph_title': 1} + R 70 H bbox=[2760,4057,4115,5702] 1355x1645 mem=11 {'doc_title': 1, 'text': 9, 'image': 1} + R 93 H bbox=[1216,4710,2752,5939] 1536x1229 mem=10 {'text': 5, 'image': 1, 'doc_title': 1, 'paragraph_title': 2, 'figure_title': 1} + R136 P bbox=[95,4735,1161,5529] 1066x794 mem=3 {'paragraph_title': 1, 'text': 1, 'image': 1} + R 44 F bbox=[650,4979,1128,5185] 478x206 mem=1 {'text': 1} + R 92 F bbox=[653,5252,1127,5744] 474x492 mem=3 {'text': 2, 'paragraph_title': 1} + R111 P bbox=[120,5618,603,6628] 483x1010 mem=3 {'paragraph_title': 1, 'text': 2} + R 98 H bbox=[2798,5710,4115,6696] 1317x986 mem=6 {'doc_title': 1, 'text': 4, 'image': 1} + R 90 F bbox=[685,5924,1099,6145] 414x221 mem=1 {'text': 1} + R 77 H bbox=[1147,5969,2752,6682] 1605x713 mem=4 {'text': 3, 'doc_title': 1} + R110 F bbox=[668,6258,1106,6617] 438x359 mem=1 {'text': 1} + >>> image-only orphans REMAINING: [] + ─ Sakshi rule-split: R41 → 3 bands at rules [1066, 2252] + ─ Sakshi rule-split: R74 → 2 bands at rules [1066] + 📐 Sakshi headline WIDEN: R77 [1217, 5969, 2690, 6109] → [1147, 5969, 2690, 6109] (un-boxed kicker ink alongside) + 🖼 Orphan photo: R38 → article R127 (an image must belong to an article) + 🖼 Orphan photo: R61 → article R82 (an image must belong to an article) + 🗞️ Masthead: dropped 4 Sakshi top-band region(s) [68, 71, 84, 104] (above first headline y=499) + +DONE -> output/Sakshi_Jangaon_District_20260520_20260609_210104/all_article_crops_review diff --git a/_sak_oldrun3.log b/_sak_oldrun3.log new file mode 100644 index 0000000..a225e8d --- /dev/null +++ b/_sak_oldrun3.log @@ -0,0 +1,55 @@ + +=== page 1: 16 crops, 16 datelines === + R 55 H bbox=[2763,1049,4114,2732] 1351x1683 mem=12 {'text': 9, 'image': 1, 'doc_title': 1, 'paragraph_title': 1} + R 52 H bbox=[688,1067,2755,3314] 2067x2247 mem=17 {'text': 9, 'image': 2, 'paragraph_title': 5, 'doc_title': 1} + R 83 H bbox=[92,1079,686,2507] 594x1428 mem=4 {'doc_title': 1, 'text': 1, 'image': 1, 'paragraph_title': 1} + R 72 H bbox=[92,2536,685,3404] 593x868 mem=2 {'doc_title': 1, 'text': 1} + R 77 H bbox=[2793,2740,4114,3303] 1321x563 mem=4 {'doc_title': 1, 'text': 1, 'image': 1, 'figure_title': 1} + R100 H bbox=[2113,3353,4113,4297] 2000x944 mem=7 {'doc_title': 1, 'text': 5, 'image': 1} + R 99 H bbox=[750,3357,2069,4605] 1319x1248 mem=8 {'doc_title': 1, 'text': 5, 'image': 1, 'figure_title': 1} + R 67 H bbox=[92,3432,686,4743] 594x1311 mem=3 {'doc_title': 1, 'text': 1, 'image': 1} + R 50 H bbox=[2112,4357,2752,5581] 640x1224 mem=5 {'text': 3, 'image': 1, 'doc_title': 1} + R 54 H bbox=[2793,4368,4114,6094] 1321x1726 mem=9 {'doc_title': 1, 'text': 6, 'image': 1, 'paragraph_title': 1} + R 78 H bbox=[749,4622,2070,6049] 1321x1427 mem=7 {'doc_title': 1, 'text': 6} + R 9 DL bbox=[92,4771,685,5982] 593x1211 mem=2 {'text': 1, 'paragraph_title': 1} + R 5 DL bbox=[2112,5609,2753,6686] 641x1077 mem=2 {'text': 1, 'paragraph_title': 1} + R 75 H bbox=[91,6012,688,6640] 597x628 mem=2 {'doc_title': 1, 'text': 1} + R 79 P bbox=[772,6082,2049,6637] 1277x555 mem=3 {'text': 2, 'paragraph_title': 1} + R 98 H bbox=[2793,6114,4114,6696] 1321x582 mem=3 {'doc_title': 1, 'text': 2} + >>> image-only orphans REMAINING: [] + 📐 Sakshi headline SYNTH: R100 doc_title [2126, 3353, 4097, 3452] (un-boxed display headline band, ink=0.35) + 📐 Sakshi headline SYNTH: R99 doc_title [763, 3357, 2068, 3452] (un-boxed display headline band, ink=0.30) + 📐 Sakshi headline WIDEN: R52 [1182, 1067, 2738, 1327] → [688, 1067, 2738, 1327] (un-boxed kicker ink alongside) + 🗞️ Masthead: dropped 5 Sakshi top-band region(s) [59, 62, 63, 64, 71] (above first headline y=1049) + +=== page 2: 17 crops, 11 datelines === + R 82 H bbox=[777,474,2801,2212] 2024x1738 mem=20 {'text': 7, 'image': 8, 'figure_title': 4, 'doc_title': 1} + R127 H bbox=[2755,485,4086,2232] 1331x1747 mem=17 {'text': 8, 'image': 6, 'figure_title': 2, 'doc_title': 1} + R129 P bbox=[95,499,741,2209] 646x1710 mem=6 {'paragraph_title': 1, 'image': 1, 'text': 3, 'figure_title': 1} + R118 H bbox=[2795,2298,4115,2957] 1320x659 mem=5 {'text': 2, 'image': 2, 'doc_title': 1} + R 91 H bbox=[72,2313,707,3101] 635x788 mem=5 {'text': 2, 'image': 1, 'doc_title': 1, 'figure_title': 1} + R 41 H bbox=[753,2316,2753,3600] 2000x1284 mem=11 {'text': 9, 'doc_title': 1, 'image': 1} + R 3 DL bbox=[2761,2958,3467,4049] 706x1091 mem=3 {'text': 1, 'image': 1, 'figure_title': 1} + R 14 DL bbox=[3475,3022,4115,3999] 640x977 mem=4 {'text': 2, 'image': 1, 'paragraph_title': 1} + R113 H bbox=[72,3129,709,4641] 637x1512 mem=5 {'doc_title': 1, 'text': 3, 'image': 1} + R 99 H bbox=[752,3656,2071,4649] 1319x993 mem=5 {'image': 1, 'text': 3, 'doc_title': 1} + R 88 P bbox=[2113,3658,2752,4643] 639x985 mem=4 {'text': 2, 'image': 1, 'paragraph_title': 1} + R 70 H bbox=[2760,4057,4115,5702] 1355x1645 mem=11 {'doc_title': 1, 'text': 9, 'image': 1} + R 93 H bbox=[1216,4710,2752,5939] 1536x1229 mem=10 {'text': 5, 'image': 1, 'doc_title': 1, 'paragraph_title': 2, 'figure_title': 1} + R136 P bbox=[95,4735,1161,5529] 1066x794 mem=3 {'paragraph_title': 1, 'text': 1, 'image': 1} + R 44 F bbox=[650,4979,1128,5185] 478x206 mem=1 {'text': 1} + R 92 F bbox=[653,5252,1127,5744] 474x492 mem=3 {'text': 2, 'paragraph_title': 1} + R111 P bbox=[120,5618,603,6628] 483x1010 mem=3 {'paragraph_title': 1, 'text': 2} + R 98 H bbox=[2798,5710,4115,6696] 1317x986 mem=6 {'doc_title': 1, 'text': 4, 'image': 1} + R 90 F bbox=[685,5924,1099,6145] 414x221 mem=1 {'text': 1} + R 77 H bbox=[1147,5969,2752,6682] 1605x713 mem=4 {'text': 3, 'doc_title': 1} + R110 F bbox=[668,6258,1106,6617] 438x359 mem=1 {'text': 1} + >>> image-only orphans REMAINING: [] + ─ Sakshi rule-split: R41 → 3 bands at rules [1066, 2252] + ─ Sakshi rule-split: R74 → 2 bands at rules [1066] + 📐 Sakshi headline WIDEN: R77 [1217, 5969, 2690, 6109] → [1147, 5969, 2690, 6109] (un-boxed kicker ink alongside) + 🖼 Orphan photo: R38 → article R127 (an image must belong to an article) + 🖼 Orphan photo: R61 → article R82 (an image must belong to an article) + 🗞️ Masthead: dropped 4 Sakshi top-band region(s) [68, 71, 84, 104] (above first headline y=499) + +DONE -> output/Sakshi_Jangaon_District_20260520_20260609_210104/all_article_crops_review diff --git a/_sak_oldrun4.log b/_sak_oldrun4.log new file mode 100644 index 0000000..a225e8d --- /dev/null +++ b/_sak_oldrun4.log @@ -0,0 +1,55 @@ + +=== page 1: 16 crops, 16 datelines === + R 55 H bbox=[2763,1049,4114,2732] 1351x1683 mem=12 {'text': 9, 'image': 1, 'doc_title': 1, 'paragraph_title': 1} + R 52 H bbox=[688,1067,2755,3314] 2067x2247 mem=17 {'text': 9, 'image': 2, 'paragraph_title': 5, 'doc_title': 1} + R 83 H bbox=[92,1079,686,2507] 594x1428 mem=4 {'doc_title': 1, 'text': 1, 'image': 1, 'paragraph_title': 1} + R 72 H bbox=[92,2536,685,3404] 593x868 mem=2 {'doc_title': 1, 'text': 1} + R 77 H bbox=[2793,2740,4114,3303] 1321x563 mem=4 {'doc_title': 1, 'text': 1, 'image': 1, 'figure_title': 1} + R100 H bbox=[2113,3353,4113,4297] 2000x944 mem=7 {'doc_title': 1, 'text': 5, 'image': 1} + R 99 H bbox=[750,3357,2069,4605] 1319x1248 mem=8 {'doc_title': 1, 'text': 5, 'image': 1, 'figure_title': 1} + R 67 H bbox=[92,3432,686,4743] 594x1311 mem=3 {'doc_title': 1, 'text': 1, 'image': 1} + R 50 H bbox=[2112,4357,2752,5581] 640x1224 mem=5 {'text': 3, 'image': 1, 'doc_title': 1} + R 54 H bbox=[2793,4368,4114,6094] 1321x1726 mem=9 {'doc_title': 1, 'text': 6, 'image': 1, 'paragraph_title': 1} + R 78 H bbox=[749,4622,2070,6049] 1321x1427 mem=7 {'doc_title': 1, 'text': 6} + R 9 DL bbox=[92,4771,685,5982] 593x1211 mem=2 {'text': 1, 'paragraph_title': 1} + R 5 DL bbox=[2112,5609,2753,6686] 641x1077 mem=2 {'text': 1, 'paragraph_title': 1} + R 75 H bbox=[91,6012,688,6640] 597x628 mem=2 {'doc_title': 1, 'text': 1} + R 79 P bbox=[772,6082,2049,6637] 1277x555 mem=3 {'text': 2, 'paragraph_title': 1} + R 98 H bbox=[2793,6114,4114,6696] 1321x582 mem=3 {'doc_title': 1, 'text': 2} + >>> image-only orphans REMAINING: [] + 📐 Sakshi headline SYNTH: R100 doc_title [2126, 3353, 4097, 3452] (un-boxed display headline band, ink=0.35) + 📐 Sakshi headline SYNTH: R99 doc_title [763, 3357, 2068, 3452] (un-boxed display headline band, ink=0.30) + 📐 Sakshi headline WIDEN: R52 [1182, 1067, 2738, 1327] → [688, 1067, 2738, 1327] (un-boxed kicker ink alongside) + 🗞️ Masthead: dropped 5 Sakshi top-band region(s) [59, 62, 63, 64, 71] (above first headline y=1049) + +=== page 2: 17 crops, 11 datelines === + R 82 H bbox=[777,474,2801,2212] 2024x1738 mem=20 {'text': 7, 'image': 8, 'figure_title': 4, 'doc_title': 1} + R127 H bbox=[2755,485,4086,2232] 1331x1747 mem=17 {'text': 8, 'image': 6, 'figure_title': 2, 'doc_title': 1} + R129 P bbox=[95,499,741,2209] 646x1710 mem=6 {'paragraph_title': 1, 'image': 1, 'text': 3, 'figure_title': 1} + R118 H bbox=[2795,2298,4115,2957] 1320x659 mem=5 {'text': 2, 'image': 2, 'doc_title': 1} + R 91 H bbox=[72,2313,707,3101] 635x788 mem=5 {'text': 2, 'image': 1, 'doc_title': 1, 'figure_title': 1} + R 41 H bbox=[753,2316,2753,3600] 2000x1284 mem=11 {'text': 9, 'doc_title': 1, 'image': 1} + R 3 DL bbox=[2761,2958,3467,4049] 706x1091 mem=3 {'text': 1, 'image': 1, 'figure_title': 1} + R 14 DL bbox=[3475,3022,4115,3999] 640x977 mem=4 {'text': 2, 'image': 1, 'paragraph_title': 1} + R113 H bbox=[72,3129,709,4641] 637x1512 mem=5 {'doc_title': 1, 'text': 3, 'image': 1} + R 99 H bbox=[752,3656,2071,4649] 1319x993 mem=5 {'image': 1, 'text': 3, 'doc_title': 1} + R 88 P bbox=[2113,3658,2752,4643] 639x985 mem=4 {'text': 2, 'image': 1, 'paragraph_title': 1} + R 70 H bbox=[2760,4057,4115,5702] 1355x1645 mem=11 {'doc_title': 1, 'text': 9, 'image': 1} + R 93 H bbox=[1216,4710,2752,5939] 1536x1229 mem=10 {'text': 5, 'image': 1, 'doc_title': 1, 'paragraph_title': 2, 'figure_title': 1} + R136 P bbox=[95,4735,1161,5529] 1066x794 mem=3 {'paragraph_title': 1, 'text': 1, 'image': 1} + R 44 F bbox=[650,4979,1128,5185] 478x206 mem=1 {'text': 1} + R 92 F bbox=[653,5252,1127,5744] 474x492 mem=3 {'text': 2, 'paragraph_title': 1} + R111 P bbox=[120,5618,603,6628] 483x1010 mem=3 {'paragraph_title': 1, 'text': 2} + R 98 H bbox=[2798,5710,4115,6696] 1317x986 mem=6 {'doc_title': 1, 'text': 4, 'image': 1} + R 90 F bbox=[685,5924,1099,6145] 414x221 mem=1 {'text': 1} + R 77 H bbox=[1147,5969,2752,6682] 1605x713 mem=4 {'text': 3, 'doc_title': 1} + R110 F bbox=[668,6258,1106,6617] 438x359 mem=1 {'text': 1} + >>> image-only orphans REMAINING: [] + ─ Sakshi rule-split: R41 → 3 bands at rules [1066, 2252] + ─ Sakshi rule-split: R74 → 2 bands at rules [1066] + 📐 Sakshi headline WIDEN: R77 [1217, 5969, 2690, 6109] → [1147, 5969, 2690, 6109] (un-boxed kicker ink alongside) + 🖼 Orphan photo: R38 → article R127 (an image must belong to an article) + 🖼 Orphan photo: R61 → article R82 (an image must belong to an article) + 🗞️ Masthead: dropped 4 Sakshi top-band region(s) [68, 71, 84, 104] (above first headline y=499) + +DONE -> output/Sakshi_Jangaon_District_20260520_20260609_210104/all_article_crops_review diff --git a/_sak_val7.log b/_sak_val7.log new file mode 100644 index 0000000..a225e8d --- /dev/null +++ b/_sak_val7.log @@ -0,0 +1,55 @@ + +=== page 1: 16 crops, 16 datelines === + R 55 H bbox=[2763,1049,4114,2732] 1351x1683 mem=12 {'text': 9, 'image': 1, 'doc_title': 1, 'paragraph_title': 1} + R 52 H bbox=[688,1067,2755,3314] 2067x2247 mem=17 {'text': 9, 'image': 2, 'paragraph_title': 5, 'doc_title': 1} + R 83 H bbox=[92,1079,686,2507] 594x1428 mem=4 {'doc_title': 1, 'text': 1, 'image': 1, 'paragraph_title': 1} + R 72 H bbox=[92,2536,685,3404] 593x868 mem=2 {'doc_title': 1, 'text': 1} + R 77 H bbox=[2793,2740,4114,3303] 1321x563 mem=4 {'doc_title': 1, 'text': 1, 'image': 1, 'figure_title': 1} + R100 H bbox=[2113,3353,4113,4297] 2000x944 mem=7 {'doc_title': 1, 'text': 5, 'image': 1} + R 99 H bbox=[750,3357,2069,4605] 1319x1248 mem=8 {'doc_title': 1, 'text': 5, 'image': 1, 'figure_title': 1} + R 67 H bbox=[92,3432,686,4743] 594x1311 mem=3 {'doc_title': 1, 'text': 1, 'image': 1} + R 50 H bbox=[2112,4357,2752,5581] 640x1224 mem=5 {'text': 3, 'image': 1, 'doc_title': 1} + R 54 H bbox=[2793,4368,4114,6094] 1321x1726 mem=9 {'doc_title': 1, 'text': 6, 'image': 1, 'paragraph_title': 1} + R 78 H bbox=[749,4622,2070,6049] 1321x1427 mem=7 {'doc_title': 1, 'text': 6} + R 9 DL bbox=[92,4771,685,5982] 593x1211 mem=2 {'text': 1, 'paragraph_title': 1} + R 5 DL bbox=[2112,5609,2753,6686] 641x1077 mem=2 {'text': 1, 'paragraph_title': 1} + R 75 H bbox=[91,6012,688,6640] 597x628 mem=2 {'doc_title': 1, 'text': 1} + R 79 P bbox=[772,6082,2049,6637] 1277x555 mem=3 {'text': 2, 'paragraph_title': 1} + R 98 H bbox=[2793,6114,4114,6696] 1321x582 mem=3 {'doc_title': 1, 'text': 2} + >>> image-only orphans REMAINING: [] + 📐 Sakshi headline SYNTH: R100 doc_title [2126, 3353, 4097, 3452] (un-boxed display headline band, ink=0.35) + 📐 Sakshi headline SYNTH: R99 doc_title [763, 3357, 2068, 3452] (un-boxed display headline band, ink=0.30) + 📐 Sakshi headline WIDEN: R52 [1182, 1067, 2738, 1327] → [688, 1067, 2738, 1327] (un-boxed kicker ink alongside) + 🗞️ Masthead: dropped 5 Sakshi top-band region(s) [59, 62, 63, 64, 71] (above first headline y=1049) + +=== page 2: 17 crops, 11 datelines === + R 82 H bbox=[777,474,2801,2212] 2024x1738 mem=20 {'text': 7, 'image': 8, 'figure_title': 4, 'doc_title': 1} + R127 H bbox=[2755,485,4086,2232] 1331x1747 mem=17 {'text': 8, 'image': 6, 'figure_title': 2, 'doc_title': 1} + R129 P bbox=[95,499,741,2209] 646x1710 mem=6 {'paragraph_title': 1, 'image': 1, 'text': 3, 'figure_title': 1} + R118 H bbox=[2795,2298,4115,2957] 1320x659 mem=5 {'text': 2, 'image': 2, 'doc_title': 1} + R 91 H bbox=[72,2313,707,3101] 635x788 mem=5 {'text': 2, 'image': 1, 'doc_title': 1, 'figure_title': 1} + R 41 H bbox=[753,2316,2753,3600] 2000x1284 mem=11 {'text': 9, 'doc_title': 1, 'image': 1} + R 3 DL bbox=[2761,2958,3467,4049] 706x1091 mem=3 {'text': 1, 'image': 1, 'figure_title': 1} + R 14 DL bbox=[3475,3022,4115,3999] 640x977 mem=4 {'text': 2, 'image': 1, 'paragraph_title': 1} + R113 H bbox=[72,3129,709,4641] 637x1512 mem=5 {'doc_title': 1, 'text': 3, 'image': 1} + R 99 H bbox=[752,3656,2071,4649] 1319x993 mem=5 {'image': 1, 'text': 3, 'doc_title': 1} + R 88 P bbox=[2113,3658,2752,4643] 639x985 mem=4 {'text': 2, 'image': 1, 'paragraph_title': 1} + R 70 H bbox=[2760,4057,4115,5702] 1355x1645 mem=11 {'doc_title': 1, 'text': 9, 'image': 1} + R 93 H bbox=[1216,4710,2752,5939] 1536x1229 mem=10 {'text': 5, 'image': 1, 'doc_title': 1, 'paragraph_title': 2, 'figure_title': 1} + R136 P bbox=[95,4735,1161,5529] 1066x794 mem=3 {'paragraph_title': 1, 'text': 1, 'image': 1} + R 44 F bbox=[650,4979,1128,5185] 478x206 mem=1 {'text': 1} + R 92 F bbox=[653,5252,1127,5744] 474x492 mem=3 {'text': 2, 'paragraph_title': 1} + R111 P bbox=[120,5618,603,6628] 483x1010 mem=3 {'paragraph_title': 1, 'text': 2} + R 98 H bbox=[2798,5710,4115,6696] 1317x986 mem=6 {'doc_title': 1, 'text': 4, 'image': 1} + R 90 F bbox=[685,5924,1099,6145] 414x221 mem=1 {'text': 1} + R 77 H bbox=[1147,5969,2752,6682] 1605x713 mem=4 {'text': 3, 'doc_title': 1} + R110 F bbox=[668,6258,1106,6617] 438x359 mem=1 {'text': 1} + >>> image-only orphans REMAINING: [] + ─ Sakshi rule-split: R41 → 3 bands at rules [1066, 2252] + ─ Sakshi rule-split: R74 → 2 bands at rules [1066] + 📐 Sakshi headline WIDEN: R77 [1217, 5969, 2690, 6109] → [1147, 5969, 2690, 6109] (un-boxed kicker ink alongside) + 🖼 Orphan photo: R38 → article R127 (an image must belong to an article) + 🖼 Orphan photo: R61 → article R82 (an image must belong to an article) + 🗞️ Masthead: dropped 4 Sakshi top-band region(s) [68, 71, 84, 104] (above first headline y=499) + +DONE -> output/Sakshi_Jangaon_District_20260520_20260609_210104/all_article_crops_review diff --git a/_sim71.py b/_sim71.py new file mode 100644 index 0000000..daa0b34 --- /dev/null +++ b/_sim71.py @@ -0,0 +1,50 @@ +"""Sandbox simulation of the Sakshi Jangaon page-1/2 crop run. +Telugu tesseract is unavailable here, so dateline starts are INJECTED with the +exact anchor set the real 20260609_210104 run found (DL anchors from the saved +crop filenames). Usage: python3 _sim71.py +Writes block table to stdout and crops to output/.../_sim_crops_/.""" +import sys, types +sys.modules.setdefault("fitz", types.ModuleType("fitz")) +import os, tempfile, contextlib, io, json, shutil +tempfile.tempdir = os.path.abspath("_tess_tmp"); os.environ["TMPDIR"] = tempfile.tempdir +os.makedirs("_tess_tmp", exist_ok=True) +from pathlib import Path +import smart_extractor as se +from _lines import separator_barriers + +TAG = sys.argv[1] if len(sys.argv) > 1 else "base" +RUN = Path("output/Sakshi_Jangaon_District_20260520_20260609_210104"); D = RUN / "pages" +OUT = RUN / f"_sim_crops_{TAG}"; P = "sakshi" +DL_IDS = {1: [71, 11, 4, 3, 9, 5], 2: []} # real run's headline-less datelines +if OUT.exists(): shutil.rmtree(OUT) + +for pg in (1, 2): + png = str(D / f"page_{pg:03d}.png") + regs = json.loads((D / f"page_{pg:03d}.regions.json").read_text())["regions"] + buf = io.StringIO() + with contextlib.redirect_stdout(buf): + regs = se.dedup_overlapping_regions(regs) + regs = se.merge_stacked_title_lines(regs, png) + regs = se.drop_masthead_regions(regs, P, pg, png) + regs = se.drop_classifieds_regions(regs, png) + regs = se.tag_legal_notices(regs, png) + regs = se.promote_paragraph_titles(regs, png, P) + regs = se.mark_bullet_titles(regs, png) + rm = {r["id"]: r for r in regs} + ds = [{"region_id": i, "dateline": "sim:", "bbox": rm[i]["bbox"], + "headline_region": None} for i in DL_IDS[pg] if i in rm] + bars = separator_barriers(png, regs) + blocks = se.cluster_all_article_blocks(regs, ds, sep_lines=bars) + n = se.crop_all_article_blocks(png, regs, str(OUT), pg, + dateline_starts=ds, sep_lines=bars) + print(f"=== page {pg}: {n} crops, {len(ds)} injected datelines ===") + for b in sorted(blocks, key=lambda b: (b["bbox"][1], b["bbox"][0])): + x1, y1, x2, y2 = b["bbox"] + types = {} + for m in b["members"]: + t = rm.get(m, {}).get("type", "?"); types[t] = types.get(t, 0) + 1 + print(f" R{b['anchor_id']:>3} {b['kind']} bbox=[{x1},{y1},{x2},{y2}] " + f"{x2-x1}x{y2-y1} mem={len(b['members'])} {types}") + for l in buf.getvalue().splitlines(): + if "Masthead" in l: print(" ", l.strip()) +print("DONE ->", OUT) diff --git a/_sim_base.log b/_sim_base.log new file mode 100644 index 0000000..e69de29 diff --git a/_tess_tmp/tess_14ktolcb b/_tess_tmp/tess_14ktolcb new file mode 100644 index 0000000..e69de29 diff --git a/_tess_tmp/tess_14ktolcb.txt b/_tess_tmp/tess_14ktolcb.txt new file mode 100644 index 0000000..e69de29 diff --git a/_tess_tmp/tess_14ktolcb_input.PNG b/_tess_tmp/tess_14ktolcb_input.PNG new file mode 100644 index 0000000..65f39c9 Binary files /dev/null and b/_tess_tmp/tess_14ktolcb_input.PNG differ diff --git a/_tess_tmp/tess_1511jiz9 b/_tess_tmp/tess_1511jiz9 new file mode 100644 index 0000000..e69de29 diff --git a/_tess_tmp/tess_1511jiz9.txt b/_tess_tmp/tess_1511jiz9.txt new file mode 100644 index 0000000..71dca2d --- /dev/null +++ b/_tess_tmp/tess_1511jiz9.txt @@ -0,0 +1,18 @@ +ADEA ASEXE shotloo Sr crsrAs +Boas dof datwera, Ads Soseor +weak? Hiirtht dod Siro. Kew0e vOt +BORD arLBSCOS TS SUNG HOTS +SS SHS EH ert. SYS OED HU, +Brag SomoSodS Dweakl SASHyw +TY. 80 OY GSo DHWBwHS S%So Bra +oer BS BP wid Sard dren Sewn +Sndke dSKo Hnowt, & Ysos! +BraS Zodoh soerr Gardsivg +DUS HEH OZ dbx Som Soebrey +Perle SH Spor Sad Seared. ac +GAS SOShowsh dso SO advert +Sto, SEBS SrerS ZokoH Sodh ws +B%o BLISS SS SyPHHS Ser +UPSPAE SOY. wisp Ero HS Koha +SE SK0. Stot dSthen Kar aeh'B + \ No newline at end of file diff --git a/_tess_tmp/tess_1511jiz9_input.PNG b/_tess_tmp/tess_1511jiz9_input.PNG new file mode 100644 index 0000000..75e4cf9 Binary files /dev/null and b/_tess_tmp/tess_1511jiz9_input.PNG differ diff --git a/_tess_tmp/tess_1onw_da9 b/_tess_tmp/tess_1onw_da9 new file mode 100644 index 0000000..e69de29 diff --git a/_tess_tmp/tess_1onw_da9.txt b/_tess_tmp/tess_1onw_da9.txt new file mode 100644 index 0000000..58d10c5 --- /dev/null +++ b/_tess_tmp/tess_1onw_da9.txt @@ -0,0 +1,12 @@ +BS: GSS Sel Ste (ohowS) +Srghoh SES wQos*s eer wpideSo +& NODA. 2025 ws8 LeeS* Snifo Tox +©%8 S0508 ZHf6* ToXoT09 s0A8 +SeghoF SQ erp, 90.59 wo Karciws +8008. 20m 72,194 0 LY soshoh +wokofire 408. warshdS(SgE ssp +espe) Soms 2,33,75 Tre, 31.93 wdor +608. vert & 3018 5 d45* HOR ay +Segboh SOB 3.214 Sow wonsen +weberthen BDar%. + \ No newline at end of file diff --git a/_tess_tmp/tess_1onw_da9_input.PNG b/_tess_tmp/tess_1onw_da9_input.PNG new file mode 100644 index 0000000..aa6f326 Binary files /dev/null and b/_tess_tmp/tess_1onw_da9_input.PNG differ diff --git a/_tess_tmp/tess_1sf6wilh b/_tess_tmp/tess_1sf6wilh new file mode 100644 index 0000000..e69de29 diff --git a/_tess_tmp/tess_1sf6wilh.txt b/_tess_tmp/tess_1sf6wilh.txt new file mode 100644 index 0000000..7a60fb8 --- /dev/null +++ b/_tess_tmp/tess_1sf6wilh.txt @@ -0,0 +1,18 @@ +BPSASOU: DeogS adFgroo wdhen L608 +woxso maaan. oY SAijo mddos +Dio AGSL0B. wd Gesey wwe od: +BOE nO AE SHE wSsr¥o ¥Oy0N0 +&. SiSHoE NDS Gadag we nod +CRE AHIYHasryon. wond Neds’ +PH SrSo nddo woo SASS ors +Gets. OY aro{go SpSaras Higher, OF +ROARS, DAS PASS, Rrcsorer Bo S*ar +doS HS Sy b3de Se, Smaysd +aPOE URES DS WES BdaryWS SEyhe +S850 wdgsore Boorres Ga GE & Hearrs +RS Yond OWGE doryim Sod. oHo8 +wont DeiegS Soxseo ae ads’, bf es +8, éé a ALa6, dé 25K4495 aod +Bo Exp 0% J actyso Sodd davrd) s08. +BaSSoo8 nado BE 4408. 3&% &%7.0 + \ No newline at end of file diff --git a/_tess_tmp/tess_1sf6wilh_input.PNG b/_tess_tmp/tess_1sf6wilh_input.PNG new file mode 100644 index 0000000..68b1fe4 Binary files /dev/null and b/_tess_tmp/tess_1sf6wilh_input.PNG differ diff --git a/_tess_tmp/tess_3ayv267z b/_tess_tmp/tess_3ayv267z new file mode 100644 index 0000000..e69de29 diff --git a/_tess_tmp/tess_3ayv267z.txt b/_tess_tmp/tess_3ayv267z.txt new file mode 100644 index 0000000..71dca2d --- /dev/null +++ b/_tess_tmp/tess_3ayv267z.txt @@ -0,0 +1,18 @@ +ADEA ASEXE shotloo Sr crsrAs +Boas dof datwera, Ads Soseor +weak? Hiirtht dod Siro. Kew0e vOt +BORD arLBSCOS TS SUNG HOTS +SS SHS EH ert. SYS OED HU, +Brag SomoSodS Dweakl SASHyw +TY. 80 OY GSo DHWBwHS S%So Bra +oer BS BP wid Sard dren Sewn +Sndke dSKo Hnowt, & Ysos! +BraS Zodoh soerr Gardsivg +DUS HEH OZ dbx Som Soebrey +Perle SH Spor Sad Seared. ac +GAS SOShowsh dso SO advert +Sto, SEBS SrerS ZokoH Sodh ws +B%o BLISS SS SyPHHS Ser +UPSPAE SOY. wisp Ero HS Koha +SE SK0. Stot dSthen Kar aeh'B + \ No newline at end of file diff --git a/_tess_tmp/tess_3ayv267z_input.PNG b/_tess_tmp/tess_3ayv267z_input.PNG new file mode 100644 index 0000000..75e4cf9 Binary files /dev/null and b/_tess_tmp/tess_3ayv267z_input.PNG differ diff --git a/_tess_tmp/tess_4yfor0nu b/_tess_tmp/tess_4yfor0nu new file mode 100644 index 0000000..e69de29 diff --git a/_tess_tmp/tess_4yfor0nu.txt b/_tess_tmp/tess_4yfor0nu.txt new file mode 100644 index 0000000..dc1bf58 --- /dev/null +++ b/_tess_tmp/tess_4yfor0nu.txt @@ -0,0 +1,19 @@ +Sorisarote: SONS Srsowird dir +wdho OY Siro Sho vbsrthen Go +wy GS ONSey BOB. donBI*G +oped wostren adie Stgods +DRaho BORod. ak DoxRebo sod by +G0dS GSS FONE DoF, HAtyeo Dory», +ProreosD Aor BOAS Swydrcen, +RES BL FORH, Divwr pasos aided +SEfoh wOwrcre Berk Ago dasep +Seirardo. 180 Sars! sorgsties wxosoe +©, BRAS Sbxfeo Ntglrodocss Bots +BrHoxood SSarigen Syoxsew BOLO +8. wr Goss armen Sorl¥arwo & +Ahoo AKowo Sk O50 waisr¥o 6%) +ey BSrardo. Bots 2S sossoWo 5 +Kotor PSE darosoiiyee Seord¥od. Sao +oO S8orsed woerss 180 See’ wh +DO dxoen Grd JasD) dotwod. + \ No newline at end of file diff --git a/_tess_tmp/tess_4yfor0nu_input.PNG b/_tess_tmp/tess_4yfor0nu_input.PNG new file mode 100644 index 0000000..e787aed Binary files /dev/null and b/_tess_tmp/tess_4yfor0nu_input.PNG differ diff --git a/_tess_tmp/tess_5o3xjw_o b/_tess_tmp/tess_5o3xjw_o new file mode 100644 index 0000000..e69de29 diff --git a/_tess_tmp/tess_5o3xjw_o.txt b/_tess_tmp/tess_5o3xjw_o.txt new file mode 100644 index 0000000..e69de29 diff --git a/_tess_tmp/tess_5o3xjw_o_input.PNG b/_tess_tmp/tess_5o3xjw_o_input.PNG new file mode 100644 index 0000000..d3ee835 Binary files /dev/null and b/_tess_tmp/tess_5o3xjw_o_input.PNG differ diff --git a/_tess_tmp/tess_6mykaz2z b/_tess_tmp/tess_6mykaz2z new file mode 100644 index 0000000..e69de29 diff --git a/_tess_tmp/tess_6mykaz2z.txt b/_tess_tmp/tess_6mykaz2z.txt new file mode 100644 index 0000000..c14da0d --- /dev/null +++ b/_tess_tmp/tess_6mykaz2z.txt @@ -0,0 +1,19 @@ +Beso a6 JUS 44 D0dH Sowb 88, +Brto & So 316% addo Gkcd HO +Bao. DyS SoYor LOHR asyg +adder 268A visio Sty sodasod. +nddo woo BASS PSGe. axspes +088 Ch Be Exons 408. Be &%5y0 +& Ad'yo Sodd der doemodd ws +SF DeogS wOsrcoen, SS'yoeo seRe, Ja +Exo FF TOs SENG Oy Ge +Bosses’ SEHSp wie Depo +worth, GE%roe bg IHS SHS dows +Daw. CAS ariglo, Sqvoosso Sve +Gigas Avfor 26A08. word 068 dake +Seren Tverd) soci. ctormsivrarshy +Sen Soda srensioreron. 08% dor +Son sds wad dciiowop syrorer +Shobayws. Gawss* arvigo adordsoors +Gos Ad Ser%o mObo2% SSO. + \ No newline at end of file diff --git a/_tess_tmp/tess_6mykaz2z_input.PNG b/_tess_tmp/tess_6mykaz2z_input.PNG new file mode 100644 index 0000000..da8f545 Binary files /dev/null and b/_tess_tmp/tess_6mykaz2z_input.PNG differ diff --git a/_tess_tmp/tess_71zcinaf b/_tess_tmp/tess_71zcinaf new file mode 100644 index 0000000..e69de29 diff --git a/_tess_tmp/tess_71zcinaf.txt b/_tess_tmp/tess_71zcinaf.txt new file mode 100644 index 0000000..e69de29 diff --git a/_tess_tmp/tess_71zcinaf_input.PNG b/_tess_tmp/tess_71zcinaf_input.PNG new file mode 100644 index 0000000..acb5162 Binary files /dev/null and b/_tess_tmp/tess_71zcinaf_input.PNG differ diff --git a/_tess_tmp/tess_80f9u6zp b/_tess_tmp/tess_80f9u6zp new file mode 100644 index 0000000..e69de29 diff --git a/_tess_tmp/tess_80f9u6zp.txt b/_tess_tmp/tess_80f9u6zp.txt new file mode 100644 index 0000000..4f6f9ed --- /dev/null +++ b/_tess_tmp/tess_80f9u6zp.txt @@ -0,0 +1,18 @@ +Domes: aS 81 Sov a8 sohergiar +09-1 Sxen > eS SHH wohorres +AaYDs LEcho BSO Somods wosrds +0% Slows. LosoosN dorfoss +(rm IPHS‘O HOB wrasro AQ So 25 +BOSD OH-1 SoSH Poy Sarows +Hor¥ardo wowten, woeryé, Gere’ +BOS? KO SOMO. wPSosirWorr +aWiOeyy Sop sUliro SYodme. se +QAS Show SoHE dor Sosio eS SHH wohorres +AaYDs LEcho BSO Somods wosrds +0% Slows. LosoosN dorfoss +(rm IPHS‘O HOB wrasro AQ So 25 +BOSD OH-1 SoSH Poy Sarows +Hor¥ardo wowten, woeryé, Gere’ +BOS? KO SOMO. wPSosirWorr +aWiOeyy Sop sUliro SYodme. se +QAS Show SoHE dor Sosio=2] + cls=[l.strip() for l in buf.getvalue().splitlines() if 'Classifieds:' in l] + o.write(f" pg{pg}: {len(blocks)} blocks | 2+dl={over} | {cls[0] if cls else 'no classifieds'}\n") + except Exception as e: + o.write(f" pg{pg}: ERROR {e}\n") +o.close();print("WROTE") diff --git a/_testcls.txt b/_testcls.txt new file mode 100644 index 0000000..e69de29 diff --git a/_trace76.py b/_trace76.py new file mode 100644 index 0000000..3153da2 --- /dev/null +++ b/_trace76.py @@ -0,0 +1,27 @@ +import sys, json, contextlib, io +sys.path.insert(0, ".") +from pathlib import Path +import smart_extractor as se +from _lines import separator_barriers + +D = Path("output/AndhraJyothi_Siddipet District_20260602_20260603_144919/pages") +png = D / "page_001.png" +regs0 = json.loads((D / "page_001.regions.json").read_text())["regions"] +with contextlib.redirect_stdout(io.StringIO()): + regs = se.drop_masthead_regions(regs0, "andhra_jyothi", 1, png) + ds = se.find_article_starts_by_dateline(regs, str(png), "andhra_jyothi") + bars = separator_barriers(str(png), regs) + bl = se.cluster_all_article_blocks(regs, dateline_starts=ds, sep_lines=bars) + +owner = {} +for b in bl: + for m in b["members"]: + owner[m] = b["anchor_id"] +for rid in (76, 68, 46): + print(f"R{rid} -> block anchor {owner.get(rid)}") + +# manual barrier check R32 vs R76/R68 +byid = {r["id"]: r for r in regs} +print("bars (full-width-ish) near y4227/4915:", [b for b in bars if b["y"] in (4227,4915) or 4200", owner.get(76), " R68 ->", owner.get(68)) diff --git a/_vwalls_test.py b/_vwalls_test.py new file mode 100644 index 0000000..949fa56 --- /dev/null +++ b/_vwalls_test.py @@ -0,0 +1,62 @@ +"""TEST (read-only): identify & map the BIG vertical lines on a page — the long column +dividers that separate articles — ignoring short verticals (in-article box edges, stray +strokes). Draws a blank-canvas 'vertical wall map' per page. No smart_extractor changes, +no API. Test on 1-2 pages only.""" +import os, tempfile, json +tempfile.tempdir = os.path.abspath("_tess_tmp"); os.environ["TMPDIR"] = tempfile.tempdir +os.makedirs("_tess_tmp", exist_ok=True) +from pathlib import Path +from PIL import Image, ImageDraw, ImageFont +from _lines import detect_separator_lines + +# A vertical line counts as a BIG article-separating wall when it runs at least this +# fraction of the page height. Column dividers span most of a column; box edges are short. +BIG_FRAC = 0.45 + +def big_vertical_walls(png_path, min_frac=BIG_FRAC): + res = detect_separator_lines(png_path) + W, H = res["size"] + walls = [] + for L in res["v"]: + y1, y2 = min(L["y1"], L["y2"]), max(L["y1"], L["y2"]) + length = y2 - y1 + if length >= min_frac * H: + walls.append({"x": (L["x1"] + L["x2"]) // 2, "y1": y1, "y2": y2, + "len": length, "frac": round(length / H, 2)}) + walls.sort(key=lambda w: w["x"]) + return walls, (W, H) + +def draw_walls(png_path, out_path): + walls, (W, H) = big_vertical_walls(png_path) + res = detect_separator_lines(png_path) # for the 'ignored' short ones + small = [L for L in res["v"] + if (max(L["y1"], L["y2"]) - min(L["y1"], L["y2"])) < BIG_FRAC * H] + img = Image.new("RGB", (W, H), "white") + d = ImageDraw.Draw(img) + try: + font = ImageFont.truetype("DejaVuSans-Bold.ttf", 30) + except Exception: + font = ImageFont.load_default() + # ignored short verticals = thin gray + for L in small: + d.line([ (L["x1"]+L["x2"])//2, min(L["y1"],L["y2"]), + (L["x1"]+L["x2"])//2, max(L["y1"],L["y2"]) ], fill=(200,200,200), width=3) + # big walls = thick red, labelled with height fraction + for w in walls: + d.line([w["x"], w["y1"], w["x"], w["y2"]], fill=(220, 30, 30), width=10) + d.text((w["x"] + 14, w["y1"] + 10), f'WALL x={w["x"]} {int(w["frac"]*100)}%H', + fill=(220, 30, 30), font=font) + img.save(out_path) + return walls + +TESTS = [ + ("output/NamastheTelangana_Siddipet_20260602_20260604_214937/pages", 2, "NT"), + ("output/AndhraJyothi_Siddipet District_20260602_20260603_144919/pages", 1, "AJ"), +] +for pdir, pg, tag in TESTS: + png = f"{pdir}/page_{pg:03d}.png" + out = f"{pdir}/page_{pg:03d}_vwalls_debug.png" + walls = draw_walls(png, out) + print(f"{tag} pg{pg}: {len(walls)} BIG vertical wall(s) -> {out}") + for w in walls: + print(f" x={w['x']:>4} y[{w['y1']},{w['y2']}] height={int(w['frac']*100)}% of page") diff --git a/app.py b/app.py new file mode 100644 index 0000000..9b6033f --- /dev/null +++ b/app.py @@ -0,0 +1,240 @@ +""" +Telugu Newspaper Article Extractor — local web app. + +Pipeline: + 1. User uploads a PDF. + 2. App renders each page to PNG at a target pixel size (~4200x7400). + 3. PaddleOCR (PP-Structure) finds layout regions per page. + 4. Grouping function joins regions into articles, anchored on the + ", మే (ఆంధ్రజ్యోతి):" dateline pattern. + 5. Each article is cropped to its own PNG. + 6. Each article's text is OCR'd with Tesseract (telugu). + 7. The user can browse results in the browser and download a zip. + +Run: + pip install -r requirements.txt + python app.py + -> open http://localhost:5000 +""" +import io +import os +import re +import json +import shutil +import zipfile +import tempfile +import threading +import traceback +from pathlib import Path +from datetime import datetime + +from flask import ( + Flask, request, render_template, redirect, url_for, + send_from_directory, send_file, abort, jsonify, +) + +from extractor import process_pdf, JOBS, JOB_LOCK + +ROOT = Path(__file__).parent +UPLOAD_DIR = ROOT / "uploads" +OUTPUT_DIR = ROOT / "output" +UPLOAD_DIR.mkdir(exist_ok=True) +OUTPUT_DIR.mkdir(exist_ok=True) + +app = Flask(__name__) +app.config["MAX_CONTENT_LENGTH"] = 200 * 1024 * 1024 # 200 MB upload cap + + +@app.route("/") +def index(): + # list existing jobs newest-first + jobs = [] + for d in sorted(OUTPUT_DIR.iterdir(), reverse=True): + if d.is_dir(): + meta_path = d / "meta.json" + meta = {} + if meta_path.exists(): + try: + meta = json.loads(meta_path.read_text()) + except Exception: + pass + jobs.append({ + "id": d.name, + "filename": meta.get("filename", "?"), + "status": meta.get("status", "unknown"), + "pages": meta.get("pages", 0), + "articles": meta.get("articles", 0), + "created": meta.get("created", ""), + }) + return render_template("index.html", jobs=jobs) + + +@app.route("/upload", methods=["POST"]) +def upload(): + f = request.files.get("pdf") + if not f or not f.filename.lower().endswith(".pdf"): + return "Upload a PDF file", 400 + + try: + target_w = int(request.form.get("width", "4200")) + target_h = int(request.form.get("height", "7400")) + except ValueError: + target_w, target_h = 4200, 7400 + + # Job folder + job_id = datetime.now().strftime("%Y%m%d_%H%M%S") + job_dir = OUTPUT_DIR / job_id + job_dir.mkdir() + + pdf_path = job_dir / "input.pdf" + f.save(str(pdf_path)) + + # Write initial meta + meta = { + "id": job_id, + "filename": f.filename, + "status": "queued", + "target_width": target_w, + "target_height": target_h, + "pages": 0, + "articles": 0, + "created": datetime.now().isoformat(timespec="seconds"), + } + (job_dir / "meta.json").write_text(json.dumps(meta, indent=2)) + + # Run in background thread so UI stays responsive + t = threading.Thread( + target=_run_job, + args=(job_id, str(pdf_path), str(job_dir), target_w, target_h), + daemon=True, + ) + t.start() + + return redirect(url_for("job_view", job_id=job_id)) + + +def _run_job(job_id, pdf_path, job_dir, target_w, target_h): + try: + with JOB_LOCK: + JOBS[job_id] = {"status": "running", "message": "Starting..."} + result = process_pdf(pdf_path, job_dir, target_w, target_h, job_id) + meta_path = Path(job_dir) / "meta.json" + meta = json.loads(meta_path.read_text()) + meta.update({ + "status": "done", + "pages": result["pages"], + "articles": result["articles"], + }) + meta_path.write_text(json.dumps(meta, indent=2)) + with JOB_LOCK: + JOBS[job_id] = {"status": "done", "message": "Finished."} + except Exception as e: + traceback.print_exc() + meta_path = Path(job_dir) / "meta.json" + try: + meta = json.loads(meta_path.read_text()) + meta["status"] = "error" + meta["error"] = str(e) + meta_path.write_text(json.dumps(meta, indent=2)) + except Exception: + pass + with JOB_LOCK: + JOBS[job_id] = {"status": "error", "message": str(e)} + + +@app.route("/job/") +def job_view(job_id): + job_dir = OUTPUT_DIR / job_id + if not job_dir.exists(): + abort(404) + + meta = json.loads((job_dir / "meta.json").read_text()) + + # collect article info if available + articles = [] + for art_dir in sorted((job_dir / "articles").glob("*")) if (job_dir / "articles").exists() else []: + if art_dir.is_dir(): + info_path = art_dir / "info.json" + info = {} + if info_path.exists(): + info = json.loads(info_path.read_text()) + articles.append({ + "name": art_dir.name, + "page": info.get("page", "?"), + "headline_preview": info.get("headline_preview", ""), + "dateline": info.get("dateline", ""), + "grouped_by": info.get("grouped_by", ""), + "has_image": (art_dir / "article.png").exists(), + "has_text": (art_dir / "article.txt").exists(), + }) + + return render_template("job.html", meta=meta, job_id=job_id, articles=articles) + + +@app.route("/job//status") +def job_status(job_id): + with JOB_LOCK: + status = JOBS.get(job_id, {"status": "unknown"}) + meta_path = OUTPUT_DIR / job_id / "meta.json" + if meta_path.exists(): + try: + meta = json.loads(meta_path.read_text()) + status["meta"] = meta + except Exception: + pass + return jsonify(status) + + +@app.route("/job//page/") +def page_image(job_id, page_num): + page_path = OUTPUT_DIR / job_id / "pages" / f"page_{page_num:03d}.png" + if not page_path.exists(): + abort(404) + return send_file(str(page_path)) + + +@app.route("/job//article//image") +def article_image(job_id, art_name): + p = OUTPUT_DIR / job_id / "articles" / art_name / "article.png" + if not p.exists(): + abort(404) + return send_file(str(p)) + + +@app.route("/job//article//text") +def article_text(job_id, art_name): + p = OUTPUT_DIR / job_id / "articles" / art_name / "article.txt" + if not p.exists(): + return "(no text)", 404 + return p.read_text(encoding="utf-8"), 200, {"Content-Type": "text/plain; charset=utf-8"} + + +@app.route("/job//download") +def download_zip(job_id): + job_dir = OUTPUT_DIR / job_id + if not job_dir.exists(): + abort(404) + + buf = io.BytesIO() + with zipfile.ZipFile(buf, "w", zipfile.ZIP_DEFLATED) as zf: + for root, dirs, files in os.walk(job_dir): + # skip the source pdf and big page images to keep the zip lean + for fn in files: + if fn == "input.pdf": + continue + full = Path(root) / fn + rel = full.relative_to(job_dir) + zf.write(full, arcname=str(rel)) + buf.seek(0) + return send_file( + buf, + mimetype="application/zip", + as_attachment=True, + download_name=f"{job_id}_articles.zip", + ) + + +if __name__ == "__main__": + print("Telugu Article Extractor") + print("Open http://localhost:5000") + app.run(host="127.0.0.1", port=5000, debug=False, threaded=True) diff --git a/claude_ocr.py b/claude_ocr.py new file mode 100644 index 0000000..08a9a7a --- /dev/null +++ b/claude_ocr.py @@ -0,0 +1,124 @@ +""" +Read Telugu text from cropped article images using Claude Vision API. +Run after a job completes to get accurate text extraction. + +Usage: + python claude_ocr.py # processes latest job + python claude_ocr.py 20260526_002006 # processes specific job +""" +import os +import sys +import json +import base64 +import anthropic +from pathlib import Path + +OUTPUT_DIR = Path(__file__).parent / "output" + + +def read_article_image(client, img_path, model="claude-sonnet-4-6"): + """Send article image to Claude and get Telugu text back.""" + with open(img_path, "rb") as f: + img_data = base64.standard_b64encode(f.read()).decode("utf-8") + + response = client.messages.create( + model=model, + max_tokens=4096, + messages=[{ + "role": "user", + "content": [ + { + "type": "image", + "source": { + "type": "base64", + "media_type": "image/png", + "data": img_data, + }, + }, + { + "type": "text", + "text": ( + "Read all Telugu text from this newspaper article image. " + "Output ONLY the Telugu text in correct reading order " + "(top to bottom, reading each column fully before moving to the next). " + "Include headlines first, then subheadlines, then body text. " + "Do not translate. Do not add any explanation. Just the Telugu text." + ), + }, + ], + }], + ) + return response.content[0].text + + +def process_job(job_id=None): + api_key = os.environ.get("ANTHROPIC_API_KEY") + if not api_key: + print("Error: Set ANTHROPIC_API_KEY environment variable") + sys.exit(1) + + # Find job directory + if job_id: + job_dir = OUTPUT_DIR / job_id + else: + # Use latest job + jobs = sorted(OUTPUT_DIR.iterdir(), reverse=True) + if not jobs: + print("No jobs found") + sys.exit(1) + job_dir = jobs[0] + job_id = job_dir.name + + articles_dir = job_dir / "articles" + if not articles_dir.exists(): + print(f"No articles found in {job_dir}") + sys.exit(1) + + client = anthropic.Anthropic(api_key=api_key) + article_dirs = sorted(articles_dir.iterdir()) + + print(f"Job: {job_id}") + print(f"Articles: {len(article_dirs)}") + print("-" * 50) + + all_text = [] + + for art_dir in article_dirs: + img_path = art_dir / "article.png" + if not img_path.exists(): + continue + + name = art_dir.name + print(f"Reading {name}...", end=" ", flush=True) + + try: + text = read_article_image(client, img_path) + # Save to article.txt (overwrites old OCR output) + (art_dir / "article.txt").write_text(text, encoding="utf-8") + print(f"OK ({len(text)} chars)") + + all_text.append(f"--- {name} ---\n{text}\n") + + # Update info.json headline preview + info_path = art_dir / "info.json" + if info_path.exists(): + info = json.loads(info_path.read_text()) + first_line = next((ln.strip() for ln in text.splitlines() if ln.strip()), "") + info["headline_preview"] = first_line[:120] + info["ocr_method"] = "claude_vision" + info_path.write_text(json.dumps(info, indent=2, ensure_ascii=False)) + + except Exception as e: + print(f"FAILED: {e}") + all_text.append(f"--- {name} ---\n[Error: {e}]\n") + + # Save combined file + combined_path = job_dir / "all_articles_claude.txt" + combined_path.write_text("\n".join(all_text), encoding="utf-8") + print("-" * 50) + print(f"Done! Combined text saved to: {combined_path}") + + +if __name__ == "__main__": + job_id = sys.argv[1] if len(sys.argv) > 1 else None + process_job(job_id) diff --git a/crop_political_articles.py b/crop_political_articles.py new file mode 100644 index 0000000..8bf414c --- /dev/null +++ b/crop_political_articles.py @@ -0,0 +1,170 @@ +""" +crop_political_articles.py +========================== + +Standalone re-cropping pass for political articles in the latest extractor run. + +What it does +------------ +- Picks the most recent run folder under `output/`. +- For each article folder under `output//articles/pXXX_aYYY/` + (these are the political articles — the extractor only writes folders for them), + reads `info.json` to get the page number and the member region IDs. +- Opens `output//pages/page_XXX.png` and computes a TIGHT crop: + union(member region bboxes) with a tiny 5px padding, + clamped to the page image. This is tighter than the extractor's default + 15px padding around the same union. +- Writes the result to `output//cropped_articles/pXXX_aYYY.png`. + +Guarantees +---------- +- Does NOT touch any existing files (articles/, all_political_images/, + pages/, political_articles.json, political_articles.pdf, etc). +- Only creates a new folder `cropped_articles/` inside the run folder. + +Usage +----- + python crop_political_articles.py + python crop_political_articles.py --run output/20260528_203315 + python crop_political_articles.py --padding 0 +""" + +from __future__ import annotations + +import argparse +import json +import sys +from pathlib import Path + +from PIL import Image + + +def find_latest_run(output_root: Path) -> Path: + if not output_root.exists(): + sys.exit(f"output folder not found: {output_root}") + runs = sorted( + [p for p in output_root.iterdir() if p.is_dir() and p.name[0].isdigit()], + key=lambda p: p.name, + ) + if not runs: + sys.exit(f"no run folders found under {output_root}") + return runs[-1] + + +def load_regions(pages_dir: Path, page_num: int) -> dict[int, list[int]]: + """Return {region_id: [x1, y1, x2, y2]} for the given page.""" + regions_path = pages_dir / f"page_{page_num:03d}.regions.json" + if not regions_path.exists(): + return {} + with regions_path.open() as f: + data = json.load(f) + out: dict[int, list[int]] = {} + for r in data.get("regions", []): + rid = r.get("id") + bbox = r.get("bbox") + if rid is None or not bbox or len(bbox) != 4: + continue + out[int(rid)] = [int(v) for v in bbox] + return out + + +def tight_bbox(member_ids: list[int], region_map: dict[int, list[int]], + fallback_bbox: list[int] | None) -> list[int] | None: + """Compute union bbox of member regions; fall back to info.json bbox.""" + boxes = [region_map[rid] for rid in member_ids if rid in region_map] + if boxes: + x1 = min(b[0] for b in boxes) + y1 = min(b[1] for b in boxes) + x2 = max(b[2] for b in boxes) + y2 = max(b[3] for b in boxes) + return [x1, y1, x2, y2] + if fallback_bbox and len(fallback_bbox) == 4: + return [int(v) for v in fallback_bbox] + return None + + +def crop_one(article_dir: Path, pages_dir: Path, out_dir: Path, + padding: int) -> tuple[bool, str]: + info_path = article_dir / "info.json" + if not info_path.exists(): + return False, f"{article_dir.name}: no info.json" + with info_path.open() as f: + info = json.load(f) + + page_num = int(info.get("page", 0)) + member_ids = [int(x) for x in info.get("member_region_ids", []) or []] + fallback = info.get("bbox") + + page_img_path = pages_dir / f"page_{page_num:03d}.png" + if not page_img_path.exists(): + return False, f"{article_dir.name}: page image missing ({page_img_path.name})" + + region_map = load_regions(pages_dir, page_num) + bbox = tight_bbox(member_ids, region_map, fallback) + if bbox is None: + return False, f"{article_dir.name}: could not compute bbox" + + with Image.open(page_img_path) as im: + pw, ph = im.size + x1 = max(0, bbox[0] - padding) + y1 = max(0, bbox[1] - padding) + x2 = min(pw, bbox[2] + padding) + y2 = min(ph, bbox[3] + padding) + if x2 <= x1 or y2 <= y1: + return False, f"{article_dir.name}: empty crop {[x1,y1,x2,y2]}" + cropped = im.crop((x1, y1, x2, y2)) + out_path = out_dir / f"{article_dir.name}.png" + cropped.save(out_path) + return True, f"{article_dir.name}: {x2-x1}x{y2-y1}px -> {out_path.name}" + + +def main() -> int: + ap = argparse.ArgumentParser(description=__doc__) + ap.add_argument( + "--run", type=Path, default=None, + help="Path to a specific run folder. Default: latest under ./output", + ) + ap.add_argument( + "--output-root", type=Path, default=Path(__file__).parent / "output", + help="Root output dir (default: ./output next to this script)", + ) + ap.add_argument( + "--padding", type=int, default=5, + help="Pixels of padding around the tight region union (default: 5)", + ) + args = ap.parse_args() + + run_dir = args.run if args.run is not None else find_latest_run(args.output_root) + run_dir = run_dir.resolve() + print(f"Run folder: {run_dir}") + + articles_dir = run_dir / "articles" + pages_dir = run_dir / "pages" + if not articles_dir.is_dir(): + sys.exit(f"missing articles/ in {run_dir}") + if not pages_dir.is_dir(): + sys.exit(f"missing pages/ in {run_dir}") + + out_dir = run_dir / "cropped_articles" + out_dir.mkdir(exist_ok=True) + print(f"Writing to: {out_dir}") + + article_folders = sorted(p for p in articles_dir.iterdir() if p.is_dir()) + if not article_folders: + print("No political article folders found.") + return 0 + + ok = 0 + for adir in article_folders: + success, msg = crop_one(adir, pages_dir, out_dir, args.padding) + print((" ok " if success else " -- ") + msg) + if success: + ok += 1 + + print(f"\nDone. {ok}/{len(article_folders)} articles cropped into " + f"{out_dir.relative_to(run_dir.parent)}") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/extractor.py b/extractor.py new file mode 100644 index 0000000..b1fa034 --- /dev/null +++ b/extractor.py @@ -0,0 +1,1105 @@ +""" +extractor.py — the pipeline. + + Stage 1: render_pdf_pages() PDF -> high-DPI PNGs + Stage 2: detect_regions() page PNG -> regions.json + Stage 3: group_into_articles() regions.json -> articles.json + Primary: Claude API (spatial reasoning over regions) + Fallback: column-aware + dateline-anchored Python rules + Stage 4: crop_articles() articles.json -> article PNGs + Stage 5: ocr_articles() article PNGs -> article TXTs +""" +import os +import re +import json +import base64 +import threading +from io import BytesIO +from pathlib import Path + +import fitz # PyMuPDF +from PIL import Image + +JOBS = {} +JOB_LOCK = threading.Lock() + +_LAYOUT = None +_LAYOUT_LOCK = threading.Lock() + + +def _set_status(job_id, msg): + with JOB_LOCK: + if job_id in JOBS: + JOBS[job_id]["message"] = msg + + +# --------------------------------------------------------------------------- +# Stage 1 — render PDF pages +# --------------------------------------------------------------------------- +def render_pdf_pages(pdf_path, out_dir, target_w, target_h): + out_dir = Path(out_dir) + out_dir.mkdir(parents=True, exist_ok=True) + + doc = fitz.open(pdf_path) + pages = [] + for i, page in enumerate(doc, start=1): + rect = page.rect + scale = min(target_w / rect.width, target_h / rect.height) + mat = fitz.Matrix(scale, scale) + pix = page.get_pixmap(matrix=mat, alpha=False) + out_path = out_dir / f"page_{i:03d}.png" + pix.save(str(out_path)) + pages.append({"page": i, "path": str(out_path), + "width": pix.width, "height": pix.height}) + doc.close() + return pages + + +# --------------------------------------------------------------------------- +# Stage 2 — detect regions with PaddleOCR PP-Structure +# --------------------------------------------------------------------------- +def _load_layout_model(): + global _LAYOUT + if _LAYOUT is not None: + return _LAYOUT + with _LAYOUT_LOCK: + if _LAYOUT is not None: + return _LAYOUT + import paddleocr + # PaddleOCR v3.5.0 — use LayoutDetection for layout analysis. + # Stock ~0.5 confidence cutoff (no threshold override) — this is what + # Claude grouping is tuned to see. + try: + _LAYOUT = paddleocr.LayoutDetection() + except Exception: + try: + _LAYOUT = paddleocr.PPStructureV3() + except Exception: + _LAYOUT = paddleocr.PaddleOCR(lang='te') + return _LAYOUT + + +def detect_regions(page_png_path): + layout = _load_layout_model() + import numpy as np + img = np.array(Image.open(page_png_path).convert("RGB")) + + # PaddleOCR v3.5.0 uses .predict() instead of direct call + if hasattr(layout, 'predict'): + raw = layout.predict(page_png_path) + elif callable(layout): + raw = layout(img) + else: + raw = layout.predict(img) + + regions = [] + + # PaddleOCR v3.5.0: raw is a list of DetResult objects (dict-like). + # Each DetResult has 'boxes' key containing list of detected regions. + # Each box has: 'cls_id', 'label', 'score', 'coordinate' (4 floats: x1,y1,x2,y2) + if isinstance(raw, list) and len(raw) > 0: + for det_result in raw: + # Get boxes from the DetResult + boxes = None + if hasattr(det_result, 'get'): + boxes = det_result.get('boxes', None) + elif hasattr(det_result, 'boxes'): + boxes = det_result.boxes + elif isinstance(det_result, dict) and 'boxes' in det_result: + boxes = det_result['boxes'] + + if boxes is not None: + for box in boxes: + rid = len(regions) + 1 + # Extract coordinate + coord = None + if isinstance(box, dict): + coord = box.get('coordinate', box.get('bbox', None)) + label = box.get('label', box.get('type', 'unknown')).lower() + score = float(box.get('score', box.get('confidence', 0))) + elif hasattr(box, 'coordinate'): + coord = box.coordinate + label = getattr(box, 'label', 'unknown').lower() + score = float(getattr(box, 'score', 0)) + else: + label = 'unknown' + score = 0.0 + + bbox = [0, 0, 0, 0] + if coord is not None: + if hasattr(coord, 'tolist'): + coord = coord.tolist() + coord = [float(v) for v in coord] + if len(coord) == 4: + bbox = [int(v) for v in coord] + elif len(coord) >= 8: + xs = [coord[j] for j in range(0, len(coord), 2)] + ys = [coord[j] for j in range(1, len(coord), 2)] + bbox = [int(min(xs)), int(min(ys)), int(max(xs)), int(max(ys))] + + regions.append({"id": rid, "type": label, "bbox": bbox, "confidence": score}) + + # Detailed logging + print(f"\n{'='*70}") + print(f"LAYOUT DETECTION RESULTS: {len(regions)} regions found") + print(f"{'='*70}") + + # Count by type + type_counts = {} + for r in regions: + t = r["type"] + type_counts[t] = type_counts.get(t, 0) + 1 + print(f"Region types: {dict(sorted(type_counts.items(), key=lambda x: -x[1]))}") + print() + + # Print each region + for r in regions: + x1, y1, x2, y2 = r["bbox"] + w = x2 - x1 + h = y2 - y1 + print(f" Region {r['id']:3d}: type={r['type']:15s} bbox=[{x1:5d},{y1:5d},{x2:5d},{y2:5d}] size={w:5d}x{h:5d}px conf={r['confidence']:.2f}") + print(f"{'='*70}\n") + + # Draw bounding boxes on page image and save as debug image + try: + from PIL import ImageDraw, ImageFont + debug_img = Image.open(page_png_path).copy() + draw = ImageDraw.Draw(debug_img) + + # Color map for different region types + colors = { + "text": "blue", "image": "green", "title": "red", "doc_title": "red", + "header": "gray", "footer": "gray", "paragraph_title": "orange", + "table": "purple", "figure": "green", "list": "cyan", + } + + for r in regions: + x1, y1, x2, y2 = r["bbox"] + color = colors.get(r["type"], "yellow") + # Draw rectangle + draw.rectangle([x1, y1, x2, y2], outline=color, width=3) + # Draw label + label_text = f"{r['id']}:{r['type']}" + draw.text((x1 + 5, y1 + 5), label_text, fill=color) + + # Save debug image next to the page image + debug_path = page_png_path.replace(".png", "_regions_debug.png") + debug_img.save(debug_path) + print(f"DEBUG IMAGE saved: {debug_path}") + print(f" Open this image to see all {len(regions)} bounding boxes drawn on the page") + print(f" Colors: text=blue, title/doc_title=red, image/figure=green, header/footer=gray, paragraph_title=orange") + print() + except Exception as e: + print(f" Could not create debug image: {e}") + + return regions + + +# --------------------------------------------------------------------------- +# Stage 3 — group regions into articles +# +# Primary: Claude API (set ANTHROPIC_API_KEY env var). +# Fallback: column-aware + dateline Python rules. +# --------------------------------------------------------------------------- + +DATELINE_RE = re.compile( + r"^[\u0C00-\u0C7F\u0964\u0965A-Za-z0-9\s.\-]{1,40}?," + r"\s*[\u0C00-\u0C7F]{1,8}\s*\d{1,2}" + r"[^:]{0,30}\(ఆంధ్రజ్యోతి\)[\s\u0C00-\u0C7F]{0,4}:", + re.UNICODE, +) + + +def looks_like_dateline(text): + if not text: + return None + snippet = text.strip()[:120] + m = DATELINE_RE.match(snippet) + if m: + return snippet[:m.end()] + if "ఆంధ్రజ్యోతి" in snippet[:100] and ":" in snippet[:120]: + return snippet[:snippet.find(":") + 1] + return None + + +def _bbox_union(a, b): + return [min(a[0], b[0]), min(a[1], b[1]), max(a[2], b[2]), max(a[3], b[3])] + + +def _make_topband_ocr(page_img): + try: + import pytesseract + except ImportError: + return None + + def topband(reg): + l, t, r, b = reg["bbox"] + strip_h = min(140, b - t) + if strip_h <= 0: + return "" + strip = page_img.crop((l, t, r, t + strip_h)) + try: + return pytesseract.image_to_string(strip, lang="tel") + except Exception: + return "" + return topband + + +def _enrich_regions_with_text(regions, topband_ocr): + enriched = [] + for r in regions: + rec = dict(r) + if topband_ocr and r["type"] in ("text", "paragraph", "list"): + try: + txt = topband_ocr(r) or "" + except Exception: + txt = "" + rec["top_text"] = txt.strip()[:160] + d = looks_like_dateline(rec["top_text"]) + rec["dateline"] = d if d else None + else: + rec["top_text"] = "" + rec["dateline"] = None + enriched.append(rec) + return enriched + + +# ---------- Claude-based grouping ---------- +def _make_page_thumbnail(page_png_path, max_dim=900): + img = Image.open(page_png_path) + img.thumbnail((max_dim, max_dim), Image.LANCZOS) + buf = BytesIO() + img.save(buf, format="JPEG", quality=80) + return base64.standard_b64encode(buf.getvalue()).decode("ascii") + + +CLAUDE_GROUPING_PROMPT = """You group newspaper page regions into INDIVIDUAL articles. Each news story is a SEPARATE article. + +You receive: +1. A thumbnail of the full page (for spatial context). +2. A JSON list of regions on the page. Each has: + - id (int) + - type ("title", "text", "figure", "table", etc.) + - bbox [left, top, right, bottom] in pixels (top-left origin) + - top_text: a Tesseract OCR snippet from the top of text regions + - dateline: if non-null, the region starts an article body (strong signal) + +CRITICAL RULES — BE CONSERVATIVE, SPLIT MORE: +1. A `title` or `doc_title` region ALWAYS starts a NEW article. Every title = separate article. +2. A region with a non-null `dateline` ALWAYS starts a new article body. +3. Text regions attach to the NEAREST title above them IN THE SAME COLUMN ONLY. +4. Newspapers have COLUMNS. A region in column 1 NEVER belongs to an article in column 2, even if vertically close. +5. When in doubt, SPLIT into separate articles rather than merge. Too many small articles is better than one giant article containing multiple stories. +6. An article should typically have 1-5 regions. If an article has more than 6 regions, you are probably merging multiple stories — split them. +7. A banner title spanning multiple columns owns ONLY the text directly below it, not the entire page. +8. Types "header", "footer", "page_number" — make these their own separate article. +9. Figures/images attach to the nearest title above them in THEIR column only. +10. NEVER merge two articles that have different headlines or topics into one. + +Return ONLY valid JSON, no prose, no markdown fences: +{ + "articles": [ + { + "article_id": 1, + "title_region_id": , + "member_region_ids": [, ...], + "headline_hint": "", + "dateline": "" + } + ] +} + +Order articles top-to-bottom, then left-to-right by topmost member region. +Every region (except headers/footers) appears in exactly one article. +AIM FOR 15-25 articles per page for a typical Telugu newspaper front page.""" + + +def _group_with_claude(page_png_path, regions, api_key, model="claude-opus-4-8"): + try: + import anthropic + except ImportError: + raise RuntimeError("anthropic package not installed — pip install anthropic") + + client = anthropic.Anthropic(api_key=api_key) + thumbnail_b64 = _make_page_thumbnail(page_png_path) + + compact = [ + { + "id": r["id"], + "type": r["type"], + "bbox": r["bbox"], + "top_text": r.get("top_text", "")[:160], + "dateline": r.get("dateline"), + } + for r in regions + ] + + user_content = [ + {"type": "image", + "source": {"type": "base64", "media_type": "image/jpeg", "data": thumbnail_b64}}, + {"type": "text", + "text": "Regions:\n" + json.dumps(compact, ensure_ascii=False, indent=2)}, + ] + + msg = client.messages.create( + model=model, + max_tokens=4000, + system=CLAUDE_GROUPING_PROMPT, + messages=[{"role": "user", "content": user_content}], + ) + + raw = "".join(b.text for b in msg.content if hasattr(b, "text")).strip() + if raw.startswith("```"): + raw = raw.split("\n", 1)[1] if "\n" in raw else raw + if raw.endswith("```"): + raw = raw[:raw.rfind("```")] + data = json.loads(raw) + + region_by_id = {r["id"]: r for r in regions} + articles = [] + for art in data.get("articles", []): + ids = art.get("member_region_ids", []) + if not ids: + continue + bbox = None + for rid in ids: + r = region_by_id.get(rid) + if not r: + continue + bbox = list(r["bbox"]) if bbox is None else _bbox_union(bbox, r["bbox"]) + if bbox is None: + continue + articles.append({ + "article_id": art.get("article_id", len(articles) + 1), + "title_region": art.get("title_region_id"), + "member_regions": ids, + "bbox": bbox, + "dateline": art.get("dateline"), + "headline_hint": art.get("headline_hint"), + "grouped_by": "claude", + }) + return articles + + +# ---------- Fallback Python heuristic ---------- +def _assign_columns(regions, page_width): + if not regions: + return {0: regions} + threshold = page_width * 0.04 + lefts = sorted({r["bbox"][0] for r in regions}) + column_starts = [lefts[0]] + for prev, curr in zip(lefts, lefts[1:]): + if curr - prev > threshold: + column_starts.append(curr) + columns = {i: [] for i in range(len(column_starts))} + for r in regions: + left = r["bbox"][0] + col_idx = 0 + for i, cs in enumerate(column_starts): + if left >= cs - threshold: + col_idx = i + columns[col_idx].append(r) + return columns + + +def _group_with_rules(regions, page_width): + columns = _assign_columns(regions, page_width) + articles = [] + next_id = 1 + for col_idx in sorted(columns.keys()): + col_regs = sorted(columns[col_idx], key=lambda r: r["bbox"][1]) + current = None + for reg in col_regs: + rtype = reg["type"] + dateline = reg.get("dateline") + starts = (rtype in ("title", "header")) or (dateline is not None) + if starts: + if current is not None and current["member_regions"]: + articles.append(current) + current = { + "article_id": next_id, + "title_region": reg["id"] if rtype in ("title", "header") else None, + "member_regions": [reg["id"]], + "bbox": list(reg["bbox"]), + "dateline": dateline, + "grouped_by": "rules", + } + next_id += 1 + else: + if current is None: + current = { + "article_id": next_id, "title_region": None, + "member_regions": [reg["id"]], "bbox": list(reg["bbox"]), + "dateline": None, "grouped_by": "rules", + } + next_id += 1 + else: + current["member_regions"].append(reg["id"]) + current["bbox"] = _bbox_union(current["bbox"], reg["bbox"]) + if current is not None and current["member_regions"]: + articles.append(current) + return articles + + +def group_into_articles(page_png_path, regions, page_width, page_img=None): + """Claude first; rules fallback. Returns (articles, enriched_regions).""" + topband_ocr = _make_topband_ocr(page_img) if page_img else None + enriched = _enrich_regions_with_text(regions, topband_ocr) + + api_key = os.environ.get("ANTHROPIC_API_KEY") + if api_key: + try: + articles = _group_with_claude(page_png_path, enriched, api_key) + if articles: + return articles, enriched + except Exception as e: + print(f"[claude grouping failed, falling back to rules]: {e}") + + articles = _group_with_rules(enriched, page_width) + return articles, enriched + + +# --------------------------------------------------------------------------- +# Stage 4 — crop each article +# --------------------------------------------------------------------------- +def crop_articles(page_png_path, articles, out_dir, page_num): + out_dir = Path(out_dir) + out_dir.mkdir(parents=True, exist_ok=True) + img = Image.open(page_png_path) + + results = [] + for art in articles: + l, t, r, b = art["bbox"] + margin = 6 + l = max(0, l - margin); t = max(0, t - margin) + r = min(img.width, r + margin); b = min(img.height, b + margin) + + crop = img.crop((l, t, r, b)) + name = f"p{page_num:03d}_a{art['article_id']:03d}" + art_dir = out_dir / name + art_dir.mkdir(parents=True, exist_ok=True) + crop.save(art_dir / "article.png") + + info = { + "page": page_num, + "article_id": art["article_id"], + "bbox": [l, t, r, b], + "member_regions": art["member_regions"], + "title_region": art.get("title_region"), + "dateline": art.get("dateline"), + "headline_hint": art.get("headline_hint"), + "grouped_by": art.get("grouped_by", "rules"), + } + (art_dir / "info.json").write_text(json.dumps(info, indent=2, ensure_ascii=False)) + results.append({"name": name, "dir": str(art_dir), **info}) + return results + + +# --------------------------------------------------------------------------- +# Stage 5 — OCR each cropped article (Telugu) +# --------------------------------------------------------------------------- +def _ocr_with_paddleocr(img_path): + """Use PaddleOCR for better reading-order-aware Telugu OCR.""" + try: + import paddleocr + ocr = paddleocr.PaddleOCR(lang='te') + result = ocr.predict(str(img_path)) + + lines = [] + if isinstance(result, list) and len(result) > 0: + det_result = result[0] + # PaddleOCR v3.5 returns DetResult with 'rec_texts' or nested structure + rec_texts = None + if hasattr(det_result, 'get'): + rec_texts = det_result.get('rec_texts', None) + if rec_texts: + lines = list(rec_texts) + else: + # Try to extract from boxes/text pairs + boxes = det_result.get('dt_polys', []) if hasattr(det_result, 'get') else [] + texts = det_result.get('rec_texts', []) if hasattr(det_result, 'get') else [] + if texts: + # Sort by y-coordinate (top to bottom), then x (left to right) + # to get proper reading order + pairs = [] + for i, txt in enumerate(texts): + if i < len(boxes): + box = boxes[i] + if hasattr(box, 'tolist'): + box = box.tolist() + # Get top-left y coordinate for sorting + if isinstance(box, (list, tuple)) and len(box) >= 4: + y = min(box[j] for j in range(1, len(box), 2)) if len(box) >= 8 else box[1] + x = min(box[j] for j in range(0, len(box), 2)) if len(box) >= 8 else box[0] + else: + y, x = 0, 0 + pairs.append((y, x, txt)) + else: + pairs.append((0, 0, txt)) + + # Sort: primarily by y (row), then x (column within row) + # Group into rows based on y proximity + if pairs: + pairs.sort(key=lambda p: (p[0], p[1])) + row_threshold = 20 # pixels + rows = [] + current_row = [pairs[0]] + for p in pairs[1:]: + if abs(p[0] - current_row[0][0]) < row_threshold: + current_row.append(p) + else: + current_row.sort(key=lambda p: p[1]) # sort by x within row + rows.append(current_row) + current_row = [p] + current_row.sort(key=lambda p: p[1]) + rows.append(current_row) + + for row in rows: + lines.append(" ".join(p[2] for p in row)) + else: + # Last resort: try str representation + try: + s = str(det_result) + if len(s) > 10: + return s + except: + pass + + return "\n".join(lines) if lines else None + except Exception as e: + print(f"PaddleOCR text extraction failed: {e}") + return None + + +def _ocr_with_tesseract(img_path): + """Fallback: use Tesseract with column-aware PSM for Telugu OCR.""" + try: + import pytesseract + # PSM 1 = Automatic page segmentation with OSD (better for multi-column) + # PSM 3 = Fully automatic (default, often jumbles columns) + config = '--psm 1 --oem 3' + text = pytesseract.image_to_string(Image.open(img_path), lang="tel", config=config) + if not text.strip(): + # Fallback to PSM 4 (single column) if PSM 1 gives nothing + config = '--psm 4 --oem 3' + text = pytesseract.image_to_string(Image.open(img_path), lang="tel", config=config) + return text + except Exception as e: + return f"[OCR error: {e}]" + + +def ocr_articles(article_records): + for rec in article_records: + art_dir = Path(rec["dir"]) + img_path = art_dir / "article.png" + if not img_path.exists(): + continue + + # Try PaddleOCR first (better reading order), fall back to Tesseract + text = _ocr_with_paddleocr(img_path) + if not text or len(text.strip()) < 10: + text = _ocr_with_tesseract(img_path) + + (art_dir / "article.txt").write_text(text or "", encoding="utf-8") + + info_path = art_dir / "info.json" + if info_path.exists(): + info = json.loads(info_path.read_text()) + first_line = next((ln.strip() for ln in text.splitlines() if ln.strip()), "") + info["headline_preview"] = first_line[:120] + info_path.write_text(json.dumps(info, indent=2, ensure_ascii=False)) + + +# --------------------------------------------------------------------------- +# Political Filter — crop first, then filter with Claude +# --------------------------------------------------------------------------- +def _filter_articles_with_claude(page_img_path, article_ids_with_locations, page_num, job_dir=None): + """Send page image + article IDs to Claude, get back only politically significant IDs.""" + import base64 + from io import BytesIO + api_key = os.environ.get("ANTHROPIC_API_KEY") + if not api_key: + return None + + try: + import anthropic + client = anthropic.Anthropic(api_key=api_key) + + # Resize page image for API + img = Image.open(page_img_path) + max_w = 1500 + if img.width > max_w: + ratio = max_w / img.width + img = img.resize((max_w, int(img.height * ratio)), Image.LANCZOS) + + buf = BytesIO() + img.convert("RGB").save(buf, format="JPEG", quality=80) + img_data = base64.standard_b64encode(buf.getvalue()).decode("utf-8") + + # Build article list text + article_list = "\n".join( + f"- {aid}: located at {loc}" for aid, loc in article_ids_with_locations + ) + + prompt = f"""You are a senior opposition party strategist in Telangana state, India. + +I have already cropped the following articles from this newspaper page: + +{article_list} + +For EACH article, ask yourself these TWO questions: + +QUESTION 1: "Does this article describe actual DAMAGE, LOSS, DEATH, CORRUPTION, or SUFFERING happening to people in Telangana? And can the government (state OR central) be BLAMED for it — either for causing it, failing to prevent it, or failing to respond to it?" +- The article must describe a NEGATIVE EVENT that has already occurred — not a future plan, not an announcement, not a statistic. +- There must be a VICTIM — someone who died, lost money, was cheated, is suffering, or was harmed. +- The government (Telangana state OR central/BJP) can be blamed — for causing it, failing to prevent it, or failing to respond to it. +- Central government failures also qualify IF the article describes actual harm to Telangana citizens (e.g., NEET paper leak harming Telangana students, central policy causing job losses in Telangana). +- If the article describes something POSITIVE (new scheme, development, achievement) → answer is NO. +- If the article describes a PLAN or ANNOUNCEMENT (not something bad that happened) → answer is NO. +- If there is NO victim and NO damage described → answer is NO. +- RESPONSIBILITY CHECK: Ask "WHO caused this damage?" If the damage was caused by nature (fire, flood, lightning), individual negligence (reckless driving, personal accident), criminals (theft, murder), or private parties — and the government had NO role in causing or preventing it — answer is NO. Only answer YES if the government is CLEARLY responsible through its policy, negligence, corruption, or failure to act. +- If YES → proceed to Question 2 +- If NO → REJECT this article immediately + +QUESTION 2: "Can the opposition party DIRECTLY USE this article to attack the Telangana government in a press conference, assembly session, or public rally?" +- If YES → SELECT this article +- If NO → REJECT this article + +BOTH answers must be YES to select. If either answer is NO, reject. + +DEDUPLICATION: If two or more articles cover the SAME issue or event (e.g., same farmer protest, same accident, same scam), only SELECT the ONE with the most complete coverage or the larger article. REJECT the duplicates. Do not keep two articles about the same news story. + +NOW go through EACH article ID I gave you. Ask Q1 and Q2 for each. Only select if BOTH are YES. + +IMPORTANT: This application is for TELANGANA STATE politics only. Farmer issues are the MOST IMPORTANT — always select if farmers are suffering. + +DO NOT SELECT — automatically REJECT: +- International news (wars, foreign affairs, immigration) +- Government achievements, new schemes, welfare announcements, development projects +- Newspaper masthead, headers, advertisements, classifieds +- Sports, entertainment, cinema, horoscopes +- General statistics without actual damage (birth rate, population, GDP) +- News about other states +- Resignations — ALWAYS REJECT unless the article explicitly contains words like corruption, scam, fired, forced out, or political pressure. A person resigning for personal, health, family, or workload reasons is NOT a government failure. +- PM Modi speeches or party meetings (unless specific policy failure hurting Telangana people) +- Individual accidents (road accidents, electric shocks, drownings, fire accidents) — these are NOT government failures, they can happen due to personal negligence or bad luck. Only select if there is a MASS disaster (10+ deaths) or CLEAR systemic government negligence (e.g., bridge collapse, building collapse, factory fire due to no safety inspections). + +Return a JSON object with TWO lists: + +1. "reasoning": for EVERY article (selected or not), explain your thinking: + - "id": the article ID + - "headline_english": English translation + - "q1": "YES" or "NO" + - "q1_reason": why (1 sentence) + - "q2": "YES" or "NO" (skip if q1 is NO) + - "q2_reason": why (1 sentence, skip if q1 is NO) + - "decision": "SELECT" or "REJECT" + +2. "selected": only the articles where both Q1 and Q2 are YES: + - "id": the article ID + - "headline_english": English translation of the headline + - "q1_answer": your answer to Question 1 (1 sentence) + - "q2_answer": your answer to Question 2 (1 sentence) + - "attack_angle": what opposition should say in a press conference (1 sentence) + - "priority": "high" or "medium" + - "category": one of corruption, governance_failure, public_grievance, law_order, health, education, infrastructure, ruling_party_crisis, farmer_issues + +Return ONLY valid JSON. No markdown. If no articles pass both questions, return {{"reasoning": [...], "selected": []}}""" + + response = client.messages.create( + model="claude-opus-4-8", max_tokens=4096, + messages=[{"role": "user", "content": [ + {"type": "image", "source": {"type": "base64", "media_type": "image/jpeg", "data": img_data}}, + {"type": "text", "text": prompt}, + ]}], + ) + + text = response.content[0].text.strip() + if text.startswith("```"): + text = text.split("\n", 1)[1] + if text.endswith("```"): + text = text[:-3] + + result = json.loads(text.strip()) + + # Log full reasoning to file and terminal + reasoning = result.get("reasoning", []) + if reasoning: + print(f"\n=== POLITICAL FILTER REASONING (Page {page_num}) ===") + for r in reasoning: + status = "✅ SELECT" if r.get("decision") == "SELECT" else "❌ REJECT" + print(f" {r.get('id','?')}: {r.get('headline_english','')[:50]}") + print(f" Q1: {r.get('q1','?')} — {r.get('q1_reason','')}") + if r.get('q2'): + print(f" Q2: {r.get('q2','?')} — {r.get('q2_reason','')}") + print(f" → {status}") + print(f"=== END REASONING ===\n") + + # Save reasoning log to job directory + from pathlib import Path + log_path = Path(job_dir) / f"filter_reasoning_page{page_num}.json" if job_dir else None + if log_path: + log_path.write_text(json.dumps(result, indent=2, ensure_ascii=False), encoding="utf-8") + + return result.get("selected", []) + + except Exception as e: + print(f"Political filter failed: {e}") + import traceback; traceback.print_exc() + return None + + +def _generate_opposition_brief(selected_articles, job_dir): + """Generate a readable opposition daily brief text file.""" + high = [a for a in selected_articles if a.get("priority") == "high"] + medium = [a for a in selected_articles if a.get("priority") == "medium"] + + brief_path = Path(job_dir) / "opposition_daily_brief.txt" + with open(brief_path, "w", encoding="utf-8") as f: + f.write("=" * 70 + "\n") + f.write(" OPPOSITION DAILY BRIEF\n") + f.write("=" * 70 + "\n\n") + f.write(f"Total significant articles: {len(selected_articles)}\n") + f.write(f"HIGH: {len(high)} | MEDIUM: {len(medium)}\n") + f.write("-" * 70 + "\n\n") + + for label, group in [("HIGH PRIORITY — IMMEDIATE ACTION", high), + ("MEDIUM PRIORITY", medium)]: + if group: + f.write(f">>> {label} <<<\n\n") + for i, a in enumerate(group, 1): + f.write(f"{i}. [{a.get('id', '?')}] {a.get('headline_english', 'N/A')}\n") + f.write(f" Category: {a.get('category', '?')}\n") + f.write(f" WHY: {a.get('political_significance', 'N/A')}\n") + f.write(f" ATTACK: {a.get('attack_angle', 'N/A')}\n\n") + f.write("-" * 70 + "\n") + + f.write("END OF BRIEF\n") + + +# --------------------------------------------------------------------------- +# Orchestrator +# --------------------------------------------------------------------------- +def process_pdf(pdf_path, job_dir, target_w, target_h, job_id): + job_dir = Path(job_dir) + pages_dir = job_dir / "pages" + articles_root = job_dir / "articles" + pages_dir.mkdir(parents=True, exist_ok=True) + articles_root.mkdir(parents=True, exist_ok=True) + + # Step 1: Render PDF pages + _set_status(job_id, "Rendering PDF pages...") + pages = render_pdf_pages(pdf_path, pages_dir, target_w, target_h) + + all_records = [] + all_selected = [] + + for p in pages: + n = p["page"] + page_img_path = p["path"] + + # Step 2: PaddleOCR layout detection — precise bounding boxes + _set_status(job_id, f"Page {n}: detecting layout regions...") + regions = detect_regions(page_img_path) + (pages_dir / f"page_{n:03d}.regions.json").write_text( + json.dumps({"page": n, "regions": regions}, indent=2, ensure_ascii=False) + ) + print(f" Page {n}: {len(regions)} layout regions detected") + + # Step 3: Group regions into articles + _set_status(job_id, f"Page {n}: grouping articles...") + page_img = Image.open(page_img_path) + articles, enriched = group_into_articles(page_img_path, regions, p["width"], page_img=page_img) + + (pages_dir / f"page_{n:03d}.articles.json").write_text( + json.dumps({"page": n, "articles": articles, "enriched_regions": enriched}, + indent=2, ensure_ascii=False) + ) + + # --- DETAILED GROUPING LOG --- + print(f"\n{'='*70}") + print(f"GROUPING RESULTS — PAGE {n}") + print(f"{'='*70}") + print(f" Input: {len(regions)} regions → Output: {len(articles)} articles") + print() + + # Build region lookup + region_map = {r["id"]: r for r in regions} + + for art in articles: + aid = art.get("article_id", "?") + member_ids = art.get("member_region_ids", art.get("member_regions", [])) + title_id = art.get("title_region_id", art.get("title_region")) + hint = (art.get("headline_hint") or "")[:60] + dateline = art.get("dateline", "") + grouped_by = art.get("grouped_by", "") + bbox = art.get("bbox", []) + + print(f" ARTICLE {aid}: \"{hint}\"") + if dateline: + print(f" Dateline: {dateline}") + if title_id: + title_r = region_map.get(title_id, {}) + print(f" Title region: ID {title_id} (type={title_r.get('type','?')}, bbox={title_r.get('bbox',[])})") + print(f" Member regions ({len(member_ids)}): {member_ids}") + + # Show each member region's details + if member_ids: + min_x, min_y, max_x, max_y = 99999, 99999, 0, 0 + for rid in member_ids: + r = region_map.get(rid, {}) + rb = r.get("bbox", [0,0,0,0]) + print(f" Region {rid:3d}: type={r.get('type','?'):15s} bbox=[{rb[0]:5d},{rb[1]:5d},{rb[2]:5d},{rb[3]:5d}] size={rb[2]-rb[0]:5d}x{rb[3]-rb[1]:5d}px") + min_x = min(min_x, rb[0]) + min_y = min(min_y, rb[1]) + max_x = max(max_x, rb[2]) + max_y = max(max_y, rb[3]) + print(f" Merged bbox: [{min_x},{min_y},{max_x},{max_y}] size={max_x-min_x}x{max_y-min_y}px") + if bbox: + print(f" Article bbox: {bbox}") + print() + + # Summary stats + region_counts = [len(art.get("member_region_ids", art.get("member_regions", []))) for art in articles] + if region_counts: + print(f" STATS:") + print(f" Articles: {len(articles)}") + print(f" Regions per article: min={min(region_counts)}, max={max(region_counts)}, avg={sum(region_counts)/len(region_counts):.1f}") + large = [i+1 for i,c in enumerate(region_counts) if c > 6] + if large: + print(f" ⚠️ Large articles (>6 regions): {large} — may have merged multiple stories") + print(f"{'='*70}\n") + + # Draw grouped articles on debug image with different colors per article + try: + from PIL import ImageDraw + debug_img = Image.open(page_img_path).copy() + draw = ImageDraw.Draw(debug_img) + + # Dark distinct colors for each article + article_colors = [ + (255, 0, 0), # red + (0, 0, 255), # blue + (0, 150, 0), # dark green + (180, 0, 180), # purple + (200, 100, 0), # dark orange + (0, 150, 150), # teal + (150, 0, 0), # dark red + (0, 0, 150), # dark blue + (100, 100, 0), # olive + (200, 0, 100), # magenta + (0, 100, 200), # steel blue + (150, 75, 0), # brown + (100, 0, 150), # dark purple + (0, 130, 80), # dark teal + (180, 50, 50), # brick red + (50, 50, 180), # indigo + (0, 180, 0), # green + (180, 0, 0), # crimson + (0, 0, 180), # navy + (150, 150, 0), # dark yellow + (200, 50, 150), # pink + (50, 150, 50), # forest green + (100, 50, 200), # violet + (200, 150, 50), # gold + (50, 100, 150), # slate + ] + + for idx, art in enumerate(articles): + color = article_colors[idx % len(article_colors)] + aid = art.get("article_id", idx+1) + member_ids = art.get("member_region_ids", art.get("member_regions", [])) + + for rid in member_ids: + r = region_map.get(rid, {}) + rb = r.get("bbox", [0,0,0,0]) + # Draw thick border with article color + draw.rectangle([rb[0], rb[1], rb[2], rb[3]], outline=color, width=5) + # Label with article ID + draw.text((rb[0]+8, rb[1]+8), f"A{aid}", fill=color) + + # Draw merged article bbox as thick dark outer border + if member_ids: + all_bboxes = [region_map.get(rid, {}).get("bbox", [0,0,0,0]) for rid in member_ids] + mx1 = min(b[0] for b in all_bboxes) + my1 = min(b[1] for b in all_bboxes) + mx2 = max(b[2] for b in all_bboxes) + my2 = max(b[3] for b in all_bboxes) + # Thick dark outer border around the whole article + draw.rectangle([mx1-15, my1-15, mx2+15, my2+15], outline=color, width=15) + # Dark background label + label = f"Article {aid}: {(art.get('headline_hint') or '')[:30]}" + draw.rectangle([mx1-10, my1-40, mx1 + len(label)*12, my1-10], fill=color) + draw.text((mx1-5, my1-38), label, fill="white") + + debug_path = page_img_path.replace(".png", "_articles_debug.png") + debug_img.save(debug_path) + print(f"ARTICLES DEBUG IMAGE saved: {debug_path}") + print(f" Each article has a unique color. Regions with same color = same article.") + print() + except Exception as e: + print(f" Could not create articles debug image: {e}") + # + # # Step 4: Crop ALL articles with precise PaddleOCR boundaries + # _set_status(job_id, f"Page {n}: cropping {len(articles)} articles...") + # records = crop_articles(page_img_path, articles, articles_root, n) + # print(f" Page {n}: cropped {len(records)} articles") + # + # # Build article ID + location + headline list for Claude + # article_ids_with_locations = [] + # for rec in records: + # art_dir = Path(rec["dir"]) + # art_name = art_dir.name + # info_path = art_dir / "info.json" + # if info_path.exists(): + # info = json.loads(info_path.read_text()) + # bbox = info.get("bbox", [0, 0, 0, 0]) + # hint = info.get("headline_hint", "") + # pw, ph = page_img.size + # loc = f"headline: \"{hint}\", position: x:{bbox[0]}-{bbox[2]}, y:{bbox[1]}-{bbox[3]}" + # else: + # loc = "unknown" + # article_ids_with_locations.append((art_name, loc)) + # + # # Step 5: Political filter — send page image + article IDs to Claude + # if article_ids_with_locations: + # _set_status(job_id, f"Page {n}: filtering {len(records)} articles for political significance...") + # selected = _filter_articles_with_claude(page_img_path, article_ids_with_locations, n, job_dir=str(job_dir)) + # + # if selected: + # selected_ids = {s["id"] for s in selected} + # print(f" Page {n}: {len(selected)} politically significant articles: {selected_ids}") + # + # # Step 6: Keep only selected articles, remove the rest + # for rec in records: + # art_dir = Path(rec["dir"]) + # art_name = art_dir.name + # if art_name in selected_ids: + # # Update info.json with political analysis + # sel_info = next((s for s in selected if s["id"] == art_name), {}) + # info_path = art_dir / "info.json" + # if info_path.exists(): + # info = json.loads(info_path.read_text()) + # info["headline_english"] = sel_info.get("headline_english", "") + # info["political_significance"] = sel_info.get("political_significance", "") + # info["attack_angle"] = sel_info.get("attack_angle", "") + # info["priority"] = sel_info.get("priority", "") + # info["category"] = sel_info.get("category", "") + # info["grouped_by"] = "paddleocr + political_filter" + # info_path.write_text(json.dumps(info, indent=2, ensure_ascii=False)) + # all_records.append(rec) + # all_selected.append(sel_info) + # else: + # # Delete non-political article + # import shutil + # shutil.rmtree(str(art_dir), ignore_errors=True) + # else: + # # No political articles on this page — delete all cropped articles + # print(f" Page {n}: no political articles found, removing all {len(records)} articles") + # import shutil + # for rec in records: + # shutil.rmtree(str(Path(rec["dir"])), ignore_errors=True) + # else: + # all_records.extend(records) + + print(f" Page {n}: Steps 3-6 skipped (commented out). Check regions JSON.") + + # --- SUMMARY ACROSS ALL PAGES --- + print(f"\n{'='*70}") + print(f"LAYOUT DETECTION SUMMARY — ALL PAGES") + print(f"{'='*70}") + + # Reload all regions from saved JSON + all_page_summaries = [] + grand_total = 0 + grand_type_counts = {} + + for p in pages: + n = p["page"] + regions_file = pages_dir / f"page_{n:03d}.regions.json" + if regions_file.exists(): + data = json.loads(regions_file.read_text()) + page_regions = data.get("regions", []) + grand_total += len(page_regions) + + # Count by type + type_counts = {} + for r in page_regions: + t = r["type"] + type_counts[t] = type_counts.get(t, 0) + 1 + grand_type_counts[t] = grand_type_counts.get(t, 0) + 1 + + # Estimate articles: doc_title + paragraph_title = potential article starts + doc_titles = type_counts.get("doc_title", 0) + para_titles = type_counts.get("paragraph_title", 0) + + print(f"\n PAGE {n}:") + print(f" Total regions: {len(page_regions)}") + print(f" Types: {dict(sorted(type_counts.items(), key=lambda x: -x[1]))}") + print(f" Potential articles (doc_title): {doc_titles}") + print(f" Potential sub-articles (paragraph_title): {para_titles}") + print(f" Estimated article range: {doc_titles} (min) to {doc_titles + para_titles} (max)") + + all_page_summaries.append({ + "page": n, + "total_regions": len(page_regions), + "type_counts": type_counts, + "doc_titles": doc_titles, + "paragraph_titles": para_titles, + }) + + print(f"\n GRAND TOTAL:") + print(f" Pages: {len(pages)}") + print(f" Total regions: {grand_total}") + print(f" All types: {dict(sorted(grand_type_counts.items(), key=lambda x: -x[1]))}") + total_doc = grand_type_counts.get("doc_title", 0) + total_para = grand_type_counts.get("paragraph_title", 0) + print(f" Estimated articles across all pages: {total_doc} (min) to {total_doc + total_para} (max)") + print(f"{'='*70}\n") + + # Save summary JSON + summary_path = job_dir / "layout_summary.json" + summary_path.write_text(json.dumps({ + "total_pages": len(pages), + "total_regions": grand_total, + "grand_type_counts": grand_type_counts, + "estimated_articles_min": total_doc, + "estimated_articles_max": total_doc + total_para, + "pages": all_page_summaries, + }, indent=2, ensure_ascii=False), encoding="utf-8") + print(f"Summary saved: {summary_path}") + + # # Save political filter results + # if all_selected: + # (job_dir / "political_articles.json").write_text( + # json.dumps(all_selected, indent=2, ensure_ascii=False), encoding="utf-8" + # ) + # _generate_opposition_brief(all_selected, job_dir) + + # # Copy all kept article images into one folder + # all_images_dir = job_dir / "all_political_images" + # all_images_dir.mkdir(parents=True, exist_ok=True) + # import shutil + # for rec in all_records: + # art_dir = Path(rec["dir"]) + # img_path = art_dir / "article.png" + # if img_path.exists(): + # art_name = art_dir.name + # shutil.copy2(str(img_path), str(all_images_dir / f"{art_name}.png")) + # print(f" Copied {len(all_records)} images to {all_images_dir}") + + # # Create a single PDF with all political article images + # if all_records: + # pdf_images = [] + # for rec in sorted(all_records, key=lambda r: r.get("article_id", 0)): + # art_dir = Path(rec["dir"]) + # img_path = art_dir / "article.png" + # if img_path.exists(): + # pdf_images.append(Image.open(img_path).convert("RGB")) + # if pdf_images: + # pdf_path_out = job_dir / "political_articles.pdf" + # pdf_images[0].save(str(pdf_path_out), save_all=True, append_images=pdf_images[1:]) + # print(f" Created PDF with {len(pdf_images)} articles: {pdf_path_out}") + + _set_status(job_id, f"Done! Steps 1-2 complete. {len(regions)} regions detected.") + return {"pages": len(pages), "articles": 0} diff --git a/generate_pdf.py b/generate_pdf.py new file mode 100644 index 0000000..e69de29 diff --git a/political_filter.py b/political_filter.py new file mode 100644 index 0000000..b78d9c0 --- /dev/null +++ b/political_filter.py @@ -0,0 +1,229 @@ +""" +Political Article Filter — scans newspaper PDF pages using Claude Vision +to identify politically significant articles that opposition can use. + +Run after PDF is uploaded and pages are rendered to images. + +Usage: + python political_filter.py # processes latest job + python political_filter.py 20260526_012054 # specific job +""" +import os +import sys +import json +import base64 +import anthropic +from pathlib import Path + +OUTPUT_DIR = Path(__file__).parent / "output" + + +def encode_image(img_path): + with open(img_path, "rb") as f: + return base64.standard_b64encode(f.read()).decode("utf-8") + + +def scan_pages_for_political_articles(client, page_images, model="claude-sonnet-4-6"): + """Send all page images to Claude and get politically significant articles.""" + + content = [] + for i, img_path in enumerate(page_images, 1): + content.append({ + "type": "image", + "source": { + "type": "base64", + "media_type": "image/png", + "data": encode_image(img_path), + }, + }) + content.append({ + "type": "text", + "text": f"[This is page {i} of the newspaper]" + }) + + content.append({ + "type": "text", + "text": """You are a political analyst working for an opposition party in Telangana, India. + +Scan ALL pages of this Telugu newspaper carefully. Identify EVERY article that is politically significant — specifically issues that an opposition party can use for political mileage. + +INCLUDE articles about: +- Government failures, delays, broken promises +- Corruption, scams, misuse of funds by ruling party leaders or officials +- Price hikes, inflation affecting common people +- Farmer distress, crop losses, loan issues +- Infrastructure failures (roads, water, electricity, hospitals) +- Law and order failures, crime, police issues +- Unemployment, job losses +- Public grievances against government departments +- Controversial government policies or decisions +- Ruling party internal conflicts or embarrassments +- Any issue that can be raised in assembly, press conference, or public rally + +IGNORE articles about: +- Sports, entertainment, cinema +- Horoscopes, puzzles, classifieds +- International news with no local political angle +- Advertisements +- Purely factual/neutral government announcements with no controversy + +For each politically significant article, provide: +1. page_number: which page it's on +2. headline: the Telugu headline (copy exactly from image) +3. headline_english: English translation of headline +4. location: describe where on the page (top-left, center, bottom-right, etc.) +5. political_significance: why this matters for opposition (2-3 sentences) +6. attack_angle: how opposition can use this (1-2 sentences) +7. priority: "high", "medium", or "low" +8. category: one of ["corruption", "governance_failure", "public_grievance", "economy", "law_order", "health", "education", "infrastructure", "ruling_party_crisis", "policy_controversy", "other"] + +Return ONLY valid JSON array. No markdown, no explanation. Example: +[ + { + "page_number": 1, + "headline": "Telugu headline here", + "headline_english": "English translation", + "location": "top-right, large article with photo", + "political_significance": "Why this matters...", + "attack_angle": "How to use this...", + "priority": "high", + "category": "corruption" + } +] + +Be thorough. Don't miss any politically useful article. A good opposition analyst finds ammunition in everything.""" + }) + + response = client.messages.create( + model=model, + max_tokens=8192, + messages=[{"role": "user", "content": content}], + ) + + response_text = response.content[0].text.strip() + # Clean up if wrapped in markdown + if response_text.startswith("```"): + response_text = response_text.split("\n", 1)[1] + if response_text.endswith("```"): + response_text = response_text[:-3] + + return json.loads(response_text) + + +def process_job(job_id=None): + api_key = os.environ.get("ANTHROPIC_API_KEY") + if not api_key: + print("Error: Set ANTHROPIC_API_KEY environment variable") + sys.exit(1) + + # Find job directory + if job_id: + job_dir = OUTPUT_DIR / job_id + else: + jobs = sorted(OUTPUT_DIR.iterdir(), reverse=True) + if not jobs: + print("No jobs found") + sys.exit(1) + job_dir = jobs[0] + job_id = job_dir.name + + pages_dir = job_dir / "pages" + if not pages_dir.exists(): + print(f"No pages found in {job_dir}") + sys.exit(1) + + # Collect page images + page_images = sorted(pages_dir.glob("page_*.png")) + if not page_images: + print("No page images found") + sys.exit(1) + + print(f"Job: {job_id}") + print(f"Pages: {len(page_images)}") + print("Scanning for politically significant articles...") + print() + + client = anthropic.Anthropic(api_key=api_key) + articles = scan_pages_for_political_articles(client, page_images) + + # Sort by priority + priority_order = {"high": 0, "medium": 1, "low": 2} + articles.sort(key=lambda a: priority_order.get(a.get("priority", "low"), 3)) + + # Save results + output_path = job_dir / "political_articles.json" + with open(output_path, "w", encoding="utf-8") as f: + json.dump(articles, f, indent=2, ensure_ascii=False) + + # Print summary + print(f"Found {len(articles)} politically significant articles:") + print("=" * 70) + + high = [a for a in articles if a.get("priority") == "high"] + medium = [a for a in articles if a.get("priority") == "medium"] + low = [a for a in articles if a.get("priority") == "low"] + + print(f" HIGH priority: {len(high)}") + print(f" MEDIUM priority: {len(medium)}") + print(f" LOW priority: {len(low)}") + print() + + for i, article in enumerate(articles, 1): + priority_tag = f"[{article.get('priority', '?').upper()}]" + category = article.get("category", "other") + print(f"{i}. {priority_tag} [{category}] (Page {article.get('page_number', '?')})") + print(f" {article.get('headline', 'N/A')}") + print(f" ({article.get('headline_english', 'N/A')})") + print(f" Political angle: {article.get('attack_angle', 'N/A')}") + print() + + # Generate brief text file + brief_path = job_dir / "opposition_daily_brief.txt" + with open(brief_path, "w", encoding="utf-8") as f: + f.write("=" * 70 + "\n") + f.write(" OPPOSITION DAILY BRIEF\n") + f.write(f" Source: {job_id}\n") + f.write("=" * 70 + "\n\n") + + f.write(f"Total politically significant articles: {len(articles)}\n") + f.write(f"HIGH priority: {len(high)} | MEDIUM: {len(medium)} | LOW: {len(low)}\n") + f.write("-" * 70 + "\n\n") + + if high: + f.write(">>> HIGH PRIORITY — IMMEDIATE ACTION NEEDED <<<\n\n") + for i, a in enumerate(high, 1): + f.write(f"{i}. {a.get('headline', 'N/A')}\n") + f.write(f" ({a.get('headline_english', 'N/A')})\n") + f.write(f" Page: {a.get('page_number', '?')} | Category: {a.get('category', '?')}\n") + f.write(f" WHY IT MATTERS: {a.get('political_significance', 'N/A')}\n") + f.write(f" ATTACK ANGLE: {a.get('attack_angle', 'N/A')}\n\n") + + if medium: + f.write("-" * 70 + "\n") + f.write(">>> MEDIUM PRIORITY <<<\n\n") + for i, a in enumerate(medium, 1): + f.write(f"{i}. {a.get('headline', 'N/A')}\n") + f.write(f" ({a.get('headline_english', 'N/A')})\n") + f.write(f" Page: {a.get('page_number', '?')} | Category: {a.get('category', '?')}\n") + f.write(f" WHY IT MATTERS: {a.get('political_significance', 'N/A')}\n") + f.write(f" ATTACK ANGLE: {a.get('attack_angle', 'N/A')}\n\n") + + if low: + f.write("-" * 70 + "\n") + f.write(">>> LOW PRIORITY — TRACK FOR LATER <<<\n\n") + for i, a in enumerate(low, 1): + f.write(f"{i}. {a.get('headline', 'N/A')}\n") + f.write(f" ({a.get('headline_english', 'N/A')})\n") + f.write(f" ATTACK ANGLE: {a.get('attack_angle', 'N/A')}\n\n") + + f.write("=" * 70 + "\n") + f.write("END OF BRIEF\n") + + print("=" * 70) + print(f"Saved: {output_path}") + print(f"Saved: {brief_path}") + + +if __name__ == "__main__": + job_id = sys.argv[1] if len(sys.argv) > 1 else None + process_job(job_id) diff --git a/requirements.txt b/requirements.txt new file mode 100644 index 0000000..0d1e398 --- /dev/null +++ b/requirements.txt @@ -0,0 +1,8 @@ +Flask>=3.0 +PyMuPDF>=1.23 +Pillow>=10.0 +numpy>=1.24 +paddleocr==3.5.0 +paddlepaddle==3.0.0 +pytesseract>=0.3.10 +anthropic>=0.40 diff --git a/run_pipeline.py b/run_pipeline.py new file mode 100644 index 0000000..46d30fe --- /dev/null +++ b/run_pipeline.py @@ -0,0 +1,101 @@ +import os +import shutil +import argparse +from pathlib import Path +from datetime import datetime +import extractor + +def main(): + parser = argparse.ArgumentParser(description="Run the News-Scan end-to-end extraction pipeline.") + parser.add_argument( + "--max-pages", + type=int, + help="Limit processing to the first N pages." + ) + parser.add_argument( + "--pages", + type=str, + help="Comma-separated list of specific 1-based page numbers to process (e.g. 1,3,5)." + ) + + args = parser.parse_args() + + input_dir = Path("input") + processed_dir = Path("processed") + output_dir = Path("output") + + # Ensure directories exist + input_dir.mkdir(parents=True, exist_ok=True) + processed_dir.mkdir(parents=True, exist_ok=True) + output_dir.mkdir(parents=True, exist_ok=True) + + # Check for API key early + if not os.environ.get("ANTHROPIC_API_KEY"): + print("WARNING: 'ANTHROPIC_API_KEY' environment variable is not set!") + print("The political filter and AI insights generation will be SKIPPED.") + print("To generate the final PDF report, you must set this key.") + print("-" * 60) + + # Find the first PDF in the input folder + pdf_files = list(input_dir.glob("*.pdf")) + if not pdf_files: + print(f"Error: No PDF files found in {input_dir.absolute()}") + return + + pdf_path = pdf_files[0] + print(f"Found file to process: {pdf_path.name}") + + job_id = datetime.now().strftime("%Y%m%d_%H%M%S") + job_dir = output_dir / job_id + job_dir.mkdir(parents=True, exist_ok=True) + + # Parse selected pages + pages_to_process = None + if args.pages: + try: + pages_to_process = [int(p.strip()) for p in args.pages.split(",") if p.strip()] + print(f"Limiting processing to specific pages: {pages_to_process}") + except ValueError: + print("Error: --pages must be a comma-separated list of integers (e.g., 1,3,5)") + return + elif args.max_pages: + print(f"Limiting processing to first {args.max_pages} pages") + else: + print("Processing all pages of the document") + + print(f"Starting full pipeline extraction on {pdf_path}") + print(f"Output directory: {job_dir}") + print() + + # Run the full pipeline (extract, group, crop, OCR, filter) + result = extractor.process_pdf( + str(pdf_path), + str(job_dir), + 4200, + 7400, + job_id=job_id, + max_pages=args.max_pages, + pages_to_process=pages_to_process + ) + print(f"\nExtraction complete! Found {result['articles']} articles across {result['pages']} pages.") + + # Move the file from input to processed + dest_path = processed_dir / pdf_path.name + + # Ensure no collision in processed folder + if dest_path.exists(): + timestamp = datetime.now().strftime("%H%M%S") + dest_path = processed_dir / f"{pdf_path.stem}_{timestamp}{pdf_path.suffix}" + + shutil.move(str(pdf_path), str(dest_path)) + print(f"\nMoved processed file to: {dest_path}") + + # Check if the PDF report was generated + pdf_report = job_dir / "political_articles_insights.pdf" + if pdf_report.exists(): + print(f"\nSUCCESS: AI Insights PDF generated at {pdf_report}") + else: + print(f"\nNOTE: The AI Insights PDF was not generated. This usually means no political articles were found, or the ANTHROPIC_API_KEY was missing.") + +if __name__ == "__main__": + main() diff --git a/scratch_py/_blocks_snapshot.py b/scratch_py/_blocks_snapshot.py new file mode 100644 index 0000000..4edc24f --- /dev/null +++ b/scratch_py/_blocks_snapshot.py @@ -0,0 +1,21 @@ +import sys, json, contextlib, io +sys.path.insert(0, ".") +from pathlib import Path +import smart_extractor as se +from _lines import separator_barriers +D = Path("output/AndhraJyothi_Siddipet District_20260602_20260603_144919/pages") +out={} +for pg in (1,2,3,4): + png=D/f"page_{pg:03d}.png" + regs=json.loads((D/f"page_{pg:03d}.regions.json").read_text())["regions"] + with contextlib.redirect_stdout(io.StringIO()): + regs=se.drop_masthead_regions(regs,"andhra_jyothi",pg,png) + ds=se.find_article_starts_by_dateline(regs,str(png),"andhra_jyothi") + bars=separator_barriers(str(png),regs) + bl=se.cluster_all_article_blocks(regs,dateline_starts=ds,sep_lines=bars) + out[pg]={b["anchor_id"]:[b["kind"],tuple(sorted(b["members"]))] for b in bl} +tag=sys.argv[1] if len(sys.argv)>1 else "base" +Path(f"scratch_py/_snap_{tag}.json").write_text(json.dumps(out,indent=0)) +print(f"saved _snap_{tag}.json") +for pg in (1,2,3,4): + print(f"PAGE{pg}: {len(out[pg])} blocks anchors={sorted(out[pg])}") diff --git a/scratch_py/_boundaries.py b/scratch_py/_boundaries.py new file mode 100644 index 0000000..796d1ea --- /dev/null +++ b/scratch_py/_boundaries.py @@ -0,0 +1,204 @@ +"""Construct one boundary box per article by fusing three signals: + + 1. DATELINE pattern -> the MINIMUM set of articles (one per ', date:'). + 2. HEADLINE above -> the article's top anchor / column span (doc_title). + 3. STRAIGHT RULES -> hard walls: snap top/bottom to horizontal rules and + left/right to vertical rules between stories. + +Pure geometry on saved fixtures + the page PNG. No Claude, no API. + +construct_boundaries(regions, datelines, lines, size) -> list of: + {region_id, dateline, headline_region, bbox:[l,t,r,b], walls:{...}} +""" + + +def _cx(b): + return (b[0] + b[2]) / 2 + + +def construct_boundaries(regions, datelines, lines, size): + W, H = size + rmap = {r["id"]: r for r in regions} + h_rules = sorted(lines["h"], key=lambda L: L["y1"]) + v_rules = sorted(lines["v"], key=lambda L: L["x1"]) + dls = sorted(datelines, key=lambda s: s["bbox"][1]) + + # ---- duplicate-dateline dedup: PaddleOCR sometimes boxes the SAME physical text + # twice (and a false dateline match — an ad/phone line — then appears as two + # near-identical bodies). Two datelines are one article when EITHER their bodies + # overlap heavily (IoU>0.6) OR one body is mostly nested inside the other AND the + # dateline text is identical (the same town double-boxed as a thin line + a tall + # paragraph — IoU stays low but containment is near 1). Keep the larger body and + # inherit a headline from the dropped twin if the keeper had none. + def _area(b): + return (b[2]-b[0]) * (b[3]-b[1]) + + def _same_dateline(sa, sb): + a, b = sa["bbox"], sb["bbox"] + ix = max(0, min(a[2], b[2]) - max(a[0], b[0])) + iy = max(0, min(a[3], b[3]) - max(a[1], b[1])) + inter = ix * iy + if not inter: + return False + aa, ba = _area(a), _area(b) + iou = inter / (aa + ba - inter) if (aa + ba - inter) else 0.0 + if iou > 0.6: + return True + contain = inter / min(aa, ba) if min(aa, ba) else 0.0 + ta = (sa.get("dateline") or "").strip() + tb = (sb.get("dateline") or "").strip() + return contain > 0.8 and ta != "" and ta == tb + + deduped = [] + for s in dls: + dup = next((k for k in deduped if _same_dateline(s, k)), None) + if dup is None: + deduped.append(s) + else: # keep larger body; inherit a missing headline + keep, drop = (s, dup) if _area(s["bbox"]) > _area(dup["bbox"]) else (dup, s) + if keep.get("headline_region") not in rmap and drop.get("headline_region") in rmap: + keep = {**keep, "headline_region": drop["headline_region"]} + deduped[deduped.index(dup)] = keep + dls = deduped + + # ---- shared-headline dedup: a doc_title can only head ONE article. When two + # datelines both matched the same headline (stacked in a column, or a 2-col + # headline with a missed neighbour), keep it for the dateline with the SMALLEST + # positive gap below the headline; the others lose it (anchor on their own body). + by_hl = {} + for s in dls: + h = s.get("headline_region") + if h in rmap: + by_hl.setdefault(h, []).append(s) + hid_of = {id(s): s.get("headline_region") for s in dls} + for h, group in by_hl.items(): + if len(group) <= 1: + continue + hb = rmap[h]["bbox"] + below = [s for s in group if s["bbox"][1] >= hb[3] - 30] + keeper = min(below, key=lambda s: s["bbox"][1] - hb[3]) if below \ + else min(group, key=lambda s: abs(s["bbox"][1] - hb[3])) + for s in group: + if s is not keeper: + hid_of[id(s)] = None + + def _anchor_rect(ns): + """A neighbour's article ANCHOR = its headline (if it owns one) UNION its + dateline body. Floors are computed against this rect — not the bare body — + so a story ends at the TOP of the next story's HEADLINE, and a wide 2-column + headline that reaches sideways into this column still acts as a wall.""" + b = ns["bbox"] + h = hid_of.get(id(ns)) + if h in rmap: + hb = rmap[h]["bbox"] + return (min(b[0], hb[0]), min(b[1], hb[1]), + max(b[2], hb[2]), max(b[3], hb[3])) + return (b[0], b[1], b[2], b[3]) + + def _covers(L, y0, y1): + # the rule must span the MAJORITY of [y0,y1], not merely touch an edge + ov = min(L["y2"], y1) - max(L["y1"], y0) + return ov >= 0.5 * max(1, y1 - y0) + + def vwall_left(cx, y0, y1): + cands = [L for L in v_rules if L["x1"] < cx - 20 and _covers(L, y0, y1)] + return max((L["x1"] for L in cands), default=0) + + def vwall_right(cx, y0, y1): + cands = [L for L in v_rules if L["x1"] > cx + 20 and _covers(L, y0, y1)] + return min((L["x1"] for L in cands), default=W) + + def hrule_between(x0, x1, y_lo, y_hi, want_above): + """Return the horizontal rule whose span covers [x0,x1] within (y_lo,y_hi). + want_above=True -> the LOWEST such rule (closest above the body's top is the + bottom-most of those above); we pick nearest to the reference edge.""" + hits = [L for L in h_rules + if L["y1"] > y_lo and L["y1"] < y_hi + and L["x1"] <= x1 - 30 and L["x2"] >= x0 + 30] + if not hits: + return None + # nearest to the reference edge: above -> max y (closest below ceiling ref), + # below -> min y (closest above floor ref) + return max(hits, key=lambda L: L["y1"]) if want_above \ + else min(hits, key=lambda L: L["y1"]) + + out = [] + for i, s in enumerate(dls): + db = s["bbox"] + cx = _cx(db) + hid = hid_of[id(s)] + hb = rmap[hid]["bbox"] if (hid in rmap) else None + + # ---- horizontal span: UNION of headline + dateline body so the body is + # always contained; top anchor is the headline top if present. + if hb: + left0, right0 = min(hb[0], db[0]), max(hb[2], db[2]) + top_anchor = hb[1] + else: + left0, right0 = db[0], db[2] + top_anchor = db[1] + + # ---- CEILING: snap up to a horizontal rule just above the anchor (but no + # higher than the previous article's dateline in this column). + prev_y = 0 + for ps in dls: + if ps is s: + continue + pb = ps["bbox"] + if pb[0] - 40 <= cx <= pb[2] + 40 and pb[3] <= top_anchor: + prev_y = max(prev_y, pb[3]) + ceil_rule = hrule_between(left0, right0, prev_y, top_anchor + 20, want_above=True) + # a ceiling rule only counts if it sits a sane distance ABOVE the anchor + # (a far-away rule is a different story's divider, not this article's top). + top = top_anchor + if ceil_rule and 0 <= (top_anchor - ceil_rule["y1"]) <= 500: + top = ceil_rule["y1"] + else: + ceil_rule = None + top = min(top, top_anchor) # never cut INTO the article from above + + # ---- FLOOR: nearest boundary directly below in THIS dateline's OWN column + # (own column, not the wide headline span — a neighbour under a 2-col + # headline must clip the SIDE, not act as the floor). + own_l, own_r = db[0], db[2] + next_y = H + for ns in dls: + if ns is s: + continue + na = _anchor_rect(ns) # headline ∪ body of neighbour + if na[1] > db[1] + 30 and not (na[2] < own_l + 20 or na[0] > own_r - 20): + next_y = min(next_y, na[1]) + floor_rule = hrule_between(own_l, own_r, db[3], next_y + 5, want_above=False) + bottom = floor_rule["y1"] if floor_rule else next_y + bottom = max(bottom, db[3]) # never cut INTO the article from below + + # ---- SIDE-CLIP: a neighbour dateline that sits beside this body within the + # (wide) headline span clips the corresponding edge, so a left-column + # story never swallows the right column's separate story. + for ns in dls: + if ns is s: + continue + nb = ns["bbox"] + if nb[3] > top_anchor + 20 and nb[1] < bottom - 20: # vertical overlap + if own_r - 10 <= nb[0] < right0: # neighbour to the right + right0 = min(right0, nb[0]) + if left0 < nb[2] <= own_l + 10: # neighbour to the left + left0 = max(left0, nb[2]) + + # ---- side walls: snap to vertical rules spanning the article's y-band + lwall = vwall_left(cx, top, bottom) + rwall = vwall_right(cx, top, bottom) + left = max(left0 - 20, lwall) + right = min(right0 + 20, rwall) + left = min(left, db[0]) # guarantee the dateline body fits + right = max(right, db[2]) + + out.append({ + "region_id": s["region_id"], + "dateline": s.get("dateline", ""), + "headline_region": hid, + "bbox": [int(left), int(top), int(right), int(bottom)], + "walls": {"ceil_rule": bool(ceil_rule), "floor_rule": bool(floor_rule), + "lwall": int(lwall), "rwall": int(rwall)}, + }) + return out diff --git a/scratch_py/_continuation.py b/scratch_py/_continuation.py new file mode 100644 index 0000000..0a945ab --- /dev/null +++ b/scratch_py/_continuation.py @@ -0,0 +1,462 @@ +"""Cross-page article-continuation linker (classical, no Claude / no API). + +Telugu papers "jump" long stories across pages: + + • bottom of the source article : 'మిగతా వ పేజీలో...' (rest is on page N) + • top of the continuation page : the SAME headline repeated, plus the marker + '(మొదటి పేజీ తరువాయి)' (continued from page 1) + +We REQUIRE BOTH signals: the jump-from marker on the source page (which also tells +us the target page N) AND a repeated headline on page N that matches the source +article's headline. We then build a boundary box for the continuation by injecting +a SYNTHETIC dateline anchored on that repeated headline and running the existing +boundary engine (so the continuation floors correctly against page-N's own +articles), and finally STITCH the source crop + continuation crop into ONE tall +image per article. + +Pure geometry + Tesseract on saved fixtures (page PNGs + regions.json + the +per-article info.json the pipeline already wrote). No Claude. + + link_continuations(run_dir) -> list of link records (also written to + /continuations/links.json, stitched PNGs alongside). +""" +import difflib +import json +import re +import unicodedata +from pathlib import Path + +from PIL import Image + +from _lines import detect_separator_lines + +try: + from smart_extractor import _dateline_in_text +except Exception: # keep the module importable in isolation + def _dateline_in_text(_t, _paper=None): + return None + +_ZW = "‌‍" # zero-width non-joiner / joiner +_TERMINATORS = ".।?!" # sentence-enders: full stop, danda, ?, ! +_TELUGU_DIGITS = {ord("౦") + i: str(i) for i in range(10)} + +# jump-FROM: 'మిగతా వ పేజీ...' — tolerate OCR noise between the tokens. +_JUMP_FROM = re.compile(r"మిగ[తథధ]ా.{0,10}?([0-9౦-౯]{1,2})\s*వ?\s*పే[జీిౌ]", re.UNICODE) +# jump-TO: 'మొదటి పేజీ తరువాయి' — the continuation marker on the target page. +_JUMP_TO = re.compile(r"మొదట[ిి]?\s*పే[జీి].{0,6}?తరు[వ]?ా[యి]", re.UNICODE) + + +def _paper_of_height(h): + if 7040 <= h <= 7060: + return "sakshi" + if h in (6833, 6549): + return "andhra_jyothi" + if h == 6422: + return "namaste_telangana" + return f"h{h}" + + +def _ocr(page, box): + import pytesseract + try: + return pytesseract.image_to_string(page.crop((int(box[0]), int(box[1]), + int(box[2]), int(box[3]))), + lang="tel") + except Exception: + return "" + + +def _page_num_from(text): + m = _JUMP_FROM.search(text or "") + if not m: + return None + digits = m.group(1).translate(_TELUGU_DIGITS) + dm = re.search(r"\d+", digits) + if not dm: + return None + d = dm.group() + # OCR frequently DOUBLE-RECOGNISES a single glyph as the Latin digit followed + # by its Telugu twin (e.g. '3౩' → '33'). District editions only ever run a + # handful of pages, so a run of identical digits is that artefact, not page 33. + if len(d) == 2 and d[0] == d[1]: + d = d[0] + return int(d) + + +def _norm(s): + s = unicodedata.normalize("NFC", s or "") + s = "".join(c for c in s if c not in _ZW) + return re.sub(r"[\s.,!?:;\-()।\"'‘’]", "", s) + + +def _similar(a, b): + a, b = _norm(a), _norm(b) + if not a or not b: + return 0.0 + return difflib.SequenceMatcher(None, a, b).ratio() + + +def _find_jump_from(page, art_bbox): + """Scan the BOTTOM band of the source article for the 'మిగతా Nవ పేజీలో' marker; + fall back to the whole article box. Returns the target page number or None.""" + l, t, r, b = art_bbox + band_top = max(t, b - 340) + n = _page_num_from(_ocr(page, (l, band_top, r, b))) + if n is None: # marker may sit a little higher + n = _page_num_from(_ocr(page, (l, t, r, b))) + return n + + +def _best_headline(regs, page, headline_tel, thresh=0.60): + """Among the doc_titles on the target page, return (region, score, has_marker) + for the one whose OCR text best matches the source headline (>= thresh).""" + best, best_score = None, 0.0 + for r in regs: + if r.get("type") != "doc_title": + continue + score = _similar(_ocr(page, r["bbox"]), headline_tel) + if score > best_score: + best, best_score = r, score + if not best or best_score < thresh: + return None, best_score, False + hb = best["bbox"] + below = _ocr(page, (hb[0], hb[3], hb[2], min(hb[3] + 240, page.height))) + has_marker = bool(_JUMP_TO.search(below) or _JUMP_TO.search(_ocr(page, hb))) + return best, best_score, has_marker + + +def _ends_terminated(text): + """True if the column's last word ends a sentence (., danda, ?, !), ignoring + trailing quotes/brackets. This is the (noisy, OCR-based) 'article concluded' + signal — used only AFTER the structural stop-tests have had their say.""" + t = (text or "").rstrip() + t = t.rstrip("\"'’”)］]》」』 ") + return bool(t) and t[-1] in _TERMINATORS + + +def _col_width(regs): + """Robust single-column width = median width of the page's text boxes.""" + ws = sorted(r["bbox"][2] - r["bbox"][0] for r in regs if r.get("type") == "text") + return ws[len(ws) // 2] if ws else 700 + + +def _has_body_below(regs, b): + """True if a text box sits below box `b` within its horizontal span — i.e. `b` + is a headline opening its OWN article (a new story), not a trailing label or an + inline emphasis run. Used to let a paragraph_title act as an article floor.""" + for r in regs: + if r.get("type") == "text": + rb = r["bbox"] + if rb[1] > b[1] + 5 and rb[0] < b[2] and rb[2] > b[0]: + return True + return False + + +def _is_floor_head(regs, r, hid): + """A region that floors a continuation: any doc_title, OR a paragraph_title that + has its own body beneath it (the start of a DIFFERENT article — e.g. a + 'continued from page N' sub-head sitting below our continuation block).""" + if r.get("id") == hid: + return False + t = r.get("type") + if t == "doc_title": + return True + return t == "paragraph_title" and _has_body_below(regs, r["bbox"]) + + +def _floor_under(regs, lines, hb, hid, x0, x1, H): + """Lowest boundary below the header within the corridor [x0,x1]: the top of the + next headline (doc_title, or an article-starting paragraph_title), or a + horizontal rule, whichever comes first (else page bottom).""" + f = H + for r in regs: + b = r["bbox"] + if (_is_floor_head(regs, r, hid) + and b[1] > hb[3] + 20 and b[0] < x1 and b[2] > x0): + f = min(f, b[1]) + for ln in (lines.get("h") or []): + y = ln["y1"] + if hb[3] + 20 < y < f and ln["x1"] < x1 and ln["x2"] > x0: + f = min(f, y) + return f + + +def _floor_below_body(regs, lines, hb, hid, x0, x1, body, H): + """Lowest article boundary in corridor [x0,x1]. A next-headline or horizontal + rule only counts as the floor when NO included body text continues *below* it in + an overlapping column — that distinguishes the next article's wide headline (no + body beneath, within this block) from an interior sub-head / column-top rule + (article body keeps going underneath it).""" + def _no_body_below(bx0, bx2, yt): + for bb in body: + if bb[3] > yt + 10 and bb[0] < bx2 and bb[2] > bx0: + return False + return True + + f = H + for r in regs: + b = r["bbox"] + if (_is_floor_head(regs, r, hid) + and b[1] > hb[3] + 20 and b[0] < x1 and b[2] > x0 + and _no_body_below(b[0], b[2], b[1])): + f = min(f, b[1]) + for ln in (lines.get("h") or []): + y = ln["y1"] + if hb[3] + 20 < y < f and ln["x1"] < x1 and ln["x2"] > x0 \ + and _no_body_below(ln["x1"], ln["x2"], y): + f = min(f, y) + return f + + +def _vrule_in_gutter(lines, x0, x1, y0, y1): + """A printed vertical rule sitting in the gutter [x0,x1] across most of [y0,y1].""" + for ln in (lines.get("v") or []): + if x0 - 8 <= ln["x1"] <= x1 + 8: + ov = min(ln["y2"], y1) - max(ln["y1"], y0) + if ov > 0.4 * (y1 - y0): + return True + return False + + +def _track_starts_with_title(regs, items, hb): + """STOP-test 5: does a headline / sub-head box open the next column?""" + xs = min(it["bbox"][0] for it in items) + xe = max(it["bbox"][2] for it in items) + ytop = min(it["bbox"][1] for it in items) + for r in regs: + if r.get("type") in ("doc_title", "paragraph_title"): + b = r["bbox"] + if b[0] < xe and b[2] > xs and b[1] > hb[3] + 20 and abs(b[1] - ytop) < 130: + return True + return False + + +def _continuation_crop(regs, png_path, paper, headline_region): + """Crop the FULL continuation article by GROWING RIGHT from the headline. + + A continuation's body is usually wider than its header and spans several columns. + We start at the header's column and walk right column-by-column, absorbing the + next column unless a STOP-test fires: + + 1 no next column 5 next column opens with a headline/sub-head + 2 vertical-rule wall 6 current column's last word ends with a period + 3 horizontal-rule break 7 grammatical bridge fails (approx: 4+6 absent) + 4 next column opens a dateline + + Structural tests (1-5) outrank the OCR period (6). The crop is the single + rectangle enclosing every absorbed column (+ any photo inside the block), + floored at the next headline / horizontal rule. + + Returns (PIL.Image, debug_bbox) or (None, None).""" + page = Image.open(png_path).convert("RGB") + H = page.height + hb = headline_region["bbox"] + hid = headline_region.get("id") + cx = (hb[0] + hb[2]) / 2 + lines = detect_separator_lines(png_path) + colW = _col_width(regs) + + # provisional floor under the header's own span gives us the band to cluster in + F0 = _floor_under(regs, lines, hb, hid, hb[0], hb[2], H) + + # body text boxes in the band, clustered into left-to-right COLUMN TRACKS + band = [r for r in regs if r.get("type") == "text" + and (r["bbox"][1] + r["bbox"][3]) / 2 < F0 + and r["bbox"][3] > hb[1] + and r["bbox"][0] >= hb[0] - colW * 0.5] # drop columns left of the header + if not band: + return None, None + tracks = [] + for r in sorted(band, key=lambda r: r["bbox"][0]): + for tr in tracks: + if abs(r["bbox"][0] - tr["x0"]) < colW * 0.5: + tr["items"].append(r) + break + else: + tracks.append({"x0": r["bbox"][0], "items": [r]}) + tracks.sort(key=lambda t: t["x0"]) + + # START column = the track straddling the headline centre (else nearest) + start = None + for i, t in enumerate(tracks): + if min(it["bbox"][0] for it in t["items"]) <= cx <= max(it["bbox"][2] for it in t["items"]): + start = i + break + if start is None: + start = min(range(len(tracks)), key=lambda i: abs(tracks[i]["x0"] - cx)) + + import os + DBG = os.environ.get("CONT_DEBUG") + if DBG: + print(f"[dbg] hb={hb} cx={cx} colW={colW} F0={F0}") + for ti, t in enumerate(tracks): + print(f"[dbg] track {ti} x0={t['x0']} items=" + + ",".join(f"{it.get('id')}{it['bbox']}" for it in t['items'])) + print(f"[dbg] start track = {start}") + + # GROW RIGHT, applying the stop-tests at each gutter + included = [start] + i = start + while True: + cur = tracks[i]["items"] + cur_r = max(it["bbox"][2] for it in cur) + nxt = next((j for j in range(i + 1, len(tracks)) + if tracks[j]["x0"] > cur_r - colW * 0.3), None) + if nxt is None: # 1 no next column + if DBG: print(f"[dbg] from {i}: STOP test1 no next column") + break + nt = tracks[nxt]["items"] + nt_l = min(it["bbox"][0] for it in nt) + y0, y1 = hb[1], F0 + if _vrule_in_gutter(lines, cur_r, nt_l, y0, y1): # 2 vertical wall + if DBG: print(f"[dbg] from {i} to {nxt}: STOP test2 vrule wall {cur_r}-{nt_l}") + break + top_reg = min(nt, key=lambda it: it["bbox"][1]) + tb = top_reg["bbox"] + head_txt = _ocr(page, (tb[0], tb[1], tb[2], min(tb[1] + 150, tb[3]))) + if _dateline_in_text(head_txt, paper): # 4 dateline opens next col + if DBG: print(f"[dbg] from {i} to {nxt}: STOP test4 dateline '{head_txt[:40]}'") + break + if _track_starts_with_title(regs, nt, hb): # 5 headline opens next col + if DBG: print(f"[dbg] from {i} to {nxt}: STOP test5 title opens next") + break + bot_reg = max(cur, key=lambda it: it["bbox"][3]) + bot_txt = _ocr(page, bot_reg["bbox"]) + if _ends_terminated(bot_txt): # 6 period concludes + if DBG: print(f"[dbg] from {i} to {nxt}: STOP test6 period; tail='{bot_txt[-30:]}'") + break + if DBG: print(f"[dbg] grow {i} -> {nxt}") + included.append(nxt) # 7 bridge holds -> grow + i = nxt + if DBG: print(f"[dbg] included tracks = {included}") + + # FINAL rectangle over the absorbed columns, floored against the real corridor + left = hb[0] + right = max(max(it["bbox"][2] for it in tracks[k]["items"]) for k in included) + body = [it["bbox"] for k in included for it in tracks[k]["items"]] + F = _floor_below_body(regs, lines, hb, hid, left, right, body, H) + if DBG: print(f"[dbg] left={left} right={right} F={F}") + + members = list(body) + for r in regs: # sweep in photos / captions inside the block + if r.get("type") in ("image", "figure_title", "paragraph_title", "doc_title"): + b = r["bbox"] + bcx = (b[0] + b[2]) / 2 + if left - 10 <= bcx <= right + 10 and b[3] <= F and b[3] > hb[1] - 320: + members.append(b) + + left = min(left, min(m[0] for m in members)) + top = min(hb[1], min(m[1] for m in members)) + bot = min(int(F) - 8, max(m[3] for m in members) + 22) + box = (int(left), int(max(0, top - 6)), int(right), int(bot)) + return page.crop(box), list(box) + + +def _stitch(top_png, cont_img, out_path): + """Stack the source page-1 crop above the composed continuation image.""" + top = Image.open(top_png).convert("RGB") + W = max(top.width, cont_img.width) + sep = 18 + combo = Image.new("RGB", (W, top.height + sep + cont_img.height), "white") + combo.paste(top, (0, 0)) + combo.paste(cont_img, (0, top.height + sep)) + combo.save(out_path) + return combo.size + + +def link_continuations(run_dir, out_dir=None, match_thresh=0.60, verbose=True): + run = Path(run_dir) + pages_dir = run / "pages" + arts_dir = run / "articles" + out_dir = Path(out_dir) if out_dir else (run / "continuations") + out_dir.mkdir(parents=True, exist_ok=True) + + p1 = pages_dir / "page_001.png" + paper = _paper_of_height(Image.open(p1).height) if p1.exists() else "unknown" + + results = [] + for info_path in sorted(arts_dir.glob("*/info.json")): + info = json.loads(info_path.read_text()) + P, bbox = info.get("page"), info.get("bbox") + headline = info.get("headline_telugu") or "" + if not P or not bbox: + continue + # Continuations only ever START on page 1 — the front page jumps long lead + # stories onto inside pages. Articles on pages 2+ are self-contained, so we + # never scan them for a jump-from marker. + if P != 1: + continue + src_png = pages_dir / f"page_{P:03d}.png" + if not src_png.exists(): + continue + + N = _find_jump_from(Image.open(src_png), bbox) # SIGNAL 1: jump-from + if not N or N == P: + continue + + # The jump-from page DIGIT is OCR-noisy (Telugu ౪/౬ confusion) and can point + # at a page outside this extraction (here the front page said '6వ పేజీ' but + # the run only has 4 pages). So treat the marker as "this front-page lead + # continues on a later page" and resolve the ACTUAL target by the repeated- + # headline signal: score every OTHER page's doc_titles against the source + # headline and take the best match, breaking ties toward the OCR'd page N. + existing = sorted( + int(m.group(1)) + for q in pages_dir.glob("page_*.png") + if (m := re.match(r"page_(\d+)\.png$", q.name))) + best = None # (score, page, tregs, tgt_png_str, hl, marker) + for q in existing: + if q == P: + continue + qr = pages_dir / f"page_{q:03d}.regions.json" + qp = pages_dir / f"page_{q:03d}.png" + if not qr.exists() or not qp.exists(): + continue + qregs = json.loads(qr.read_text())["regions"] + hl_q, score_q, marker_q = _best_headline(qregs, Image.open(qp), headline, match_thresh) + if hl_q is None: + continue + if best is None or score_q > best[0] or (score_q == best[0] and q == N): + best = (score_q, q, qregs, str(qp), hl_q, marker_q) + + if best is None: + rec = {"article": info_path.parent.name, "src_page": P, "headline": headline, + "target_page": N, "match_score": 0.0, "taruvaayi_marker": False, + "linked": False, + "reason": "no matching repeated headline on any inside page"} + results.append(rec) + if verbose: + print(f" ✗ {rec['article']} → p{N}: jump-from found but no repeated " + f"headline match on any inside page (NOT linked)") + continue + + score, N, tregs, tgt_png, hl, marker = best + rec = {"article": info_path.parent.name, "src_page": P, "headline": headline, + "target_page": N, "match_score": round(score, 2), + "taruvaayi_marker": marker, "linked": False} + + cont_img, cbox = _continuation_crop(tregs, tgt_png, paper, hl) + if cont_img is None: + rec["reason"] = "no body under repeated headline" + results.append(rec) + continue + out_png = out_dir / f"{info_path.parent.name}__cont_p{N:03d}.png" + size = _stitch(arts_dir / info_path.parent.name / "article.png", + cont_img, out_png) + rec.update({"linked": True, "cont_bbox": cbox, + "stitched": str(out_png), "stitched_size": size}) + results.append(rec) + if verbose: + print(f" ✓ {rec['article']} '{headline[:26]}' → p{N} " + f"(headline {score:.2f}, తరువాయి={marker}) → {out_png.name}") + + (out_dir / "links.json").write_text(json.dumps(results, indent=2, ensure_ascii=False)) + linked = sum(1 for r in results if r["linked"]) + if verbose: + print(f"\n {linked} continuation(s) linked & stitched; report: {out_dir/'links.json'}") + return results + + +if __name__ == "__main__": + import sys + link_continuations(sys.argv[1] if len(sys.argv) > 1 else "output/20260601_234709") diff --git a/scratch_py/_detect_rules.py b/scratch_py/_detect_rules.py new file mode 100644 index 0000000..bf1fccb --- /dev/null +++ b/scratch_py/_detect_rules.py @@ -0,0 +1,74 @@ +"""PROOF-OF-CONCEPT: detect the printed grey separator rules on a newspaper page +using classical CV (no ML, no API). Horizontal + vertical thin straight lines are +extracted by morphological opening with long thin kernels, then overlaid on the +page so we can see whether they recover article boundaries. + +Usage: venv/bin/python3 _detect_rules.py [out_png] +""" +import sys +from pathlib import Path + +import cv2 +import numpy as np + + +def detect_rules(gray, min_frac=0.10, thickness=9): + """Return (h_lines, v_lines) as lists of (x1,y1,x2,y2). + min_frac: a rule must be at least this fraction of page width/height long. + """ + H, W = gray.shape + # The rules are mid-grey on white. Invert + threshold so lines become white. + # Use a gentle adaptive-ish binarisation: anything clearly darker than paper. + blur = cv2.GaussianBlur(gray, (3, 3), 0) + bw = cv2.threshold(blur, 0, 255, cv2.THRESH_BINARY_INV + cv2.THRESH_OTSU)[1] + + def extract(kernel_len, horizontal): + if horizontal: + k = cv2.getStructuringElement(cv2.MORPH_RECT, (kernel_len, 1)) + else: + k = cv2.getStructuringElement(cv2.MORPH_RECT, (1, kernel_len)) + opened = cv2.morphologyEx(bw, cv2.MORPH_OPEN, k, iterations=1) + # thicken slightly so near-collinear fragments merge + opened = cv2.dilate(opened, cv2.getStructuringElement( + cv2.MORPH_RECT, (thickness, thickness))) + cnts, _ = cv2.findContours(opened, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE) + lines = [] + for c in cnts: + x, y, w, h = cv2.boundingRect(c) + if horizontal and w >= min_frac * W and h <= 60: + lines.append((x, y + h // 2, x + w, y + h // 2, w)) + if (not horizontal) and h >= min_frac * H and w <= 60: + lines.append((x + w // 2, y, x + w // 2, y + h, h)) + return lines + + h_lines = extract(int(min_frac * W), horizontal=True) + v_lines = extract(int(min_frac * H), horizontal=False) + return h_lines, v_lines + + +def main(png_path, out_path=None): + png_path = Path(png_path) + out_path = Path(out_path) if out_path else Path(f"/tmp/rules_{png_path.stem}.png") + img = cv2.imread(str(png_path)) + gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY) + H, W = gray.shape + h_lines, v_lines = detect_rules(gray) + + print(f"page {png_path.name} {W}x{H}") + print(f" horizontal rules: {len(h_lines)}") + for x1, y1, x2, y2, ln in sorted(h_lines, key=lambda L: L[1]): + print(f" y={y1:>5} x[{x1:>4}..{x2:>4}] len={ln}") + print(f" vertical rules: {len(v_lines)}") + for x1, y1, x2, y2, ln in sorted(v_lines, key=lambda L: L[0]): + print(f" x={x1:>5} y[{y1:>4}..{y2:>4}] len={ln}") + + for x1, y1, x2, y2, ln in h_lines: + cv2.line(img, (x1, y1), (x2, y2), (0, 0, 255), 6) # red = horizontal + for x1, y1, x2, y2, ln in v_lines: + cv2.line(img, (x1, y1), (x2, y2), (255, 0, 0), 6) # blue = vertical + cv2.imwrite(str(out_path), img) + print(f"\noverlay written: {out_path}") + + +if __name__ == "__main__": + main(sys.argv[1], sys.argv[2] if len(sys.argv) > 2 else None) diff --git a/scratch_py/_diff_snap.py b/scratch_py/_diff_snap.py new file mode 100644 index 0000000..8c10f04 --- /dev/null +++ b/scratch_py/_diff_snap.py @@ -0,0 +1,16 @@ +import json +from pathlib import Path +b=json.loads(Path("scratch_py/_snap_base.json").read_text()) +n=json.loads(Path("scratch_py/_snap_new.json").read_text()) +any_change=False +for pg in ("1","2","3","4"): + bp,np_=b[pg],n[pg] + keys=set(bp)|set(np_) + for k in sorted(keys,key=int): + if bp.get(k)!=np_.get(k): + any_change=True + print(f"PAGE{pg} anchor {k}:") + print(f" base={bp.get(k)}") + print(f" new ={np_.get(k)}") +if not any_change: + print("NO CHANGE across all 4 pages — column-walk is a no-op on this fixture.") diff --git a/scratch_py/_find_banner.py b/scratch_py/_find_banner.py new file mode 100644 index 0000000..fc2f39b --- /dev/null +++ b/scratch_py/_find_banner.py @@ -0,0 +1,14 @@ +import json +from pathlib import Path +import numpy as np +D = Path("output/AndhraJyothi_Siddipet District_20260602_20260603_144919/pages") +for pg in (1,2,3,4): + regs = json.loads((D/f"page_{pg:03d}.regions.json").read_text())["regions"] + texts=[r for r in regs if r.get("type")=="text"] + ws=[r["bbox"][2]-r["bbox"][0] for r in texts] + pitch=int(np.median(ws)) if ws else 0 + docs=[r for r in regs if r.get("type")=="doc_title"] + print(f"== PAGE {pg}: textmedian(pitch)={pitch}, {len(docs)} doc_titles ==") + for r in sorted(docs,key=lambda r:r["bbox"][1]): + b=r["bbox"]; w=b[2]-b[0] + print(f" doc#{r['id']:>3} bbox={b} w={w} (~{w/max(1,pitch):.1f} col)") diff --git a/scratch_py/_ocr_page4.py b/scratch_py/_ocr_page4.py new file mode 100644 index 0000000..dde81d8 --- /dev/null +++ b/scratch_py/_ocr_page4.py @@ -0,0 +1,21 @@ +import paddleocr +from PIL import Image +from pathlib import Path + +# Find the latest page 4 image +img_path = next(Path("output").glob("20260612_003429/pages/page_004.png")) +print(f"Running PaddleOCR on {img_path}") + +ocr = paddleocr.PaddleOCR(lang='te') +result = ocr.predict(str(img_path)) + +print("OCR lines on Page 4 containing 'చెరువు' or 'మంత్రి' or numbers:") +if isinstance(result, list) and len(result) > 0: + det_result = result[0] + # Check if it has boxes and texts + texts = det_result.get('rec_texts', []) + boxes = det_result.get('dt_polys', []) + for i, txt in enumerate(texts): + if any(w in txt for w in ["చెరువు", "మంత్రి", "36", "ఏళ్ల", "ఉత్తమ్", "పల్లా", "సిద్ధపేట", "గూడెం"]): + box = boxes[i].tolist() if hasattr(boxes[i], 'tolist') else boxes[i] + print(f"Text: '{txt}' at box: {box}") diff --git a/scratch_py/_peek.py b/scratch_py/_peek.py new file mode 100644 index 0000000..583e167 --- /dev/null +++ b/scratch_py/_peek.py @@ -0,0 +1,11 @@ +import sys +sys.path.insert(0,".") +from PIL import Image +from pathlib import Path +base=Path("output/AndhraJyothi_Siddipet District_20260602_20260603_144919/pages") +img=Image.open(base/"page_001.png").convert("RGB") +print("page size",img.size) +# bottom region holding H38 + neighbours +crop=img.crop((1550,5300,4135,6800)) +crop.save("scratch_py/_peek_bottom.png") +print("saved scratch_py/_peek_bottom.png", crop.size) diff --git a/scratch_py/_peek2.py b/scratch_py/_peek2.py new file mode 100644 index 0000000..b000dfa --- /dev/null +++ b/scratch_py/_peek2.py @@ -0,0 +1,14 @@ +import sys, json +sys.path.insert(0,".") +from PIL import Image +from pathlib import Path +base=Path("output/AndhraJyothi_Siddipet District_20260602_20260603_144919/pages") +img=Image.open(base/"page_001.png").convert("RGB") +img.crop((2750,4750,4150,6800)).save("scratch_py/_peek_rightcol.png") +# list regions in right column area +regs=json.loads((base/"page_001.regions.json").read_text())["regions"] +print("regions with cx in [3100,4150] and y in [4750,6800]:") +for r in sorted(regs,key=lambda r:r["bbox"][1]): + b=r["bbox"]; cx=(b[0]+b[2])/2 + if 3100<=cx<=4150 and b[1]<6800 and b[3]>4750: + print(f" R{r['id']:>3} {r.get('type'):<14} bbox={b}") diff --git a/scratch_py/_probe_walk.py b/scratch_py/_probe_walk.py new file mode 100644 index 0000000..fd151bc --- /dev/null +++ b/scratch_py/_probe_walk.py @@ -0,0 +1,29 @@ +import sys, json, contextlib, io +sys.path.insert(0,".") +from pathlib import Path +import smart_extractor as se +from _lines import separator_barriers +D=Path("output/AndhraJyothi_Siddipet District_20260602_20260603_144919/pages") +def cx(b): return (b[0]+b[2])/2 +for pg in (1,2,3,4): + png=D/f"page_{pg:03d}.png" + regs0=json.loads((D/f"page_{pg:03d}.regions.json").read_text())["regions"] + with contextlib.redirect_stdout(io.StringIO()): + regs=se.drop_masthead_regions(regs0,"andhra_jyothi",pg,png) + ds=se.find_article_starts_by_dateline(regs,str(png),"andhra_jyothi") + bars=separator_barriers(str(png),regs) + bl=se.cluster_all_article_blocks(regs,dateline_starts=ds,sep_lines=bars) + tws=sorted(r["bbox"][2]-r["bbox"][0] for r in regs if r.get("type")=="text") + pitch=tws[len(tws)//2] if tws else 0 + dl=set(s["region_id"] for s in ds) + docids=set(r["id"] for r in regs if r.get("type")=="doc_title") + hard=dl|docids + vsep=[L for L in bars if L.get("vert")] + print(f"== PAGE{pg} pitch={pitch} vrules={len(vsep)} ==") + for blk in bl: + b=blk["bbox"]; w=b[2]-b[0] + if blk["kind"]=="H" and w>=1.6*pitch: + lo,hi=b[2],b[2]+int(1.4*pitch) + col=[r for r in regs if r.get("type") not in ("header","number") and lo+5<=cx(r["bbox"])<=hi] + hardhit=[r["id"] for r in col if r["id"] in hard] + print(f" banner R{blk['anchor_id']} bbox={b} w={w} (~{w/pitch:.1f}col) rightcol_regs={[r['id'] for r in col]} hard={hardhit}") diff --git a/scratch_py/_regress_final.py b/scratch_py/_regress_final.py new file mode 100644 index 0000000..13f7c12 --- /dev/null +++ b/scratch_py/_regress_final.py @@ -0,0 +1,147 @@ +"""Production-path regression harness for grow_articles_final. + +Unlike _regress_grow.py (which calls _grow_article_downward directly with +dateline_starts=None and so over-estimates wrapping), this mirrors the REAL +pipeline tail: it computes dateline_starts via find_article_starts_by_dateline +and runs grow_articles_final exactly as production does. A diff of two snapshots +therefore isolates what the code change does ON THE SHIPPING PATH. + +Usage: + venv/bin/python3 _regress_final.py snapshot.json # snapshot current code + venv/bin/python3 _regress_final.py --diff a.json b.json # compare two snapshots +""" +import json +import sys +from collections import Counter +from pathlib import Path + +OUT = Path("output") + + +def paper_of_height(h): + if 7040 <= h <= 7060: + return "sakshi" + if h in (6833, 6549): + return "andhra_jyothi" + if h == 6422: + return "namaste_telangana" + return "unknown" + + +def union_bbox(member_ids, region_map): + bs = [region_map[r]["bbox"] for r in member_ids if r in region_map] + if not bs: + return None + return [min(b[0] for b in bs), min(b[1] for b in bs), + max(b[2] for b in bs), max(b[3] for b in bs)] + + +def run(): + import copy + from PIL import Image + from smart_extractor import find_article_starts_by_dateline, grow_articles_final + + runs = sorted([d for d in OUT.iterdir() if d.is_dir()], + key=lambda d: d.stat().st_mtime, reverse=True) + snapshot = {} + papers = Counter() + for run_dir in runs: + pages_dir = run_dir / "pages" + if not pages_dir.exists(): + continue + for reg_path in sorted(pages_dir.glob("page_*.regions.json")): + pol_path = reg_path.with_name(reg_path.name.replace( + ".regions.json", ".political_result.json")) + png = reg_path.with_name(reg_path.name.replace(".regions.json", ".png")) + if not pol_path.exists() or not png.exists(): + continue + regions = json.loads(reg_path.read_text())["regions"] + region_map = {r["id"]: r for r in regions} + political = json.loads(pol_path.read_text()).get("political_articles", []) + if not political: + continue + try: + paper = paper_of_height(Image.open(png).height) + dateline_starts = find_article_starts_by_dateline( + regions, str(png), paper) + grown = grow_articles_final( + copy.deepcopy(political), regions, dateline_starts) + except Exception as e: + print(f" SKIP {run_dir.name}/{reg_path.stem}: {e}") + continue + + for i, art in enumerate(grown): + key = f"{run_dir.name}/{reg_path.stem}/art{i}" + members = sorted(art.get("member_region_ids", []) or []) + snapshot[key] = { + "paper": paper, + "headline": art.get("headline_english", "")[:50], + "members": members, + "bbox": union_bbox(members, region_map), + } + papers[paper] += 1 + print(f" papers: {dict(papers)}") + return snapshot + + +def diff(path_a, path_b): + a = json.loads(Path(path_a).read_text()) + b = json.loads(Path(path_b).read_text()) + keys = sorted(set(a) | set(b)) + + per_total = Counter() + per_changed = Counter() + grew = Counter() + shrank = Counter() + samples = [] + + for k in keys: + ra, rb = a.get(k), b.get(k) + paper = (rb or ra).get("paper", "unknown") + per_total[paper] += 1 + if ra is None or rb is None: + per_changed[paper] += 1 + samples.append((k, paper, "ONLY IN ONE SNAPSHOT", None, None)) + continue + sa, sb = set(ra["members"]), set(rb["members"]) + if sa != sb: + per_changed[paper] += 1 + added = sorted(sb - sa) + removed = sorted(sa - sb) + if added: + grew[paper] += 1 + if removed: + shrank[paper] += 1 + samples.append((k, paper, rb["headline"], added, removed)) + + print("\n=== PRODUCTION-PATH DIFF (A=baseline/OLD -> B=new) ===") + print(f"{'paper':<18}{'articles':>10}{'changed':>9}{'grew':>7}{'shrank':>8}") + for paper in sorted(per_total): + print(f"{paper:<18}{per_total[paper]:>10}{per_changed[paper]:>9}" + f"{grew[paper]:>7}{shrank[paper]:>8}") + tot = sum(per_total.values()) + chg = sum(per_changed.values()) + print(f"{'TOTAL':<18}{tot:>10}{chg:>9}{sum(grew.values()):>7}{sum(shrank.values()):>8}") + + if samples: + print(f"\n--- {len(samples)} changed article(s) ---") + for k, paper, hl, added, removed in samples: + print(f"\n[{paper}] {k}") + print(f" headline: {hl}") + if added: + print(f" + added : {added}") + if removed: + print(f" - removed: {removed}") + else: + print("\nNo differences — member sets identical across all fixtures.") + return chg + + +if __name__ == "__main__": + if len(sys.argv) >= 4 and sys.argv[1] == "--diff": + diff(sys.argv[2], sys.argv[3]) + else: + out_path = sys.argv[1] if len(sys.argv) > 1 else "/tmp/final_snapshot.json" + snap = run() + Path(out_path).write_text(json.dumps(snap, indent=2, ensure_ascii=False)) + print(f"wrote {len(snap)} article snapshots to {out_path}") diff --git a/scratch_py/_regress_grow.py b/scratch_py/_regress_grow.py new file mode 100644 index 0000000..94437d7 --- /dev/null +++ b/scratch_py/_regress_grow.py @@ -0,0 +1,173 @@ +"""Regression harness for _grow_article_downward. + +Runs the grow function over the saved fixtures of EVERY cached run, with IDENTICAL +auxiliary inputs (claimed = union of OTHER articles' members on the page, +dateline_starts = None). Because old and new code receive the same inputs, a diff +of the two output snapshots isolates exactly what the code change does. + +Records per article: the grown member-id set and its union bbox, tagged with the +source newspaper (from meta.json) so results can be broken down per paper. + +Usage: + python3 _regress_grow.py snapshot.json # snapshot current code + python3 _regress_grow.py --diff a.json b.json # compare two snapshots +""" +import json +import re +import sys +from collections import Counter +from pathlib import Path + +OUT = Path("output") + + +def union_bbox(member_ids, region_map): + bs = [region_map[r]["bbox"] for r in member_ids if r in region_map] + if not bs: + return None + return [min(b[0] for b in bs), min(b[1] for b in bs), + max(b[2] for b in bs), max(b[3] for b in bs)] + + +def paper_of(run_dir): + meta = run_dir / "meta.json" + if meta.exists(): + try: + fn = json.loads(meta.read_text()).get("filename", "") + name = re.split(r"[_0-9]", fn)[0].strip() + if name: + return name + except Exception: + pass + # Cached fixtures carry no meta.json; identify the paper by page-1 image height, + # which is distinct per edition (verified against the printed masthead nameplate). + png = run_dir / "pages" / "page_001.png" + if png.exists(): + try: + from PIL import Image + h = Image.open(png).height + if 7040 <= h <= 7060: + return "Sakshi" + if h in (6833, 6549): + return "Andhra Jyothi" + if h == 6422: + return "Namaste Telangana" + except Exception: + pass + return "unknown" + + +def run(): + from smart_extractor import _grow_article_downward + + runs = sorted([d for d in OUT.iterdir() if d.is_dir()], + key=lambda d: d.stat().st_mtime, reverse=True) + snapshot = {} + papers = Counter() + for run_dir in runs: + pages_dir = run_dir / "pages" + if not pages_dir.exists(): + continue + paper = paper_of(run_dir) + for reg_path in sorted(pages_dir.glob("page_*.regions.json")): + pol_path = reg_path.with_name(reg_path.name.replace(".regions.json", + ".political_result.json")) + if not pol_path.exists(): + continue + regions = json.loads(reg_path.read_text())["regions"] + region_map = {r["id"]: r for r in regions} + political = json.loads(pol_path.read_text()).get("political_articles", []) + + # claimed = every member of every article (mirrors the real caller's + # global claimed set built before growth). + all_members = set() + for a in political: + all_members.update(a.get("member_region_ids", [])) + + for i, art in enumerate(political): + members = art.get("member_region_ids", []) or [] + if not members: + continue + # claimed for THIS article = everything claimed by OTHER articles. + claimed = set(all_members) - set(members) + grown, added = _grow_article_downward( + members, region_map, regions, claimed, dateline_starts=None + ) + key = f"{run_dir.name}/{reg_path.stem}/art{i}" + snapshot[key] = { + "paper": paper, + "headline": art.get("headline_english", "")[:50], + "in_members": sorted(members), + "grown_members": sorted(grown), + "added": sorted(added), + "in_bbox": union_bbox(members, region_map), + "grown_bbox": union_bbox(grown, region_map), + } + papers[paper] += 1 + print(f" papers: {dict(papers)}") + return snapshot + + +def diff(path_a, path_b): + """Compare snapshot A (baseline/OLD) against B (NEW). Reports per-paper how many + articles changed their grown member set, plus a sample of the changes.""" + a = json.loads(Path(path_a).read_text()) + b = json.loads(Path(path_b).read_text()) + keys = sorted(set(a) | set(b)) + + per_paper_total = Counter() + per_paper_changed = Counter() + grew = Counter() # NEW added regions OLD didn't + shrank = Counter() # NEW dropped regions OLD had + samples = [] + + for k in keys: + ra, rb = a.get(k), b.get(k) + paper = (rb or ra).get("paper", "unknown") + per_paper_total[paper] += 1 + if ra is None or rb is None: + per_paper_changed[paper] += 1 + samples.append((k, paper, "ONLY IN ONE SNAPSHOT", None, None)) + continue + sa, sb = set(ra["grown_members"]), set(rb["grown_members"]) + if sa != sb: + per_paper_changed[paper] += 1 + added = sorted(sb - sa) + removed = sorted(sa - sb) + if added: + grew[paper] += 1 + if removed: + shrank[paper] += 1 + samples.append((k, paper, rb["headline"], added, removed)) + + print("\n=== REGRESSION DIFF (A=baseline/OLD → B=new) ===") + print(f"{'paper':<18}{'articles':>10}{'changed':>9}{'grew':>7}{'shrank':>8}") + for paper in sorted(per_paper_total): + print(f"{paper:<18}{per_paper_total[paper]:>10}{per_paper_changed[paper]:>9}" + f"{grew[paper]:>7}{shrank[paper]:>8}") + tot = sum(per_paper_total.values()) + chg = sum(per_paper_changed.values()) + print(f"{'TOTAL':<18}{tot:>10}{chg:>9}{sum(grew.values()):>7}{sum(shrank.values()):>8}") + + if samples: + print(f"\n--- {len(samples)} changed article(s) (showing up to 40) ---") + for k, paper, hl, added, removed in samples[:40]: + print(f"\n[{paper}] {k}") + print(f" headline: {hl}") + if added: + print(f" + added : {added}") + if removed: + print(f" - removed: {removed}") + else: + print("\nNo differences — grown member sets are identical across all fixtures.") + return chg + + +if __name__ == "__main__": + if len(sys.argv) >= 4 and sys.argv[1] == "--diff": + diff(sys.argv[2], sys.argv[3]) + else: + out_path = sys.argv[1] if len(sys.argv) > 1 else "/tmp/grow_snapshot.json" + snap = run() + Path(out_path).write_text(json.dumps(snap, indent=2, ensure_ascii=False)) + print(f"wrote {len(snap)} article snapshots to {out_path}") diff --git a/scratch_py/_regress_masthead.py b/scratch_py/_regress_masthead.py new file mode 100644 index 0000000..3313aa9 --- /dev/null +++ b/scratch_py/_regress_masthead.py @@ -0,0 +1,139 @@ +"""Regression harness for drop_masthead_regions. + +drop_masthead_regions runs inside main() BEFORE political_result.json is saved, so +the saved fixtures never capture its effect. This harness replays it directly over +every cached page's regions.json, mirroring the real call +(drop_masthead_regions(regions, paper, page_num, page_img_path)). + +The cached fixtures carry no meta.json, so the paper is recovered from the page-1 +image height (verified against the printed masthead nameplate): + 7053/7055 -> sakshi 6833/6549 -> andhra_jyothi 6422 -> namaste_telangana + +Two views: + • SHIPPED (gated): what the function actually drops in production. Expect the + Namaste Telangana page-1 masthead to be dropped (incl. a doc_title nameplate, + the mis-group seed) and NOTHING dropped on inner pages or other papers. + • UNGATED PROBE: the same geometric rule with the paper/page gate removed, run on + every page-1. Confirms the gate is the only thing keeping the rule off other + papers AND that the rule would never strip a genuine page-1 headline. +""" +import glob +import json +import sys +from collections import Counter +from pathlib import Path + +from PIL import Image + +import smart_extractor +from smart_extractor import drop_masthead_regions + +OUT = Path("output") + + +def paper_of(run_dir): + png = run_dir / "pages" / "page_001.png" + if png.exists(): + try: + h = Image.open(png).height + if 7040 <= h <= 7060: + return "sakshi" + if h in (6833, 6549): + return "andhra_jyothi" + if h == 6422: + return "namaste_telangana" + except Exception: + pass + return "unknown" + + +def page_num_of(reg_path): + # page_001.regions.json -> 1 + stem = reg_path.name.split(".")[0] # page_001 + return int(stem.split("_")[1]) + + +def types_of(regions, ids): + m = {r["id"]: r.get("type") for r in regions} + return Counter(m.get(i) for i in ids) + + +def run(): + dirs = sorted([d for d in OUT.iterdir() if d.is_dir()]) + shipped = [] # rows: (paper, page, run, dropped_ids, dropped_types) + probe = [] # ungated, page-1 only + fired = Counter() # paper -> #page-1 runs where shipped drop fired + p1_total = Counter() # paper -> #page-1 runs + false_pos = [] # shipped drops on non-NT or page>1 (should be empty) + + for d in dirs: + pages_dir = d / "pages" + if not pages_dir.exists(): + continue + paper = paper_of(d) + for reg_path in sorted(pages_dir.glob("page_*.regions.json")): + page_num = page_num_of(reg_path) + png = pages_dir / reg_path.name.replace(".regions.json", ".png") + regions = json.loads(reg_path.read_text())["regions"] + + # ---- SHIPPED (gated) ---- + kept = drop_masthead_regions(list(regions), paper, page_num, str(png)) + kept_ids = {r["id"] for r in kept} + dropped = sorted(r["id"] for r in regions if r["id"] not in kept_ids) + if dropped: + shipped.append((paper, page_num, d.name, dropped, types_of(regions, dropped))) + if not (paper == "namaste_telangana" and page_num == 1): + false_pos.append((paper, page_num, d.name, dropped)) + + if page_num == 1: + p1_total[paper] += 1 + if paper == "namaste_telangana" and dropped: + fired[paper] += 1 + + # ---- UNGATED PROBE (force the rule to run) ---- + probe_kept = drop_masthead_regions(list(regions), "namaste_telangana", 1, str(png)) + probe_kept_ids = {r["id"] for r in probe_kept} + pdropped = sorted(r["id"] for r in regions if r["id"] not in probe_kept_ids) + if pdropped: + probe.append((paper, d.name, pdropped, types_of(regions, pdropped))) + + return shipped, probe, fired, p1_total, false_pos + + +def report(shipped, probe, fired, p1_total, false_pos): + print("\n=== SHIPPED (gated) — Namaste Telangana page-1 masthead drop ===") + nt = [s for s in shipped if s[0] == "namaste_telangana" and s[1] == 1] + print(f"NT page-1 runs: {p1_total['namaste_telangana']} | masthead fired in: {fired['namaste_telangana']}") + seedless = [] + for paper, page, run, dropped, types in sorted(nt, key=lambda x: x[2]): + has_title = types.get("doc_title", 0) > 0 + flag = "" if has_title else " ⚠️ no doc_title dropped (seed not removed?)" + if not has_title: + seedless.append(run) + print(f" {run}: dropped {dropped} types={dict(types)}{flag}") + + misses = p1_total['namaste_telangana'] - fired['namaste_telangana'] + print(f"\n NT page-1 misses (nothing dropped): {misses}") + print(f" NT page-1 drops WITHOUT a doc_title nameplate seed: {len(seedless)} {seedless}") + + print("\n=== FALSE POSITIVES (shipped drops on non-NT, or page>1) ===") + if false_pos: + for paper, page, run, dropped in false_pos: + print(f" ⚠️ {paper} p{page} {run}: dropped {dropped}") + else: + print(" none — gate held: 0 drops on inner pages and on Sakshi/Andhra Jyothi.") + + print("\n=== UNGATED PROBE — would the bare geometric rule strip a real headline? ===") + by_paper = Counter(p[0] for p in probe) + print(f"page-1 runs where the ungated rule drops something, by paper: {dict(by_paper)}") + for paper in ("sakshi", "andhra_jyothi"): + hits = [p for p in probe if p[0] == paper] + print(f"\n --- {paper}: {len(hits)} page-1 run(s) would drop something ungated ---") + for _, run, dropped, types in hits[:12]: + print(f" {run}: {dropped} types={dict(types)}") + if not hits: + print(" none — rule is inert on this paper's page-1 even ungated (safe).") + + +if __name__ == "__main__": + report(*run()) diff --git a/scratch_py/_replay_crop.py b/scratch_py/_replay_crop.py new file mode 100644 index 0000000..47bc4dc --- /dev/null +++ b/scratch_py/_replay_crop.py @@ -0,0 +1,70 @@ +"""Replay the post-Claude pipeline tail on a saved run's fixtures, WITHOUT any +billable Claude calls. Loads regions.json + political_result.json (already +post-safeguard), then applies ONLY the new steps that change the crop: + - grow_articles_final (Fix #1 floor threshold + Fix #3 final grow pass) + - crop_political_articles (Fix #2 column-aware foreign-headline clip) +and writes regenerated crops to a /tmp dir for visual inspection. + +Usage: python3 _replay_crop.py +""" +import json +import sys +from pathlib import Path + +import smart_extractor +from smart_extractor import ( + find_article_starts_by_dateline, + grow_articles_final, + crop_political_articles, +) + + +def paper_of_height(h): + if 7040 <= h <= 7060: + return "sakshi" + if h in (6833, 6549): + return "andhra_jyothi" + if h == 6422: + return "namaste_telangana" + return "unknown" + + +def main(run_dir, page_num): + run = Path(run_dir) + pages = run / "pages" + png = pages / f"page_{page_num:03d}.png" + reg_path = pages / f"page_{page_num:03d}.regions.json" + pol_path = pages / f"page_{page_num:03d}.political_result.json" + + regions = json.loads(reg_path.read_text())["regions"] + political = json.loads(pol_path.read_text())["political_articles"] + + from PIL import Image + paper = paper_of_height(Image.open(png).height) + print(f"paper={paper} regions={len(regions)} articles={len(political)}") + + dateline_starts = find_article_starts_by_dateline(regions, str(png), paper) + + print("\n--- BEFORE grow_articles_final (saved member sets) ---") + for i, a in enumerate(political, 1): + m = sorted(a.get("member_region_ids", []) or []) + print(f" a{i:03d}: {a.get('headline_english','?')[:38]!r:42} members={m}") + + political = grow_articles_final(political, regions, dateline_starts) + + print("\n--- AFTER grow_articles_final ---") + for i, a in enumerate(political, 1): + m = sorted(a.get("member_region_ids", []) or []) + print(f" a{i:03d}: {a.get('headline_english','?')[:38]!r:42} members={m}") + + out = Path(f"/tmp/replay_{run.name}_p{page_num:03d}") + out.mkdir(parents=True, exist_ok=True) + print(f"\n--- CROP (writing to {out}) ---") + crop_political_articles(str(png), political, regions, out, page_num) + print(f"\nCrops written to: {out}") + for f in sorted(out.glob("*.png")): + print(f" {f.name}") + + +if __name__ == "__main__": + main(sys.argv[1], int(sys.argv[2])) diff --git a/scratch_py/_replay_pg1_all.py b/scratch_py/_replay_pg1_all.py new file mode 100644 index 0000000..9f83196 --- /dev/null +++ b/scratch_py/_replay_pg1_all.py @@ -0,0 +1,19 @@ +import sys, json, shutil, contextlib, io +sys.path.insert(0,".") +from pathlib import Path +import smart_extractor as se +from _lines import separator_barriers +base=Path("output/AndhraJyothi_Siddipet District_20260602_20260603_144919") +D=base/"pages"; out=base/"pg1_all_current" +if out.exists(): shutil.rmtree(out) +png=D/"page_001.png" +regs=json.loads((D/"page_001.regions.json").read_text())["regions"] +with contextlib.redirect_stdout(io.StringIO()): + regs=se.drop_masthead_regions(regs,"andhra_jyothi",1,png) + ds=se.find_article_starts_by_dateline(regs,str(png),"andhra_jyothi") + bars=separator_barriers(str(png),regs) + bl=se.cluster_all_article_blocks(regs,dateline_starts=ds,sep_lines=bars) + n=se.crop_all_article_blocks(str(png),regs,str(out),1,dateline_starts=ds,sep_lines=bars) +for b in sorted(bl,key=lambda d:(d["bbox"][1],d["bbox"][0])): + print(f"{b['kind']:>2} anchor={b['anchor_id']:>3} bbox={b['bbox']} w={b['bbox'][2]-b['bbox'][0]}") +print(f"\n{n} crops -> {out}/page_001/") diff --git a/scratch_py/_replay_pg2_all.py b/scratch_py/_replay_pg2_all.py new file mode 100644 index 0000000..efd6304 --- /dev/null +++ b/scratch_py/_replay_pg2_all.py @@ -0,0 +1,19 @@ +import sys, json, shutil, contextlib, io +sys.path.insert(0,".") +from pathlib import Path +import smart_extractor as se +from _lines import separator_barriers +base=Path("output/AndhraJyothi_Siddipet District_20260602_20260603_144919") +D=base/"pages"; out=base/"pg2_all_current" +if out.exists(): shutil.rmtree(out) +png=D/"page_002.png" +regs=json.loads((D/"page_002.regions.json").read_text())["regions"] +with contextlib.redirect_stdout(io.StringIO()): + regs=se.drop_masthead_regions(regs,"andhra_jyothi",2,png) + ds=se.find_article_starts_by_dateline(regs,str(png),"andhra_jyothi") + bars=separator_barriers(str(png),regs) + bl=se.cluster_all_article_blocks(regs,dateline_starts=ds,sep_lines=bars) + n=se.crop_all_article_blocks(str(png),regs,str(out),2,dateline_starts=ds,sep_lines=bars) +for b in sorted(bl,key=lambda d:(d["bbox"][1],d["bbox"][0])): + print(f"{b['kind']:>2} anchor={b['anchor_id']:>3} bbox={b['bbox']} w={b['bbox'][2]-b['bbox'][0]}") +print(f"\n{n} crops -> {out}/page_002/") diff --git a/scratch_py/_replay_pg3_all.py b/scratch_py/_replay_pg3_all.py new file mode 100644 index 0000000..127826b --- /dev/null +++ b/scratch_py/_replay_pg3_all.py @@ -0,0 +1,19 @@ +import sys, json, shutil, contextlib, io +sys.path.insert(0,".") +from pathlib import Path +import smart_extractor as se +from _lines import separator_barriers +base=Path("output/AndhraJyothi_Siddipet District_20260602_20260603_144919") +D=base/"pages"; out=base/"pg3_all_current" +if out.exists(): shutil.rmtree(out) +png=D/"page_003.png" +regs=json.loads((D/"page_003.regions.json").read_text())["regions"] +with contextlib.redirect_stdout(io.StringIO()): + regs=se.drop_masthead_regions(regs,"andhra_jyothi",3,png) + ds=se.find_article_starts_by_dateline(regs,str(png),"andhra_jyothi") + bars=separator_barriers(str(png),regs) + bl=se.cluster_all_article_blocks(regs,dateline_starts=ds,sep_lines=bars) + n=se.crop_all_article_blocks(str(png),regs,str(out),3,dateline_starts=ds,sep_lines=bars) +for b in sorted(bl,key=lambda d:(d["bbox"][1],d["bbox"][0])): + print(f"{b['kind']:>2} anchor={b['anchor_id']:>3} bbox={b['bbox']} w={b['bbox'][2]-b['bbox'][0]}") +print(f"\n{n} crops -> {out}/page_003/") diff --git a/scratch_py/_replay_pg4_all.py b/scratch_py/_replay_pg4_all.py new file mode 100644 index 0000000..cd5eaff --- /dev/null +++ b/scratch_py/_replay_pg4_all.py @@ -0,0 +1,19 @@ +import sys, json, shutil, contextlib, io +sys.path.insert(0,".") +from pathlib import Path +import smart_extractor as se +from _lines import separator_barriers +base=Path("output/AndhraJyothi_Siddipet District_20260602_20260603_144919") +D=base/"pages"; out=base/"pg4_all_current" +if out.exists(): shutil.rmtree(out) +png=D/"page_004.png" +regs=json.loads((D/"page_004.regions.json").read_text())["regions"] +with contextlib.redirect_stdout(io.StringIO()): + regs=se.drop_masthead_regions(regs,"andhra_jyothi",4,png) + ds=se.find_article_starts_by_dateline(regs,str(png),"andhra_jyothi") + bars=separator_barriers(str(png),regs) + bl=se.cluster_all_article_blocks(regs,dateline_starts=ds,sep_lines=bars) + n=se.crop_all_article_blocks(str(png),regs,str(out),4,dateline_starts=ds,sep_lines=bars) +for b in sorted(bl,key=lambda d:(d["bbox"][1],d["bbox"][0])): + print(f"{b['kind']:>2} anchor={b['anchor_id']:>3} bbox={b['bbox']} w={b['bbox'][2]-b['bbox'][0]}") +print(f"\n{n} crops -> {out}/page_004/") diff --git a/scratch_py/_replay_regions.py b/scratch_py/_replay_regions.py new file mode 100644 index 0000000..287aaf2 --- /dev/null +++ b/scratch_py/_replay_regions.py @@ -0,0 +1,61 @@ +"""Re-run ONLY PaddleOCR layout detection (no Claude calls) on a saved run's +page PNG, at the default 0.5 cutoff and at a lowered threshold, and write the +all-regions debug image for each so the effect of the threshold is visible. + +Usage: venv/bin/python3 _replay_regions.py [threshold] + (threshold defaults to 0.3) +""" +import sys +from collections import Counter +from pathlib import Path + +from smart_extractor import detect_regions, draw_all_regions_debug + + +def summarize(regions): + c = Counter(r.get("type", "unknown") for r in regions) + return ", ".join(f"{k}={v}" for k, v in sorted(c.items(), key=lambda kv: -kv[1])) + + +def main(run_dir, page_num, threshold=0.3): + run = Path(run_dir) + png = run / "pages" / f"page_{page_num:03d}.png" + out = Path(f"/tmp/replay_regions_{run.name}_p{page_num:03d}") + out.mkdir(parents=True, exist_ok=True) + + print(f"=== page {png} -> {out} ===\n") + + # BASELINE: stock 0.5 cutoff + print("--- detection @ default (0.5) ---") + base = detect_regions(str(png), threshold=0.5) + print(f" {len(base)} regions: {summarize(base)}") + draw_all_regions_debug(str(png), base, out, page_num) + Path(out / f"page_{page_num:03d}_allregions_debug.png").rename( + out / f"page_{page_num:03d}_thr050.png") + + # LOWERED: threshold under test + print(f"\n--- detection @ {threshold} ---") + low = detect_regions(str(png), threshold=threshold) + print(f" {len(low)} regions: {summarize(low)}") + draw_all_regions_debug(str(png), low, out, page_num) + Path(out / f"page_{page_num:03d}_allregions_debug.png").rename( + out / f"page_{page_num:03d}_thr{int(threshold*100):03d}.png") + + # DELTA: new regions gained by lowering the cutoff + print(f"\n--- delta: {len(low) - len(base)} more region(s) at {threshold} ---") + base_boxes = {tuple(r["bbox"]) for r in base} + gained = [r for r in low if tuple(r["bbox"]) not in base_boxes] + gained.sort(key=lambda r: (r["bbox"][1], r["bbox"][0])) + for r in gained: + b = r["bbox"] + print(f" + R{r['id']:>3} {r.get('type','?'):>16} conf={r.get('confidence',0):.2f} " + f"bbox={b} ({b[2]-b[0]}x{b[3]-b[1]})") + + print(f"\nImages written to: {out}") + for f in sorted(out.glob("*.png")): + print(f" {f.name}") + + +if __name__ == "__main__": + thr = float(sys.argv[3]) if len(sys.argv) > 3 else 0.3 + main(sys.argv[1], int(sys.argv[2]), thr) diff --git a/scratch_py/_search_pdf.py b/scratch_py/_search_pdf.py new file mode 100644 index 0000000..145a5c1 --- /dev/null +++ b/scratch_py/_search_pdf.py @@ -0,0 +1,13 @@ +import fitz +import sys +from pathlib import Path + +# Find the PDF in the processed directory +pdf_path = next(Path("processed").glob("Jyothi_SiddipetDistrict_17-05-2026*.pdf")) +doc = fitz.open(str(pdf_path)) +if len(doc) >= 4: + page = doc[3] # 0-indexed page 4 is index 3 + print("PAGE 4 TEXT:") + print(page.get_text()) +else: + print(f"PDF only has {len(doc)} pages") diff --git a/scratch_py/_snap_base.json b/scratch_py/_snap_base.json new file mode 100644 index 0000000..cd9070f --- /dev/null +++ b/scratch_py/_snap_base.json @@ -0,0 +1,683 @@ +{ +"1": { +"32": [ +"H", +[ +1, +4, +6, +9, +20, +23, +31, +32, +36, +42, +45, +46, +56, +64 +] +], +"61": [ +"H", +[ +13, +25, +26, +28, +39, +47, +61 +] +], +"55": [ +"P", +[ +22, +35, +55, +65 +] +], +"5": [ +"DL", +[ +5, +18, +53 +] +], +"59": [ +"H", +[ +2, +59 +] +], +"60": [ +"H", +[ +15, +60, +66 +] +], +"3": [ +"DL", +[ +3, +21, +75 +] +], +"16": [ +"O", +[ +16 +] +], +"12": [ +"O", +[ +12, +33, +34, +58 +] +], +"76": [ +"O", +[ +76 +] +], +"68": [ +"O", +[ +68 +] +], +"48": [ +"H", +[ +14, +17, +19, +29, +44, +48, +54, +62, +67, +71 +] +], +"50": [ +"H", +[ +8, +11, +27, +30, +43, +49, +50, +57, +63, +73 +] +], +"38": [ +"H", +[ +7, +10, +37, +38, +41, +51, +52, +69, +70, +74 +] +] +}, +"2": { +"68": [ +"H", +[ +7, +27, +34, +36, +47, +65, +68, +69, +75, +76 +] +], +"4": [ +"DL", +[ +4, +63 +] +], +"61": [ +"P", +[ +32, +61 +] +], +"64": [ +"P", +[ +6, +41, +62, +64 +] +], +"10": [ +"DL", +[ +10, +43 +] +], +"56": [ +"H", +[ +1, +3, +5, +14, +56, +58, +79, +83, +89 +] +], +"87": [ +"H", +[ +2, +22, +40, +42, +87, +93, +94 +] +], +"28": [ +"DL", +[ +12, +28, +38, +85, +92, +96 +] +], +"71": [ +"O", +[ +71, +77, +80, +88 +] +], +"39": [ +"O", +[ +39, +72 +] +], +"50": [ +"H", +[ +8, +9, +16, +50, +73 +] +], +"18": [ +"DL", +[ +11, +18, +24, +26, +46, +51, +66, +70 +] +], +"19": [ +"DL", +[ +19 +] +], +"78": [ +"H", +[ +17, +35, +78 +] +], +"13": [ +"DL", +[ +13, +55 +] +], +"86": [ +"H", +[ +23, +25, +29, +57, +86 +] +], +"20": [ +"DL", +[ +20, +59 +] +], +"49": [ +"H", +[ +30, +37, +44, +49, +60, +74, +91 +] +], +"53": [ +"H", +[ +15, +31, +33, +53, +81, +84, +90 +] +], +"54": [ +"H", +[ +21, +54, +82 +] +] +}, +"3": { +"48": [ +"H", +[ +18, +22, +27, +36, +48, +49, +60, +72 +] +], +"63": [ +"H", +[ +6, +31, +63, +71, +73 +] +], +"32": [ +"O", +[ +32 +] +], +"56": [ +"P", +[ +20, +28, +41, +56, +66 +] +], +"75": [ +"H", +[ +2, +14, +47, +69, +75 +] +], +"15": [ +"DL", +[ +15, +23, +51 +] +], +"53": [ +"P", +[ +9, +11, +24, +53 +] +], +"19": [ +"O", +[ +19, +29, +61 +] +], +"26": [ +"O", +[ +26, +64 +] +], +"33": [ +"H", +[ +1, +3, +12, +21, +33, +34, +43, +46, +62 +] +], +"59": [ +"H", +[ +4, +17, +45, +59 +] +], +"42": [ +"H", +[ +13, +25, +30, +42, +54, +55, +57 +] +], +"35": [ +"H", +[ +7, +10, +16, +35, +40, +44, +58, +67, +70, +76, +78, +79 +] +], +"38": [ +"H", +[ +5, +38 +] +], +"50": [ +"H", +[ +8, +50, +68, +77 +] +] +}, +"4": { +"48": [ +"H", +[ +2, +3, +9, +10, +24, +25, +27, +32, +33, +34, +45, +48, +49, +59, +71, +87 +] +], +"92": [ +"P", +[ +12, +39, +64, +83, +92 +] +], +"97": [ +"H", +[ +11, +13, +14, +16, +46, +51, +52, +68, +69, +76, +77, +97, +107 +] +], +"26": [ +"O", +[ +26, +66 +] +], +"96": [ +"H", +[ +5, +38, +41, +42, +65, +86, +96 +] +], +"91": [ +"H", +[ +1, +6, +29, +31, +36, +37, +60, +73, +78, +91 +] +], +"89": [ +"H", +[ +17, +35, +40, +47, +56, +80, +89, +102 +] +], +"62": [ +"H", +[ +7, +8, +54, +57, +62, +75, +82, +94 +] +], +"85": [ +"H", +[ +4, +23, +67, +74, +85 +] +], +"93": [ +"H", +[ +18, +93 +] +], +"22": [ +"O", +[ +22 +] +], +"21": [ +"DL", +[ +21, +63 +] +], +"19": [ +"DL", +[ +19, +30, +50, +58, +88 +] +], +"103": [ +"H", +[ +28, +43, +101, +103 +] +], +"44": [ +"O", +[ +44 +] +], +"81": [ +"H", +[ +15, +81 +] +], +"72": [ +"H", +[ +20, +72 +] +], +"98": [ +"O", +[ +98 +] +], +"90": [ +"O", +[ +90 +] +] +} +} \ No newline at end of file diff --git a/scratch_py/_snap_new.json b/scratch_py/_snap_new.json new file mode 100644 index 0000000..c8f4c15 --- /dev/null +++ b/scratch_py/_snap_new.json @@ -0,0 +1,683 @@ +{ +"1": { +"61": [ +"H", +[ +13, +23, +25, +26, +28, +39, +47, +61, +64 +] +], +"55": [ +"P", +[ +22, +35, +55, +65 +] +], +"5": [ +"DL", +[ +5, +18, +53 +] +], +"32": [ +"H", +[ +1, +4, +6, +9, +20, +31, +32, +36, +42, +45, +46, +56 +] +], +"59": [ +"H", +[ +2, +59 +] +], +"60": [ +"H", +[ +15, +60, +66 +] +], +"3": [ +"DL", +[ +3, +21, +75 +] +], +"16": [ +"O", +[ +16 +] +], +"12": [ +"O", +[ +12, +33, +34, +58 +] +], +"76": [ +"O", +[ +76 +] +], +"68": [ +"O", +[ +68 +] +], +"48": [ +"H", +[ +17, +48 +] +], +"38": [ +"H", +[ +7, +10, +14, +19, +29, +37, +38, +41, +44, +51, +52, +54, +62, +67, +69, +70, +71, +74 +] +], +"50": [ +"H", +[ +8, +11, +27, +30, +43, +49, +50, +57, +63, +73 +] +] +}, +"2": { +"68": [ +"H", +[ +7, +27, +34, +36, +47, +65, +68, +69, +75, +76 +] +], +"4": [ +"DL", +[ +4, +63 +] +], +"61": [ +"P", +[ +32, +61 +] +], +"64": [ +"P", +[ +6, +41, +62, +64 +] +], +"10": [ +"DL", +[ +10, +43 +] +], +"56": [ +"H", +[ +1, +3, +5, +14, +56, +58, +79, +83, +89 +] +], +"87": [ +"H", +[ +2, +22, +40, +42, +87, +93, +94 +] +], +"28": [ +"DL", +[ +12, +28, +38, +85, +92, +96 +] +], +"71": [ +"O", +[ +71, +77, +80, +88 +] +], +"39": [ +"O", +[ +39, +72 +] +], +"50": [ +"H", +[ +8, +9, +16, +50, +73 +] +], +"18": [ +"DL", +[ +11, +18, +24, +26, +46, +51, +66, +70 +] +], +"19": [ +"DL", +[ +19 +] +], +"78": [ +"H", +[ +17, +35, +78 +] +], +"13": [ +"DL", +[ +13, +55 +] +], +"86": [ +"H", +[ +23, +25, +29, +57, +86 +] +], +"20": [ +"DL", +[ +20, +59 +] +], +"49": [ +"H", +[ +30, +37, +44, +49, +60, +74, +91 +] +], +"53": [ +"H", +[ +15, +31, +33, +53, +81, +84, +90 +] +], +"54": [ +"H", +[ +21, +54, +82 +] +] +}, +"3": { +"48": [ +"H", +[ +18, +22, +27, +36, +48, +49, +60, +72 +] +], +"63": [ +"H", +[ +6, +31, +63, +71, +73 +] +], +"32": [ +"O", +[ +32 +] +], +"56": [ +"P", +[ +20, +28, +41, +56, +66 +] +], +"75": [ +"H", +[ +2, +14, +47, +69, +75 +] +], +"15": [ +"DL", +[ +15, +23, +51 +] +], +"53": [ +"P", +[ +9, +11, +24, +53 +] +], +"19": [ +"O", +[ +19, +29, +61 +] +], +"26": [ +"O", +[ +26, +64 +] +], +"33": [ +"H", +[ +1, +3, +12, +21, +33, +34, +43, +46, +62 +] +], +"59": [ +"H", +[ +4, +17, +45, +59 +] +], +"42": [ +"H", +[ +13, +25, +30, +42, +54, +55, +57 +] +], +"35": [ +"H", +[ +7, +10, +16, +35, +40, +44, +58, +67, +70, +76, +78, +79 +] +], +"38": [ +"H", +[ +5, +38 +] +], +"50": [ +"H", +[ +8, +50, +68, +77 +] +] +}, +"4": { +"48": [ +"H", +[ +2, +3, +9, +10, +24, +25, +27, +32, +33, +34, +45, +48, +49, +59, +71, +87 +] +], +"92": [ +"P", +[ +12, +39, +64, +83, +92 +] +], +"97": [ +"H", +[ +11, +13, +14, +16, +46, +51, +52, +68, +69, +76, +77, +97, +107 +] +], +"26": [ +"O", +[ +26, +66 +] +], +"96": [ +"H", +[ +5, +38, +41, +42, +65, +86, +96 +] +], +"91": [ +"H", +[ +1, +6, +29, +31, +36, +37, +60, +73, +78, +91 +] +], +"89": [ +"H", +[ +17, +35, +40, +47, +56, +80, +89, +102 +] +], +"62": [ +"H", +[ +7, +8, +54, +57, +62, +75, +82, +94 +] +], +"85": [ +"H", +[ +4, +23, +67, +74, +85 +] +], +"93": [ +"H", +[ +18, +93 +] +], +"22": [ +"O", +[ +22 +] +], +"21": [ +"DL", +[ +21, +63 +] +], +"19": [ +"DL", +[ +19, +30, +50, +58, +88 +] +], +"103": [ +"H", +[ +28, +43, +101, +103 +] +], +"44": [ +"O", +[ +44 +] +], +"81": [ +"H", +[ +15, +81 +] +], +"72": [ +"H", +[ +20, +72 +] +], +"98": [ +"O", +[ +98 +] +], +"90": [ +"O", +[ +90 +] +] +} +} \ No newline at end of file diff --git a/scratch_py/_synth_walk.py b/scratch_py/_synth_walk.py new file mode 100644 index 0000000..d351ccd --- /dev/null +++ b/scratch_py/_synth_walk.py @@ -0,0 +1,18 @@ +import sys +sys.path.insert(0,".") +import smart_extractor as se +# pitch ~ 700. Banner spans cols 1-2 (x 0..1500). Article band y 0..2000. +R=[ + {"id":1,"type":"doc_title","bbox":[0,0,1500,150],"confidence":0.9}, # banner (cols1-2) + {"id":2,"type":"text","bbox":[0,200,700,2000],"confidence":0.9}, # col1 body (has dateline) + {"id":3,"type":"text","bbox":[760,200,1460,2000],"confidence":0.9}, # col2 body + {"id":4,"type":"text","bbox":[1520,200,2220,2000],"confidence":0.9}, # col3 CONTINUATION (no dateline) + {"id":5,"type":"paragraph_title","bbox":[1520,200,2120,260],"confidence":0.9}, # col3 subhead +] +# dateline only in col1 body (region 2) +ds=[{"region_id":2,"bbox":[0,200,700,260],"headline_region":1}] +bl=se.cluster_all_article_blocks(R,dateline_starts=ds,sep_lines=[]) +for b in bl: + print(b["kind"],"anchor",b["anchor_id"],"members",sorted(b["members"]),"bbox",b["bbox"]) +assert any(b["anchor_id"]==1 and 4 in b["members"] and 5 in b["members"] for b in bl), "col3 NOT absorbed" +print("\nPASS: col3 (regions 4,5) absorbed into banner block anchor=1") diff --git a/scratch_py/_test_boundaries.py b/scratch_py/_test_boundaries.py new file mode 100644 index 0000000..b0f66db --- /dev/null +++ b/scratch_py/_test_boundaries.py @@ -0,0 +1,87 @@ +"""Single-page boundary test: datelines + rules -> article boxes, overlaid. +Usage: venv/bin/python3 _test_boundaries.py +""" +import json +import sys +from pathlib import Path + +import cv2 +from PIL import Image + +from _lines import detect_separator_lines +from _boundaries import construct_boundaries +from smart_extractor import find_article_starts_by_dateline + + +def paper_of_height(h): + if 7040 <= h <= 7060: + return "sakshi" + if h in (6833, 6549): + return "andhra_jyothi" + if h == 6422: + return "namaste_telangana" + return "unknown" + + +def iou(a, b): + ix = max(0, min(a[2], b[2]) - max(a[0], b[0])) + iy = max(0, min(a[3], b[3]) - max(a[1], b[1])) + inter = ix * iy + if inter == 0: + return 0.0 + ua = (a[2] - a[0]) * (a[3] - a[1]) + (b[2] - b[0]) * (b[3] - b[1]) - inter + return inter / ua if ua else 0.0 + + +def main(run_dir, page_num): + run = Path(run_dir) + png = run / "pages" / f"page_{page_num:03d}.png" + regs = json.loads((run / "pages" / f"page_{page_num:03d}.regions.json").read_text())["regions"] + rmap = {r["id"]: r for r in regs} + paper = paper_of_height(Image.open(png).height) + + lines = detect_separator_lines(png) + dls = find_article_starts_by_dateline(regs, str(png), paper) + boxes = construct_boundaries(regs, dls, lines, lines["size"]) + + print(f"\npaper={paper} datelines={len(dls)} boxes={len(boxes)} " + f"rules H={len(lines['h'])} V={len(lines['v'])}") + + # validation + fails = [] + for bx in boxes: + db = rmap[bx["region_id"]]["bbox"] + bb = bx["bbox"] + contains = bb[0] <= db[0] and bb[1] <= db[1] and bb[2] >= db[2] and bb[3] >= db[3] + if not contains: + fails.append(f"R{bx['region_id']} box does NOT contain its dateline body") + if bb[2] - bb[0] < 50 or bb[3] - bb[1] < 50: + fails.append(f"R{bx['region_id']} degenerate box {bb}") + for i in range(len(boxes)): + for j in range(i + 1, len(boxes)): + ov = iou(boxes[i]["bbox"], boxes[j]["bbox"]) + if ov > 0.25: + fails.append(f"R{boxes[i]['region_id']} & R{boxes[j]['region_id']} overlap iou={ov:.2f}") + + for bx in sorted(boxes, key=lambda z: z["bbox"][1]): + print(f" R{bx['region_id']:>3} hl=R{bx['headline_region']} {bx['dateline'][:22]!r:24} " + f"bbox={bx['bbox']} walls={bx['walls']['ceil_rule']}/{bx['walls']['floor_rule']}") + print(f"\nVALIDATION: {'PASS' if not fails else f'{len(fails)} ISSUE(S)'}") + for f in fails: + print(" ⚠", f) + + img = cv2.imread(str(png)) + for L in lines["h"]: + cv2.line(img, (L["x1"], L["y1"]), (L["x2"], L["y2"]), (0, 0, 255), 4) + for L in lines["v"]: + cv2.line(img, (L["x1"], L["y1"]), (L["x2"], L["y2"]), (255, 0, 0), 4) + for bx in boxes: + b = bx["bbox"] + cv2.rectangle(img, (b[0], b[1]), (b[2], b[3]), (0, 180, 0), 8) + out = Path(f"/tmp/boundaries_{run.name}_p{page_num:03d}.png") + cv2.imwrite(str(out), img) + print("overlay:", out) + + +if __name__ == "__main__": + main(sys.argv[1], int(sys.argv[2])) diff --git a/scratch_py/_test_boundaries50.py b/scratch_py/_test_boundaries50.py new file mode 100644 index 0000000..8306c61 --- /dev/null +++ b/scratch_py/_test_boundaries50.py @@ -0,0 +1,170 @@ +"""50-run validation harness (no Claude). For every page with a regions.json in +the last N runs: detect rules -> dateline scan -> construct article boundaries -> +validate. Writes overlays for FAILING pages and a per-paper pass/fail report. + +Usage: venv/bin/python3 _test_boundaries50.py [N=50] [out_dir=/tmp/bnd50] +""" +import json +import sys +import traceback +from collections import Counter +from pathlib import Path + +import cv2 +from PIL import Image + +from _lines import detect_separator_lines +from _boundaries import construct_boundaries +from smart_extractor import find_article_starts_by_dateline + +OUT = Path("output") +DL_CACHE = Path("/tmp/dl_cache") +DL_CACHE.mkdir(parents=True, exist_ok=True) + + +def cached_datelines(regs, png, paper, key): + """Datelines depend only on the (fixed) regions + page image, so cache the + Tesseract scan to disk — re-validation after a _boundaries.py edit is then + instant (no OCR).""" + cf = DL_CACHE / f"{key}.json" + if cf.exists(): + return json.loads(cf.read_text()) + dls = find_article_starts_by_dateline(regs, png, paper) + cf.write_text(json.dumps(dls)) + return dls + + +def paper_of_height(h): + if 7040 <= h <= 7060: + return "sakshi" + if h in (6833, 6549): + return "andhra_jyothi" + if h == 6422: + return "namaste_telangana" + return f"h{h}" + + +def iou(a, b): + ix = max(0, min(a[2], b[2]) - max(a[0], b[0])) + iy = max(0, min(a[3], b[3]) - max(a[1], b[1])) + inter = ix * iy + if not inter: + return 0.0 + ua = (a[2]-a[0])*(a[3]-a[1]) + (b[2]-b[0])*(b[3]-b[1]) - inter + return inter / ua if ua else 0.0 + + +def validate(boxes, rmap): + issues = [] + for bx in boxes: + db = rmap[bx["region_id"]]["bbox"] + bb = bx["bbox"] + if not (bb[0] <= db[0]+2 and bb[1] <= db[1]+2 and bb[2] >= db[2]-2 and bb[3] >= db[3]-2): + issues.append(("no-contain", bx["region_id"])) + if bb[2]-bb[0] < 60 or bb[3]-bb[1] < 60: + issues.append(("degenerate", bx["region_id"])) + for i in range(len(boxes)): + for j in range(i+1, len(boxes)): + if iou(boxes[i]["bbox"], boxes[j]["bbox"]) > 0.25: + issues.append(("overlap", (boxes[i]["region_id"], boxes[j]["region_id"]))) + return issues + + +def draw(png, lines, boxes, out_path): + img = cv2.imread(str(png)) + for L in lines["h"]: + cv2.line(img, (L["x1"], L["y1"]), (L["x2"], L["y2"]), (0, 0, 255), 4) + for L in lines["v"]: + cv2.line(img, (L["x1"], L["y1"]), (L["x2"], L["y2"]), (255, 0, 0), 4) + for bx in boxes: + b = bx["bbox"] + cv2.rectangle(img, (b[0], b[1]), (b[2], b[3]), (0, 180, 0), 8) + cv2.imwrite(str(out_path), img) + + +def main(n=50, out_dir="/tmp/bnd50"): + out_dir = Path(out_dir) + out_dir.mkdir(parents=True, exist_ok=True) + runs = sorted([d for d in OUT.iterdir() if d.is_dir()], + key=lambda d: d.stat().st_mtime, reverse=True)[:n] + + per_paper_pages = Counter() + per_paper_fail = Counter() + issue_kinds = Counter() + fail_list = [] + total_pages = 0 + total_boxes = 0 + total_dls = 0 + + for run in runs: + pages = run / "pages" + if not pages.exists(): + continue + for reg_path in sorted(pages.glob("page_*.regions.json")): + png = reg_path.with_name(reg_path.name.replace(".regions.json", ".png")) + if not png.exists(): + continue + try: + regs = json.loads(reg_path.read_text())["regions"] + rmap = {r["id"]: r for r in regs} + paper = paper_of_height(Image.open(png).height) + lines = detect_separator_lines(png) + dls = cached_datelines(regs, str(png), paper, + f"{run.name}_{reg_path.stem}") + if not dls: + continue # no datelines -> nothing to bound + boxes = construct_boundaries(regs, dls, lines, lines["size"]) + issues = validate(boxes, rmap) + except Exception: + issues = [("exception", reg_path.name)] + boxes, dls, lines = [], [], {"h": [], "v": []} + traceback.print_exc() + + total_pages += 1 + total_boxes += len(boxes) + total_dls += len(dls) + per_paper_pages[paper] += 1 + if issues: + per_paper_fail[paper] += 1 + for k, _ in issues: + issue_kinds[k] += 1 + tag = f"{run.name}_{reg_path.stem}" + fail_list.append((tag, paper, len(dls), len(boxes), issues)) + try: + draw(png, lines, boxes, out_dir / f"FAIL_{tag}.png") + except Exception: + pass + + print("\n==================== BOUNDARY VALIDATION (last %d runs) ====================" % n) + print(f"{'paper':<20}{'pages':>8}{'fail':>7}{'pass%':>8}") + for paper in sorted(per_paper_pages): + pg = per_paper_pages[paper] + fl = per_paper_fail[paper] + print(f"{paper:<20}{pg:>8}{fl:>7}{100*(pg-fl)/pg:>7.0f}%") + tot = total_pages + fl = sum(per_paper_fail.values()) + print(f"{'TOTAL':<20}{tot:>8}{fl:>7}{100*(tot-fl)/max(1,tot):>7.0f}%") + print(f"\ndatelines total={total_dls} boxes total={total_boxes} " + f"(boxes should equal datelines: {'OK' if total_dls==total_boxes else 'MISMATCH'})") + print(f"issue kinds: {dict(issue_kinds)}") + + if fail_list: + print(f"\n--- {len(fail_list)} failing page(s) (overlays in {out_dir}/FAIL_*.png) ---") + for tag, paper, nd, nb, issues in fail_list[:60]: + ic = Counter(k for k, _ in issues) + print(f" [{paper}] {tag} dls={nd} boxes={nb} {dict(ic)}") + # machine-readable + (out_dir / "report.json").write_text(json.dumps({ + "pages": total_pages, "fail": fl, + "per_paper": {p: [per_paper_pages[p], per_paper_fail[p]] for p in per_paper_pages}, + "issue_kinds": dict(issue_kinds), + "fails": [[t, p, nd, nb, [list(x) if isinstance(x, tuple) else x for _, x in iss]] + for t, p, nd, nb, iss in fail_list], + }, indent=2)) + print(f"\nreport: {out_dir/'report.json'}") + + +if __name__ == "__main__": + n = int(sys.argv[1]) if len(sys.argv) > 1 else 50 + od = sys.argv[2] if len(sys.argv) > 2 else "/tmp/bnd50" + main(n, od) diff --git a/scratch_py/vl_compare.py b/scratch_py/vl_compare.py new file mode 100644 index 0000000..06bf0e7 --- /dev/null +++ b/scratch_py/vl_compare.py @@ -0,0 +1,294 @@ +#!/usr/bin/env python3 +""" +vl_compare.py — Run PaddleOCR-VL on one newspaper page and compare its +layout / reading-order output against the existing PaddleOCR LayoutDetection +regions (page_XXX.regions.json) produced by extractor.py. + +Goal: see whether VL's reading-order blocks line up with article boundaries +better than the current "box soup" before deciding to swap engines. + +USAGE + # auto-pick the most recent output//pages/page_001.png and its regions.json + python vl_compare.py + + # or point at a specific page + regions file + python vl_compare.py output/20260529_004651/pages/page_001.png \ + output/20260529_004651/pages/page_001.regions.json + +OUTPUT (written next to the page image) + page_XXX.vl_layout.json normalized VL blocks (id, type, bbox, reading order, text) + page_XXX.vl_debug.png page with VL boxes + reading-order numbers drawn + page_XXX.vl_vs_paddle.json side-by-side summary (counts by type, totals) + page_XXX.vl_raw.json raw VL result (best-effort, for debugging) + +NOTES + * PaddleOCR-VL is a ~0.9B vision-language model — expect a GPU and a model + download on first run. It is SEPARATE from the lightweight LayoutDetection + used in extractor.py. + * The VL entry point has moved across releases. This script tries several + known import paths and prints a clear hint if none are found so you can + adapt the one line to your installed version. + * Run inside your project venv: ./venv/bin/python vl_compare.py +""" + +import os +import sys +import glob +import json +import traceback + + +# -------------------------------------------------------------------------- +# 1. Resolve input page image + matching regions.json +# -------------------------------------------------------------------------- +def resolve_inputs(argv): + if len(argv) >= 2: + page_png = argv[1] + regions_json = argv[2] if len(argv) >= 3 else os.path.splitext(page_png)[0] + ".regions.json" + return page_png, regions_json + + # auto-discover: most recently modified page_001.png under output/ + candidates = sorted( + glob.glob("output/*/pages/page_001.png"), + key=os.path.getmtime, + reverse=True, + ) + if not candidates: + sys.exit("No page_001.png found under output/*/pages/. " + "Pass a page image path explicitly: python vl_compare.py [regions.json]") + page_png = candidates[0] + regions_json = os.path.splitext(page_png)[0] + ".regions.json" + return page_png, regions_json + + +# -------------------------------------------------------------------------- +# 2. Load the VL model (defensive across releases) +# -------------------------------------------------------------------------- +def load_vl_model(): + import paddleocr + print(f"paddleocr version: {getattr(paddleocr, '__version__', 'unknown')}") + + # Try the documented entry points, newest-first. Adjust if your release differs. + attempts = [ + ("paddleocr.PaddleOCRVL", lambda: paddleocr.PaddleOCRVL()), + ("paddleocr.PPDocVL", lambda: paddleocr.PPDocVL()), + ("paddleocr.DocVLM", lambda: paddleocr.DocVLM()), + ] + for name, ctor in attempts: + try: + model = ctor() + print(f"Loaded VL model via: {name}") + return model + except Exception as e: # noqa: BLE001 + print(f" - {name} unavailable: {e}") + + # Last resort: PaddleX pipeline name + try: + from paddlex import create_pipeline + model = create_pipeline(pipeline="PaddleOCR-VL") + print("Loaded VL model via: paddlex.create_pipeline('PaddleOCR-VL')") + return model + except Exception as e: # noqa: BLE001 + print(f" - paddlex pipeline unavailable: {e}") + + sys.exit( + "\nCould not load PaddleOCR-VL with any known entry point.\n" + "Check the exact class name for your installed version, e.g.:\n" + " python -c \"import paddleocr; print([n for n in dir(paddleocr) if 'VL' in n or 'Doc' in n])\"\n" + "then edit the `attempts` list near the top of load_vl_model()." + ) + + +# -------------------------------------------------------------------------- +# 3. Run VL + normalize its output to the same shape as regions.json +# -------------------------------------------------------------------------- +def to_bbox(coord): + """Accept [x1,y1,x2,y2] or polygon [x1,y1,...] -> [x1,y1,x2,y2] ints.""" + if coord is None: + return [0, 0, 0, 0] + if hasattr(coord, "tolist"): + coord = coord.tolist() + coord = [float(v) for v in coord] + if len(coord) == 4: + return [int(v) for v in coord] + xs = coord[0::2] + ys = coord[1::2] + return [int(min(xs)), int(min(ys)), int(max(xs)), int(max(ys))] + + +def run_vl(model, page_png): + if hasattr(model, "predict"): + raw = model.predict(page_png) + elif callable(model): + raw = model(page_png) + else: + raw = model.predict(input=page_png) + + # raw is typically a list of result objects (dict-like). Iterate generically. + results = list(raw) if not isinstance(raw, list) else raw + + blocks = [] + raw_dump = [] + + for res in results: + # try to get a plain dict for dumping/parsing + d = None + for attr in ("json", "res", "_to_dict"): + try: + v = getattr(res, attr, None) + d = v() if callable(v) else v + if d: + break + except Exception: # noqa: BLE001 + pass + if d is None and isinstance(res, dict): + d = res + raw_dump.append(_jsonable(d if d is not None else str(res))) + + # VL layout blocks commonly live under one of these keys + layout = None + if isinstance(d, dict): + for key in ("layout_det_res", "parsing_res_list", "layout", "blocks", "boxes"): + if key in d and d[key]: + layout = d[key] + break + if layout is None: + continue + + # layout may itself be a dict containing 'boxes' + if isinstance(layout, dict): + layout = layout.get("boxes", layout.get("parsing_res_list", [])) + + for i, b in enumerate(layout): + if not isinstance(b, dict): + continue + coord = b.get("coordinate", b.get("bbox", b.get("block_bbox", b.get("layout_bbox")))) + label = str(b.get("label", b.get("type", b.get("block_label", "unknown")))).lower() + score = float(b.get("score", b.get("confidence", 0)) or 0) + order = b.get("reading_order", b.get("order", b.get("index", i))) + text = b.get("text", b.get("block_content", b.get("rec_text", ""))) + if isinstance(text, list): + text = " ".join(map(str, text)) + blocks.append({ + "id": len(blocks) + 1, + "type": label, + "bbox": to_bbox(coord), + "confidence": round(score, 4), + "reading_order": order, + "text_preview": (str(text)[:120] if text else ""), + }) + + # sort by reading order if available + try: + blocks.sort(key=lambda x: (x["reading_order"] if isinstance(x["reading_order"], (int, float)) else 0)) + for i, b in enumerate(blocks, 1): + b["id"] = i + except Exception: # noqa: BLE001 + pass + + return blocks, raw_dump + + +def _jsonable(obj): + try: + json.dumps(obj) + return obj + except Exception: # noqa: BLE001 + return str(obj) + + +# -------------------------------------------------------------------------- +# 4. Compare against existing PaddleOCR regions +# -------------------------------------------------------------------------- +def type_counts(regions): + c = {} + for r in regions: + t = r.get("type", "unknown") + c[t] = c.get(t, 0) + 1 + return dict(sorted(c.items(), key=lambda x: -x[1])) + + +def draw_debug(page_png, blocks, out_png): + try: + from PIL import Image, ImageDraw, ImageFont + except Exception: # noqa: BLE001 + print("Pillow not available — skipping debug image.") + return + img = Image.open(page_png).convert("RGB") + draw = ImageDraw.Draw(img) + try: + font = ImageFont.truetype("DejaVuSans-Bold.ttf", 60) + except Exception: # noqa: BLE001 + font = ImageFont.load_default() + for b in blocks: + x1, y1, x2, y2 = b["bbox"] + draw.rectangle([x1, y1, x2, y2], outline=(220, 30, 30), width=6) + tag = f"{b['id']}:{b['type']}" + draw.text((x1 + 8, y1 + 8), tag, fill=(220, 30, 30), font=font) + img.save(out_png) + print(f" wrote {out_png}") + + +def main(): + page_png, regions_json = resolve_inputs(sys.argv) + if not os.path.exists(page_png): + sys.exit(f"Page image not found: {page_png}") + print(f"Page image : {page_png}") + print(f"Regions JSON: {regions_json} (exists={os.path.exists(regions_json)})") + + stem = os.path.splitext(page_png)[0] + + # existing paddle regions + paddle_regions = [] + if os.path.exists(regions_json): + with open(regions_json) as f: + paddle_regions = json.load(f).get("regions", []) + + # run VL + print("\nLoading PaddleOCR-VL (first run downloads the model)...") + model = load_vl_model() + print("Running VL on the page...") + blocks, raw_dump = run_vl(model, page_png) + + # write outputs + with open(stem + ".vl_layout.json", "w") as f: + json.dump({"page_image": os.path.basename(page_png), "blocks": blocks}, f, indent=2, ensure_ascii=False) + with open(stem + ".vl_raw.json", "w") as f: + json.dump(raw_dump, f, indent=2, ensure_ascii=False) + + summary = { + "page_image": os.path.basename(page_png), + "paddle_layoutdetection": { + "total_regions": len(paddle_regions), + "by_type": type_counts(paddle_regions), + }, + "paddleocr_vl": { + "total_blocks": len(blocks), + "by_type": type_counts(blocks), + "has_reading_order": any(isinstance(b.get("reading_order"), (int, float)) for b in blocks), + }, + } + with open(stem + ".vl_vs_paddle.json", "w") as f: + json.dump(summary, f, indent=2, ensure_ascii=False) + + draw_debug(page_png, blocks, stem + ".vl_debug.png") + + print("\n" + "=" * 60) + print("COMPARISON SUMMARY") + print("=" * 60) + print(json.dumps(summary, indent=2, ensure_ascii=False)) + print("\nWrote:") + for ext in (".vl_layout.json", ".vl_raw.json", ".vl_vs_paddle.json", ".vl_debug.png"): + print(" " + stem + ext) + print("\nNext: open page_XXX_regions_debug.png (existing) next to page_XXX.vl_debug.png") + print("and eyeball whether VL's blocks track article boundaries better.") + + +if __name__ == "__main__": + try: + main() + except SystemExit: + raise + except Exception: # noqa: BLE001 + traceback.print_exc() + sys.exit("\nFailed. If the error is about the VL class name, see load_vl_model().") diff --git a/smart_extractor.py b/smart_extractor.py new file mode 100644 index 0000000..137bb88 --- /dev/null +++ b/smart_extractor.py @@ -0,0 +1,6657 @@ +""" +Smart Extractor — One Claude call per page for political article detection + cropping. + +PaddleOCR finds precise regions → Claude identifies ONLY political region groups → crop. + +Usage: + python smart_extractor.py newspaper.pdf + python smart_extractor.py newspaper.pdf --output ./my_output +""" +import os +import sys +import json +import base64 +import argparse +from pathlib import Path +from io import BytesIO +from datetime import datetime + +import fitz # PyMuPDF +from PIL import Image +from difflib import SequenceMatcher + + +# --------------------------------------------------------------------------- +# Step 1: PDF to page images +# --------------------------------------------------------------------------- +def render_pdf_pages(pdf_path, pages_dir, target_w=4200, target_h=7400): + doc = fitz.open(pdf_path) + pages = [] + for i, page in enumerate(doc, 1): + scale_x = target_w / page.rect.width + scale_y = target_h / page.rect.height + scale = min(scale_x, scale_y) + mat = fitz.Matrix(scale, scale) + pix = page.get_pixmap(matrix=mat) + out_path = str(pages_dir / f"page_{i:03d}.png") + pix.save(out_path) + pages.append({"page": i, "path": out_path, "width": pix.width, "height": pix.height}) + print(f" Page {i}: {pix.width}x{pix.height} px") + doc.close() + return pages + + +# --------------------------------------------------------------------------- +# Step 2: PaddleOCR layout detection +# --------------------------------------------------------------------------- +def detect_regions(page_png_path, threshold=None): + import paddleocr + import numpy as np + + # Load model (cached after first call). Default (threshold=None) uses + # PP-DocLayout's stock ~0.5 cutoff — what Claude grouping is tuned to see. + # A lower threshold can be passed for experimentation (e.g. via the + # _replay_regions.py harness), but is NOT used in the production pipeline: + # lowering it floods grouping with low-confidence + duplicate boxes. + if threshold is None: + layout = paddleocr.LayoutDetection() + else: + layout = paddleocr.LayoutDetection(threshold=threshold) + + raw = layout.predict(page_png_path) + + regions = [] + if isinstance(raw, list) and len(raw) > 0: + for det_result in raw: + boxes = None + if hasattr(det_result, 'get'): + boxes = det_result.get('boxes', None) + elif hasattr(det_result, 'boxes'): + boxes = det_result.boxes + if boxes is not None: + for box in boxes: + rid = len(regions) + 1 + coord = None + if isinstance(box, dict): + coord = box.get('coordinate', box.get('bbox', None)) + label = box.get('label', box.get('type', 'unknown')).lower() + score = float(box.get('score', 0)) + elif hasattr(box, 'coordinate'): + coord = box.coordinate + label = getattr(box, 'label', 'unknown').lower() + score = float(getattr(box, 'score', 0)) + else: + label, score = 'unknown', 0.0 + + bbox = [0, 0, 0, 0] + if coord is not None: + if hasattr(coord, 'tolist'): + coord = coord.tolist() + coord = [float(v) for v in coord] + if len(coord) == 4: + bbox = [int(v) for v in coord] + regions.append({"id": rid, "type": label, "bbox": bbox, "confidence": score}) + + print(f" {len(regions)} layout regions detected") + return regions + + +def log_region_inventory(regions, page_num, pre_counts=None): + """Diagnostic log: a clearly-sectioned, column-aligned inventory of the page — + counts per region type, then a table of every doc_title and paragraph_title + (id, the four bbox numbers, width, height, confidence). Region boxes carry no + text, so this is the structural inventory only.""" + from collections import Counter + bar = "=" * 64 + counts = Counter(r.get("type", "unknown") for r in regions) + + print() + print(" " + bar) + print(f" PAGE {page_num} — REGION INVENTORY ({len(regions)} regions total)") + print(" " + bar) + + # --- counts by type, dotted leaders, right-aligned numbers --- + if pre_counts is not None: + print(" Counts by type (before promotion -> after promotion)") + all_types = sorted(set(counts) | set(pre_counts), + key=lambda t: (-counts.get(t, 0), t)) + for t in all_types: + before, after = pre_counts.get(t, 0), counts.get(t, 0) + dots = "." * max(2, 20 - len(t)) + mark = " <== changed by promotion" if before != after else "" + print(f" {t} {dots} {before:>3d} -> {after:>3d}{mark}") + else: + print(" Counts by type") + for t, c in sorted(counts.items(), key=lambda kv: (-kv[1], kv[0])): + dots = "." * max(2, 20 - len(t)) + print(f" {t} {dots} {c:>3d}") + + # --- aligned table for headline-type regions --- + def table(label): + items = sorted([r for r in regions if r.get("type") == label], + key=lambda z: (z["bbox"][1], z["bbox"][0])) + print() + print(f" {label.upper()} ({len(items)})") + print(f" {'ID':>3} {'X1':>5} {'Y1':>5} {'X2':>5} {'Y2':>5} | {'W':>5} {'H':>4} {'CONF':>4} FLAG") + print(" " + "-" * 58) + for r in items: + b = r["bbox"] + w, h = b[2] - b[0], b[3] - b[1] + # a full-width paragraph_title is very likely a real headline mis-tagged + flag = "<- wide, likely a headline" if (label == "paragraph_title" and w >= 1000) else "" + print(f" {r['id']:>3} {b[0]:>5} {b[1]:>5} {b[2]:>5} {b[3]:>5} | " + f"{w:>5} {h:>4} {r.get('confidence', 0):>4.2f} {flag}") + + table("doc_title") + table("paragraph_title") + print(" " + bar) + print() + + +# --------------------------------------------------------------------------- +# Headline promotion: fix mis-tagged headlines (paragraph_title -> doc_title) +# --------------------------------------------------------------------------- +def _dateline_in_text(text, paper): + """Return the dateline string (incl. its colon) if the first line opens with one, + else None. Sakshi = ':' (no date). Andhra Jyothi = ', (...):'. + + A dateline often WRAPS across several lines in narrow columns — e.g. + 'హుస్నా బాద్,' / 'జూన్ 1 (ఆంధ్ర' / 'జ్యోతి): …'. Accumulate the first few lines + until the closing colon appears, capped at 3 lines so a colon deep in body text + is never pulled in (the prefix length/word guards below reject false matches).""" + lines = [l.strip() for l in (text or "").splitlines() if l.strip()] + if not lines: + return None + # Andhra Jyothi's closing colon is sometimes OCR'd as '=' (e.g. '(ఆంధ్రజ్యోతి)='), + # so accept '=' as a delimiter for that paper only — other papers stay strict. + # Tesseract also routinely misreads the dateline colon as a SEMICOLON ('మే 19 ;'), so ';' + # is accepted everywhere (the digit + short-prefix + no-sentence-punctuation guards below + # keep a mid-body semicolon from matching). This recovers e.g. JG R18 'చిల్పూరు, మే 19 ;'. + delims = ":：=;" if paper == "andhra_jyothi" else ":：;" + line = lines[0] + for nxt in lines[1:3]: # join up to 3 leading lines (narrow cols) + if any(c in line for c in delims): + break + line = line + " " + nxt + pos = next((i for i, c in enumerate(line) if c in delims), -1) + if pos <= 0: + return None + prefix = line[:pos].strip() + # Multi-town datelines join several towns with '/' ('గజ్వేల్ రూరల్/మర్కూక్, …'), + # which pushes the prefix past the normal 45-char cap — allow more length when a + # slash is present (the digit/masthead guard below still gates false matches). + max_len = 70 if "/" in prefix else 45 + if not (0 < len(prefix) <= max_len) or len(prefix.split()) > 8: + return None + # A colon matched mid-sentence (prefix carrying sentence punctuation) is not a + # dateline — '!?।' always disqualify. The separator comma after the town is often + # OCR'd as a '.', so for ANDHRA JYOTHI ONLY a single '.' OR ',' is tolerated — but + # not both (both together signals a real sentence). Other papers keep the strict + # rule: any '.' disqualifies. + if any(p in prefix for p in "!?।"): + return None + if paper == "andhra_jyothi": + if "." in prefix and "," in prefix: + return None + elif "." in prefix: + return None + if paper == "andhra_jyothi": + # Andhra Jyothi datelines carry a date (digits) and/or the masthead. + if not (any(ch.isdigit() for ch in prefix) or "ఆంధ్రజ్యోతి" in line[:pos + 2]): + return None + elif paper == "namaste_telangana": + # Namaste Telangana: ', :' — a date (digits), no masthead. + if not any(ch.isdigit() for ch in prefix): + return None + return line[:pos + 1] + + +def _bbox_overlap_ratio(a, b): + ix = max(0, min(a[2], b[2]) - max(a[0], b[0])) + iy = max(0, min(a[3], b[3]) - max(a[1], b[1])) + inter = ix * iy + if inter == 0: + return 0.0 + area_a = (a[2] - a[0]) * (a[3] - a[1]) + area_b = (b[2] - b[0]) * (b[3] - b[1]) + return inter / max(1, min(area_a, area_b)) + + +def _bbox_iou(a, b): + ix = max(0, min(a[2], b[2]) - max(a[0], b[0])) + iy = max(0, min(a[3], b[3]) - max(a[1], b[1])) + inter = ix * iy + if inter == 0: + return 0.0 + area_a = (a[2] - a[0]) * (a[3] - a[1]) + area_b = (b[2] - b[0]) * (b[3] - b[1]) + return inter / max(1, area_a + area_b - inter) + + +def dedup_overlapping_regions(regions, thresh=0.85): + """Drop near-duplicate detections (PaddleOCR sometimes emits the SAME element + twice at the identical box, e.g. one heading tagged both doc_title and + paragraph_title). Uses IoU so only NEARLY-IDENTICAL boxes are merged — a small + region that merely sits inside a bigger one (text over a photo, a caption) is + NOT dropped. Keep the better one: doc_title > paragraph_title, then higher conf.""" + rank = {"doc_title": 2, "paragraph_title": 1} + drop = set() + for i in range(len(regions)): + if regions[i]["id"] in drop: + continue + for j in range(i + 1, len(regions)): + if regions[j]["id"] in drop: + continue + if _bbox_iou(regions[i]["bbox"], regions[j]["bbox"]) > thresh: + a, b = regions[i], regions[j] + ka = (rank.get(a.get("type"), 0), a.get("confidence", 0)) + kb = (rank.get(b.get("type"), 0), b.get("confidence", 0)) + loser, keep = (b, a) if ka >= kb else (a, b) + drop.add(loser["id"]) + print(f" · dedup: dropped region {loser['id']} ({loser.get('type')}, " + f"conf={loser.get('confidence',0):.2f}) — near-duplicate of region " + f"{keep['id']} ({keep.get('type')}, conf={keep.get('confidence',0):.2f})") + return [r for r in regions if r["id"] not in drop] + + +def _box_ink_color(page_img, bbox, cache): + """Return (ink_density, mean_dark_color) for a box — used to compare font + weight (how much ink) and text color between two stacked title lines.""" + key = tuple(bbox) + if key in cache: + return cache[key] + c = page_img.crop(key).convert("RGB") + if c.width == 0 or c.height == 0: + cache[key] = (0.0, (0, 0, 0)); return cache[key] + s = c.resize((min(200, c.width), min(60, c.height))) + px = s.load() + dark = [] + for y in range(s.height): + for x in range(s.width): + r, g, b = px[x, y][:3] + if 0.299 * r + 0.587 * g + 0.114 * b < 140: # dark = ink + dark.append((r, g, b)) + density = len(dark) / max(1, s.width * s.height) + if dark: + n = len(dark) + col = (sum(p[0] for p in dark) / n, sum(p[1] for p in dark) / n, sum(p[2] for p in dark) / n) + else: + col = (0, 0, 0) + cache[key] = (density, col) + return cache[key] + + +def merge_stacked_title_lines(regions, page_img_path=None, max_gap=90, x_frac=0.5, + h_ratio=1.8, color_max=60, density_max=0.10): + """Merge consecutive title-type boxes that are really the lines of ONE headline + (PaddleOCR boxes each wrapped line of a headline separately). Conservative: + both title-type, same column (x-overlap), small vertical gap, similar height, + lower line not wider, NOTHING between them, AND (if the page image is given) + the two lines share font weight (ink density) and text color.""" + page_img = None + if page_img_path: + try: + page_img = Image.open(page_img_path).convert("RGB") + except Exception: + page_img = None + feat_cache = {} + + def is_title(r): + return r.get("type") in ("doc_title", "paragraph_title") + + def x_overlap(a, b): + ov = max(0, min(a[2], b[2]) - max(a[0], b[0])) + return ov / max(1, min(a[2] - a[0], b[2] - b[0])) + + def something_between(a, b): + x1, x2 = max(a[0], b[0]), min(a[2], b[2]) + for r in regions: + if is_title(r): + continue + rb = r["bbox"] + cy = (rb[1] + rb[3]) / 2 + if a[3] < cy < b[1] and not (rb[2] <= x1 or rb[0] >= x2): + return True + return False + + regions = list(regions) + changed = True + while changed: + changed = False + titles = sorted([r for r in regions if is_title(r)], key=lambda z: z["bbox"][1]) + for A in titles: + best, best_gap = None, 1e9 + for B in titles: + if B is A: + continue + a, b = A["bbox"], B["bbox"] + if b[1] < a[3] - 5: # B must sit below A + continue + gap = b[1] - a[3] + if gap > max_gap or x_overlap(a, b) < x_frac: + continue + ha, hb = a[3] - a[1], b[3] - b[1] + if max(ha, hb) / max(1, min(ha, hb)) > h_ratio: + continue + # Wrapped headline lines: the lower line is the same width or + # SHORTER than the upper (line 1 fills the column, line 2 is + # shorter). If the lower box is clearly WIDER, the upper one is a + # section label / kicker sitting above a wider headline — do NOT merge. + wa, wb = a[2] - a[0], b[2] - b[0] + if wb > wa * 1.15: + continue + if something_between(a, b): + continue + # Style match: same font weight (ink density) and same text color. + if page_img is not None: + da, ca = _box_ink_color(page_img, a, feat_cache) + db, cb = _box_ink_color(page_img, b, feat_cache) + color_dist = sum((p - q) ** 2 for p, q in zip(ca, cb)) ** 0.5 + if color_dist > color_max or abs(da - db) > density_max: + continue + if gap < best_gap: + best, best_gap = B, gap + if best is not None: + a, b = A["bbox"], best["bbox"] + A["bbox"] = [min(a[0], b[0]), min(a[1], b[1]), max(a[2], b[2]), max(a[3], b[3])] + if best.get("type") == "doc_title": + A["type"] = "doc_title" + A["confidence"] = max(A.get("confidence", 0), best.get("confidence", 0)) + A.setdefault("_merged_lines", [A["id"]]).append(best["id"]) + print(f" ⊕ merged stacked headline line: region {best['id']} into region " + f"{A['id']} (gap={best_gap:.0f}px) → bbox {A['bbox']}") + regions = [r for r in regions if r is not best] + changed = True + break + return regions + + +def drop_masthead_regions(regions, paper, page_num, page_img_path): + """Remove the front-page masthead (the fixed newspaper header) so it is never + mis-grouped as a news article. + + Namaste Telangana opens page 1 with a banner holding the giant edition + nameplate (e.g. 'జనగామ'). PaddleOCR detects it three different ways, all + handled here: + • an explicit 'header'-typed region, + • an 'image' region with the nameplate as a doc_title sitting INSIDE it, or + • a bare nameplate doc_title at the very top (banner not detected at all). + Either way Claude sometimes seeds an 'article' from the nameplate (which then + over-grows into the real story beside it). A genuine page-1 headline never sits + in the top ~10% strip (real stories start well below the masthead), never lives + inside an image, and a 'header' is never a story — so we drop the masthead + banner, the nameplate, and the date-line beneath it. + + Gated to page 1 (mastheads are front-page) so inner-page content is untouched. + """ + if paper == "andhra_jyothi": + # Andhra Jyothi section fronts open with a nameplate band — the 'ఆంధ్రజ్యోతి' + # logo + the coloured 'సిద్దిపేట / date' banner — as image/header regions + # pinned to the very TOP strip of the page. Real article photos start well + # below it, so any image/header whose TOP sits in the top ~5% is masthead and + # must never be folded into a story. Drop them on every page (inner pages + # simply have nothing up there). + try: + page_h = Image.open(page_img_path).height + except Exception: + page_h = 6833 + strip = page_h * 0.05 + drop_ids = {r["id"] for r in regions + if r["bbox"][1] < strip and r.get("type") in ("image", "header")} + if drop_ids: + print(f" 🗞️ Masthead: dropped {len(drop_ids)} Andhra Jyothi nameplate " + f"region(s) {sorted(drop_ids)} (top-strip banner)") + return [r for r in regions if r["id"] not in drop_ids] + if paper == "sakshi": + # Sakshi opens every page with a furniture band ABOVE the first headline: + # page 1 = nameplate + weather box + photo-brief teasers (with '-8లో' jump + # pointers) + the full-width date strip; inner pages = the section header. + # With no Sakshi masthead rule this band folds into the top articles (the + # weather box into the lead story) and the date strip even SEEDS a fake + # dateline article ('బుధవారం 20 మే 2026' OCRs as a dateline). Everything + # that ends above the topmost headline-type region is that band → drop. + tops = [r["bbox"][1] for r in regions + if r.get("type") in ("doc_title", "paragraph_title")] + try: + page_h = Image.open(page_img_path).height + except Exception: + page_h = 7055 + if tops: + cut = min(tops) + if cut < page_h * 0.30: # sanity: band must be page furniture + drop_ids = {r["id"] for r in regions if r["bbox"][3] <= cut} + if drop_ids: + print(f" 🗞️ Masthead: dropped {len(drop_ids)} Sakshi top-band " + f"region(s) {sorted(drop_ids)} (above first headline y={cut})") + return [r for r in regions if r["id"] not in drop_ids] + return regions + if paper != "namaste_telangana" or page_num != 1: + return regions + try: + page_h = Image.open(page_img_path).height + except Exception: + page_h = 7400 + top_band = page_h * 0.18 + nameplate_band = page_h * 0.10 # the very top strip the masthead occupies + + def _inside(inner, outer, frac=0.8): + ix = max(0, min(inner[2], outer[2]) - max(inner[0], outer[0])) + iy = max(0, min(inner[3], outer[3]) - max(inner[1], outer[1])) + ia = (inner[2] - inner[0]) * (inner[3] - inner[1]) + return ia > 0 and (ix * iy) / ia >= frac + + def _hoverlap(a, b): + return max(0, min(a[2], b[2]) - max(a[0], b[0])) + + # Banner regions: explicit header, or an image hosting the nameplate doc_title. + banner_regs = [ + r for r in regions if r["bbox"][1] < top_band and ( + r.get("type") == "header" + or (r.get("type") == "image" + and any(d.get("type") == "doc_title" and _inside(d["bbox"], r["bbox"]) + for d in regions)) + ) + ] + # Bare nameplate: a doc_title whose top sits in the top ~10% strip. + nameplate_titles = [ + r for r in regions + if r.get("type") == "doc_title" and r["bbox"][1] < nameplate_band + ] + if not banner_regs and not nameplate_titles: + return regions + + drop_ids = set() + for im in banner_regs: + drop_ids.add(im["id"]) + for r in regions: + if r["id"] != im["id"] and _inside(r["bbox"], im["bbox"]): + drop_ids.add(r["id"]) + for nt in nameplate_titles: + drop_ids.add(nt["id"]) + nb = nt["bbox"] + # Sweep the date-line text directly beneath the nameplate (same x-span). + for r in regions: + rb = r["bbox"] + if (r["id"] not in drop_ids and r.get("type") == "text" + and nb[3] <= rb[1] <= nb[3] + 250 + and _hoverlap(rb, nb) > 0): + drop_ids.add(r["id"]) + if drop_ids: + print(f" 🗞️ Masthead: dropped {len(drop_ids)} header region(s) " + f"{sorted(drop_ids)} (Namaste Telangana page-1 nameplate)") + return [r for r in regions if r["id"] not in drop_ids] + + +def drop_classifieds_regions(regions, page_img_path): + """Drop CLASSIFIEDS blocks — the 'Classified Ads / phone-number listing' columns — which + are page FURNITURE, not news. Left in, a neighbouring article's column-fold grows down + into them (e.g. JG R65 swallowing the whole left column). A region is classifieds when + its OCR text carries a classifieds keyword OR ≥5 phone-like numbers (a news article + rarely has >3). Only TALL regions are OCR-scanned (a classifieds listing is a tall + column), which keeps the cost down and avoids touching ordinary text. Court/legal + notices are intentionally NOT dropped here.""" + try: + import pytesseract + import re as _re + from PIL import Image as _Image + if "eng" not in set(pytesseract.get_languages(config="")): + return regions + except Exception: + return regions + try: + img = _Image.open(page_img_path).convert("RGB") + except Exception: + return regions + KW = ("classified", "classifieds", "contact", "క్లాసిఫైడ") + drop = set() + for r in regions: + b = r["bbox"] + if (b[3] - b[1]) < 400: # classifieds listings are tall columns + continue + try: + # Cheap OCR: phone-runs (\d{6,}) and the English keyword are the real signals, + # so a half-res ENG-only pass suffices — ~2× faster than full-res tel+eng and + # verified to give the IDENTICAL drop set on the JG/AJ/NT corpus. + c = img.crop(tuple(b)) + c = c.resize((max(1, c.width // 2), max(1, c.height // 2))) + t = pytesseract.image_to_string(c, lang="eng") + except Exception: + continue + tl = t.lower() + phones = len(_re.findall(r"\d{6,}", t)) + if any(k in tl for k in KW) or phones >= 5: + drop.add(r["id"]) + if drop: + print(f" 🗞️ Classifieds: dropped {len(drop)} region(s) {sorted(drop)} " + f"(phone-listing / 'Classified Ads' furniture)") + return [r for r in regions if r["id"] not in drop] + + +def tag_legal_notices(regions, page_img_path): + """Tag COURT / LEGAL-NOTICE regions (set r['_legal']='core'|'support'). A legal notice + (court summons, name-change / matrimonial petition, public 'by order of the court' notice) + is page FURNITURE like classifieds — but, per the brief, it is KEPT and cropped as its OWN + section rather than dropped or absorbed into a neighbouring article (e.g. JG R65 swallowed a + court summons stacked in its left column). These notices are set in English legal + boilerplate, so a half-res ENG-only OCR detects them reliably and cheaply. The actual + carve-into-its-own-box happens later inside `cluster_all_article_blocks` (which reads these + tags, no OCR), so every consumer — the crops AND the debug boundary map — sees the section.""" + try: + import pytesseract + import statistics as _st + from PIL import Image as _Image + if "eng" not in set(pytesseract.get_languages(config="")): + return regions + except Exception: + return regions + try: + img = _Image.open(page_img_path).convert("RGB") + except Exception: + return regions + # CORE = unambiguous court/petition boilerplate; SUPPORT = corroborating (won't seed a + # cluster on its own — a carve requires ≥1 CORE hit — but is folded in when adjacent). + CORE = ("hon'ble", "honble", "honourable", "in the court of", "by order of", "plaintiff", + "defendant", "petitioner", "respondent", "summons", "decree", "civil judge", + "district judge", "munsif", "o.s.no", "o.s. no", "i.a.no", "i.a. no", "sub court", + "appeal suit", "notice is hereby") + SUPPORT = ("advocate", "vakalat", "vakalath", "whereas") + _tw = [r["bbox"][2] - r["bbox"][0] for r in regions if r.get("type") == "text"] + pitch = _st.median(_tw) if _tw else 600 + for r in regions: + if r.get("type") not in ("text", "paragraph_title"): + continue + b = r["bbox"]; w = b[2] - b[0]; h = b[3] - b[1] + if h < 30 or w < 60 or w > 1.4 * pitch: # single-column notice lines only + continue + try: + c = img.crop(tuple(b)) + c = c.resize((max(1, c.width // 2), max(1, c.height // 2))) + t = pytesseract.image_to_string(c, lang="eng").lower() + except Exception: + continue + if any(k in t for k in CORE): + r["_legal"] = "core" + elif any(k in t for k in SUPPORT): + r["_legal"] = "support" + return regions + + +def recover_orphan_text_headlines(blocks, regions, page_img_path): + """Recover a headline PaddleOCR mis-tagged as `text` — but ONLY when it ended up as a + stray ORPHAN block. That orphan-ness is the real signature of a lost headline: a + photo CAPTION is also short + bold + above a photo, but it always FOLDS into its + article (never orphans), so scoping to orphans leaves captions untouched. For a + qualifying orphan (short + bold vs. body ink + sitting above a photo), re-tag its text + region as a doc_title and return the re-tagged ids — the caller re-clusters so the + recovered headline anchors its article.""" + try: + page = Image.open(page_img_path).convert("RGB") + except Exception: + return [] + cache = {} + rmap = {r["id"]: r for r in regions} + texts = [r for r in regions if r.get("type") == "text"] + if not texts: + return [] + dens = sorted(_box_ink_color(page, r["bbox"], cache)[0] for r in texts) + med_dens = dens[len(dens) // 2] if dens else 0 # body-text ink baseline + tws = sorted((r["bbox"][2] - r["bbox"][0]) for r in texts) + pitch = tws[len(tws) // 2] if tws else 0 + + def _xov(a, b): + return max(0, min(a[2], b[2]) - max(a[0], b[0])) + + retag = [] + for blk in blocks: + if blk.get("kind") != "O": + continue # ONLY stray orphans + for m in blk.get("members", []): + r = rmap.get(m) + if not r or r.get("type") != "text": + continue + b = r["bbox"] + w, h = b[2] - b[0], b[3] - b[1] + if h > 190 or w < 0.45 * max(1, pitch): # short (≤~2 lines), column-ish wide + continue + if _box_ink_color(page, b, cache)[0] < 1.4 * max(0.01, med_dens): + continue # bold (≥1.4× body ink) + if not any(o.get("type") == "image" and o["bbox"][1] >= b[3] - 10 + and o["bbox"][1] - b[3] <= 220 and _xov(b, o["bbox"]) >= 0.4 * w + for o in regions): + continue # must sit above a photo + r["type"] = "doc_title" + r["_headline_from_text"] = True + retag.append(m) + print(f" ⬆ ORPHAN HEADLINE: R{m} re-tagged text→doc_title " + f"(bold orphan sitting above a photo)") + return retag + + +def promote_paragraph_titles(regions, page_img_path, paper, width_min=1000, + col_frac=0.8, abs_floor=250): + """Reclassify a paragraph_title to a doc_title ONLY when BOTH hold: + (1) it is full-column wide (>= width_min px), AND + (2) a fresh dateline body starts directly beneath it. + Logs each promotion with the dateline text found, for debugging.""" + try: + import pytesseract + if "tel" not in set(pytesseract.get_languages(config="")): + print(" (Headline promotion skipped — Telugu Tesseract 'tel' not installed)") + return regions + except Exception: + print(" (Headline promotion skipped — pytesseract not available)") + return regions + + page = Image.open(page_img_path) + + def first_line(bbox): + l, t, r, b = bbox + strip_h = min(220, b - t) # tall enough for a 3-line wrap in narrow columns + if strip_h <= 0: + return "" + try: + return pytesseract.image_to_string(page.crop((l, t, r, t + strip_h)), lang="tel") + except Exception: + return "" + + promoted = 0 + for reg in regions: + if reg.get("type") != "paragraph_title": + continue + b = reg["bbox"] + w = b[2] - b[0] + if w < abs_floor: # ignore tiny inline labels outright + continue + # skip if it duplicates an existing doc_title (PaddleOCR double-detects the + # same heading as both doc_title and paragraph_title — promoting would make + # two doc_titles for one heading) + if any(_bbox_overlap_ratio(b, dt["bbox"]) > 0.6 + for dt in regions if dt is not reg and dt.get("type") == "doc_title"): + print(f" · not promoted: region {reg['id']} (W={w}) — already covered by an existing doc_title") + continue + # Body blocks beneath the heading. Use horizontal OVERLAP, not centre + # containment: a wide headline often sits above a MULTI-COLUMN body split + # around a photo, so its centre can land on the photo with no text under it. + below = [x for x in regions if x.get("type") == "text" + and x["bbox"][1] >= b[3] - 10 + and max(0, min(x["bbox"][2], b[2]) - max(x["bbox"][0], b[0])) >= 0.2 * w] + if not below: + print(f" · not promoted: region {reg['id']} (W={w}) — no body block below it") + continue + body = min(below, key=lambda x: x["bbox"][1] - b[3]) + body_w = body["bbox"][2] - body["bbox"][0] + # condition 1 (column-relative): the heading spans its own column (≈ the + # width of the body beneath it), OR is full-page wide. This catches a real + # headline sitting in a narrow column (which an absolute px rule misses). + # For a multi-column body the heading is far wider than one column, so this + # holds comfortably. + spans_column = (w >= col_frac * body_w) or (w >= width_min) + if not spans_column: + print(f" · not promoted: region {reg['id']} (W={w}) — not full-column width " + f"(body region {body['id']} W={body_w}; needs ≥ {int(col_frac * body_w)})") + continue + # condition 2: a fresh dateline opens the body. With a multi-column body the + # dateline is in the first column, which may not be the centremost block, so + # scan the blocks of the immediate row beneath the heading (top-then-left). + gap_top = body["bbox"][1] - b[3] + # SAME-COLUMN only: the dateline that promotes this heading must sit in a body + # that lies under the heading's OWN column — the heading must cover at least + # half of that body's width — not merely clip a neighbouring column. + def _under_heading(xb): + ov = max(0, min(b[2], xb[2]) - max(b[0], xb[0])) + return ov >= 0.5 * max(1, xb[2] - xb[0]) + row = sorted([x for x in below + if x["bbox"][1] - b[3] <= gap_top + 120 and _under_heading(x["bbox"])], + key=lambda x: (x["bbox"][1], x["bbox"][0])) + + def _dateline_in_body_top(bbox): + # The dateline that OPENS the article may sit a line or two BELOW a merged + # photo caption / sub-headline inside the same body region — so scan the + # first several lines, not just the top one. + l, t, rr, bo = bbox + sh = min(360, bo - t) + if sh <= 0: + return None + try: + txt = pytesseract.image_to_string(page.crop((l, t, rr, t + sh)), lang="tel") + except Exception: + return None + lines = [x for x in txt.splitlines() if x.strip()] + for i in range(min(len(lines), 6)): + d = _dateline_in_text("\n".join(lines[i:i + 3]), paper) + if d: + return d + return None + + dl = None + for x in row[:4]: + dl = _dateline_in_body_top(x["bbox"]) + if dl: + break + if dl: # condition 2: dateline directly below + reg["type"] = "doc_title" + reg["_promoted_from"] = "paragraph_title" + reg["_dateline"] = dl + promoted += 1 + print(f" ⬆ PROMOTED region {reg['id']}: paragraph_title → doc_title " + f"[paper={paper}] (W={w} spans column [body {body['id']} W={body_w}]; " + f"dateline below in region {body['id']}: {dl!r})") + else: + print(f" · not promoted: region {reg['id']} (W={w} spans column) — " + f"[paper={paper}] no dateline under it (body region {body['id']})") + if promoted: + print(f" → headline promotion: {promoted} paragraph_title(s) reclassified as doc_title") + return regions + + +def detect_sakshi_headline_bands(regions, page_img_path): + """SAKSHI ONLY: pixel-level recovery of display headlines PaddleOCR never boxed. + + Sakshi sets very large display headlines that the layout model frequently + misses outright (JG pg1: the lift-1 and SIR-mapping banners have NO region at + all) or boxes only partially (the 'అనూహ్య స్పందన' headline is missing its red + kicker half). Sakshi also has no ruled grid, so a missing headline makes the + whitespace clusterer fold that article's photo/body into the story above or + below it — the single biggest source of bad Sakshi crops. Two pixel passes: + + 1. WIDEN — a doc_title grows sideways over adjacent large-ink columns that + no other region covers (the un-boxed kicker half of its own headline). + Widest titles first, and each extension joins the coverage map, so a + narrow neighbour can never steal a wide headline's kicker. + 2. SYNTHESIZE — a band of big-type ink covered by NO region, at least a + column wide and 55-280px tall, with article content following below it, + becomes a synthetic doc_title anchor so the clusterer splits there. + + Bands above the first real headline are never synthesized (that strip is the + masthead/weather furniture, already dropped). Returns the updated regions. + """ + import numpy as np + try: + img = Image.open(page_img_path).convert("L") + except Exception: + return regions + W, H = img.size + g = np.asarray(img) + ink = g < 160 + cov = np.zeros((H, W), dtype=bool) + for r in regions: + b = r["bbox"] + cov[max(0, b[1] - 6):min(H, b[3] + 6), max(0, b[0] - 6):min(W, b[2] + 6)] = True + + _tw = sorted(r["bbox"][2] - r["bbox"][0] for r in regions if r.get("type") == "text") + col_w = _tw[len(_tw) // 2] if _tw else 600 + + # ---- pass 1: WIDEN partially-boxed doc_titles over uncovered kicker ink ---- + titles = sorted([r for r in regions if r.get("type") == "doc_title"], + key=lambda r: -(r["bbox"][2] - r["bbox"][0])) + for t in titles: + b = t["bbox"] + y1, y2 = max(0, b[1]), min(H, b[3]) + bh = max(1, y2 - y1) + band_ink = ink[y1:y2, :] + # columns covered by OTHER regions in this y-band + ocov = np.zeros(W, dtype=bool) + for r in regions: + if r is t: + continue + rb = r["bbox"] + if rb[1] < y2 and rb[3] > y1: + ocov[max(0, rb[0] - 6):min(W, rb[2] + 6)] = True + inkcol = band_ink.sum(axis=0) > 0.12 * bh + new_x = {} + for side, rng in (("L", range(b[0] - 1, -1, -1)), ("R", range(b[2] + 1, W))): + gap, inky, xstop = 0, [], None + for x in rng: + if ocov[x]: + xstop = x + break + if inkcol[x]: + inky.append(x); gap = 0 + else: + gap += 1 + if gap > 110: + break + # Ink within 25px of a foreign region's edge is that neighbour's own glyph + # overflow (PaddleOCR boxes are tight) — not kicker ink. Drop it so the + # widened headline never laps into the next column's margin. + if xstop is not None: + inky = [x for x in inky + if ((x >= xstop + 25) if side == "L" else (x <= xstop - 25))] + reach = (min(inky) if side == "L" else max(inky)) if inky else None + if reach is not None and abs(reach - (b[0] if side == "L" else b[2])) >= 60: + new_x[side] = reach + if new_x: + ob = list(b) + if "L" in new_x: + t["bbox"][0] = max(0, new_x["L"] - 8) + if "R" in new_x: + t["bbox"][2] = min(W, new_x["R"] + 8) + t["_widened"] = True + cov[max(0, b[1] - 6):min(H, b[3] + 6), + max(0, t["bbox"][0] - 6):min(W, t["bbox"][2] + 6)] = True + print(f" 📐 Sakshi headline WIDEN: R{t['id']} {ob} → {t['bbox']} " + f"(un-boxed kicker ink alongside)") + + # ---- pass 2: SYNTHESIZE doc_titles for completely un-boxed headline bands ---- + free_ink = ink & ~cov + title_tops = [r["bbox"][1] for r in regions + if r.get("type") in ("doc_title", "paragraph_title")] + floor_y = (min(title_tops) - 40) if title_tops else 0 + rowink = free_ink.sum(axis=1) + hot = rowink > 320 + bands = [] + yy = 0 + while yy < H: + if hot[yy]: + y0 = yy + gap = 0 + while yy < H and gap <= 18: + gap = 0 if hot[yy] else gap + 1 + yy += 1 + bands.append((y0, yy - gap)) + else: + yy += 1 + # Text-column LEFT edges (clustered): two different articles' headlines can share + # pixel rows with only a ~50px gutter between them — the same size as a word gap — + # so a gap alone cannot split them. A gap that CONTAINS a known column-left IS the + # gutter between two columns of text → split there; a word gap inside one headline + # never coincides with a column start → bridge it. + _clefts = [] + for r in regions: + if r.get("type") != "text": + continue + x0 = r["bbox"][0] + for c in _clefts: + if abs(c[0] / c[1] - x0) <= 25: + c[0] += x0; c[1] += 1 + break + else: + _clefts.append([x0, 1]) + col_lefts = [c[0] / c[1] for c in _clefts if c[1] >= 2] + next_id = max((r["id"] for r in regions), default=0) + 1 + new_regs = [] + for (by1, by2) in bands: + if by1 < floor_y: + continue + colhot = free_ink[by1:by2, :].sum(axis=0) > 0.25 * (by2 - by1) + # split the row band into independent column runs (two headlines can share rows) + x = 0 + runs = [] + seg = None + gapstart = None + while x <= W: + on = colhot[x] if x < W else False + if on: + if seg is None: + seg = [x, x] + elif gapstart is not None: + gw = x - gapstart + crosses_col = any(gapstart - 5 <= c <= x + 25 for c in col_lefts) + if gw > 150 or (gw >= 35 and crosses_col): + runs.append((seg[0], seg[1] + 1)) + seg = [x, x] + else: + seg[1] = x + gapstart = None + seg[1] = x + else: + if seg is not None and gapstart is None: + gapstart = x + if seg is not None and gapstart is not None and x - gapstart > 150: + runs.append((seg[0], seg[1] + 1)) + seg, gapstart = None, None + x += 1 + if seg is not None: + runs.append((seg[0], seg[1] + 1)) + for (rx1, rx2) in runs: + if (rx2 - rx1) < 0.75 * col_w: + continue + sub = free_ink[by1:by2, rx1:rx2] + rows_any = np.where(sub.sum(axis=1) > 40)[0] + if rows_any.size == 0: + continue + sy1, sy2 = by1 + int(rows_any.min()), by1 + int(rows_any.max()) + 1 + if not (55 <= (sy2 - sy1) <= 400): + continue # 400 admits tall display/script faces (NT 'నమ్మిండు..') + # while still rejecting photos (their frac fails too) + frac = free_ink[sy1:sy2, rx1:rx2].mean() + if not (0.05 <= frac <= 0.45): + continue # too sparse = stray marks; too dense = a filled + # box / photo (the గమనిక notice header is 0.51; + # real display headlines measure 0.30-0.35) + if not any(r.get("type") in ("text", "paragraph_title") + and sy2 <= r["bbox"][1] <= sy2 + 500 + and min(rx2, r["bbox"][2]) - max(rx1, r["bbox"][0]) + >= 0.3 * (r["bbox"][2] - r["bbox"][0]) + for r in regions): + continue # no article content below → not a headline + nb = [max(0, rx1 - 6), max(0, sy1 - 6), min(W, rx2 + 6), min(H, sy2 + 6)] + new_regs.append({"id": next_id, "type": "doc_title", "bbox": nb, + "confidence": 0.50, "_synth_band": True}) + print(f" 📐 Sakshi headline SYNTH: R{next_id} doc_title {nb} " + f"(un-boxed display headline band, ink={frac:.2f})") + next_id += 1 + return regions + new_regs + + +def mark_bullet_titles(regions, page_img_path): + """Flag title regions that VISUALLY begin with a bullet glyph (●/•). A bullet is a + deck / sub-point, NEVER a real headline — so the clusterer can demote it when a real + headline sits above it. Detected by pixels (not OCR, which mangles '●' into a stray + 'అ'): in the title's first-line strip, look for a narrow, vertically-filled dark blob + at the very left (the disk) followed by a whitespace gap before the text begins.""" + try: + page = Image.open(page_img_path).convert("L") + except Exception: + return regions + px = page.load() + W, H = page.size + for r in regions: + if r.get("type") not in ("doc_title", "paragraph_title"): + continue + x1, y1, x2, y2 = r["bbox"] + h = min(y2 - y1, 90) + sx2 = min(x2, x1 + 140) + if h <= 0 or sx2 - x1 < 20: + continue + cols = [sum(1 for y in range(y1, y1 + h) if 0 <= y < H and px[min(x, W - 1), y] < 110) + for x in range(x1, sx2)] + i = 0 + while i < len(cols) and cols[i] < 2: # skip leading blank margin + i += 1 + j = i + while j < len(cols) and cols[j] >= 2: # first ink run (the bullet disk) + j += 1 + k = j + while k < len(cols) and cols[k] < 2: # whitespace gap after it + k += 1 + run_w, gap = j - i, k - j + fill = (max(cols[i:j]) / h) if j > i else 0 + # bullet disk: a narrow run (8–55px), well filled vertically (≥0.5), a clear gap + if 8 <= run_w <= 55 and gap >= 8 and fill >= 0.5: + r["_bullet"] = True + n = sum(1 for r in regions if r.get("_bullet")) + if n: + print(f" • bullet titles flagged: {n} (deck/sub-point, never a headline)") + return regions + + +def find_article_starts_by_dateline(regions, page_img_path, paper): + """CLUE scan (Tesseract, no Claude): OCR the first line of every text region; + any that OPENS with a dateline (', మే 19:' etc.) is the START of an + article — even if PaddleOCR never detected that article's headline. This is a + diagnostic cross-check: the number of dateline starts should match the number + of articles, exposing headlines the layout model missed.""" + try: + import pytesseract + if "tel" not in set(pytesseract.get_languages(config="")): + print(" (dateline clue scan skipped — Telugu Tesseract 'tel' not installed)") + return [] + except Exception: + print(" (dateline clue scan skipped — pytesseract not available)") + return [] + + page = Image.open(page_img_path) + starts = [] + for r in regions: + if r.get("type") != "text": + continue + b = r["bbox"] + sh = min(220, b[3] - b[1]) # tall enough for a 3-line wrap in narrow columns + if sh <= 0: + continue + try: + txt = pytesseract.image_to_string(page.crop((b[0], b[1], b[2], b[1] + sh)), lang="tel") + except Exception: + txt = "" + dl = _dateline_in_text(txt, paper) + if dl: + starts.append({"region_id": r["id"], "dateline": dl, "bbox": b}) + + n = len(starts) + print(f" 📰 DATELINE CLUE [paper={paper}]: {n} dateline(s) by the ', date:' pattern " + f"→ MINIMUM {n} article(s) on this page (actual may be more; never fewer).") + for s in sorted(starts, key=lambda z: z["bbox"][1]): + print(f" body R{s['region_id']} opens an article — dateline {s['dateline']!r}") + + # --- DETECTED vs MISSED: does each dateline article have a detected headline? --- + start_bboxes = [s["bbox"] for s in starts] + + def headline_above(body, max_gap=1000): + cx = (body[0] + body[2]) / 2 + cands = [r for r in regions + if (r.get("type") == "doc_title" or r.get("_promoted_from")) + and r["bbox"][0] <= cx <= r["bbox"][2] + and r["bbox"][3] <= body[1] + 40 + and (body[1] - r["bbox"][3]) <= max_gap] # must be reasonably close above + if not cands: + return None + h = max(cands, key=lambda r: r["bbox"][3]) # nearest headline above + for ob in start_bboxes: # another article's dateline between? + if ob is body: + continue + ocx = (ob[0] + ob[2]) / 2 + if h["bbox"][3] < ob[1] < body[1] and cx - 120 <= ocx <= cx + 120: + return None + return h["id"] + + detected, missed = [], [] + for s in starts: + h = headline_above(s["bbox"]) + s["headline_region"] = h # doc_title id above it, or None if missed + (detected if h else missed).append((s, h)) + + n_dt = sum(1 for r in regions if r.get("type") == "doc_title") + print(f" 🔎 HEADLINE COVERAGE: {len(detected)}/{n} articles have a detected headline; " + f"{len(missed)} MISSED by PaddleOCR (doc_title count={n_dt}).") + if missed: + print(f" ⚠ {len(missed)} article(s) Claude must catch by EYE (no headline box) — these would " + f"be lost if we only used doc_titles:") + for s, _ in sorted(missed, key=lambda z: z[0]["bbox"][1]): + print(f" body R{s['region_id']} {s['dateline']!r} ← headline not detected") + return starts + + +def draw_titles_debug(page_img_path, regions, pages_dir, page_num): + """Save a per-page image highlighting every headline-type region: + blue = doc_title, orange = paragraph_title, purple = figure_title, + green = promoted headline. Each box is labelled R W.""" + try: + from PIL import ImageDraw, ImageFont + except Exception as e: + print(f" (titles debug image skipped: {e})") + return + try: + font = ImageFont.truetype("DejaVuSans-Bold.ttf", 30) + except Exception: + try: + font = ImageFont.load_default(size=30) + except Exception: + font = ImageFont.load_default() + + img = Image.open(page_img_path).convert("RGB") + draw = ImageDraw.Draw(img) + n_doc = n_pt = n_fig = n_promo = 0 + for r in regions: + t = r.get("type") + if t not in ("doc_title", "paragraph_title", "figure_title"): + continue + b = r["bbox"] + w = b[2] - b[0] + if r.get("_promoted_from"): + color, lw, tag = (0, 160, 0), 10, f"R{r['id']} PROMOTED->doc_title W{w}" + n_promo += 1 + elif t == "doc_title": + color, lw, tag = (0, 90, 220), 9, f"R{r['id']} doc_title W{w}" + n_doc += 1 + elif t == "paragraph_title": + color, lw, tag = (230, 120, 0), 5, f"R{r['id']} paragraph_title W{w}" + n_pt += 1 + else: # figure_title + color, lw, tag = (150, 50, 200), 5, f"R{r['id']} figure_title W{w}" + n_fig += 1 + draw.rectangle([b[0], b[1], b[2], b[3]], outline=color, width=lw) + ty = max(0, b[1] - 34) + draw.rectangle([b[0], ty, b[0] + len(tag) * 16 + 12, ty + 32], fill=color) + draw.text((b[0] + 6, ty + 4), tag, fill="white", font=font) + + out = str(pages_dir / f"page_{page_num:03d}_titles_debug.png") + img.save(out) + print(f" TITLES DEBUG IMAGE saved: {out} " + f"(doc_title={n_doc}, paragraph_title={n_pt}, figure_title={n_fig}, promoted={n_promo})") + + +# distinct color per PaddleOCR region type (RGB) +_TYPE_COLORS = { + "doc_title": (0, 90, 220), # blue + "paragraph_title": (230, 120, 0), # orange + "figure_title": (150, 50, 200), # purple + "text": (110, 120, 135), # grey + "image": (20, 160, 90), # teal-green + "figure": (20, 160, 90), # teal-green + "table": (200, 40, 40), # red + "header": (210, 160, 0), # gold + "footer": (140, 90, 40), # brown + "number": (90, 90, 90), # dark grey +} + + +def draw_all_regions_debug(page_img_path, regions, pages_dir, page_num): + """Save a per-page image highlighting EVERY region from PaddleOCR, color-coded + by type and labelled R . A complete map of what was detected.""" + try: + from PIL import ImageDraw, ImageFont + except Exception as e: + print(f" (all-regions debug image skipped: {e})") + return + try: + font = ImageFont.truetype("DejaVuSans-Bold.ttf", 26) + except Exception: + try: + font = ImageFont.load_default(size=26) + except Exception: + font = ImageFont.load_default() + + from collections import Counter + img = Image.open(page_img_path).convert("RGB") + draw = ImageDraw.Draw(img) + for r in regions: + t = r.get("type", "unknown") + b = r["bbox"] + color = _TYPE_COLORS.get(t, (90, 90, 90)) + # promoted headlines stand out in green regardless of base type + if r.get("_promoted_from"): + color = (0, 160, 0) + lw = 7 if t in ("doc_title", "paragraph_title", "figure_title") else 4 + tag = f"R{r['id']} {t}" + draw.rectangle([b[0], b[1], b[2], b[3]], outline=color, width=lw) + ty = max(0, b[1] - 30) + draw.rectangle([b[0], ty, b[0] + len(tag) * 14 + 10, ty + 28], fill=color) + draw.text((b[0] + 5, ty + 3), tag, fill="white", font=font) + + out = str(pages_dir / f"page_{page_num:03d}_allregions_debug.png") + img.save(out) + counts = Counter(r.get("type", "unknown") for r in regions) + summary = ", ".join(f"{k}={v}" for k, v in sorted(counts.items(), key=lambda kv: -kv[1])) + print(f" ALL-REGIONS DEBUG IMAGE saved: {out} ({len(regions)} regions: {summary})") + + +def _wall_between_datelines(upper_box, lower_box, doc_titles, h_lines): + """Is there a real article WALL — a headline (doc_title) or a horizontal + separator line — in the vertical gap between two datelines (upper above lower), + within their shared column? A wall means the lower dateline starts a genuinely + separate article; no wall means continuous body text (one roundup story).""" + top, bot = upper_box[3], lower_box[1] # bottom of upper dl → top of lower dl + if bot <= top: + return False + cx0, cx1 = max(upper_box[0], lower_box[0]), min(upper_box[2], lower_box[2]) + cw = cx1 - cx0 + if cw <= 0: # no column overlap → use lower column + cx0, cx1 = lower_box[0], lower_box[2] + cw = cx1 - cx0 + for rb in doc_titles: # a headline in the gap + if rb[1] >= top - 10 and rb[3] <= bot + 10 \ + and max(0, min(cx1, rb[2]) - max(cx0, rb[0])) >= 0.3 * max(1, cw): + return True + for L in h_lines: # a separator line crossing the gap + if top <= L["y"] <= bot \ + and max(0, min(cx1, L["x2"]) - max(cx0, L["x1"])) >= 0.3 * max(1, cw): + return True + return False + + +def split_blocks_on_datelines(blocks, regions, dateline_starts, sep_lines=None): + """Apply the one-dateline-per-crop rule to the geometric article blocks (the + source of all_article_crops), using the SAME wall rule as the political-article + splitter: a block holding 2+ datelines is split at every same-column dateline + that has a real WALL (headline / separator line) above it. Datelines in a + different column always split; same-column datelines with only body text + between them (a roundup) stay in one block.""" + if not dateline_starts: + return blocks + region_map = {r["id"]: r for r in regions} + dl_ids = {s["region_id"] for s in dateline_starts} + doc_titles = [r["bbox"] for r in regions if r.get("type") == "doc_title"] + h_lines = [L for L in (sep_lines or []) if not L.get("vert") and "y" in L] + # A BANNER (a doc_title ≥1.6 column-widths) heads ONE article even when its body + # carries several mandal datelines (a roundup). Compute the column pitch so we can + # recognise such banners and keep their sub-datelines together. + _tw = sorted((r["bbox"][2] - r["bbox"][0]) for r in regions if r.get("type") == "text") + _pitch = _tw[len(_tw) // 2] if _tw else 0 + + out = [] + for blk in blocks: + members = blk.get("members", []) + dls = [rid for rid in members if rid in dl_ids and rid in region_map] + if len(dls) <= 1: + out.append(blk) + continue + + # Banner-roundup exception: if this block's OWN banner headline (a doc_title + # ≥1.6 column-widths, member of the block) sits ABOVE all its datelines and + # spans their columns (each dateline's centre lies under the banner), the whole + # thing is ONE roundup story under one headline — keep it together, do NOT split. + dl_cx = [((region_map[d]["bbox"][0] + region_map[d]["bbox"][2]) / 2) for d in dls] + dl_top = min(region_map[d]["bbox"][1] for d in dls) + block_banners = [region_map[m]["bbox"] for m in members + if region_map.get(m, {}).get("type") == "doc_title" + and _pitch and (region_map[m]["bbox"][2] - region_map[m]["bbox"][0]) >= 1.6 * _pitch] + if any(bt[3] <= dl_top + 40 and all(bt[0] - 30 <= cx <= bt[2] + 30 for cx in dl_cx) + for bt in block_banners): + out.append(blk) + continue + + dl_boxes = [(rid, region_map[rid]["bbox"]) for rid in dls] + + def owner(rid): + if rid not in region_map: + return dls[0] + b = region_map[rid]["bbox"] + cx = (b[0] + b[2]) / 2 + in_col = [(d, db) for d, db in dl_boxes if db[0] - 40 <= cx <= db[2] + 40] + if region_map[rid].get("type") == "doc_title": # headline owns dl below it + below = [(d, db) for d, db in in_col if db[1] >= b[3] - 30] + if below: + return min(below, key=lambda z: z[1][1])[0] + above = [(d, db) for d, db in in_col if db[1] <= b[1] + 30] # body → dl above + if above: + return max(above, key=lambda z: z[1][1])[0] + if in_col: + return min(in_col, key=lambda z: z[1][1])[0] + return min(dls, key=lambda d: region_map[d]["bbox"][1]) + + groups = {} + for rid in members: + groups.setdefault(owner(rid), []).append(rid) + if len(groups) <= 1: + out.append(blk) + continue + + # keeper = group holding the block's anchor (else the topmost dateline group) + keeper = blk["anchor_id"] if blk["anchor_id"] in groups \ + else (owner(blk["anchor_id"]) if blk["anchor_id"] in region_map + else min(dls, key=lambda d: region_map[d]["bbox"][1])) + if keeper not in groups: + keeper = max(groups, key=lambda k: len(groups[k])) + + keeper_ids = list(groups[keeper]) + split_groups = [] + for dl, ids in groups.items(): + if dl == keeper: + continue + db = region_map[dl]["bbox"] + cx = (db[0] + db[2]) / 2 + above = [region_map[d]["bbox"] for d in dls + if d != dl and region_map[d]["bbox"][1] < db[1] + and region_map[d]["bbox"][0] - 40 <= cx <= region_map[d]["bbox"][2] + 40] + upper = max(above, key=lambda b: b[1]) if above else None + # Same-column dateline above → wall decides. None above → different column + # (side-by-side) → always its own block. + split = _wall_between_datelines(upper, db, doc_titles, h_lines) \ + if upper is not None else True + (split_groups.append((dl, ids)) if split else keeper_ids.extend(ids)) + + if not split_groups: + out.append(blk) + continue + + def _bbox_of(ids): + bxs = [region_map[i]["bbox"] for i in ids if i in region_map] + return [min(b[0] for b in bxs), min(b[1] for b in bxs), + max(b[2] for b in bxs), max(b[3] for b in bxs)] + + print(f" ✂ Block dateline split: block R{blk['anchor_id']} held {len(dls)} " + f"datelines → 1 + {len(split_groups)} (headline/separator-line boundary)") + out.append({**blk, "members": sorted(keeper_ids), "bbox": _bbox_of(keeper_ids)}) + for dl, ids in sorted(split_groups, key=lambda kv: region_map[kv[0]]["bbox"][1]): + out.append({"anchor_id": dl, "kind": "DL", "bbox": _bbox_of(ids), + "members": sorted(ids)}) + return out + + +# Paper currently being processed (set by process_pdf / the replay scripts before clustering). +# Gates paper-specific clustering passes — e.g. SAKSHI (a no-ruled-grid, headline-light paper) +# needs aggressive horizontal-rule splitting that would over-split the AJ/NT boxed grids. +_ACTIVE_PAPER = None + + +def cluster_all_article_blocks(regions, dateline_starts=None, sep_lines=None): + """Reconstruct EVERY candidate article block on a page (political and not), + purely geometrically, BEFORE any Stage-3 filtering. Shared by the all-articles + debug overlay and the per-article crop dump so both stay in sync. + + Each headline (doc_title or promoted) and each headline-less dateline start is + an article 'anchor'; a paragraph_title that sits at the very top of its column + (no headline above it in that column) is promoted to an anchor too. Every other + region is folded into the nearest anchor above it in the same column (else the + nearest below, else the globally nearest). Page furniture (header/number) is + dropped. + + Returns a list of blocks ordered top-to-bottom, each a dict: + {"anchor_id", "kind" ('H'|'DL'|'P'), "bbox" [x1,y1,x2,y2], "members" [ids]} + """ + if not regions: + return [] + + # Drop printer colour-registration bars: short image strips that sit BELOW all + # article text at the page bottom (CMYK dots, etc.) — page furniture, never an + # article. No real photo sits beneath all body text without a caption. + _txt_bottom = max((r["bbox"][3] for r in regions if r.get("type") == "text"), default=0) + if _txt_bottom > 0: + regions = [r for r in regions if not ( + r.get("type") == "image" + and r["bbox"][1] >= _txt_bottom - 5 + and (r["bbox"][3] - r["bbox"][1]) < 150)] + + def _cx(b): + return (b[0] + b[2]) / 2 + + def _col_overlap(ab, b): + return (ab[0] <= _cx(b) <= ab[2]) or (b[0] <= _cx(ab) <= b[2]) + + sep = sep_lines or [] + vsep = [L for L in sep if L.get("vert")] # vertical column-divider rules + sep = [L for L in sep if not L.get("vert")] # horizontal article separators + + # A printed rule whose midpoint lands well inside a photo box is the photo's own + # internal/border rule, NOT an article separator. PaddleOCR inflates photo boxes, + # so such a rule can sit inside the photo of the SAME article. Body text still + # respects every rule, but a photo/caption is allowed to fold up to its article + # across one of these in-photo rules (see the fold loop's `ignore_inphoto`). + _img_boxes = [r["bbox"] for r in regions if r.get("type") == "image"] + + def _in_photo(L, margin=20): + mx = (L["x1"] + L["x2"]) / 2 + for ix1, iy1, ix2, iy2 in _img_boxes: + if ix1 + margin <= mx <= ix2 - margin and iy1 + margin <= L["y"] <= iy2 - margin: + return True + # SAKSHI: a rule hugging an image's BOTTOM edge (within 14px below it, + # horizontally inside the photo span ±25px) is the photo's own frame / + # caption rule, never a story separator. PaddleOCR's photo box ends a few + # px above the printed border, so the interior test alone misses it — and + # the stray border then fences the article's own caption and column body + # off their anchor (JG pg1 lift-1: R61/R46/R23 fell into the story below). + if (_ACTIVE_PAPER == "sakshi" + and iy2 - 6 <= L["y"] <= iy2 + 14 + and L["x1"] >= ix1 - 25 and L["x2"] <= ix2 + 25): + return True + return False + + sep_real = [L for L in sep if not _in_photo(L)] # rules that aren't inside a photo + + def _is_box_edge(L): + """SAKSHI ONLY: a horizontal rule with a PARALLEL partner rule 80-400px away + (≥70% x-overlap) is the top/bottom EDGE of a small boxed insert (quote box / + bullet box) INSIDE an article — never a story separator. Without this, the + orange bullet-box borders under a Sakshi headline fence the article's own + column bodies off their anchor (JG pg1 lift-1: R46/R23/R61 fell into the + story below). Not applied to the ruled papers, where two stacked stories + can legitimately sit between parallel rules this close.""" + if _ACTIVE_PAPER != "sakshi": + return False + w = max(1, L["x2"] - L["x1"]) + for M in sep: + if M is L or "y" not in M: + continue + if not (80 <= abs(M["y"] - L["y"]) <= 400): + continue + ov = min(L["x2"], M["x2"]) - max(L["x1"], M["x1"]) + if ov >= 0.7 * min(w, max(1, M["x2"] - M["x1"])): + return True + return False + + def _barrier_between(top_box, bot_box, lines=None): + """True if a separator rule cuts between an upper box and a lower box — i.e. + they're different articles and must not be folded/grown together. The rule + must sit below the upper box's bottom and no deeper than the lower box's + vertical midpoint (OCR inflates photo boxes UP past the rule, so we can't + require the rule to clear the lower box's top), and overlap the lower box's + column (line fragments are short, so any real overlap counts). + + Default line set is `sep_real` (rules NOT inside a photo): a rule whose + midpoint lands inside an image box is the photo's own internal/border mark, + never an article separator, so it must not fence anything — body text + included. Pass an explicit list to override.""" + bx1, by1, bx2, by2 = bot_box + bmid = (by1 + by2) / 2 + for L in (sep_real if lines is None else lines): + if top_box[3] <= L["y"] <= bmid: + ov = max(0, min(bx2, L["x2"]) - max(bx1, L["x1"])) + if ov >= 0.25 * min(bx2 - bx1, L["x2"] - L["x1"]) and not _is_box_edge(L): + return True + return False + + dl_starts = dateline_starts or [] + missed_ids = {s["region_id"] for s in dl_starts if not s.get("headline_region")} + + # A narrow doc_title sitting directly beneath a WIDER doc_title (no body text + # between them) is that headline's kicker/deck — not a separate article. It must + # NOT become its own anchor: otherwise a multi-column-header article gets split, + # and a dateline under it binds to the kicker instead of the real (wide) header. + # Demoting the kicker lets it fold into the wide header so the article stays whole + # and grows across all of the header's columns. + _doc_titles = [r for r in regions if r.get("type") == "doc_title"] + + def _is_kicker(k): + kb = k["bbox"] + kw = kb[2] - kb[0] + for w in _doc_titles: + if w["id"] == k["id"]: + continue + wb = w["bbox"] + if (wb[2] - wb[0]) < kw * 1.4: + continue # not meaningfully wider + if not (wb[0] - 30 <= kb[0] and kb[2] <= wb[2] + 30): + continue # wide header must span kicker + if not (0 <= kb[1] - wb[3] <= 130): + continue # kicker sits directly below + if any(t.get("type") == "text" + and wb[3] <= t["bbox"][1] and t["bbox"][3] <= kb[1] + 5 + and _col_overlap(kb, t["bbox"]) for t in regions): + continue # body text between → separate + return True + return False + + kicker_ids = {r["id"] for r in _doc_titles if _is_kicker(r)} + + # A doc_title that VISUALLY starts with a bullet (flagged by mark_bullet_titles) is a + # deck/sub-point, never a headline. If a real doc_title sits ABOVE it in the same + # column, demote the bullet so it folds into that article instead of anchoring its + # own block (fixes a banner article being split by its bulleted sub-head). Unlike + # the kicker rule, this holds even when a photo/body sits between them. + bullet_demoted = { + r["id"] for r in _doc_titles + if r.get("_bullet") and any( + o["id"] != r["id"] and o.get("type") == "doc_title" and not o.get("_bullet") + and _col_overlap(o["bbox"], r["bbox"]) and o["bbox"][1] < r["bbox"][1] + for o in _doc_titles)} + + def _is_base(r): + return ((r.get("type") == "doc_title" or r.get("_promoted_from") + or r["id"] in missed_ids) + and r["id"] not in kicker_ids and r["id"] not in bullet_demoted) + + base_anchors = [r for r in regions if _is_base(r)] + base_anchor_ids = {a["id"] for a in base_anchors} + + # --- Grow-up recovery for headless ('missed') datelines -------------------- + # An article whose bold headline PaddleOCR did NOT tag as doc_title has that + # headline (plus any bullet sub-title, caption, and photo) sitting ABOVE its + # dateline body. Walk UP from each such dateline and claim the contiguous run + # of title-type + image boxes that overlap the dateline's column — allowing + # boxes WIDER than the column (multi-column banner headlines). Stop at the + # first body 'text' box, another headline (doc_title/promoted), or another + # dateline. Those boxes attach DOWN into this dateline's block instead of + # folding into the article above, so the crop becomes complete. + def _hoverlap_frac(a, b): + ix = max(0, min(a[2], b[2]) - max(a[0], b[0])) + return ix / max(1, min(a[2] - a[0], b[2] - b[0])) + + dl_body_ids = {s["region_id"] for s in dl_starts} + grow_claims = {} # region_id -> dateline-anchor id that claimed it upward + for s in dl_starts: + if s.get("headline_region"): + continue # already has a headline + dl_id, dl_box = s["region_id"], s["bbox"] + cands = [r for r in regions + if r["id"] != dl_id + and r["bbox"][3] <= dl_box[1] + 20 # bottom at/above dateline top + and _hoverlap_frac(dl_box, r["bbox"]) >= 0.5] + cands.sort(key=lambda r: r["bbox"][1], reverse=True) # nearest above first + + def _is_caption(box): + """A short text line sitting directly under a photo is that photo's + caption (PaddleOCR often tags it 'text', not 'figure_title'). Treat it as + claimable so grow-up can pass THROUGH it to reach the photo above.""" + if box[3] - box[1] > 200: + return False # too tall → body paragraph + for o in regions: + if o.get("type") != "image": + continue + ob = o["bbox"] + if (ob[3] <= box[1] and (box[1] - ob[3]) <= 150 + and _hoverlap_frac(box, ob) >= 0.5): + return True + return False + + claimed_img = False # an article has one photo group + # Photo-unit detection: when the nearest content region directly above the + # dateline is a photo (or its caption), the separator just under it is the + # caption/photo UNDERLINE — part of THIS article, not an article boundary — + # so grow-up may cross it. Narrowly gated: if the nearest region above is + # ordinary body text, the separator is a real boundary and still stops grow-up. + cross_underline = False + for r in cands: + t0 = r.get("type") + if t0 in ("header", "number", "footer"): + continue + cross_underline = (t0 == "image" + or (t0 == "text" and _is_caption(r["bbox"]))) + break # inspect only the nearest region + for r in cands: + t = r.get("type") + if _barrier_between(r["bbox"], dl_box): + _photo_part = (t == "image" or t in ("paragraph_title", "figure_title") + or (t == "text" and _is_caption(r["bbox"]))) + if not (cross_underline and _photo_part): + break # real boundary → top of article + # else: photo/caption underline → cross it, keep claiming the photo unit + if t in ("header", "number", "footer"): + continue # skip furniture, keep going up + if (r["id"] in dl_body_ids or r["id"] in base_anchor_ids): + break # another article → stop + if t == "text": + if _is_caption(r["bbox"]): + grow_claims[r["id"]] = dl_id # photo caption → claim & go up + continue + break # body text → stop + if t == "image": + if claimed_img: + break # 2nd photo → next article's → stop + claimed_img = True + grow_claims[r["id"]] = dl_id + elif t == "paragraph_title": + grow_claims[r["id"]] = dl_id # the story's own bold heading + break # the heading IS the article's top — anything above it + # (the previous story's column body, its photos) is not + # ours (Sakshi pg2: R14's walk climbed past subhead R121 + # and stole R20, the భక్తులకు article's column body) + elif t == "figure_title": + grow_claims[r["id"]] = dl_id # photo caption → keep climbing + else: + break + + # --- Stacked-headline recovery ABOVE a detected headline anchor (SAME COLUMN) -- + # A wrapped headline (or a header + sub-heading stack) is split into several title + # boxes: the lowest becomes the doc_title anchor, the upper line(s) sit above it. + # They belong to the anchor's article, not the previous same-column story. Grow UP + # from each doc_title anchor claiming contiguous bold title-type boxes; stop at the + # first normal body text (the previous article's), a separator rule, or another + # anchor. NO gap-size rule: the gap INSIDE a headline can equal the gap to the + # previous article (both ~40px observed), so only row TYPE — bold title vs normal + # body text — reliably separates them. + _TITLE_UP = ("paragraph_title", "figure_title", "header") + headline_stack_claims = {} # region_id -> doc_title anchor id + for a in base_anchors: + if a.get("type") != "doc_title": + continue + ab = a["bbox"] + cur_top = ab[1] + while True: + ups = [r for r in regions + if r["id"] != a["id"] + and r["id"] not in headline_stack_claims + and r["id"] not in base_anchor_ids + and r["id"] not in grow_claims + and r.get("type") in _TITLE_UP + and r["bbox"][3] <= cur_top + 8 + and _hoverlap_frac(ab, r["bbox"]) >= 0.5] # same column only + if not ups: + break + r = max(ups, key=lambda r: r["bbox"][1]) # nearest title above + rb = r["bbox"] + # Normal body text between this title and the stack → previous article → stop. + if any(o.get("type") == "text" + and rb[3] <= o["bbox"][1] and o["bbox"][3] <= cur_top + 5 + and _hoverlap_frac(ab, o["bbox"]) >= 0.5 for o in regions): + break + if _barrier_between(rb, [ab[0], cur_top, ab[2], cur_top + 1]): + break + # A figure_title sitting directly UNDER an image is that photo's CAPTION, + # never a wrapped headline line — it ends the stack (Sakshi: the story + # above's photo caption must not be pulled into a synthesized headline). + if (r.get("type") == "figure_title" + and any(o.get("type") == "image" + and 0 <= rb[1] - o["bbox"][3] <= 90 + and (min(rb[2], o["bbox"][2]) - max(rb[0], o["bbox"][0])) + >= 0.4 * (rb[2] - rb[0]) + for o in regions)): + break + headline_stack_claims[r["id"]] = a["id"] + cur_top = rb[1] + + extra_ids = set() + extra_anchors = [] + for r in regions: + if _is_base(r) or r.get("type") != "paragraph_title": + continue + if r["id"] in grow_claims or r["id"] in headline_stack_claims: # claimed below/stack + continue + b = r["bbox"] + # A paragraph_title sitting DIRECTLY ABOVE a base doc_title (same column, + # tiny gap, no rule between) is the upper line of that two-line headline — + # not a new article start. Let it fold down into that doc_title's block. + head_below = any( + a.get("type") == "doc_title" and _col_overlap(a["bbox"], b) + and 0 <= a["bbox"][1] - b[3] <= 120 + and not _barrier_between(b, a["bbox"]) + for a in base_anchors) + if head_below: + continue + _above = [a for a in base_anchors + if _col_overlap(a["bbox"], b) and a["bbox"][1] <= b[1] + and not _barrier_between(a["bbox"], b)] + if not _above: + extra_ids.add(r["id"]) + extra_anchors.append(r) + elif _ACTIVE_PAPER == "sakshi": + # SAKSHI: a column-wide paragraph_title sitting FAR below the nearest + # detected headline in its column, with its own content directly + # underneath, is the HEADING of a new stacked story whose display type + # PaddleOCR mistyped (pg2 R125 'విమానయాన రంగంలో..' sat 1200px under the + # statue story's doc_title and the whole column merged into one + # page-tall crop). A deck/quote-box pt near its parent stays demoted. + _tw3 = sorted(q["bbox"][2] - q["bbox"][0] for q in regions + if q.get("type") == "text") + _colw3 = _tw3[len(_tw3) // 2] if _tw3 else 600 + _gap3 = b[1] - max(a["bbox"][3] for a in _above) + # 900px ≈ 1.4 columns of body: a feature package's own sub-head sits + # closer to its headline than that (pg1 R94 at 614px must stay folded); + # a genuinely separate stacked story sits further (pg2 R125 at 1201px). + if (_gap3 > 900 and (b[2] - b[0]) >= 0.55 * _colw3 + and any(q.get("type") in ("text", "image") + and 0 <= q["bbox"][1] - b[3] <= 150 + and (min(q["bbox"][2], b[2]) - max(q["bbox"][0], b[0])) + >= 0.4 * (b[2] - b[0]) + for q in regions)): + extra_ids.add(r["id"]) + extra_anchors.append(r) + print(f" ⚓ Sakshi stacked-story heading: paragraph_title R{r['id']} " + f"promoted to anchor ({_gap3}px below the headline above)") + + anchors = base_anchors + extra_anchors + if not anchors: + return [] + anchor_ids = {a["id"] for a in anchors} + + # Deck-band protection: whatever sits BELOW a headline and ABOVE that article's + # dateline (same column) is the article's deck — bullet/summary lines, a lead + # photo, a caption. It is PART OF THE ARTICLE and can never be split off as an + # orphan, even when a decorative bullet underline reads as a separator rule. + # Claim every region inside that header→dateline band for the headline. + _aby = {a["id"]: a for a in anchors} + band_bot_for = {} # headline id -> top y of its dateline body + for s in dl_starts: + h = s.get("headline_region") + if h in anchor_ids: + band_bot_for[h] = min(band_bot_for.get(h, s["bbox"][1]), s["bbox"][1]) + band_claims = {} # region_id -> headline anchor id + for h, band_bot in band_bot_for.items(): + hb = _aby[h]["bbox"] + band_top = hb[3] + if band_bot <= band_top: + continue + for r in regions: + if r["id"] in anchor_ids or r.get("type") in ("header", "number", "footer"): + continue + rb = r["bbox"] + if (rb[1] >= band_top - 10 and rb[3] <= band_bot + 10 + and _col_overlap(hb, rb)): + band_claims[r["id"]] = h + + # Faint top-boundary rules above multi-column headers (tagged "mc" by + # separator_barriers): anchor_id -> boundary y. A region whose box is wholly + # above this y must not fold into that multi-column anchor's block. + mc_bound = {L["anchor"]: L["y"] for L in sep + if L.get("mc") and L.get("anchor") in anchor_ids} + + def _mc_blocked(a, box): + y0 = mc_bound.get(a["id"]) + return y0 is not None and box[3] <= y0 + + clusters = {a["id"]: [(a["id"], a["bbox"])] for a in anchors} + orphans = [] + for r in regions: + if r["id"] in anchor_ids or r.get("type") in ("header", "number", "footer"): + continue + b = r["bbox"] + if r["id"] in band_claims and band_claims[r["id"]] in clusters: + clusters[band_claims[r["id"]]].append((r["id"], b)) # deck band → headline + continue + if r["id"] in grow_claims and grow_claims[r["id"]] in clusters: + clusters[grow_claims[r["id"]]].append((r["id"], b)) # grown upward + continue + if r["id"] in headline_stack_claims and headline_stack_claims[r["id"]] in clusters: + clusters[headline_stack_claims[r["id"]]].append((r["id"], b)) # upper headline line + continue + col = [a for a in anchors if _col_overlap(a["bbox"], b)] + # A separator rule blocks folding ACROSS it for body text. Two exceptions: + # (1) a dateline body must always stay with its own headline (a rule never + # sits between a headline and its dateline); (2) a photo or its caption may + # fold up to its article across an IN-PHOTO rule (the photo's own border), + # but still respects real gutter separators — so it uses `sep_real`. + is_dl_body = r["id"] in dl_body_ids + is_photo = r.get("type") in ("image", "figure_title") + bar_lines = sep_real if is_photo else None + # A multi-column header's faint top-boundary rule clamps ITS OWN block: a + # region lying entirely above that rule must not fold down into the + # multi-column article below it. This is scoped to the multi-col anchor only + # (via mc_bound), so ordinary body text elsewhere is unaffected. + above = [a for a in col if a["bbox"][1] <= b[1] + 40 + and (is_dl_body or not _barrier_between(a["bbox"], b, bar_lines)) + and not _mc_blocked(a, b)] + if above: + best = max(above, key=lambda a: a["bbox"][1]) + else: + # No anchor above in-column: fall through to the nearest anchor, prefer + # one whose column overlaps it. But a region fenced off from a candidate + # by a printed separator rule belongs to a DIFFERENT article — exclude + # those. A region cut off from EVERY candidate is a true orphan (e.g. a + # self-contained sub-article whose headline OCR mis-tagged): it must not + # be force-folded into a dateline story, so it goes to its own block. + cands = [a for a in anchors if not _mc_blocked(a, b)] or anchors + col_c = [a for a in cands if _col_overlap(a["bbox"], b)] + pool0 = col_c or cands + + def _fenced(a): + ab = a["bbox"] + return (_barrier_between(ab, b, bar_lines) if ab[1] <= b[1] + else _barrier_between(b, ab, bar_lines)) + + pool = [a for a in pool0 if not _fenced(a)] + if not pool: + orphans.append((r["id"], b)) + continue + best = min(pool, + key=lambda a: (_cx(a["bbox"]) - _cx(b)) ** 2 + + (a["bbox"][1] - b[1]) ** 2) + clusters[best["id"]].append((r["id"], b)) + + # Group the fenced-off regions by column-overlap + vertical adjacency. A group is + # only treated as a genuine ORPHAN ARTICLE (kept out of the dateline stories and + # emitted on its own) when it looks self-contained: it carries a title-type region + # AND has more than one member (e.g. a photo + mis-tagged headline + body whose + # OCR headline wasn't detected). A lone stray text/image fragment is NOT an article + # — it falls back to its nearest in-column anchor so dateline body isn't stripped. + rtype = {r["id"]: r.get("type") for r in regions} + orphans.sort(key=lambda z: z[1][1]) + groups = [] + for rid, ob in orphans: + placed = False + for grp in groups: + gb = grp["bbox"] + if _col_overlap(gb, ob) and -40 <= ob[1] - gb[3] <= 320: + grp["members"].append((rid, ob)) + grp["bbox"] = [min(gb[0], ob[0]), min(gb[1], ob[1]), + max(gb[2], ob[2]), max(gb[3], ob[3])] + placed = True + break + if not placed: + groups.append({"anchor_id": rid, "bbox": list(ob), + "members": [(rid, ob)]}) + + _title_t = ("figure_title", "paragraph_title", "doc_title") + + def _cluster_ubbox(aid): + ms = clusters[aid] + return [min(bb[0] for _, bb in ms), min(bb[1] for _, bb in ms), + max(bb[2] for _, bb in ms), max(bb[3] for _, bb in ms)] + + def _wide_rule_between(ab, gtop, frac=0.55): + """True if a WIDE article-boundary rule (spanning ≥frac of the header's + width) sits between the header bottom `ab[3]` and `gtop`. Narrow bullet / + caption underlines do not count — only a full-width rule that separates two + stacked stories. Used to stop folds from crossing a real article boundary.""" + y0 = ab[3] + if gtop <= y0: + return False + hw = ab[2] - ab[0] + for L in sep: + if L.get("vert") or "y" not in L: + continue + if y0 - 4 <= L["y"] <= gtop + 4: + ov = (min(ab[2], L.get("x2", 0)) - max(ab[0], L.get("x1", 0))) + if ov >= frac * hw: + return True + return False + + orphan_groups = [] + for grp in groups: + mem_ids = [m for m, _ in grp["members"]] + # A real (mis-tagged) article carries a headline-ish title AND body text. A + # bare photo+caption (title-type but no body text) is just a figure belonging + # to a neighbouring story, so it folds back rather than orphaning. + is_article = (any(rtype.get(m) in _title_t for m in mem_ids) + and any(rtype.get(m) == "text" for m in mem_ids)) + if is_article: + orphan_groups.append(grp) + continue + gb = grp["bbox"] + # A photo (+caption) sitting geometrically INSIDE an article's bounding box + # belongs to that article: a banner header spans the whole article width, so + # any image beneath it within that span — e.g. a 2nd photo in another column — + # is part of the story, not a separate crop. Fold it into the tightest + # containing cluster, overriding any decorative / in-photo rule between them — + # but NOT across a real article-boundary rule (a wide full-width rule that + # separates two stacked stories), otherwise a lower article's lead photo would + # be swallowed by the article above it. + if any(rtype.get(m) == "image" for m in mem_ids): + host, host_area = None, None + for a in anchors: + cb = _cluster_ubbox(a["id"]) + if (cb[0] - 15 <= gb[0] and gb[2] <= cb[2] + 15 + and cb[1] - 15 <= gb[1] and gb[3] <= cb[3] + 15 + and not _wide_rule_between(a["bbox"], gb[1])): + area = (cb[2] - cb[0]) * (cb[3] - cb[1]) + if host is None or area < host_area: + host, host_area = a["id"], area + if host is not None: + for rid, ob in grp["members"]: + clusters[host].append((rid, ob)) + continue + # --- In-column body belongs to its header (dateline-independent) --------- + # MOST IMPORTANT RULE: every paragraph / bullet / text line BELOW a header, + # in the header's columns, is that article's own body — it must NEVER become + # an orphan just because a decorative bullet / sub-heading underline reads as + # a separator rule. The ONLY thing that ends an article downward is a REAL + # article-boundary rule: a WIDE horizontal rule spanning most of the header's + # width. A narrow bullet underline does not. This does NOT depend on dateline + # detection (often absent), so it holds even with zero datelines. + # Scope to TEXT bodies (paragraphs / bullets). A group carrying an image is a + # self-contained block — a photo, an advertisement / greetings poster — not + # flowing article text, so it is NOT pulled up by this rule (inline article + # photos are handled separately by the containment fold above). + _text_body = (any(rtype.get(m) == "text" for m in mem_ids) + and not any(rtype.get(m) == "image" for m in mem_ids)) + bhost = None + if _text_body: + for a in sorted((x for x in anchors if x.get("type") == "doc_title"), + key=lambda x: -x["bbox"][3]): # nearest header above first + ab = a["bbox"] + if not _col_overlap(ab, gb) or gb[1] < ab[3] - 10: + continue # group must sit below this header + if _wide_rule_between(ab, gb[1]): + continue # real boundary rule → different story + bhost = a["id"] + break + if bhost is not None: + for rid, ob in grp["members"]: + clusters[bhost].append((rid, ob)) + continue + # Not a clearly self-contained article. Try to fold the group back into a + # neighbouring story — but a width-matched grey rule BOUNDS an article, so the + # fold-back must respect it: only anchors NOT fenced from the group are valid + # targets. If the group is fenced from EVERY in-column anchor (e.g. a promo / + # teaser box below a story's full-width bottom rule, whose bold heading OCR + # merged into plain text), it is a separate 'other' article → emit on its own. + gcands = [a for a in anchors + if not (mc_bound.get(a["id"]) is not None and gb[3] <= mc_bound[a["id"]])] + gcands = gcands or anchors + gcol = [a for a in gcands if _col_overlap(a["bbox"], gb)] or gcands + + def _grp_fenced(a): + ab = a["bbox"] + return (_barrier_between(ab, gb) if ab[1] <= gb[1] + else _barrier_between(gb, ab)) + + unfenced = [a for a in gcol if not _grp_fenced(a)] + if not unfenced: + orphan_groups.append(grp) # fenced from all → separate article + continue + for rid, ob in grp["members"]: # fold each member to nearest unfenced + best = min(unfenced, key=lambda a: (_cx(a["bbox"]) - _cx(ob)) ** 2 + + (a["bbox"][1] - ob[1]) ** 2) + clusters[best["id"]].append((rid, ob)) + + # --- Grow-right column walk for multi-column (banner) headers --------------- + # A banner headline only sets the article's MINIMUM width — the body can run + # into further columns to the right (overflow text, a photo) that the header + # does not span. Starting from a multi-column header block, walk RIGHTWARD one + # column-pitch at a time and absorb the next column when it is a genuine + # continuation: it carries NO dateline/headline of its own anywhere in the + # column, is not fenced by a printed vertical rule, is ~one column wide + # (±10%, photos exempt), and its content covers ≥75% of the article's band. + # Stop at the first column holding a dateline/headline, a vertical rule, or + # the page edge. + rby = {r["id"]: r for r in regions} + tws = sorted((r["bbox"][2] - r["bbox"][0]) for r in regions + if r.get("type") == "text") + pitch = tws[len(tws) // 2] if tws else 0 + hard_ids = set(dl_body_ids) | base_anchor_ids # datelines + headlines + dissolved = set() + + def _ubbox(members): + xs = [b for _, b in members] + return [min(z[0] for z in xs), min(z[1] for z in xs), + max(z[2] for z in xs), max(z[3] for z in xs)] + + # current owner of every region (its cluster anchor, or an orphan group index) + loc = {} + for aid, mem in clusters.items(): + for i, _ in mem: + loc[i] = ("c", aid) + for gi, grp in enumerate(orphan_groups): + for i, _ in grp["members"]: + loc[i] = ("o", gi) + + if pitch > 0: + # Grow-right starters: any article anchor — a doc_title of ANY width (a banner + # need not be ≥1.6×pitch; a narrow ~1.5-column headline like లారీలు grows too) + # OR a headless dateline article (గజ్వేల్ with no detected headline). The strict + # column-twin guards below (same width ±10% / photo-exempt, same gutter, band + # alignment ≥85%, stop at a vertical rule or a column owning its own headline/ + # dateline) are what keep this from over-merging — so the START can be liberal. + banners = [a for a in anchors + if a["id"] not in dissolved + and (a.get("type") == "doc_title" or a["id"] in missed_ids)] + banners.sort(key=lambda a: a["bbox"][1]) + for H in banners: + if H["id"] in dissolved: + continue + hid = H["id"] + + def _relocate(rid): + where = loc.get(rid) + if where is None or where == ("c", hid): + return + kind, key = where + if kind == "c": + clusters[key] = [(i, b) for i, b in clusters[key] if i != rid] + else: + grp = orphan_groups[key] + grp["members"] = [(i, b) for i, b in grp["members"] if i != rid] + clusters[hid].append((rid, rby[rid]["bbox"])) + loc[rid] = ("c", hid) + + def _absorb(rid): + # An extra (paragraph_title) anchor in the column is an in-article + # subhead: move its WHOLE cluster and dissolve it. Plain regions + # move individually. + if rid in anchor_ids and rid != hid: + for i, _ in list(clusters.get(rid, [])): + _relocate(i) + clusters[rid] = [] + dissolved.add(rid) + else: + _relocate(rid) + + hb = _ubbox(clusters[hid]) + band_top, band_bot, right = hb[1], hb[3], hb[2] + # This article's OWN column geometry (rules 2 & 3). Derive the column + # WIDTH and inter-column GUTTER from the header's existing body columns + # so the grow-right tests compare against THIS article's metrics — not a + # page-wide median that mixes in narrow side stories. Single-column + # spanning members (the banner header itself, ≥1.5×pitch wide) are + # excluded so they don't merge two columns into one. + _bodies = [b for i, b in clusters[hid] + if i != hid and (b[2] - b[0]) < 1.5 * pitch] + _acols = [] # [left, right] per body column + for bx in sorted(_bodies, key=lambda z: z[0]): + for c in _acols: + ov = min(c[1], bx[2]) - max(c[0], bx[0]) + if ov > 0.5 * min(c[1] - c[0], bx[2] - bx[0]): + c[0] = min(c[0], bx[0]); c[1] = max(c[1], bx[2]) + break + else: + _acols.append([bx[0], bx[2]]) + _acols.sort() + art_colw = max((c[1] - c[0] for c in _acols), default=pitch) + _gaps = [_acols[i + 1][0] - _acols[i][1] for i in range(len(_acols) - 1)] + art_gutter = min(_gaps) if _gaps else None + while True: + lo, hi = right, right + int(1.4 * pitch) + col = [r for r in regions + if r.get("type") not in ("header", "number", "footer") + and lo + 5 <= _cx(r["bbox"]) <= hi] + if not col: + break # nothing → page edge + # A dateline/headline stops the walk only when it sits WITHIN this + # article's vertical band. A different story stacked ABOVE/BELOW in + # the same column (its headline outside our band) is not a barrier — + # only the column-segment beside our own rows decides ownership. + if any(r["id"] in hard_ids + and min(r["bbox"][3], band_bot) > max(r["bbox"][1], band_top) + for r in col): + break # dateline/headline → stop + inband = [r for r in col + if (min(r["bbox"][3], band_bot) - max(r["bbox"][1], band_top)) + >= 0.75 * max(1, r["bbox"][3] - r["bbox"][1])] + if not inband: + break + cb = _ubbox([(r["id"], r["bbox"]) for r in inband]) + # A vertical rule in the GUTTER *before* this column (between our current + # right edge and the column's left edge) is the article boundary → stop. + # A rule at the column's own RIGHT edge is just its divider — absorb the + # column; the next iteration's gutter check stops the walk there. + if any(right - 5 <= L["x"] <= cb[0] + 5 + and L["y1"] <= band_bot and L["y2"] >= band_top + for L in vsep): + break # vertical rule in gutter → stop + cw = cb[2] - cb[0] + gutter = cb[0] - right # gap to this column + has_img = any(r.get("type") == "image" for r in inband) + # Rule 3 — the next column's WIDTH must be within ±10% of THIS + # article's column width (photos exempt: a photo column need not + # match the text column width). A different width = different story. + if not (has_img or abs(cw - art_colw) <= 0.10 * art_colw): + break # width mismatch → stop + # Rule 2 — the GUTTER to the next column must be within ±10% of the + # article's own inter-column gutter (photos exempt). A wider gap is + # the whitespace before a DIFFERENT story, so stop the walk there. + if (art_gutter is not None and not has_img + and abs(gutter - art_gutter) > 0.10 * art_gutter): + break # gutter widened → stop + # The column must SPAN the article's band height (header + the + # first columns) within ±15% — paragraph gaps inside the column do + # not count against it; only the top-to-bottom extent matters. + col_top = min(r["bbox"][1] for r in inband) + col_bot = max(r["bbox"][3] for r in inband) + band_h = band_bot - band_top + if band_h > 0 and (col_bot - col_top) < 0.85 * band_h: + break # column too short → stop + for r in inband: + _absorb(r["id"]) + nb = _ubbox(clusters[hid]) + band_top, band_bot = nb[1], nb[3] + if cb[2] <= right: + break # no progress → stop + right = cb[2] + + blocks = [] + for a in anchors: + if a["id"] in dissolved: + continue + members = clusters[a["id"]] + if not members: + continue + bxs = [b for _, b in members] + bbox = [min(z[0] for z in bxs), min(z[1] for z in bxs), + max(z[2] for z in bxs), max(z[3] for z in bxs)] + if a["id"] in extra_ids: + kind = "P" + elif a["id"] in missed_ids and a.get("type") != "doc_title": + kind = "DL" + else: + kind = "H" + # Missed-headline article: its stylised banner headline wasn't tagged as a + # region, so it sits in the whitespace ABOVE the topmost detected member with + # nothing to anchor on. Extend the block's top UP to the article-boundary rule + # directly above — but ONLY across a band that is empty of any other region's + # content. If a foreign region sits in the gap, the rule belongs to a different + # story above and extending would drag it in, so we leave the top as-is. + if kind == "DL": + member_ids = {m for m, _ in members} + cand = [L["y"] for L in sep + if not L.get("vert") and "y" in L + and bbox[1] - 420 <= L["y"] <= bbox[1] - 20 + and max(0, min(bbox[2], L["x2"]) - max(bbox[0], L["x1"])) + >= 0.5 * (bbox[2] - bbox[0])] + if cand: + ry = max(cand) + band_clear = not any( + r["id"] not in member_ids + and _col_overlap(bbox, r["bbox"]) + and r["bbox"][3] > ry + 6 and r["bbox"][1] < bbox[1] - 6 + for r in regions) + if band_clear: + bbox[1] = ry + 6 + blocks.append({"anchor_id": a["id"], "kind": kind, "bbox": bbox, + "members": [m for m, _ in members]}) + orphan_blocks = [] + for grp in orphan_groups: + if not grp["members"]: + continue + gbxs = [b for _, b in grp["members"]] + gbbox = [min(z[0] for z in gbxs), min(z[1] for z in gbxs), + max(z[2] for z in gbxs), max(z[3] for z in gbxs)] + orphan_blocks.append({"anchor_id": grp["anchor_id"], "kind": "O", + "bbox": gbbox, + "members": [m for m, _ in grp["members"]]}) + + # Orphan blocks must not overlap each other. When two overlap, the smaller is + # almost always a subset/fragment of the bigger main block — keep the larger + # and drop the smaller. Process biggest-first; drop any orphan that overlaps a + # kept (larger) orphan by ≥50% of its own area. + def _ob_area(b): + return max(0, b[2] - b[0]) * max(0, b[3] - b[1]) + orphan_blocks.sort(key=lambda b: _ob_area(b["bbox"]), reverse=True) + kept_orphans = [] + for ob in orphan_blocks: + bigger = next((k for k in kept_orphans + if _bbox_overlap_ratio(ob["bbox"], k["bbox"]) >= 0.5), None) + if bigger is not None: + print(f" · orphan overlap: block R{ob['anchor_id']} overlaps larger " + f"orphan R{bigger['anchor_id']} — dropped (smaller/subset)") + continue + kept_orphans.append(ob) + blocks.extend(kept_orphans) + + # Enforce one-dateline-per-crop on the geometric blocks too (same wall rule as + # the political-article splitter), so all_article_crops never merge two stories. + blocks = split_blocks_on_datelines(blocks, regions, dateline_starts, sep_lines=sep_lines) + + # Multi-column body under a WIDE photo: a plain-body orphan block sitting directly + # BELOW a photo that belongs to an article (within the photo's column span, small + # gap, no separator between) is that article's continuation in the next column — + # merge it in, so the crop isn't left with a masked-white gap where that text sits. + _rby = {r["id"]: r for r in regions} + + def _plain_body(blk): + return blk["kind"] == "O" and blk["members"] and all( + _rby.get(m, {}).get("type") == "text" and m not in dl_body_ids + for m in blk["members"]) + + photo_owner = {} # photo region id -> its (non-orphan) article block + for blk in blocks: + if blk["kind"] == "O": + continue + for m in blk["members"]: + if _rby.get(m, {}).get("type") == "image": + photo_owner[m] = blk + kept = [] + for blk in blocks: + if _plain_body(blk): + ob = blk["bbox"] + owner = None + for pid, ow in photo_owner.items(): + pb = _rby[pid]["bbox"] + xov = max(0, min(pb[2], ob[2]) - max(pb[0], ob[0])) + if (pb[3] <= ob[1] + 20 and ob[1] - pb[3] <= 200 # photo directly above + and xov >= 0.5 * max(1, ob[2] - ob[0]) # orphan under the photo + and not _barrier_between(pb, ob)): # no separator between + owner = ow + break + if owner is not None: + owner["members"] = sorted(set(owner["members"]) | set(blk["members"])) + mb = [_rby[m]["bbox"] for m in owner["members"] if m in _rby] + owner["bbox"] = [min(b[0] for b in mb), min(b[1] for b in mb), + max(b[2] for b in mb), max(b[3] for b in mb)] + print(f" ⇲ Multi-column body: orphan R{blk['anchor_id']} folded under photo " + f"into article R{owner['anchor_id']} (next-column continuation)") + continue + kept.append(blk) + blocks = kept + + # Caption binding + suppression: a caption — a `figure_title` (any height) OR a short + # `text` (≤ ~2 lines / 200px) sitting directly under an image — that ended up as its + # OWN orphan block belongs to the article that owns the IMAGE above it. Bind it there. + # Any caption-only orphan that can't be bound to a photo's article is page furniture, + # never an article → drop it (e.g. the stray "మాట్లాడుతున్న ప్రొఫెసర్…" orphan). + _rbc = {r["id"]: r for r in regions} + + def _xovc(a, b): + return max(0, min(a[2], b[2]) - max(a[0], b[0])) + + def _img_above(cb): + cw = cb[2] - cb[0] + cands = [_rbc[i]["bbox"] for i in _rbc if _rbc[i].get("type") == "image" + and _rbc[i]["bbox"][3] <= cb[1] + 10 and cb[1] - _rbc[i]["bbox"][3] <= 150 + and _xovc(cb, _rbc[i]["bbox"]) >= 0.4 * max(1, cw)] + if not cands: + return None + return max(cands, key=lambda ib: (_xovc(cb, ib), -(cb[1] - ib[3]))) + + def _is_caption_member(r): + t = r.get("type") + if t == "figure_title": + return True + return t == "text" and (r["bbox"][3] - r["bbox"][1]) <= 200 and _img_above(r["bbox"]) is not None + + img_owner = {} # image region id -> the block that holds it + for blk in blocks: + for m in blk["members"]: + if _rbc.get(m, {}).get("type") == "image": + img_owner[m] = blk + + kept_c = [] + for blk in blocks: + members = [m for m in blk["members"] if m in _rbc] + if blk["kind"] == "O" and members and all(_is_caption_member(_rbc[m]) for m in members): + bound = 0 + for m in members: + ib = _img_above(_rbc[m]["bbox"]) + owner = next((img_owner[i] for i in img_owner + if _rbc[i]["bbox"] == ib and img_owner[i] is not blk), None) if ib else None + if owner is not None: + owner["members"] = sorted(set(owner["members"]) | {m}) + bx = [_rbc[x]["bbox"] for x in owner["members"] if x in _rbc] + owner["bbox"] = [min(b[0] for b in bx), min(b[1] for b in bx), + max(b[2] for b in bx), max(b[3] for b in bx)] + bound += 1 + if bound: + print(f" ⊞ Caption bound: orphan R{blk['anchor_id']} → folded into its photo's article") + else: + print(f" ⊠ Caption orphan dropped: R{blk['anchor_id']} (caption with no photo article)") + continue # caption-only orphan never survives as a crop + kept_c.append(blk) + blocks = kept_c + + # Banner fill: a true banner (a wide doc_title, ≥1.6×pitch) owns EVERY column under + # its width, down to the article's bottom. A column that got intercepted — e.g. a + # caption that anchored its own headless fragment block (R78: caption + col-3 body) — + # is pulled back into the banner. Only HEADLESS FRAGMENT blocks (no dateline, no + # doc_title/promoted headline of their own) that lie ENTIRELY within the banner's + # x-span (+ a half-column overflow margin) and inside its body band are absorbed, so + # a real neighbouring article (which has its own headline/dateline) is never merged. + _rbf = {r["id"]: r for r in regions} + for blk in blocks: + if blk["kind"] != "H": + continue + a = _rbf.get(blk["anchor_id"]) + if not a or a.get("type") != "doc_title" or (a["bbox"][2] - a["bbox"][0]) < 1.6 * pitch: + continue # banner only + bx1, bx2 = a["bbox"][0] - 270, a["bbox"][2] + 270 # banner span + ½-column overflow + btop, bbot = a["bbox"][3] - 10, blk["bbox"][3] + 10 # body band: below headline → bottom + for other in blocks: + if other is blk or not other.get("members"): + continue + if any(_rbf.get(m, {}).get("type") == "doc_title" + or _rbf.get(m, {}).get("_promoted_from") or m in dl_body_ids + for m in other["members"]): + continue # has its own headline/dateline → real article + if all((_rbf[m]["bbox"][0] + _rbf[m]["bbox"][2]) / 2 >= bx1 + and (_rbf[m]["bbox"][0] + _rbf[m]["bbox"][2]) / 2 <= bx2 + and _rbf[m]["bbox"][1] >= btop and _rbf[m]["bbox"][3] <= bbot + for m in other["members"] if m in _rbf): + blk["members"] = sorted(set(blk["members"]) | set(other["members"])) + bx = [_rbf[m]["bbox"] for m in blk["members"] if m in _rbf] + blk["bbox"] = [min(b[0] for b in bx), min(b[1] for b in bx), + max(b[2] for b in bx), max(b[3] for b in bx)] + print(f" ▦ Banner fill: fragment R{other['anchor_id']} pulled under " + f"banner R{blk['anchor_id']} (column below the banner)") + other["members"] = [] # emptied → dropped below + blocks = [b for b in blocks if b.get("members")] + + # Headless-DL headline-strip recovery (NO OCR): a dateline article with NO detected + # headline has its real headline in the UNDETECTED strip ABOVE the block's top edge — + # whether that top is a PHOTO (anganwadi case) or the BODY (top-of-page articles whose + # banner headline PaddleOCR missed entirely). Extend the block's TOP up over that strip + # so the headline pixels land in the crop. Bound it by a separator rule above, the + # previous article's content, or a detected header/masthead (a hard floor — never + # extend INTO it). For a top-of-page article with nothing above, extend up by a capped + # amount to the page margin. Scoped strictly to HEADLESS DL articles. + _rb3 = {r["id"]: r for r in regions} + _hsep = [L for L in (sep_lines or []) if not L.get("vert") and "y" in L] + HSTRIP_CAP = 300 + + def _xov3(a, b): + return max(0, min(a[2], b[2]) - max(a[0], b[0])) + + for blk in blocks: + if blk["kind"] != "DL": + continue + if any(_rb3.get(m, {}).get("type") in ("doc_title", "paragraph_title") + for m in blk["members"]): + continue # has a headline already → not headless + bb = blk["bbox"] + top, w = bb[1], bb[2] - bb[0] + # Boundary above the block top, in its column: separators + the bottoms of + # NON-member regions (previous article / a header). A header/masthead is a hard + # floor — never extend into it. + ups, header_floor = [], None + for r in regions: + if r["id"] in blk["members"]: + continue + rb = r["bbox"] + if rb[3] <= top + 5 and top - rb[3] <= HSTRIP_CAP and _xov3(bb, rb) >= 0.3 * w: + ups.append(rb[3]) + if r.get("type") in ("header", "footer", "number"): + header_floor = max(header_floor or 0, rb[3]) + seps = [L["y"] for L in _hsep + if top - HSTRIP_CAP <= L["y"] <= top + 5 + and _xov3(bb, [L["x1"], 0, L["x2"], 0]) >= 0.3 * w] + cands = ups + seps + if cands: + boundary = max(cands) + elif top <= HSTRIP_CAP: # top-of-page article → page margin + boundary = max(0, top - HSTRIP_CAP) + else: + continue # mid-page, nothing above → no strip + if header_floor is not None: + boundary = max(boundary, header_floor) # never cross a header/masthead + # No detected image inside the strip (boundary→top) — keep it a pure headline band. + if any(_rb3.get(r["id"], {}).get("type") == "image" + and boundary < r["bbox"][3] <= top - 5 and _xov3(bb, r["bbox"]) >= 0.3 * w + for r in regions): + continue + strip_h = top - boundary + if not (45 <= strip_h <= HSTRIP_CAP): # must be a headline-sized band + continue + new_top = int(boundary) + 4 + if new_top < top: + blk["bbox"] = [bb[0], new_top, bb[2], bb[3]] + print(f" ⬆ Headline-strip: DL block R{blk['anchor_id']} top {top}→{new_top} " + f"(captured undetected headline above the article)") + + # Floor-trim: drop trailing page-bottom furniture (a footer line, an empty/tiny text + # fragment) that sits FAR below a block's real content, separated by a large empty + # gap. Such regions fold into the nearest anchor above and inflate the crop with + # white space. If a big vertical gap (>300px) separates the lowest members from the + # rest AND everything below the gap is small/furniture, drop them and pull the bbox + # up to the real content. + _rb4 = {r["id"]: r for r in regions} + for blk in blocks: + mem = sorted([(m, _rb4[m]["bbox"]) for m in blk["members"] if m in _rb4], + key=lambda x: x[1][1]) + if len(mem) < 2: + continue + for i in range(len(mem) - 1, 0, -1): + gap = mem[i][1][1] - max(b[3] for _, b in mem[:i]) + below = mem[i:] + small_furniture = all( + (b[3] - b[1]) < 150 + and _rb4[m].get("type") in ("text", "footer", "number", "figure_title") + for m, b in below) + if gap > 300 and small_furniture: + keep = [m for m, _ in mem[:i]] + bx = [_rb4[m]["bbox"] for m in keep] + blk["members"] = keep + blk["bbox"] = [min(b[0] for b in bx), min(b[1] for b in bx), + max(b[2] for b in bx), max(b[3] for b in bx)] + print(f" ⬓ Floor-trim: block R{blk['anchor_id']} dropped {len(below)} " + f"trailing furniture region(s) below a {int(gap)}px gap") + break + + # Ceiling-trim: the mirror of floor-trim, for LEADING furniture. A headless dateline + # (DL) block's real content is at/below the dateline — nothing above belongs to it. But + # because every region folds into the nearest anchor, a tall top-of-column banner/photo + # that has no anchor of its own falls DOWN into the first dated article beneath it, + # ballooning the crop up to the page top. If the members ABOVE the dateline are + # separated from it by a big gap (>300px), drop them and pull the bbox down. Gated to + # DL blocks + >300px gap, so a normal photo/headline just above a dateline (small gap) + # is never touched. + _dlc = {s["region_id"] for s in (dateline_starts or [])} + for blk in blocks: + if blk["kind"] != "DL": + continue + anc = _rb4.get(blk["anchor_id"], {}).get("bbox") + if not anc: + continue + atop = anc[1] + above = [(m, _rb4[m]["bbox"]) for m in blk["members"] + if m in _rb4 and m != blk["anchor_id"] and _rb4[m]["bbox"][3] <= atop + 5] + if not above: + continue + gap = atop - max(b[3] for _, b in above) + if gap > 300: + drop = {m for m, _ in above} + keep = [m for m in blk["members"] if m not in drop] + bx = [_rb4[m]["bbox"] for m in keep if m in _rb4] + blk["members"] = keep + blk["bbox"] = [min(b[0] for b in bx), min(b[1] for b in bx), + max(b[2] for b in bx), max(b[3] for b in bx)] + print(f" ⬒ Ceiling-trim: block R{blk['anchor_id']} dropped {len(above)} " + f"leading furniture region(s) above a {int(gap)}px gap") + + # Lead-photo-with-caption rebind: a PHOTO → CAPTION → HEADLINE vertical stack is the + # standard lead-photo layout (the photo illustrates the headline's story below it). When + # the photo/caption folded UP into the article above instead, move BOTH the photo and + # its caption down into the headline's block. Gated to that exact 3-region stack (image, + # a figure_title directly below it, a doc_title directly below that) so a photo at the + # BOTTOM of an article — which has body text (not a headline) under its caption — is + # never moved (that was the H93-type false positive). + _rlp = {r["id"]: r for r in regions} + _blkof = {} + for bl in blocks: + for m in bl["members"]: + _blkof[m] = bl + _imgs = [r for r in regions if r.get("type") == "image"] + _caps = [r for r in regions if r.get("type") == "figure_title"] + _docs = [r for r in regions if r.get("type") == "doc_title"] + + def _hovl(a, b): + return max(0, min(a[2], b[2]) - max(a[0], b[0])) + for im in _imgs: + ib = im["bbox"] + cap = next((c for c in _caps if 0 <= c["bbox"][1] - ib[3] <= 90 + and _hovl(ib, c["bbox"]) >= 0.4 * (ib[2] - ib[0])), None) + if not cap: + continue + cb = cap["bbox"] + dt = next((d for d in _docs if 0 <= d["bbox"][1] - cb[3] <= 90 + and _hovl(cb, d["bbox"]) >= 0.3 * (cb[2] - cb[0])), None) + if not dt: + continue + if dt.get("_synth_band"): + continue # a SYNTHESIZED headline's lead photo sits BELOW its band by + # construction; a photo+caption ABOVE it is the tail of the + # article above (Sakshi R36/R69 must stay with the story above) + src = _blkof.get(im["id"]); tgt = _blkof.get(dt["id"]) + if src is None or tgt is None or src is tgt: + continue + moved = [r for r in (im, cap) if _blkof.get(r["id"]) is src] + for r in moved: + src["members"].remove(r["id"]); tgt["members"].append(r["id"]); _blkof[r["id"]] = tgt + for blk in (src, tgt): + bx = [_rlp[m]["bbox"] for m in blk["members"] if m in _rlp] + if bx: + blk["bbox"] = [min(b[0] for b in bx), min(b[1] for b in bx), + max(b[2] for b in bx), max(b[3] for b in bx)] + print(f" 📸 Lead-photo rebind: photo R{im['id']} (+cap R{cap['id']}) → headline " + f"R{dt['id']}'s block (from R{src['anchor_id']})") + + # NOTE: a "below-divider cut" (cut content sitting below a vertical divider's END out of + # the right-column article above — e.g. JG pg1 DL2 wrongly holding the maize story's + # wide photo R30 that straddles the divider) was implemented and REVERTED. The cut/merge + # destabilised the dense JG front page: the freed content merged into the wrong neighbour + # (R49 instead of the maize story R29), and the emptied DL2 regrew (Step 4) and re-absorbed + # stray regions. It is SAFE for AJ/NT (the straddle gate matched only JG R2), but the + # JG merge-target + regrow interaction needs more work before it can land. Deferred. + + # Body-less headline merge: a large headline sometimes lands in its OWN H block + # together with the photo/caption ABOVE it, leaving the bullets + dated body as a + # SEPARATE block right underneath. Such a headline owns no story text of its own, yet + # a dated story sits directly below it in the same columns — the headline clearly + # belongs to THAT story. Fold the headline (and its photo) down into the story below. + # Gated tight: A must be body-LESS, its bottom member must be the headline (doc_title), + # and B must sit ≤90px below with ≥60% x-overlap AND carry real body text. The overlap + # gate means a WIDE section banner over a single narrow column won't merge (roundups). + _rbm = {r["id"]: r for r in regions} + _dropped = set() + _TITLE = ("doc_title", "paragraph_title") + for A in blocks: + # A body-less HEADER block — a lone doc_title (kind H) OR a lone paragraph_title / + # quote-box header (kind P) with no story text of its own — that sits directly above + # a block carrying real body text belongs to that story. (P covers cases like a + # pull-quote/sidebar header whose body folded into the article, leaving the header + # cropped as a tiny strip.) + if A["kind"] not in ("H", "P") or id(A) in _dropped: + continue + amem = [(m, _rbm[m]) for m in A["members"] if m in _rbm] + if any(r["type"] == "text" for _, r in amem): + continue # A must be body-LESS + if not any(r["type"] in _TITLE for _, r in amem): + continue + bottom_m = max(amem, key=lambda t: t[1]["bbox"][3]) + if bottom_m[1]["type"] not in _TITLE: + continue # header must touch the body below + ax1, ay1, ax2, ay2 = A["bbox"] + best = None + for B in blocks: + if B is A or id(B) in _dropped: + continue + bx1, by1, bx2, by2 = B["bbox"] + if not (-10 <= by1 - ay2 <= 90): + continue + if max(0, min(ax2, bx2) - max(ax1, bx1)) < 0.6 * (ax2 - ax1): + continue + if not any(_rbm.get(m, {}).get("type") == "text" for m in B["members"]): + continue + if best is None or by1 < best[0]: + best = (by1, B) + if best: + B = best[1] + B["members"] = list(dict.fromkeys(A["members"] + B["members"])) + bx = [_rbm[m]["bbox"] for m in B["members"] if m in _rbm] + B["bbox"] = [min(b[0] for b in bx), min(b[1] for b in bx), + max(b[2] for b in bx), max(b[3] for b in bx)] + _dropped.add(id(A)) + print(f" ⬇ Headline-merge: body-less headline R{A['anchor_id']} folded into " + f"dated story R{B['anchor_id']}") + if _dropped: + blocks = [b for b in blocks if id(b) not in _dropped] + + # Containment-merge: if a block X sits almost entirely (≥90% of ITS area) inside a + # strictly LARGER block Y, X is a duplicate fragment of Y — a split-off column, an + # orphan tail, or a sub-story sitting under a roundup banner. Fold X's members into Y + # and drop X, so that content is cropped ONCE (inside Y) instead of twice. Runs BEFORE + # the rectangle pass so a reunited article (e.g. a paragraph_title header whose 2nd + # column had split off) again reads as multi-column and can be rectangled. The 0.9 + # containment + larger-Y gate is strict: two side-by-side articles never qualify + # (neither is ~entirely inside the other), so unrelated articles are untouched. + _rbc = {r["id"]: r for r in regions} + def _area(b): + return max(1, (b[2] - b[0]) * (b[3] - b[1])) + _gone = set() + for X in blocks: + if id(X) in _gone: + continue + a = X["bbox"] + best = None + for Y in blocks: + if Y is X or id(Y) in _gone: + continue + c = Y["bbox"] + ox = max(0, min(a[2], c[2]) - max(a[0], c[0])) + oy = max(0, min(a[3], c[3]) - max(a[1], c[1])) + if _area(c) > _area(a) and ox * oy / _area(a) >= 0.9: + if best is None or _area(c) < _area(best["bbox"]): + best = Y # tightest wrapping container + if best is not None: + best["members"] = list(dict.fromkeys(best["members"] + X["members"])) + bx = [_rbc[m]["bbox"] for m in best["members"] if m in _rbc] + best["bbox"] = [min(b[0] for b in bx), min(b[1] for b in bx), + max(b[2] for b in bx), max(b[3] for b in bx)] + _gone.add(id(X)) + print(f" ⊎ Containment-merge: R{X['anchor_id']}({X['kind']}) folded into " + f"R{best['anchor_id']}({best['kind']})") + if _gone: + blocks = [b for b in blocks if id(b) not in _gone] + + # Header-split: a headline (doc_title) heads the story BELOW it. If an H block also + # holds a dateline sitting clearly ABOVE its headline, that dateline belongs to a + # SEPARATE earlier article that got glued on — either the column above had no anchor of + # its own and folded down into this block, or a contained DL article was absorbed into + # this block's over-extended box. Cut at the headline: everything above the headline + # becomes its own dated article; the headline and everything below it stay. Gated to a + # dateline >20px above the headline — the exact over-merge signature (measured to never + # match a correctly-clustered single article on AJ/NT). + _rbh = {r["id"]: r for r in regions} + _dlh = {s["region_id"] for s in (dateline_starts or [])} + _hs_new = [] + for blk in list(blocks): + if blk["kind"] != "H": + continue + hb = _rbh.get(blk["anchor_id"], {}).get("bbox") + if not hb: + continue + htop = hb[1] + dl_above = [m for m in blk["members"] + if m in _dlh and _rbh[m]["bbox"][1] < htop - 20] + if not dl_above: + continue + above = [m for m in blk["members"] + if m in _rbh and m != blk["anchor_id"] and _rbh[m]["bbox"][3] <= htop + 10] + below = [m for m in blk["members"] if m not in set(above)] + if not above or len(below) < 1: + continue + top_dl = min(dl_above, key=lambda m: _rbh[m]["bbox"][1]) + abx = [_rbh[m]["bbox"] for m in above if m in _rbh] + _hs_new.append({"anchor_id": top_dl, "kind": "DL", "members": sorted(above), + "bbox": [min(b[0] for b in abx), min(b[1] for b in abx), + max(b[2] for b in abx), max(b[3] for b in abx)]}) + bbx = [_rbh[m]["bbox"] for m in below if m in _rbh] + blk["members"] = sorted(below) + blk["bbox"] = [min(b[0] for b in bbx), min(b[1] for b in bbx), + max(b[2] for b in bbx), max(b[3] for b in bbx)] + blk.pop("_raw_rect", None) # geometry changed; re-evaluate below + print(f" ✂ Header-split: R{blk['anchor_id']}(H) split off the article ABOVE its " + f"headline (new DL anchor R{top_dl}, {len(above)} region(s))") + blocks.extend(_hs_new) + + # Banner crop-the-rectangle: a BANNER-headed MULTI-COLUMN article is cropped as the + # FULL rectangle — the banner's top, across its columns, down to the detected bottom + # boundary — taking everything in the box (raw, no masking). This fills any residual + # gap and keeps a section header's stories together (one crop per banner, by design). + # Strictly gated: (a) banner ≥1.6 columns AND article spans ≥2 columns, and (b) the + # bottom boundary is SURE — a separator rule spanning the banner, the next banner + # below, or the page bottom. If the boundary is uncertain, leave the member-based crop. + _rbr = {r["id"]: r for r in regions} + _hbar = [L for L in (sep_lines or []) if not L.get("vert") and "y" in L] + _page_bot = max((r["bbox"][3] for r in regions), default=0) + _dlids = {s["region_id"] for s in (dateline_starts or [])} + _sec_dropped = set() + for blk in blocks: + # A doc_title-headed block (kind H) is banner-detected on its doc_title; a + # paragraph_title-headed block (kind P) on its paragraph_title. H behaviour is + # unchanged — P just lets a wide paragraph_title header act as the banner too, so + # multi-column paragraph_title articles crop as one rectangle like doc_title ones. + if blk["kind"] == "H": + _btype = "doc_title" + elif blk["kind"] == "P": + _btype = "paragraph_title" + else: + continue + bts = [_rbr[m]["bbox"] for m in blk["members"] + if _rbr.get(m, {}).get("type") == _btype + and (_rbr[m]["bbox"][2] - _rbr[m]["bbox"][0]) >= 1.6 * pitch] + if not bts: + continue + banner = max(bts, key=lambda b: b[2] - b[0]) + + # ── Wide SECTION banner (roundup) ────────────────────────────────────── + # A banner ≥2.5 columns that heads MULTIPLE separate dated sub-stories is a + # SECTION header, not a single article. Crop the whole section as ONE raw + # rectangle: banner-top → the next full-width separator that spans the banner, + # and absorb (drop) every sub-block sitting inside it. The "≥2 headed sub-blocks + # inside" gate is what separates a section header from a wide SINGLE-article + # banner (which has no dated sub-stories beneath it) — so single articles and + # already-merged roundups are never caught. + b0l, b0t, b0r, b0b = banner + ban_w = b0r - b0l + if blk["kind"] == "H" and ban_w >= 2.5 * pitch: + spanning = [L["y"] for L in _hbar if L["y"] > b0b + 20 + and max(0, min(b0r, L["x2"]) - max(b0l, L["x1"])) >= 0.8 * ban_w] + sec_bot = min(spanning) if spanning else None + if sec_bot is not None: + inside = [o for o in blocks if o is not blk + and o["bbox"][0] >= b0l - 30 and o["bbox"][2] <= b0r + 30 + and o["bbox"][1] >= b0t - 10 and o["bbox"][3] <= sec_bot + 10] + headed_inside = [o for o in inside + if any(m in _dlids for m in o["members"]) + or any(_rbr.get(m, {}).get("type") in ("doc_title", "paragraph_title") + for m in o["members"])] + if len(headed_inside) >= 2: + blk["bbox"] = [int(b0l), int(b0t), int(b0r), int(sec_bot)] + blk["_raw_rect"] = True + blk["_section"] = True # TRUE Type-1 section: a wide banner + a spanning + # bottom RULE (sec_bot) — never Type-2 split this. + for o in inside: + _sec_dropped.add(id(o)) + print(f" ▭▭ Section banner: R{blk['anchor_id']} cropped as ONE rectangle " + f"x[{int(b0l)},{int(b0r)}] y[{int(b0t)},{int(sec_bot)}], " + f"absorbed {len(inside)} sub-block(s)") + continue + # ─────────────────────────────────────────────────────────────────────── + + # x-span = UNION of banner AND every member: the banner is sometimes detected + # narrower than the article's body columns, so clamping to the banner edge would + # clip the outer column(s). Take the widest extent that the article actually fills. + mboxes = [_rbr[m]["bbox"] for m in blk["members"] if m in _rbr] + bl = min([banner[0]] + [b[0] for b in mboxes]) + brt = max([banner[2]] + [b[2] for b in mboxes]) + btop = min([banner[1]] + [b[1] for b in mboxes]) # incl. a photo ABOVE the banner + cols = {round(((_rbr[m]["bbox"][0] + _rbr[m]["bbox"][2]) / 2) / max(1, pitch)) + for m in blk["members"] if _rbr.get(m, {}).get("type") == "text"} + if len(cols) < 2: + continue # must be multi-column + + # The article's REAL bottom comes from THIS block's own clustered members — + # NOT every region geometrically under the banner's wide x-span (that would pull + # a top banner all the way to the page bottom and swallow the stories below it). + content_bot = max((_rbr[m]["bbox"][3] for m in blk["members"] if m in _rbr), + default=blk["bbox"][3]) + # SURE bottom boundary: a clean cut must exist JUST below the content (within + # ~70px) — a separator rule spanning the banner, the next banner down, or the + # page bottom. This confirms the rectangle ends with the article, not mid-story. + WIN = 70 + sep_below = [L["y"] for L in _hbar + if content_bot - 12 <= L["y"] <= content_bot + WIN + and max(0, min(brt, L["x2"]) - max(bl, L["x1"])) >= 0.5 * (brt - bl)] + next_banner = [r["bbox"][1] for r in regions + if r.get("type") == "doc_title" + and (r["bbox"][2] - r["bbox"][0]) >= 1.6 * pitch + and content_bot - 12 <= r["bbox"][1] <= content_bot + WIN + 40 + and max(0, min(brt, r["bbox"][2]) - max(bl, r["bbox"][0])) >= 0.3 * (brt - bl)] + near_page_bot = content_bot >= _page_bot - 80 + if not (sep_below or next_banner or near_page_bot): + continue # boundary NOT sure → leave as-is + bottom = min([content_bot + 12] + sep_below + [(nb - 8) for nb in next_banner]) + bottom = max(bottom, content_bot + 4) # never cut into the content + blk["bbox"] = [int(bl), int(btop), int(brt), int(bottom)] + blk["_raw_rect"] = True + print(f" ▭ Banner rectangle: R{blk['anchor_id']} cropped top-to-bottom " + f"x[{int(bl)},{int(brt)}] y[{int(btop)},{int(bottom)}] (raw, {len(cols)} cols)") + + if _sec_dropped: + blocks = [b for b in blocks if id(b) not in _sec_dropped] + + # Rightmost-column page-edge extension: an article in the RIGHTMOST column has the + # page's right margin as its natural right boundary (a "dead-end"). If a block's right + # edge already sits within half a column of the page-content right edge, extend it out + # to that page edge so the rightmost story isn't cropped tight / clipped. Strictly the + # right edge only — never a column-add, and INTERIOR articles (right edge well left of + # the page edge) are never touched. + _cright = max((r["bbox"][2] for r in regions), default=0) + for blk in blocks: + x1, y1, x2, y2 = blk["bbox"] + if _cright - 0.5 * pitch <= x2 < _cright: + blk["bbox"] = [x1, y1, int(_cright), y2] + print(f" ▶ Right-edge→page: R{blk['anchor_id']} right {int(x2)}→{int(_cright)} " + f"(rightmost column, +{int(_cright - x2)}px)") + + # Drop graphic-label orphans: an ORPHAN (O) block that is pure text and sits almost + # entirely (≥80%) inside a FOREIGN image region is a label / caption baked into a photo + # or banner graphic — not an article. Remove it. (Measured: only 1 such block across + # AJ+NT+JG — a stray label inside a front-page banner image; AJ/NT have none.) + _rbj = {r["id"]: r for r in regions} + _imgsj = [(r["id"], r["bbox"]) for r in regions if r.get("type") == "image"] + _junk = set() + for blk in blocks: + if blk["kind"] != "O": + continue + if any(_rbj.get(m, {}).get("type") != "text" for m in blk["members"]): + continue # pure-text orphans only + a = blk["bbox"]; aarea = max(1, (a[2] - a[0]) * (a[3] - a[1])) + for iid, c in _imgsj: + if iid in blk["members"]: + continue + ox = max(0, min(a[2], c[2]) - max(a[0], c[0])) + oy = max(0, min(a[3], c[3]) - max(a[1], c[1])) + if ox * oy / aarea >= 0.8: + _junk.add(id(blk)) + print(f" ✗ Drop graphic-label orphan: R{blk['anchor_id']} (pure text " + f"inside foreign image R{iid})") + break + if _junk: + blocks = [b for b in blocks if id(b) not in _junk] + + # NOTE: orphan-TEXT absorb (fold a pure-text orphan up into the article above) was tried + # and REVERTED — its only match (NT pg4 R75→R132) was a CLASSIFIEDS "change of name" + # notice where it could not be verified whether the orphan was part of that ad or a + # separate adjacent notice. Merging distinct classifieds is a regression risk, so this + # remains deferred (see CROP_STRATEGY.md backlog). + + # No-overlap reconciliation: when two block boxes overlap and exactly ONE of them has + # NO members inside the overlap, that box is merely over-reaching into its neighbour's + # territory (empty space for a masked crop, a DUPLICATE for a raw crop). Clip the + # over-reaching EDGE of the 0-member box back to its own members so the boxes no longer + # overlap. Only the 0-member side is trimmed (never grown); pairs where BOTH sides have + # members in the overlap are genuine interleaves (L-shaped articles) and are left alone. + _rbo = {r["id"]: r for r in regions} + + def _mbb(blk): + bx = [_rbo[m]["bbox"] for m in blk["members"] if m in _rbo] + return ([min(b[0] for b in bx), min(b[1] for b in bx), + max(b[2] for b in bx), max(b[3] for b in bx)] if bx else list(blk["bbox"])) + + def _memcount(blk, O): + return sum(1 for m in blk["members"] if m in _rbo + and not (_rbo[m]["bbox"][2] < O[0] or _rbo[m]["bbox"][0] > O[2] + or _rbo[m]["bbox"][3] < O[1] or _rbo[m]["bbox"][1] > O[3])) + + for i in range(len(blocks)): + for j in range(i + 1, len(blocks)): + A, B = blocks[i], blocks[j] + a, b = A["bbox"], B["bbox"] + O = [max(a[0], b[0]), max(a[1], b[1]), min(a[2], b[2]), min(a[3], b[3])] + if O[0] >= O[2] or O[1] >= O[3]: + continue + iarea = (O[2] - O[0]) * (O[3] - O[1]) + if iarea < 0.10 * min((a[2]-a[0])*(a[3]-a[1]), (b[2]-b[0])*(b[3]-b[1])): + continue + ca, cb = _memcount(A, O), _memcount(B, O) + if (ca == 0) == (cb == 0): + continue # both empty or both occupied → leave + X = A if ca == 0 else B # the over-reaching, content-free box + other = B if X is A else A + xb = list(X["bbox"]) + mx = _mbb(X) # X's own members (must NOT be cut) + # neighbour's OCCUPIED region within the overlap (the content X is covering) + oboxes = [_rbo[mm]["bbox"] for mm in other["members"] + if mm in _rbo and not (_rbo[mm]["bbox"][2] < O[0] or _rbo[mm]["bbox"][0] > O[2] + or _rbo[mm]["bbox"][3] < O[1] or _rbo[mm]["bbox"][1] > O[3])] + if not oboxes: + continue + oc = [min(b[0] for b in oboxes), min(b[1] for b in oboxes), + max(b[2] for b in oboxes), max(b[3] for b in oboxes)] + # Retreat X's facing edge to just clear `oc`, WITHOUT cutting any of X's members. + cands = [] + if oc[1] - 4 >= mx[3]: cands.append((xb[3] - (oc[1] - 4), [xb[0], xb[1], xb[2], oc[1] - 4])) # clip bottom (X above) + if oc[3] + 4 <= mx[1]: cands.append(((oc[3] + 4) - xb[1], [xb[0], oc[3] + 4, xb[2], xb[3]])) # clip top (X below) + if oc[0] - 4 >= mx[2]: cands.append((xb[2] - (oc[0] - 4), [xb[0], xb[1], oc[0] - 4, xb[3]])) # clip right (X left) + if oc[2] + 4 <= mx[0]: cands.append(((oc[2] + 4) - xb[0], [oc[2] + 4, xb[1], xb[2], xb[3]])) # clip left (X right) + cands = [(loss, nb) for loss, nb in cands if loss > 0 and nb[2] > nb[0] and nb[3] > nb[1]] + if not cands: + continue # X wraps the neighbour → genuine, leave alone + X["bbox"] = min(cands, key=lambda t: t[0])[1] + print(f" ▢ No-overlap: trimmed R{X['anchor_id']}({X['kind']}) to clear overlap " + f"with R{other['anchor_id']}") + + # Ragged-bottom sidebar split: an L-shaped article whose ONE column runs FAR below the + # others, where that trailing tail is a SELF-CONTAINED sub-box (starts with its own + # paragraph_title sub-head) and the vacated corner is occupied by a DIFFERENT article + # (so one rectangle would mask that neighbour as white space). Split the tail into its + # own crop so the main article becomes a clean rectangle. STRICT gate: the MAIN part + # (block minus tail) must REMAIN multi-column (≥2 text columns) — so a block whose main + # would collapse to a lone column is left intact (avoids mis-splitting e.g. AJ R53). + _rs = {r["id"]: r for r in regions} + _tws = sorted((r["bbox"][2] - r["bbox"][0]) for r in regions if r.get("type") == "text") + _pitch_s = _tws[len(_tws) // 2] if _tws else 530 + _split_new = [] + for blk in list(blocks): + mem = [(m, _rs[m]["bbox"]) for m in blk["members"] if m in _rs] + if len(mem) < 4: + continue + colbot = {} + for m, bb in mem: + if _rs[m]["type"] in ("text", "paragraph_title"): + c = round(((bb[0] + bb[2]) / 2) / _pitch_s) + colbot[c] = max(colbot.get(c, -1), bb[3]) + if len(colbot) < 2: + continue + tailcol = max(colbot, key=lambda c: colbot[c]) + others_bot = max(v for c, v in colbot.items() if c != tailcol) + if colbot[tailcol] - others_bot < 250: + continue + tail = [(m, bb) for m, bb in mem + if round(((bb[0] + bb[2]) / 2) / _pitch_s) == tailcol and bb[1] >= others_bot - 30] + if not tail: + continue + tail.sort(key=lambda t: t[1][1]) + if _rs[tail[0][0]]["type"] != "paragraph_title": + continue + if tail[-1][1][3] - tail[0][1][1] < 200: + continue + tail_ids = {m for m, _ in tail} + main_ids = [m for m in blk["members"] if m not in tail_ids] + maincols = {round(((_rs[m]["bbox"][0] + _rs[m]["bbox"][2]) / 2) / _pitch_s) + for m in main_ids if _rs.get(m, {}).get("type") == "text"} + if len(maincols) < 2: + continue # main would collapse → leave intact + ty1 = tail[0][1][1]; ty2 = max(bb[3] for _, bb in tail); minx = min(bb[0] for _, bb in tail) + if not any(o is not blk and not (o["bbox"][3] < ty1 or o["bbox"][1] > ty2) + and o["bbox"][0] < minx - 20 for o in blocks): + continue # nothing in the vacated corner → no white space + mbx = [_rs[m]["bbox"] for m in main_ids if m in _rs] + blk["members"] = sorted(main_ids) + blk["bbox"] = [min(b[0] for b in mbx), min(b[1] for b in mbx), + max(b[2] for b in mbx), max(b[3] for b in mbx)] + tbx = [bb for _, bb in tail] + _split_new.append({"anchor_id": tail[0][0], "kind": "P", "members": sorted(tail_ids), + "bbox": [min(b[0] for b in tbx), min(b[1] for b in tbx), + max(b[2] for b in tbx), max(b[3] for b in tbx)]}) + print(f" ⑃ Sidebar-split: R{blk['anchor_id']} → main + sidebar R{tail[0][0]} " + f"({len(tail_ids)} regs, clears L-shape white space)") + blocks.extend(_split_new) + + # Seed / lock CONFIDENT cells: a block is high-confidence ("locked") when it is a + # COMPLETE article — carries BOTH a dateline AND a detected headline + # (doc_title/paragraph_title) — OR it is FULLY BOXED (≥3 of its 4 sides coincide with a + # printed separator rule). These are the partition's fixed seeds; later boundary work + # operates on the UN-locked remainder. + _hb_l = [L for L in (sep_lines or []) if not L.get("vert") and "y" in L] + _vb_l = [L for L in (sep_lines or []) if L.get("vert")] + _dl_l = {s["region_id"] for s in (dateline_starts or [])} + _rl = {r["id"]: r for r in regions} + for blk in blocks: + x1, y1, x2, y2 = blk["bbox"]; bw = max(1, x2 - x1); bhh = max(1, y2 - y1) + n_dl = sum(1 for m in blk["members"] if m in _dl_l) + (1 if blk["anchor_id"] in _dl_l + and blk["anchor_id"] not in blk["members"] else 0) + single_dl = (n_dl == 1) # exactly one dated story = not an over-merge + has_head = any(_rl.get(m, {}).get("type") in ("doc_title", "paragraph_title") + for m in blk["members"]) + sides = sum([ + any(abs(L["y"] - y1) <= 45 and max(0, min(x2, L["x2"]) - max(x1, L["x1"])) >= 0.6 * bw for L in _hb_l), + any(abs(L["y"] - y2) <= 45 and max(0, min(x2, L["x2"]) - max(x1, L["x1"])) >= 0.6 * bw for L in _hb_l), + any(abs(L["x"] - x1) <= 45 and max(0, min(y2, L["y2"]) - max(y1, L["y1"])) >= 0.6 * bhh for L in _vb_l), + any(abs(L["x"] - x2) <= 45 and max(0, min(y2, L["y2"]) - max(y1, L["y1"])) >= 0.6 * bhh for L in _vb_l), + ]) + blk["_locked"] = bool((single_dl and has_head) or sides >= 3) + + # Carve COURT / LEGAL NOTICES into their OWN section box. Regions tagged r['_legal'] by + # tag_legal_notices are court summons / name-change petitions / 'by order of the court' + # notices — page furniture that must be cropped as its own block, not absorbed into a + # neighbour (JG R65 swallowed a court summons in its left column). Group adjacent tagged + # regions; carve a cluster out ONLY when it is a real multi-part notice (≥2 tagged regions, + # ≥1 of them 'core') that sits ENTIRELY inside ONE WIDE (masked) host block — so the host + # whites the carved area out and nothing is duplicated. (A notice straddling a NARROW + # straight-cropped block, e.g. NT pg3, is intentionally left alone: carving it would + # duplicate, since a narrow block is cropped verbatim with no white-out.) Runs AFTER + # seed/lock so the new box is force-locked and Step 4 treats it as a fixed obstacle. + _legal_regs = [r for r in regions if r.get("_legal")] + if _legal_regs: + _legal_regs.sort(key=lambda r: r["bbox"][1]) + _used = set(); _clusters = [] + for r in _legal_regs: + if r["id"] in _used: + continue + grp = [r]; _used.add(r["id"]); changed = True + while changed: + changed = False + gx1 = min(g["bbox"][0] for g in grp); gx2 = max(g["bbox"][2] for g in grp) + gy1 = min(g["bbox"][1] for g in grp); gy2 = max(g["bbox"][3] for g in grp) + for s in _legal_regs: + if s["id"] in _used: + continue + sb = s["bbox"] + ox = min(gx2, sb[2]) - max(gx1, sb[0]) + if ox < 0.4 * min(gx2 - gx1, sb[2] - sb[0]): + continue # different column + if sb[1] > gy2 + 180 or sb[3] < gy1 - 180: + continue # too far vertically + grp.append(s); _used.add(s["id"]); changed = True + _clusters.append(grp) + _lbyid = {r["id"]: r for r in regions} + for grp in _clusters: + ids = {g["id"] for g in grp} + if len(grp) < 2 or not any(g.get("_legal") == "core" for g in grp): + continue # need a real multi-part court notice + hosts = [b for b in blocks if ids & (set(b["members"]) | {b["anchor_id"]})] + if len(hosts) != 1: + continue # spread across blocks → leave alone + host = hosts[0] + if not ids <= (set(host["members"]) | {host["anchor_id"]}): + continue + hx1, _hy1, hx2, _hy2 = host["bbox"] + if (hx2 - hx1) <= 1100: + continue # NARROW host crops verbatim → would duplicate + mem = sorted(ids) + lbx = [_lbyid[i]["bbox"] for i in mem] + lbbox = [min(b[0] for b in lbx), min(b[1] for b in lbx), + max(b[2] for b in lbx), max(b[3] for b in lbx)] + anchor = min(mem, key=lambda i: _lbyid[i]["bbox"][1]) # topmost region = header + host["members"] = [m for m in host["members"] if m not in ids] + blocks.append({"anchor_id": anchor, "kind": "L", "bbox": lbbox, + "members": mem, "_locked": True, "_legal": True}) + print(f" ⚖️ Legal notice: carved R{anchor} ({len(mem)} regs) out of " + f"R{host['anchor_id']} as its own section {lbbox}") + + # ── Type-2 SPLIT: separate stacked articles that share NO common bottom rule ────────── + # Two layouts both look like one big box but need OPPOSITE treatment (AJ / Namaste — ruled + # papers): + # • Type 1 — several sub-stories under ONE wide banner with a COMMON BOTTOM horizontal rule. + # That banner+rule made the block a `_section` (the section-banner pass) → crop as ONE + # rectangle. NEVER split it. + # • Type 2 — independent articles stacked one below another, EACH with its OWN headline AND + # its own dateline, with NO common bottom rule between them (e.g. JG R65: కలెక్టర్/సుందరయ్య/ + # పోస్టుమార్టం; R64: జాతీయ రహదారి / ఉపసర్పంచుల; R97: గుండె చికిత్స / మృతుడి పరామర్శ). Over-merge + # → split into one article per headline. + # DISCRIMINATOR = the `_section` flag (a real spanning bottom RULE), NOT the coarse `_raw_rect` + # flag — a single wide headline raw'd by the banner-rectangle pass (which merely extends to the + # next banner, NO rule) can still hide a 2nd article below it (R64), so those ARE eligible. + # ANCHORS = doc_title OR paragraph_title (PaddleOCR often tags the 2nd headline a + # paragraph_title — R84, R58) that OWNS a dateline. Ownership is COLUMN-AWARE: each dateline + # goes to the nearest headline above it WHOSE x-span covers the dateline (so a caption in a + # different column never steals a dateline). A single article with a headline + a sub-headline + # sharing ONE dateline yields ONE owner ⇒ not split (AJ R50/R57, R35/R79). + _new_split = [] + _split_done = [] + for blk in list(blocks): + if blk.get("_section") or blk.get("_legal"): + continue # true Type-1 section / legal → keep whole + dts = sorted([m for m in blk["members"] + if _rl.get(m, {}).get("type") in ("doc_title", "paragraph_title")], + key=lambda m: _rl[m]["bbox"][1]) + _owns = {} + for d in [m for m in blk["members"] if m in _dl_l]: + db = _rl[d]["bbox"]; dy = db[1]; dcx = (db[0] + db[2]) / 2 + above = [h for h in dts if _rl[h]["bbox"][1] <= dy + 8 + and _rl[h]["bbox"][0] - 40 <= dcx <= _rl[h]["bbox"][2] + 40] # same column + if above: + _owns.setdefault(max(above, key=lambda h: _rl[h]["bbox"][1]), []).append(d) + anchors = sorted(_owns.keys(), key=lambda h: _rl[h]["bbox"][1]) + if len(anchors) < 2: + continue + # require the headlines to be VERTICALLY STACKED (≥120px apart) — not two columns headed + # side-by-side at the same height (that is one section, not a vertical over-merge). + ys = [_rl[a]["bbox"][1] for a in anchors] + if any(abs(ys[i + 1] - ys[i]) < 120 for i in range(len(ys) - 1)): + continue + # Assign each member to its article using BOTH y and x: the nearest headline at/above it + # (the y-band), but only if the member sits within that headline's COLUMN x-span (± margin). + # A member that lands in a headline's y-band but a DIFFERENT column is unrelated content + # sharing the page real-estate — e.g. JG R65's left column below the carved court notice + # holds a classifieds logo + a small job ad + a గమనిక notice, none of which belong to the + # centre-right సుందరయ్య / పోస్టుమార్టం stories. Such members fall out as LEFTOVER and become a + # `_furniture` section: excluded from the article crops (a separate block masks white), drawn + # on the debug boundary map, and NOT saved as an article crop. + _marg = 0.3 * pitch + groups = {a: [] for a in anchors} + leftover = [] + for m in set(blk["members"]) | {blk["anchor_id"]}: + if m not in _rl: + continue + mb = _rl[m]["bbox"]; mcx = (mb[0] + mb[2]) / 2; my = mb[1] + above = [a for a in anchors if _rl[a]["bbox"][1] <= my + 8] + tgt = max(above, key=lambda a: _rl[a]["bbox"][1]) if above else anchors[0] + hb = _rl[tgt]["bbox"] + if hb[0] - _marg <= mcx <= hb[2] + _marg: + groups[tgt].append(m) + else: + leftover.append(m) + for a in anchors: + mem = sorted(set(groups[a])) + if not mem: + continue + bxs = [_rl[m]["bbox"] for m in mem] + _new_split.append({"anchor_id": a, "kind": "H", + "bbox": [min(b[0] for b in bxs), min(b[1] for b in bxs), + max(b[2] for b in bxs), max(b[3] for b in bxs)], + "members": mem, "_locked": True}) + # group the leftover (out-of-column) regions into furniture section(s) by column + + # proximity — but NEVER across a separator rule: two stacked items in the leftover + # column with a rule between them are different stories (pg2 col4: the విద్యార్థిని + # girl's story and the శ్రీనివాస్ tribute tail were one group until the y=4679 rule + # was honoured). + _fclusters = [] + for m in sorted(leftover, key=lambda m: _rl[m]["bbox"][1]): + mb = _rl[m]["bbox"] + placed = False + for fc in _fclusters: + fx1 = min(_rl[i]["bbox"][0] for i in fc); fx2 = max(_rl[i]["bbox"][2] for i in fc) + fy1 = min(_rl[i]["bbox"][1] for i in fc) + fy2 = max(_rl[i]["bbox"][3] for i in fc) + if (min(fx2, mb[2]) - max(fx1, mb[0])) >= 0.3 * min(fx2 - fx1, mb[2] - mb[0]) \ + and mb[1] <= fy2 + 260 \ + and not (_ACTIVE_PAPER == "sakshi" + and _barrier_between([fx1, fy1, fx2, fy2], mb)): + fc.append(m); placed = True; break + if not placed: + _fclusters.append([m]) + for fc in _fclusters: + bxs = [_rl[i]["bbox"] for i in fc] + fbb = [min(b[0] for b in bxs), min(b[1] for b in bxs), + max(b[2] for b in bxs), max(b[3] for b in bxs)] + top_id = min(fc, key=lambda i: _rl[i]["bbox"][1]) + # A leftover group that LEADS with its own column-wide heading and carries body + # text is a real stacked ARTICLE whose column none of the split anchors span — + # emit it as an article, not furniture (Sakshi pg2: the విద్యార్థిని ఆత్మహత్య story, + # heading R88 on top, fell out of the R41 split and was never cropped). The + # width gate matters twice over: a narrow bold label is not a story heading (a + # classifieds WANTED box stays furniture), and a narrow SUB-head atop a story's + # continuation tail ('పలువురి నివాళి') must fall through to the continuation + # check below, not start a phantom article. This check runs BEFORE band-attach: + # a group with its own real heading is a new story even when it sits gap-close + # beside another article's band. + if (_rl[top_id].get("type") in ("doc_title", "paragraph_title") + and any(_rl[i].get("type") == "text" for i in fc) + and (_rl[top_id]["bbox"][2] - _rl[top_id]["bbox"][0]) + >= 0.6 * max(1, fbb[2] - fbb[0])): + _new_split.append({"anchor_id": top_id, "kind": "P", "bbox": fbb, + "members": sorted(fc), "_locked": True}) + print(f" ✂️ Type-2 leftover → ARTICLE R{top_id} (leads with its own " + f"heading; {len(fc)} regions)") + continue + # A heading-less leftover group whose vertical band MATCHES an adjacent-column + # split article (top within 90px, ≥80% overlap, facing gap ≤80px) is that + # article's column CONTINUATION — its text even resumes mid-sentence (pg2: + # R119/R4/R97/R7 carry the విషాదయాత్ర story's ending + tributes, beside H93). + fh = max(1, fbb[3] - fbb[1]) + host = None + for ns in (_new_split if _ACTIVE_PAPER == "sakshi" else []): + if ns.get("_furniture"): + continue + nb = ns["bbox"] + gap = (fbb[0] - nb[2]) if fbb[0] >= nb[2] else (nb[0] - fbb[2]) + vov = min(fbb[3], nb[3]) - max(fbb[1], nb[1]) + if (-20 <= gap <= 80 and abs(nb[1] - fbb[1]) <= 90 + and vov >= 0.8 * min(fh, max(1, nb[3] - nb[1]))): + host = ns + break + if host is not None: + host["members"] = sorted(set(host["members"]) | set(fc)) + host["bbox"] = [min(host["bbox"][0], fbb[0]), min(host["bbox"][1], fbb[1]), + max(host["bbox"][2], fbb[2]), max(host["bbox"][3], fbb[3])] + print(f" ✂️ Type-2 leftover → CONTINUATION of R{host['anchor_id']} " + f"(same band, adjacent column; {len(fc)} regions)") + continue + _new_split.append({"anchor_id": min(fc), "kind": "F", "bbox": fbb, + "members": sorted(fc), "_locked": True, "_furniture": True}) + _split_done.append(id(blk)) + print(f" ✂️ Type-2 split: R{blk['anchor_id']} → {len(anchors)} articles " + f"R{anchors}" + (f" + {len(_fclusters)} furniture section(s)" if _fclusters else "") + + " (stacked headlines, no common bottom rule)") + if _split_done: + blocks = [b for b in blocks if id(b) not in _split_done] + _new_split + + # ── SAKSHI / NT horizontal-rule split ──────────────────────────────────────────────── + # Sakshi is a no-grid, headline-light paper, so the whitespace clusterer over-merges stacked + # stories into one block. But Sakshi DOES print horizontal RULES between stacked articles + # (full-width and ~1-column). Per the brief: wherever a rule LONGER than a caption (≥0.85 col) + # crosses a block's interior with real content both above and below, split the block there. + # NT (Namaste Telangana) gets the SAME split but only at rules ≥1.8 columns wide — per the + # line-pattern study a ≥2-col rule is an article boundary on NT (the boxed grid's ~1-col + # caption/box rules must NOT cut; NT Adilabad's feature-heavy flood front page over-merged + # the whole bottom band across a 4.4-col rule). AJ is untouched (whitespace layout). + # Each band's anchor is its own headline if it has one, else its topmost member. + if _ACTIVE_PAPER in ("sakshi", "namaste_telangana"): + _sk_minw = 0.85 if _ACTIVE_PAPER == "sakshi" else 1.8 + _sk_tw = sorted(r["bbox"][2] - r["bbox"][0] for r in regions if r.get("type") == "text") + _sk_col = _sk_tw[len(_sk_tw) // 2] if _sk_tw else 600 + _sk_hbars = sorted([L for L in (sep_lines or []) if not L.get("vert") and "y" in L], + key=lambda L: L["y"]) + _sk_new = [] + _sk_done = [] + for blk in list(blocks): + if blk.get("_raw_rect") or blk.get("_legal") or blk.get("_furniture"): + continue + x1, y1, x2, y2 = blk["bbox"]; bw = max(1, x2 - x1) + mems = [m for m in (set(blk["members"]) | {blk["anchor_id"]}) if m in _rl] + if len(mems) < 2: + continue + # y-tops of every "article start" in the block (a detected headline or a dateline) + starts_y = [_rl[m]["bbox"][1] for m in mems + if _rl[m].get("type") in ("doc_title", "paragraph_title") or m in _dl_l] + # Cut ONLY at a rule that (a) is longer than a caption & spans the block, (b) is NOT a + # caption underline, AND (c) has a NEW article starting just below it (a headline/dateline + # within ~170px). Condition (c) is what stops a HEADLINE-underline rule from severing a + # headline from its own body (the old bug: R67 → 47px headline + orphan body). + cuts = [L["y"] for L in _sk_hbars + if y1 + 80 < L["y"] < y2 - 80 + and (L["x2"] - L["x1"]) >= _sk_minw * _sk_col + and (min(x2, L["x2"]) - max(x1, L["x1"])) >= 0.6 * bw + and not _line_is_underline(L["y"], L["x1"], L["x2"], regions) + and (any(L["y"] - 10 <= sy <= L["y"] + 170 for sy in starts_y) + # NT: a ≥2.2-col rule is a STRUCTURAL band divider; it cuts even + # when the next band's headline isn't a member yet (NT Adilabad's + # bottom-band headline only merges in later via containment) + or (_ACTIVE_PAPER == "namaste_telangana" + and (L["x2"] - L["x1"]) >= 2.2 * _sk_col))] + if not cuts: + continue + bounds = [y1 - 1] + cuts + [y2 + 1] + bands = [] + for i in range(len(bounds) - 1): + lo, hi = bounds[i], bounds[i + 1] + bm = [m for m in mems if lo <= (_rl[m]["bbox"][1] + _rl[m]["bbox"][3]) / 2 < hi] + if bm: + bands.append(bm) + if len(bands) < 2: + continue + for bm in bands: + bxs = [_rl[m]["bbox"] for m in bm] + heads = [m for m in bm if _rl[m].get("type") in ("doc_title", "paragraph_title")] + anch = (min(heads, key=lambda m: _rl[m]["bbox"][1]) if heads + else min(bm, key=lambda m: _rl[m]["bbox"][1])) + _sk_new.append({"anchor_id": anch, "kind": ("H" if heads else blk.get("kind", "O")), + "bbox": [min(b[0] for b in bxs), min(b[1] for b in bxs), + max(b[2] for b in bxs), max(b[3] for b in bxs)], + "members": sorted(bm), "_locked": True}) + _sk_done.append(id(blk)) + print(f" ─ Sakshi rule-split: R{blk['anchor_id']} → {len(bands)} bands at rules {cuts}") + if _sk_done: + blocks = [b for b in blocks if id(b) not in _sk_done] + _sk_new + + # Step 4 — grow OPEN (non-confident) cells into surrounding EMPTY space, bounded by the + # nearest other cell (locked OR open), big vertical walls, and the page frame. LOCKED + # cells are NEVER moved. Each open cell extends each edge until just before the nearest + # neighbour/wall/frame in that direction, filling its gap WITHOUT overlapping anything. + # Grow-only (never shrinks). Greedy (uses already-grown neighbours) so two adjacent open + # cells meet at the gap instead of overlapping. + _GAP = 8 + _frame_l = min((r["bbox"][0] for r in regions), default=0) + _frame_r = max((r["bbox"][2] for r in regions), default=0) + _frame_b = max((r["bbox"][3] for r in regions), default=0) + _frame_t = min((b["bbox"][1] for b in blocks), default=0) # content top (below masthead) + _vwx = [((L["x1"] + L["x2"]) // 2 if "x" not in L else L["x"], + min(L.get("y1", 0), L.get("y2", 0)), max(L.get("y1", 0), L.get("y2", 0))) + for L in (sep_lines or []) if L.get("vert")] + for X in [b for b in blocks if not b.get("_locked")]: + x1, y1, x2, y2 = X["bbox"] + others = [b["bbox"] for b in blocks if b is not X] + + def _yband(c): + return not (c[3] <= y1 or c[1] >= y2) + + def _xband(c): + return not (c[2] <= x1 or c[0] >= x2) + # right: nearest cell-left / wall to the right, else frame right + rb = [c[0] for c in others if _yband(c) and c[0] >= x2] \ + + [wx for wx, wy1, wy2 in _vwx if wx >= x2 and not (wy2 <= y1 or wy1 >= y2)] + nx2 = min([_frame_r] + rb) - _GAP + # left + lb = [c[2] for c in others if _yband(c) and c[2] <= x1] \ + + [wx for wx, wy1, wy2 in _vwx if wx <= x1 and not (wy2 <= y1 or wy1 >= y2)] + nx1 = max([_frame_l] + lb) + _GAP + # down + db = [c[1] for c in others if _xband(c) and c[1] >= y2] + ny2 = min([_frame_b] + db) - _GAP + # up + ub = [c[3] for c in others if _xband(c) and c[3] <= y1] + ny1 = max([_frame_t] + ub) + _GAP + # grow-only; keep current edge if the computed bound would shrink + gb = [min(x1, nx1), min(y1, ny1), max(x2, nx2), max(y2, ny2)] + # Clip-after-grow: independent edge growth can poke a CORNER into a diagonal + # neighbour. Retreat any grown edge back out of every other cell — but never below + # X's ORIGINAL box — so the grown box overlaps NOTHING (locked or open). + for c in others: + if gb[2] <= c[0] or gb[0] >= c[2] or gb[3] <= c[1] or gb[1] >= c[3]: + continue + if c[0] >= x2: gb[2] = min(gb[2], max(x2, c[0] - _GAP)) # neighbour to the right + elif c[2] <= x1: gb[0] = max(gb[0], min(x1, c[2] + _GAP)) # to the left + elif c[1] >= y2: gb[3] = min(gb[3], max(y2, c[1] - _GAP)) # below + elif c[3] <= y1: gb[1] = max(gb[1], min(y1, c[3] + _GAP)) # above + # else: c overlaps X's ORIGINAL box (pre-existing) — leave it to other passes + # STRICT GUARD: Step 4 must never increase overlap with a LOCKED (confident) cell. + # If the grown box would touch any locked cell more than the original did, abandon + # the growth entirely for this cell (revert to original). + def _ovl(p, q): + return max(0, min(p[2], q[2]) - max(p[0], q[0])) * max(0, min(p[3], q[3]) - max(p[1], q[1])) + orig = [x1, y1, x2, y2] + if any(b.get("_locked") and b["bbox"] is not X["bbox"] + and _ovl(gb, b["bbox"]) > _ovl(orig, b["bbox"]) + 1 for b in blocks if b is not X): + gb = orig + X["bbox"] = gb + + # Step 5 — assign every REMAINING (unassigned) content region to a cell: the cell whose + # box CONTAINS the region's centre (smallest such). This catches regions a raw section + # rectangle covers visually but never recorded as members, and any stray content sitting + # inside an existing crop. Containment-only ⇒ the receiving box already covers it, so no + # box grows and no new overlap is created. Masthead-band / far furniture is left out. + _CONTENT5 = {"text", "image", "figure_title", "doc_title", "paragraph_title"} + _assigned5 = set() + for b in blocks: + _assigned5.update(b["members"]) + for r in regions: + if r["id"] in _assigned5 or r.get("type") not in _CONTENT5: + continue + rb = r["bbox"]; rcx = (rb[0] + rb[2]) / 2; rcy = (rb[1] + rb[3]) / 2 + if rcy < _frame_t - 5: + continue # above the content frame → furniture + cont = [b for b in blocks + if b["bbox"][0] <= rcx <= b["bbox"][2] and b["bbox"][1] <= rcy <= b["bbox"][3]] + if not cont: + continue # not inside any crop → leave (nearest-assign skipped: bbox-grow risk) + tgt = min(cont, key=lambda b: (b["bbox"][2] - b["bbox"][0]) * (b["bbox"][3] - b["bbox"][1])) + tgt["members"] = sorted(set(tgt["members"]) | {r["id"]}) + _assigned5.add(r["id"]) + + # ── Bind ORPHAN PHOTOS — an image may NEVER stand alone; it must belong to an article. ────── + # An orphan block (no doc_title/paragraph_title headline AND no dateline) that is PHOTO-DOMINANT + # (its image area ≥ 50% of the block, or it is image-only) is a lead/inline photo that lost its + # article (common on Sakshi, whose headlines PaddleOCR often misses — JG R28, Sakshi pg2 photo + # row). Attach it to the best article: the one whose top sits just BELOW the photo in the same + # column (the photo is that article's lead photo), else just above, else the nearest + # column-overlapping article. A TEXT/body orphan is LEFT ALONE — an article may be orphan, only + # a photo may not. + def _is_article(b): + return any(_rl.get(m, {}).get("type") in ("doc_title", "paragraph_title") + for m in b["members"]) or any(m in _dl_l for m in b["members"]) + _articles = [b for b in blocks if _is_article(b) + and not b.get("_furniture") and not b.get("_legal")] # photos bind to NEWS, not furniture + # A photo must not bind ACROSS a horizontal rule that is longer than a caption (>0.6 col): such + # a rule is an article separator (the user's cross-paper line rule). Used to keep e.g. a Sakshi + # top "student photo" from binding down through full-width rules into an unrelated story. + _op_hbars = [L for L in (sep_lines or []) if not L.get("vert") and "y" in L] + _op_tw = sorted(r["bbox"][2] - r["bbox"][0] for r in regions if r.get("type") == "text") + _op_col = _op_tw[len(_op_tw) // 2] if _op_tw else 600 + + def _sep_between(p, q): + lo, hi = (p[3], q[1]) if q[1] >= p[3] else (q[3], p[1]) + if hi <= lo: + return False # boxes overlap vertically → no gap rule + return any(lo - 5 < L["y"] < hi + 5 and (L["x2"] - L["x1"]) > 0.6 * _op_col + and (min(p[2], L["x2"]) - max(p[0], L["x1"])) >= 0.3 * (p[2] - p[0]) + for L in _op_hbars) + if _articles: + _ph_absorbed = [] + for ob in blocks: + if _is_article(ob) or ob.get("_furniture") or ob.get("_legal") or not ob["members"]: + continue + imgs = [m for m in ob["members"] if _rl.get(m, {}).get("type") == "image"] + if not imgs: + continue # text-only orphan → an article may orphan + ox1, oy1, ox2, oy2 = ob["bbox"] + oa = max(1, (ox2 - ox1) * (oy2 - oy1)) + ia = sum(max(0, _rl[m]["bbox"][2] - _rl[m]["bbox"][0]) + * max(0, _rl[m]["bbox"][3] - _rl[m]["bbox"][1]) for m in imgs) + body = [m for m in ob["members"] if _rl.get(m, {}).get("type") == "text"] + if body and ia < 0.5 * oa: + continue # has BODY text and not image-dominant = article → leave + # else: image-only, or image + caption (figure_title) only, or image-dominant → a PHOTO + + def _col(a): + return (min(ox2, a["bbox"][2]) - max(ox1, a["bbox"][0])) >= 0.4 * (ox2 - ox1) + # ADJACENT article (small gap ≤200px): the photo is its lead/inline photo — bind even if + # a rule sits between (that rule is the article's own headline border, not a separator). + below = [a for a in _articles if _col(a) and oy2 - 40 <= a["bbox"][1] <= oy2 + 200] + above = [a for a in _articles if _col(a) and oy1 - 200 <= a["bbox"][3] <= oy1 + 40] + if below: + tgt = min(below, key=lambda a: a["bbox"][1] - oy2) # lead photo: article just below + elif above: + tgt = min(above, key=lambda a: oy1 - a["bbox"][3]) # inline/tail photo just above + else: + # FAR fallback: nearest column-overlapping article, but NEVER across a separator rule + # (>0.6 col) — that is what kept the Sakshi student photo R52 out of R118 (a story two + # full-width rules + a page of white space away). + colov = [a for a in _articles if _col(a) and not _sep_between(ob["bbox"], a["bbox"])] + tgt = min(colov, key=lambda a: abs((a["bbox"][1] + a["bbox"][3]) / 2 + - (oy1 + oy2) / 2)) if colov else None + if tgt is None: + continue + tgt["members"] = sorted(set(tgt["members"]) | set(ob["members"])) + tgt["bbox"] = [min(tgt["bbox"][0], ox1), min(tgt["bbox"][1], oy1), + max(tgt["bbox"][2], ox2), max(tgt["bbox"][3], oy2)] + _ph_absorbed.append(id(ob)) + print(f" 🖼 Orphan photo: R{ob['anchor_id']} → article R{tgt['anchor_id']} " + f"(an image must belong to an article)") + if _ph_absorbed: + blocks = [b for b in blocks if id(b) not in _ph_absorbed] + + # ── SAKSHI: fold a text-only CONTINUATION column into the boxed article beside it. ── + # Sakshi prints multi-column boxed briefs with no internal rules; when the right + # column carries no headline/dateline of its own, the clusterer leaves it as a text + # ORPHAN (pg1 R18 = the right half of the R79 box — its first word even continues + # mid-word from the left column). Merge a text-only orphan into an article block + # sitting DIRECTLY beside it with the SAME vertical band: that is the other column + # of the same box, never a different story (a story in the next column starts at + # its own y and has its own anchor, so the band test rejects it). + if _ACTIVE_PAPER == "sakshi": + _cont_done = [] + for ob in blocks: + if (id(ob) in _cont_done or ob.get("_furniture") or ob.get("_legal") + or _is_article(ob) or not ob["members"]): + continue + if any(_rl.get(m, {}).get("type") == "image" for m in ob["members"]): + continue # photos are handled by the photo bind + ox1, oy1, ox2, oy2 = ob["bbox"] + oh = max(1, oy2 - oy1) + best = None + for tb in blocks: + if (tb is ob or id(tb) in _cont_done or not _is_article(tb) + or tb.get("_furniture") or tb.get("_legal")): + continue + tx1, ty1, tx2, ty2 = tb["bbox"] + gap = (ox1 - tx2) if ox1 >= tx2 else (tx1 - ox2) + if not (-20 <= gap <= 80): + continue # must be the directly adjacent column + vov = min(oy2, ty2) - max(oy1, ty1) + if abs(ty1 - oy1) > 90 or vov < 0.8 * min(oh, max(1, ty2 - ty1)): + continue # same vertical band = same box + fx1, fx2 = (tx2, ox1) if ox1 >= tx2 else (ox2, tx1) + if any(fx1 - 6 <= (L["x"] if "x" in L else (L["x1"] + L["x2"]) // 2) <= fx2 + 6 + and (min(max(L.get("y1", 0), L.get("y2", 0)), oy2) + - max(min(L.get("y1", 0), L.get("y2", 0)), oy1)) >= 0.5 * vov + for L in vsep): + continue # a drawn wall between → different stories + if best is None or abs(ty1 - oy1) < abs(best["bbox"][1] - oy1): + best = tb + if best is not None: + best["members"] = sorted(set(best["members"]) | set(ob["members"])) + best["bbox"] = [min(best["bbox"][0], ox1), min(best["bbox"][1], oy1), + max(best["bbox"][2], ox2), max(best["bbox"][3], oy2)] + _cont_done.append(id(ob)) + print(f" ⇤ Sakshi continuation column: orphan R{ob['anchor_id']} folded into " + f"R{best['anchor_id']} (same band, adjacent column)") + if _cont_done: + blocks = [b for b in blocks if id(b) not in _cont_done] + + # ── SAKSHI: re-home a STRAY IMAGE a wide banner anchor claimed by the column rule. ── + # A synthesized 3-column headline column-overlaps everything beneath it, so a small + # inline image far below — whose own narrow article headline does NOT column-overlap + # it — falls to the banner (JG pg1 R44: the R50 story's 'సాక్షి ఎఫెక్ట్' clipping fell + # to the SIR banner 1100px above). If an image member sits vertically DETACHED from + # the rest of its block and fully INSIDE another article's box, it belongs there. + # Also enabled for NT (JG pg1: the heat feature's photos/continuation fell into the + # H28 story 2200px below; the same re-home family cures it). AJ stays untouched. + if _ACTIVE_PAPER in ("sakshi", "namaste_telangana"): + for B in blocks: + if B.get("_raw_rect") or B.get("_section") or B.get("_furniture"): + continue + for m in [x for x in list(B["members"]) if _rl.get(x, {}).get("type") == "image"]: + mb = _rl[m]["bbox"] + # The photo's own caption travels WITH it: a figure_title directly + # below (≤90px, ≥40% overlap) is part of the unit — and must be + # excluded from the envelope test, or a stolen photo+caption pair at + # a block's bottom never reads as detached (pg2 R47+R115: the dead + # farmer's portrait stuck to the briefs column above his story). + cap = next((x for x in B["members"] + if _rl.get(x, {}).get("type") == "figure_title" + and 0 <= _rl[x]["bbox"][1] - mb[3] <= 90 + and (min(_rl[x]["bbox"][2], mb[2]) - max(_rl[x]["bbox"][0], mb[0])) + >= 0.4 * (mb[2] - mb[0])), None) + unit = [m] + ([cap] if cap is not None else []) + ubox = [min(_rl[u]["bbox"][0] for u in unit), min(_rl[u]["bbox"][1] for u in unit), + max(_rl[u]["bbox"][2] for u in unit), max(_rl[u]["bbox"][3] for u in unit)] + rest = [_rl[x]["bbox"] for x in B["members"] if x not in unit and x in _rl] + if not rest: + continue + ub = [min(b[0] for b in rest), min(b[1] for b in rest), + max(b[2] for b in rest), max(b[3] for b in rest)] + if not (ubox[1] >= ub[3] - 10 or ubox[3] <= ub[1] + 10): + continue # unit inside its block's envelope → fine + for C in blocks: + if C is B or C.get("_furniture") or C.get("_legal") or not _is_article(C): + continue + cb = C["bbox"] + if (cb[0] - 15 <= ubox[0] and ubox[2] <= cb[2] + 15 + and cb[1] - 15 <= ubox[1] and ubox[3] <= cb[3] + 15): + for u in unit: + B["members"].remove(u) + C["members"] = sorted(set(C["members"]) | set(unit)) + B["bbox"] = ub + C["bbox"] = [min(cb[0], ubox[0]), min(cb[1], ubox[1]), + max(cb[2], ubox[2]), max(cb[3], ubox[3])] + print(f" 🖼 Sakshi stray image re-home: R{unit} " + f"R{B['anchor_id']} → R{C['anchor_id']} (inside its box)") + break + + # A CAPTION (figure_title) belongs with the photo it abuts. When the photo + # folded into one block but its caption into another (the dance-photo caption + # R104 fell 3800px down its column into the statue story H106, stretching that + # crop page-tall), move the caption to the photo's block and shrink the donor + # back to its remaining members. + _cap_owner = {} + for Bc in blocks: + for m in Bc["members"]: + _cap_owner[m] = Bc + for B in blocks: + if B.get("_raw_rect") or B.get("_section") or B.get("_furniture"): + continue + for m in [x for x in list(B["members"]) + if _rl.get(x, {}).get("type") == "figure_title"]: + mb = _rl[m]["bbox"] + img = next((q for q in regions if q.get("type") == "image" + # -30 tolerance: PaddleOCR's photo box often overlaps the + # caption strip by a few px (new-run R104 vs R44: -11px) + and (-30 <= mb[1] - q["bbox"][3] <= 90 # caption below photo + or -30 <= q["bbox"][1] - mb[3] <= 90) # caption above photo + and (min(mb[2], q["bbox"][2]) - max(mb[0], q["bbox"][0])) + >= 0.4 * (mb[2] - mb[0])), None) + if img is None: + continue + C = _cap_owner.get(img["id"]) + if C is None or C is B or C.get("_furniture") or C.get("_legal"): + continue + B["members"].remove(m) + C["members"] = sorted(set(C["members"]) | {m}) + _cap_owner[m] = C + rest = [_rl[x]["bbox"] for x in B["members"] if x in _rl] + if rest: + B["bbox"] = [min(b[0] for b in rest), min(b[1] for b in rest), + max(b[2] for b in rest), max(b[3] for b in rest)] + cb = C["bbox"] + C["bbox"] = [min(cb[0], mb[0]), min(cb[1], mb[1]), + max(cb[2], mb[2]), max(cb[3], mb[3])] + print(f" 🏷 Sakshi stray caption re-home: R{m} " + f"R{B['anchor_id']} → R{C['anchor_id']} (its photo R{img['id']} lives there)") + + # A member sitting entirely ABOVE its own block's HEADLINE is never that + # article's content unless it is title-typed (kicker/deck) — body text, photos + # and captions up there are the tail of the story above that fell down-column + # to this anchor (Sakshi pg2: the NPDCL story's last lines into H100; NT JG + # pg1: the heat feature's vendor/ice-cream photos + continuation column into + # H28, 2200px above its 'ఆలస్యంపై రైతుల ఆగ్రహం' headline). If another + # article's box contains it, move it there and shrink the donor. + for B in blocks: + if (B.get("kind") != "H" or B.get("_raw_rect") or B.get("_section") + or B.get("_furniture")): + continue + _ab = _rl.get(B["anchor_id"], {}).get("bbox") + if not _ab: + continue + for m in [x for x in list(B["members"]) + if _rl.get(x, {}).get("type") in ("text", "image", "figure_title") + and _rl[x]["bbox"][3] <= _ab[1] + 10]: + mb = _rl[m]["bbox"] + host = None + for C in blocks: + if C is B or C.get("_furniture") or C.get("_legal") or not _is_article(C): + continue + cb = C["bbox"] + if (cb[0] - 15 <= mb[0] and mb[2] <= cb[2] + 15 + and cb[1] - 15 <= mb[1] and mb[3] <= cb[3] + 15): + if host is None or ((cb[2] - cb[0]) * (cb[3] - cb[1]) + < (host["bbox"][2] - host["bbox"][0]) + * (host["bbox"][3] - host["bbox"][1])): + host = C + if host is None: + continue + B["members"].remove(m) + host["members"] = sorted(set(host["members"]) | {m}) + rest = [_rl[x]["bbox"] for x in B["members"] if x in _rl] + if rest: + B["bbox"] = [min(b[0] for b in rest), min(b[1] for b in rest), + max(b[2] for b in rest), max(b[3] for b in rest)] + hb = host["bbox"] + host["bbox"] = [min(hb[0], mb[0]), min(hb[1], mb[1]), + max(hb[2], mb[2]), max(hb[3], mb[3])] + print(f" 📄 Sakshi stray text re-home: R{m} above R{B['anchor_id']}'s " + f"headline → R{host['anchor_id']} (inside its box)") + + # ── NT LATE structural-rule cut ────────────────────────────────────────────── + # The mid-pipeline rule-split runs BEFORE the grow / containment-merge / orphan- + # photo passes, so a block that only became page-tall THROUGH those later passes + # is never re-checked (NT Adilabad pg1: DL5 swallowed the flood feature and the + # bottom band across a 4.4-column rule). Re-cut FINAL blocks at structural rules: + # ≥1.8 col wide, spanning the block, not a caption underline, and either a new + # article start just below or simply ≥2.2 col (a rule that wide IS a band divider + # on NT per the line study). Locked / raw-rect / section / furniture stay whole. + if _ACTIVE_PAPER == "namaste_telangana": + _lt_tw = sorted(r["bbox"][2] - r["bbox"][0] for r in regions if r.get("type") == "text") + _lt_col = _lt_tw[len(_lt_tw) // 2] if _lt_tw else 600 + _lt_h = sorted([L for L in (sep_lines or []) if not L.get("vert") and "y" in L], + key=lambda L: L["y"]) + _lt_new, _lt_done = [], [] + for blk in list(blocks): + if (blk.get("_raw_rect") or blk.get("_section") or blk.get("_legal") + or blk.get("_furniture") or blk.get("_locked")): + continue + x1, y1, x2, y2 = blk["bbox"]; bw = max(1, x2 - x1) + mems = [m for m in (set(blk["members"]) | {blk["anchor_id"]}) if m in _rl] + if len(mems) < 2: + continue + starts_y = [_rl[m]["bbox"][1] for m in mems + if _rl[m].get("type") in ("doc_title", "paragraph_title") or m in _dl_l] + cuts = [L["y"] for L in _lt_h + if y1 + 80 < L["y"] < y2 - 80 + and (L["x2"] - L["x1"]) >= 1.8 * _lt_col + and (min(x2, L["x2"]) - max(x1, L["x1"])) >= 0.6 * bw + and not _line_is_underline(L["y"], L["x1"], L["x2"], regions) + and (any(L["y"] - 10 <= sy <= L["y"] + 170 for sy in starts_y) + or (L["x2"] - L["x1"]) >= 2.2 * _lt_col)] + if not cuts: + continue + bounds = [y1 - 1] + sorted(set(cuts)) + [y2 + 1] + bands = [] + for i in range(len(bounds) - 1): + lo, hi = bounds[i], bounds[i + 1] + bm = [m for m in mems if lo <= (_rl[m]["bbox"][1] + _rl[m]["bbox"][3]) / 2 < hi] + if bm: + bands.append(bm) + if len(bands) < 2: + continue + for bm in bands: + bxs = [_rl[m]["bbox"] for m in bm] + heads = [m for m in bm if _rl[m].get("type") in ("doc_title", "paragraph_title")] + anch = (min(heads, key=lambda m: _rl[m]["bbox"][1]) if heads + else min(bm, key=lambda m: _rl[m]["bbox"][1])) + _lt_new.append({"anchor_id": anch, "kind": ("H" if heads else blk.get("kind", "O")), + "bbox": [min(b[0] for b in bxs), min(b[1] for b in bxs), + max(b[2] for b in bxs), max(b[3] for b in bxs)], + "members": sorted(bm), "_locked": True}) + _lt_done.append(id(blk)) + print(f" ─ NT late rule-split: R{blk['anchor_id']} → {len(bands)} bands " + f"at structural rules {sorted(set(cuts))}") + if _lt_done: + blocks = [b for b in blocks if id(b) not in _lt_done] + _lt_new + + blocks.sort(key=lambda blk: (blk["bbox"][1], blk["bbox"][0])) + return blocks + + +def draw_all_article_boundaries_debug(page_img_path, regions, pages_dir, page_num, + dateline_starts=None, sep_lines=None): + """Save a per-page image outlining EVERY candidate article block on the page + BEFORE the political filter runs (see cluster_all_article_blocks). Each block + gets a distinct colour and a '# [H|DL|P] ()' label — the + full set of articles the page contains, political and not, so you can see the + raw layout that the Stage-3 political filter then narrows down.""" + try: + from PIL import ImageDraw, ImageFont + except Exception as e: + print(f" (all-article-boundaries debug image skipped: {e})") + return + blocks = cluster_all_article_blocks(regions, dateline_starts, sep_lines=sep_lines) + if not blocks: + return + try: + font = ImageFont.truetype("DejaVuSans-Bold.ttf", 34) + except Exception: + try: + font = ImageFont.load_default(size=34) + except Exception: + font = ImageFont.load_default() + + img = Image.open(page_img_path).convert("RGB") + draw = ImageDraw.Draw(img) + for idx, blk in enumerate(blocks): + x1, y1, x2, y2 = blk["bbox"] + color = _ARTICLE_COLORS[idx % len(_ARTICLE_COLORS)] + draw.rectangle([x1, y1, x2, y2], outline=color, width=12) + label = f"#{blk['anchor_id']} [{blk['kind']}] ({len(blk['members'])})" + ty = max(0, y1 - 42) + draw.rectangle([x1, ty, x1 + len(label) * 17 + 14, ty + 40], fill=color) + draw.text((x1 + 7, ty + 4), label, fill="white", font=font) + + out = str(pages_dir / f"page_{page_num:03d}_all_articles_debug.png") + img.save(out) + print(f" ALL-ARTICLE-BOUNDARIES DEBUG IMAGE saved: {out} " + f"({len(blocks)} article blocks before political filter)") + + +def draw_boundary_map_debug(page_img_path, regions, pages_dir, page_num, + dateline_starts=None, sep_lines=None): + """PRE-CROP DEBUG MAP: draw every article block's BOUNDARY + its HEADER on a BLANK + white canvas (an 'empty page' map), so each job leaves a per-page picture of exactly + where the article boxes landed BEFORE cropping — to eyeball overlaps, gaps, the + masthead band, and which header anchors each box. White = empty body; solid bar = + the header/dateline anchor; coloured outline = the article boundary. No paid API.""" + try: + from PIL import ImageDraw, ImageFont + except Exception as e: + print(f" (boundary-map debug image skipped: {e})") + return + try: + W, H = Image.open(page_img_path).size + except Exception as e: + print(f" (boundary-map debug image skipped: {e})") + return + blocks = cluster_all_article_blocks(regions, dateline_starts, sep_lines=sep_lines) + rmap = {r["id"]: r for r in regions} + dl_ids = {s["region_id"] for s in (dateline_starts or [])} + try: + font = ImageFont.truetype("DejaVuSans-Bold.ttf", 30) + except Exception: + try: + font = ImageFont.load_default(size=30) + except Exception: + font = ImageFont.load_default() + + # Local Telugu OCR (best-effort) to print each article's actual HEADER text on its box. + _ocr = None + try: + import pytesseract + if "tel" in set(pytesseract.get_languages(config="")): + _ocr = pytesseract + except Exception: + _ocr = None + _hfont = font + for _tf in ("NotoSansTelugu-Regular.ttf", + "/System/Library/Fonts/Supplemental/Telugu Sangam MN.ttc", + "/System/Library/Fonts/KohinoorTelugu.ttc", + "/usr/share/fonts/truetype/noto/NotoSansTelugu-Regular.ttf"): + try: + _hfont = ImageFont.truetype(_tf, 30) + break + except Exception: + continue + src = Image.open(page_img_path).convert("RGB") if _ocr else None + + def _header_text(blk): + """OCR the article's header: the headline region for H/P, else the top strip of + the dateline body for DL — so every box shows what story it is.""" + if not _ocr: + return "" + ar = rmap.get(blk["anchor_id"], {}) + b = ar.get("bbox") + if not b: + return "" + strip_h = min(170, b[3] - b[1]) + if strip_h <= 0: + return "" + try: + t = _ocr.image_to_string(src.crop((b[0], b[1], b[2], b[1] + strip_h)), lang="tel") + return " ".join(t.split())[:42] + except Exception: + return "" + + img = Image.new("RGB", (W, H), "white") + draw = ImageDraw.Draw(img) + + # Masthead / page-header band = everything above the topmost article box. + top_wall = min((b["bbox"][1] for b in blocks), default=0) + if top_wall > 4: + draw.rectangle([0, 0, W, top_wall], fill=(232, 232, 232)) + draw.line([0, top_wall, W, top_wall], fill=(140, 140, 140), width=5) + draw.text((14, max(2, top_wall // 2 - 16)), + "MASTHEAD / page-header band", fill=(110, 110, 110), font=font) + + for idx, blk in enumerate(blocks): + x1, y1, x2, y2 = blk["bbox"] + color = _ARTICLE_COLORS[idx % len(_ARTICLE_COLORS)] + # Furniture (classifieds / notice / unrelated content) — drawn as a hatched GREY section + # so it is visible on the map but clearly NOT an article (it is never saved as a crop). + if blk.get("_furniture"): + fc = (150, 150, 150) + for sx in range(x1, x2, 30): + draw.line([sx, y1, min(sx + 30, x2), y2], fill=fc, width=2) # diagonal hatch + draw.rectangle([x1, y1, x2, y2], outline=fc, width=6) + flab = f"FURNITURE #{blk['anchor_id']} ({len(blk['members'])})" + draw.rectangle([x1 + 5, y1 + 6, x1 + 16 + len(flab) * 14, y1 + 40], fill=fc) + draw.text((x1 + 11, y1 + 10), flab, fill="white", font=font) + continue + # header anchor: solid bar for H/P headline; outlined box for a DL dateline. + ar = rmap.get(blk["anchor_id"]) + if ar and ar.get("type") in ("doc_title", "paragraph_title"): + draw.rectangle(ar["bbox"], fill=color) + elif ar and blk["anchor_id"] in dl_ids: + draw.rectangle(ar["bbox"], outline=color, width=6) + # article boundary — LOCKED (confident) cells get a solid box; OPEN cells dashed. + locked = blk.get("_locked") + if locked: + draw.rectangle([x1, y1, x2, y2], outline=color, width=8) + else: + for sx in range(x1, x2, 40): # dashed outline for OPEN cells + draw.line([sx, y1, min(sx + 22, x2), y1], fill=color, width=8) + draw.line([sx, y2, min(sx + 22, x2), y2], fill=color, width=8) + for sy in range(y1, y2, 40): + draw.line([x1, sy, x1, min(sy + 22, y2)], fill=color, width=8) + draw.line([x2, sy, x2, min(sy + 22, y2)], fill=color, width=8) + lock_tag = "LOCK " if locked else "open " + label = f"{lock_tag}#{blk['anchor_id']} [{blk['kind']}] ({len(blk['members'])})" + ty = max(top_wall, y1) + 6 + draw.rectangle([x1 + 5, ty, x1 + 16 + len(label) * 15, ty + 36], fill=color) + draw.text((x1 + 11, ty + 3), label, fill="white", font=font) + # actual header text (OCR) on a white strip just below the id label + head = _header_text(blk) + if head: + hy = ty + 40 + draw.rectangle([x1 + 5, hy, min(x2 - 4, x1 + 24 + len(head) * 16), hy + 38], + fill="white", outline=color, width=2) + draw.text((x1 + 11, hy + 4), head, fill=(20, 20, 20), font=_hfont) + + out = str(pages_dir / f"page_{page_num:03d}_boundary_map_debug.png") + img.save(out) + print(f" BOUNDARY-MAP DEBUG IMAGE saved: {out} " + f"({len(blocks)} article boxes on blank canvas)") + + +def log_identification_breakdown(page_img_path, regions, dateline_starts, + political, rejected, page_num, sep_lines=None): + """Per-page log: the headlines CLAUDE identified (political + rejected) vs the + article blocks the CODE identified geometrically (cluster_all_article_blocks), + with each code block's anchor heading read by local Telugu OCR. No paid API. + Lets you eyeball, per page, what the model saw vs what the layout logic carved.""" + blocks = cluster_all_article_blocks(regions, dateline_starts, sep_lines=sep_lines) + rmap = {r["id"]: r for r in regions} + + # Local OCR for the code-block anchor heading (best-effort; regions store no text). + _ocr = None + try: + import pytesseract + if "tel" in set(pytesseract.get_languages(config="")): + _ocr = pytesseract + except Exception: + _ocr = None + page_img = Image.open(page_img_path) if _ocr else None + + def anchor_heading(b): + if not _ocr: + return "" + sh = min(180, b[3] - b[1]) + if sh <= 0: + return "" + try: + t = _ocr.image_to_string( + page_img.crop((b[0], b[1], b[2], b[1] + sh)), lang="tel") + return " ".join(t.split())[:60] + except Exception: + return "" + + claude_total = len(political) + len(rejected) + print(f"\n{'='*70}") + print(f"PAGE {page_num} — IDENTIFICATION BREAKDOWN") + print(f"{'='*70}") + print(f" CLAUDE IDENTIFIED COUNT : {claude_total} " + f"({len(political)} political + {len(rejected)} rejected)") + for i, a in enumerate(political, 1): + print(f" P{i:<2} {a.get('headline_telugu','?')}") + for i, r in enumerate(rejected, 1): + print(f" R{i:<2} {r.get('headline','?')}") + print(f" CODE IDENTIFIED COUNT : {len(blocks)}") + for i, blk in enumerate(blocks, 1): + a = rmap.get(blk["anchor_id"], {}) + head = anchor_heading(a.get("bbox", [0, 0, 0, 0])) or "(no OCR text)" + print(f" {i:<2} [{blk['kind']:<2}] R{blk['anchor_id']:<3} {head}") + print(f"{'='*70}\n") + + +def check_dateline_crop_integrity(regions, dateline_starts, page_num, sep_lines=None): + """Dateline-vs-crop integrity check (deterministic, no paid API). + + Rule: every OCR-detected dateline is a guaranteed article (the minimum-count + rule), and a properly cropped article carries BOTH its headline and its + dateline body. This flags all-article blocks that violate that — i.e. crops + that are split/incomplete: + • INCOMPLETE — a block anchored on a dateline body with NO detected headline + (kind 'DL'); its headline + photo most likely got cropped into a neighbour. + • MERGED — one block holds 2+ datelines (multiple articles in one crop). + Returns a list of dicts {page, anchor_id, kind, reason}; also prints a summary.""" + from collections import defaultdict + blocks = cluster_all_article_blocks(regions, dateline_starts, sep_lines=sep_lines) + dl_ids = {s["region_id"] for s in (dateline_starts or [])} + rtype = {r["id"]: r.get("type") for r in regions} + title_types = ("doc_title", "paragraph_title", "figure_title", "image") + + per_block_dls = defaultdict(list) + for blk in blocks: + for mid in blk["members"]: + if mid in dl_ids: + per_block_dls[blk["anchor_id"]].append(mid) + + issues = [] + for blk in blocks: + a = blk["anchor_id"] + if blk["kind"] == "DL": + # After grow-up, a DL block that recovered a headline/photo (a title or + # image member) is complete; flag only those still carrying none. + has_title = any(rtype.get(m) in title_types for m in blk["members"]) + if not has_title: + issues.append({"page": page_num, "anchor_id": a, "kind": "INCOMPLETE", + "reason": "dateline body with NO headline/photo found " + "above it — crop is just the body"}) + dls = per_block_dls.get(a, []) + if len(dls) >= 2: + issues.append({"page": page_num, "anchor_id": a, "kind": "MERGED", + "reason": f"{len(dls)} datelines in one crop " + f"(regions {sorted(dls)}) — articles merged"}) + + n_dl = len(dl_ids) + if issues: + print(f" ⚠ DATELINE/CROP INTEGRITY (page {page_num}): {len(issues)} issue(s) " + f"across {len(blocks)} blocks / {n_dl} datelines") + for it in issues: + print(f" [{it['kind']}] block R{it['anchor_id']}: {it['reason']}") + else: + print(f" ✅ DATELINE/CROP INTEGRITY (page {page_num}): every dateline maps " + f"to a headline-bearing crop ({len(blocks)} blocks / {n_dl} datelines)") + return issues + + +def crop_all_article_blocks(page_img_path, regions, out_base_dir, page_num, + dateline_starts=None, pad=12, sep_lines=None, + only_anchor_ids=None, overlay=False): + """Crop EVERY candidate article block (no political filtering) to its own image + so each page's articles can be eyeballed in isolation. Writes one PNG per block + into /page_/ named a_.png (order + is top-to-bottom on the page). Uses the same clustering as the debug overlay. + + When `only_anchor_ids` is given, only blocks with those anchors are SAVED, but + every other block is still whited out of the saved crops (so a dateline crop + stays clean of neighbouring orphan/other articles).""" + from PIL import ImageDraw + import numpy as np + from _lines import _faint_grey_band_below + blocks = cluster_all_article_blocks(regions, dateline_starts, sep_lines=sep_lines) + if not blocks: + return 0 + page_dir = Path(out_base_dir) / f"page_{page_num:03d}" + page_dir.mkdir(parents=True, exist_ok=True) + img = Image.open(page_img_path).convert("RGB") + W, H = img.size + grayarr = np.asarray(img.convert("L")) + rmap = {r["id"]: r for r in regions} + owner = {} + for blk in blocks: + for mid in blk["members"] + [blk["anchor_id"]]: + owner[mid] = blk["anchor_id"] + + def _intersect(a, b): + ix = min(a[2], b[2]) - max(a[0], b[0]) + iy = min(a[3], b[3]) - max(a[1], b[1]) + return ix > 0 and iy > 0 + + # A block whose rectangle is no wider than ~1.5 columns sits in a single + # column, so its bbox IS the article's true boundary — crop it straight from + # the page (no white-out) to keep display headlines PaddleOCR never boxed. + # Wider multi-column blocks are L-shaped: their rectangle overlaps neighbour + # articles, so those neighbours still need whiting out. + NARROW_W = 1100 + + saved = 0 + for order, blk in enumerate(blocks, 1): + if only_anchor_ids is not None and blk["anchor_id"] not in only_anchor_ids: + continue + if blk.get("_furniture"): + continue # classifieds / notice / unrelated furniture — NOT an article crop + # (still present in `blocks`, so it whites out of neighbour crops and + # shows as a section on the debug boundary map) + bx1, by1, bx2, by2 = blk["bbox"] + # Drop a DETACHED top banner photo. When the topmost member of a wide block + # is an image that does NOT span the article width AND a tall pure-white + # pixel gutter separates it from the content below, that photo belongs to a + # neighbouring story sharing the banner row — keeping it would leave a big + # blank corner (the neighbour's blanked column). Start the crop where the + # content resumes below the gutter instead. + mems = sorted((rmap[m] for m in blk["members"] + [blk["anchor_id"]] + if m in rmap), key=lambda r: r["bbox"][1]) + if mems and mems[0].get("type") == "image" and (bx2 - bx1) > NARROW_W: + top = mems[0]["bbox"] + below = [r for r in mems if r["bbox"][1] >= top[3] - 2] + if (top[2] - top[0]) < 0.7 * (bx2 - bx1) and below: + ny = min(r["bbox"][1] for r in below) + white = (grayarr[top[3]:ny, bx1:bx2] > 235).mean(axis=1) >= 0.99 + run = 0; resume = None + for i, w in enumerate(white): + if w: + run += 1 + else: + if run >= 30: + resume = top[3] + i + run = 0 + if resume is not None: + by1 = resume + x1 = max(0, bx1 - pad); y1 = max(0, by1 - pad) + x2 = min(W, bx2 + pad) + # If a faint grey separator rule sits just below the article (spanning its + # column width), cut the bottom edge precisely AT that line; otherwise keep + # the body-extent bottom. + band_y = _faint_grey_band_below(grayarr, blk["bbox"]) + y2 = min(H, band_y) if band_y is not None else min(H, by2 + pad) + rect = (x1, y1, x2, y2) + crop = img.crop(rect).copy() + + if not blk.get("_raw_rect"): + pristine = crop.copy() + draw = ImageDraw.Draw(crop) + + def _wbox(b, gpad=0): + cx1 = max(0, b[0] - x1 - gpad); cy1 = max(0, b[1] - y1 - gpad) + cx2 = min(x2 - x1, b[2] - x1 + gpad); cy2 = min(y2 - y1, b[3] - y1 + gpad) + if cx2 > cx1 and cy2 > cy1: + draw.rectangle([cx1, cy1, cx2, cy2], fill=(255, 255, 255)) + + if (bx2 - bx1) > NARROW_W: + for ob in blocks: + if ob["anchor_id"] == blk["anchor_id"]: + continue + if _intersect(rect, ob["bbox"]): + _wbox(ob["bbox"], gpad=8) + for r in regions: + if r["id"] in owner: + continue + if _intersect(rect, r["bbox"]): + _wbox(r["bbox"], gpad=4) + else: + # NARROW single-column crop: the rect is the article's true + # boundary, but the ±pad margin (or a clustering overlap) pulls + # in slivers of NEIGHBOUR articles — sliced glyph columns at the + # crop edges, a foreign headline poking in at the bottom. White + # out regions OWNED by other blocks; leave un-owned ink alone + # (it may be this article's own display headline that PaddleOCR + # never boxed). Members are repasted below, so an overlapping + # foreign box can never erase this article's own content. + for r in regions: + oa = owner.get(r["id"]) + if oa is None or oa == blk["anchor_id"]: + continue + if _intersect(rect, r["bbox"]): + _wbox(r["bbox"], gpad=2) + for mid in blk["members"] + [blk["anchor_id"]]: + r = rmap.get(mid) + if not r: + continue + b = r["bbox"] + cx1 = max(0, b[0] - x1); cy1 = max(0, b[1] - y1) + cx2 = min(x2 - x1, b[2] - x1); cy2 = min(y2 - y1, b[3] - y1) + if cx2 > cx1 and cy2 > cy1: + crop.paste(pristine.crop((cx1, cy1, cx2, cy2)), (cx1, cy1)) + + # Trim the outer white margin (the blank column gutter to the next + # article on the right, plus any white border on the other sides). + # A border row/column is trimmed unless ≥1% of its pixels are ink, + # so a faint sparse strip bleeding in from a neighbour is dropped + # while genuine text columns and full-width grey rules survive. + ca = np.asarray(crop.convert("L")) + ink = ca < 245 + colf = ink.mean(axis=0) + rowf = ink.mean(axis=1) + cols = np.where(colf > 0.01)[0] + rows = np.where(rowf > 0.01)[0] + trim_x = trim_y = 0 + if cols.size and rows.size: + trim_x, trim_y = int(cols.min()), int(rows.min()) + crop = crop.crop((trim_x, trim_y, + int(cols.max()) + 1, int(rows.max()) + 1)) + + if overlay: + # REVIEW overlay: draw every member region on the crop so coverage can + # be eyeballed — blue = anchor, magenta = synthesized headline band, + # green = other members. (The paid pipeline saves plain crops; only the + # replay/review path passes overlay=True.) + od = ImageDraw.Draw(crop) + offx, offy = x1 + trim_x, y1 + trim_y + for mid in blk["members"] + [blk["anchor_id"]]: + r = rmap.get(mid) + if not r: + continue + b = r["bbox"] + box = [b[0] - offx, b[1] - offy, b[2] - offx, b[3] - offy] + if box[2] <= 0 or box[3] <= 0 or box[0] >= crop.width or box[1] >= crop.height: + continue + colr = ((200, 0, 160) if r.get("_synth_band") + else (0, 90, 220) if mid == blk["anchor_id"] + else (0, 160, 0)) + od.rectangle(box, outline=colr, width=4) + od.text((box[0] + 6, box[1] + 4), + f"R{mid} {r.get('type', '')[:14]}", fill=colr) + + name = f"a{order:02d}_{blk['kind']}{blk['anchor_id']}.png" + crop.save(str(page_dir / name)) + saved += 1 + print(f" PER-ARTICLE CROPS saved: {page_dir} ({saved} article images)") + return saved + + +def _line_is_underline(ly, lx1, lx2, regions, band=45): + """RULE: a horizontal rule with a photo/figure or its caption (figure_title) + sitting DIRECTLY ABOVE it is an UNDERLINE for that image/caption — NOT an + article separator. Such a line must never be used to split or bound articles + (Rule-B floor corroboration, banner bottom rule, etc.). + + Returns True when an image / figure / figure_title region's BOTTOM edge lies + just above the line (within `band` px) and the line is roughly the WIDTH of + that image/caption. A real underline is drawn to the width of the photo it + sits under, so the line and the region span the same horizontal extent. A + full-width structural rule (e.g. a multi-column banner's bottom border) is + far wider than any single photo above it — it sticks out well past the + image on both sides — so it is NOT treated as an underline. + """ + lw = max(lx2 - lx1, 1) + for r in regions: + if str(r.get("type")) not in ("image", "figure", "figure_title"): + continue + b = r["bbox"] + if not (ly - band <= b[3] <= ly + 5): # region bottom hugs the line + continue + iw = max(b[2] - b[0], 1) + ov = min(b[2], lx2) - max(b[0], lx1) + # underline ⟺ image sits above the line AND the line is not much wider + # than the image (line width within 1.5× the image width). A banner + # bottom rule is several times wider than the photo above it -> skip. + if ov > 0.5 * min(iw, lw) and lw <= 1.5 * iw: + return True + return False + + +# distinct, high-contrast colours cycled per article boundary (RGB) +_ARTICLE_COLORS = [ + (220, 30, 30), (30, 110, 220), (0, 160, 70), (210, 120, 0), + (150, 50, 200), (0, 170, 180), (200, 30, 130), (120, 90, 30), + (60, 60, 200), (90, 160, 0), +] + + +def draw_article_boundaries_debug(page_img_path, records, pages_dir, page_num): + """Save a per-page image outlining the FINAL crop rectangle of every article + (after all geometric rules: foreign-headline clip, Rule-B floor, grow-right, + multi-column banner, and the calibration-strip bottom clamp). Each article + gets a distinct colour and a label 'p_a [BANNER] '. This is + the ground-truth view of what each article crop actually contains. + + `records` are the dicts returned by crop_political_articles (each carries the + final 'bbox'); when bbox is absent it is read back from the article info.json. + """ + try: + from PIL import ImageDraw, ImageFont + except Exception as e: + print(f" (article-boundaries debug image skipped: {e})") + return + page_recs = [r for r in records if r.get("page") == page_num] + if not page_recs: + return + try: + font = ImageFont.truetype("DejaVuSans-Bold.ttf", 34) + except Exception: + try: + font = ImageFont.load_default(size=34) + except Exception: + font = ImageFont.load_default() + + img = Image.open(page_img_path).convert("RGB") + draw = ImageDraw.Draw(img) + for idx, rec in enumerate(sorted(page_recs, key=lambda r: r.get("article_id", 0))): + b = rec.get("bbox") + if b is None: # fall back to the stored info.json + try: + info = json.loads((Path(rec["dir"]) / "info.json").read_text()) + b = info["bbox"] + except Exception: + continue + color = _ARTICLE_COLORS[idx % len(_ARTICLE_COLORS)] + draw.rectangle([b[0], b[1], b[2], b[3]], outline=color, width=14) + tag = rec.get("name", f"p{page_num:03d}_a{rec.get('article_id', idx + 1):03d}") + if rec.get("is_banner"): + tag += " [BANNER]" + hl = (rec.get("headline_telugu") or "")[:22] + label = f"{tag} {hl}" if hl else tag + ty = max(0, b[1] - 44) + draw.rectangle([b[0], ty, b[0] + len(label) * 17 + 16, ty + 42], fill=color) + try: + draw.text((b[0] + 8, ty + 4), label, fill="white", font=font) + except Exception: + draw.text((b[0] + 8, ty + 4), tag, fill="white", font=font) + + out = str(pages_dir / f"page_{page_num:03d}_articles_debug.png") + img.save(out) + print(f" ARTICLE-BOUNDARIES DEBUG IMAGE saved: {out} ({len(page_recs)} articles)") + + +# --------------------------------------------------------------------------- +# Step 3: Claude — identify political articles + group regions (ONE call) +# --------------------------------------------------------------------------- +POLITICAL_PROMPT = """You are a senior opposition party strategist in Telangana state, India. + +I am showing you a Telugu newspaper page image. I have also detected {num_regions} text/image regions on this page using OCR layout detection. Each region has an ID and precise pixel coordinates. + +REGIONS: +{region_list} + +{dateline_starts} + +YOUR TASK: Look at the newspaper page and identify ONLY the politically significant articles. For each political article, tell me which region IDs belong to it so I can crop them together precisely. + +For EACH article on the page, ask: + +QUESTION 1: "Does this article describe actual DAMAGE, LOSS, DEATH, CORRUPTION, or SUFFERING happening to people in Telangana? And can the government (state OR central) be BLAMED for it?" +- Must be a NEGATIVE EVENT that already occurred — not a plan, announcement, or statistic. +- Must have a VICTIM — someone who died, lost money, was cheated, is suffering. +- Government (Telangana state OR central/BJP) can be blamed — for causing, failing to prevent, or failing to respond. +- Central government failures qualify IF actual harm to Telangana citizens. +- RESPONSIBILITY CHECK: If damage caused by nature, individual negligence, criminals, or private parties with NO government role → NO. +- If POSITIVE (new scheme, development, achievement) → NO. +- If PLAN or ANNOUNCEMENT → NO. +- If NO victim and NO damage → NO. + +DO NOT RATIONALIZE A GOVERNMENT ANGLE THAT ISN'T REALLY THERE: +The proximate cause of the damage must clearly be a government act, a specific government policy, government inaction on a specific known issue, or systemic government failure. You may NOT stretch INDIVIDUAL CRIME articles to qualify by arguing things like: +- "The government should have provided better welfare for this community" (when the actual event was an individual crime) +- "The government should have prevented this" (when the actual cause was a private quarrel, criminal act, accident, or natural event with no government role) +- "This community suffers in general and the government doesn't help" (when the actual article is about ONE individual incident) +If you find yourself reaching for a generic "community welfare" or "should have done better" angle on an individual crime, REJECT it. + +DO QUALIFY — ongoing policy-effect articles. These count as government damage even though they aren't a one-time "event": +- Petrol/diesel/LPG price hikes that hurt consumers, drivers, transporters (central government tax policy) +- Electricity tariff hikes that burden households (state policy) +- Procurement-center failures hurting farmers (state agriculture/civil-supplies policy) +- Scheme implementation failures: pension delays, ration shortages, scholarship non-payment (specific state/central scheme) +- Inflation/price rise of essential commodities tied to specific government policy +For these, "specific damage" can be a quantified ongoing burden (Rs.X per day, Rs.Y per family per month, N% increase) rather than a one-time incident. ACCEPT them as long as a named government policy or department is the cause and there are real victims with real numbers. + +PROTESTS ARE EVIDENCE OF DAMAGE — QUALIFY THEM: +Farmer/citizen PROTESTS, rasta-rokos (road blockades), and dharnas held OVER A SPECIFIC GOVERNMENT FAILURE (procurement not happening, paddy/crops spoiling, unpaid dues, ration/pension/scholarship denial, water not released) DO QUALIFY. The protest is not "just a protest" — it is direct evidence that the underlying government failure has already harmed real victims (e.g. farmers blocking a road because grain has been sitting unprocured for a month and is sprouting). Do NOT reject these as "small", "brief", "minor", or "overlapping with another story". If it has its own headline, list it as its OWN article. (This is different from purely political/party rallies, organizational meetings, or student union protests with no specific government-caused damage — those still do NOT qualify.) + +QUESTION 2: "Can the opposition party DIRECTLY USE this article to attack the government?" + +Only include articles where BOTH answers are YES. + +GROUPING RULES — READ CAREFULLY (this is critical for correct cropping): +- An "article" is ONE story occupying ONE contiguous rectangular block on the page (headline + its body text + its photo/caption directly under or beside that headline). +- A REAL ARTICLE has a main headline (OCR will label it as `doc_title`). Every entry you return in political_articles MUST include exactly one `doc_title` region in its member_region_ids — EXCEPT for the missed-headline articles flagged in the DATELINE SCAN below, which you anchor on their dateline body region instead. +- The member regions of a single article MUST be spatially adjacent — they touch or sit immediately next to each other. Use the bbox coordinates I gave you to verify. +- DO NOT merge regions from different columns of the page into one article unless they are literally the continuation of the same article (same headline, text flows column-to-column). +- DO NOT merge two visually separate stories just because they cover similar topics. If there are two headlines, that is TWO articles — list them separately (or reject one). +- If you are unsure whether two regions belong to the same article, KEEP THEM SEPARATE. Over-grouping is worse than under-grouping. +- Before finalizing each article, mentally draw the bounding box of all its member regions and ask: "Does this rectangle contain ONLY this one story, with no other articles inside it?" If not, remove the outlier regions. + +ARTICLE BODY DATELINE — a strong signal for "this is where an article begins": +{dateline_rule} + +SUB-BOXES AND QUOTE BOXES — DO NOT EMIT THESE AS SEPARATE ARTICLES: +Newspapers often embed sub-sections inside a larger article: victim testimonials, official statements, quote-boxes, infoboxes, "what they said" panels, side-stories. These typically have: +- A small `paragraph_title` heading instead of a big `doc_title` (no doc_title at all) +- A small portrait photo of one person plus a few sentences of text +- They sit INSIDE or DIRECTLY ADJACENT to the main article's bounding area (same column, same vertical zone, often in the rightmost narrow column of the parent article) +These are PART OF THE PARENT ARTICLE — include their regions in the parent's member_region_ids. DO NOT create a separate political_articles entry for them. If you cannot find a `doc_title` to anchor an "article", it is almost certainly a sub-box of another article — fold it in or drop it. + +TELANGANA STATE politics only. Farmer issues are MOST IMPORTANT. + +DO NOT SELECT: +- International news, other states news +- Government achievements, new schemes, welfare announcements, development projects +- Newspaper masthead, headers, ads, classifieds +- Sports, entertainment, cinema, horoscopes +- General statistics without actual damage +- Resignations (unless corruption/scam/fired/forced out) +- PM Modi speeches (unless specific policy failure hurting Telangana) +- Individual accidents (road, electric shock, drowning, fire) — only if MASS disaster (10+ deaths) or CLEAR systemic government negligence +- INDIVIDUAL CRIMES AND VIOLENCE: murders, assaults, robberies, kidnappings, domestic violence, family disputes, gang violence, personal enmity attacks, suicides — these are matters between private parties or criminals and the victim. REJECT them. They qualify ONLY if there is a clear, specific, named government failure that directly caused or enabled the event (e.g., police themselves committed the crime, a specific government policy directly produced the violence, or 10+ victims in one systemic incident). A generic "the community needs more government protection" or "welfare hasn't reached them" angle is NOT enough — REJECT. +- COMMUNITY-LEVEL GENERIC GRIEVANCES: articles that just describe a community's hardships in general without a specific named government action or inaction that caused a specific damage event — REJECT. + +For each selected political article, return: +- "headline_telugu": exact Telugu headline from the page +- "headline_english": English translation +- "member_region_ids": list of region IDs that belong to this article (for precise cropping) +- "q1_reason": why this is government failure (1 sentence) +- "q2_reason": how opposition can use this (1 sentence) +- "attack_angle": what to say in press conference (1 sentence) +- "priority": "high" or "medium" +- "category": one of corruption, governance_failure, public_grievance, law_order, health, education, infrastructure, ruling_party_crisis, farmer_issues + +Also return "rejected" list with brief reasoning for EVERY rejected article. + +Return ONLY valid JSON: +{{ + "political_articles": [...], + "rejected": [ + {{"headline": "...", "reason": "..."}} + ] +}} + +If no political articles found, return {{"political_articles": [], "rejected": [...]}}""" + + +# --- Paper-specific dateline rules ----------------------------------------- +# The dateline (the place-prefix that opens an article's body) is the most +# reliable "a new article starts here" signal, but its exact form differs by +# paper. We pick the rule from the PDF filename. Common shared rules (a new +# dateline forces a split; it overrides a mis-typed headline) are appended to +# every variant. +_DATELINE_COMMON = """A text region whose first words match this dateline pattern is the START of an article's body — it belongs to the headline directly above it. + +Use the dateline as follows: +- A NEW dateline marks the START of a NEW article. If two datelines appear under what looks like one story, that is TWO articles — split them into separate entries, EVEN IF they cover the same topic and sit side by side. +- This OVERRIDES the region type label. PaddleOCR sometimes mislabels a real headline as `paragraph_title` instead of `doc_title`. If a heading sits directly above a fresh dateline, treat it as a real article headline and emit a SEPARATE article — do NOT fold it into a neighbour as a sub-box. +- A text region with NO dateline of its own (it continues the flow from the region above) is a continuation of the same article — keep it together. +- The sub-box / fold-in rule below applies ONLY to blocks that have NO dateline of their own.""" + +DATELINE_RULES = { + "sakshi": ( + "This is a SAKSHI page. Sakshi opens each article body with just a place " + "name followed by a COLON (no date):\n" + " Examples: `జనగామ:` `జనగామ రూరల్:` `:`\n" + "The reliable marker is a short place-prefix ending in a colon \":\" at the " + "start of a body block.\n\n" + _DATELINE_COMMON + ), + "andhra_jyothi": ( + "This is an ANDHRA JYOTHI page. Andhra Jyothi opens each article body with " + "place + date + (ఆంధ్రజ్యోతి) + COLON:\n" + " Examples: `వరంగల్, మే 14 (ఆంధ్రజ్యోతి):` `, (ఆంధ్రజ్యోతి):`\n" + "The reliable marker is a prefix ending in `(ఆంధ్రజ్యోతి):` — a place, a " + "date, the masthead in brackets, then a colon.\n\n" + _DATELINE_COMMON + ), + "namaste_telangana": ( + "This is a NAMASTE TELANGANA page. Namaste Telangana opens each article body " + "with place + date (Telugu month + day, e.g. మే 19) + COLON, with NO masthead:\n" + " Examples: `వరంగల్, మే 19:` `జనగామ, మే 19:` `, :`\n" + "The reliable marker is a short place-and-date prefix ending in a colon \":\" " + "(it contains a date but, unlike Andhra Jyothi, no `(ఆంధ్రజ్యోతి)` masthead).\n\n" + + _DATELINE_COMMON + ), + "unknown": ( + "Telugu newspapers open each article body with a location dateline ending in " + "a colon. Two common formats:\n" + " - Andhra Jyothi: `, (ఆంధ్రజ్యోతి):`\n" + " - Sakshi: `:` (place name + colon, no date)\n" + "The reliable marker is a short place-prefix ending in a colon \":\" at the " + "start of a body block.\n\n" + _DATELINE_COMMON + ), +} + + +def detect_paper(pdf_path): + """Identify the newspaper from the PDF filename so we can pick its dateline rule.""" + name = str(pdf_path).lower() + if "sakshi" in name: + return "sakshi" + if "namaste" in name or "namasthe" in name or "_nt_" in name or "nt_" in name: + return "namaste_telangana" + if "andhra" in name or "jyoth" in name or "_aj_" in name or "_aj" in name or "aj_" in name: + return "andhra_jyothi" + return "unknown" + + +def _build_dateline_block(dateline_starts): + """Turn the deterministic dateline scan into prompt text that tells Claude how + many articles there are and which ones have a headline PaddleOCR missed.""" + if not dateline_starts: + return ("DATELINE SCAN: (none available for this page — rely on the image and " + "region types.)") + lines = [ + "DATELINE SCAN (deterministic, reliable): each region below OPENS its body with a " + "dateline, so each is a SEPARATE article. This is a MINIMUM article count — do NOT " + "return fewer political articles than the political ones among these:", + ] + n_missed = 0 + for s in sorted(dateline_starts, key=lambda z: z["bbox"][1]): + if s.get("headline_region"): + lines.append(f" - Region {s['region_id']} (dateline \"{s['dateline']}\"): headline " + f"already detected as region {s['headline_region']}.") + else: + n_missed += 1 + lines.append(f" - Region {s['region_id']} (dateline \"{s['dateline']}\"): *** NO headline " + f"box detected *** — PaddleOCR MISSED this article's headline (it is large " + f"stylised text directly ABOVE region {s['region_id']}, which you CAN see in " + f"the image). Emit this as its OWN article anchored on region " + f"{s['region_id']}; read its headline from the image. Do NOT fold or drop it.") + if n_missed: + lines.append(f"IMPORTANT: {n_missed} article(s) above have NO detected headline box. They are " + f"real articles you must still consider — anchor each on its dateline body region.") + return "\n".join(lines) + + +def _extract_json_obj(text): + """Pull the outermost JSON object out of a model response that may be empty, + wrapped in ``` / ```json fences, or padded with prose before/after the JSON. + Raises ValueError on an empty response; lets json.JSONDecodeError propagate + for genuinely malformed JSON.""" + import re as _re + if not text or not text.strip(): + raise ValueError("empty response") + t = text.strip() + # Prefer content inside a fenced block if one exists. + m = _re.search(r"```(?:json)?\s*(.*?)```", t, _re.DOTALL) + if m: + t = m.group(1).strip() + else: + t = t.replace("```json", "").replace("```", "").strip() + # Trim any leading/trailing prose around the object braces. + if not t.startswith("{"): + i = t.find("{") + if i != -1: + t = t[i:] + if not t.endswith("}"): + j = t.rfind("}") + if j != -1: + t = t[:j + 1] + if not t: + raise ValueError("no JSON object found in response") + return json.loads(t) + + +def _salvage_political_array(text): + """Last-resort recovery when the whole JSON object will not parse — typically a + stray character somewhere in the (log-only) ``rejected`` list breaks the document + even though the ``political_articles`` array itself is well-formed. Locate just + that array by string-aware bracket matching and parse it on its own. Returns the + list of articles, or None if it cannot be recovered.""" + import re as _re + if not text: + return None + m = _re.search(r'"political_articles"\s*:\s*\[', text) + if not m: + return None + start = m.end() - 1 # index of the opening '[' + depth = 0 + in_str = False + esc = False + for j in range(start, len(text)): + c = text[j] + if in_str: + if esc: + esc = False + elif c == "\\": + esc = True + elif c == '"': + in_str = False + continue + if c == '"': + in_str = True + elif c == "[": + depth += 1 + elif c == "]": + depth -= 1 + if depth == 0: + try: + arr = json.loads(text[start:j + 1]) + return arr if isinstance(arr, list) else None + except json.JSONDecodeError: + return None + return None + + +def identify_and_group_political_articles(page_img_path, regions, page_num, paper="unknown", + dateline_starts=None): + api_key = os.environ.get("ANTHROPIC_API_KEY") + if not api_key: + print("ERROR: Set ANTHROPIC_API_KEY environment variable") + sys.exit(1) + + import anthropic + client = anthropic.Anthropic(api_key=api_key) + + # Resize image for API + img = Image.open(page_img_path) + max_w = 1500 + if img.width > max_w: + ratio = max_w / img.width + img = img.resize((max_w, int(img.height * ratio)), Image.LANCZOS) + + buf = BytesIO() + img.convert("RGB").save(buf, format="JPEG", quality=80) + img_data = base64.standard_b64encode(buf.getvalue()).decode("utf-8") + + # Build region list + region_lines = [] + for r in regions: + region_lines.append(f" ID {r['id']}: type={r['type']}, bbox=[{r['bbox'][0]}, {r['bbox'][1]}, {r['bbox'][2]}, {r['bbox'][3]}]") + region_list = "\n".join(region_lines) + + dateline_rule = DATELINE_RULES.get(paper, DATELINE_RULES["unknown"]) + prompt = POLITICAL_PROMPT.format(num_regions=len(regions), region_list=region_list, + dateline_rule=dateline_rule, + dateline_starts=_build_dateline_block(dateline_starts)) + + print(f" Sending page {page_num} to Claude Opus 4.8 ({len(regions)} regions)...") + + def _call(strict=False): + ptext = prompt + if strict: + ptext += ("\n\nIMPORTANT: Return ONLY the raw JSON object described above — " + "no explanation, no markdown code fences, nothing before or after it.") + return client.messages.create( + model="claude-opus-4-8", + max_tokens=8192, + messages=[{"role": "user", "content": [ + {"type": "image", "source": {"type": "base64", "media_type": "image/jpeg", "data": img_data}}, + {"type": "text", "text": ptext}, + ]}], + ) + + def _text_of(resp): + return "".join(getattr(b, "text", "") for b in resp.content + if getattr(b, "type", None) == "text").strip() + + dbg = Path(page_img_path).with_name(f"page_{page_num:03d}.badresponse.txt") + + response = _call() + text = _text_of(response) + first_text = text + try: + result = _extract_json_obj(text) + except (ValueError, json.JSONDecodeError) as e: + try: + dbg.write_text(f"stop_reason={response.stop_reason}\n\n{text}") + except Exception: + pass + print(f" ⚠️ Page {page_num}: response was not valid JSON ({e}); " + f"stop_reason={response.stop_reason}. Retrying once (strict)...") + response = _call(strict=True) + text = _text_of(response) + try: + result = _extract_json_obj(text) + except (ValueError, json.JSONDecodeError) as e2: + try: + dbg.write_text(f"stop_reason={response.stop_reason}\n\n{text}") + except Exception: + pass + # Both parses failed outright. Before giving up, try to salvage just the + # political_articles array from either response — a stray char in the + # log-only `rejected` list should not cost us a whole page of articles. + salvaged = _salvage_political_array(first_text) or _salvage_political_array(text) + if salvaged is not None: + print(f" ♻️ Page {page_num}: full JSON was malformed but recovered " + f"{len(salvaged)} political article(s) from the partial response. " + f"Raw response saved to {dbg.name}") + return {"political_articles": salvaged, "rejected": []} + print(f" ❌ Page {page_num}: still no valid JSON after retry " + f"(stop_reason={response.stop_reason}). Treating page as having no " + f"political articles so the rest of the run continues. " + f"Raw response saved to {dbg.name}") + return {"political_articles": [], "rejected": []} + + # Print reasoning + political = result.get("political_articles", []) + rejected = result.get("rejected", []) + + print(f"\n=== PAGE {page_num} RESULTS ===") + print(f" Selected: {len(political)} political articles") + for i, art in enumerate(political, 1): + print(f" ✅ {i}. {art.get('headline_english', '?')}") + print(f" Regions: {art.get('member_region_ids', [])}") + print(f" Q1: {art.get('q1_reason', '')}") + print(f" Q2: {art.get('q2_reason', '')}") + print(f" Rejected: {len(rejected)} articles") + for r in rejected: + print(f" ❌ {r.get('headline', '?')[:50]} — {r.get('reason', '')[:60]}") + print(f"=== END PAGE {page_num} ===\n") + + return result + + +# --------------------------------------------------------------------------- +# Safeguard: split articles whose member regions form spatially disjoint clusters. +# +# Claude sometimes bundles regions from different stories into one article +# (e.g. p001_a001 covering both "Petrol Shock" and the Indiramma housing article +# in the same group). Geometrically, those member regions form two or more +# disconnected clusters. This function detects that and splits them up. +# --------------------------------------------------------------------------- +def _bboxes_connected(a, b, gap=60): + """Two bboxes are connected if they overlap or sit within `gap` pixels of each other.""" + ax1, ay1, ax2, ay2 = a + bx1, by1, bx2, by2 = b + dx = max(0, max(bx1 - ax2, ax1 - bx2)) + dy = max(0, max(by1 - ay2, ay1 - by2)) + return dx <= gap and dy <= gap + + +def _cluster_regions(member_ids, region_map, gap=60): + """Group region IDs into connected components via union-find on bbox proximity.""" + ids = [rid for rid in member_ids if rid in region_map] + if not ids: + return [] + parent = {rid: rid for rid in ids} + + def find(x): + while parent[x] != x: + parent[x] = parent[parent[x]] + x = parent[x] + return x + + def union(x, y): + rx, ry = find(x), find(y) + if rx != ry: + parent[rx] = ry + + for i, a in enumerate(ids): + for b in ids[i + 1:]: + if _bboxes_connected(region_map[a]["bbox"], region_map[b]["bbox"], gap): + union(a, b) + + clusters = {} + for rid in ids: + clusters.setdefault(find(rid), []).append(rid) + # Sort clusters by area (largest first) — main story should come first. + def area(group): + bs = [region_map[r]["bbox"] for r in group] + return (max(b[2] for b in bs) - min(b[0] for b in bs)) * \ + (max(b[3] for b in bs) - min(b[1] for b in bs)) + return sorted(clusters.values(), key=area, reverse=True) + + +# --------------------------------------------------------------------------- +# Headline verification via Claude vision (catches headline hallucinations). +# +# Claude sometimes claims a doc_title region contains one headline while the +# region actually contains a totally different article (e.g. labelling a +# US-Iran peace deal as "Deputy Collector's illegal assets"). The region IDs +# are honest but the text content is fabricated, so geometric safeguards +# cannot catch it. We crop the doc_title region and send JUST that crop to +# Claude with a tightly scoped prompt — "transcribe the Telugu text" — and +# compare to the headline Claude originally reported. PaddleOCR's Telugu +# model is unreliable, so we use Claude vision for this check. +# --------------------------------------------------------------------------- + + +def _normalize_telugu(s): + """Strip whitespace and punctuation for comparison.""" + if not s: + return "" + return "".join(ch for ch in s if ch.isalnum() or ord(ch) > 127) + + +def _claude_transcribe_region(client, page_img_path, bbox, pad=6): + """Crop the bbox from the page image and ask Claude to transcribe its text.""" + import anthropic + img = Image.open(page_img_path) + x1, y1, x2, y2 = bbox + crop = img.crop((max(0, x1 - pad), max(0, y1 - pad), + min(img.width, x2 + pad), min(img.height, y2 + pad))) + buf = BytesIO() + crop.convert("RGB").save(buf, format="JPEG", quality=90) + img_data = base64.standard_b64encode(buf.getvalue()).decode("utf-8") + + prompt = ( + "This image is a cropped newspaper headline region. " + "Transcribe the visible Telugu text EXACTLY as it appears. " + "Return ONLY the transcribed Telugu text, no English translation, " + "no explanation, no quotes. If the image is blank or unreadable, " + "return exactly the word: UNREADABLE" + ) + + resp = client.messages.create( + model="claude-haiku-4-5-20251001", + max_tokens=200, + messages=[{"role": "user", "content": [ + {"type": "image", "source": {"type": "base64", + "media_type": "image/jpeg", + "data": img_data}}, + {"type": "text", "text": prompt}, + ]}], + ) + return resp.content[0].text.strip() + + +def verify_headlines_with_claude(political_articles, regions, page_img_path, + min_similarity=0.40): + """ + For each political article, crop the doc_title region Claude pointed to and + ask Claude (Haiku) to transcribe the actual Telugu text. Compare against + the headline Claude originally claimed. + + If the claimed region's text doesn't match the claimed headline, SEARCH + every other doc_title region on the page for a match. If a different + doc_title's transcription matches the claimed headline, REMAP the article + onto that region (the downward-growth safeguard will then fill in body + regions). If no region matches, drop the article as hallucinated. + """ + if not political_articles: + return political_articles + + api_key = os.environ.get("ANTHROPIC_API_KEY") + if not api_key: + print(" (Headline verification skipped — ANTHROPIC_API_KEY not set)") + return political_articles + + try: + import anthropic + except ImportError: + print(" (Headline verification skipped — anthropic package not installed)") + return political_articles + + client = anthropic.Anthropic(api_key=api_key) + region_map = {r["id"]: r for r in regions} + all_doc_titles = [r for r in regions if r.get("type") == "doc_title"] + + # Spot oversized display banners (lead/section headlines set in large + # stylized type). OCR reliably mangles such headlines, so a low similarity + # there is NOT evidence of a hallucination and must not trigger a drop. + # The earlier median*1.8 rule was brittle: on a page with several big + # headers the median rises and a genuine banner (e.g. a 493px headline on + # a page whose median is 278) falls just under the bar. Use the SMALLEST + # doc_title (the typical body headline) as the baseline instead, plus an + # absolute floor, so a banner is caught regardless of what else is on the + # page. + _dt_heights = sorted(r["bbox"][3] - r["bbox"][1] for r in all_doc_titles) + _min_dt_h = _dt_heights[0] if _dt_heights else 0 + _median_dt_h = _dt_heights[len(_dt_heights) // 2] if _dt_heights else 0 + def _is_display_banner(rid): + h = region_map[rid]["bbox"][3] - region_map[rid]["bbox"][1] + if h < 220: # normal column headlines never qualify + return False + return (h >= 300 # absolute display floor + or (_min_dt_h and h >= 2.0 * _min_dt_h) # 2x the body headline + or (_median_dt_h and h >= 1.6 * _median_dt_h)) + + def _ocr_unreadable(txt): + # OCR failed to produce a usable transcription — treat as "can't verify", + # never as proof of a hallucination. + if not txt: + return True + t = txt.strip() + return t.upper() == "UNREADABLE" or len(_normalize_telugu(t)) < 6 + + # Cache transcriptions across articles so we don't re-transcribe the same + # doc_title region multiple times. + cache = {} + def transcribe(rid): + if rid in cache: + return cache[rid] + try: + txt = _claude_transcribe_region(client, page_img_path, + region_map[rid]["bbox"]) + except Exception as e: + print(f" (transcription error for region {rid}: {e})") + txt = None + cache[rid] = txt + return txt + + def sim_to_claim(rid, claim_norm): + actual = transcribe(rid) + if not actual or actual.strip().upper() == "UNREADABLE": + return 0.0, actual + return SequenceMatcher(None, _normalize_telugu(actual), claim_norm).ratio(), actual + + kept = [] + for art in political_articles: + member_ids = art.get("member_region_ids", []) or [] + titles_in = [rid for rid in member_ids + if region_map.get(rid, {}).get("type") == "doc_title"] + claimed = (art.get("headline_telugu") or "").strip() + label = (art.get("headline_english") or "?")[:50] + + if not titles_in or not claimed: + kept.append(art) + continue + + primary_id = min(titles_in, key=lambda r: region_map[r]["bbox"][1]) + claim_norm = _normalize_telugu(claimed) + if not claim_norm: + kept.append(art) + continue + + sim, actual = sim_to_claim(primary_id, claim_norm) + if sim >= min_similarity: + print(f" ✓ headline verified: '{label}' at region {primary_id} (sim={sim:.2f})") + kept.append(art) + continue + + # Wrapped-headline guard: OCR of the anchor often captures only the + # first line of a headline that wraps across several regions (e.g. the + # anchor reads "వంట గ్యాస్" while the full headline is "వంట గ్యాస్ + # కొరతను నివారించాలి"), which scores low against the full claim. If the + # anchor's OCR is contained in the claim (or vice-versa), it IS the + # right headline — keep it and do NOT go hunting for another region to + # remap onto. Skipping this is what let the cooking-gas article get + # remapped onto the health headline. + actual_norm = _normalize_telugu(actual or "") + if (actual_norm and len(actual_norm) >= 5 + and (actual_norm in claim_norm or claim_norm in actual_norm)): + print(f" ✓ headline partial-match kept: '{label}' at region {primary_id} " + f"(anchor OCR {actual_norm[:18]!r} is a fragment of the claim; sim={sim:.2f})") + kept.append(art) + continue + + # Mismatch — search other doc_titles for the actual headline. + print(f" ⚠ headline mismatch for '{label}' at region {primary_id} " + f"(sim={sim:.2f}): claimed={claimed[:40]!r}, " + f"actual={(actual or '')[:40]!r}") + print(f" searching {len(all_doc_titles)-1} other doc_titles for the real region...") + + best_id, best_sim, best_text = None, sim, actual + for r in all_doc_titles: + if r["id"] == primary_id: + continue + s, txt = sim_to_claim(r["id"], claim_norm) + if s > best_sim: + best_sim, best_id, best_text = s, r["id"], txt + + # Remap needs a clear bar, but not so high that Telugu OCR noise on a + # GENUINE match (which routinely lands ~0.65-0.80, since OCR garbles a few + # characters) gets rejected. Unrelated headlines that merely share the + # "...చాలి / ...ించాలి" ending top out around ~0.52, so a 0.62 bar plus the + # requirement that the match clearly beats the wrong anchor (by >=0.15) + # accepts a real, OCR-degraded match while still rejecting suffix collisions. + remap_min = 0.62 + if best_id is not None and best_sim >= remap_min and (best_sim - sim) >= 0.15: + print(f" ↻ REMAPPED '{label}' from region {primary_id} → region {best_id} " + f"(sim={best_sim:.2f}, text={(best_text or '')[:40]!r})") + # Start fresh with just the new doc_title; downward growth will + # fill in the article's body regions. + art["member_region_ids"] = [best_id] + art["_remapped_from"] = primary_id + art["_remapped_to"] = best_id + kept.append(art) + elif _is_display_banner(primary_id) or _ocr_unreadable(best_text): + # Either an oversized display banner, or OCR simply couldn't read + # the headline. In both cases the low similarity is an OCR failure, + # NOT evidence that Claude hallucinated the article — every article + # is anchored to a real doc_title region Claude saw in the layout. + # Dropping here is what made legit articles (e.g. the stylized + # "సూక్ష్మ సేద్యం.. తగ్గిన రాయితీ" banner) vanish between runs. + # Trust Claude's selection and keep the article. + reason = ("display banner too large for reliable OCR" + if _is_display_banner(primary_id) + else "OCR returned unreadable/garbage text") + h = region_map[primary_id]['bbox'][3] - region_map[primary_id]['bbox'][1] + print(f" 🛡 KEPT (unverified): '{label}' at region {primary_id} — {reason} " + f"(sim={best_sim:.2f}, h={h}px, min/median doc_title h={_min_dt_h}/{_median_dt_h}px); " + f"trusting Claude's selection") + art["_headline_unverified"] = True + kept.append(art) + else: + # OCR produced a confident, readable headline that clearly differs + # from the claim AND matches no other region — a real hallucination. + print(f" ✗ HALLUCINATION DROPPED: '{label}' — region {primary_id} OCRs to " + f"a different readable headline and no doc_title matches the claim " + f"(best sim={best_sim:.2f}, ocr={(best_text or '')[:40]!r})") + return kept + + +def _drop_blocked_by_other_doc_title(member_ids, region_map, all_regions, + col_overlap_threshold=0.3): + """ + Drop member regions that "belong" to a different article based on column-aware + headline blocking. For each non-headline member region R of this article: + + Look at every OTHER doc_title D on the page (not this article's headline). + If D sits vertically between this article's headline and R, AND D's column + overlaps significantly with R's column, AND D's column does NOT overlap + with this article's headline → R is on the wrong side of D's headline and + most likely belongs to D's article. Drop R. + + Also drop any doc_title region in `member_ids` that isn't this article's + primary (topmost) doc_title — a second headline is always wrong here. + """ + titles_in = [rid for rid in member_ids + if region_map.get(rid, {}).get("type") == "doc_title"] + if not titles_in: + return member_ids, [] + + primary_id = min(titles_in, key=lambda r: region_map[r]["bbox"][1]) + primary_bbox = region_map[primary_id]["bbox"] + pt_x1, pt_y1, pt_x2, pt_y2 = primary_bbox + + other_doc_titles = [r for r in all_regions + if r.get("type") == "doc_title" + and r["id"] != primary_id] + + def col_overlap_ratio(a, b): + ov = max(0, min(a[2], b[2]) - max(a[0], b[0])) + return ov / max(1, min(a[2] - a[0], b[2] - b[0])) + + kept, dropped = [], [] + for rid in member_ids: + r = region_map.get(rid) + if r is None: + continue + + # Always keep the primary headline. + if rid == primary_id: + kept.append(rid) + continue + + # Drop any OTHER doc_title in the member list — it's another article's headline. + if r.get("type") == "doc_title": + dropped.append(rid) + continue + + r_bbox = r["bbox"] + blocked = False + for ot in other_doc_titles: + ot_bbox = ot["bbox"] + # ot must sit vertically between the article's headline and R. + if not (pt_y2 < ot_bbox[1] and ot_bbox[3] <= r_bbox[3]): + continue + # ot's column must overlap R's column. + if col_overlap_ratio(ot_bbox, r_bbox) < col_overlap_threshold: + continue + # ot's column must NOT overlap this article's headline column — + # otherwise it could be a continuation, not a barrier. + if col_overlap_ratio(ot_bbox, primary_bbox) >= col_overlap_threshold: + continue + blocked = True + break + + if blocked: + dropped.append(rid) + else: + kept.append(rid) + + return kept, dropped + + +def _trim_above_headline(member_ids, region_map, tolerance=30): + """ + Drop member regions that sit entirely ABOVE the article's headline. + In a newspaper, an article's `doc_title` marks the TOP of the story — + anything whose bottom edge is above the headline belongs to the article + above and must not be included. `tolerance` allows a small overlap + (e.g. a decorative line or bullet that sits a few pixels above). + """ + titles = [ + rid for rid in member_ids + if rid in region_map and region_map[rid].get("type") == "doc_title" + ] + if not titles: + return member_ids, [] # no headline to anchor on — leave as-is + + # Topmost headline = smallest y1 + top_y1 = min(region_map[rid]["bbox"][1] for rid in titles) + boundary = top_y1 - tolerance + + kept, dropped = [], [] + for rid in member_ids: + r = region_map.get(rid) + if not r: + continue + # Always keep doc_title regions themselves. + if r.get("type") == "doc_title": + kept.append(rid) + continue + y2 = r["bbox"][3] + if y2 < boundary: + dropped.append(rid) + else: + kept.append(rid) + return kept, dropped + + +def _grow_article_downward(member_ids, region_map, all_regions, claimed_globally, + gap=50, x_tolerance=80, headline_continuation_gap=80, + dateline_starts=None): + """ + Pull in UNCLAIMED regions below the article's headline that are spatially + adjacent to the existing members. Bounded by: + - top: article's headline (with small tolerance) + - bottom: the next doc_title on the page (or page bottom) + - sides: the article's current horizontal extent (so we don't grow + into a neighboring column that belongs to another article) + + Also handles multi-line headlines where OCR splits one headline into + multiple stacked doc_title regions: any doc_title sitting immediately + below the article's headline in the same column (gap <= 80 px) is treated + as a headline-continuation line, folded into the article, and NOT counted + as the next-article floor. + + Claude sometimes under-includes body text regions; this catches them. + """ + titles_in = [rid for rid in member_ids + if region_map.get(rid, {}).get("type") == "doc_title"] + if not titles_in: + return member_ids, [] + + primary_id = min(titles_in, key=lambda r: region_map[r]["bbox"][1]) + headline_y1 = region_map[primary_id]["bbox"][1] + + # Horizontal extent of the article's current members — this defines the + # columns the article occupies. We allow `x_tolerance` slack on either side. + member_boxes = [region_map[rid]["bbox"] for rid in member_ids if rid in region_map] + art_x1 = min(b[0] for b in member_boxes) - x_tolerance + art_x2 = max(b[2] for b in member_boxes) + x_tolerance + + def _shares_column(other_bbox): + return max(0, min(art_x2, other_bbox[2]) - max(art_x1, other_bbox[0])) > 0 + + def _shares_column_floor(other_bbox): + # A FLOOR (next article's headline/dateline) must genuinely sit in our + # columns — not merely clip the x_tolerance padding margin. A neighbouring + # column's article whose left edge falls within our slop overlaps by only a + # few px; treating it as a floor wrongly truncates our OWN body that extends + # a little past it. Require the overlap to exceed the padding we added. + return max(0, min(art_x2, other_bbox[2]) - max(art_x1, other_bbox[0])) > x_tolerance + + # ── Detect multi-line headline ────────────────────────────────────────── + # Any doc_title sitting immediately below the current headline bottom in + # the same column is treated as a continuation line of the same headline. + # Iterate so we catch 3+ line headlines. + continuation_ids = [] + current_bottom = region_map[primary_id]["bbox"][3] + candidates_for_continuation = [ + r for r in all_regions + if r.get("type") == "doc_title" + and r["id"] != primary_id + and _shares_column(r["bbox"]) + ] + progressed = True + while progressed: + progressed = False + for r in candidates_for_continuation: + if r["id"] in continuation_ids: + continue + top, bot = r["bbox"][1], r["bbox"][3] + gap_above = top - current_bottom + if 0 <= gap_above <= headline_continuation_gap: + continuation_ids.append(r["id"]) + current_bottom = max(current_bottom, bot) + progressed = True + headline_bottom = current_bottom + + # Find the next REAL doc_title floor — one that's NOT a headline continuation. + other_titles_y = sorted( + r["bbox"][1] for r in all_regions + if r.get("type") == "doc_title" + and r["id"] != primary_id + and r["id"] not in continuation_ids + and r["bbox"][1] > headline_bottom + 30 + and _shares_column_floor(r["bbox"]) + ) + floor = other_titles_y[0] if other_titles_y else float("inf") + + # ── Identify an ORPHANED headline's own body dateline ──────────────────── + # When Claude returns an article as HEADLINE ONLY (no body member of its own), + # the body that flows directly beneath the headline usually opens with this + # article's OWN dateline. That dateline must NOT act as a downward floor (it is + # our body, not the next story). Treat the topmost same-column dateline below the + # headline — provided no doc_title sits between it and the headline — as our own + # opening dateline. + _title_types = {"doc_title", "paragraph_title", "figure_title"} + has_body = any(region_map.get(rid, {}).get("type") not in _title_types + for rid in member_ids) + own_dateline_id = None + if not has_body and dateline_starts: + # Prefer a dateline body the scan already matched to THIS headline. + _own = [s for s in dateline_starts + if s.get("headline_region") == primary_id + and _shares_column_floor(s["bbox"]) + and s["bbox"][1] > headline_bottom - 10 + and s["region_id"] not in member_ids] + if _own: + own_dateline_id = min(_own, key=lambda s: s["bbox"][1])["region_id"] + else: + # Otherwise the topmost same-column dateline below the headline, as long + # as no doc_title sits between it and the headline, is our own body. + _col_dls = sorted( + (s for s in dateline_starts + if _shares_column_floor(s["bbox"]) + and s["bbox"][1] > headline_bottom - 10 + and s["region_id"] not in member_ids + and s.get("headline_region") != primary_id), + key=lambda s: s["bbox"][1]) + if _col_dls and _col_dls[0]["bbox"][1] <= floor: + own_dateline_id = _col_dls[0]["region_id"] + + # DATELINE FLOOR: the next article's dateline is a hard downward boundary — it + # works even when that next article's HEADLINE was missed by PaddleOCR. We only + # treat a dateline as a floor if it belongs to a DIFFERENT article: skip this + # article's own opening dateline (it sits right under the headline, and/or its + # scan-matched headline is this one). + if dateline_starts: + for s in dateline_starts: + b = s["bbox"] + if not _shares_column_floor(b): + continue + if b[1] <= headline_bottom + 150: # too close = this article's own dateline + continue + if s.get("headline_region") == primary_id: # explicitly this article's own + continue + if s["region_id"] == own_dateline_id: # orphaned headline's own body dateline + continue + floor = min(floor, b[1]) + + def _center_in_article_columns(r): + b = r["bbox"] + center_x = (b[0] + b[2]) / 2 + return art_x1 <= center_x <= art_x2 + + # Candidate pool: unclaimed regions in the article's y-band AND within + # the article's horizontal column span. + candidates = [ + r for r in all_regions + if r["id"] not in claimed_globally + and r["id"] not in member_ids + and r["id"] not in continuation_ids + and r["bbox"][1] >= headline_y1 - 30 + and r["bbox"][3] <= floor + 30 + and _center_in_article_columns(r) + ] + + # Seed the kept set with the original members + any continuation doc_titles + # (multi-line headline pieces). They count as "added by growth" so they + # surface in logs and show up in the crop. + kept = list(member_ids) + kept_set = set(kept) + added = [] + for cid in continuation_ids: + if cid not in kept_set: + kept.append(cid) + kept_set.add(cid) + added.append(cid) + + # ORPHANED-HEADLINE SEED: when Claude returns an article as headline-only, the + # first body line sits a little too far below the headline to connect directly + # (sub-head / standfirst spacing exceeds `gap`). The dateline scan, however, + # already matched a body region to THIS headline (own_dateline_id). Seed it so + # adjacency growth can chain the rest of the body from it. This anchors on a + # KNOWN body region rather than loosening the gap, so growth still stops at the + # gap before the next (possibly OCR-missed) article's photo/headline. + if (own_dateline_id is not None and own_dateline_id not in kept_set + and own_dateline_id not in claimed_globally + and own_dateline_id in region_map): + kept.append(own_dateline_id) + kept_set.add(own_dateline_id) + added.append(own_dateline_id) + + # Iteratively grow: any candidate that touches the current set is added. + changed = True + while changed: + changed = False + for r in list(candidates): + if r["id"] in kept_set: + continue + for k in kept_set: + if k not in region_map: + continue + if _bboxes_connected(region_map[k]["bbox"], r["bbox"], gap): + kept.append(r["id"]) + kept_set.add(r["id"]) + added.append(r["id"]) + changed = True + break + + # ── Rightward column extension ────────────────────────────────────────── + # The growth above is bounded by the article's CURRENT column span, so a body + # that wraps into a column the HEADLINE does not sit over is never reached + # (headline over cols 1–2, body actually uses cols 1–3). Step into the column + # immediately to the right and pull in unclaimed continuation blocks. + # + # Purely ADDITIVE: it only runs after the span-bounded growth, and only adds + # regions that pass every gate below — so an article with no adjacent + # continuation column is returned unchanged. Each candidate must: + # • be unclaimed and not already kept, + # • NOT be a headline itself (doc_title) and NOT open its own dateline + # (either would mean a different article begins there), + # • sit in the next column to the right (its left edge just past our right + # edge) and touch the kept set (adjacency), and + # • sit ABOVE the first headline/dateline in ITS OWN column (per-column + # floor) — so a different story stacked lower in that column is left out. + def _hoverlap(a, b): + return max(0, min(a[2], b[2]) - max(a[0], b[0])) + + def _voverlap(a, b): + return max(0, min(a[3], b[3]) - max(a[1], b[1])) + + dateline_body_ids = {ds["region_id"] for ds in (dateline_starts or [])} + + def _column_floor(r): + """Top of the first NEW-article headline/dateline in r's column, below our + headline. r is only eligible if it sits entirely above this line.""" + # A neighbour headline can start at nearly the SAME y as our own (parallel + # columns), so we exclude our own titles by IDENTITY (not by y) and accept + # any other title at/below our headline top. The `headline_y1 - 30` gate + # keeps mastheads ABOVE the headline from acting as a floor. + ys = [t["bbox"][1] for t in all_regions + if t.get("type") == "doc_title" + and t["id"] != primary_id and t["id"] not in continuation_ids + and t["id"] not in member_ids + and t["bbox"][1] > headline_y1 - 30 + and _hoverlap(t["bbox"], r["bbox"]) > 0] + # A parallel/neighbouring article's headline is often a paragraph_title + # (not a doc_title) — e.g. two stories side by side, or a boxed item heading + # the next column. Treat any FOREIGN paragraph_title the same as a doc_title + # floor: if it hoverlaps the candidate's column and sits at/below our headline + # top, the candidate must stay ABOVE it. This means we only step into a + # continuation column from its TOP (unheaded text flowing from our column); + # if the column is headed by someone else's title, we stop. Under-reaching a + # continuation is safe — bleeding a neighbour's article is not. + for t in all_regions: + if (t.get("type") == "paragraph_title" + and t["id"] != primary_id and t["id"] not in continuation_ids + and t["id"] not in member_ids + and t["bbox"][1] > headline_y1 - 30 + and _hoverlap(t["bbox"], r["bbox"]) > 0): + ys.append(t["bbox"][1]) + for ds in (dateline_starts or []): + db = ds["bbox"] + if (ds.get("headline_region") != primary_id + and ds["region_id"] != own_dateline_id + and db[1] > headline_y1 - 30 + and _hoverlap(db, r["bbox"]) > 0): + ys.append(db[1]) + return min(ys) if ys else float("inf") + + col_gap = 160 # gutter + slack between adjacent columns + changed = True + while changed: + changed = False + for r in all_regions: + rid = r["id"] + if (rid in kept_set or rid in claimed_globally or rid in member_ids + or r.get("type") == "doc_title" or rid in dateline_body_ids): + continue + rb = r["bbox"] + # Within the article's vertical band, above its column's next headline. + if rb[1] < headline_y1 - 30 or rb[3] > _column_floor(r) + 30: + continue + # The reading flow must WRAP into this column: a kept *body text* member + # has to sit immediately to its LEFT and overlap it vertically. This is + # what separates a genuine continuation column from an unrelated story + # that merely sits adjacent (whose left neighbour is a masthead, photo, + # or another article — not our body text). + wraps = any( + k in region_map + and region_map[k].get("type") == "text" + and 0 <= rb[0] - region_map[k]["bbox"][2] <= col_gap + and _voverlap(region_map[k]["bbox"], rb) >= 60 + for k in kept_set + ) + if not wraps: + continue + # GUARD: don't wrap into a region that is HEADED by its own column header. + # A neighbour article's headline is frequently mis-tagged as plain `text` + # by OCR (so it never becomes a doc_title/paragraph_title floor). If any + # FOREIGN region sits directly above `r` in its column — tightly stacked + # and covering most of r's width — then r begins that neighbour's story, + # not our continuation. Genuine continuation text has nothing foreign + # directly above it (only our own headline, or the column top). + r_w = max(1, rb[2] - rb[0]) + headed_by_foreign = any( + t["id"] != primary_id + and t["id"] not in continuation_ids + and t["id"] not in member_ids + and t["id"] not in kept_set + and t.get("type") in ("text", "doc_title", "paragraph_title", "figure_title") + and 0 <= rb[1] - t["bbox"][3] <= 60 + and _hoverlap(t["bbox"], rb) >= 0.5 * r_w + for t in all_regions + ) + if headed_by_foreign: + continue + kept.append(rid) + kept_set.add(rid) + added.append(rid) + changed = True + + return kept, added + + +def split_disjoint_articles(political_articles, regions, gap=60, min_regions=2, + dateline_starts=None): + """For each Claude-returned article, split into spatially-connected sub-articles.""" + region_map = {r["id"]: r for r in regions} + + # ------ Pass 0: drop regions blocked by another doc_title in same column ------ + # For each article, remove members that sit behind another article's + # headline (column-aware), and any duplicate doc_titles in the member list. + for art in political_articles: + member_ids = art.get("member_region_ids", []) + member_ids, dropped_blocked = _drop_blocked_by_other_doc_title( + member_ids, region_map, regions + ) + if dropped_blocked: + print(f" ⛔ Column blocker: '{art.get('headline_english','?')[:40]}' — " + f"dropped {len(dropped_blocked)} region(s) blocked by another " + f"doc_title in their column: {dropped_blocked}") + art["member_region_ids"] = member_ids + + # ------ Pass 1: trim above-headline strays from every article ------ + trimmed_articles = [] + for art in political_articles: + member_ids = art.get("member_region_ids", []) + member_ids, dropped_above = _trim_above_headline(member_ids, region_map) + if dropped_above: + print(f" ⚠️ Headline trim: '{art.get('headline_english','?')[:40]}' — " + f"dropped {len(dropped_above)} region(s) above the headline: {dropped_above}") + new_art = dict(art) + new_art["member_region_ids"] = member_ids + trimmed_articles.append(new_art) + + # ------ Pass 2: grow each article downward into unclaimed regions ------ + # Build the global "claimed" set so growth never steals from another article. + claimed = set() + for a in trimmed_articles: + claimed.update(a["member_region_ids"]) + for art in trimmed_articles: + grown, added = _grow_article_downward( + art["member_region_ids"], region_map, regions, claimed, + dateline_starts=dateline_starts + ) + if added: + print(f" ⬇ Growth: '{art.get('headline_english','?')[:40]}' — " + f"added {len(added)} adjacent region(s) below headline: {added}") + claimed.update(added) + art["member_region_ids"] = grown + + # ------ Pass 3: geometric cluster split (catches remaining disjoint cases) ------ + out = [] + for art in trimmed_articles: + member_ids = art["member_region_ids"] + clusters = _cluster_regions(member_ids, region_map, gap) + if len(clusters) <= 1: + out.append(art) + continue + + # Multi-cluster article. Headline (doc_title) is the article's identity — + # a real article must contain one. Use it to classify each cluster: + # - Cluster with a doc_title → real article content, keep + # - Cluster with NO doc_title → misattributed strays, drop + # - Multiple clusters each with their own doc_title → Claude bundled + # two real articles together; split them into separate output items. + with_title, without_title = [], [] + for cluster in clusters: + has_title = any( + region_map.get(rid, {}).get("type") == "doc_title" + for rid in cluster + ) + (with_title if has_title else without_title).append(cluster) + + stray_count = sum(len(c) for c in without_title) + headline = art.get("headline_english", "?")[:40] + + if len(with_title) == 0: + # No headline in any cluster — likely an OCR mislabel. Fall back to + # the largest cluster (matches previous behavior). + keeper = clusters[0] + print(f" ⚠️ Geometry split: '{headline}' — no doc_title in any cluster, " + f"kept largest ({len(keeper)} regions), dropped {len(member_ids)-len(keeper)}") + new_art = dict(art) + new_art["member_region_ids"] = keeper + new_art["_split_note"] = "no doc_title in any cluster; kept largest" + out.append(new_art) + elif len(with_title) == 1: + # One real cluster + N stray clusters → keep real, drop strays. + keeper = with_title[0] + if stray_count: + print(f" ⚠️ Geometry split: '{headline}' — kept the {len(keeper)} " + f"regions around the headline, dropped {stray_count} stray " + f"region(s) from {len(without_title)} disconnected cluster(s) " + f"with no doc_title") + new_art = dict(art) + new_art["member_region_ids"] = keeper + new_art["_split_note"] = ( + f"dropped {stray_count} stray region(s) without doc_title" + ) + out.append(new_art) + else: + # Multiple real articles bundled together → split into separate items. + print(f" ⚠️ Geometry split: '{headline}' — found {len(with_title)} " + f"doc_titles in disjoint clusters, splitting into {len(with_title)} " + f"articles (also dropped {stray_count} strays from " + f"{len(without_title)} cluster(s) without doc_title)") + for idx, cluster in enumerate(with_title, 1): + new_art = dict(art) + new_art["member_region_ids"] = cluster + if idx > 1: + new_art["headline_english"] = ( + f"{art.get('headline_english','?')} (split {idx})" + ) + new_art["_split_note"] = ( + f"original group had {len(with_title)} doc_titles; this is part {idx}" + ) + out.append(new_art) + + # ------ Pass 4: merge sub-boxes (no doc_title) into their parent article ------ + # Quote-boxes, testimonial boxes, and infoboxes inside a larger article have + # only a paragraph_title — no doc_title. If such an "article" sits inside or + # directly adjacent to another article's bounding box, fold it into that + # parent rather than emitting it as a separate (duplicate) crop. + out = _merge_subboxes_into_parents(out, region_map, dateline_starts=dateline_starts) + + # ------ Pass 5: final re-growth over orphaned regions ------------------- + # The Pass 2 growth used a `claimed` set built BEFORE the geometry split. + # If Claude mis-assigned a body region to a neighboring article, it counted + # as claimed and blocked the rightful article from growing into it — then + # Pass 3 dropped it from the wrong article as a disconnected stray, leaving + # it orphaned (unclaimed) with no owner. Rebuild `claimed` from the final + # article set and run one more growth pass so each article absorbs the + # still-unclaimed regions directly below its headline. + claimed = set() + for a in out: + claimed.update(a["member_region_ids"]) + for art in out: + grown, added = _grow_article_downward( + art["member_region_ids"], region_map, regions, claimed, + dateline_starts=dateline_starts + ) + if added: + print(f" ⬇ Re-growth: '{art.get('headline_english','?')[:40]}' — " + f"recovered {len(added)} orphaned region(s) below headline: {added}") + claimed.update(added) + art["member_region_ids"] = grown + + return out + + +def split_articles_on_datelines(political_articles, regions, dateline_starts, sep_lines=None): + """Correct Claude's OVER-grouping: if one article's member regions contain TWO + (or more) dateline bodies, it has swallowed >1 story — split it. Each region is + assigned to the dateline that OWNS it (nearest dateline above it in its column); + members above all datelines attach to the topmost. The group holding the + article's headline keeps the original metadata; the others are emitted as + new articles flagged _needs_review (their Q1/Q2 belong to the parent). + + A non-keeper dateline group is split off as a SEPARATE article only when a real + article WALL — a headline (doc_title) or a horizontal separator line — sits in + the gap between it and the dateline directly above it. If the gap is just + continuous body text (no wall), the datelines belong to one roundup story + (same headline, several mandals) and are folded back together.""" + if not dateline_starts: + return political_articles + region_map = {r["id"]: r for r in regions} + dl_ids = {s["region_id"] for s in dateline_starts} + + # Article walls: detected headlines and horizontal separator lines. + doc_titles = [r["bbox"] for r in regions if r.get("type") == "doc_title"] + h_lines = [L for L in (sep_lines or []) if not L.get("vert") and "y" in L] + + def wall_between(upper_box, lower_box): + return _wall_between_datelines(upper_box, lower_box, doc_titles, h_lines) + + out = [] + for art in political_articles: + members = art.get("member_region_ids", []) or [] + dls = [rid for rid in members if rid in dl_ids and rid in region_map] + if len(dls) <= 1: + out.append(art) + continue + + dl_boxes = [(rid, region_map[rid]["bbox"]) for rid in dls] + + def owner(rid): + if rid not in region_map: + return dls[0] + b = region_map[rid]["bbox"] + cx = (b[0] + b[2]) / 2 + in_col = [(d, db) for d, db in dl_boxes if db[0] - 40 <= cx <= db[2] + 40] + # A HEADLINE owns the dateline directly BELOW it (its body starts there), + # not the dateline above it. + if region_map[rid].get("type") == "doc_title": + below = [(d, db) for d, db in in_col if db[1] >= b[3] - 30] + if below: + return min(below, key=lambda z: z[1][1])[0] # nearest below + # A BODY region belongs to the nearest dateline ABOVE it. + above = [(d, db) for d, db in in_col if db[1] <= b[1] + 30] + if above: + return max(above, key=lambda z: z[1][1])[0] + if in_col: # above all → topmost in its column + return min(in_col, key=lambda z: z[1][1])[0] + return min(dls, key=lambda d: region_map[d]["bbox"][1]) + + groups = {} + for rid in members: + groups.setdefault(owner(rid), []).append(rid) + if len(groups) <= 1: + out.append(art) + continue + + # which group keeps the parent metadata? the one holding the article's headline + doc_ids = [rid for rid in members + if region_map.get(rid, {}).get("type") == "doc_title"] + keeper = owner(min(doc_ids, key=lambda r: region_map[r]["bbox"][1])) if doc_ids \ + else min(dls, key=lambda d: region_map[d]["bbox"][1]) + if keeper not in groups: + keeper = max(groups, key=lambda k: len(groups[k])) + + # Split a non-keeper dateline group off as its own article only when a real + # WALL (headline or separator line) sits between it and the dateline directly + # above it. No wall → continuous body text → same roundup story → fold back + # into the keeper rather than emit a spurious headless crop. + keeper_ids = list(groups[keeper]) + review_groups = [] + for dl, ids in groups.items(): + if dl == keeper: + continue + db = region_map[dl]["bbox"] + cx = (db[0] + db[2]) / 2 + above = [region_map[d]["bbox"] for d in dls + if d != dl and region_map[d]["bbox"][1] < db[1] + and region_map[d]["bbox"][0] - 40 <= cx <= region_map[d]["bbox"][2] + 40] + upper = max(above, key=lambda b: b[1]) if above else None + # Fold-back only applies WITHIN a column. If there is a same-column + # dateline above, decide by the wall in the gap. If there is none, this + # group sits in a DIFFERENT column (a side-by-side story) → always its own + # article; never fold two datelines together across columns. + has_wall = wall_between(upper, db) if upper is not None else True + if has_wall: + review_groups.append((dl, ids)) + else: + keeper_ids.extend(ids) + + if not review_groups: + out.append(art) # nothing was really a separate story + continue + + print(f" ✂ Dateline split: '{art.get('headline_english','?')[:40]}' contained " + f"{len(dls)} datelines → kept 1 + split off {len(review_groups)} article(s) " + f"(headline/separator-line boundary)") + na = dict(art) + na["member_region_ids"] = sorted(keeper_ids) + out.append(na) + for dl, ids in sorted(review_groups, key=lambda kv: region_map[kv[0]]["bbox"][1]): + out.append({ + "headline_telugu": "", "headline_english": "(dateline-split — needs analysis)", + "member_region_ids": sorted(ids), "priority": "medium", "category": "review", + "q1_reason": "", "q2_reason": "", "attack_angle": "", + "_split_on_dateline": True, "_needs_review": True, + }) + return out + + +def trim_above_anchor(political_articles, regions, dateline_starts=None): + """DATELINE CEILING (the mirror of the downward floor): an article must not + include regions that sit ABOVE its own start — those belong to the article + above it. The article's start (anchor) is its headline (`doc_title`), or, if + PaddleOCR missed the headline, its own dateline body. Any member region whose + bottom is above the anchor's top is dropped.""" + region_map = {r["id"]: r for r in regions} + dl_list = [(s["region_id"], s["bbox"]) for s in (dateline_starts or []) + if s["region_id"] in region_map] + dl_ids = {d for d, _ in dl_list} + for art in political_articles: + members = art.get("member_region_ids", []) or [] + doc_ids = [rid for rid in members + if region_map.get(rid, {}).get("type") == "doc_title"] + if doc_ids: + anchor = min(doc_ids, key=lambda r: region_map[r]["bbox"][1]) + else: + dls = [rid for rid in members if rid in dl_ids and rid in region_map] + if not dls: + continue # no headline and no dateline → can't place a ceiling + anchor = min(dls, key=lambda r: region_map[r]["bbox"][1]) + ab = region_map[anchor]["bbox"] + ceiling, acx = ab[1], (ab[0] + ab[2]) / 2 + + # the article's OWN dateline = the nearest dateline at/below the anchor in its column + own = set() + below_anchor = [(d, db) for d, db in dl_list + if db[0] - 40 <= acx <= db[2] + 40 and db[1] >= ab[3] - 30] + if below_anchor: + own.add(min(below_anchor, key=lambda z: z[1][1])[0]) + + kept, dropped = [], [] + for rid in members: + r = region_map.get(rid) + if rid == anchor or not r: + kept.append(rid) + continue + b = r["bbox"] + cx = (b[0] + b[2]) / 2 + above_anchor = b[3] <= ceiling + 5 + # Is there a FOREIGN dateline at/above this region's level? (Not column- + # restricted, so it also catches the OTHER columns of a multi-column + # article sitting above this one.) + foreign_above = any(db[1] <= b[1] + 30 for d, db in dl_list if d not in own) + if dl_list and above_anchor and foreign_above: + dropped.append(rid) # below a foreign dateline → previous article (any type) + elif not dl_list and above_anchor and r.get("type") == "text": + dropped.append(rid) # no datelines → conservative: only body text + else: + kept.append(rid) + if dropped: + print(f" ⬆ Dateline ceiling: '{art.get('headline_english','?')[:40]}' — dropped " + f"{len(dropped)} region(s) above its start (belong to the article above): " + f"{sorted(dropped)}") + art["member_region_ids"] = sorted(kept) + return political_articles + + +def trim_below_floor(political_articles, regions, dateline_starts=None): + """Drop a member region that sits ENTIRELY BELOW the article's own text AND at/below + the point where a FOREIGN article (a doc_title or dateline not in this article) has + already started. Catches a stray photo/box from the next article that got grouped in + even though there's no foreign headline directly above it in its own column (e.g. a + portrait placed beside the next article's headline). Conservative: only regions that + are below all of THIS article's text are eligible, so body text is never touched.""" + region_map = {r["id"]: r for r in regions} + all_docs = [r for r in regions if r.get("type") == "doc_title"] + dl_list = [(s["region_id"], s["bbox"]) for s in (dateline_starts or [])] + for art in political_articles: + members = art.get("member_region_ids", []) or [] + mset = set(members) + doc_ids = [i for i in members if region_map.get(i, {}).get("type") == "doc_title"] + text_boxes = [region_map[i]["bbox"] for i in members + if region_map.get(i, {}).get("type") == "text"] + if not doc_ids or not text_boxes: + continue + anchor_top = min(region_map[i]["bbox"][1] for i in doc_ids) + text_bottom = max(b[3] for b in text_boxes) + foreign_tops = [r["bbox"][1] for r in all_docs + if r["id"] not in mset and r["bbox"][1] > anchor_top] + foreign_tops += [db[1] for d, db in dl_list + if d not in mset and db[1] > anchor_top] + if not foreign_tops: + continue + kept, dropped = [], [] + for rid in members: + r = region_map.get(rid) + if not r or rid in doc_ids: + kept.append(rid) + continue + b = r["bbox"] + below_text = b[1] > text_bottom # below ALL of this article's text + foreign_started = any(ft <= b[1] + 30 for ft in foreign_tops) # next article began at/above it + if below_text and foreign_started: + dropped.append(rid) + else: + kept.append(rid) + if dropped: + print(f" ⬇ Floor trim: '{art.get('headline_english','?')[:40]}' — dropped " + f"{len(dropped)} region(s) below its text that belong to the article below: " + f"{sorted(dropped)}") + art["member_region_ids"] = sorted(kept) + return political_articles + + +def grow_articles_final(political_articles, regions, dateline_starts=None): + """Final downward+rightward growth pass on the SETTLED article shapes. + + The first growth runs inside split_disjoint_articles, but the dateline splits + and trims that follow reshape articles (and mint new ones from dateline + splits), leaving those final shapes un-grown — so a body region adjacent to a + headline can be left out of the crop. Re-grow here, after all reshaping, so + every final article pulls in its adjacent body before cropping. + + Strictly additive: `claimed` is the union of all articles' members (each + article's own members are excluded as candidates inside the grow), and newly + grown regions are claimed immediately, so no article ever steals another's. + """ + region_map = {r["id"]: r for r in regions} + claimed = set() + for a in political_articles: + claimed.update(a.get("member_region_ids", []) or []) + for art in political_articles: + members = art.get("member_region_ids", []) or [] + if not members: + continue + grown, added = _grow_article_downward( + members, region_map, regions, claimed, dateline_starts=dateline_starts + ) + if added: + print(f" ⬇ Final growth: '{art.get('headline_english','?')[:40]}' — " + f"added {len(added)} region(s): {sorted(added)}") + art["member_region_ids"] = grown + claimed.update(added) + return political_articles + + +def _bbox_of(member_ids, region_map): + bs = [region_map[r]["bbox"] for r in member_ids if r in region_map] + if not bs: + return None + return [min(b[0] for b in bs), min(b[1] for b in bs), + max(b[2] for b in bs), max(b[3] for b in bs)] + + +def _bbox_relation(child, parent, adjacency=80): + """ + Return ratio of `child` bbox area that is inside `parent` bbox. + If they don't overlap directly, also consider proximity: + if `child` is within `adjacency` pixels of `parent` (horizontally and + vertically) and shares column space, treat as adjacent and return 0.5. + """ + cx1, cy1, cx2, cy2 = child + px1, py1, px2, py2 = parent + x_ov = max(0, min(cx2, px2) - max(cx1, px1)) + y_ov = max(0, min(cy2, py2) - max(cy1, py1)) + child_area = max(1, (cx2 - cx1) * (cy2 - cy1)) + contained_ratio = (x_ov * y_ov) / child_area + if contained_ratio > 0: + return contained_ratio + # No direct overlap — check adjacency in same column. + horiz_close = (x_ov > 0) or (min(abs(cx1 - px2), abs(px1 - cx2)) <= adjacency) + vert_close = (y_ov > 0) or (min(abs(cy1 - py2), abs(py1 - cy2)) <= adjacency) + horiz_shares_column = x_ov >= 0.5 * min(cx2 - cx1, px2 - px1) + if horiz_close and vert_close and horiz_shares_column: + return 0.5 + return 0.0 + + +def _merge_subboxes_into_parents(articles, region_map, min_relation=0.4, dateline_starts=None): + dl_ids = {s["region_id"] for s in (dateline_starts or [])} + parents, subs = [], [] + for art in articles: + has_title = any( + region_map.get(rid, {}).get("type") == "doc_title" + for rid in art.get("member_region_ids", []) + ) + (parents if has_title else subs).append(art) + + if not subs: + return articles # nothing to merge + if not parents: + # No doc_title parent anywhere on the page. Keep a headless sub ONLY if it is + # a plausible standalone article — anchored on a dateline, or carrying a + # title-ish region. A sub that is just plain body text with neither is an + # orphan fragment (often a non-political block onto which Claude pasted a + # headline borrowed from another page) — drop it rather than emit junk. + kept = [] + for s in subs: + mids = s.get("member_region_ids", []) or [] + has_anchor = any(rid in dl_ids for rid in mids) + has_titleish = any( + region_map.get(rid, {}).get("type") in ("doc_title", "paragraph_title") + for rid in mids + ) + if has_anchor or has_titleish: + s["_subbox_note"] = "no doc_title and no parent article candidate" + print(f" ⚠️ Sub-box without parent: '{s.get('headline_english','?')[:40]}' " + f"— kept as standalone (dateline-anchored or has title region)") + kept.append(s) + else: + print(f" ✗ Orphan sub-box dropped: '{s.get('headline_english','?')[:40]}' " + f"— no doc_title, no dateline anchor, no title region " + f"(likely a body fragment mislabeled as an article)") + return kept + + # For each sub, find the parent with highest containment/adjacency ratio. + for sub in subs: + sub_bbox = _bbox_of(sub.get("member_region_ids", []), region_map) + if sub_bbox is None: + continue + best_parent, best_score = None, 0.0 + for p in parents: + p_bbox = _bbox_of(p["member_region_ids"], region_map) + if p_bbox is None: + continue + score = _bbox_relation(sub_bbox, p_bbox) + if score > best_score: + best_score = score + best_parent = p + + sub_label = sub.get("headline_english", "?")[:40] + if best_parent is not None and best_score >= min_relation: + parent_label = best_parent.get("headline_english", "?")[:40] + parent_ids = set(best_parent["member_region_ids"]) + parent_ids.update(sub.get("member_region_ids", [])) + best_parent["member_region_ids"] = list(parent_ids) + print(f" ⤴ Sub-box merge: '{sub_label}' folded into parent " + f"'{parent_label}' (overlap/adjacency ratio={best_score:.2f})") + else: + # No doc_title and no clear parent — this is almost certainly an + # internal sub-box of some article that Claude split out. Drop it + # rather than emit a duplicate / mislabeled crop. + print(f" ✗ Sub-box dropped: '{sub_label}' (no doc_title, " + f"best parent overlap={best_score:.2f}) — likely a quote-box " + f"or testimonial inside another article") + + return parents + + +# --------------------------------------------------------------------------- +# Step 4: Crop using PaddleOCR's precise boxes +# --------------------------------------------------------------------------- +def crop_political_articles(page_img_path, political_articles, regions, output_dir, page_num, + dateline_starts=None, sep_lines=None): + img = Image.open(page_img_path) + pw, ph = img.size + + # ── BLOCK-SNAP: prefer the validated geometric block crop ───────────────────── + # The geometric clusterer + crop_all_article_blocks carry ALL the per-paper + # cropping fixes (furniture drop, un-boxed headline recovery, neighbour masking, + # box-edge rules, continuation folds — see CROP_STRATEGY.md). Instead of + # re-deriving an article rectangle from Claude's member list with parallel + # ad-hoc geometry, map each political article onto the geometric block that owns + # the MAJORITY of its members and ship that block's crop. The legacy path below + # remains the fallback when no block matches (and for any article whose block + # was already claimed by a higher-priority article on the same page). + _snap_files, _snap_blocks, _snap_owner = {}, {}, {} + try: + import re as _re, tempfile as _tf, shutil as _sh + _blocks = cluster_all_article_blocks(regions, dateline_starts, sep_lines=sep_lines) + for _b in _blocks: + if _b.get("_furniture"): + continue + _snap_blocks[_b["anchor_id"]] = _b + for _m in _b["members"] + [_b["anchor_id"]]: + _snap_owner[_m] = _b["anchor_id"] + _bdir = Path(_tf.mkdtemp(prefix="polblocks_")) + crop_all_article_blocks(page_img_path, regions, str(_bdir), page_num, + dateline_starts, sep_lines=sep_lines) + for _p in (_bdir / f"page_{page_num:03d}").glob("a*.png"): + _mm = _re.match(r"a\d+_[A-Z]+(\d+)\.png$", _p.name) + if _mm: + _snap_files[int(_mm.group(1))] = _p + except Exception as _e: + print(f" (block-snap unavailable, falling back to legacy crops: {_e})") + _snap_files, _snap_blocks, _snap_owner = {}, {}, {} + _snap_used = set() + + # Bottom of real content: y above which the page holds only the printer's + # CMYK colour-calibration strip + paper margin. Article crops are clamped to + # this so the registration dots never bleed into an article image. Returns the + # full height when there is no such strip (namaste, photo-filled bottoms). + try: + from _lines import detect_content_bottom + content_bottom = detect_content_bottom(page_img_path) + if not content_bottom: + content_bottom = ph + except Exception: + content_bottom = ph + + # Build region lookup + region_map = {r["id"]: r for r in regions} + + # Every detected headline on the page. A `doc_title` that does NOT belong to + # the article being cropped marks the START of a DIFFERENT story, so the crop + # must stop above it — otherwise the next article's headline + body bleed in. + all_doc_titles = [r for r in regions if str(r.get("type")) == "doc_title"] + + records = [] + for i, art in enumerate(political_articles, 1): + member_ids = list(art.get("member_region_ids", [])) + if not member_ids: + continue + + # Member regions that actually exist on the page. + member_regs = [region_map[rid] for rid in member_ids if rid in region_map] + if not member_regs: + continue + + # --- block-snap: majority-vote the owning geometric block ------------ + _votes = {} + for _r in member_regs: + _a = _snap_owner.get(_r["id"]) + if _a is not None: + _votes[_a] = _votes.get(_a, 0) + 1 + _snap = None + if _votes: + _best, _n = max(_votes.items(), key=lambda kv: kv[1]) + if (_n >= 0.5 * len(member_regs) and _best in _snap_files + and _best not in _snap_used): + _snap = _best + if _snap is not None: + _blk = _snap_blocks[_snap] + art_name = f"p{page_num:03d}_a{i:03d}" + art_dir = output_dir / art_name + art_dir.mkdir(parents=True, exist_ok=True) + _sh.copyfile(_snap_files[_snap], art_dir / "article.png") + _bb = list(_blk["bbox"]) + info = { + "page": page_num, + "article_id": i, + "bbox": _bb, + "member_region_ids": sorted(set(_blk["members"]) | {_snap}), + "headline_telugu": art.get("headline_telugu", ""), + "headline_english": art.get("headline_english", ""), + "q1_reason": art.get("q1_reason", ""), + "q2_reason": art.get("q2_reason", ""), + "attack_angle": art.get("attack_angle", ""), + "priority": art.get("priority", ""), + "category": art.get("category", ""), + "method": "smart_extractor_blocksnap", + } + (art_dir / "info.json").write_text(json.dumps(info, indent=2, ensure_ascii=False)) + records.append({"dir": str(art_dir), "page": page_num, "article_id": i, + "name": art_name, "bbox": _bb, + "headline_telugu": art.get("headline_telugu", ""), + "is_banner": False}) + art["_page"] = page_num + art["_art_name"] = art_name + art["_crop_area"] = (_bb[2] - _bb[0]) * (_bb[3] - _bb[1]) + _snap_used.add(_snap) + print(f" 🧲 Block-snap: {art_name} → geometric block R{_snap} " + f"({_n}/{len(member_regs)} members, bbox={_bb})") + continue + + # Raw union of member bounding boxes (no padding yet). + rx1 = min(r["bbox"][0] for r in member_regs) + ry1 = min(r["bbox"][1] for r in member_regs) + rx2 = max(r["bbox"][2] for r in member_regs) + ry2 = max(r["bbox"][3] for r in member_regs) + member_ids0 = list(member_ids) # original members (for banner restore) + + # --- Multi-column BANNER headline: crop the whole multi-column block -- + # A single WIDE headline spanning several columns heads ONE article; every + # column beneath it down to its bottom rule belongs to that article even + # though it contains internal sub-headlines and photos. The per-column + # clip / floor rules below would wrongly split such a banner, so when we + # recognise a banner bounded by side walls AND a bottom rule we take the + # whole block and skip those rules. If there is NO bottom rule we fall + # back to the normal per-column logic. + banner_box = None + _bt0 = sorted([r for r in member_regs if str(r.get("type")) == "doc_title"], + key=lambda r: r["bbox"][1]) + if _bt0 and (_bt0[0]["bbox"][2] - _bt0[0]["bbox"][0]) >= 0.40 * pw: + hb0 = _bt0[0]["bbox"] + htop, hbot = hb0[1], hb0[3] + try: + from _lines import detect_separator_lines + _ls0 = detect_separator_lines(page_img_path) + vlines0, hlines0 = _ls0.get("v", []), _ls0.get("h", []) + except Exception: + vlines0, hlines0 = [], [] + + def _vcov(ln): # v-rule reaches up alongside headline + return ln.get("y1", 0) <= hbot and ln.get("y2", 0) >= htop - 20 + lw = [ln.get("x1", ln.get("x", 0)) for ln in vlines0 + if _vcov(ln) and ln.get("x1", ln.get("x", 0)) <= hb0[0] + 30] + rw = [ln.get("x1", ln.get("x", 0)) for ln in vlines0 + if _vcov(ln) and ln.get("x1", ln.get("x", 0)) >= hb0[2] - 30] + left_wall = max(lw) if lw else rx1 + right_wall = min(rw) if rw else rx2 + bw = right_wall - left_wall + # Bottom rule: a horizontal line spanning most of the banner width, the + # first one BELOW the grouped content. The width test (>60% of banner) + # rejects short caption/photo underlines; _line_is_underline() also + # rejects a WIDE rule that merely underlines an image/caption (a line + # with a photo or figure_title directly above it is not a separator). + cands = [ln.get("y1", ln.get("y", 0)) for ln in hlines0 + if ln.get("y1", ln.get("y", 0)) > max(hbot, ry2) + 5 + and (min(ln.get("x2", 0), right_wall) + - max(ln.get("x1", 0), left_wall)) > 0.6 * bw + and not _line_is_underline(ln.get("y1", ln.get("y", 0)), + ln.get("x1", 0), ln.get("x2", 0), regions)] + if cands and bw > 0: + bottom_rule = min(cands) + if bottom_rule > htop + 200: + banner_box = [left_wall, htop, right_wall, bottom_rule] + + # --- Clip out intruding foreign headlines (column-aware) ------------ + # An article has ONE headline at the top. Any OTHER headline below it marks + # the START of the next story IN THE COLUMN(S) IT SPANS. Drop only the + # members that overlap that headline's column AND sit at/below it — body + # flowing in the article's OTHER columns is kept. A page-wide y-clip would + # truncate a multi-column article whenever a short neighbouring headline + # appears in just ONE of its columns (e.g. a col-1 headline below the + # article while its body continues lower in cols 2–4). + member_ids_set = {r["id"] for r in member_regs} + own_titles = sorted([r for r in member_regs if str(r.get("type")) == "doc_title"], + key=lambda r: r["bbox"][1]) + primary_bottom = own_titles[0]["bbox"][3] if own_titles else ry1 + + # Span of this article's TEXTUAL content (excludes images/figures). Used to + # recognise a disjoint trailing member (e.g. a misgrouped far-side photo) + # that sits past the article's text AND below a foreign headline. + text_regs = [r for r in member_regs + if str(r.get("type")) in ("text", "doc_title", "paragraph_title")] + text_x1 = min((r["bbox"][0] for r in text_regs), default=rx1) + text_x2 = max((r["bbox"][2] for r in text_regs), default=rx2) + text_bottom = max((r["bbox"][3] for r in text_regs), default=ry2) + + def _hov(a, b): + return max(0, min(a[2], b[2]) - max(a[0], b[0])) + + drop_ids = set() + for dt in all_doc_titles: + if dt["id"] in member_ids_set: # our own headline / continuation + continue + db = dt["bbox"] + if db[1] <= primary_bottom: # must start below our headline + continue + if _hov(db, [rx1, ry1, rx2, ry2]) <= 0: # must fall in our columns + continue + # A foreign headline that sits BELOW all of our text marks pure trailing + # foreign content: anything starting at/below it is the next story. Drop + # such a member only if it's horizontally disjoint from our text column, + # so a legitimate bottom photo sitting under the text is never cut. + trailing = db[1] >= text_bottom + for r in member_regs: + rb = r["bbox"] + if rb[1] >= db[1] - 30: + if _hov(rb, db) > 0: + drop_ids.add(r["id"]) + elif trailing and _hov(rb, [text_x1, 0, text_x2, 0]) <= 0: + drop_ids.add(r["id"]) + if drop_ids: + kept = [r for r in member_regs if r["id"] not in drop_ids] + if kept: + member_regs = kept + member_ids = [r["id"] for r in kept] + rx1 = min(r["bbox"][0] for r in member_regs) + ry1 = min(r["bbox"][1] for r in member_regs) + rx2 = max(r["bbox"][2] for r in member_regs) + ry2 = max(r["bbox"][3] for r in member_regs) + print(f" ✂️ Clipped {len(drop_ids)} member(s) under a foreign headline " + f"out of p{page_num:03d}_a{i:03d}: {sorted(drop_ids)}") + + # --- Clip members across a vertical separator rule ------------------- + # A printed vertical rule running the height of the article marks a + # column/article boundary (the gutter between two stories). Drop members + # that fall entirely on the FAR side of such a rule from the article's + # primary headline — e.g. a neighbouring story's photo/caption bleeding + # in past the gutter line. The headline must sit cleanly on one side of + # the rule (a rule passing THROUGH the headline spans the article and is + # not a boundary). + prim = sorted([r for r in member_regs if str(r.get("type")) == "doc_title"], + key=lambda r: r["bbox"][1]) + prim = prim[0] if prim else (sorted(member_regs, key=lambda r: r["bbox"][1])[0] + if member_regs else None) + if prim is not None and len(member_regs) > 1: + try: + from _lines import detect_separator_lines + vlines = detect_separator_lines(page_img_path).get("v", []) + except Exception: + vlines = [] + hb = prim["bbox"] + vdrop = set() + for ln in vlines: + vx = ln.get("x1", ln.get("x", 0)) + ly1, ly2 = ln.get("y1", 0), ln.get("y2", 0) + if (ly2 - ly1) < 600: # only long structural rules + continue + if hb[0] < vx < hb[2]: # rule passes through headline + continue + head_left = hb[2] <= vx + for r in member_regs: + if r["id"] == prim["id"]: + continue + b = r["bbox"] + if b[3] < ly1 or b[1] > ly2: # member must overlap rule span + continue + on_far_side = (b[0] >= vx) if head_left else (b[2] <= vx) + if on_far_side: + vdrop.add(r["id"]) + if vdrop: + kept = [r for r in member_regs if r["id"] not in vdrop] + if kept: + member_regs = kept + member_ids = [r["id"] for r in kept] + rx1 = min(r["bbox"][0] for r in member_regs) + ry1 = min(r["bbox"][1] for r in member_regs) + rx2 = max(r["bbox"][2] for r in member_regs) + ry2 = max(r["bbox"][3] for r in member_regs) + print(f" ✂️ Clipped {len(vdrop)} member(s) across a vertical rule " + f"out of p{page_num:03d}_a{i:03d}: {sorted(vdrop)}") + + # --- Floor at a second in-column headline ---------------------------- + # An article has ONE headline. A SECOND headline-class member (doc_title + # or paragraph_title) below it, in the same column AND with its own body + # text beneath, is the start of a different story that was over-merged + # (e.g. a 'continued from page N' header). Treat the topmost such member + # as a floor: drop it and every member at/below it that overlaps the + # headline's column. Members in the article's OTHER columns are kept. + prim2 = sorted([r for r in member_regs if str(r.get("type")) == "doc_title"], + key=lambda r: r["bbox"][1]) + if prim2: + pb = prim2[0]["bbox"] + pid = prim2[0]["id"] + # Horizontal rules on the page — a TRUE second-article boundary in the + # column is printed under a horizontal rule. An in-article sub-head / + # byline (a narrow paragraph_title with body beneath it, like a + # dateline) has NO rule above it and must NOT floor the crop. + try: + from _lines import detect_separator_lines + hlines = detect_separator_lines(page_img_path).get("h", []) + except Exception: + hlines = [] + + def _rule_above(cb): + ct, cw = cb[1], (cb[2] - cb[0]) + for ln in hlines: + ly = ln.get("y1", ln.get("y", 0)) + if pb[3] < ly <= ct + 20: + ox = (min(ln.get("x2", 0), cb[2]) - max(ln.get("x1", 0), cb[0])) + # A caption/photo underline (image or figure_title directly + # above the line) is NOT a section separator — skip it. + if ox > 0.4 * cw and not _line_is_underline( + ly, ln.get("x1", 0), ln.get("x2", 0), regions): + return True + return False + + def _qualifies(r): + t = str(r.get("type")) + if t == "doc_title": + return True + # paragraph_title only floors with a corroborating rule above it + return t == "paragraph_title" and _rule_above(r["bbox"]) + + sec = [r for r in member_regs + if r["id"] != pid + and _qualifies(r) + and r["bbox"][1] > pb[3] + 5 + and _hov(r["bbox"], pb) > 0.3 * (pb[2] - pb[0])] + if sec: + floor = min(r["bbox"][1] for r in sec) + has_body_below = any( + str(r.get("type")) == "text" + and r["bbox"][1] > floor + and _hov(r["bbox"], [pb[0], 0, pb[2], 0]) > 0 + for r in member_regs) + if has_body_below: + fdrop = {r["id"] for r in member_regs + if r["bbox"][1] >= floor - 10 + and _hov(r["bbox"], [pb[0], 0, pb[2], 0]) > 0} + if fdrop: + kept = [r for r in member_regs if r["id"] not in fdrop] + if kept: + member_regs = kept + member_ids = [r["id"] for r in kept] + rx1 = min(r["bbox"][0] for r in member_regs) + ry1 = min(r["bbox"][1] for r in member_regs) + rx2 = max(r["bbox"][2] for r in member_regs) + ry2 = max(r["bbox"][3] for r in member_regs) + print(f" ✂️ Floored p{page_num:03d}_a{i:03d} at a 2nd in-column " + f"headline (y={floor}); dropped {sorted(fdrop)}") + + # --- Grow RIGHT to a vertical separator wall ------------------------- + # Claude sometimes under-groups an article, omitting its right-most + # column even though a printed vertical rule clearly bounds the story on + # the right. If a long structural rule runs just past the crop's right + # edge within the article's vertical band, and the strip between the + # current right edge and that wall contains NO foreign headline + # (doc_title), the strip belongs to this article — absorb its text / + # image / caption members and extend the crop to the wall. + right_cap = None + if member_regs: + try: + from _lines import detect_separator_lines + vlines2 = detect_separator_lines(page_img_path).get("v", []) + except Exception: + vlines2 = [] + art_h = ry2 - ry1 + current_ids2 = {r["id"] for r in member_regs} + walls = [ln for ln in vlines2 + if (ln.get("y2", 0) - ln.get("y1", 0)) >= 600 + and ln.get("x1", ln.get("x", 0)) > rx2 + 10 + and (min(ln.get("y2", 0), ry2) - max(ln.get("y1", 0), ry1)) > 0.5 * art_h] + if walls and art_h > 0: + wall_x = min(ln.get("x1", ln.get("x", 0)) for ln in walls) + # Bound the grow distance: a missing right-hand COLUMN cannot be + # wider than the article it attaches to. A far wall at the page's + # right margin would otherwise swallow whole neighbouring stories + # that happen to lack a detected gutter rule. + art_w = rx2 - rx1 + too_far = (wall_x - rx2) > max(art_w, 1) + # No FOREIGN headline may live in the strip — that would mean the + # strip is a different story, not this article's missing column. + foreign = too_far or any(str(r.get("type")) == "doc_title" + and r["id"] not in current_ids2 + and r["bbox"][2] > rx2 and r["bbox"][0] < wall_x + and r["bbox"][3] > ry1 and r["bbox"][1] < ry2 + for r in regions) + if not foreign: + absorb = [r for r in regions + if str(r.get("type")) in ("text", "image", + "figure_title", "paragraph_title") + and r["id"] not in current_ids2 + and rx2 < (r["bbox"][0] + r["bbox"][2]) / 2 < wall_x + and r["bbox"][3] > ry1 and r["bbox"][1] < ry2] + if absorb: + member_regs = member_regs + absorb + member_ids = [r["id"] for r in member_regs] + rx1 = min(r["bbox"][0] for r in member_regs) + ry1 = min(r["bbox"][1] for r in member_regs) + rx2 = max(max(r["bbox"][2] for r in member_regs), int(wall_x)) + ry2 = max(r["bbox"][3] for r in member_regs) + right_cap = int(wall_x) + print(f" ➡️ Grew p{page_num:03d}_a{i:03d} right to a vertical " + f"wall (x={int(wall_x)}); absorbed {sorted(r['id'] for r in absorb)}") + + # A recognised banner overrides the per-column clip/floor/grow results: + # take the whole multi-column block (side walls + bottom rule) and restore + # the full member set the clips may have pruned. + if banner_box is not None: + rx1, ry1, rx2, ry2 = banner_box + right_cap = None + member_ids = list(member_ids0) + print(f" 🪧 Banner p{page_num:03d}_a{i:03d}: cropped full multi-column " + f"block x[{rx1}..{rx2}] y[{ry1}..{ry2}]") + + # Pad and clamp to page bounds. + x1 = max(0, rx1 - 15) + y1 = max(0, ry1 - 15) + x2 = min(pw, rx2 + 15) + y2 = min(ph, ry2 + 15, content_bottom) + if right_cap is not None: + x2 = min(x2, right_cap) + + if (x2 - x1) < 50 or (y2 - y1) < 50: + continue + + # Crop + art_name = f"p{page_num:03d}_a{i:03d}" + art_dir = output_dir / art_name + art_dir.mkdir(parents=True, exist_ok=True) + + cropped = img.crop((x1, y1, x2, y2)) + cropped.save(str(art_dir / "article.png")) + + info = { + "page": page_num, + "article_id": i, + "bbox": [x1, y1, x2, y2], + "member_region_ids": member_ids, + "headline_telugu": art.get("headline_telugu", ""), + "headline_english": art.get("headline_english", ""), + "q1_reason": art.get("q1_reason", ""), + "q2_reason": art.get("q2_reason", ""), + "attack_angle": art.get("attack_angle", ""), + "priority": art.get("priority", ""), + "category": art.get("category", ""), + "method": "smart_extractor", + } + (art_dir / "info.json").write_text(json.dumps(info, indent=2, ensure_ascii=False)) + records.append({"dir": str(art_dir), "page": page_num, "article_id": i, + "name": art_name, "bbox": [x1, y1, x2, y2], + "headline_telugu": art.get("headline_telugu", ""), + "is_banner": banner_box is not None}) + + # Annotate the article dict so the cross-page de-dup step (Step 5) can + # compare instances of the same story and keep the most complete crop. + art["_page"] = page_num + art["_art_name"] = art_name + art["_crop_area"] = (x2 - x1) * (y2 - y1) + + print(f" Cropped: {art_name} ({len(member_ids)} regions, bbox={[x1,y1,x2,y2]})") + + if _snap_files: + try: + _sh.rmtree(next(iter(_snap_files.values())).parent.parent) + except Exception: + pass + return records + + +# --------------------------------------------------------------------------- +# Generate output files +# --------------------------------------------------------------------------- +def generate_brief(all_articles, output_dir): + high = [a for a in all_articles if a.get("priority") == "high"] + medium = [a for a in all_articles if a.get("priority") == "medium"] + + brief_path = output_dir / "opposition_daily_brief.txt" + with open(brief_path, "w", encoding="utf-8") as f: + f.write("=" * 70 + "\n") + f.write(" OPPOSITION DAILY BRIEF\n") + f.write("=" * 70 + "\n\n") + f.write(f"Total: {len(all_articles)} | HIGH: {len(high)} | MEDIUM: {len(medium)}\n") + f.write("-" * 70 + "\n\n") + + for label, group in [("HIGH PRIORITY", high), ("MEDIUM PRIORITY", medium)]: + if group: + f.write(f">>> {label} <<<\n\n") + for i, a in enumerate(group, 1): + f.write(f"{i}. {a.get('headline_telugu', '')}\n") + f.write(f" ({a.get('headline_english', '')})\n") + f.write(f" WHY: {a.get('q1_reason', '')}\n") + f.write(f" ATTACK: {a.get('attack_angle', '')}\n\n") + f.write("-" * 70 + "\n") + f.write("END OF BRIEF\n") + + print(f" Brief saved: {brief_path}") + + +def generate_pdf(all_records, output_dir): + images = [] + for rec in sorted(all_records, key=lambda r: (r["page"], r["article_id"])): + img_path = Path(rec["dir"]) / "article.png" + if img_path.exists(): + images.append(Image.open(img_path).convert("RGB")) + if images: + pdf_path = output_dir / "political_articles.pdf" + images[0].save(str(pdf_path), save_all=True, append_images=images[1:]) + print(f" PDF saved: {pdf_path}") + + +def generate_combined_pdf(crop_paths, out_path): + """Stitch every cropped article image across ALL processed PDFs into one PDF, + one crop per page, in the given order.""" + images = [] + for p in crop_paths: + try: + images.append(Image.open(p).convert("RGB")) + except Exception as e: + print(f" ⚠️ Skipping unreadable crop {p}: {e}") + if images: + images[0].save(str(out_path), save_all=True, append_images=images[1:]) + print(f" Combined PDF saved: {out_path} ({len(images)} crops)") + else: + print(" No crops found to combine into a PDF.") + + +def copy_all_images(all_records, output_dir): + import shutil + img_dir = output_dir / "all_political_images" + img_dir.mkdir(exist_ok=True) + for rec in all_records: + src = Path(rec["dir"]) / "article.png" + if src.exists(): + shutil.copy2(str(src), str(img_dir / f"{rec['name']}.png")) + print(f" All images copied to: {img_dir}") + + +def write_review_report(all_political, pages_dir, output_dir, tiny_area=500000, + crop_integrity_issues=None): + """Lightweight post-run validator. Flags crops that probably need a human look — + empty Telugu headline, a suspiciously small crop area, or a _needs_review marker — + plus any page whose model response failed to parse (a *.badresponse.txt exists). + Writes /_REVIEW.txt. Returns the number of flagged items.""" + lines = [] + flagged = 0 + + bad_pages = sorted(Path(pages_dir).glob("*.badresponse.txt")) + if bad_pages: + lines.append("PAGES WITH UNPARSEABLE MODEL RESPONSE (articles may be missing):") + for bp in bad_pages: + lines.append(f" - {bp.name}") + lines.append("") + flagged += len(bad_pages) + + art_issues = [] + for art in all_political: + reasons = [] + if not (art.get("headline_telugu") or "").strip(): + reasons.append("empty headline_telugu") + area = art.get("_crop_area") + if isinstance(area, (int, float)) and area < tiny_area: + reasons.append(f"tiny crop area ({int(area):,} < {tiny_area:,})") + if art.get("_needs_review"): + reasons.append("flagged _needs_review") + if reasons: + art_issues.append((art, reasons)) + + if art_issues: + lines.append("CROPS TO REVIEW:") + for art, reasons in art_issues: + name = art.get("_art_name", "?") + page = art.get("_page", "?") + hl = (art.get("headline_telugu") or art.get("headline_english") or "").strip() or "(no headline)" + lines.append(f" - p{page} {name}: {hl}") + lines.append(f" → {'; '.join(reasons)}") + lines.append("") + flagged += len(art_issues) + + if crop_integrity_issues: + lines.append("INCOMPLETE / IMPROPER CROPS (dateline-vs-crop integrity):") + for it in crop_integrity_issues: + lines.append(f" - p{it['page']} block R{it['anchor_id']} [{it['kind']}]: {it['reason']}") + lines.append("") + flagged += len(crop_integrity_issues) + + header = (f"REVIEW REPORT — {flagged} item(s) flagged\n" + f"{'=' * 48}\n\n") if flagged else "REVIEW REPORT — no issues flagged.\n" + (Path(output_dir) / "_REVIEW.txt").write_text(header + "\n".join(lines), encoding="utf-8") + if flagged: + print(f" ⚠️ Review report: {flagged} item(s) flagged → {output_dir / '_REVIEW.txt'}") + else: + print(f" ✅ Review report: no issues flagged → {output_dir / '_REVIEW.txt'}") + return flagged + + +# --------------------------------------------------------------------------- +# Main +# --------------------------------------------------------------------------- +def process_pdf(pdf_path, output_dir, target_w=4200, target_h=7400): + """Run the full extraction pipeline on ONE PDF, writing everything into output_dir.""" + pdf_path = Path(pdf_path) + output_dir = Path(output_dir) + pages_dir = output_dir / "pages" + articles_dir = output_dir / "articles" + pages_dir.mkdir(parents=True, exist_ok=True) + articles_dir.mkdir(parents=True, exist_ok=True) + + print(f"PDF: {pdf_path}") + print(f"Output: {output_dir}") + print() + + # Step 1: PDF → page images + print("STEP 1: Rendering PDF pages...") + pages = render_pdf_pages(pdf_path, pages_dir, target_w, target_h) + paper = detect_paper(pdf_path) + global _ACTIVE_PAPER + _ACTIVE_PAPER = paper # gates paper-specific clustering passes (e.g. Sakshi) + print(f" Detected paper: {paper} (dateline rule selected accordingly)") + print() + + all_records = [] + all_political = [] + page_summary = [] # per-page counts: (page, all_blocks/crops, political) + crop_integrity_issues = [] # dateline-vs-crop integrity flags (split/merged crops) + + for p in pages: + n = p["page"] + page_img_path = p["path"] + + # Step 2: PaddleOCR layout detection + print(f"STEP 2: Page {n} — detecting layout regions...") + regions = detect_regions(page_img_path) + + # Drop near-duplicate detections (same heading detected twice). + regions = dedup_overlapping_regions(regions) + + # Merge the lines of a headline that wrapped onto 2+ lines (boxed separately). + regions = merge_stacked_title_lines(regions, page_img_path) + + # Drop the fixed front-page masthead (Namaste Telangana edition nameplate) + # so it is never mis-grouped as a news article. + regions = drop_masthead_regions(regions, paper, n, page_img_path) + + # Drop CLASSIFIEDS columns (phone-listing furniture) so a neighbouring article's + # column-fold can't grow down into them (JG R65 mega-over-merge). News untouched. + regions = drop_classifieds_regions(regions, page_img_path) + + # Tag COURT / LEGAL NOTICES (kept, but carved into their own section by the clusterer + # so they aren't absorbed into a neighbouring article). + regions = tag_legal_notices(regions, page_img_path) + + # Snapshot type counts BEFORE promotion so the inventory can show before/after. + from collections import Counter as _Counter + pre_promo_counts = _Counter(r.get("type", "unknown") for r in regions) + + # Reclassify mis-tagged headlines: a full-width paragraph_title that has a + # fresh dateline body directly beneath it is really a headline → doc_title. + regions = promote_paragraph_titles(regions, page_img_path, paper) + + # SAKSHI/NT: recover display headlines PaddleOCR never boxed (pixel pass) — + # widen partially-boxed headlines over their kicker, synthesize anchors + # for completely un-boxed headline bands (NT JG pg1: the script-font + # 'నమ్మిండు..' and DL2's heading have no region at all). NT is FRONT PAGE + # ONLY: inner NT pages are dense and well-detected; the synth there split + # protected roundup sections (Siddipet pg4 25→28). + if paper == "sakshi" or (paper == "namaste_telangana" and n == 1): + regions = detect_sakshi_headline_bands(regions, page_img_path) + + # Save regions (post-promotion) + (pages_dir / f"page_{n:03d}.regions.json").write_text( + json.dumps({"page": n, "regions": regions}, indent=2, ensure_ascii=False) + ) + + # Diagnostic: per-page region-type counts (before/after promotion) + headline list + log_region_inventory(regions, n, pre_promo_counts) + + # Flag bullet-prefixed titles (decks/sub-points) so the clusterer demotes them + # when a real headline sits above — a bullet is never an article headline. + regions = mark_bullet_titles(regions, page_img_path) + + # Article-boundary rules: horizontal separator lines (without a bold headline + # directly above) mark where one stacked story ends and the next begins. Used + # as hard barriers so the clusterer never folds/grows a block across them. + try: + from _lines import separator_barriers + sep_lines = separator_barriers(page_img_path, regions) + except Exception as e: + print(f" (separator-line detection skipped: {e})") + sep_lines = [] + + # CLUE: find article-starts by the dateline pattern (catches articles whose + # headline PaddleOCR missed). Fed into the Stage-3 prompt so Claude emits them. + dateline_starts = find_article_starts_by_dateline(regions, page_img_path, paper) + + # Orphan-scoped headline recovery: cluster once, then re-tag any BOLD orphan text + # sitting above a photo as a doc_title (a lost headline PaddleOCR mis-typed). Only + # orphans qualify, so captions — which always fold into an article — are untouched. + # If anything was recovered, re-detect datelines (they can now link to the headline) + # and the downstream cluster/crop runs use the corrected regions. + _probe_blocks = cluster_all_article_blocks(regions, dateline_starts, sep_lines=sep_lines) + if recover_orphan_text_headlines(_probe_blocks, regions, page_img_path): + dateline_starts = find_article_starts_by_dateline(regions, page_img_path, paper) + + # Diagnostic image: highlight headline-type boxes (doc/paragraph/figure_title) + draw_titles_debug(page_img_path, regions, pages_dir, n) + # Diagnostic image: EVERY region from PaddleOCR, color-coded by type + draw_all_regions_debug(page_img_path, regions, pages_dir, n) + # Diagnostic image: EVERY candidate article block BEFORE political filtering + draw_all_article_boundaries_debug(page_img_path, regions, pages_dir, n, + dateline_starts, sep_lines=sep_lines) + # Diagnostic image: BLANK-canvas boundary map (headers + boundaries) before cropping + draw_boundary_map_debug(page_img_path, regions, pages_dir, n, + dateline_starts, sep_lines=sep_lines) + # Per-article crops (all articles, unfiltered) into output_dir/all_article_crops/page_NNN/ + n_all_crops = crop_all_article_blocks(page_img_path, regions, + Path(output_dir) / "all_article_crops", n, dateline_starts, + sep_lines=sep_lines) + # Flag crops that violate the dateline minimum-count rule (split/merged). + crop_integrity_issues += check_dateline_crop_integrity(regions, dateline_starts, n, + sep_lines=sep_lines) + + # Step 3: Claude — identify political articles + group regions (ONE call) + print(f"STEP 3: Page {n} — identifying political articles (1 Claude call)...") + result = identify_and_group_political_articles(page_img_path, regions, n, paper, + dateline_starts) + + political = result.get("political_articles", []) + rejected = result.get("rejected", []) + + # Headline verification: crop each doc_title region and ask Claude + # (via a small, scoped vision call) to transcribe the actual Telugu + # text. Drop articles where that transcription doesn't match the + # headline Claude originally reported. Catches the case where Claude + # hallucinates content (e.g. labelling a US-Iran peace deal article + # as "Deputy Collector's illegal assets"). + print(f"STEP 3b: Page {n} — verifying {len(political)} headline(s) against image...") + political = verify_headlines_with_claude(political, regions, page_img_path) + + # Geometry safeguard: split or trim Claude's groupings if member regions + # form spatially disjoint clusters (catches the multi-article bundling bug). + political = split_disjoint_articles(political, regions, dateline_starts=dateline_starts) + + # Correct over-grouping: split any article whose regions span 2+ datelines. + political = split_articles_on_datelines(political, regions, dateline_starts, sep_lines=sep_lines) + + # Dateline ceiling: drop regions above an article's start (they belong to + # the article above). Mirror of the downward floor; needs no datelines when + # the headline was detected. + political = trim_above_anchor(political, regions, dateline_starts) + + # Floor trim: drop a stray region below the article's text that belongs to + # the next article (e.g. a portrait beside the next headline). + political = trim_below_floor(political, regions, dateline_starts) + + # Final growth on the SETTLED shapes: the in-split grow pass can't see the + # reshaping done by the dateline split + trims, so under-grown articles get + # one more additive downward/rightward pass here. + political = grow_articles_final(political, regions, dateline_starts) + + # Save full result (post-safeguard) + (pages_dir / f"page_{n:03d}.political_result.json").write_text( + json.dumps({"political_articles": political, "rejected": rejected}, + indent=2, ensure_ascii=False), + encoding="utf-8", + ) + + # Per-page count log. Two independent counters, never blended: + # • Code side — datelines from the OCR scan + article blocks cropped, + # broken down by anchor kind (H headline / P paragraph / DL dateline- + # only / O orphan-or-other). + # • Claude side — political kept, rejected, and their total. + from collections import Counter as _KindCounter + _blocks = cluster_all_article_blocks(regions, dateline_starts, sep_lines=sep_lines) + _kind = _KindCounter(b["kind"] for b in _blocks) + n_datelines = len(dateline_starts) + n_headline = _kind.get("H", 0) + n_para = _kind.get("P", 0) + n_dl_block = _kind.get("DL", 0) + n_orphan = _kind.get("O", 0) + n_claude = len(political) + n_rejected = len(rejected) + print(f"\n 📊 PAGE {n} COUNTS:") + print(f" Code — datelines found : {n_datelines}") + print(f" Code — article blocks cropped : {n_all_crops}") + print(f" headline-anchored (H) : {n_headline}") + print(f" paragraph-anchored (P) : {n_para}") + print(f" dateline-only (DL) : {n_dl_block}") + print(f" orphans / other (O) : {n_orphan}") + print(f" Claude — political (kept) : {n_claude}") + print(f" Claude — rejected : {n_rejected}") + print(f" Claude — total identified : {n_claude + n_rejected} (kept + rejected)") + + # Per-page: Claude-identified headings vs Code-identified block headings. + log_identification_breakdown(page_img_path, regions, dateline_starts, + political, rejected, n, sep_lines=sep_lines) + + # --- DETAILED LOGGING --- + region_map = {r["id"]: r for r in regions} + + print(f"\n{'='*70}") + print(f"SMART EXTRACTOR — PAGE {n} RESULTS") + print(f"{'='*70}") + print(f" Input: {len(regions)} regions") + print(f" Output: {len(political)} political articles selected, {len(rejected)} rejected") + print() + + # Log each selected political article with region details + for i, art in enumerate(political, 1): + member_ids = art.get("member_region_ids", []) + print(f" ✅ POLITICAL ARTICLE {i}: \"{art.get('headline_english', '?')}\"") + print(f" Telugu: {art.get('headline_telugu', '?')}") + print(f" Priority: {art.get('priority', '?')} | Category: {art.get('category', '?')}") + print(f" Q1: {art.get('q1_reason', '?')}") + print(f" Q2: {art.get('q2_reason', '?')}") + print(f" Attack: {art.get('attack_angle', '?')}") + print(f" Member regions ({len(member_ids)}): {member_ids}") + + if member_ids: + min_x, min_y, max_x, max_y = 99999, 99999, 0, 0 + for rid in member_ids: + r = region_map.get(rid, {}) + rb = r.get("bbox", [0,0,0,0]) + print(f" Region {rid:3d}: type={r.get('type','?'):15s} bbox=[{rb[0]:5d},{rb[1]:5d},{rb[2]:5d},{rb[3]:5d}] size={rb[2]-rb[0]:5d}x{rb[3]-rb[1]:5d}px") + min_x = min(min_x, rb[0]) + min_y = min(min_y, rb[1]) + max_x = max(max_x, rb[2]) + max_y = max(max_y, rb[3]) + print(f" Merged bbox: [{min_x},{min_y},{max_x},{max_y}] size={max_x-min_x}x{max_y-min_y}px") + print() + + # Log rejected articles + if rejected: + print(f" REJECTED ARTICLES:") + for r in rejected: + print(f" ❌ {r.get('headline', '?')[:60]} — {r.get('reason', '?')[:60]}") + print() + + # Stats + if political: + region_counts = [len(art.get("member_region_ids", [])) for art in political] + print(f" STATS:") + print(f" Political articles: {len(political)}") + print(f" Regions per article: min={min(region_counts)}, max={max(region_counts)}, avg={sum(region_counts)/len(region_counts):.1f}") + large = [i+1 for i,c in enumerate(region_counts) if c > 6] + if large: + print(f" ⚠️ Large articles (>6 regions): {large} — may have merged multiple stories") + print(f"{'='*70}\n") + + # Draw debug image — political articles only with colored boundaries + try: + from PIL import ImageDraw + debug_img = Image.open(page_img_path).copy() + draw = ImageDraw.Draw(debug_img) + + article_colors = [ + (255, 0, 0), (0, 0, 255), (0, 150, 0), (180, 0, 180), + (200, 100, 0), (0, 150, 150), (150, 0, 0), (0, 0, 150), + (100, 100, 0), (200, 0, 100), (0, 100, 200), (150, 75, 0), + ] + + for idx, art in enumerate(political): + color = article_colors[idx % len(article_colors)] + member_ids = art.get("member_region_ids", []) + + # Draw each member region + for rid in member_ids: + r = region_map.get(rid, {}) + rb = r.get("bbox", [0,0,0,0]) + draw.rectangle([rb[0], rb[1], rb[2], rb[3]], outline=color, width=5) + draw.text((rb[0]+8, rb[1]+8), f"R{rid}", fill=color) + + # Draw merged outer boundary + if member_ids: + all_bboxes = [region_map.get(rid, {}).get("bbox", [0,0,0,0]) for rid in member_ids] + mx1 = min(b[0] for b in all_bboxes) + my1 = min(b[1] for b in all_bboxes) + mx2 = max(b[2] for b in all_bboxes) + my2 = max(b[3] for b in all_bboxes) + draw.rectangle([mx1-15, my1-15, mx2+15, my2+15], outline=color, width=15) + # Label + label = f"POLITICAL {idx+1}: {art.get('headline_english','')[:40]}" + draw.rectangle([mx1-15, my1-50, mx1 + len(label)*11, my1-15], fill=color) + draw.text((mx1-10, my1-48), label, fill="white") + + debug_path = page_img_path.replace(".png", "_political_debug.png") + debug_img.save(debug_path) + print(f"POLITICAL DEBUG IMAGE saved: {debug_path}") + print(f" Only political articles shown with colored boundaries.") + print() + except Exception as e: + print(f" Could not create debug image: {e}") + + page_summary.append((n, n_datelines, n_all_crops, n_orphan, n_claude, n_rejected)) + + if not political: + print(f" Page {n}: no political articles found") + continue + + # Step 4: Crop using precise PaddleOCR boxes + print(f"STEP 4: Page {n} — cropping {len(political)} political articles...") + records = crop_political_articles(page_img_path, political, regions, articles_dir, n, + dateline_starts=dateline_starts, sep_lines=sep_lines) + all_records.extend(records) + all_political.extend(political) + # Diagnostic image: final crop rectangle of every article on the page. + draw_article_boundaries_debug(page_img_path, records, pages_dir, n) + + print() + + # Per-page article-count summary table (logged for every job). + # "Actual articles" mirrors the code-identified article-block count. + if page_summary: + print(f"{'='*84}") + print("PER-PAGE ARTICLE COUNT SUMMARY") + print(f"{'='*84}") + print(f" {'Page':<6}{'Datelines':<12}{'Blocks (code)':<16}{'Orphans/Other':<16}" + f"{'Claude kept':<14}{'Rejected':<10}") + t_dl = t_bl = t_or = t_cl = t_rj = 0 + for n, n_dl, n_bl, n_or, n_cl, n_rj in page_summary: + print(f" {n:<6}{n_dl:<12}{n_bl:<16}{n_or:<16}{n_cl:<14}{n_rj:<10}") + t_dl += n_dl; t_bl += n_bl; t_or += n_or; t_cl += n_cl; t_rj += n_rj + print(f" {'-'*78}") + print(f" {'Total':<6}{t_dl:<12}{t_bl:<16}{t_or:<16}{t_cl:<14}{t_rj:<10}") + print(f"{'='*84}\n") + + # Cross-page handling: KEEP BOTH (per-page). + # A story that shares a headline across pages (e.g. a front-page lead plus + # the inside-page body) is NOT de-duplicated here. Deleting the smaller crop + # was wrong: an over-merged crop on one page could "win" on raw area and + # silently delete a correctly-cropped article on another page (front-page + # leads were disappearing this way). True continuations are stitched + # separately by _continuation.py; every page's selected article is retained. + if all_political: + # Informational only — report same-headline matches without dropping any. + seen_by_key = {} + for art in all_political: + key = _normalize_telugu(art.get("headline_telugu", "")) \ + or (art.get("headline_english", "") or "").strip().lower() + if not key: + continue + seen_by_key.setdefault(key, []).append(art) + for key, arts in seen_by_key.items(): + if len(arts) > 1: + pages_str = ", ".join(str(a.get("_page", "?")) for a in arts) + print(f" ↔ Same headline on pages {pages_str}: " + f"'{arts[0].get('headline_english','?')[:40]}' — keeping all (per-page).") + + # Step 5: Generate outputs + if all_political: + print("STEP 5: Generating outputs...") + (output_dir / "political_articles.json").write_text( + json.dumps(all_political, indent=2, ensure_ascii=False), encoding="utf-8" + ) + generate_brief(all_political, output_dir) + copy_all_images(all_records, output_dir) + generate_pdf(all_records, output_dir) + + # Post-run validator — surface crops/pages that likely need a human look. + write_review_report(all_political, pages_dir, output_dir, + crop_integrity_issues=crop_integrity_issues) + + print() + print(f"DONE! {len(all_records)} political articles found across {len(pages)} pages.") + print(f"Output: {output_dir}") + return output_dir + + +def _collect_pdfs(paths): + """Expand a mix of PDF files and folders into a de-duplicated, ordered PDF list.""" + pdfs = [] + for raw in paths: + p = Path(raw) + if p.is_dir(): + pdfs.extend(sorted(p.glob("*.pdf"))) + elif p.is_file() and p.suffix.lower() == ".pdf": + pdfs.append(p) + else: + print(f" ⚠️ Skipping (not a PDF or folder): {p}") + seen, unique = set(), [] + for p in pdfs: + rp = p.resolve() + if rp not in seen: + seen.add(rp) + unique.append(p) + return unique + + +# --------------------------------------------------------------------------- +# Main +# --------------------------------------------------------------------------- +def main(): + parser = argparse.ArgumentParser(description="Smart Extractor — political article detection") + parser.add_argument("paths", nargs="+", + help="One or more PDF files and/or folders containing PDFs") + parser.add_argument("--output", default=None, + help="Output base directory (default: ./output). " + "Each PDF gets its own subfolder _.") + parser.add_argument("--target-w", type=int, default=4200, help="Target page width in pixels") + parser.add_argument("--target-h", type=int, default=7400, help="Target page height in pixels") + args = parser.parse_args() + + pdfs = _collect_pdfs(args.paths) + if not pdfs: + print("ERROR: no PDF files found in the given path(s).") + sys.exit(1) + + base = Path(args.output) if args.output else Path("output") + print(f"Found {len(pdfs)} PDF(s) to process:") + for p in pdfs: + print(f" - {p.name}") + print() + + batch_ts = datetime.now().strftime("%Y%m%d_%H%M%S") + results = [] + all_crop_paths = [] + for i, pdf in enumerate(pdfs, 1): + timestamp = datetime.now().strftime("%Y%m%d_%H%M%S") + out_dir = base / f"{pdf.stem}_{timestamp}" + print("=" * 70) + print(f"[{i}/{len(pdfs)}] {pdf.name}") + print("=" * 70) + try: + process_pdf(pdf, out_dir, args.target_w, args.target_h) + all_crop_paths.extend(sorted((out_dir / "all_political_images").glob("*.png"))) + results.append(("ok", pdf.name, out_dir)) + except Exception as e: + import traceback + traceback.print_exc() + print(f" ❌ Failed on {pdf.name}: {e} — continuing with the next PDF.") + results.append(("FAILED", pdf.name, out_dir)) + + # One combined PDF of every cropped article across all processed PDFs. + if all_crop_paths: + print() + print(f"Building combined PDF of {len(all_crop_paths)} crop(s) from all PDFs...") + generate_combined_pdf(all_crop_paths, base / f"ALL_crops_{batch_ts}.pdf") + + print() + print("=" * 70) + print(f"BATCH COMPLETE — {len(pdfs)} PDF(s)") + for status, name, out_dir in results: + print(f" {status:>7} {name} → {out_dir}") + if all_crop_paths: + print(f" Combined crops PDF → {base / f'ALL_crops_{batch_ts}.pdf'}") + print("=" * 70) + + +if __name__ == "__main__": + main() diff --git a/static/style.css b/static/style.css new file mode 100644 index 0000000..b983987 --- /dev/null +++ b/static/style.css @@ -0,0 +1,157 @@ +* { box-sizing: border-box; } +body { + margin: 0; + font-family: system-ui, -apple-system, "Segoe UI", Roboto, sans-serif; + background: #faf8f3; + color: #222; + line-height: 1.55; +} +.wrap { max-width: 1100px; margin: 0 auto; padding: 0 24px; } + +header { + background: #fff; + border-bottom: 1px solid #e7e3d8; + padding: 16px 0; + margin-bottom: 32px; +} +header .wrap { + display: flex; align-items: baseline; gap: 16px; +} +.brand { + font-size: 20px; + font-weight: 600; + color: #222; + text-decoration: none; +} +.tag { color: #888; font-size: 13px; } + +main { min-height: 60vh; } + +.card { + background: #fff; + border: 1px solid #e7e3d8; + border-radius: 10px; + padding: 24px; + margin-bottom: 24px; +} +h1 { margin: 0 0 8px; font-size: 22px; } +h2 { margin: 0 0 12px; font-size: 18px; } +p { margin: 0 0 12px; } +.muted { color: #888; font-size: 14px; } + +form label { display: block; margin: 12px 0 6px; font-weight: 500; } +form label span { display: block; font-size: 14px; margin-bottom: 4px; } +.row { display: flex; gap: 16px; } +.row > label { flex: 1; } +input[type="file"], input[type="number"] { + width: 100%; + padding: 10px; + border: 1px solid #d9d4c4; + border-radius: 6px; + background: #fff; + font: inherit; +} +button { + margin-top: 16px; + padding: 10px 18px; + background: #2c2c2a; + color: #fff; + border: 0; + border-radius: 6px; + font: inherit; + cursor: pointer; +} +button:hover { background: #444; } + +.btn { + display: inline-block; + padding: 8px 14px; + background: #2c2c2a; + color: #fff; + text-decoration: none; + border-radius: 6px; + font-size: 14px; +} +.btn:hover { background: #444; } + +table { width: 100%; border-collapse: collapse; } +th, td { + padding: 10px 8px; + text-align: left; + border-bottom: 1px solid #f0ece1; +} +th { font-size: 13px; color: #888; font-weight: 500; } +code { background: #f4f0e4; padding: 1px 5px; border-radius: 3px; font-size: 13px; } + +.status { + display: inline-block; + padding: 2px 8px; + border-radius: 999px; + font-size: 12px; + text-transform: uppercase; + letter-spacing: 0.04em; + font-weight: 500; +} +.status-queued, .status-running { background: #ffeebb; color: #7a4500; } +.status-done { background: #d8ecd0; color: #205820; } +.status-error { background: #fbd9d4; color: #842018; } +.status-unknown { background: #e4e4e4; color: #555; } + +.grid { + display: grid; + grid-template-columns: repeat(auto-fill, minmax(280px, 1fr)); + gap: 16px; +} +.art { + border: 1px solid #e7e3d8; + border-radius: 8px; + overflow: hidden; + background: #fff; + display: flex; + flex-direction: column; +} +.art-img { + background: #f4f0e4; + max-height: 220px; + overflow: hidden; + display: flex; + align-items: center; + justify-content: center; +} +.art-img img { + max-width: 100%; + max-height: 220px; + display: block; +} +.art-meta { + padding: 12px; + font-size: 14px; + display: flex; + flex-direction: column; + gap: 6px; +} +.dateline { + font-size: 12px; + color: #2a6; + background: #eaf6ee; + padding: 4px 8px; + border-radius: 4px; + display: inline-block; +} +.head { + font-size: 14px; + color: #333; + font-weight: 500; + /* Telugu text may render in a fallback font; just leave native */ +} +.art-meta .row a { + font-size: 13px; + color: #2c2c2a; + text-decoration: none; + padding: 4px 10px; + border: 1px solid #ddd; + border-radius: 4px; +} +.art-meta .row a:hover { background: #f0ece1; } + +footer { color: #888; font-size: 13px; margin: 48px 0 24px; } diff --git a/templates/base.html b/templates/base.html new file mode 100644 index 0000000..ce3c439 --- /dev/null +++ b/templates/base.html @@ -0,0 +1,23 @@ + + + + + + {% block title %}Telugu Article Extractor{% endblock %} + + + +
+
+ Telugu Article Extractor + PDF → high-DPI pages → articles → text +
+
+
+ {% block content %}{% endblock %} +
+
+ Local-only. Files stay on this machine. +
+ + diff --git a/templates/index.html b/templates/index.html new file mode 100644 index 0000000..8fa527a --- /dev/null +++ b/templates/index.html @@ -0,0 +1,51 @@ +{% extends "base.html" %} +{% block content %} + +
+

Upload a PDF

+

Each page will be rendered to fit inside the pixel box below, then split into per-article images and text.

+
+ +
+ + +
+ +
+
+ +
+

Recent jobs

+ {% if jobs %} + + + + + + {% for j in jobs %} + + + + + + + + + {% endfor %} + +
IDFilePagesArticlesStatus
{{ j.id }}{{ j.filename }}{{ j.pages }}{{ j.articles }}{{ j.status }}Open
+ {% else %} +

No jobs yet.

+ {% endif %} +
+ +{% endblock %} diff --git a/templates/job.html b/templates/job.html new file mode 100644 index 0000000..afa4889 --- /dev/null +++ b/templates/job.html @@ -0,0 +1,76 @@ +{% extends "base.html" %} +{% block content %} + +
+

Job {{ job_id }}

+

{{ meta.filename }}

+

+ Status: {{ meta.status }} + +

+

+ Target render: {{ meta.target_width }} × {{ meta.target_height }} px · + Pages found: {{ meta.pages }} · + Articles found: {{ articles|length }} +

+

+ ⬇ Download all (zip) +

+
+ +{% if articles %} +
+

Articles

+
+ {% for a in articles %} +
+
+ {% if a.has_image %} + + {% endif %} +
+
+ {{ a.name }} + page {{ a.page }}{% if a.grouped_by %} · grouped by {{ a.grouped_by }}{% endif %} + {% if a.dateline %}
{{ a.dateline }}
{% endif %} + {% if a.headline_preview %}
{{ a.headline_preview }}
{% endif %} +
+ {% if a.has_image %}image{% endif %} + {% if a.has_text %}text{% endif %} +
+
+
+ {% endfor %} +
+
+{% endif %} + + + +{% endblock %} diff --git a/test_padding.py b/test_padding.py new file mode 100644 index 0000000..e69de29 diff --git a/test_pdf.py b/test_pdf.py new file mode 100644 index 0000000..e69de29