================================================================================
TELUGU POLITICAL-ARTICLE EXTRACTOR - EDGE CASES, BUGS & RULES
================================================================================
Pipeline:  PDF -> PaddleOCR layout regions -> Claude grouping
           (identify_and_group_political_articles) -> deterministic geometric
           safeguards in crop_political_articles() -> per-page crops ->
           political_articles.json.  Separator/line + page-bottom detection
           lives in _lines.py (classical CV, no ML, no API).

This document lists every edge case / bug / rule handled so far, plus the ones
still in progress.  Each point: PROBLEM -> RULE/FIX -> STATUS -> VERIFICATION.

Papers seen (identified by page-1 pixel height):
  6422            = namaste_telangana   (no printer calibration strip)
  6549 / 6833     = andhra_jyothi       (CMYK calibration strip at bottom)
  7040-7060       = sakshi              (CMYK calibration strip at bottom)

================================================================================
1. GROW-RIGHT TO THE BOUNDING VERTICAL WALL   [FIXED]
================================================================================
PROBLEM:
  An under-grouped article was cropped too NARROW - its body clearly continued
  into the next column to the right, but Claude grouped only the left column, so
  the crop stopped mid-article.
  Example: run 20260602_093820  p003_a003  "రైతన్న లోడెత్తేదెలా?".

RULE / FIX (in crop_political_articles, after the Rule-B floor block):
  - Detect a long vertical separator line (len >= 600 px) that sits to the RIGHT
    of the current crop (x > rx2 + 10) and overlaps the article vertically
    (> 50% y-overlap).  That line is the article's right "wall".
  - Absorb every text / image / figure_title / paragraph_title region whose
    center-x falls between the current right edge (rx2) and the wall.
  - Push the right edge out to the wall: rx2 = wall_x, right_cap = wall_x.
  - The padding step clamps x2 = min(x2, right_cap) so it never overshoots.

WIDTH-CAP GUARD (prevents over-grab):
  too_far = (wall_x - rx2) > max(art_w, 1)
  If the wall is farther away than the article's own width, it is a page-margin
  rule swallowing a NEIGHBOUR, not the article's wall -> do NOT grow.
  Also bail if a FOREIGN doc_title (a different story's headline) lies inside the
  strip we are about to absorb.
  Regression that drove this guard: run 20260601_234709 p007_a001 grew from
  [77,634,1035,1609] to [77,634,4127,2581], eating neighbour "భూముల విలువ పెంపు".

STATUS: Fixed & committed (HEAD 2345453).  Prints "➡️ Grew ...".

================================================================================
2. ROBUST JSON PARSING OF CLAUDE'S GROUPING RESPONSE   [FIXED]
================================================================================
PROBLEM:
  One page's grouping response was empty / truncated / fence-wrapped, and the
  fragile parse of content[0].text crashed the WHOLE run:
     json.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
  Example: a run crashed on page 10.

RULE / FIX (smart_extractor.py):
  - New helper _extract_json_obj(text): strips ```json fences / prose, extracts
    the outermost {...} object, raises ValueError on empty/garbage.
  - Gather text from ALL response content blocks (not just block 0).
  - max_tokens raised to 8192 to avoid truncated JSON.
  - On (ValueError, json.JSONDecodeError): save page_NNN.badresponse.txt for
    debugging, then RETRY once in strict mode ("Return ONLY the raw JSON ...").
  - If the retry also fails: return {"political_articles": [], "rejected": []}
    so that ONE bad page degrades gracefully instead of killing the run.
  (json.JSONDecodeError is a subclass of ValueError - both are caught.)

STATUS: Fixed.

================================================================================
3. RULE-B OVER-FIRE: in-article subhead wrongly treated as a new headline [FIXED]
================================================================================
PROBLEM:
  Rule B floors a crop at the next headline that starts a DIFFERENT story in the
  same column.  But a paragraph_title that is really an IN-ARTICLE byline /
  subhead (narrow, with body text continuing below it) was being treated as a
  new-article headline, so the crop was floored far too high and dropped 9 of the
  article's member regions.
  Example: run 20260603_112014 p003_a001 "తీరని కొనుగోళ్ల వెతలు" - paragraph_title
  R79 [1193,5226,2035,5271] (an internal subhead) floored the crop at y5226.

RULE / FIX (Rule B block in crop_political_articles):
  - A doc_title floors the crop UNCONDITIONALLY (a real new story headline).
  - A paragraph_title floors ONLY if there is a HORIZONTAL SEPARATOR RULE
    directly above it (a printed line above a heading => genuine new section).
    Implemented via _rule_above(): a detected h-line whose y is between the
    primary headline bottom and the candidate top, overlapping > 40% of the
    candidate's width.
  - R79 has NO rule above -> no longer floors (correct).

VERIFICATION (offline, no API):
  - 20260603_112014 p003_a001 now keeps all members; bbox grew correctly down.
  - 20260602_093820 R81 [1162,1879,2063,1926] HAS a rule above (y1707,
    overlap 787) -> STILL floors correctly (legitimate case preserved).
  - 20260601_234709 known-good crops unaffected.

STATUS: Fixed.

NOTE / FOLLOW-UP: the cross-run downward-growth sweep (>250 px) flagged a few
crops that grew DOWN after this change (they were previously floored by a
paragraph_title without a rule above).  Largest legit one is the 112014 fix
itself.  Others to re-confirm are NOT over-merges (swallowing a separate article
below):
   20260602_081720 p004_a001  DOWN+1951
   20260531_231056 p002_a004/a005/a006, p002_a001
   20260601_005917 p002_a006
   20260531_205325 p002_a005
[STILL TO VALIDATE]

================================================================================
4. PAGE-BOTTOM PRINTER CALIBRATION STRIP DETECTION   [FIXED - needs wiring]
================================================================================
PROBLEM:
  Andhra Jyothi & Sakshi print a CMYK colour-calibration strip (registration
  dots + grey bars) just above the bottom paper margin.  Article crops whose
  member regions reach the page bottom (esp. footer-type regions) bleed this
  strip of coloured dots into the article image.

RULE / FIX (_lines.py: detect_content_bottom):
  Returns the y at the TOP of the calibration band (clamp crops to it), or the
  full height H when there is no strip (nothing to clamp).
  Algorithm:
   a. Search the bottom search_frac (12%) of the page.
   b. Per-row colour fraction col_frac (saturated CMYK: sat>55 & max>80) and ink
      fraction; the strip = the bottom-most run of rows whose col_frac >= 0.06.
   c. Require a WHITE PAPER MARGIN below the band down to the page edge
      (>=90% blank) - a real strip is followed by margin.
   d. Walk UP from the band bottom while still coloured, tolerating small white
      gaps between dot rows (<=12 px).
   e. Guard max_band (160 px): if the band is too tall it is fused with a photo
      -> bail (return H).
   f. PEAK-COLOUR DISCRIMINATOR (key false-positive fix): require the band's
      peak col_frac >= 0.15.  Real registration strips are dense multi-hue CMYK
      dots peaking 0.22-0.45; a single-hue coloured ad/notice box peaks ~0.11.

FALSE POSITIVE that drove the peak-colour gate:
  namaste 20260603_113847 p004 has a GREEN-bordered notice box near the bottom
  (col peak 0.113) embedded in article text.  An earlier "white gutter above"
  gate rejected it but WRONGLY rejected real AJ strips on busy pages (p3/p6 had a
  photo extending into the gutter).  The peak-colour gate (0.15) cleanly
  separates: namaste box 0.11 -> reject; AJ 0.25 / Sakshi 0.22 / AJ-6549 0.45 ->
  accept.

VERIFICATION (offline):
  AJ-6833  -> clamps ~6674-6687  (drop ~150)
  AJ-6549  -> clamps ~6377-6401  (drop ~150-172)   (was a false NEGATIVE before)
  Sakshi   -> clamps ~6720-6721  (drop ~332)
  namaste  -> H (no clamp)        (correct - no strip; green box not clamped)
  AJ pages with a full-bleed bottom photo -> H (acceptable: no clean strip).

STATUS: Detector fixed & verified across all 4 papers.
PENDING: wire into crop_political_articles as a per-page y2 clamp
  (y2 = min(y2, content_bottom), computed once per page); then re-verify
  112014 p003_a001 no longer includes calibration dots + regression-check.

================================================================================
5. MULTI-COLUMN BANNER HEADLINE = ONE ARTICLE SPANNING ALL ITS COLUMNS
   [FIXED]
================================================================================
PROBLEM:
  When a single wide headline (banner) spans MULTIPLE columns, everything beneath
  it (down to its bottom rule) is ONE article - even though it contains several
  internal sub-headlines / sub-sections and photos.  The foreign-headline clip
  (Rule A) and Rule B wrongly treat those internal sub-headlines as new stories
  and truncate the crop, so the banner article comes out severely under-cropped.
  Example: namaste 20260603_113847 p004 article 1
    "రైతులను వీడని అన్‌లోడింగ్ సమస్య"
    Banner headline id=64 [1264,423,3500,628]  (width 2236 = 0.53 of page,
    crosses the internal vertical rule at x=2388 -> multi-column banner).
    Bounding vertical walls: x=1232 (left), x=3545 (right).
    Bottom rule: full-banner-width h-line at y=2756 (x[1268..3512], len 2244).
    Correct crop ~= [1232, 423, 3545, 2756].
    Current (buggy) crop = [1264,423,3500,2183] - cut off ~570 px too short and
    not extended to the side walls.

RULE (to implement):
  - BANNER TEST: an article's top headline (doc_title) is a banner if it CROSSES
    at least one internal vertical separator line (i.e. it spans >1 column).
    (Equivalently width >= ~0.4 of page width.)
  - For a banner article:
      * Horizontal extent = bounded by the vertical separator lines just LEFT and
        just RIGHT of the headline (left_wall .. right_wall).
      * Bottom = the LOWEST/own bottom horizontal rule that spans most of the
        banner width (x-overlap > ~0.6 of banner width) and lies below the
        headline -> the first such full-width rule below the grouped content.
      * Crop = [left_wall, headline_top-pad, right_wall, bottom_rule].
      * SKIP the internal foreign-headline clip and Rule B floor for this article
        (its sub-headlines are part of it).
  - "If the multi-column header HAS a bottom line, crop to it; if it does NOT,
    fall back to the previous (member-union + Rule A/B) logic." (user's words)

IMPLEMENTED (crop_political_articles, banner_box branch):
  - BANNER TEST: topmost member doc_title width >= 0.40 * page_width.
  - left_wall / right_wall = vertical separator lines reaching alongside the
    headline, just left of / right of it (fallback to member edge if none).
  - bottom_rule = the first horizontal line BELOW the grouped content whose
    x-span overlaps > 60% of the banner width (the >60% width test rejects short
    caption/photo underlines - see point 6). Require bottom_rule > headline_top+200.
  - Only when BOTH walls-ish extent AND a bottom_rule exist do we set
    banner_box = [left_wall, headline_top, right_wall, bottom_rule]; this OVERRIDES
    the per-column clip / Rule-B floor / grow-right results and restores the full
    member set. No bottom rule -> banner_box stays None -> normal logic (fallback).

VERIFICATION (offline, namaste 20260603_113847 p4 art1):
  - Banner fired: x[1232..3545] y[423..2756] -> padded crop [1217,408,3560,2771].
  - Visual check: crop now contains the full headline + every column beneath it
    down to the bottom rule (previously cut at y~2183).
  - Regression: banner does NOT fire on normal single-column-headline articles
    (threshold 0.40*pw); other pages' crops unchanged.

STATUS: Fixed & verified offline.

================================================================================
6. HORIZONTAL RULE WITH TEXT/IMAGE DIRECTLY ABOVE IT = UNDERLINE, NOT SEPARATOR
   [PARTIALLY APPLIED - new rule, affects points 3, 4, 5]
================================================================================
PROBLEM:
  Not every horizontal printed line separates two articles.  A line that has text
  or an image DIRECTLY ABOVE it is an UNDERLINE - a caption underline beneath a
  photo, or a decorative underline under a heading.  Using such a line as an
  article boundary (a Rule-B "rule above", or a banner BOTTOM rule, or a page
  separator) is wrong.
  Examples (user-supplied): photo captions like "జక్కల్ శివారులో నిర్మించిన ..."
  and "నర్సాపూర్‌లో మాట్లాడుతున్న మాజీ మంత్రి ..." each sit ABOVE a thin rule -
  those rules underline the caption, they do not divide articles.

RULE (to implement):
  When deciding whether a horizontal line is an ARTICLE SEPARATOR (for Rule B's
  "rule above" check, for the banner bottom rule in point 5, and anywhere a
  h-line is used as a boundary):
    - Reject the line if the tight band immediately ABOVE it is mostly INK
      (text) or is an image/figure region -> it is an underline, not a separator.
    - A genuine article separator sits in a whitespace gutter (paper above AND
      below), OR has a NEW headline starting just below it.
  (This is analogous to the existing _isolated() whitespace-gutter test in
  _lines.py used for separator-line detection.)

APPLIED SO FAR:
  - Banner bottom rule (point 5): only horizontal lines spanning > 60% of the
    banner width qualify, which already excludes short caption/photo underlines
    (those are < ~0.5 column wide).
  - Note: _lines.detect_separator_lines() already runs the _isolated() gutter
    test, so the h-lines fed to Rule B / banner are whitespace-gutter lines, not
    arbitrary text strokes.
STILL TO DO:
  - Add an explicit "ink/image directly above" rejection to Rule-B's
    _rule_above() so a caption underline beneath a photo can never corroborate a
    paragraph_title floor, even if a future detector change lets one through.

STATUS: Partially applied (via width test + existing gutter detection);
        explicit ink-above guard for Rule B still to add.

================================================================================
CROSS-CUTTING NOTES / CONVENTIONS
================================================================================
- BILLING/SECURITY CONSTRAINT: do NOT call the Claude APIs while iterating.  All
  verification is done OFFLINE via fixture replay of saved page PNGs +
  regions.json + cached per-page political_result.json.
- Calibration strip is NOT an OCR region; crops only reach it via member-region
  bboxes (especially footer/header/number page-furniture regions) or padding.
  Consider dropping footer/header/number members from crops as well.
- Test harnesses (in /tmp): replay_crop.py, replay_run.py, replay_regress.py
  (new-vs-stored bbox diff), sweep_floor.py (flags downward growth >250 px),
  test_bottom.py / test_all.py (detect_content_bottom across papers).
================================================================================

can u creat img with boundaris of ech articl in th  empty whit pg wit diffrnt colr ech articl so tht i can see if thy r overlppng r ?
