267 lines
15 KiB
Plaintext
267 lines
15 KiB
Plaintext
================================================================================
|
||
TELUGU POLITICAL-ARTICLE EXTRACTOR - EDGE CASES, BUGS & RULES
|
||
================================================================================
|
||
Pipeline: PDF -> PaddleOCR layout regions -> Claude grouping
|
||
(identify_and_group_political_articles) -> deterministic geometric
|
||
safeguards in crop_political_articles() -> per-page crops ->
|
||
political_articles.json. Separator/line + page-bottom detection
|
||
lives in _lines.py (classical CV, no ML, no API).
|
||
|
||
This document lists every edge case / bug / rule handled so far, plus the ones
|
||
still in progress. Each point: PROBLEM -> RULE/FIX -> STATUS -> VERIFICATION.
|
||
|
||
Papers seen (identified by page-1 pixel height):
|
||
6422 = namaste_telangana (no printer calibration strip)
|
||
6549 / 6833 = andhra_jyothi (CMYK calibration strip at bottom)
|
||
7040-7060 = sakshi (CMYK calibration strip at bottom)
|
||
|
||
================================================================================
|
||
1. GROW-RIGHT TO THE BOUNDING VERTICAL WALL [FIXED]
|
||
================================================================================
|
||
PROBLEM:
|
||
An under-grouped article was cropped too NARROW - its body clearly continued
|
||
into the next column to the right, but Claude grouped only the left column, so
|
||
the crop stopped mid-article.
|
||
Example: run 20260602_093820 p003_a003 "రైతన్న లోడెత్తేదెలా?".
|
||
|
||
RULE / FIX (in crop_political_articles, after the Rule-B floor block):
|
||
- Detect a long vertical separator line (len >= 600 px) that sits to the RIGHT
|
||
of the current crop (x > rx2 + 10) and overlaps the article vertically
|
||
(> 50% y-overlap). That line is the article's right "wall".
|
||
- Absorb every text / image / figure_title / paragraph_title region whose
|
||
center-x falls between the current right edge (rx2) and the wall.
|
||
- Push the right edge out to the wall: rx2 = wall_x, right_cap = wall_x.
|
||
- The padding step clamps x2 = min(x2, right_cap) so it never overshoots.
|
||
|
||
WIDTH-CAP GUARD (prevents over-grab):
|
||
too_far = (wall_x - rx2) > max(art_w, 1)
|
||
If the wall is farther away than the article's own width, it is a page-margin
|
||
rule swallowing a NEIGHBOUR, not the article's wall -> do NOT grow.
|
||
Also bail if a FOREIGN doc_title (a different story's headline) lies inside the
|
||
strip we are about to absorb.
|
||
Regression that drove this guard: run 20260601_234709 p007_a001 grew from
|
||
[77,634,1035,1609] to [77,634,4127,2581], eating neighbour "భూముల విలువ పెంపు".
|
||
|
||
STATUS: Fixed & committed (HEAD 2345453). Prints "➡️ Grew ...".
|
||
|
||
================================================================================
|
||
2. ROBUST JSON PARSING OF CLAUDE'S GROUPING RESPONSE [FIXED]
|
||
================================================================================
|
||
PROBLEM:
|
||
One page's grouping response was empty / truncated / fence-wrapped, and the
|
||
fragile parse of content[0].text crashed the WHOLE run:
|
||
json.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
|
||
Example: a run crashed on page 10.
|
||
|
||
RULE / FIX (smart_extractor.py):
|
||
- New helper _extract_json_obj(text): strips ```json fences / prose, extracts
|
||
the outermost {...} object, raises ValueError on empty/garbage.
|
||
- Gather text from ALL response content blocks (not just block 0).
|
||
- max_tokens raised to 8192 to avoid truncated JSON.
|
||
- On (ValueError, json.JSONDecodeError): save page_NNN.badresponse.txt for
|
||
debugging, then RETRY once in strict mode ("Return ONLY the raw JSON ...").
|
||
- If the retry also fails: return {"political_articles": [], "rejected": []}
|
||
so that ONE bad page degrades gracefully instead of killing the run.
|
||
(json.JSONDecodeError is a subclass of ValueError - both are caught.)
|
||
|
||
STATUS: Fixed.
|
||
|
||
================================================================================
|
||
3. RULE-B OVER-FIRE: in-article subhead wrongly treated as a new headline [FIXED]
|
||
================================================================================
|
||
PROBLEM:
|
||
Rule B floors a crop at the next headline that starts a DIFFERENT story in the
|
||
same column. But a paragraph_title that is really an IN-ARTICLE byline /
|
||
subhead (narrow, with body text continuing below it) was being treated as a
|
||
new-article headline, so the crop was floored far too high and dropped 9 of the
|
||
article's member regions.
|
||
Example: run 20260603_112014 p003_a001 "తీరని కొనుగోళ్ల వెతలు" - paragraph_title
|
||
R79 [1193,5226,2035,5271] (an internal subhead) floored the crop at y5226.
|
||
|
||
RULE / FIX (Rule B block in crop_political_articles):
|
||
- A doc_title floors the crop UNCONDITIONALLY (a real new story headline).
|
||
- A paragraph_title floors ONLY if there is a HORIZONTAL SEPARATOR RULE
|
||
directly above it (a printed line above a heading => genuine new section).
|
||
Implemented via _rule_above(): a detected h-line whose y is between the
|
||
primary headline bottom and the candidate top, overlapping > 40% of the
|
||
candidate's width.
|
||
- R79 has NO rule above -> no longer floors (correct).
|
||
|
||
VERIFICATION (offline, no API):
|
||
- 20260603_112014 p003_a001 now keeps all members; bbox grew correctly down.
|
||
- 20260602_093820 R81 [1162,1879,2063,1926] HAS a rule above (y1707,
|
||
overlap 787) -> STILL floors correctly (legitimate case preserved).
|
||
- 20260601_234709 known-good crops unaffected.
|
||
|
||
STATUS: Fixed.
|
||
|
||
NOTE / FOLLOW-UP: the cross-run downward-growth sweep (>250 px) flagged a few
|
||
crops that grew DOWN after this change (they were previously floored by a
|
||
paragraph_title without a rule above). Largest legit one is the 112014 fix
|
||
itself. Others to re-confirm are NOT over-merges (swallowing a separate article
|
||
below):
|
||
20260602_081720 p004_a001 DOWN+1951
|
||
20260531_231056 p002_a004/a005/a006, p002_a001
|
||
20260601_005917 p002_a006
|
||
20260531_205325 p002_a005
|
||
[STILL TO VALIDATE]
|
||
|
||
================================================================================
|
||
4. PAGE-BOTTOM PRINTER CALIBRATION STRIP DETECTION [FIXED - needs wiring]
|
||
================================================================================
|
||
PROBLEM:
|
||
Andhra Jyothi & Sakshi print a CMYK colour-calibration strip (registration
|
||
dots + grey bars) just above the bottom paper margin. Article crops whose
|
||
member regions reach the page bottom (esp. footer-type regions) bleed this
|
||
strip of coloured dots into the article image.
|
||
|
||
RULE / FIX (_lines.py: detect_content_bottom):
|
||
Returns the y at the TOP of the calibration band (clamp crops to it), or the
|
||
full height H when there is no strip (nothing to clamp).
|
||
Algorithm:
|
||
a. Search the bottom search_frac (12%) of the page.
|
||
b. Per-row colour fraction col_frac (saturated CMYK: sat>55 & max>80) and ink
|
||
fraction; the strip = the bottom-most run of rows whose col_frac >= 0.06.
|
||
c. Require a WHITE PAPER MARGIN below the band down to the page edge
|
||
(>=90% blank) - a real strip is followed by margin.
|
||
d. Walk UP from the band bottom while still coloured, tolerating small white
|
||
gaps between dot rows (<=12 px).
|
||
e. Guard max_band (160 px): if the band is too tall it is fused with a photo
|
||
-> bail (return H).
|
||
f. PEAK-COLOUR DISCRIMINATOR (key false-positive fix): require the band's
|
||
peak col_frac >= 0.15. Real registration strips are dense multi-hue CMYK
|
||
dots peaking 0.22-0.45; a single-hue coloured ad/notice box peaks ~0.11.
|
||
|
||
FALSE POSITIVE that drove the peak-colour gate:
|
||
namaste 20260603_113847 p004 has a GREEN-bordered notice box near the bottom
|
||
(col peak 0.113) embedded in article text. An earlier "white gutter above"
|
||
gate rejected it but WRONGLY rejected real AJ strips on busy pages (p3/p6 had a
|
||
photo extending into the gutter). The peak-colour gate (0.15) cleanly
|
||
separates: namaste box 0.11 -> reject; AJ 0.25 / Sakshi 0.22 / AJ-6549 0.45 ->
|
||
accept.
|
||
|
||
VERIFICATION (offline):
|
||
AJ-6833 -> clamps ~6674-6687 (drop ~150)
|
||
AJ-6549 -> clamps ~6377-6401 (drop ~150-172) (was a false NEGATIVE before)
|
||
Sakshi -> clamps ~6720-6721 (drop ~332)
|
||
namaste -> H (no clamp) (correct - no strip; green box not clamped)
|
||
AJ pages with a full-bleed bottom photo -> H (acceptable: no clean strip).
|
||
|
||
STATUS: Detector fixed & verified across all 4 papers.
|
||
PENDING: wire into crop_political_articles as a per-page y2 clamp
|
||
(y2 = min(y2, content_bottom), computed once per page); then re-verify
|
||
112014 p003_a001 no longer includes calibration dots + regression-check.
|
||
|
||
================================================================================
|
||
5. MULTI-COLUMN BANNER HEADLINE = ONE ARTICLE SPANNING ALL ITS COLUMNS
|
||
[FIXED]
|
||
================================================================================
|
||
PROBLEM:
|
||
When a single wide headline (banner) spans MULTIPLE columns, everything beneath
|
||
it (down to its bottom rule) is ONE article - even though it contains several
|
||
internal sub-headlines / sub-sections and photos. The foreign-headline clip
|
||
(Rule A) and Rule B wrongly treat those internal sub-headlines as new stories
|
||
and truncate the crop, so the banner article comes out severely under-cropped.
|
||
Example: namaste 20260603_113847 p004 article 1
|
||
"రైతులను వీడని అన్లోడింగ్ సమస్య"
|
||
Banner headline id=64 [1264,423,3500,628] (width 2236 = 0.53 of page,
|
||
crosses the internal vertical rule at x=2388 -> multi-column banner).
|
||
Bounding vertical walls: x=1232 (left), x=3545 (right).
|
||
Bottom rule: full-banner-width h-line at y=2756 (x[1268..3512], len 2244).
|
||
Correct crop ~= [1232, 423, 3545, 2756].
|
||
Current (buggy) crop = [1264,423,3500,2183] - cut off ~570 px too short and
|
||
not extended to the side walls.
|
||
|
||
RULE (to implement):
|
||
- BANNER TEST: an article's top headline (doc_title) is a banner if it CROSSES
|
||
at least one internal vertical separator line (i.e. it spans >1 column).
|
||
(Equivalently width >= ~0.4 of page width.)
|
||
- For a banner article:
|
||
* Horizontal extent = bounded by the vertical separator lines just LEFT and
|
||
just RIGHT of the headline (left_wall .. right_wall).
|
||
* Bottom = the LOWEST/own bottom horizontal rule that spans most of the
|
||
banner width (x-overlap > ~0.6 of banner width) and lies below the
|
||
headline -> the first such full-width rule below the grouped content.
|
||
* Crop = [left_wall, headline_top-pad, right_wall, bottom_rule].
|
||
* SKIP the internal foreign-headline clip and Rule B floor for this article
|
||
(its sub-headlines are part of it).
|
||
- "If the multi-column header HAS a bottom line, crop to it; if it does NOT,
|
||
fall back to the previous (member-union + Rule A/B) logic." (user's words)
|
||
|
||
IMPLEMENTED (crop_political_articles, banner_box branch):
|
||
- BANNER TEST: topmost member doc_title width >= 0.40 * page_width.
|
||
- left_wall / right_wall = vertical separator lines reaching alongside the
|
||
headline, just left of / right of it (fallback to member edge if none).
|
||
- bottom_rule = the first horizontal line BELOW the grouped content whose
|
||
x-span overlaps > 60% of the banner width (the >60% width test rejects short
|
||
caption/photo underlines - see point 6). Require bottom_rule > headline_top+200.
|
||
- Only when BOTH walls-ish extent AND a bottom_rule exist do we set
|
||
banner_box = [left_wall, headline_top, right_wall, bottom_rule]; this OVERRIDES
|
||
the per-column clip / Rule-B floor / grow-right results and restores the full
|
||
member set. No bottom rule -> banner_box stays None -> normal logic (fallback).
|
||
|
||
VERIFICATION (offline, namaste 20260603_113847 p4 art1):
|
||
- Banner fired: x[1232..3545] y[423..2756] -> padded crop [1217,408,3560,2771].
|
||
- Visual check: crop now contains the full headline + every column beneath it
|
||
down to the bottom rule (previously cut at y~2183).
|
||
- Regression: banner does NOT fire on normal single-column-headline articles
|
||
(threshold 0.40*pw); other pages' crops unchanged.
|
||
|
||
STATUS: Fixed & verified offline.
|
||
|
||
================================================================================
|
||
6. HORIZONTAL RULE WITH TEXT/IMAGE DIRECTLY ABOVE IT = UNDERLINE, NOT SEPARATOR
|
||
[PARTIALLY APPLIED - new rule, affects points 3, 4, 5]
|
||
================================================================================
|
||
PROBLEM:
|
||
Not every horizontal printed line separates two articles. A line that has text
|
||
or an image DIRECTLY ABOVE it is an UNDERLINE - a caption underline beneath a
|
||
photo, or a decorative underline under a heading. Using such a line as an
|
||
article boundary (a Rule-B "rule above", or a banner BOTTOM rule, or a page
|
||
separator) is wrong.
|
||
Examples (user-supplied): photo captions like "జక్కల్ శివారులో నిర్మించిన ..."
|
||
and "నర్సాపూర్లో మాట్లాడుతున్న మాజీ మంత్రి ..." each sit ABOVE a thin rule -
|
||
those rules underline the caption, they do not divide articles.
|
||
|
||
RULE (to implement):
|
||
When deciding whether a horizontal line is an ARTICLE SEPARATOR (for Rule B's
|
||
"rule above" check, for the banner bottom rule in point 5, and anywhere a
|
||
h-line is used as a boundary):
|
||
- Reject the line if the tight band immediately ABOVE it is mostly INK
|
||
(text) or is an image/figure region -> it is an underline, not a separator.
|
||
- A genuine article separator sits in a whitespace gutter (paper above AND
|
||
below), OR has a NEW headline starting just below it.
|
||
(This is analogous to the existing _isolated() whitespace-gutter test in
|
||
_lines.py used for separator-line detection.)
|
||
|
||
APPLIED SO FAR:
|
||
- Banner bottom rule (point 5): only horizontal lines spanning > 60% of the
|
||
banner width qualify, which already excludes short caption/photo underlines
|
||
(those are < ~0.5 column wide).
|
||
- Note: _lines.detect_separator_lines() already runs the _isolated() gutter
|
||
test, so the h-lines fed to Rule B / banner are whitespace-gutter lines, not
|
||
arbitrary text strokes.
|
||
STILL TO DO:
|
||
- Add an explicit "ink/image directly above" rejection to Rule-B's
|
||
_rule_above() so a caption underline beneath a photo can never corroborate a
|
||
paragraph_title floor, even if a future detector change lets one through.
|
||
|
||
STATUS: Partially applied (via width test + existing gutter detection);
|
||
explicit ink-above guard for Rule B still to add.
|
||
|
||
================================================================================
|
||
CROSS-CUTTING NOTES / CONVENTIONS
|
||
================================================================================
|
||
- BILLING/SECURITY CONSTRAINT: do NOT call the Claude APIs while iterating. All
|
||
verification is done OFFLINE via fixture replay of saved page PNGs +
|
||
regions.json + cached per-page political_result.json.
|
||
- Calibration strip is NOT an OCR region; crops only reach it via member-region
|
||
bboxes (especially footer/header/number page-furniture regions) or padding.
|
||
Consider dropping footer/header/number members from crops as well.
|
||
- Test harnesses (in /tmp): replay_crop.py, replay_run.py, replay_regress.py
|
||
(new-vs-stored bbox diff), sweep_floor.py (flags downward growth >250 px),
|
||
test_bottom.py / test_all.py (detect_content_bottom across papers).
|
||
================================================================================
|
||
|
||
can u creat img with boundaris of ech articl in th empty whit pg wit diffrnt colr ech articl so tht i can see if thy r overlppng r ?
|