Telugu Article Extractor

A local web app that takes a Telugu newspaper PDF and produces one PNG + one TXT file per article.

Pipeline

PDF
 │ Stage 1  PyMuPDF  →  page_001.png  (target: 4200×7400 px, fits aspect)
 ▼
Pages
 │ Stage 2  PaddleOCR PP-Structure  →  regions.json
 │            (title / text / figure rectangles with bboxes)
 ▼
Regions
 │ Stage 3  Article grouping  →  articles.json
 │            Primary: Claude API (spatial reasoning over regions,
 │                     sees page thumbnail + region JSON)
 │            Fallback: column-aware + dateline-anchored Python rules
 │                     • title region starts a new article
 │                     • "<place>, మే <day> (ఆంధ్రజ్యోతి):" dateline
 │                       also starts a new article
 │                     • regions partitioned by column first to avoid
 │                       cross-column contamination
 ▼
Articles (bboxes)
 │ Stage 4  PIL crop  →  article.png  per article
 ▼
Article PNGs
 │ Stage 5  Tesseract (lang=tel)  →  article.txt  per article
 ▼
Done. Browse in the UI, download all as zip.

Setup

1. System dependencies

# Ubuntu / Debian
sudo apt update
sudo apt install -y tesseract-ocr tesseract-ocr-tel libgl1

tesseract-ocr-tel is the Telugu language data — required for the per-article OCR step.

libgl1 is needed by paddlepaddle on headless Linux.

2. Python environment

cd telugu_extractor
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

First run of paddlepaddle/paddleocr will download model weights (~400 MB) into ~/.paddleocr/. This happens once.

3. (Optional) Enable Claude grouping

Stage 3 (article grouping) works two ways:

• Without an API key: falls back to column-aware + dateline Python rules. Works fine for tidy pages; struggles on banner headlines and cross-column layouts.
• With an API key: Claude looks at a thumbnail of the page plus the regions JSON and does the spatial reasoning. Much better on complex layouts. Costs ~$0.01–0.05 per page.

To enable, set the environment variable before running:

export ANTHROPIC_API_KEY="your-api-key-here"

Get a key at https://console.anthropic.com/. If the variable isn't set or the API call fails, the app silently falls back to the Python rules.

4. Run

python app.py

Then open http://localhost:5000.

How to use

1. On the homepage, choose a PDF and pick a target render size. Defaults: 4200 × 7400 px (≈ 600 DPI for a tabloid newspaper page). Each page is scaled to fit inside this box preserving aspect.

2. Click "Start extraction". A background job kicks off and the page redirects to a job view that polls for status.

3. When the job finishes, every article is shown as a card with:

   • The cropped image (article.png)
   • A link to its OCR'd text (article.txt)
   • The detected dateline (e.g. చిన్నగూడూరు, మే 15 (ఆంధ్రజ్యోతి):) if the grouper found one
   • The first non-empty line of OCR text as a headline preview

4. Click "Download all (zip)" to grab everything in one file.

What's in each job folder

output/<timestamp>/
├── input.pdf                  the uploaded PDF
├── meta.json                  job metadata
├── pages/
│   ├── page_001.png           rendered page
│   ├── page_001.regions.json  raw PaddleOCR layout output
│   └── page_001.articles.json grouping result
└── articles/
    └── p001_a001/
        ├── article.png        per-article crop
        ├── article.txt        per-article OCR
        └── info.json          bbox + dateline + member regions

Everything is on disk and inspectable. If a crop looks wrong, you can look at pages/page_NNN.articles.json to see exactly which regions were grouped together.

Tuning the grouping

Stage 3 has two implementations in extractor.py:

Path A: Claude API (_group_with_claude)

The system prompt is in CLAUDE_GROUPING_PROMPT. It tells Claude:

• Use title regions as article starts
• Use the dateline field as a strong "this is an article body start" signal
• Respect column boundaries
• Handle banner headlines that span columns
• Pull quotes inside another article's bbox belong to that article

To tune, edit the prompt. To debug, look at pages/page_NNN.articles.json and check the grouped_by field — "claude" means the API call worked, "rules" means it fell back.

Path B: Python rules (_group_with_rules)

Used when no API key is set or the API call fails.

1. Column partition — regions clustered by left coordinate.
2. Within each column: title region OR dateline match starts a new article; everything else attaches to the current one.

To tune the rules:

• Adjust DATELINE_RE for different paper conventions.
• Adjust the column-clustering threshold in _assign_columns.

Known limitations

• L-shaped articles: an article whose body wraps around a photo into a non-rectangular shape can't be captured by one bbox. The current pipeline produces the smallest rect that covers everything, which may include a sliver of a neighboring article. You'd need a polygon-based crop to fix that — out of scope here.

• Cross-page continuations (మిగతా 3వ పేజీలో): not stitched. Each page is processed independently.

• PaddleOCR layout model is trained mostly on English/Chinese documents. Expect 80–95% of articles to be bounded correctly out of the box. Edge cases are why the per-job folder keeps the raw regions.json — so you can audit and tune.

License & data

All processing is local. The app never sends your PDF or anything extracted from it to a remote service. The PaddleOCR model weights are downloaded once from PaddlePaddle's CDN on first run.