sundeep-news-scan/README.md

# Telugu Article Extractor

A local web app that takes a Telugu newspaper PDF and produces one PNG +
one TXT file per article.

## Pipeline

```
PDF
 │ Stage 1  PyMuPDF  →  page_001.png  (target: 4200×7400 px, fits aspect)
 ▼
Pages
 │ Stage 2  PaddleOCR PP-Structure  →  regions.json
 │            (title / text / figure rectangles with bboxes)
 ▼
Regions
 │ Stage 3  Article grouping  →  articles.json
 │            Primary: Claude API (spatial reasoning over regions,
 │                     sees page thumbnail + region JSON)
 │            Fallback: column-aware + dateline-anchored Python rules
 │                     • title region starts a new article
 │                     • "<place>, మే <day> (ఆంధ్రజ్యోతి):" dateline
 │                       also starts a new article
 │                     • regions partitioned by column first to avoid
 │                       cross-column contamination
 ▼
Articles (bboxes)
 │ Stage 4  PIL crop  →  article.png  per article
 ▼
Article PNGs
 │ Stage 5  Tesseract (lang=tel)  →  article.txt  per article
 ▼
Done. Browse in the UI, download all as zip.
```

## Setup

### 1. System dependencies

```bash
# Ubuntu / Debian
sudo apt update
sudo apt install -y tesseract-ocr tesseract-ocr-tel libgl1
```

`tesseract-ocr-tel` is the Telugu language data — required for the per-article
OCR step.

`libgl1` is needed by paddlepaddle on headless Linux.

### 2. Python environment

```bash
cd telugu_extractor
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
```

First run of paddlepaddle/paddleocr will download model weights (~400 MB)
into `~/.paddleocr/`. This happens once.

### 3. (Optional) Enable Claude grouping

Stage 3 (article grouping) works two ways:

- **Without an API key**: falls back to column-aware + dateline Python
  rules. Works fine for tidy pages; struggles on banner headlines and
  cross-column layouts.
- **With an API key**: Claude looks at a thumbnail of the page plus
  the regions JSON and does the spatial reasoning. Much better on
  complex layouts. Costs ~$0.01–0.05 per page.

To enable, set the environment variable before running:

```bash
export ANTHROPIC_API_KEY="your-api-key-here"
```

Get a key at https://console.anthropic.com/. If the variable isn't set
or the API call fails, the app silently falls back to the Python rules.

### 4. Run

```bash
python app.py
```

Then open <http://localhost:5000>.

## How to use

1. On the homepage, choose a PDF and pick a target render size.
   Defaults: 4200 × 7400 px (≈ 600 DPI for a tabloid newspaper page).
   Each page is scaled to fit *inside* this box preserving aspect.

2. Click "Start extraction". A background job kicks off and the page
   redirects to a job view that polls for status.

3. When the job finishes, every article is shown as a card with:
   - The cropped image (`article.png`)
   - A link to its OCR'd text (`article.txt`)
   - The detected dateline (e.g. `చిన్నగూడూరు, మే 15 (ఆంధ్రజ్యోతి):`)
     if the grouper found one
   - The first non-empty line of OCR text as a headline preview

4. Click "Download all (zip)" to grab everything in one file.

## What's in each job folder

```
output/<timestamp>/
├── input.pdf                  the uploaded PDF
├── meta.json                  job metadata
├── pages/
│   ├── page_001.png           rendered page
│   ├── page_001.regions.json  raw PaddleOCR layout output
│   └── page_001.articles.json grouping result
└── articles/
    └── p001_a001/
        ├── article.png        per-article crop
        ├── article.txt        per-article OCR
        └── info.json          bbox + dateline + member regions
```

Everything is on disk and inspectable. If a crop looks wrong, you can
look at `pages/page_NNN.articles.json` to see exactly which regions
were grouped together.

## Tuning the grouping

Stage 3 has two implementations in `extractor.py`:

### Path A: Claude API (`_group_with_claude`)

The system prompt is in `CLAUDE_GROUPING_PROMPT`. It tells Claude:
- Use `title` regions as article starts
- Use the `dateline` field as a strong "this is an article body start" signal
- Respect column boundaries
- Handle banner headlines that span columns
- Pull quotes inside another article's bbox belong to that article

To tune, edit the prompt. To debug, look at `pages/page_NNN.articles.json`
and check the `grouped_by` field — `"claude"` means the API call worked,
`"rules"` means it fell back.

### Path B: Python rules (`_group_with_rules`)

Used when no API key is set or the API call fails.

1. **Column partition** — regions clustered by `left` coordinate.
2. **Within each column**: `title` region OR dateline match starts a
   new article; everything else attaches to the current one.

To tune the rules:
- Adjust `DATELINE_RE` for different paper conventions.
- Adjust the column-clustering threshold in `_assign_columns`.

## Known limitations

- **L-shaped articles**: an article whose body wraps around a photo into
  a non-rectangular shape can't be captured by one bbox. The current
  pipeline produces the smallest rect that covers everything, which may
  include a sliver of a neighboring article. You'd need a polygon-based
  crop to fix that — out of scope here.

- **Cross-page continuations** (`మిగతా 3వ పేజీలో`): not stitched. Each
  page is processed independently.

- **PaddleOCR layout model is trained mostly on English/Chinese
  documents.** Expect 80–95% of articles to be bounded correctly out of
  the box. Edge cases are why the per-job folder keeps the raw
  `regions.json` — so you can audit and tune.

## License & data

All processing is local. The app never sends your PDF or anything
extracted from it to a remote service. The PaddleOCR model weights are
downloaded once from PaddlePaddle's CDN on first run.