sundeep-news-scan/ARCHITECTURE.md

News-Scan Architecture

This application is an automated end-to-end pipeline designed to extract, process, OCR, and semantically analyze Telugu newspaper PDFs for political insights. It leverages computer vision for layout detection, deep learning for Optical Character Recognition (OCR), and Large Language Models (LLMs) to identify and analyze politically significant articles from raw newspapers.

High-Level Pipeline (extractor.py)

The extraction pipeline is executed in a series of distinct stages. It is orchestrated primarily by extractor.py and can be triggered via the CLI (run_pipeline.py) or the Web UI (app.py).

Stage 1: PDF to Image Rendering

• Tool: PyMuPDF (fitz)
• Process: Converts uploaded newspaper PDFs into high-resolution PNG images.

Stage 2: Layout Detection

• Tool: Surya Layout Model (surya)
• Process: Detects bounding boxes for various elements on the page (e.g., Text, Title, Picture, Section-header, Caption).
• Resiliency: Surya has a known int32 overflow issue on Windows when processing very large images. The system intercepts the image, downscales it to a safe 2048px maximum bound before inference, and scales the resulting bounding boxes back up to match the original high-resolution image.

Stage 3: Article Grouping

• Tool: geometric_grouper.py
• Process: Consolidates raw, disconnected text blocks and images into cohesive "articles." It uses spatial heuristics (column-aware logic, dateline anchoring) to determine which headline belongs to which text columns.

Stage 4: Image Cropping

• Tool: Pillow (PIL)
• Process: Crops bounding boxes of headlines and full articles from the high-resolution source images. It adds dynamic padding to prevent overlap with neighboring articles.

Stage 5: Optical Character Recognition (OCR)

• Tool: PaddleOCR (configured for Telugu lang='te')
• Process: Performs text extraction on the cropped images.
• Resiliency: PaddleOCR is prone to deadlocks, memory exhaustion, or native crashes on extremely large article images. The system proactively checks pixel counts and gracefully downscales any image exceeding 1,000,000 pixels before passing it to PaddleOCR, preserving enough resolution for text extraction while ensuring pipeline stability.

Stage 6: Semantic Political Filtering

• Tool: Anthropic API (claude-sonnet-4-6)
• Process: Runs a two-phase analysis:
  1. Triage: Quickly evaluates extracted headlines to discard obvious non-political content (ads, sports, entertainment).
  2. Deep Analysis: Reads the full Telugu OCR text of the triaged articles to answer specific political strategy questions (e.g., assessing government blame, finding farmer issues). Deduplicates stories covering the same event.

Stage 7: PDF Report Generation

• Tool: FPDF (generate_pdf.py)
• Process: Compiles the selected high-priority political articles, their translated English headlines, AI-generated attack angles, and article crops into a comprehensive Insights PDF report.

---

Directory Structure

Core Application Files

• run_pipeline.py - Command-line interface to process PDFs automatically from the input/ folder to the processed/ folder.
• app.py - Flask web application providing a user interface for uploading PDFs and viewing extracted insights.
• extractor.py - The core pipeline orchestrator containing all the stages described above.
• geometric_grouper.py - Core grouping logic for clustering Surya regions.
• generate_pdf.py - PDF rendering logic for the final report.

Working Directories

• input/ - Directory for PDFs awaiting processing (used by the CLI).
• output/ - Directory where all generated images, JSON files, OCR text files, and final PDFs are saved.
• processed/ - Directory where original PDFs are moved after processing is complete.

Debug & Testing Scripts

This workspace contains a suite of internal testing, validation, and debugging tools (e.g. _lines.py, _partition_debug.py, _vwalls_test.py, _replay_*.py). These scripts are used for isolating layout bugs, tuning column gaps, and replaying specific newspaper extraction behaviors.

---

Recent Stability Improvements

To improve the robustness of the application on Windows systems, we implemented critical architectural guardrails in the image processing layers:

1. Surya Layout Int32 Overflow Fix: Added automatic downscaling and upscaling of bounding boxes in Stage 2 to prevent native crashes in the Surya module when processing extremely large broadsheet pages.
2. PaddleOCR Deadlock Prevention: Added an aggressive pixel threshold in extractor.py (lowered to 1,000,000 pixels). If a merged article crop exceeds this size, it is proactively resized using Image.LANCZOS. This prevents PaddleOCR from silently crashing or permanently hanging the entire Python pipeline when attempting to extract text from large, multi-column stories.