feat: add documentation, improve thread-safe job status management, and harden image processing pipeline against memory overflows

.gitignore

@@ -1,27 +1,40 @@
 # Virtual environment
 venv/
 .venv/
+.venv*/
 
 # Python cache
 __pycache__/
 *.py[cod]
 *.egg-info/
 
-# Pipeline outputs (large, regenerated each run)
+# Pipeline inputs and outputs (large, regenerated each run)
+input/
 output/
+processed/
 uploads/
+NewsPapers/
 
 # Input newspaper PDFs (large binary data)
 *.pdf
 
-# Scratch / diagnostic preview images
+# Scratch / diagnostic files and directories
 _*.png
 pg-*.jpg
 q-*.jpg
 gp-*.jpg
 gq-*.jpg
+_*.log
+_*.py
+_*.txt
+_tess_tmp/
+scratch_py/
+test_*.py
+fix_pdf.py
+app_py_utf8.txt
+original_app_py.txt
 
-# Generated HTML guides
+# Generated guides
 Extractor_Guide.html
 HOW_IT_WORKS.html
 

ARCHITECTURE.md

@@ -0,0 +1,67 @@
+# News-Scan Architecture
+
+This application is an automated end-to-end pipeline designed to extract, process, OCR, and semantically analyze Telugu newspaper PDFs for political insights. It leverages computer vision for layout detection, deep learning for Optical Character Recognition (OCR), and Large Language Models (LLMs) to identify and analyze politically significant articles from raw newspapers.
+
+## High-Level Pipeline (`extractor.py`)
+
+The extraction pipeline is executed in a series of distinct stages. It is orchestrated primarily by `extractor.py` and can be triggered via the CLI (`run_pipeline.py`) or the Web UI (`app.py`).
+
+### Stage 1: PDF to Image Rendering
+- **Tool**: PyMuPDF (`fitz`)
+- **Process**: Converts uploaded newspaper PDFs into high-resolution PNG images.
+
+### Stage 2: Layout Detection
+- **Tool**: Surya Layout Model (`surya`)
+- **Process**: Detects bounding boxes for various elements on the page (e.g., `Text`, `Title`, `Picture`, `Section-header`, `Caption`).
+- **Resiliency**: Surya has a known int32 overflow issue on Windows when processing very large images. The system intercepts the image, downscales it to a safe 2048px maximum bound before inference, and scales the resulting bounding boxes back up to match the original high-resolution image.
+
+### Stage 3: Article Grouping
+- **Tool**: `geometric_grouper.py`
+- **Process**: Consolidates raw, disconnected text blocks and images into cohesive "articles." It uses spatial heuristics (column-aware logic, dateline anchoring) to determine which headline belongs to which text columns.
+
+### Stage 4: Image Cropping
+- **Tool**: Pillow (`PIL`)
+- **Process**: Crops bounding boxes of headlines and full articles from the high-resolution source images. It adds dynamic padding to prevent overlap with neighboring articles.
+
+### Stage 5: Optical Character Recognition (OCR)
+- **Tool**: PaddleOCR (configured for Telugu `lang='te'`)
+- **Process**: Performs text extraction on the cropped images.
+- **Resiliency**: PaddleOCR is prone to deadlocks, memory exhaustion, or native crashes on extremely large article images. The system proactively checks pixel counts and gracefully downscales any image exceeding 1,000,000 pixels before passing it to PaddleOCR, preserving enough resolution for text extraction while ensuring pipeline stability.
+
+### Stage 6: Semantic Political Filtering
+- **Tool**: Anthropic API (`claude-sonnet-4-6`)
+- **Process**: Runs a two-phase analysis:
+  1. **Triage**: Quickly evaluates extracted headlines to discard obvious non-political content (ads, sports, entertainment).
+  2. **Deep Analysis**: Reads the full Telugu OCR text of the triaged articles to answer specific political strategy questions (e.g., assessing government blame, finding farmer issues). Deduplicates stories covering the same event.
+
+### Stage 7: PDF Report Generation
+- **Tool**: FPDF (`generate_pdf.py`)
+- **Process**: Compiles the selected high-priority political articles, their translated English headlines, AI-generated attack angles, and article crops into a comprehensive Insights PDF report.
+
+---
+
+## Directory Structure
+
+### Core Application Files
+* **`run_pipeline.py`** - Command-line interface to process PDFs automatically from the `input/` folder to the `processed/` folder.
+* **`app.py`** - Flask web application providing a user interface for uploading PDFs and viewing extracted insights.
+* **`extractor.py`** - The core pipeline orchestrator containing all the stages described above.
+* **`geometric_grouper.py`** - Core grouping logic for clustering Surya regions.
+* **`generate_pdf.py`** - PDF rendering logic for the final report.
+
+### Working Directories
+* **`input/`** - Directory for PDFs awaiting processing (used by the CLI).
+* **`output/`** - Directory where all generated images, JSON files, OCR text files, and final PDFs are saved.
+* **`processed/`** - Directory where original PDFs are moved after processing is complete.
+
+### Debug & Testing Scripts
+This workspace contains a suite of internal testing, validation, and debugging tools (e.g. `_lines.py`, `_partition_debug.py`, `_vwalls_test.py`, `_replay_*.py`). These scripts are used for isolating layout bugs, tuning column gaps, and replaying specific newspaper extraction behaviors.
+
+---
+
+## Recent Stability Improvements
+
+To improve the robustness of the application on Windows systems, we implemented critical architectural guardrails in the image processing layers:
+
+1. **Surya Layout Int32 Overflow Fix**: Added automatic downscaling and upscaling of bounding boxes in Stage 2 to prevent native crashes in the Surya module when processing extremely large broadsheet pages.
+2. **PaddleOCR Deadlock Prevention**: Added an aggressive pixel threshold in `extractor.py` (lowered to 1,000,000 pixels). If a merged article crop exceeds this size, it is proactively resized using `Image.LANCZOS`. This prevents PaddleOCR from silently crashing or permanently hanging the entire Python pipeline when attempting to extract text from large, multi-column stories.

app.py

@@ -41,7 +41,8 @@ if hasattr(sys.stderr, 'reconfigure'):
 
 from extractor import process_pdf
 
-
+JOBS = {}
+JOB_LOCK = threading.Lock()
 
 ROOT = Path(__file__).parent
 UPLOAD_DIR = ROOT / "uploads"
@@ -123,12 +124,8 @@ def upload():
 
 def _run_job(job_id, pdf_path, job_dir, target_w, target_h):
     try:
-        meta_path = Path(job_dir) / "meta.json"
+        with JOB_LOCK:
-        meta = json.loads(meta_path.read_text())
+            JOBS[job_id] = {"status": "running", "message": "Starting..."}
-        meta["status"] = "running"
-        meta["message"] = "Starting..."
-        meta_path.write_text(json.dumps(meta, indent=2))
-        
         result = process_pdf(pdf_path, job_dir, target_w, target_h, job_id)
         meta_path = Path(job_dir) / "meta.json"
         meta = json.loads(meta_path.read_text())
@@ -137,9 +134,9 @@ def _run_job(job_id, pdf_path, job_dir, target_w, target_h):
             "pages": result["pages"],
             "articles": result["articles"],
         })
-        meta["status"] = "done"
-        meta["message"] = "Finished."
         meta_path.write_text(json.dumps(meta, indent=2))
+        with JOB_LOCK:
+            JOBS[job_id] = {"status": "done", "message": "Finished."}
     except Exception as e:
         traceback.print_exc()
         meta_path = Path(job_dir) / "meta.json"
@@ -147,10 +144,11 @@ def _run_job(job_id, pdf_path, job_dir, target_w, target_h):
             meta = json.loads(meta_path.read_text())
             meta["status"] = "error"
             meta["error"] = str(e)
-            meta["message"] = str(e)
             meta_path.write_text(json.dumps(meta, indent=2))
         except Exception:
             pass
+        with JOB_LOCK:
+            JOBS[job_id] = {"status": "error", "message": str(e)}
 
 
 @app.route("/job/<job_id>")
@@ -184,17 +182,16 @@ def job_view(job_id):
 
 @app.route("/job/<job_id>/status")
 def job_status(job_id):
+    with JOB_LOCK:
+        status = JOBS.get(job_id, {"status": "unknown"})
     meta_path = OUTPUT_DIR / job_id / "meta.json"
-    status_data = {"status": "unknown"}
     if meta_path.exists():
         try:
             meta = json.loads(meta_path.read_text())
-            status_data["status"] = meta.get("status", "unknown")
+            status["meta"] = meta
-            status_data["message"] = meta.get("message", "")
-            status_data["meta"] = meta
         except Exception:
             pass
-    return jsonify(status_data)
+    return jsonify(status)
 
 
 @app.route("/job/<job_id>/page/<int:page_num>")

claude_ocr.py

@@ -18,23 +18,8 @@ OUTPUT_DIR = Path(__file__).parent / "output"
 
 def read_article_image(client, img_path, model="claude-sonnet-4-6"):
     """Send article image to Claude and get Telugu text back."""
-    from PIL import Image
+    with open(img_path, "rb") as f:
-    import io
+        img_data = base64.standard_b64encode(f.read()).decode("utf-8")
-    
-    with Image.open(img_path) as img:
-        # Resize if too large to fit in 10MB limit (usually > 4000x4000)
-        max_dim = 2800
-        if max(img.width, img.height) > max_dim:
-            ratio = max_dim / max(img.width, img.height)
-            new_w = int(img.width * ratio)
-            new_h = int(img.height * ratio)
-            print(f"  [Claude OCR] Resizing {img.width}x{img.height} to {new_w}x{new_h} to fit API limits")
-            img = img.resize((new_w, new_h), Image.LANCZOS)
-        
-        # Save to bytes
-        buffer = io.BytesIO()
-        img.save(buffer, format="PNG", optimize=True)
-        img_data = base64.standard_b64encode(buffer.getvalue()).decode("utf-8")
 
     response = client.messages.create(
         model=model,

extractor.py

@@ -40,21 +40,17 @@ if not os.environ.get("ANTHROPIC_API_KEY"):
 import fitz  # PyMuPDF
 from PIL import Image
 
-
+JOBS = {}
+JOB_LOCK = threading.Lock()
 
 _LAYOUT = None
 _LAYOUT_LOCK = threading.Lock()
 
 
-def _set_status(job_dir, msg):
+def _set_status(job_id, msg):
-    try:
+    with JOB_LOCK:
-        meta_path = Path(job_dir) / "meta.json"
+        if job_id in JOBS:
-        if meta_path.exists():
+            JOBS[job_id]["message"] = msg
-            meta = json.loads(meta_path.read_text(encoding="utf-8"))
-            meta["message"] = msg
-            meta_path.write_text(json.dumps(meta, indent=2, ensure_ascii=False), encoding="utf-8")
-    except Exception:
-        pass
 
 
 # ---------------------------------------------------------------------------
@@ -114,19 +110,7 @@ def detect_regions(page_png_path):
     from surya.layout import batch_layout_detection
     img = Image.open(page_png_path).convert("RGB")
 
-    orig_w, orig_h = img.size
+    layout_predictions = batch_layout_detection([img], model, processor)
-    scale_factor = 1.0
-    # Surya has a numpy int32 overflow bug on Windows for very large images.
-    # Downscale for layout detection (Surya internally downscales anyway).
-    if max(orig_w, orig_h) > 2048:
-        scale_factor = 2048.0 / max(orig_w, orig_h)
-        new_w = int(orig_w * scale_factor)
-        new_h = int(orig_h * scale_factor)
-        img_for_surya = img.resize((new_w, new_h), Image.LANCZOS)
-    else:
-        img_for_surya = img
-
-    layout_predictions = batch_layout_detection([img_for_surya], model, processor)
 
     regions = []
 
@@ -147,8 +131,7 @@ def detect_regions(page_png_path):
         layout = layout_predictions[0]
         if hasattr(layout, 'bboxes'):
             for rid, box in enumerate(layout.bboxes, start=1):
-                # Scale coordinates back up to original image dimensions
+                bbox = [int(v) for v in box.bbox]
-                bbox = [int(v / scale_factor) for v in box.bbox]
                 raw_label = box.label
                 label = label_map.get(raw_label, "text").lower()
                 # Surya doesn't expose confidence in this API level easily, default to 1.0
@@ -334,22 +317,103 @@ def crop_full_articles(page_png_path, selected_ids, articles, out_dir, page_num)
 # ---------------------------------------------------------------------------
 # Stage 5 — OCR each cropped article (Telugu)
 # ---------------------------------------------------------------------------
-def _ocr_with_claude(img_path):
+_paddle_ocr_instance = None
-    """Use Claude Vision API for Telugu OCR as a fallback since PaddleOCR is missing."""
+
-    import os
+def _ocr_with_paddleocr(img_path):
-    api_key = os.environ.get("ANTHROPIC_API_KEY")
+    """Use PaddleOCR for better reading-order-aware Telugu OCR."""
-    if not api_key:
+    global _paddle_ocr_instance
-        print("ANTHROPIC_API_KEY not set. Skipping OCR.")
+    temp_path = None
-        return None
-    
     try:
-        import anthropic
+        from PIL import Image
-        import claude_ocr
+        with Image.open(img_path) as im:
-        client = anthropic.Anthropic(api_key=api_key)
+            px = im.width * im.height
-        return claude_ocr.read_article_image(client, str(img_path))
+            if px > 1000000:
+                # Instead of skipping, resize the article image to prevent PaddleOCR deadlock or crash
+                ratio = (900000 / px) ** 0.5
+                new_w = int(im.width * ratio)
+                new_h = int(im.height * ratio)
+                print(f"Article image {img_path.name} is too large ({im.width}x{im.height} = {px}px). Resizing to {new_w}x{new_h} for safe PaddleOCR.")
+                resized_im = im.resize((new_w, new_h), Image.LANCZOS)
+                temp_path = img_path.parent / f"temp_resized_{img_path.name}"
+                resized_im.save(temp_path)
+        
+        import paddleocr
+        if _paddle_ocr_instance is None:
+            _paddle_ocr_instance = paddleocr.PaddleOCR(lang='te')
+        
+        path_to_ocr = str(temp_path) if temp_path else str(img_path)
+        result = _paddle_ocr_instance.predict(path_to_ocr)
+
+        lines = []
+        if isinstance(result, list) and len(result) > 0:
+            det_result = result[0]
+            # PaddleOCR v3.5 returns DetResult with 'rec_texts' or nested structure
+            rec_texts = None
+            if hasattr(det_result, 'get'):
+                rec_texts = det_result.get('rec_texts', None)
+            if rec_texts:
+                lines = list(rec_texts)
+            else:
+                # Try to extract from boxes/text pairs
+                boxes = det_result.get('dt_polys', []) if hasattr(det_result, 'get') else []
+                texts = det_result.get('rec_texts', []) if hasattr(det_result, 'get') else []
+                if texts:
+                    # Sort by y-coordinate (top to bottom), then x (left to right)
+                    # to get proper reading order
+                    pairs = []
+                    for i, txt in enumerate(texts):
+                        if i < len(boxes):
+                            box = boxes[i]
+                            if hasattr(box, 'tolist'):
+                                box = box.tolist()
+                            # Get top-left y coordinate for sorting
+                            if isinstance(box, (list, tuple)) and len(box) >= 4:
+                                y = min(box[j] for j in range(1, len(box), 2)) if len(box) >= 8 else box[1]
+                                x = min(box[j] for j in range(0, len(box), 2)) if len(box) >= 8 else box[0]
+                            else:
+                                y, x = 0, 0
+                            pairs.append((y, x, txt))
+                        else:
+                            pairs.append((0, 0, txt))
+
+                    # Sort: primarily by y (row), then x (column within row)
+                    # Group into rows based on y proximity
+                    if pairs:
+                        pairs.sort(key=lambda p: (p[0], p[1]))
+                        row_threshold = 20  # pixels
+                        rows = []
+                        current_row = [pairs[0]]
+                        for p in pairs[1:]:
+                            if abs(p[0] - current_row[0][0]) < row_threshold:
+                                current_row.append(p)
+                            else:
+                                current_row.sort(key=lambda p: p[1])  # sort by x within row
+                                rows.append(current_row)
+                                current_row = [p]
+                        current_row.sort(key=lambda p: p[1])
+                        rows.append(current_row)
+
+                        for row in rows:
+                            lines.append(" ".join(p[2] for p in row))
+                else:
+                    # Last resort: try str representation
+                    try:
+                        s = str(det_result)
+                        if len(s) > 10:
+                            return s
+                    except:
+                        pass
+
+        return "\n".join(lines) if lines else None
     except Exception as e:
-        print(f"Claude text extraction failed: {e}")
+        print(f"PaddleOCR text extraction failed: {e}")
         return None
+    finally:
+        if temp_path and temp_path.exists():
+            try:
+                temp_path.unlink()
+            except:
+                pass
 
 
 def ocr_headlines(headline_records):
@@ -358,7 +422,7 @@ def ocr_headlines(headline_records):
         if not img_path.exists():
             rec["headline_text"] = ""
             continue
-        text = _ocr_with_claude(img_path)
+        text = _ocr_with_paddleocr(img_path)
         rec["headline_text"] = (text or "").strip()
         (img_path.parent / "headline.txt").write_text(rec["headline_text"], encoding="utf-8")
 
@@ -369,14 +433,12 @@ def ocr_articles(article_records):
         if not img_path.exists():
             continue
 
-        text = _ocr_with_claude(img_path)
+        text = _ocr_with_paddleocr(img_path)
         (art_dir / "article.txt").write_text(text or "", encoding="utf-8")
 
         info_path = art_dir / "info.json"
         if info_path.exists():
-            import json
             info = json.loads(info_path.read_text(encoding="utf-8"))
-            info["ocr_method"] = "claude_vision"
             info_path.write_text(json.dumps(info, indent=2, ensure_ascii=False), encoding="utf-8")
 
 
@@ -618,7 +680,7 @@ def process_pdf(pdf_path, job_dir, target_w, target_h, job_id, max_pages=None, p
     articles_root.mkdir(parents=True, exist_ok=True)
 
     # Step 1: Render PDF pages
-    _set_status(job_dir, "Rendering PDF pages...")
+    _set_status(job_id, "Rendering PDF pages...")
     pages = render_pdf_pages(pdf_path, pages_dir, target_w, target_h, max_pages=max_pages, pages_to_process=pages_to_process)
 
     all_records = []
@@ -629,7 +691,7 @@ def process_pdf(pdf_path, job_dir, target_w, target_h, job_id, max_pages=None, p
         page_img_path = p["path"]
 
         # Step 2: PaddleOCR layout detection — precise bounding boxes
-        _set_status(job_dir, f"Page {n}: detecting layout regions...")
+        _set_status(job_id, f"Page {n}: detecting layout regions...")
         regions = detect_regions(page_img_path)
         (pages_dir / f"page_{n:03d}.regions.json").write_text(
             json.dumps({"page": n, "regions": regions}, indent=2, ensure_ascii=False), encoding="utf-8"
@@ -637,7 +699,7 @@ def process_pdf(pdf_path, job_dir, target_w, target_h, job_id, max_pages=None, p
         print(f"  Page {n}: {len(regions)} layout regions detected")
 
         # Step 3: Group regions into articles
-        _set_status(job_dir, f"Page {n}: grouping articles...")
+        _set_status(job_id, f"Page {n}: grouping articles...")
         page_img = Image.open(page_img_path)
         articles, enriched = group_into_articles(page_img_path, regions, p["width"], page_img=page_img)
 
@@ -770,14 +832,14 @@ def process_pdf(pdf_path, job_dir, target_w, target_h, job_id, max_pages=None, p
         except Exception as e:
             print(f"  Could not create articles debug image: {e}")
         # Step 4: Crop and OCR HEADLINES only
-        _set_status(job_dir, f"Page {n}: cropping {len(articles)} headlines...")
+        _set_status(job_id, f"Page {n}: cropping {len(articles)} headlines...")
         headline_records = crop_headlines(page_img_path, articles, regions, articles_root, n)
         
-        _set_status(job_dir, f"Page {n}: running fast OCR on {len(headline_records)} headlines...")
+        _set_status(job_id, f"Page {n}: running fast OCR on {len(headline_records)} headlines...")
         ocr_headlines(headline_records)
 
         # Step 5: Triage with Claude Haiku based on headlines
-        _set_status(job_dir, f"Page {n}: triaging headlines with Claude...")
+        _set_status(job_id, f"Page {n}: triaging headlines with Claude...")
         selected_ids = _triage_headlines_with_claude(headline_records, n, job_dir=str(job_dir))
 
         # Cleanup non-selected directories
@@ -790,16 +852,16 @@ def process_pdf(pdf_path, job_dir, target_w, target_h, job_id, max_pages=None, p
             continue
 
         # Step 6: Crop FULL articles for the selected ones
-        _set_status(job_dir, f"Page {n}: cropping {len(selected_ids)} FULL articles...")
+        _set_status(job_id, f"Page {n}: cropping {len(selected_ids)} FULL articles...")
         records = crop_full_articles(page_img_path, selected_ids, articles, articles_root, n)
         print(f"  Page {n}: cropped {len(records)} full articles")
 
         # Step 7: OCR full articles
-        _set_status(job_dir, f"Page {n}: running heavy OCR on {len(records)} full articles...")
+        _set_status(job_id, f"Page {n}: running heavy OCR on {len(records)} full articles...")
         ocr_articles(records)
 
         # Step 8: Deep Analysis with Claude Opus
-        _set_status(job_dir, f"Page {n}: deep political analysis with Claude...")
+        _set_status(job_id, f"Page {n}: deep political analysis with Claude...")
         selected = _analyze_articles_with_claude(records, n, job_dir=str(job_dir))
 
         if selected:
@@ -921,5 +983,5 @@ def process_pdf(pdf_path, job_dir, target_w, target_h, job_id, max_pages=None, p
     if all_selected and generate_political_pdf:
         generate_political_pdf(all_selected, job_dir)
 
-    _set_status(job_dir, "Done!")
+    _set_status(job_id, "Done!")
     return {"pages": len(pages), "articles": len(all_records)}

requirements.txt

@@ -1,15 +1,8 @@
 Flask>=3.0
 PyMuPDF>=1.23
 Pillow>=10.0
-numpy>=1.24,<2.0.0
+numpy>=1.24
+paddleocr==3.5.0
+paddlepaddle==3.0.0
+pytesseract>=0.3.10
 anthropic>=0.40
-surya-ocr==0.4.15
-fpdf2>=2.7.0
-gunicorn>=21.0.0
-opencv-python-headless>=4.8.0
-
-# Missing Surya dependencies (useful if installed with --no-deps)
-ftfy>=6.0.0
-filetype>=1.2.0
-pypdfium2>=4.0.0
-tabulate>=0.9.0