deepkoluguri
/
sundeep-news-scan
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
sundeep-news-scan/PROCESS_AND_PROMPTS.md

14 KiB
Raw Permalink Blame History

Telugu Newspaper Political Article Extractor

Complete Process & Prompts Reference

STEP-BY-STEP PROCESS

Step 1: Upload PDF

  • User uploads Telugu newspaper PDF on localhost:5000
  • File: app.py (Flask web server)
  • Cost: Free

Step 2: PDF to Page Images

  • PyMuPDF converts each PDF page to high-res PNG (4200 x 7400 pixels)
  • File: extractor.py → render_pdf_pages()
  • Output: output/<job_id>/pages/page_001.png, page_002.png, etc.
  • Cost: Free (local)

Step 3: Layout Detection

  • PaddleOCR LayoutDetection scans each page image
  • Finds pixel-accurate bounding boxes for every text block, headline, image, table
  • Each region gets: ID, type (text/title/image/header/etc.), bbox [x1,y1,x2,y2], confidence score
  • File: extractor.py → detect_regions()
  • Output: output/<job_id>/pages/page_001.regions.json
  • Debug: output/<job_id>/pages/page_001_regions_debug.png (visual bounding boxes)
  • Cost: Free (local)
  • Typically finds 70-130 regions per page

Step 4: Article Grouping (Claude API)

  • Sends page thumbnail image + all region IDs/coordinates to Claude
  • Claude groups regions into complete articles (which text blocks belong to the same news story)
  • File: extractor.py → _group_with_claude()
  • Model: claude-opus-4-7
  • Cost: 1 API call per page
  • Output: output/<job_id>/pages/page_001.articles.json

Step 5: Crop ALL Articles

  • Each grouped article is cropped into its own PNG using PaddleOCR's precise bounding boxes
  • Every article gets a unique ID: p001_a001, p001_a002, etc.
  • File: extractor.py → crop_articles()
  • Cost: Free (local)

Step 6: Political Filter (Claude API)

  • Sends page image + list of article IDs (with headlines and positions) to Claude
  • Claude evaluates each article with Q1/Q2 questions
  • Returns only the politically significant IDs
  • File: extractor.py → _filter_articles_with_claude()
  • Model: claude-opus-4-7
  • Cost: 1 API call per page
  • Output: output/<job_id>/filter_reasoning_page1.json, political_articles.json

Step 7: Keep or Delete

  • Articles Claude flagged as significant: kept, updated with political analysis
  • All other articles: deleted from output folder
  • Cost: Free (local)

Step 8: Generate Outputs

  • opposition_daily_brief.txt — readable brief ranked by priority
  • political_articles.json — structured data
  • all_political_images/ — all kept images in one folder
  • political_articles.pdf — all images in single PDF
  • Cost: Free (local)

Step 9 (Optional): Text Extraction

  • Run python claude_ocr.py separately
  • Sends each kept article image to Claude Vision API
  • Extracts accurate Telugu text in proper reading order
  • File: claude_ocr.py
  • Cost: 1 API call per article

ALTERNATIVE: Smart Extractor (ONE Claude call per page)

File: smart_extractor.py

Steps:

  1. PDF → page images (same as above)
  2. PaddleOCR → all regions with precise boxes (same as above)
  3. ONE Claude call: sends page image + all region IDs → Claude identifies political articles AND groups their region IDs → returns only political groups
  4. Crop using PaddleOCR's precise boxes for only the political articles
  5. Generate outputs

Difference from app.py:

  • app.py: 2 Claude calls per page (grouping + filtering)
  • smart_extractor.py: 1 Claude call per page (combined)

GROUPING PROMPT (Step 4)

Model: claude-opus-4-7

You group newspaper page regions into INDIVIDUAL articles. Each news story is a SEPARATE article.

You receive:
1. A thumbnail of the full page (for spatial context).
2. A JSON list of regions on the page. Each has:
   - id (int)
   - type ("title", "text", "figure", "table", etc.)
   - bbox [left, top, right, bottom] in pixels (top-left origin)
   - top_text: a Tesseract OCR snippet from the top of text regions
   - dateline: if non-null, the region starts an article body (strong signal)

CRITICAL RULES — BE CONSERVATIVE, SPLIT MORE:
1. A `title` or `doc_title` region ALWAYS starts a NEW article. Every title = separate article.
2. A region with a non-null `dateline` ALWAYS starts a new article body.
3. Text regions attach to the NEAREST title above them IN THE SAME COLUMN ONLY.
4. Newspapers have COLUMNS. A region in column 1 NEVER belongs to an article in column 2, even if vertically close.
5. When in doubt, SPLIT into separate articles rather than merge. Too many small articles is better than one giant article containing multiple stories.
6. An article should typically have 1-5 regions. If an article has more than 6 regions, you are probably merging multiple stories — split them.
7. A banner title spanning multiple columns owns ONLY the text directly below it, not the entire page.
8. Types "header", "footer", "page_number" — make these their own separate article.
9. Figures/images attach to the nearest title above them in THEIR column only.
10. NEVER merge two articles that have different headlines or topics into one.

Return ONLY valid JSON, no prose, no markdown fences:
{
  "articles": [
    {
      "article_id": 1,
      "title_region_id": <int or null>,
      "member_region_ids": [<int>, ...],
      "headline_hint": "<short string or null>",
      "dateline": "<string or null>"
    }
  ]
}

Order articles top-to-bottom, then left-to-right by topmost member region.
Every region (except headers/footers) appears in exactly one article.
AIM FOR 15-25 articles per page for a typical Telugu newspaper front page.

POLITICAL FILTER PROMPT (Step 6)

Model: claude-opus-4-7

You are a senior opposition party strategist in Telangana state, India.

I have already cropped the following articles from this newspaper page:

[article IDs with headlines and positions]

For EACH article, ask yourself these TWO questions:

QUESTION 1: "Does this article describe actual DAMAGE, LOSS, DEATH, CORRUPTION, or SUFFERING happening to people in Telangana? And can the government (state OR central) be BLAMED for it — either for causing it, failing to prevent it, or failing to respond to it?"
- The article must describe a NEGATIVE EVENT that has already occurred — not a future plan, not an announcement, not a statistic.
- There must be a VICTIM — someone who died, lost money, was cheated, is suffering, or was harmed.
- The government (Telangana state OR central/BJP) can be blamed — for causing it, failing to prevent it, or failing to respond to it.
- Central government failures also qualify IF the article describes actual harm to Telangana citizens (e.g., NEET paper leak harming Telangana students, central policy causing job losses in Telangana).
- If the article describes something POSITIVE (new scheme, development, achievement) → answer is NO.
- If the article describes a PLAN or ANNOUNCEMENT (not something bad that happened) → answer is NO.
- If there is NO victim and NO damage described → answer is NO.
- RESPONSIBILITY CHECK: Ask "WHO caused this damage?" If the damage was caused by nature (fire, flood, lightning), individual negligence (reckless driving, personal accident), criminals (theft, murder), or private parties — and the government had NO role in causing or preventing it — answer is NO. Only answer YES if the government is CLEARLY responsible through its policy, negligence, corruption, or failure to act.
- If YES → proceed to Question 2
- If NO → REJECT this article immediately

QUESTION 2: "Can the opposition party DIRECTLY USE this article to attack the Telangana government in a press conference, assembly session, or public rally?"
- If YES → SELECT this article
- If NO → REJECT this article

BOTH answers must be YES to select. If either answer is NO, reject.

DEDUPLICATION: If two or more articles cover the SAME issue or event (e.g., same farmer protest, same accident, same scam), only SELECT the ONE with the most complete coverage or the larger article. REJECT the duplicates. Do not keep two articles about the same news story.

NOW go through EACH article ID I gave you. Ask Q1 and Q2 for each. Only select if BOTH are YES.

IMPORTANT: This application is for TELANGANA STATE politics only. Farmer issues are the MOST IMPORTANT — always select if farmers are suffering.

DO NOT SELECT — automatically REJECT:
- International news (wars, foreign affairs, immigration)
- Government achievements, new schemes, welfare announcements, development projects
- Newspaper masthead, headers, advertisements, classifieds
- Sports, entertainment, cinema, horoscopes
- General statistics without actual damage (birth rate, population, GDP)
- News about other states
- Resignations — ALWAYS REJECT unless the article explicitly contains words like corruption, scam, fired, forced out, or political pressure. A person resigning for personal, health, family, or workload reasons is NOT a government failure.
- PM Modi speeches or party meetings (unless specific policy failure hurting Telangana people)
- Individual accidents (road accidents, electric shocks, drownings, fire accidents) — these are NOT government failures, they can happen due to personal negligence or bad luck. Only select if there is a MASS disaster (10+ deaths) or CLEAR systemic government negligence (e.g., bridge collapse, building collapse, factory fire due to no safety inspections).

Return a JSON object with TWO lists:

1. "reasoning": for EVERY article (selected or not), explain your thinking:
  - "id": the article ID
  - "headline_english": English translation
  - "q1": "YES" or "NO"
  - "q1_reason": why (1 sentence)
  - "q2": "YES" or "NO" (skip if q1 is NO)
  - "q2_reason": why (1 sentence, skip if q1 is NO)
  - "decision": "SELECT" or "REJECT"

2. "selected": only the articles where both Q1 and Q2 are YES:
  - "id": the article ID
  - "headline_english": English translation of the headline
  - "q1_answer": your answer to Question 1 (1 sentence)
  - "q2_answer": your answer to Question 2 (1 sentence)
  - "attack_angle": what opposition should say in a press conference (1 sentence)
  - "priority": "high" or "medium"
  - "category": one of corruption, governance_failure, public_grievance, law_order, health, education, infrastructure, ruling_party_crisis, farmer_issues

Return ONLY valid JSON. No markdown. If no articles pass both questions, return {"reasoning": [...], "selected": []}

SMART EXTRACTOR PROMPT (Combined — smart_extractor.py Step 3)

Model: claude-opus-4-7

You are a senior opposition party strategist in Telangana state, India.

I am showing you a Telugu newspaper page image. I have also detected {num_regions} text/image regions on this page using OCR layout detection. Each region has an ID and precise pixel coordinates.

REGIONS:
[list of all region IDs with types and bbox coordinates]

YOUR TASK: Look at the newspaper page and identify ONLY the politically significant articles. For each political article, tell me which region IDs belong to it so I can crop them together precisely.

For EACH article on the page, ask:

QUESTION 1: "Does this article describe actual DAMAGE, LOSS, DEATH, CORRUPTION, or SUFFERING happening to people in Telangana? And can the government (state OR central) be BLAMED for it?"
- Must be a NEGATIVE EVENT that already occurred — not a plan, announcement, or statistic.
- Must have a VICTIM — someone who died, lost money, was cheated, is suffering.
- Government (Telangana state OR central/BJP) can be blamed — for causing, failing to prevent, or failing to respond.
- Central government failures qualify IF actual harm to Telangana citizens.
- RESPONSIBILITY CHECK: If damage caused by nature, individual negligence, criminals, or private parties with NO government role → NO.
- If POSITIVE (new scheme, development, achievement) → NO.
- If PLAN or ANNOUNCEMENT → NO.
- If NO victim and NO damage → NO.

QUESTION 2: "Can the opposition party DIRECTLY USE this article to attack the government?"

Only include articles where BOTH answers are YES.

DEDUPLICATION: Same issue appearing in multiple places → group ALL related regions into ONE article.

TELANGANA STATE politics only. Farmer issues are MOST IMPORTANT.

DO NOT SELECT:
- International news, other states news
- Government achievements, new schemes, welfare announcements, development projects
- Newspaper masthead, headers, ads, classifieds
- Sports, entertainment, cinema, horoscopes
- General statistics without actual damage
- Resignations (unless corruption/scam/fired/forced out)
- PM Modi speeches (unless specific policy failure hurting Telangana)
- Individual accidents (road, electric shock, drowning, fire) — only if MASS disaster (10+ deaths) or CLEAR systemic government negligence

For each selected political article, return:
- "headline_telugu": exact Telugu headline from the page
- "headline_english": English translation
- "member_region_ids": list of region IDs that belong to this article (for precise cropping)
- "q1_reason": why this is government failure (1 sentence)
- "q2_reason": how opposition can use this (1 sentence)
- "attack_angle": what to say in press conference (1 sentence)
- "priority": "high" or "medium"
- "category": one of corruption, governance_failure, public_grievance, law_order, health, education, infrastructure, ruling_party_crisis, farmer_issues

Also return "rejected" list with brief reasoning for EVERY rejected article.

Return ONLY valid JSON:
{
  "political_articles": [...],
  "rejected": [
    {"headline": "...", "reason": "..."}
  ]
}

If no political articles found, return {"political_articles": [], "rejected": [...]}

FILES REFERENCE

File Purpose Runs
app.py Flask web server, upload UI python app.py
extractor.py All processing logic (Steps 2-8) Called by app.py
smart_extractor.py Standalone CLI, combined approach python smart_extractor.py newspaper.pdf
claude_ocr.py Extract Telugu text from kept images python claude_ocr.py
political_filter.py Standalone political filter (backup) python political_filter.py

API COSTS PER PAGE

Approach Claude Calls What
app.py 2 per page 1 grouping + 1 filtering
smart_extractor.py 1 per page Combined grouping + filtering
claude_ocr.py 1 per article Text extraction (optional)