institutional-trader/README/HYBRID_SETUP.md

Hybrid Architecture Setup Guide

This guide explains how to set up and use the hybrid Node.js + Python architecture.

Architecture Overview

┌─────────────┐
│  React UI  │
└──────┬──────┘
       │
┌──────▼──────────┐
│  Node.js API    │  ← Handles WebSockets, routing, real-time features
│  (Express)      │
└──────┬──────────┘
       │ HTTP
┌──────▼──────────┐
│  Python Service │  ← Handles complex data processing
│  (FastAPI)      │
└──────┬──────────┘
       │
┌──────▼──────────┐
│  PostgreSQL     │
└─────────────────┘

Benefits

1. Node.js: Fast, efficient for I/O-bound operations (WebSockets, API routing)
2. Python: Better for data processing, analytics, and complex calculations
3. Maintainability: Complex SQL logic moved to Python for easier debugging
4. Flexibility: Can gradually migrate more logic to Python

Setup Instructions

1. Python Service Setup

# Navigate to Python service directory
cd backend/python_service

# Create virtual environment (recommended)
python -m venv venv

# Activate virtual environment
# On Windows:
venv\Scripts\activate
# On macOS/Linux:
source venv/bin/activate

# Install dependencies
pip install -r requirements.txt

# Configure environment
cp .env.example .env
# Edit .env with your database credentials

2. Start Python Service

# Development mode (with auto-reload)
cd backend/python_service
uvicorn main:app --reload --port 8010

# Or use the Python script
python main.py

The service will be available at http://localhost:8010

3. Node.js Backend Setup

# Install dependencies (including node-fetch)
cd backend
npm install

# Configure environment
# Add to your .env file:
PYTHON_SERVICE_URL=http://localhost:8010
USE_PYTHON_SERVICE=true  # Set to 'false' to disable Python service

4. Start Node.js Backend

cd backend
npm run dev

Configuration

Environment Variables

Python Service (.env in backend/python_service/):

# Database Configuration
USE_LOCAL_DB=true
LOCAL_DB_HOST=localhost
LOCAL_DB_PORT=5432
LOCAL_DB_USER=postgres
LOCAL_DB_PASSWORD=postgres
LOCAL_DB_NAME=institutional_trader

# Or use DATABASE_URL for remote database
# DATABASE_URL=postgresql://user:password@host:port/database

Node.js Backend (.env in backend/):

# Python Service Configuration
PYTHON_SERVICE_URL=http://localhost:8010
USE_PYTHON_SERVICE=true  # Enable/disable Python service

# Existing database configuration...

How It Works

Request Flow

1. Client makes request to Node.js API: GET /api/options/flow
2. Node.js checks if Python service is enabled
3. If enabled: Node.js calls Python service via HTTP
4. Python service processes data using pandas (replaces complex SQL)
5. Python service returns processed data
6. Node.js enriches data with additional features (badges, signals)
7. Node.js returns final response to client

Fallback Mechanism

If Python service is unavailable or disabled:

• Node.js automatically falls back to the original SQL query
• No breaking changes to existing functionality
• Seamless transition

API Endpoints

Python Service

• GET /health - Health check
• GET /api/options-flow - Processed options flow data
• GET /api/options-flow/stats - Flow statistics

Node.js API (unchanged)

• GET /api/options/flow - Options flow (now calls Python service)
• All other endpoints remain the same

Testing

Test Python Service

# Health check
curl http://localhost:8010/health

# Get options flow
curl "http://localhost:8010/api/options-flow?start_date=2024-01-01&end_date=2024-01-02"

Test Node.js Integration

# Should use Python service if available
curl "http://localhost:3010/api/options/flow?startDate=2024-01-01&endDate=2024-01-02"

Migration Status

✅ Completed:

• Python service structure
• Options flow processing (replaces optionflowrockerscorer.sql)
• Node.js integration with fallback
• Health checks and error handling

⏳ In Progress:

• Additional SQL script migrations
• Performance optimization
• Full test coverage

Troubleshooting

Python Service Not Starting

1. Check database credentials in .env
2. Verify PostgreSQL is running
3. Check port 8010 is not in use
4. Review Python service logs

Node.js Can't Connect to Python Service

1. Verify Python service is running: curl http://localhost:8010/health
2. Check PYTHON_SERVICE_URL in Node.js .env
3. Check firewall/network settings
4. Node.js will automatically fallback to SQL if Python unavailable

Performance Issues

1. Python service processes data in memory (pandas)
2. For large datasets, consider:
   • Adding pagination
   • Implementing caching
   • Optimizing pandas operations
   • Using database indexes

Next Steps

1. Migrate Additional SQL Scripts:

   • intradaysignalscorer.sql
   • premarketgaprangescreener.sql
   • Other complex queries

2. Add Caching:

   • Cache processed results
   • Reduce database load

3. Add Monitoring:

   • Service health monitoring
   • Performance metrics
   • Error tracking

4. Optimize:

   • Batch processing
   • Parallel processing
   • Database query optimization

Support

For issues or questions:

1. Check service logs
2. Verify environment configuration
3. Test individual services independently
4. Review this documentation