institutional-trader/backend/HYBRID_SETUP.md

# Hybrid Architecture Setup Guide

This guide explains how to set up and use the hybrid Node.js + Python architecture.

## Architecture Overview

```
┌─────────────┐
│  React UI  │
└──────┬──────┘
       │
┌──────▼──────────┐
│  Node.js API    │  ← Handles WebSockets, routing, real-time features
│  (Express)      │
└──────┬──────────┘
       │ HTTP
┌──────▼──────────┐
│  Python Service │  ← Handles complex data processing
│  (FastAPI)      │
└──────┬──────────┘
       │
┌──────▼──────────┐
│  PostgreSQL     │
└─────────────────┘
```

## Benefits

1. **Node.js**: Fast, efficient for I/O-bound operations (WebSockets, API routing)
2. **Python**: Better for data processing, analytics, and complex calculations
3. **Maintainability**: Complex SQL logic moved to Python for easier debugging
4. **Flexibility**: Can gradually migrate more logic to Python

## Setup Instructions

### 1. Python Service Setup

```bash
# Navigate to Python service directory
cd backend/python_service

# Create virtual environment (recommended)
python -m venv venv

# Activate virtual environment
# On Windows:
venv\Scripts\activate
# On macOS/Linux:
source venv/bin/activate

# Install dependencies
pip install -r requirements.txt

# Configure environment
cp .env.example .env
# Edit .env with your database credentials
```

### 2. Start Python Service

```bash
# Development mode (with auto-reload)
cd backend/python_service
uvicorn main:app --reload --port 8000

# Or use the Python script
python main.py
```

The service will be available at `http://localhost:8000`

### 3. Node.js Backend Setup

```bash
# Install dependencies (including node-fetch)
cd backend
npm install

# Configure environment
# Add to your .env file:
PYTHON_SERVICE_URL=http://localhost:8000
USE_PYTHON_SERVICE=true  # Set to 'false' to disable Python service
```

### 4. Start Node.js Backend

```bash
cd backend
npm run dev
```

## Configuration

### Environment Variables

**Python Service (.env in `backend/python_service/`):**
```env
# Database Configuration
USE_LOCAL_DB=true
LOCAL_DB_HOST=localhost
LOCAL_DB_PORT=5432
LOCAL_DB_USER=postgres
LOCAL_DB_PASSWORD=postgres
LOCAL_DB_NAME=institutional_trader

# Or use DATABASE_URL for remote database
# DATABASE_URL=postgresql://user:password@host:port/database
```

**Node.js Backend (.env in `backend/`):**
```env
# Python Service Configuration
PYTHON_SERVICE_URL=http://localhost:8000
USE_PYTHON_SERVICE=true  # Enable/disable Python service

# Existing database configuration...
```

## How It Works

### Request Flow

1. **Client** makes request to Node.js API: `GET /api/options/flow`
2. **Node.js** checks if Python service is enabled
3. **If enabled**: Node.js calls Python service via HTTP
4. **Python service** processes data using pandas (replaces complex SQL)
5. **Python service** returns processed data
6. **Node.js** enriches data with additional features (badges, signals)
7. **Node.js** returns final response to client

### Fallback Mechanism

If Python service is unavailable or disabled:
- Node.js automatically falls back to the original SQL query
- No breaking changes to existing functionality
- Seamless transition

## API Endpoints

### Python Service

- `GET /health` - Health check
- `GET /api/options-flow` - Processed options flow data
- `GET /api/options-flow/stats` - Flow statistics

### Node.js API (unchanged)

- `GET /api/options/flow` - Options flow (now calls Python service)
- All other endpoints remain the same

## Testing

### Test Python Service

```bash
# Health check
curl http://localhost:8000/health

# Get options flow
curl "http://localhost:8000/api/options-flow?start_date=2024-01-01&end_date=2024-01-02"
```

### Test Node.js Integration

```bash
# Should use Python service if available
curl "http://localhost:3010/api/options/flow?startDate=2024-01-01&endDate=2024-01-02"
```

## Migration Status

✅ **Completed:**
- Python service structure
- Options flow processing (replaces `optionflowrockerscorer.sql`)
- Node.js integration with fallback
- Health checks and error handling

⏳ **In Progress:**
- Additional SQL script migrations
- Performance optimization
- Full test coverage

## Troubleshooting

### Python Service Not Starting

1. Check database credentials in `.env`
2. Verify PostgreSQL is running
3. Check port 8000 is not in use
4. Review Python service logs

### Node.js Can't Connect to Python Service

1. Verify Python service is running: `curl http://localhost:8000/health`
2. Check `PYTHON_SERVICE_URL` in Node.js `.env`
3. Check firewall/network settings
4. Node.js will automatically fallback to SQL if Python unavailable

### Performance Issues

1. Python service processes data in memory (pandas)
2. For large datasets, consider:
   - Adding pagination
   - Implementing caching
   - Optimizing pandas operations
   - Using database indexes

## Next Steps

1. **Migrate Additional SQL Scripts:**
   - `intradaysignalscorer.sql`
   - `premarketgaprangescreener.sql`
   - Other complex queries

2. **Add Caching:**
   - Cache processed results
   - Reduce database load

3. **Add Monitoring:**
   - Service health monitoring
   - Performance metrics
   - Error tracking

4. **Optimize:**
   - Batch processing
   - Parallel processing
   - Database query optimization

## Support

For issues or questions:
1. Check service logs
2. Verify environment configuration
3. Test individual services independently
4. Review this documentation