institutional-trader/backend/RAW_SQL_SETUP.md

# Raw SQL Query Setup Guide

## Overview

The backend now supports raw SQL queries (Script 1 style) for complex options flow analysis. This guide explains how to set it up.

## Prerequisites

1. **Supabase RPC Function**: You need to create the `exec_sql` function in your Supabase database
2. **Script 1 SQL**: Your complex SQL query needs to be parameterized and added to the codebase

## Step 1: Create RPC Function in Supabase

Run the SQL from `database/rpc_functions.sql` in your Supabase SQL Editor:

```sql
CREATE OR REPLACE FUNCTION exec_sql(
  query_text text,
  params jsonb DEFAULT '[]'::jsonb
)
RETURNS jsonb
LANGUAGE plpgsql
SECURITY DEFINER
SET search_path = public
AS $$
-- See database/rpc_functions.sql for full definition
$$;
```

## Step 2: Add Your Script 1 SQL Query

1. **Open** `backend/src/queries/optionsFlowQuery.js`
2. **Find** the `optionsFlowQuerySQL` constant (around line 20)
3. **Replace** the placeholder comment with your actual Script 1 SQL query
4. **Parameterize** all values using `$1`, `$2`, `$3`, etc.

### Example Parameterization

**Before (hardcoded):**
```sql
WHERE created_at >= '2024-01-01'
  AND created_at <= '2024-01-31'
  AND total_premium >= 1000000
LIMIT 100;
```

**After (parameterized):**
```sql
WHERE created_at >= $1::date
  AND created_at <= $2::date
  AND total_premium >= $3::numeric
LIMIT $4::integer;
```

## Step 3: Update Parameter Builder

Update `buildOptionsFlowParams()` in `optionsFlowQuery.js` to match your SQL parameter order:

```javascript
export function buildOptionsFlowParams(params = {}) {
  return [
    params.startDate,           // $1
    params.endDate,             // $2
    params.minNotional,         // $3
    params.limit                // $4
    // Add more as needed
  ];
}
```

## Step 4: Enable Raw SQL in API Calls

### In Routes

The route automatically supports raw SQL via query parameter:

```javascript
// Use raw SQL
GET /api/options-flow?useRawSQL=true&startDate=2024-01-01&endDate=2024-01-31

// Use query builder (default)
GET /api/options-flow?startDate=2024-01-01
```

### In Code

```javascript
import { OptionsFlowQuery } from './queries/optionsFlowQuery.js';

// Use raw SQL
const flows = await OptionsFlowQuery.getRecentFlow(filters, true);

// Use query builder (default)
const flows = await OptionsFlowQuery.getRecentFlow(filters, false);
```

## Parameter Types

Use these PostgreSQL type casts in your SQL:

- **Dates**: `$1::date`, `$2::timestamp`
- **Numbers**: `$3::numeric`, `$4::integer`, `$5::float`
- **Text**: `$6::text`, `$7::varchar(10)`
- **Booleans**: `$8::boolean`

## Fallback Behavior

If raw SQL fails or isn't configured:
- The system automatically falls back to the query builder
- No errors are thrown (warning logged to console)
- API continues to work normally

This ensures backward compatibility and graceful degradation.

## Testing

### Test in Supabase SQL Editor First

```sql
SELECT * FROM exec_sql(
  'SELECT * FROM options_flow WHERE created_at >= $1::date LIMIT $2::integer',
  '["2024-01-01", 10]'::jsonb
);
```

### Test via API

```bash
# With raw SQL
curl "http://localhost:3001/api/options-flow?useRawSQL=true&startDate=2024-01-01&limit=10"

# Without (query builder)
curl "http://localhost:3001/api/options-flow?startDate=2024-01-01&limit=10"
```

## Environment Variables

Ensure your `.env` has:

```env
SUPABASE_URL=your_url
SUPABASE_KEY=your_service_key
SUPABASE_ANON_KEY=your_anon_key  # Optional but recommended
```

## Troubleshooting

### Error: "exec_sql RPC function not found"

- Run `database/rpc_functions.sql` in Supabase SQL Editor
- Verify the function exists: `SELECT proname FROM pg_proc WHERE proname = 'exec_sql';`

### Error: "SQL query failed"

- Check your SQL syntax in Supabase SQL Editor first
- Verify parameter types match (e.g., `$1::date` not `$1::text`)
- Check parameter count matches your `buildOptionsFlowParams()` function

### Query returns empty results

- Test the SQL directly in Supabase SQL Editor with sample parameters
- Verify date formats are correct (YYYY-MM-DD)
- Check table names and column names match your schema

## Security Notes

⚠️ **Important**: The `exec_sql` function uses `SECURITY DEFINER` which runs with elevated privileges. 

- Only grant execute permission to trusted roles
- Always use parameterized queries (never string concatenation)
- Validate input parameters before passing to `rawQuery()`
- Consider creating specific stored procedures for each query type instead of generic `exec_sql`

## Best Practices

1. **Test First**: Always test SQL queries in Supabase SQL Editor before adding to code
2. **Parameterize Everything**: Never use string interpolation for SQL queries
3. **Type Safety**: Always use PostgreSQL type casts (`::date`, `::numeric`, etc.)
4. **Error Handling**: The system falls back gracefully, but log errors for debugging
5. **Documentation**: Document your SQL query parameters in code comments