institutional-trader/README/RAW_SQL_SETUP.md

Raw SQL Query Setup Guide

Overview

The backend now supports raw SQL queries (Script 1 style) for complex options flow analysis. This guide explains how to set it up.

Prerequisites

1. Supabase RPC Function: You need to create the exec_sql function in your Supabase database
2. Script 1 SQL: Your complex SQL query needs to be parameterized and added to the codebase

Step 1: Create RPC Function in Supabase

Run the SQL from database/rpc_functions.sql in your Supabase SQL Editor:

CREATE OR REPLACE FUNCTION exec_sql(
  query_text text,
  params jsonb DEFAULT '[]'::jsonb
)
RETURNS jsonb
LANGUAGE plpgsql
SECURITY DEFINER
SET search_path = public
AS $$
-- See database/rpc_functions.sql for full definition
$$;

Step 2: Add Your Script 1 SQL Query

1. Open backend/src/queries/optionsFlowQuery.js
2. Find the optionsFlowQuerySQL constant (around line 20)
3. Replace the placeholder comment with your actual Script 1 SQL query
4. Parameterize all values using $1, $2, $3, etc.

Example Parameterization

Before (hardcoded):

WHERE created_at >= '2024-01-01'
  AND created_at <= '2024-01-31'
  AND total_premium >= 1000000
LIMIT 100;

After (parameterized):

WHERE created_at >= $1::date
  AND created_at <= $2::date
  AND total_premium >= $3::numeric
LIMIT $4::integer;

Step 3: Update Parameter Builder

Update buildOptionsFlowParams() in optionsFlowQuery.js to match your SQL parameter order:

export function buildOptionsFlowParams(params = {}) {
  return [
    params.startDate,           // $1
    params.endDate,             // $2
    params.minNotional,         // $3
    params.limit                // $4
    // Add more as needed
  ];
}

Step 4: Enable Raw SQL in API Calls

In Routes

The route automatically supports raw SQL via query parameter:

// Use raw SQL
GET /api/options-flow?useRawSQL=true&startDate=2024-01-01&endDate=2024-01-31

// Use query builder (default)
GET /api/options-flow?startDate=2024-01-01

In Code

import { OptionsFlowQuery } from './queries/optionsFlowQuery.js';

// Use raw SQL
const flows = await OptionsFlowQuery.getRecentFlow(filters, true);

// Use query builder (default)
const flows = await OptionsFlowQuery.getRecentFlow(filters, false);

Parameter Types

Use these PostgreSQL type casts in your SQL:

• Dates: $1::date, $2::timestamp
• Numbers: $3::numeric, $4::integer, $5::float
• Text: $6::text, $7::varchar(10)
• Booleans: $8::boolean

Fallback Behavior

If raw SQL fails or isn't configured:

• The system automatically falls back to the query builder
• No errors are thrown (warning logged to console)
• API continues to work normally

This ensures backward compatibility and graceful degradation.

Testing

Test in Supabase SQL Editor First

SELECT * FROM exec_sql(
  'SELECT * FROM options_flow WHERE created_at >= $1::date LIMIT $2::integer',
  '["2024-01-01", 10]'::jsonb
);

Test via API

# With raw SQL
curl "http://localhost:3001/api/options-flow?useRawSQL=true&startDate=2024-01-01&limit=10"

# Without (query builder)
curl "http://localhost:3001/api/options-flow?startDate=2024-01-01&limit=10"

Environment Variables

Ensure your .env has:

SUPABASE_URL=your_url
SUPABASE_KEY=your_service_key
SUPABASE_ANON_KEY=your_anon_key  # Optional but recommended

Troubleshooting

Error: "exec_sql RPC function not found"

• Run database/rpc_functions.sql in Supabase SQL Editor
• Verify the function exists: SELECT proname FROM pg_proc WHERE proname = 'exec_sql';

Error: "SQL query failed"

• Check your SQL syntax in Supabase SQL Editor first
• Verify parameter types match (e.g., $1::date not $1::text)
• Check parameter count matches your buildOptionsFlowParams() function

Query returns empty results

• Test the SQL directly in Supabase SQL Editor with sample parameters
• Verify date formats are correct (YYYY-MM-DD)
• Check table names and column names match your schema

Security Notes

⚠️ Important: The exec_sql function uses SECURITY DEFINER which runs with elevated privileges.

• Only grant execute permission to trusted roles
• Always use parameterized queries (never string concatenation)
• Validate input parameters before passing to rawQuery()
• Consider creating specific stored procedures for each query type instead of generic exec_sql

Best Practices

1. Test First: Always test SQL queries in Supabase SQL Editor before adding to code
2. Parameterize Everything: Never use string interpolation for SQL queries
3. Type Safety: Always use PostgreSQL type casts (::date, ::numeric, etc.)
4. Error Handling: The system falls back gracefully, but log errors for debugging
5. Documentation: Document your SQL query parameters in code comments