institutional-trader/README/CHEDDARFLOW_SYNC_GUIDE.md

# CheddarFlow Live Option Flow Integration Guide

This guide explains how to use the CheddarFlow WebSocket service to receive live option flow data.

## Quick Start

1. **Get authentication tokens** from browser DevTools (see Authentication section below)
2. **Add to `.env` file:**
   ```env
   CHEDDARFLOW_SESSION=your_session_token
   CHEDDARFLOW_TOKEN=your_jwt_token
   ```
3. **Find the GraphQL subscription query** from WebSocket messages (see GraphQL Subscription Query section)
4. **Set the query** in `.env`:
   ```env
   CHEDDARFLOW_SUBSCRIPTION_QUERY="subscription { ... }"
   ```
5. **Start the server** - the service will automatically connect

## Overview

The CheddarFlow WebSocket service connects to `wss://app.cheddarflow.com/api/ws` using the GraphQL Transport WebSocket protocol (`graphql-transport-ws`) to receive real-time option flow data.

## Setup

### 1. Enable the Service

The service is enabled by default. To disable it, add to your `.env` file:

```env
ENABLE_CHEDDARFLOW=false
```

### 2. Database Storage

By default, received data is automatically saved to the `OptionsFlow_monthly` table. To disable database storage:

```env
CHEDDARFLOW_SAVE_TO_DB=false
```

### 3. Authentication (Required)

CheddarFlow requires authentication. You have two options:

#### Option 1: Cookies (Recommended - Easiest)

Get cookies from your browser after logging into CheddarFlow:

1. Open CheddarFlow dashboard in your browser
2. Open DevTools (F12) → Application tab → Cookies
3. Copy all cookies for `app.cheddarflow.com` or `dash.cheddarflow.com`
4. Format them as: `cookie1=value1; cookie2=value2; cookie3=value3`

Add to `.env`:
```env
CHEDDARFLOW_COOKIES=your_cookie_string_here
```

Example:
```env
CHEDDARFLOW_COOKIES=auth0_session=xxx; auth0_csrf=yyy; other_cookie=zzz
```

#### Option 2: Session + Token

If you prefer using session and token separately:

```env
# Session token
CHEDDARFLOW_SESSION=your_session_token_here

# JWT token
CHEDDARFLOW_TOKEN=your_jwt_token_here
```

**How to get your authentication:**

### Method 1: Cookies (Easiest - Recommended)

1. Open CheddarFlow dashboard in your browser (while logged in)
2. Open DevTools (F12) → Application tab → Cookies
3. Select `app.cheddarflow.com` or `dash.cheddarflow.com`
4. Copy all cookie name-value pairs
5. Format as: `name1=value1; name2=value2; name3=value3`
6. Add to `.env` as `CHEDDARFLOW_COOKIES`

**Important**: Cookies may include session tokens that expire. Refresh them if connection fails.

### Method 2: From WebSocket Messages

1. Open CheddarFlow dashboard in your browser
2. Open DevTools (F12) → Network tab
3. Filter by "WS" (WebSocket)
4. Connect to the WebSocket (refresh page or navigate to a page that uses live data)
5. Look for the first message with `type: "connection_init"`
6. Copy the `session` and `token` values from the `payload` object

Example payload structure:
```json
{
  "type": "connection_init",
  "payload": {
    "session": "C7vnUHLdb9TORyCGmdOtb",
    "token": "eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCIsImtpZCI6ImdLSEdIb1UxWTAxaE1JcmwzLUpSdiJ9..."
  }
}
```

### Method 2: From GraphQL API Response

1. Open CheddarFlow dashboard in your browser
2. Open DevTools (F12) → Network tab
3. Filter by "Fetch/XHR"
4. Look for a POST request to `https://app.cheddarflow.com/api`
5. Find the request with `operationName: "GetUserSessionAndMarketStatus"`
6. Copy the `token` from `data.session.token` in the response
7. For `session`, check cookies or use the WebSocket method above

### Method 3: Programmatic Fetch (Advanced)

You can use the helper function to fetch tokens programmatically, but this requires authentication cookies:

```javascript
import { fetchCheddarFlowSession } from './src/services/cheddarflowWebSocketService.js';

// Get cookies from browser DevTools → Application → Cookies
const cookies = 'your_cookie_string_here';

const sessionData = await fetchCheddarFlowSession({
  cookie: cookies
});

console.log('Token:', sessionData.token);
```

**Note**: These tokens may expire. If the connection fails, refresh your tokens from the browser DevTools.

### 4. Configuration Options

Add these optional settings to your `.env` file:

```env
# Enable/disable CheddarFlow service (default: true)
ENABLE_CHEDDARFLOW=true

# Save received data to database (default: true)
CHEDDARFLOW_SAVE_TO_DB=true

# WebSocket URL (default: wss://app.cheddarflow.com/api/ws)
CHEDDARFLOW_WS_URL=wss://app.cheddarflow.com/api/ws

# Origin header (default: https://dash.cheddarflow.com)
CHEDDARFLOW_ORIGIN=https://dash.cheddarflow.com

# Feed ID and Topic for options feed subscription (required)
CHEDDARFLOW_FEED_ID=1612407366
CHEDDARFLOW_FEED_TOPIC=1612407366
```

## How It Works

1. **Connection**: The service automatically connects to CheddarFlow's WebSocket on server startup
2. **GraphQL Protocol**: Uses the `graphql-transport-ws` protocol for GraphQL subscriptions
3. **Data Mapping**: Incoming data is automatically mapped to the `OptionsFlow_monthly` schema
4. **Database Storage**: Records are inserted into the database in real-time
5. **Auto-Reconnect**: Automatically reconnects if the connection is lost

## GraphQL Subscription Query

The service uses the `OnOptionsFeedEvents` subscription which provides real-time option flow data. The subscription is already configured and ready to use.

### Feed ID Configuration

The subscription requires a `feed ID` and `topic` (usually the same value). This appears to be a view/feed identifier in CheddarFlow.

**To find your feed ID:**

1. Open CheddarFlow dashboard in browser DevTools
2. Go to Network tab → Filter by "WS" (WebSocket)
3. Look for a `subscribe` message with `operationName: "OnOptionsFeedEvents"`
4. Copy the `id` and `topic` values from `payload.variables`

Example:
```json
{
  "type": "subscribe",
  "payload": {
    "variables": {
      "id": "1612407366",
      "topic": "1612407366"
    }
  }
}
```

**Add to `.env` file:**
```env
CHEDDARFLOW_FEED_ID=1612407366
CHEDDARFLOW_FEED_TOPIC=1612407366
```

If not set, the service defaults to `1612407366` (you may need to update this with your own feed ID).

## Data Format

CheddarFlow sends option flow data in a tabular format within the `lastProcessedItems` field:

```json
{
  "lastProcessedItems": "{\"ins\":[[\"headers\"],[[\"data\"]]]}"
}
```

The service automatically:
1. Parses the JSON string from `lastProcessedItems`
2. Extracts the data rows (after the header row)
3. Maps each row to the database schema
4. Inserts records into `OptionsFlow_monthly` table

### Data Fields

The subscription provides these fields (in order):
- `idx`, `id`, `contract_id`, `timestamp`, `symbol`, `expiry`, `strike`, `put_call`, `side`, `buy_sell`, `spot`, `size`, `price`, `premium`, `sweep_block_split`, `volume`, `open_int`, `conds`

These are automatically mapped to the `OptionsFlow_monthly` schema.

## Data Mapping

The service automatically maps CheddarFlow data fields to the `OptionsFlow_monthly` schema:

| Database Column | CheddarFlow Field Variations |
|----------------|------------------------------|
| `Symbol` | `symbol`, `Symbol`, `ticker`, `Ticker` |
| `CallPut` | `callPut`, `CallPut`, `optionType`, `OptionType` |
| `Strike` | `strike`, `Strike`, `strikePrice`, `StrikePrice` |
| `Premium` | `premium`, `Premium`, `totalPremium`, `TotalPremium` |
| `Volume` | `volume`, `Volume`, `vol`, `Vol`, `contracts` |
| `Spot` | `spot`, `Spot`, `underlyingPrice`, `stockPrice` |
| `Side` | `side`, `Side`, `tradeSide`, `direction` |
| `ExpirationDate` | `expirationDate`, `ExpirationDate`, `expiry`, `expiration` |
| ... and more | See `mapCheddarFlowToDatabase()` function |

## Monitoring

### Health Check

Check service status via the health endpoint:

```bash
curl http://localhost:3010/health
```

Response includes `cheddarFlow` status:

```json
{
  "cheddarFlow": {
    "running": true,
    "connected": true,
    "subscriptionId": "sub_1234567890",
    "reconnectAttempts": 0,
    "status": "connected"
  }
}
```

### Logs

The service logs connection status and data processing:

```
📡 CheddarFlow: Connecting to wss://app.cheddarflow.com/api/ws...
📡 CheddarFlow: WebSocket opened
📡 CheddarFlow: Connection acknowledged
📡 CheddarFlow: Subscription started
📡 CheddarFlow: Inserted 10 records
```

## Troubleshooting

### Service Not Connecting

**Symptoms**: Logs show connection attempts but no connection

**Solutions**:
1. Check your network connection
2. Verify the WebSocket URL is correct
3. Check if CheddarFlow requires authentication (may need to add auth headers)
4. Review firewall/proxy settings

### No Data Received

**Symptoms**: Connected but no data flowing

**Solutions**:
1. Verify the GraphQL subscription query is correct
2. Check if CheddarFlow requires authentication tokens
3. Inspect WebSocket messages to see what CheddarFlow is sending
4. Check if there's actual flow data available (market hours, etc.)

### Database Insert Errors

**Symptoms**: Connection works but records aren't being saved

**Solutions**:
1. Verify database connection is working
2. Check that `OptionsFlow_monthly` table exists
3. Review database logs for constraint violations
4. Ensure `CHEDDARFLOW_SAVE_TO_DB` is not set to `false`

### Authentication Required

CheddarFlow requires `session` and `token` in the `connection_init` payload. The service automatically includes these if set in environment variables:

```env
CHEDDARFLOW_SESSION=your_session_token
CHEDDARFLOW_TOKEN=your_jwt_token
```

If these are missing, you'll see a warning in the logs and the connection will likely fail.

**To get fresh tokens:**
1. Open CheddarFlow dashboard in browser
2. DevTools → Network → WS filter
3. Find `connection_init` message
4. Copy `session` and `token` from payload
5. Update your `.env` file

**Token expiration:** JWT tokens typically expire after some time. If connection fails, refresh your tokens from the browser.

## Manual Testing

You can test the service manually:

```javascript
import { startCheddarFlowService, stopCheddarFlowService, getCheddarFlowStatus } from './src/services/cheddarflowWebSocketService.js';

// Start service
const client = startCheddarFlowService({
  enableDatabaseInsert: true,
  onData: (mappedRecord, rawData) => {
    console.log('Received flow:', mappedRecord);
  },
  onError: (error) => {
    console.error('Error:', error);
  }
});

// Check status
setTimeout(() => {
  const status = getCheddarFlowStatus();
  console.log('Status:', status);
}, 5000);

// Stop service (when done)
// stopCheddarFlowService();
```

## Next Steps

1. **Determine GraphQL Query**: Find the actual subscription query from CheddarFlow
2. **Update Subscription**: Replace the placeholder query in `startSubscription()`
3. **Add Authentication**: If required, add authentication tokens
4. **Customize Mapping**: Adjust field mappings if CheddarFlow uses different field names
5. **Test**: Verify data is flowing and being saved correctly

## API Reference

### Functions

- `startCheddarFlowService(options)` - Start the WebSocket service
- `stopCheddarFlowService()` - Stop the service
- `getCheddarFlowStatus()` - Get current service status
- `mapCheddarFlowToDatabase(data)` - Map CheddarFlow data to database schema

### Options

```javascript
{
  url: 'wss://app.cheddarflow.com/api/ws',  // WebSocket URL
  origin: 'https://dash.cheddarflow.com',   // Origin header
  enableDatabaseInsert: true,                // Save to database
  autoReconnect: true,                       // Auto-reconnect on disconnect
  reconnectInterval: 5000,                   // Reconnect delay (ms)
  maxReconnectAttempts: 10,                  // Max reconnect attempts
  onData: (mappedRecord, rawData) => {},     // Data callback
  onError: (error) => {}                     // Error callback
}
```

## Support

If you encounter issues:

1. Check the console logs for detailed error messages
2. Verify the WebSocket connection in browser DevTools
3. Test the GraphQL subscription query manually
4. Review the data mapping function for field name mismatches