Compare commits

...

1 Commits
main ... deploy

Author SHA1 Message Date
Deep Koluguri c8deeaf9e0 added tailscale 2025-12-12 20:55:24 -05:00
13 changed files with 403 additions and 73 deletions

View File

@ -36,10 +36,19 @@ sudo bash deploy.sh
```env ```env
NODE_ENV=production NODE_ENV=production
PORT=3000 PORT=3000
SUPABASE_URL=your_supabase_url
SUPABASE_ANON_KEY=your_anon_key # Remote PostgreSQL Database
SUPABASE_SERVICE_KEY=your_service_key USE_LOCAL_DB=true
DATABASE_URL=your_database_url LOCAL_DB_HOST=100.121.163.23
LOCAL_DB_PORT=5432
LOCAL_DB_USER=postgres
LOCAL_DB_PASSWORD=your_postgres_password
LOCAL_DB_NAME=institutional_trader
# OR use DATABASE_URL format:
# DATABASE_URL=postgresql://postgres:your_password@100.121.163.23:5432/institutional_trader
# CORS Configuration
CORS_ORIGIN=https://yourdomain.com CORS_ORIGIN=https://yourdomain.com
JWT_SECRET=your_secret_key_32_chars_min JWT_SECRET=your_secret_key_32_chars_min
``` ```
@ -101,9 +110,9 @@ sudo certbot --nginx -d yourdomain.com -d www.yourdomain.com -d api.yourdomain.c
**Option B: PostgreSQL User** **Option B: PostgreSQL User**
```bash ```bash
# Run the SQL script # Connect to remote database and run the SQL script
cd /opt/institutional_trader/backend/database cd /opt/institutional_trader/backend/database
sudo -u postgres psql institutional_trader < setup_friend_access.sql psql -h 100.121.163.23 -p 5432 -U postgres -d institutional_trader < setup_friend_access.sql
# Edit the script first to set username/password! # Edit the script first to set username/password!
``` ```
@ -114,6 +123,9 @@ sudo -u postgres psql institutional_trader < setup_friend_access.sql
sudo systemctl status institutional-trader-backend sudo systemctl status institutional-trader-backend
sudo systemctl status nginx sudo systemctl status nginx
# Test database connection
psql -h 100.121.163.23 -p 5432 -U postgres -d institutional_trader
# Test health endpoint # Test health endpoint
curl https://api.yourdomain.com/health curl https://api.yourdomain.com/health

View File

@ -38,10 +38,18 @@ Complete guide for deploying the Institutional Trader platform on a Proxmox serv
- Configure: - Configure:
- **Hostname**: `institutional-trader` - **Hostname**: `institutional-trader`
- **Password**: Set root password - **Password**: Set root password
- **CPU**: 2-4 cores - **CPU**: 2-4 cores ✅ (You have 4 CPUs - perfect!)
- **Memory**: 4-8 GB RAM - **Memory**: 4-8 GB RAM ✅ (You have 31.75 GiB - excellent!)
- **Disk**: 20-50 GB - **Disk**: 20-50 GB ✅ (You have 294.23 GiB - plenty of space!)
- **Network**: Bridge with static IP (recommended) or DHCP - **Network**: Bridge with static IP (recommended) or DHCP
- **Unprivileged**: Yes/No (see note below)
**Note on Unprivileged Containers:**
- ✅ **Your configuration will work** - Unprivileged containers are fine for this deployment
- Services run on ports > 1024 (Node.js: 3000, Python: 8000) - no issues
- Nginx can run as reverse proxy (may need to bind to ports > 1024 or use host network)
- PostgreSQL can be installed and run normally
- SSL certificates work fine with Certbot
2. **Start Container:** 2. **Start Container:**
```bash ```bash
@ -80,6 +88,43 @@ Complete guide for deploying the Institutional Trader platform on a Proxmox serv
## 3. System Dependencies ## 3. System Dependencies
### Important: Unprivileged Container Considerations
If your container is **unprivileged** (which yours is), there are a few adjustments:
1. **Nginx Port Binding**:
- Unprivileged containers cannot bind to ports < 1024
- **Solution**: Use Nginx on host or configure port forwarding
- Alternative: Run Nginx on port 8080/8443 and forward from host
2. **Systemd**:
- May need to enable systemd in container
- Check: `systemctl status` should work
3. **File Permissions**:
- Some operations may need different permissions
- Usually not an issue for this application
**Your Configuration Analysis:**
- ✅ **4 CPUs** - Perfect (recommended 2-4)
- ✅ **31.75 GiB RAM** - Excellent (recommended 4-8 GB)
- ✅ **294.23 GiB Disk** - Plenty of space (recommended 20-50 GB)
- ⚠️ **Unprivileged: Yes** - Will work, but see Nginx note below
### Enabling Systemd (if needed)
If systemd doesn't work in your container, you may need to enable it:
```bash
# Check if systemd is working
systemctl status
# If it fails, you may need to enable it in Proxmox
# Edit container config on Proxmox host:
pct set <container-id> -features nesting=1
pct set <container-id> -features keyctl=1
```
Run these commands on your Ubuntu container/VM: Run these commands on your Ubuntu container/VM:
```bash ```bash
@ -93,7 +138,7 @@ apt install -y nodejs
# Install Python 3.11+ and pip # Install Python 3.11+ and pip
apt install -y python3 python3-pip python3-venv apt install -y python3 python3-pip python3-venv
# Install PostgreSQL client (if using local DB) # Install PostgreSQL client (for remote database connection)
apt install -y postgresql-client apt install -y postgresql-client
# Install Nginx # Install Nginx
@ -160,7 +205,7 @@ SUPABASE_ANON_KEY=your_anon_key
SUPABASE_SERVICE_KEY=your_service_role_key SUPABASE_SERVICE_KEY=your_service_role_key
# OR Direct PostgreSQL Connection (if using local/remote PostgreSQL) # OR Direct PostgreSQL Connection (if using local/remote PostgreSQL)
DATABASE_URL=postgresql://postgres:password@localhost:5432/institutional_trader DATABASE_URL=postgresql://postgres:password@100.121.163.23:5432/institutional_trader
# CORS Configuration # CORS Configuration
CORS_ORIGIN=https://yourdomain.com CORS_ORIGIN=https://yourdomain.com
@ -236,42 +281,40 @@ This creates a `dist` folder with production-ready files.
- Use `DATABASE_URL` or `SUPABASE_URL` + keys in backend `.env` - Use `DATABASE_URL` or `SUPABASE_URL` + keys in backend `.env`
- See [Database Access for Friend](#9-database-access-for-friend) section for sharing access - See [Database Access for Friend](#9-database-access-for-friend) section for sharing access
### Option B: Local PostgreSQL (Self-Hosted) ### Option B: Remote PostgreSQL Database
```bash Your database is hosted at `100.121.163.23:5432`. Configure connection:
# Install PostgreSQL
apt install -y postgresql postgresql-contrib
# Start PostgreSQL **Update backend `.env`:**
systemctl start postgresql
systemctl enable postgresql
# Create database and user
sudo -u postgres psql
# In PostgreSQL prompt:
CREATE DATABASE institutional_trader;
CREATE USER trader_user WITH PASSWORD 'secure_password_here';
GRANT ALL PRIVILEGES ON DATABASE institutional_trader TO trader_user;
\q
# Import schema
cd /opt/institutional_trader/backend/database
sudo -u postgres psql institutional_trader < schema.sql
sudo -u postgres psql institutional_trader < missing_tables.sql
# Import other SQL files as needed
```
Update backend `.env`:
```env ```env
USE_LOCAL_DB=true USE_LOCAL_DB=true
LOCAL_DB_HOST=localhost LOCAL_DB_HOST=100.121.163.23
LOCAL_DB_PORT=5432 LOCAL_DB_PORT=5432
LOCAL_DB_USER=trader_user LOCAL_DB_USER=postgres
LOCAL_DB_PASSWORD=secure_password_here LOCAL_DB_PASSWORD=your_postgres_password
LOCAL_DB_NAME=institutional_trader LOCAL_DB_NAME=institutional_trader
``` ```
**Or use DATABASE_URL format:**
```env
DATABASE_URL=postgresql://postgres:your_password@100.121.163.23:5432/institutional_trader
```
**Test connection:**
```bash
# Test from container
psql -h 100.121.163.23 -p 5432 -U postgres -d institutional_trader
# Or test from Proxmox host
apt install -y postgresql-client
psql -h 100.121.163.23 -p 5432 -U postgres -d institutional_trader
```
**Note:** Make sure the remote PostgreSQL server allows connections from your Proxmox container IP. You may need to:
1. Configure `pg_hba.conf` on the database server to allow your container's IP
2. Configure firewall rules to allow port 5432 from your container
3. Ensure PostgreSQL is listening on the correct interface (not just localhost)
--- ---
## 6. Systemd Services ## 6. Systemd Services
@ -288,7 +331,7 @@ Add:
```ini ```ini
[Unit] [Unit]
Description=Institutional Trader Backend API Description=Institutional Trader Backend API
After=network.target postgresql.service After=network.target
[Service] [Service]
Type=simple Type=simple
@ -326,7 +369,7 @@ Add:
```ini ```ini
[Unit] [Unit]
Description=Institutional Trader Python Service Description=Institutional Trader Python Service
After=network.target postgresql.service After=network.target
[Service] [Service]
Type=simple Type=simple
@ -368,8 +411,30 @@ sudo journalctl -u institutional-trader-* -f
## 7. Nginx Configuration ## 7. Nginx Configuration
### ⚠️ Important: Unprivileged Container Nginx Setup
If your container is **unprivileged**, Nginx cannot bind to ports 80/443 directly. You have two options:
**Option A: Run Nginx on Proxmox Host (Recommended)**
- Install Nginx on the Proxmox host
- Configure it to proxy to your container's services
- Container services run on ports 3000 (backend) and 8000 (Python)
- See "Nginx on Host" section below
**Option B: Use Port Forwarding**
- Run Nginx in container on ports 8080/8443
- Forward ports from Proxmox host to container
- Configure firewall rules
**Option C: Enable Privileged Mode (Less Secure)**
- Convert container to privileged mode
- Allows binding to ports 80/443
- Less secure but simpler
### 7.1 Create Nginx Configuration ### 7.1 Create Nginx Configuration
**For Container (if using Option B):**
```bash ```bash
sudo nano /etc/nginx/sites-available/institutional-trader sudo nano /etc/nginx/sites-available/institutional-trader
``` ```
@ -492,6 +557,77 @@ sudo nginx -t
sudo systemctl reload nginx sudo systemctl reload nginx
``` ```
### 7.3 Nginx on Proxmox Host (For Unprivileged Containers)
If your container is unprivileged, install Nginx on the Proxmox host:
**On Proxmox Host:**
```bash
# Install Nginx
apt install -y nginx
# Create configuration
nano /etc/nginx/sites-available/institutional-trader
```
**Nginx Config (on Proxmox host, pointing to container):**
```nginx
# Backend API Server
server {
listen 443 ssl http2;
server_name api.yourdomain.com;
ssl_certificate /etc/letsencrypt/live/yourdomain.com/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/yourdomain.com/privkey.pem;
location / {
proxy_pass http://<container-ip>:3000;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection 'upgrade';
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
}
location /ws {
proxy_pass http://<container-ip>:3000;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "Upgrade";
proxy_read_timeout 86400;
}
}
# Frontend Server
server {
listen 443 ssl http2;
server_name yourdomain.com www.yourdomain.com;
ssl_certificate /etc/letsencrypt/live/yourdomain.com/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/yourdomain.com/privkey.pem;
root /var/www/institutional-trader/dist;
index index.html;
location / {
try_files $uri $uri/ /index.html;
}
}
```
**Copy frontend files to host:**
```bash
# On container, create archive
cd /opt/institutional_trader/frontend
tar czf /tmp/frontend-dist.tar.gz dist/
# On Proxmox host, copy from container
pct pull <container-id> /tmp/frontend-dist.tar.gz /tmp/
tar xzf /tmp/frontend-dist.tar.gz -C /var/www/institutional-trader/
```
--- ---
## 8. SSL Certificates ## 8. SSL Certificates
@ -601,13 +737,14 @@ ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT SELECT, USAGE ON SEQUENCES TO fr
**Connection String for Friend:** **Connection String for Friend:**
``` ```
postgresql://friend_user:secure_password_here@your-server-ip:5432/institutional_trader postgresql://friend_user:secure_password_here@100.121.163.23:5432/institutional_trader
``` ```
**Important:** If your friend needs to connect from outside your network: **Important:** Make sure the PostgreSQL server at `100.121.163.23`:
1. Configure PostgreSQL to accept remote connections 1. Allows remote connections from your friend's IP
2. Set up firewall rules 2. Has `pg_hba.conf` configured to allow connections
3. Consider using SSH tunnel for security 3. Has firewall rules allowing port 5432
4. Consider using SSH tunnel for additional security
### Option C: SSH Tunnel (Most Secure) ### Option C: SSH Tunnel (Most Secure)
@ -615,12 +752,14 @@ Create an SSH tunnel for your friend:
```bash ```bash
# On your friend's machine # On your friend's machine
ssh -L 5432:localhost:5432 user@your-server-ip ssh -L 5432:100.121.163.23:5432 user@your-proxmox-server-ip
# Then they can connect using: # Then they can connect using:
# postgresql://friend_user:password@localhost:5432/institutional_trader # postgresql://friend_user:password@localhost:5432/institutional_trader
``` ```
This creates a secure tunnel through your Proxmox server to the database.
### Option D: Read-Only Access (If Friend Only Needs to Read Data) ### Option D: Read-Only Access (If Friend Only Needs to Read Data)
```sql ```sql
@ -696,13 +835,13 @@ sudo systemctl reload nginx
- Use Supabase Dashboard > Database > Backups - Use Supabase Dashboard > Database > Backups
- Or use `pg_dump` with connection string - Or use `pg_dump` with connection string
**For Local PostgreSQL:** **For Remote PostgreSQL:**
```bash ```bash
# Create backup # Create backup
sudo -u postgres pg_dump institutional_trader > backup_$(date +%Y%m%d).sql pg_dump -h 100.121.163.23 -p 5432 -U postgres -d institutional_trader > backup_$(date +%Y%m%d).sql
# Restore backup # Restore backup
sudo -u postgres psql institutional_trader < backup_20240101.sql psql -h 100.121.163.23 -p 5432 -U postgres -d institutional_trader < backup_20240101.sql
``` ```
--- ---
@ -751,8 +890,8 @@ sudo systemctl reload nginx
### Database Connection Issues ### Database Connection Issues
```bash ```bash
# Test PostgreSQL connection (local) # Test PostgreSQL connection (remote)
psql -h localhost -U trader_user -d institutional_trader psql -h 100.121.163.23 -p 5432 -U postgres -d institutional_trader
# Test Supabase connection # Test Supabase connection
psql "postgresql://postgres:password@db.project.supabase.co:5432/postgres" psql "postgresql://postgres:password@db.project.supabase.co:5432/postgres"

View File

@ -3,6 +3,7 @@
-- --
-- Usage: -- Usage:
-- For Supabase: Run in Supabase SQL Editor -- For Supabase: Run in Supabase SQL Editor
-- For Remote PostgreSQL: psql -h 100.121.163.23 -p 5432 -U postgres -d institutional_trader < setup_friend_access.sql
-- For Local PostgreSQL: sudo -u postgres psql institutional_trader < setup_friend_access.sql -- For Local PostgreSQL: sudo -u postgres psql institutional_trader < setup_friend_access.sql
-- --
-- IMPORTANT: Replace 'friend_user' and 'secure_password_here' with actual values! -- IMPORTANT: Replace 'friend_user' and 'secure_password_here' with actual values!
@ -126,13 +127,13 @@ ORDER BY grantee, table_name;
-- Connection Strings for Friend -- Connection Strings for Friend
-- ============================================ -- ============================================
-- For Remote PostgreSQL (100.121.163.23):
-- postgresql://friend_user:secure_password_here@100.121.163.23:5432/institutional_trader
--
-- For Supabase: -- For Supabase:
-- postgresql://friend_user:secure_password_here@db.[project-ref].supabase.co:5432/postgres -- postgresql://friend_user:secure_password_here@db.[project-ref].supabase.co:5432/postgres
-- --
-- For Local PostgreSQL:
-- postgresql://friend_user:secure_password_here@your-server-ip:5432/institutional_trader
--
-- For SSH Tunnel (most secure): -- For SSH Tunnel (most secure):
-- 1. Friend runs: ssh -L 5432:localhost:5432 user@your-server-ip -- 1. Friend runs: ssh -L 5432:100.121.163.23:5432 user@your-proxmox-server-ip
-- 2. Friend connects to: postgresql://friend_user:password@localhost:5432/institutional_trader -- 2. Friend connects to: postgresql://friend_user:password@localhost:5432/institutional_trader

View File

@ -10,8 +10,20 @@ SUPABASE_ANON_KEY=your_production_anon_key
SUPABASE_SERVICE_KEY=your_service_role_key SUPABASE_SERVICE_KEY=your_service_role_key
# Direct PostgreSQL Connection (optional, for heavy queries) # Direct PostgreSQL Connection (optional, for heavy queries)
# Option 1: Remote PostgreSQL Database
DATABASE_URL=postgresql://postgres:[password]@100.121.163.23:5432/institutional_trader
# Option 2: Supabase Database
# Get this from: Supabase Dashboard > Settings > Database > Connection String (URI) # Get this from: Supabase Dashboard > Settings > Database > Connection String (URI)
DATABASE_URL=postgresql://postgres:[password]@db.[project-ref].supabase.co:5432/postgres # DATABASE_URL=postgresql://postgres:[password]@db.[project-ref].supabase.co:5432/postgres
# Option 3: Use LOCAL_DB variables (for remote database)
USE_LOCAL_DB=true
LOCAL_DB_HOST=100.121.163.23
LOCAL_DB_PORT=5432
LOCAL_DB_USER=postgres
LOCAL_DB_PASSWORD=your_postgres_password
LOCAL_DB_NAME=institutional_trader
# CORS Configuration # CORS Configuration
CORS_ORIGIN=https://your-frontend-domain.com CORS_ORIGIN=https://your-frontend-domain.com

View File

@ -23,7 +23,7 @@ async def get_pool() -> asyncpg.Pool:
if use_local_db: if use_local_db:
config = { config = {
'host': os.getenv('LOCAL_DB_HOST', 'localhost'), 'host': os.getenv('LOCAL_DB_HOST', '100.121.163.23'),
'port': int(os.getenv('LOCAL_DB_PORT', '5432')), 'port': int(os.getenv('LOCAL_DB_PORT', '5432')),
'user': os.getenv('LOCAL_DB_USER', 'postgres'), 'user': os.getenv('LOCAL_DB_USER', 'postgres'),
'password': os.getenv('LOCAL_DB_PASSWORD', 'postgres'), 'password': os.getenv('LOCAL_DB_PASSWORD', 'postgres'),

View File

@ -84,10 +84,20 @@ class OptionsFlowProcessor:
def parse_timestamp(self, date_str, time_str) -> Optional[datetime]: def parse_timestamp(self, date_str, time_str) -> Optional[datetime]:
"""Parse timestamp from date and time strings""" """Parse timestamp from date and time strings"""
if pd.isna(date_str) or pd.isna(time_str): if pd.isna(date_str):
return None return None
date_str = str(date_str).strip() date_str = str(date_str).strip()
# If time_str is None/NaN, parse just the date and use midnight as default
if pd.isna(time_str) or time_str is None:
# Try to parse just the date
date_obj = self.parse_date(date_str)
if date_obj:
# Use midnight (00:00:00) as default time
return datetime.combine(date_obj.date(), datetime.min.time())
return None
time_str = str(time_str).strip() time_str = str(time_str).strip()
# Try various formats # Try various formats

View File

@ -0,0 +1,64 @@
/**
* Check Database Connection Configuration
* Shows which database the application will connect to
*
* Usage: node scripts/checkDbConnection.js
*/
import dotenv from 'dotenv';
import { testConnection } from '../src/db.js';
dotenv.config();
console.log('🔍 Database Connection Configuration Check\n');
console.log('='.repeat(50));
console.log('\n📋 Environment Variables:');
console.log(' USE_LOCAL_DB:', process.env.USE_LOCAL_DB || '❌ not set');
console.log(' LOCAL_DB_HOST:', process.env.LOCAL_DB_HOST || '❌ not set (will default to: 100.121.163.23)');
console.log(' LOCAL_DB_PORT:', process.env.LOCAL_DB_PORT || '❌ not set (will default to: 5432)');
console.log(' LOCAL_DB_USER:', process.env.LOCAL_DB_USER || '❌ not set (will default to: postgres)');
console.log(' LOCAL_DB_NAME:', process.env.LOCAL_DB_NAME || '❌ not set (will default to: institutional_trader)');
console.log(' LOCAL_DB_PASSWORD:', process.env.LOCAL_DB_PASSWORD ? '✅ set' : '❌ not set');
console.log(' DATABASE_URL:', process.env.DATABASE_URL ? '✅ set (masked)' : '❌ not set');
console.log('\n🔌 Actual Connection Details:');
const actualHost = process.env.LOCAL_DB_HOST || '100.121.163.23';
const actualPort = process.env.LOCAL_DB_PORT || '5432';
const actualUser = process.env.LOCAL_DB_USER || 'postgres';
const actualDb = process.env.LOCAL_DB_NAME || 'institutional_trader';
console.log(` Host: ${actualHost}`);
console.log(` Port: ${actualPort}`);
console.log(` User: ${actualUser}`);
console.log(` Database: ${actualDb}`);
if (actualHost === 'localhost' || actualHost === '127.0.0.1') {
console.log('\n⚠ WARNING: You are connecting to localhost!');
console.log(' If you want to use the remote database at 100.121.163.23,');
console.log(' either:');
console.log(' 1. Remove LOCAL_DB_HOST from your .env file, OR');
console.log(' 2. Set LOCAL_DB_HOST=100.121.163.23 in your .env file');
} else if (actualHost === '100.121.163.23') {
console.log('\n✅ Correct: Connecting to remote database at 100.121.163.23');
} else {
console.log(`\n📌 Connecting to: ${actualHost}`);
}
console.log('\n🧪 Testing connection...');
testConnection()
.then(connected => {
if (connected) {
console.log('\n✅ Connection test successful!');
console.log(` Data will be saved to: ${actualHost}:${actualPort}/${actualDb}`);
} else {
console.log('\n❌ Connection test failed!');
console.log(' Please check your database credentials and network connectivity.');
}
process.exit(connected ? 0 : 1);
})
.catch(error => {
console.error('\n❌ Error testing connection:', error.message);
process.exit(1);
});

View File

@ -18,7 +18,7 @@ const __dirname = path.dirname(__filename);
// Default local PostgreSQL connection (can be overridden via env) // Default local PostgreSQL connection (can be overridden via env)
const localDbConfig = { const localDbConfig = {
host: process.env.LOCAL_DB_HOST || 'localhost', host: process.env.LOCAL_DB_HOST || '100.121.163.23',
port: process.env.LOCAL_DB_PORT || 5432, port: process.env.LOCAL_DB_PORT || 5432,
user: process.env.LOCAL_DB_USER || 'postgres', user: process.env.LOCAL_DB_USER || 'postgres',
password: process.env.LOCAL_DB_PASSWORD || 'postgres', password: process.env.LOCAL_DB_PASSWORD || 'postgres',

View File

@ -9,7 +9,7 @@
*/ */
import { fetchBlackBoxFlow, mapBlackBoxToDatabase } from '../src/services/blackboxApiService.js'; import { fetchBlackBoxFlow, mapBlackBoxToDatabase } from '../src/services/blackboxApiService.js';
import { rawQuery } from '../src/db.js'; import { rawQuery, testConnection } from '../src/db.js';
import dotenv from 'dotenv'; import dotenv from 'dotenv';
dotenv.config(); dotenv.config();
@ -153,6 +153,15 @@ async function syncBlackBoxFlow() {
console.log('🚀 Starting BlackBox Stocks flow data sync...\n'); console.log('🚀 Starting BlackBox Stocks flow data sync...\n');
try { try {
// Test database connection first
console.log('🔌 Testing database connection...');
const dbConnected = await testConnection();
if (!dbConnected) {
console.error('❌ Database connection failed. Please check your .env configuration.');
process.exit(1);
}
console.log('');
// Parse command line arguments // Parse command line arguments
const args = process.argv.slice(2); const args = process.argv.slice(2);
const options = {}; const options = {};
@ -218,8 +227,11 @@ async function syncBlackBoxFlow() {
// Insert into database // Insert into database
console.log('\n💾 Inserting records into database...'); console.log('\n💾 Inserting records into database...');
const dbHost = process.env.LOCAL_DB_HOST || '100.121.163.23';
const dbName = process.env.LOCAL_DB_NAME || 'institutional_trader';
console.log(` Target: ${dbHost}/${dbName}`);
const inserted = await insertFlowRecords(mappedRecords, true); const inserted = await insertFlowRecords(mappedRecords, true);
console.log(`\n✅ Successfully synced ${inserted} records`); console.log(`\n✅ Successfully synced ${inserted} records to ${dbHost}/${dbName}`);
// Summary // Summary
console.log('\n📊 Sync Summary:'); console.log('\n📊 Sync Summary:');

View File

@ -7,7 +7,7 @@ import dotenv from 'dotenv';
dotenv.config(); dotenv.config();
const localDbConfig = { const localDbConfig = {
host: process.env.LOCAL_DB_HOST || 'localhost', host: process.env.LOCAL_DB_HOST || '100.121.163.23',
port: parseInt(process.env.LOCAL_DB_PORT || '5432'), port: parseInt(process.env.LOCAL_DB_PORT || '5432'),
user: process.env.LOCAL_DB_USER || 'postgres', user: process.env.LOCAL_DB_USER || 'postgres',
password: process.env.LOCAL_DB_PASSWORD || 'postgres', password: process.env.LOCAL_DB_PASSWORD || 'postgres',

View File

@ -13,7 +13,7 @@ let pgPool = null;
if (useLocalDb) { if (useLocalDb) {
// Use local PostgreSQL database // Use local PostgreSQL database
const localDbConfig = { const localDbConfig = {
host: process.env.LOCAL_DB_HOST || 'localhost', host: process.env.LOCAL_DB_HOST || '100.121.163.23',
port: parseInt(process.env.LOCAL_DB_PORT || '5432'), port: parseInt(process.env.LOCAL_DB_PORT || '5432'),
user: process.env.LOCAL_DB_USER || 'postgres', user: process.env.LOCAL_DB_USER || 'postgres',
password: process.env.LOCAL_DB_PASSWORD || 'postgres', password: process.env.LOCAL_DB_PASSWORD || 'postgres',
@ -30,6 +30,20 @@ if (useLocalDb) {
); );
} }
// Warn if connecting to localhost (might be wrong)
if (localDbConfig.host === 'localhost' || localDbConfig.host === '127.0.0.1') {
console.warn('⚠️ WARNING: Connecting to localhost!');
console.warn(' If you meant to use remote database at 100.121.163.23,');
console.warn(' set LOCAL_DB_HOST=100.121.163.23 in your .env file');
}
console.log('🔌 Initializing PostgreSQL connection...');
console.log(' Host:', localDbConfig.host);
console.log(' Port:', localDbConfig.port);
console.log(' User:', localDbConfig.user);
console.log(' Database:', localDbConfig.database);
console.log(' Max connections:', 20);
pgPool = new Pool({ pgPool = new Pool({
host: localDbConfig.host, host: localDbConfig.host,
port: localDbConfig.port, port: localDbConfig.port,
@ -42,13 +56,32 @@ if (useLocalDb) {
}); });
pgPool.on('error', (err) => { pgPool.on('error', (err) => {
console.error('Unexpected error on idle client', err); console.error('Unexpected error on idle PostgreSQL client:', err.message);
}); });
console.log('📦 Using local PostgreSQL database'); pgPool.on('connect', () => {
console.log('✅ PostgreSQL client connected');
});
console.log('📦 PostgreSQL connection pool created');
} else if (process.env.DATABASE_URL) { } else if (process.env.DATABASE_URL) {
// Use remote PostgreSQL via connection string // Use remote PostgreSQL via connection string
try { try {
// Parse connection string for logging (mask password)
const dbUrl = process.env.DATABASE_URL;
const urlMatch = dbUrl.match(/postgresql:\/\/([^:]+):([^@]+)@([^:]+):(\d+)\/(.+)/);
console.log('🔌 Initializing PostgreSQL connection from DATABASE_URL...');
if (urlMatch) {
console.log(' Host:', urlMatch[3]);
console.log(' Port:', urlMatch[4]);
console.log(' User:', urlMatch[1]);
console.log(' Database:', urlMatch[5]);
} else {
console.log(' Using connection string (masked)');
}
console.log(' Max connections:', 20);
pgPool = new Pool({ pgPool = new Pool({
connectionString: process.env.DATABASE_URL, connectionString: process.env.DATABASE_URL,
max: 20, max: 20,
@ -57,11 +90,17 @@ if (useLocalDb) {
}); });
pgPool.on('error', (err) => { pgPool.on('error', (err) => {
console.error('Unexpected error on idle client', err); console.error('Unexpected error on idle PostgreSQL client:', err.message);
}); });
pgPool.on('connect', () => {
console.log('✅ PostgreSQL client connected');
});
console.log('📦 PostgreSQL connection pool created from DATABASE_URL');
} catch (err) { } catch (err) {
console.warn('Failed to create PostgreSQL pool:', err.message); console.warn('⚠️ Failed to create PostgreSQL pool:', err.message);
console.warn('Will fall back to Supabase client'); console.warn(' Will fall back to Supabase client');
} }
} }
@ -75,6 +114,11 @@ if (!useLocalDb) {
const supabaseAnonKey = process.env.SUPABASE_ANON_KEY || process.env.SUPABASE_KEY; const supabaseAnonKey = process.env.SUPABASE_ANON_KEY || process.env.SUPABASE_KEY;
if (supabaseUrl && supabaseKey) { if (supabaseUrl && supabaseKey) {
console.log('🔌 Initializing Supabase connection...');
console.log(' URL:', supabaseUrl);
console.log(' Using service key:', supabaseKey ? 'Yes' : 'No');
console.log(' Using anon key:', supabaseAnonKey ? 'Yes' : 'No');
// Create Supabase client with service key for admin operations // Create Supabase client with service key for admin operations
supabase = createClient(supabaseUrl, supabaseKey, { supabase = createClient(supabaseUrl, supabaseKey, {
auth: { auth: {
@ -90,9 +134,40 @@ if (!useLocalDb) {
// Create Supabase client with anon key for regular operations // Create Supabase client with anon key for regular operations
supabaseAnon = createClient(supabaseUrl, supabaseAnonKey); supabaseAnon = createClient(supabaseUrl, supabaseAnonKey);
console.log('📦 Supabase clients created');
} }
} }
// Log connection summary with actual connection details
console.log('\n📊 Database Connection Summary:');
console.log(' USE_LOCAL_DB:', process.env.USE_LOCAL_DB || 'not set');
console.log(' LOCAL_DB_HOST:', process.env.LOCAL_DB_HOST || 'not set (default: 100.121.163.23)');
console.log(' DATABASE_URL:', process.env.DATABASE_URL ? 'set (masked)' : 'not set');
if (pgPool) {
console.log(' Type: PostgreSQL (Direct Connection)');
if (useLocalDb) {
const actualHost = process.env.LOCAL_DB_HOST || '100.121.163.23';
const actualPort = process.env.LOCAL_DB_PORT || '5432';
const actualDb = process.env.LOCAL_DB_NAME || 'institutional_trader';
console.log(` ✅ ACTUAL CONNECTION: ${actualHost}:${actualPort}/${actualDb}`);
} else if (process.env.DATABASE_URL) {
const urlMatch = process.env.DATABASE_URL.match(/postgresql:\/\/([^:]+):([^@]+)@([^:]+):(\d+)\/(.+)/);
if (urlMatch) {
console.log(` ✅ ACTUAL CONNECTION: ${urlMatch[3]}:${urlMatch[4]}/${urlMatch[5]}`);
} else {
console.log(' Using DATABASE_URL connection string');
}
}
} else if (supabase) {
console.log(' Type: Supabase');
console.log(` URL: ${process.env.SUPABASE_URL}`);
} else {
console.log(' ⚠️ No database connection configured');
}
console.log('');
export { pgPool, supabase, supabaseAnon }; export { pgPool, supabase, supabaseAnon };
/** /**
@ -158,13 +233,15 @@ export async function testConnection() {
// Prefer direct PostgreSQL connection if available // Prefer direct PostgreSQL connection if available
if (pgPool) { if (pgPool) {
try { try {
const startTime = Date.now();
await Promise.race([ await Promise.race([
pgPool.query('SELECT 1'), pgPool.query('SELECT 1'),
new Promise((_, reject) => new Promise((_, reject) =>
setTimeout(() => reject(new Error('Connection timeout')), 10000) setTimeout(() => reject(new Error('Connection timeout')), 10000)
) )
]); ]);
console.log('✅ Database connection successful (PostgreSQL direct)'); const latency = Date.now() - startTime;
console.log(`✅ Database connection successful (PostgreSQL direct) - Latency: ${latency}ms`);
return true; return true;
} catch (pgErr) { } catch (pgErr) {
if (pgErr.code === 'ENOTFOUND' || pgErr.message?.includes('ENOTFOUND')) { if (pgErr.code === 'ENOTFOUND' || pgErr.message?.includes('ENOTFOUND')) {
@ -206,12 +283,15 @@ export async function testConnection() {
return false; return false;
} }
const { data, error } = await supabase.from('options_flow').select('count').limit(1); const startTime = Date.now();
const { error } = await supabase.from('options_flow').select('count').limit(1);
const latency = Date.now() - startTime;
if (error) { if (error) {
// PGRST116 = table doesn't exist (ok for setup) // PGRST116 = table doesn't exist (ok for setup)
// PGRST301 = invalid API key // PGRST301 = invalid API key
if (error.code === 'PGRST116') { if (error.code === 'PGRST116') {
console.log('✅ Database connection successful (table may not exist yet)'); console.log(`✅ Database connection successful (Supabase) - Table may not exist yet - Latency: ${latency}ms`);
return true; return true;
} }
if (error.code === 'PGRST301' || error.message?.includes('Invalid API key')) { if (error.code === 'PGRST301' || error.message?.includes('Invalid API key')) {
@ -238,7 +318,7 @@ export async function testConnection() {
console.error('Database connection error:', error); console.error('Database connection error:', error);
return false; return false;
} }
console.log('✅ Database connection successful (Supabase)'); console.log(`✅ Database connection successful (Supabase) - Latency: ${latency}ms`);
return true; return true;
} catch (err) { } catch (err) {
console.error('Database connection failed:', err); console.error('Database connection failed:', err);