chitfund/DEPLOYMENT_TROUBLESHOOTING.md

# 🔧 Deployment Troubleshooting Guide - LuckyChit

Common deployment issues and their solutions.

---

## 🚨 Critical Issues

### 1. bcrypt "invalid ELF header" Error

**Error Message:**
```
Error: /home/luckychit/apps/chitfund/backend/node_modules/bcrypt/lib/binding/napi-v3/bcrypt_lib.node: invalid ELF header
code: 'ERR_DLOPEN_FAILED'
```

**Why This Happens:**
- You installed `node_modules` on Windows
- You deployed to a Linux server
- bcrypt has **native C++ bindings** that are platform-specific
- Windows binaries cannot run on Linux

**✅ Solution (Run on Production Server):**

```bash
cd /home/luckychit/apps/chitfund/backend

# Stop PM2 process
pm2 stop luckychit-api

# Remove all node modules
rm -rf node_modules package-lock.json

# Reinstall on Linux (this will compile native bindings for Linux)
npm install

# Restart PM2
pm2 restart luckychit-api

# Check logs
pm2 logs luckychit-api
```

**✅ Prevention:**

1. **Never copy `node_modules/` between different operating systems**
2. **Add to your `.gitignore`** (already done):
   ```
   node_modules/
   ```
3. **Always run `npm install` on the target server**
4. **Use the deployment script** which handles this automatically:
   ```bash
   ./deploy.sh production
   ```

**Alternative: Use bcryptjs (Pure JavaScript)**

If you continue having issues, you can switch to `bcryptjs` (no native bindings):

```bash
npm uninstall bcrypt
npm install bcryptjs
```

Then update your code:
```javascript
// Change this:
const bcrypt = require('bcrypt');

// To this:
const bcrypt = require('bcryptjs');

// The API is identical, no other changes needed!
```

---

## 🔥 Other Native Module Issues

**Similar errors with other packages:**
- `node-sass`
- `sharp`
- `sqlite3`
- `canvas`
- `node-gyp` related errors

**Solution (Same as bcrypt):**
```bash
rm -rf node_modules package-lock.json
npm install
pm2 restart your-app
```

---

## 📦 Common PM2 Issues

### App Won't Start - Crashes Immediately

**Check logs first:**
```bash
pm2 logs luckychit-api --lines 200
```

**Common causes:**

#### 1. Missing `.env` File
```bash
# Check if .env exists
ls -la /home/luckychit/apps/chitfund/backend/.env

# If not, create it
cp .env.example .env
nano .env  # Edit with production credentials
```

#### 2. Database Connection Failed
```bash
# Check PostgreSQL is running
sudo systemctl status postgresql

# Test connection
psql -U your_db_user -d chitfund_prod

# Check .env has correct credentials
cat backend/.env | grep DB_
```

#### 3. Port Already in Use
```bash
# Find what's using port 3000
lsof -i :3000
# or
sudo netstat -tulpn | grep 3000

# Kill the process
kill -9 <PID>

# Or change port in .env
echo "PORT=3001" >> .env
```

#### 4. Missing Dependencies
```bash
cd backend
npm install
pm2 restart luckychit-api
```

---

### PM2 Shows "errored" Status

```bash
# Check detailed error
pm2 describe luckychit-api

# View recent logs
pm2 logs luckychit-api --lines 100

# Try manual start to see error
cd backend
node server.js
```

---

### App Keeps Restarting (Crash Loop)

```bash
# Check restart count
pm2 list

# View error logs
pm2 logs luckychit-api --err

# Common causes:
# - Syntax error in code
# - Missing environment variable
# - Database not accessible
# - Port conflict
```

**Reset restart counter:**
```bash
pm2 reset luckychit-api
```

---

## 🗄️ Database Issues

### Connection Refused

**Symptoms:**
```
Error: connect ECONNREFUSED 127.0.0.1:5432
```

**Solutions:**

1. **Check PostgreSQL is running:**
   ```bash
   sudo systemctl status postgresql
   sudo systemctl start postgresql
   ```

2. **Check PostgreSQL is listening:**
   ```bash
   sudo netstat -tulpn | grep 5432
   ```

3. **Check `pg_hba.conf` allows connection:**
   ```bash
   sudo nano /etc/postgresql/*/main/pg_hba.conf
   
   # Should have:
   local   all             all                                     md5
   host    all             all             127.0.0.1/32            md5
   ```

4. **Restart PostgreSQL:**
   ```bash
   sudo systemctl restart postgresql
   ```

### Authentication Failed

**Symptoms:**
```
Error: password authentication failed for user "postgres"
```

**Solution:**
```bash
# Reset password
sudo -u postgres psql
ALTER USER your_db_user WITH PASSWORD 'new_password';
\q

# Update .env file
nano backend/.env
# Change DB_PASSWORD=new_password
```

### Database Does Not Exist

**Symptoms:**
```
Error: database "chitfund_prod" does not exist
```

**Solution:**
```bash
# Create database
sudo -u postgres psql
CREATE DATABASE chitfund_prod;
GRANT ALL PRIVILEGES ON DATABASE chitfund_prod TO your_db_user;
\q

# Run migrations
cd backend
npm run migrate  # If you have migrations
```

---

## 🌐 Network Issues

### Cannot Access from External IP

**Check firewall:**
```bash
# Allow port 3000
sudo ufw allow 3000/tcp

# Check status
sudo ufw status

# Or disable temporarily for testing
sudo ufw disable
```

**Check if app is listening on 0.0.0.0:**
```bash
sudo netstat -tulpn | grep 3000

# Should show 0.0.0.0:3000 not 127.0.0.1:3000
```

**Fix in code:**
```javascript
// In server.js, ensure:
app.listen(PORT, '0.0.0.0', () => {
  console.log(`Server running on port ${PORT}`);
});
```

### nginx 502 Bad Gateway

**Symptoms:**
- nginx returns 502 error
- Backend is running but not accessible through nginx

**Solutions:**

1. **Check backend is running:**
   ```bash
   pm2 list
   curl http://localhost:3000/api/health
   ```

2. **Check nginx configuration:**
   ```bash
   sudo nginx -t
   sudo systemctl restart nginx
   ```

3. **Check nginx error logs:**
   ```bash
   sudo tail -f /var/log/nginx/error.log
   ```

4. **Verify proxy_pass URL is correct:**
   ```bash
   sudo nano /etc/nginx/sites-available/luckychit
   # Should have: proxy_pass http://localhost:3000;
   ```

---

## 💾 Permission Issues

### EACCES: permission denied

**Symptoms:**
```
Error: EACCES: permission denied, open '/path/to/file'
```

**Solutions:**

1. **Fix ownership:**
   ```bash
   sudo chown -R $USER:$USER /home/luckychit/apps/chitfund
   ```

2. **Fix log directory permissions:**
   ```bash
   mkdir -p backend/logs
   chmod 755 backend/logs
   ```

3. **Run PM2 as correct user:**
   ```bash
   # Don't use sudo with PM2 unless necessary
   pm2 kill
   pm2 start ecosystem.config.js --env production
   ```

---

## 🔄 Auto-Start Issues

### PM2 Doesn't Start After Reboot

**Check if startup is configured:**
```bash
pm2 startup
# Run the command it outputs

pm2 save
```

**Verify startup script exists:**
```bash
# For systemd:
systemctl status pm2-$USER

# Check logs:
journalctl -u pm2-$USER -n 50
```

**Re-configure if needed:**
```bash
pm2 unstartup
pm2 startup
# Run the command it outputs
pm2 save
```

---

## 📊 Performance Issues

### High Memory Usage

**Check memory:**
```bash
pm2 monit
# or
pm2 list
```

**Solutions:**

1. **Reduce PM2 instances:**
   ```bash
   pm2 scale luckychit-api 2  # Reduce to 2 instances
   ```

2. **Increase memory limit in ecosystem.config.js:**
   ```javascript
   max_memory_restart: '2G'  // Increase from 1G
   ```

3. **Check for memory leaks:**
   ```bash
   pm2 logs luckychit-api
   # Look for memory warnings
   ```

### High CPU Usage

**Check CPU:**
```bash
top
# or
htop
```

**Solutions:**

1. **Reduce instances:**
   ```bash
   pm2 scale luckychit-api 2
   ```

2. **Check for infinite loops in code**

3. **Optimize database queries**

---

## 🔍 Debugging Tips

### Enable Debug Mode

**Temporary debugging:**
```bash
pm2 stop luckychit-api
cd backend
NODE_ENV=production DEBUG=* node server.js
```

### Check All Logs

```bash
# PM2 logs
pm2 logs --lines 500

# System logs
journalctl -xe

# nginx logs (if using)
sudo tail -f /var/log/nginx/error.log
sudo tail -f /var/log/nginx/access.log

# PostgreSQL logs
sudo tail -f /var/log/postgresql/postgresql-*.log
```

### Test API Manually

```bash
# Test from server
curl http://localhost:3000/api/health

# Test with verbose output
curl -v http://localhost:3000/api/auth/login \
  -H "Content-Type: application/json" \
  -d '{"mobile":"9999999999","password":"password123"}'
```

---

## 🆘 Emergency Reset

**If everything is broken, start fresh:**

```bash
# Stop and remove all PM2 processes
pm2 kill

# Remove node_modules
cd /home/luckychit/apps/chitfund/backend
rm -rf node_modules package-lock.json

# Fresh install
npm install

# Start fresh
pm2 start ecosystem.config.js --env production
pm2 save

# Check status
pm2 list
pm2 logs
```

---

## 📞 Getting Help

When asking for help, provide:

1. **Error logs:**
   ```bash
   pm2 logs luckychit-api --lines 200 > error.log
   ```

2. **PM2 status:**
   ```bash
   pm2 list
   pm2 describe luckychit-api
   ```

3. **Environment info:**
   ```bash
   node --version
   pm2 --version
   cat /etc/os-release
   ```

4. **Configuration:**
   ```bash
   cat backend/ecosystem.config.js
   # (Don't share .env - contains secrets!)
   ```

---

## ✅ Prevention Checklist

- [ ] Never commit `node_modules/` to Git
- [ ] Always run `npm install` on production server
- [ ] Use `.env` for environment-specific config
- [ ] Keep `.env` out of Git (in `.gitignore`)
- [ ] Use PM2 startup for auto-restart
- [ ] Setup log rotation
- [ ] Monitor memory/CPU usage
- [ ] Setup database backups
- [ ] Test deployment in staging first
- [ ] Keep dependencies updated
- [ ] Document your deployment process

---

**Most issues are solved by rebuilding native modules on the production server! 🎯**