# Production Deployment Guide

This guide covers best practices and instructions for deploying the Redmine-Gitea Sync Server in production environments.

## Table of Contents

- [Prerequisites](#prerequisites)
- [Deployment Options](#deployment-options)
- [Docker Deployment](#docker-deployment)
- [Systemd Service](#systemd-service)
- [Nginx Reverse Proxy](#nginx-reverse-proxy)
- [Security Hardening](#security-hardening)
- [Monitoring and Logging](#monitoring-and-logging)
- [Backup and Recovery](#backup-and-recovery)
- [Performance Tuning](#performance-tuning)

---

## Prerequisites

### System Requirements

**Minimum:**
- CPU: 1 core
- RAM: 512 MB
- Storage: 1 GB
- OS: Linux (Ubuntu 20.04+, CentOS 8+, Debian 11+)

**Recommended:**
- CPU: 2 cores
- RAM: 2 GB
- Storage: 5 GB
- OS: Ubuntu 22.04 LTS or later

### Software Requirements

- Node.js 18.x or later
- npm 9.x or later
- Process manager (PM2, systemd, or Docker)
- Reverse proxy (Nginx or Apache)
- SSL certificates (Let's Encrypt recommended)

---

## Deployment Options

### Option 1: PM2 (Recommended)

Node.js process manager with monitoring, auto-restart, and clustering capabilities.

### Option 2: Docker

Easiest deployment with consistent environment.

### Option 3: Systemd Service

Native Linux service for better OS integration.

### Option 4: Bare Metal

Direct execution for maximum control.

---

## PM2 Deployment (Recommended)

### Why PM2?

PM2 is a production-ready process manager for Node.js applications with:
- Automatic restarts on crashes
- Built-in load balancing
- Log management
- Monitoring dashboard
- Startup scripts for system boot
- Zero-downtime reloads
- Memory monitoring

### Installation

```bash
# Install PM2 globally
npm install -g pm2

# Verify installation
pm2 --version
```

### Initial Setup

```bash
# Navigate to application directory
cd /opt/redmine-gitea-sync

# Install dependencies
npm install --production

# Configure ecosystem file
nano ecosystem.config.cjs
```

Update the `env` section with your credentials:

```javascript
env: {
  NODE_ENV: 'production',
  PORT: 3002,
  REDMINE_API_URL: 'https://redmine.example.com',
  REDMINE_API_KEY: 'your_api_key',
  GITEA_API_URL: 'https://gitea.example.com',
  GITEA_TOKEN: 'your_token',
  GITEA_OWNER: 'your_username',
  // ... other settings
}
```

### Start the Application

```bash
# Start with PM2
pm2 start ecosystem.config.cjs

# Save the PM2 process list
pm2 save

# Generate startup script
pm2 startup

# Follow the command output, then save again
pm2 save
```

### PM2 Management Commands

```bash
# View all processes
pm2 list

# View detailed info
pm2 show redmine-gitea-sync

# View logs in real-time
pm2 logs redmine-gitea-sync

# View last 100 lines
pm2 logs redmine-gitea-sync --lines 100

# Monitor resources
pm2 monit

# Restart application
pm2 restart redmine-gitea-sync

# Reload with zero downtime (for cluster mode)
pm2 reload redmine-gitea-sync

# Stop application
pm2 stop redmine-gitea-sync

# Delete from PM2
pm2 delete redmine-gitea-sync

# Update environment variables
pm2 restart redmine-gitea-sync --update-env
```

### Log Management

```bash
# View logs
pm2 logs redmine-gitea-sync

# Flush logs
pm2 flush

# Install log rotation module
pm2 install pm2-logrotate

# Configure log rotation
pm2 set pm2-logrotate:max_size 10M
pm2 set pm2-logrotate:retain 7
pm2 set pm2-logrotate:compress true
```

### PM2 with Nginx

PM2 runs on localhost:3002, use Nginx as reverse proxy:

```nginx
server {
    listen 80;
    server_name sync.example.com;
    
    location / {
        proxy_pass http://localhost:3002;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection 'upgrade';
        proxy_set_header Host $host;
        proxy_cache_bypass $http_upgrade;
    }
}
```

### Development Mode

For development with verbose logging:

```bash
# Start in development mode
pm2 start ecosystem.config.cjs --env development

# Watch for file changes (optional)
pm2 start ecosystem.config.cjs --watch
```

### Monitoring and Alerts

```bash
# Install PM2 web interface (optional)
pm2 install pm2-server-monit

# Setup email notifications on errors
pm2 install pm2-auto-pull
```

### Update Application

```bash
# Pull latest changes
cd /opt/redmine-gitea-sync
git pull

# Install new dependencies
npm install --production

# Restart with zero downtime
pm2 reload redmine-gitea-sync

# Or restart normally
pm2 restart redmine-gitea-sync
```

### Clustering (Optional)

For high-traffic scenarios, enable cluster mode:

```javascript
// In ecosystem.config.cjs
instances: 'max',  // or specify number
exec_mode: 'cluster',
```

Then restart:
```bash
pm2 reload redmine-gitea-sync
```

### Security Best Practices

```bash
# Run PM2 as dedicated user
sudo useradd -r -s /bin/false pm2user
sudo chown -R pm2user:pm2user /opt/redmine-gitea-sync

# Start PM2 as that user
sudo su - pm2user -s /bin/bash -c "pm2 start /opt/redmine-gitea-sync/ecosystem.config.cjs"

# Save and setup startup
sudo su - pm2user -s /bin/bash -c "pm2 save"
sudo env PATH=$PATH:/usr/bin pm2 startup systemd -u pm2user --hp /home/pm2user
```

### Troubleshooting PM2

**Issue: Process keeps restarting**
```bash
# Check error logs
pm2 logs redmine-gitea-sync --err

# Increase restart delay in ecosystem.config.cjs
restart_delay: 5000
```

**Issue: High memory usage**
```bash
# Check memory
pm2 list

# Adjust memory limit in ecosystem.config.cjs
max_memory_restart: '500M'
```

**Issue: PM2 not starting on boot**
```bash
# Regenerate startup script
pm2 unstartup
pm2 startup
pm2 save
```

---

## Docker Deployment

### Dockerfile

Create `Dockerfile`:

```dockerfile
FROM node:18-alpine

# Install dependencies for native modules
RUN apk add --no-cache python3 make g++

# Create app directory
WORKDIR /app

# Copy package files
COPY package*.json ./

# Install dependencies
RUN npm ci --only=production

# Copy application files
COPY . .

# Create non-root user
RUN addgroup -g 1001 -S nodejs && \
    adduser -S nodejs -u 1001 && \
    chown -R nodejs:nodejs /app

# Switch to non-root user
USER nodejs

# Expose port
EXPOSE 3002

# Health check
HEALTHCHECK --interval=30s --timeout=10s --start-period=40s --retries=3 \
  CMD node -e "require('http').get('http://localhost:3002/health', (res) => { process.exit(res.statusCode === 200 ? 0 : 1); });"

# Start application
CMD ["node", "server.js"]
```

### Docker Compose

Create `docker-compose.yml`:

```yaml
version: '3.8'

services:
  sync-server:
    build: .
    container_name: redmine-gitea-sync
    restart: unless-stopped
    ports:
      - "3002:3002"
    environment:
      - NODE_ENV=production
      - PORT=3002
    env_file:
      - .env
    volumes:
      - ./logs:/app/logs
    networks:
      - sync-network
    logging:
      driver: "json-file"
      options:
        max-size: "10m"
        max-file: "3"

networks:
  sync-network:
    driver: bridge
```

### Build and Run

```bash
# Build image
docker build -t redmine-gitea-sync .

# Run with docker-compose
docker-compose up -d

# View logs
docker-compose logs -f

# Stop service
docker-compose down
```

### Docker Hub Deployment

```bash
# Tag image
docker tag redmine-gitea-sync:latest your-username/redmine-gitea-sync:latest

# Push to Docker Hub
docker push your-username/redmine-gitea-sync:latest

# Pull and run on production server
docker pull your-username/redmine-gitea-sync:latest
docker run -d --name sync-server \
  --env-file .env \
  -p 3002:3002 \
  --restart unless-stopped \
  your-username/redmine-gitea-sync:latest
```

---

## Systemd Service

### Create Service File

Create `/etc/systemd/system/redmine-gitea-sync.service`:

```ini
[Unit]
Description=Redmine-Gitea Sync Server
After=network.target

[Service]
Type=simple
User=sync-user
Group=sync-user
WorkingDirectory=/opt/redmine-gitea-sync
EnvironmentFile=/opt/redmine-gitea-sync/.env
ExecStart=/usr/bin/node /opt/redmine-gitea-sync/server.js
Restart=always
RestartSec=10
StandardOutput=journal
StandardError=journal
SyslogIdentifier=redmine-gitea-sync

# Security settings
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=strict
ProtectHome=true
ReadWritePaths=/opt/redmine-gitea-sync/logs

[Install]
WantedBy=multi-user.target
```

### Setup Instructions

```bash
# Create dedicated user
sudo useradd -r -s /bin/false sync-user

# Create application directory
sudo mkdir -p /opt/redmine-gitea-sync
sudo chown sync-user:sync-user /opt/redmine-gitea-sync

# Copy application files
sudo cp -r . /opt/redmine-gitea-sync/
sudo chown -R sync-user:sync-user /opt/redmine-gitea-sync

# Install dependencies
cd /opt/redmine-gitea-sync
sudo -u sync-user npm ci --only=production

# Configure environment
sudo cp .env.example /opt/redmine-gitea-sync/.env
sudo nano /opt/redmine-gitea-sync/.env
sudo chown sync-user:sync-user /opt/redmine-gitea-sync/.env
sudo chmod 600 /opt/redmine-gitea-sync/.env

# Enable and start service
sudo systemctl daemon-reload
sudo systemctl enable redmine-gitea-sync
sudo systemctl start redmine-gitea-sync

# Check status
sudo systemctl status redmine-gitea-sync

# View logs
sudo journalctl -u redmine-gitea-sync -f
```

### Service Management

```bash
# Start service
sudo systemctl start redmine-gitea-sync

# Stop service
sudo systemctl stop redmine-gitea-sync

# Restart service
sudo systemctl restart redmine-gitea-sync

# Check status
sudo systemctl status redmine-gitea-sync

# Enable auto-start
sudo systemctl enable redmine-gitea-sync

# Disable auto-start
sudo systemctl disable redmine-gitea-sync

# View logs
sudo journalctl -u redmine-gitea-sync -n 100 -f
```

---

## Nginx Reverse Proxy

### Basic Configuration

Create `/etc/nginx/sites-available/sync-server`:

```nginx
upstream sync_server {
    server 127.0.0.1:3002;
    keepalive 64;
}

server {
    listen 80;
    server_name sync.example.com;
    
    # Redirect to HTTPS
    return 301 https://$server_name$request_uri;
}

server {
    listen 443 ssl http2;
    server_name sync.example.com;

    # SSL Configuration
    ssl_certificate /etc/letsencrypt/live/sync.example.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/sync.example.com/privkey.pem;
    ssl_protocols TLSv1.2 TLSv1.3;
    ssl_ciphers HIGH:!aNULL:!MD5;
    ssl_prefer_server_ciphers on;

    # Security Headers
    add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;
    add_header X-Frame-Options "DENY" always;
    add_header X-Content-Type-Options "nosniff" always;
    add_header X-XSS-Protection "1; mode=block" always;

    # Logging
    access_log /var/log/nginx/sync-server-access.log;
    error_log /var/log/nginx/sync-server-error.log;

    # Request size limits
    client_max_body_size 10M;

    # Proxy settings
    location / {
        proxy_pass http://sync_server;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection 'upgrade';
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_cache_bypass $http_upgrade;
        proxy_read_timeout 300s;
        proxy_connect_timeout 75s;
    }

    # Health check endpoint (no auth required)
    location /health {
        proxy_pass http://sync_server;
        access_log off;
    }
}
```

### Enable Configuration

```bash
# Enable site
sudo ln -s /etc/nginx/sites-available/sync-server /etc/nginx/sites-enabled/

# Test configuration
sudo nginx -t

# Reload Nginx
sudo systemctl reload nginx
```

### SSL with Let's Encrypt

```bash
# Install Certbot
sudo apt install certbot python3-certbot-nginx

# Obtain certificate
sudo certbot --nginx -d sync.example.com

# Auto-renewal is configured automatically
# Test renewal
sudo certbot renew --dry-run
```

---

## Security Hardening

### Environment Variables

```bash
# Secure .env file
chmod 600 .env
chown sync-user:sync-user .env

# Never commit .env to version control
echo ".env" >> .gitignore
```

### Firewall Configuration

```bash
# Allow only necessary ports
sudo ufw default deny incoming
sudo ufw default allow outgoing
sudo ufw allow ssh
sudo ufw allow 80/tcp
sudo ufw allow 443/tcp
sudo ufw enable
```

### Rate Limiting (Nginx)

Add to Nginx configuration:

```nginx
# Define rate limit zone
limit_req_zone $binary_remote_addr zone=webhook:10m rate=10r/s;

# Apply to webhook endpoints
location /webhook/ {
    limit_req zone=webhook burst=20 nodelay;
    proxy_pass http://sync_server;
    # ... other proxy settings
}
```

### Webhook Signature Verification

Implement webhook signature verification for production:

```javascript
// Add to server.js
import crypto from 'crypto';

function verifyRedmineSignature(req) {
    const signature = req.headers['x-redmine-signature'];
    const secret = process.env.REDMINE_WEBHOOK_SECRET;
    
    if (!signature || !secret) return false;
    
    const hash = crypto
        .createHmac('sha256', secret)
        .update(JSON.stringify(req.body))
        .digest('hex');
    
    return signature === hash;
}

// Add verification to webhook endpoint
app.post('/webhook/redmine', (req, res, next) => {
    if (!verifyRedmineSignature(req)) {
        return res.status(401).json({ error: 'Invalid signature' });
    }
    next();
}, async (req, res) => {
    // ... existing handler
});
```

### IP Whitelisting

```nginx
# In Nginx configuration
geo $allowed_ip {
    default 0;
    192.168.1.0/24 1;  # Internal network
    10.0.0.0/8 1;       # Private network
    # Add Redmine and Gitea server IPs
}

location /webhook/ {
    if ($allowed_ip = 0) {
        return 403;
    }
    proxy_pass http://sync_server;
}
```

---

## Monitoring and Logging

### Log Rotation

Create `/etc/logrotate.d/redmine-gitea-sync`:

```
/opt/redmine-gitea-sync/logs/*.log {
    daily
    rotate 14
    compress
    delaycompress
    notifempty
    missingok
    create 0640 sync-user sync-user
    sharedscripts
    postrotate
        systemctl reload redmine-gitea-sync > /dev/null 2>&1 || true
    endscript
}
```

### Health Monitoring

Simple monitoring script `monitor.sh`:

```bash
#!/bin/bash

HEALTH_URL="http://localhost:3002/health"
MAX_RETRIES=3
RETRY_COUNT=0

while [ $RETRY_COUNT -lt $MAX_RETRIES ]; do
    RESPONSE=$(curl -s -o /dev/null -w "%{http_code}" $HEALTH_URL)
    
    if [ "$RESPONSE" -eq 200 ]; then
        echo "$(date): Health check passed"
        exit 0
    fi
    
    RETRY_COUNT=$((RETRY_COUNT + 1))
    sleep 5
done

echo "$(date): Health check failed after $MAX_RETRIES attempts"
# Send alert (email, Slack, etc.)
exit 1
```

Add to crontab:

```bash
# Run health check every 5 minutes
*/5 * * * * /opt/redmine-gitea-sync/monitor.sh >> /var/log/sync-health.log 2>&1
```

### Prometheus Metrics

Install prometheus client:

```bash
npm install prom-client
```

Add metrics endpoint:

```javascript
import promClient from 'prom-client';

// Create metrics
const register = new promClient.Registry();
const syncCounter = new promClient.Counter({
    name: 'sync_operations_total',
    help: 'Total number of sync operations',
    labelNames: ['direction', 'status'],
    registers: [register]
});

const syncDuration = new promClient.Histogram({
    name: 'sync_duration_seconds',
    help: 'Duration of sync operations',
    labelNames: ['direction'],
    registers: [register]
});

// Expose metrics endpoint
app.get('/metrics', async (req, res) => {
    res.set('Content-Type', register.contentType);
    res.end(await register.metrics());
});
```

---

## Backup and Recovery

### Configuration Backup

```bash
#!/bin/bash
# backup.sh

BACKUP_DIR="/backup/sync-server"
DATE=$(date +%Y%m%d-%H%M%S)
APP_DIR="/opt/redmine-gitea-sync"

mkdir -p $BACKUP_DIR

# Backup configuration
tar -czf "$BACKUP_DIR/config-$DATE.tar.gz" \
    "$APP_DIR/.env" \
    "$APP_DIR/package.json" \
    "$APP_DIR/package-lock.json"

# Backup logs
tar -czf "$BACKUP_DIR/logs-$DATE.tar.gz" "$APP_DIR/logs"

# Remove backups older than 30 days
find $BACKUP_DIR -name "*.tar.gz" -mtime +30 -delete

echo "Backup completed: $DATE"
```

### Disaster Recovery

```bash
#!/bin/bash
# restore.sh

BACKUP_FILE=$1
RESTORE_DIR="/opt/redmine-gitea-sync"

if [ -z "$BACKUP_FILE" ]; then
    echo "Usage: $0 <backup-file>"
    exit 1
fi

# Stop service
sudo systemctl stop redmine-gitea-sync

# Restore configuration
tar -xzf "$BACKUP_FILE" -C /

# Reinstall dependencies
cd $RESTORE_DIR
npm ci --only=production

# Start service
sudo systemctl start redmine-gitea-sync

echo "Restore completed"
```

---

## Performance Tuning

### Node.js Optimization

```bash
# Set Node.js memory limit
NODE_OPTIONS="--max-old-space-size=2048" node server.js
```

Update systemd service:

```ini
[Service]
Environment="NODE_ENV=production"
Environment="NODE_OPTIONS=--max-old-space-size=2048"
```

### Database Connection Pool

If using a database for caching (future):

```javascript
const pool = {
    min: 2,
    max: 10,
    acquireTimeoutMillis: 30000,
    idleTimeoutMillis: 30000
};
```

### HTTP Keep-Alive

Already configured in axios clients:

```javascript
const http = require('http');
const https = require('https');

const httpAgent = new http.Agent({ keepAlive: true });
const httpsAgent = new https.Agent({ keepAlive: true });

const client = axios.create({
    httpAgent,
    httpsAgent
});
```

### Nginx Tuning

```nginx
worker_processes auto;
worker_rlimit_nofile 65535;

events {
    worker_connections 4096;
    use epoll;
    multi_accept on;
}

http {
    # Enable caching
    proxy_cache_path /var/cache/nginx levels=1:2 keys_zone=my_cache:10m max_size=1g inactive=60m;
    
    # Compression
    gzip on;
    gzip_vary on;
    gzip_comp_level 6;
    gzip_types text/plain text/css application/json application/javascript;
}
```

---

## Troubleshooting Production Issues

### High Memory Usage

```bash
# Check memory usage
ps aux | grep node

# Restart service
sudo systemctl restart redmine-gitea-sync

# Monitor memory
watch -n 5 'ps aux | grep node'
```

### Connection Issues

```bash
# Test Redmine connection
curl -H "X-Redmine-API-Key: YOUR_KEY" https://redmine.example.com/issues.json

# Test Gitea connection
curl -H "Authorization: token YOUR_TOKEN" https://gitea.example.com/api/v1/repos

# Check DNS resolution
nslookup redmine.example.com
nslookup gitea.example.com
```

### Webhook Not Triggering

```bash
# Check Nginx logs
sudo tail -f /var/log/nginx/sync-server-access.log
sudo tail -f /var/log/nginx/sync-server-error.log

# Check application logs
sudo journalctl -u redmine-gitea-sync -f

# Test webhook manually
curl -X POST http://localhost:3002/webhook/redmine \
  -H "Content-Type: application/json" \
  -d @test-payload.json
```

### Performance Issues

```bash
# Check CPU usage
top -p $(pgrep -f "node.*server.js")

# Check network connections
netstat -an | grep :3002

# Analyze slow requests
tail -f /var/log/nginx/sync-server-access.log | awk '{print $4, $7}'
```

---

## Production Checklist

- [ ] SSL certificates configured and auto-renewal setup
- [ ] Firewall rules configured
- [ ] Environment variables secured (chmod 600 .env)
- [ ] Service configured to start on boot
- [ ] Log rotation configured
- [ ] Monitoring and alerting setup
- [ ] Backup strategy implemented
- [ ] Health checks configured
- [ ] Webhook signatures verified
- [ ] IP whitelisting configured
- [ ] Documentation updated with production URLs
- [ ] Disaster recovery plan documented
- [ ] Performance testing completed
- [ ] Security audit completed

---

## Support and Maintenance

### Regular Maintenance Tasks

**Weekly:**
- Check logs for errors
- Verify webhooks are functioning
- Review sync statistics

**Monthly:**
- Update dependencies: `npm update`
- Review and rotate logs
- Test disaster recovery procedures
- Check SSL certificate expiration

**Quarterly:**
- Security audit
- Performance review
- Update documentation
- Backup verification

### Getting Help

For production issues:
1. Check logs: `sudo journalctl -u redmine-gitea-sync -n 100`
2. Verify configuration: `GET /status`
3. Test connectivity to Redmine and Gitea
4. Review recent changes
5. Consult documentation
6. Open issue in repository with logs and configuration details