diff --git a/DEPLOYMENT.md b/DEPLOYMENT.md new file mode 100644 index 0000000..54bcb22 --- /dev/null +++ b/DEPLOYMENT.md @@ -0,0 +1,1042 @@ +# Production Deployment Guide + +This guide covers best practices and instructions for deploying the Redmine-Gitea Sync Server in production environments. + +## Table of Contents + +- [Prerequisites](#prerequisites) +- [Deployment Options](#deployment-options) +- [Docker Deployment](#docker-deployment) +- [Systemd Service](#systemd-service) +- [Nginx Reverse Proxy](#nginx-reverse-proxy) +- [Security Hardening](#security-hardening) +- [Monitoring and Logging](#monitoring-and-logging) +- [Backup and Recovery](#backup-and-recovery) +- [Performance Tuning](#performance-tuning) + +--- + +## Prerequisites + +### System Requirements + +**Minimum:** +- CPU: 1 core +- RAM: 512 MB +- Storage: 1 GB +- OS: Linux (Ubuntu 20.04+, CentOS 8+, Debian 11+) + +**Recommended:** +- CPU: 2 cores +- RAM: 2 GB +- Storage: 5 GB +- OS: Ubuntu 22.04 LTS or later + +### Software Requirements + +- Node.js 18.x or later +- npm 9.x or later +- Process manager (PM2, systemd, or Docker) +- Reverse proxy (Nginx or Apache) +- SSL certificates (Let's Encrypt recommended) + +--- + +## Deployment Options + +### Option 1: PM2 (Recommended) + +Node.js process manager with monitoring, auto-restart, and clustering capabilities. + +### Option 2: Docker + +Easiest deployment with consistent environment. + +### Option 3: Systemd Service + +Native Linux service for better OS integration. + +### Option 4: Bare Metal + +Direct execution for maximum control. + +--- + +## PM2 Deployment (Recommended) + +### Why PM2? + +PM2 is a production-ready process manager for Node.js applications with: +- Automatic restarts on crashes +- Built-in load balancing +- Log management +- Monitoring dashboard +- Startup scripts for system boot +- Zero-downtime reloads +- Memory monitoring + +### Installation + +```bash +# Install PM2 globally +npm install -g pm2 + +# Verify installation +pm2 --version +``` + +### Initial Setup + +```bash +# Navigate to application directory +cd /opt/redmine-gitea-sync + +# Install dependencies +npm install --production + +# Configure ecosystem file +nano ecosystem.config.cjs +``` + +Update the `env` section with your credentials: + +```javascript +env: { + NODE_ENV: 'production', + PORT: 3002, + REDMINE_API_URL: 'https://redmine.example.com', + REDMINE_API_KEY: 'your_api_key', + GITEA_API_URL: 'https://gitea.example.com', + GITEA_TOKEN: 'your_token', + GITEA_OWNER: 'your_username', + // ... other settings +} +``` + +### Start the Application + +```bash +# Start with PM2 +pm2 start ecosystem.config.cjs + +# Save the PM2 process list +pm2 save + +# Generate startup script +pm2 startup + +# Follow the command output, then save again +pm2 save +``` + +### PM2 Management Commands + +```bash +# View all processes +pm2 list + +# View detailed info +pm2 show redmine-gitea-sync + +# View logs in real-time +pm2 logs redmine-gitea-sync + +# View last 100 lines +pm2 logs redmine-gitea-sync --lines 100 + +# Monitor resources +pm2 monit + +# Restart application +pm2 restart redmine-gitea-sync + +# Reload with zero downtime (for cluster mode) +pm2 reload redmine-gitea-sync + +# Stop application +pm2 stop redmine-gitea-sync + +# Delete from PM2 +pm2 delete redmine-gitea-sync + +# Update environment variables +pm2 restart redmine-gitea-sync --update-env +``` + +### Log Management + +```bash +# View logs +pm2 logs redmine-gitea-sync + +# Flush logs +pm2 flush + +# Install log rotation module +pm2 install pm2-logrotate + +# Configure log rotation +pm2 set pm2-logrotate:max_size 10M +pm2 set pm2-logrotate:retain 7 +pm2 set pm2-logrotate:compress true +``` + +### PM2 with Nginx + +PM2 runs on localhost:3002, use Nginx as reverse proxy: + +```nginx +server { + listen 80; + server_name sync.example.com; + + location / { + proxy_pass http://localhost:3002; + proxy_http_version 1.1; + proxy_set_header Upgrade $http_upgrade; + proxy_set_header Connection 'upgrade'; + proxy_set_header Host $host; + proxy_cache_bypass $http_upgrade; + } +} +``` + +### Development Mode + +For development with verbose logging: + +```bash +# Start in development mode +pm2 start ecosystem.config.cjs --env development + +# Watch for file changes (optional) +pm2 start ecosystem.config.cjs --watch +``` + +### Monitoring and Alerts + +```bash +# Install PM2 web interface (optional) +pm2 install pm2-server-monit + +# Setup email notifications on errors +pm2 install pm2-auto-pull +``` + +### Update Application + +```bash +# Pull latest changes +cd /opt/redmine-gitea-sync +git pull + +# Install new dependencies +npm install --production + +# Restart with zero downtime +pm2 reload redmine-gitea-sync + +# Or restart normally +pm2 restart redmine-gitea-sync +``` + +### Clustering (Optional) + +For high-traffic scenarios, enable cluster mode: + +```javascript +// In ecosystem.config.cjs +instances: 'max', // or specify number +exec_mode: 'cluster', +``` + +Then restart: +```bash +pm2 reload redmine-gitea-sync +``` + +### Security Best Practices + +```bash +# Run PM2 as dedicated user +sudo useradd -r -s /bin/false pm2user +sudo chown -R pm2user:pm2user /opt/redmine-gitea-sync + +# Start PM2 as that user +sudo su - pm2user -s /bin/bash -c "pm2 start /opt/redmine-gitea-sync/ecosystem.config.cjs" + +# Save and setup startup +sudo su - pm2user -s /bin/bash -c "pm2 save" +sudo env PATH=$PATH:/usr/bin pm2 startup systemd -u pm2user --hp /home/pm2user +``` + +### Troubleshooting PM2 + +**Issue: Process keeps restarting** +```bash +# Check error logs +pm2 logs redmine-gitea-sync --err + +# Increase restart delay in ecosystem.config.cjs +restart_delay: 5000 +``` + +**Issue: High memory usage** +```bash +# Check memory +pm2 list + +# Adjust memory limit in ecosystem.config.cjs +max_memory_restart: '500M' +``` + +**Issue: PM2 not starting on boot** +```bash +# Regenerate startup script +pm2 unstartup +pm2 startup +pm2 save +``` + +--- + +## Docker Deployment + +### Dockerfile + +Create `Dockerfile`: + +```dockerfile +FROM node:18-alpine + +# Install dependencies for native modules +RUN apk add --no-cache python3 make g++ + +# Create app directory +WORKDIR /app + +# Copy package files +COPY package*.json ./ + +# Install dependencies +RUN npm ci --only=production + +# Copy application files +COPY . . + +# Create non-root user +RUN addgroup -g 1001 -S nodejs && \ + adduser -S nodejs -u 1001 && \ + chown -R nodejs:nodejs /app + +# Switch to non-root user +USER nodejs + +# Expose port +EXPOSE 3002 + +# Health check +HEALTHCHECK --interval=30s --timeout=10s --start-period=40s --retries=3 \ + CMD node -e "require('http').get('http://localhost:3002/health', (res) => { process.exit(res.statusCode === 200 ? 0 : 1); });" + +# Start application +CMD ["node", "server.js"] +``` + +### Docker Compose + +Create `docker-compose.yml`: + +```yaml +version: '3.8' + +services: + sync-server: + build: . + container_name: redmine-gitea-sync + restart: unless-stopped + ports: + - "3002:3002" + environment: + - NODE_ENV=production + - PORT=3002 + env_file: + - .env + volumes: + - ./logs:/app/logs + networks: + - sync-network + logging: + driver: "json-file" + options: + max-size: "10m" + max-file: "3" + +networks: + sync-network: + driver: bridge +``` + +### Build and Run + +```bash +# Build image +docker build -t redmine-gitea-sync . + +# Run with docker-compose +docker-compose up -d + +# View logs +docker-compose logs -f + +# Stop service +docker-compose down +``` + +### Docker Hub Deployment + +```bash +# Tag image +docker tag redmine-gitea-sync:latest your-username/redmine-gitea-sync:latest + +# Push to Docker Hub +docker push your-username/redmine-gitea-sync:latest + +# Pull and run on production server +docker pull your-username/redmine-gitea-sync:latest +docker run -d --name sync-server \ + --env-file .env \ + -p 3002:3002 \ + --restart unless-stopped \ + your-username/redmine-gitea-sync:latest +``` + +--- + +## Systemd Service + +### Create Service File + +Create `/etc/systemd/system/redmine-gitea-sync.service`: + +```ini +[Unit] +Description=Redmine-Gitea Sync Server +After=network.target + +[Service] +Type=simple +User=sync-user +Group=sync-user +WorkingDirectory=/opt/redmine-gitea-sync +EnvironmentFile=/opt/redmine-gitea-sync/.env +ExecStart=/usr/bin/node /opt/redmine-gitea-sync/server.js +Restart=always +RestartSec=10 +StandardOutput=journal +StandardError=journal +SyslogIdentifier=redmine-gitea-sync + +# Security settings +NoNewPrivileges=true +PrivateTmp=true +ProtectSystem=strict +ProtectHome=true +ReadWritePaths=/opt/redmine-gitea-sync/logs + +[Install] +WantedBy=multi-user.target +``` + +### Setup Instructions + +```bash +# Create dedicated user +sudo useradd -r -s /bin/false sync-user + +# Create application directory +sudo mkdir -p /opt/redmine-gitea-sync +sudo chown sync-user:sync-user /opt/redmine-gitea-sync + +# Copy application files +sudo cp -r . /opt/redmine-gitea-sync/ +sudo chown -R sync-user:sync-user /opt/redmine-gitea-sync + +# Install dependencies +cd /opt/redmine-gitea-sync +sudo -u sync-user npm ci --only=production + +# Configure environment +sudo cp .env.example /opt/redmine-gitea-sync/.env +sudo nano /opt/redmine-gitea-sync/.env +sudo chown sync-user:sync-user /opt/redmine-gitea-sync/.env +sudo chmod 600 /opt/redmine-gitea-sync/.env + +# Enable and start service +sudo systemctl daemon-reload +sudo systemctl enable redmine-gitea-sync +sudo systemctl start redmine-gitea-sync + +# Check status +sudo systemctl status redmine-gitea-sync + +# View logs +sudo journalctl -u redmine-gitea-sync -f +``` + +### Service Management + +```bash +# Start service +sudo systemctl start redmine-gitea-sync + +# Stop service +sudo systemctl stop redmine-gitea-sync + +# Restart service +sudo systemctl restart redmine-gitea-sync + +# Check status +sudo systemctl status redmine-gitea-sync + +# Enable auto-start +sudo systemctl enable redmine-gitea-sync + +# Disable auto-start +sudo systemctl disable redmine-gitea-sync + +# View logs +sudo journalctl -u redmine-gitea-sync -n 100 -f +``` + +--- + +## Nginx Reverse Proxy + +### Basic Configuration + +Create `/etc/nginx/sites-available/sync-server`: + +```nginx +upstream sync_server { + server 127.0.0.1:3002; + keepalive 64; +} + +server { + listen 80; + server_name sync.example.com; + + # Redirect to HTTPS + return 301 https://$server_name$request_uri; +} + +server { + listen 443 ssl http2; + server_name sync.example.com; + + # SSL Configuration + ssl_certificate /etc/letsencrypt/live/sync.example.com/fullchain.pem; + ssl_certificate_key /etc/letsencrypt/live/sync.example.com/privkey.pem; + ssl_protocols TLSv1.2 TLSv1.3; + ssl_ciphers HIGH:!aNULL:!MD5; + ssl_prefer_server_ciphers on; + + # Security Headers + add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always; + add_header X-Frame-Options "DENY" always; + add_header X-Content-Type-Options "nosniff" always; + add_header X-XSS-Protection "1; mode=block" always; + + # Logging + access_log /var/log/nginx/sync-server-access.log; + error_log /var/log/nginx/sync-server-error.log; + + # Request size limits + client_max_body_size 10M; + + # Proxy settings + location / { + proxy_pass http://sync_server; + proxy_http_version 1.1; + proxy_set_header Upgrade $http_upgrade; + proxy_set_header Connection 'upgrade'; + proxy_set_header Host $host; + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Forwarded-Proto $scheme; + proxy_cache_bypass $http_upgrade; + proxy_read_timeout 300s; + proxy_connect_timeout 75s; + } + + # Health check endpoint (no auth required) + location /health { + proxy_pass http://sync_server; + access_log off; + } +} +``` + +### Enable Configuration + +```bash +# Enable site +sudo ln -s /etc/nginx/sites-available/sync-server /etc/nginx/sites-enabled/ + +# Test configuration +sudo nginx -t + +# Reload Nginx +sudo systemctl reload nginx +``` + +### SSL with Let's Encrypt + +```bash +# Install Certbot +sudo apt install certbot python3-certbot-nginx + +# Obtain certificate +sudo certbot --nginx -d sync.example.com + +# Auto-renewal is configured automatically +# Test renewal +sudo certbot renew --dry-run +``` + +--- + +## Security Hardening + +### Environment Variables + +```bash +# Secure .env file +chmod 600 .env +chown sync-user:sync-user .env + +# Never commit .env to version control +echo ".env" >> .gitignore +``` + +### Firewall Configuration + +```bash +# Allow only necessary ports +sudo ufw default deny incoming +sudo ufw default allow outgoing +sudo ufw allow ssh +sudo ufw allow 80/tcp +sudo ufw allow 443/tcp +sudo ufw enable +``` + +### Rate Limiting (Nginx) + +Add to Nginx configuration: + +```nginx +# Define rate limit zone +limit_req_zone $binary_remote_addr zone=webhook:10m rate=10r/s; + +# Apply to webhook endpoints +location /webhook/ { + limit_req zone=webhook burst=20 nodelay; + proxy_pass http://sync_server; + # ... other proxy settings +} +``` + +### Webhook Signature Verification + +Implement webhook signature verification for production: + +```javascript +// Add to server.js +import crypto from 'crypto'; + +function verifyRedmineSignature(req) { + const signature = req.headers['x-redmine-signature']; + const secret = process.env.REDMINE_WEBHOOK_SECRET; + + if (!signature || !secret) return false; + + const hash = crypto + .createHmac('sha256', secret) + .update(JSON.stringify(req.body)) + .digest('hex'); + + return signature === hash; +} + +// Add verification to webhook endpoint +app.post('/webhook/redmine', (req, res, next) => { + if (!verifyRedmineSignature(req)) { + return res.status(401).json({ error: 'Invalid signature' }); + } + next(); +}, async (req, res) => { + // ... existing handler +}); +``` + +### IP Whitelisting + +```nginx +# In Nginx configuration +geo $allowed_ip { + default 0; + 192.168.1.0/24 1; # Internal network + 10.0.0.0/8 1; # Private network + # Add Redmine and Gitea server IPs +} + +location /webhook/ { + if ($allowed_ip = 0) { + return 403; + } + proxy_pass http://sync_server; +} +``` + +--- + +## Monitoring and Logging + +### Log Rotation + +Create `/etc/logrotate.d/redmine-gitea-sync`: + +``` +/opt/redmine-gitea-sync/logs/*.log { + daily + rotate 14 + compress + delaycompress + notifempty + missingok + create 0640 sync-user sync-user + sharedscripts + postrotate + systemctl reload redmine-gitea-sync > /dev/null 2>&1 || true + endscript +} +``` + +### Health Monitoring + +Simple monitoring script `monitor.sh`: + +```bash +#!/bin/bash + +HEALTH_URL="http://localhost:3002/health" +MAX_RETRIES=3 +RETRY_COUNT=0 + +while [ $RETRY_COUNT -lt $MAX_RETRIES ]; do + RESPONSE=$(curl -s -o /dev/null -w "%{http_code}" $HEALTH_URL) + + if [ "$RESPONSE" -eq 200 ]; then + echo "$(date): Health check passed" + exit 0 + fi + + RETRY_COUNT=$((RETRY_COUNT + 1)) + sleep 5 +done + +echo "$(date): Health check failed after $MAX_RETRIES attempts" +# Send alert (email, Slack, etc.) +exit 1 +``` + +Add to crontab: + +```bash +# Run health check every 5 minutes +*/5 * * * * /opt/redmine-gitea-sync/monitor.sh >> /var/log/sync-health.log 2>&1 +``` + +### Prometheus Metrics + +Install prometheus client: + +```bash +npm install prom-client +``` + +Add metrics endpoint: + +```javascript +import promClient from 'prom-client'; + +// Create metrics +const register = new promClient.Registry(); +const syncCounter = new promClient.Counter({ + name: 'sync_operations_total', + help: 'Total number of sync operations', + labelNames: ['direction', 'status'], + registers: [register] +}); + +const syncDuration = new promClient.Histogram({ + name: 'sync_duration_seconds', + help: 'Duration of sync operations', + labelNames: ['direction'], + registers: [register] +}); + +// Expose metrics endpoint +app.get('/metrics', async (req, res) => { + res.set('Content-Type', register.contentType); + res.end(await register.metrics()); +}); +``` + +--- + +## Backup and Recovery + +### Configuration Backup + +```bash +#!/bin/bash +# backup.sh + +BACKUP_DIR="/backup/sync-server" +DATE=$(date +%Y%m%d-%H%M%S) +APP_DIR="/opt/redmine-gitea-sync" + +mkdir -p $BACKUP_DIR + +# Backup configuration +tar -czf "$BACKUP_DIR/config-$DATE.tar.gz" \ + "$APP_DIR/.env" \ + "$APP_DIR/package.json" \ + "$APP_DIR/package-lock.json" + +# Backup logs +tar -czf "$BACKUP_DIR/logs-$DATE.tar.gz" "$APP_DIR/logs" + +# Remove backups older than 30 days +find $BACKUP_DIR -name "*.tar.gz" -mtime +30 -delete + +echo "Backup completed: $DATE" +``` + +### Disaster Recovery + +```bash +#!/bin/bash +# restore.sh + +BACKUP_FILE=$1 +RESTORE_DIR="/opt/redmine-gitea-sync" + +if [ -z "$BACKUP_FILE" ]; then + echo "Usage: $0 " + exit 1 +fi + +# Stop service +sudo systemctl stop redmine-gitea-sync + +# Restore configuration +tar -xzf "$BACKUP_FILE" -C / + +# Reinstall dependencies +cd $RESTORE_DIR +npm ci --only=production + +# Start service +sudo systemctl start redmine-gitea-sync + +echo "Restore completed" +``` + +--- + +## Performance Tuning + +### Node.js Optimization + +```bash +# Set Node.js memory limit +NODE_OPTIONS="--max-old-space-size=2048" node server.js +``` + +Update systemd service: + +```ini +[Service] +Environment="NODE_ENV=production" +Environment="NODE_OPTIONS=--max-old-space-size=2048" +``` + +### Database Connection Pool + +If using a database for caching (future): + +```javascript +const pool = { + min: 2, + max: 10, + acquireTimeoutMillis: 30000, + idleTimeoutMillis: 30000 +}; +``` + +### HTTP Keep-Alive + +Already configured in axios clients: + +```javascript +const http = require('http'); +const https = require('https'); + +const httpAgent = new http.Agent({ keepAlive: true }); +const httpsAgent = new https.Agent({ keepAlive: true }); + +const client = axios.create({ + httpAgent, + httpsAgent +}); +``` + +### Nginx Tuning + +```nginx +worker_processes auto; +worker_rlimit_nofile 65535; + +events { + worker_connections 4096; + use epoll; + multi_accept on; +} + +http { + # Enable caching + proxy_cache_path /var/cache/nginx levels=1:2 keys_zone=my_cache:10m max_size=1g inactive=60m; + + # Compression + gzip on; + gzip_vary on; + gzip_comp_level 6; + gzip_types text/plain text/css application/json application/javascript; +} +``` + +--- + +## Troubleshooting Production Issues + +### High Memory Usage + +```bash +# Check memory usage +ps aux | grep node + +# Restart service +sudo systemctl restart redmine-gitea-sync + +# Monitor memory +watch -n 5 'ps aux | grep node' +``` + +### Connection Issues + +```bash +# Test Redmine connection +curl -H "X-Redmine-API-Key: YOUR_KEY" https://redmine.example.com/issues.json + +# Test Gitea connection +curl -H "Authorization: token YOUR_TOKEN" https://gitea.example.com/api/v1/repos + +# Check DNS resolution +nslookup redmine.example.com +nslookup gitea.example.com +``` + +### Webhook Not Triggering + +```bash +# Check Nginx logs +sudo tail -f /var/log/nginx/sync-server-access.log +sudo tail -f /var/log/nginx/sync-server-error.log + +# Check application logs +sudo journalctl -u redmine-gitea-sync -f + +# Test webhook manually +curl -X POST http://localhost:3002/webhook/redmine \ + -H "Content-Type: application/json" \ + -d @test-payload.json +``` + +### Performance Issues + +```bash +# Check CPU usage +top -p $(pgrep -f "node.*server.js") + +# Check network connections +netstat -an | grep :3002 + +# Analyze slow requests +tail -f /var/log/nginx/sync-server-access.log | awk '{print $4, $7}' +``` + +--- + +## Production Checklist + +- [ ] SSL certificates configured and auto-renewal setup +- [ ] Firewall rules configured +- [ ] Environment variables secured (chmod 600 .env) +- [ ] Service configured to start on boot +- [ ] Log rotation configured +- [ ] Monitoring and alerting setup +- [ ] Backup strategy implemented +- [ ] Health checks configured +- [ ] Webhook signatures verified +- [ ] IP whitelisting configured +- [ ] Documentation updated with production URLs +- [ ] Disaster recovery plan documented +- [ ] Performance testing completed +- [ ] Security audit completed + +--- + +## Support and Maintenance + +### Regular Maintenance Tasks + +**Weekly:** +- Check logs for errors +- Verify webhooks are functioning +- Review sync statistics + +**Monthly:** +- Update dependencies: `npm update` +- Review and rotate logs +- Test disaster recovery procedures +- Check SSL certificate expiration + +**Quarterly:** +- Security audit +- Performance review +- Update documentation +- Backup verification + +### Getting Help + +For production issues: +1. Check logs: `sudo journalctl -u redmine-gitea-sync -n 100` +2. Verify configuration: `GET /status` +3. Test connectivity to Redmine and Gitea +4. Review recent changes +5. Consult documentation +6. Open issue in repository with logs and configuration details \ No newline at end of file