Redmine-Gitea-Sync/TROUBLESHOOTING.md

# Troubleshooting Guide

Common issues and their solutions for the Redmine-Gitea Sync Server.

## Table of Contents

- [422 Errors](#422-errors)
- [Connection Issues](#connection-issues)
- [Sync Loop Issues](#sync-loop-issues)
- [Performance Issues](#performance-issues)
- [PM2 Issues](#pm2-issues)

---

## Label and Field Mapping Issues

### Issue: Priority/Tracker Not Syncing from Gitea to Redmine

**Symptoms:**
- Gitea labels like "bug", "High Priority" not mapping to Redmine fields
- All Redmine issues default to Bug/Normal priority

**Cause:** Gitea uses labels for everything, but Redmine has distinct `tracker_id` and `priority_id` fields.

**Solution:** The server now automatically maps labels back to Redmine fields:

**Tracker Mapping:**
- `bug` label → Tracker ID 1 (Bug)
- `feature` label → Tracker ID 2 (Feature)  
- `support` label → Tracker ID 3 (Support)

**Priority Mapping:**
- `Low Priority` → Priority ID 1
- `Normal Priority` → Priority ID 2
- `High Priority` → Priority ID 3
- `Urgent` → Priority ID 4
- `Immediate` → Priority ID 5

**Verification:**
```bash
# Create issue in Gitea with labels "feature" and "High Priority"
# Check Redmine - should show as Feature with High priority

# Update to latest version
git pull
pm2 restart redmine-gitea-sync
```

### Issue: Comments Not Syncing from Redmine to Gitea

**Symptoms:**
- Add note/journal in Redmine
- Comment doesn't appear in Gitea
- Gitea → Redmine comments work fine

**Causes:**
1. Redmine webhook not configured for journal updates
2. Journal has no notes (only field changes)
3. Cache blocking the sync

**Solutions:**

1. **Verify Redmine Webhook Events**
   - Go to: Administration → Settings → Repositories → Webhooks
   - Ensure "Issues updated" is checked
   - Redmine should send journals as part of issue updates

2. **Check Logs**
   ```bash
   pm2 logs redmine-gitea-sync
   
   # After adding a Redmine note, you should see:
   # "Processing Redmine issue #X"
   # "Synced Redmine journal #Y to Gitea comment on issue #Z"
   ```

3. **Verify Journal Has Content**
   - Empty notes (only field changes) won't sync
   - Add actual text to the note/journal

4. **Check Journal Structure**
   ```bash
   # Enable debug logging
   pm2 start ecosystem.config.cjs --env development
   
   # Look for journal processing logs
   pm2 logs redmine-gitea-sync | grep journal
   ```

5. **Clear Cache**
   ```bash
   curl -X POST http://localhost:3002/cache/clear
   ```

**Verification:**
```bash
# 1. Add a note in Redmine with text content
# 2. Watch logs
pm2 logs redmine-gitea-sync

# Should see:
# [INFO] Processing Redmine issue #X
# [INFO] Updating existing Gitea issue #Y
# [INFO] Synced Redmine journal #Z to Gitea comment on issue #Y

# 3. Check Gitea issue - comment should appear with:
# [Redmine Journal #Z]
# **username** commented:
# 
# Your note text
```

---

## 422 Errors

### Issue: "Request failed with status code 422" (Redmine → Gitea)

**Symptoms:**
```
[ERROR] Redmine to Gitea sync failed: Request failed with status code 422
```

**Cause:** Gitea API validation failure. The request payload contains invalid data.

**Common Causes:**

1. **Invalid Label Format**
   - Labels must be set separately using label IDs, not in the issue creation payload
   - Fixed in v2.0.0+

2. **Invalid Date Format**
   - Gitea expects ISO 8601 format: `2025-12-17T00:00:00Z`
   - Fixed in v2.0.0+

3. **Invalid Assignee**
   - Must be a valid Gitea username (string)
   - User must exist in Gitea

4. **Invalid Milestone**
   - Must be a valid milestone ID (number)
   - Milestone must exist in the repository

**Solutions:**

1. **Update to Latest Version**
   ```bash
   git pull
   npm install
   pm2 restart redmine-gitea-sync
   ```

2. **Enable Verbose Logging**
   ```javascript
   // In ecosystem.config.cjs
   env: {
     LOG_LEVEL: 'debug',
     LOG_VERBOSE: 'true',
   }
   ```
   
   Restart and check logs:
   ```bash
   pm2 restart redmine-gitea-sync --update-env
   pm2 logs redmine-gitea-sync
   ```
   
   Look for: `Validation error details: {...}`

3. **Disable Problematic Features Temporarily**
   ```javascript
   // In ecosystem.config.cjs
   env: {
     SYNC_LABELS: 'false',
     SYNC_MILESTONES: 'false',
   }
   ```

4. **Verify Assignee Exists**
   ```bash
   # Test if user exists in Gitea
   curl -H "Authorization: token YOUR_TOKEN" \
     https://gitea.example.com/api/v1/users/username
   ```

5. **Check Date Formats**
   - Ensure Redmine due dates are valid
   - Server will now auto-convert to ISO format

**Verification:**
```bash
# Watch logs in real-time
pm2 logs redmine-gitea-sync

# Create a test issue in Redmine
# Should see: "Successfully synced Redmine issue #X to Gitea issue #Y"
```

### Issue: "Due date is not a valid date" (Gitea → Redmine)

**Symptoms:**
```
[ERROR] Gitea to Redmine sync failed: Request failed with status code 422
[ERROR] Validation error details: {"errors":["Due date is not a valid date"]}
```

**Cause:** Redmine expects dates in `YYYY-MM-DD` format (no time component), but Gitea sends ISO 8601 format with time.

**Solution:** This is automatically fixed in v2.0.0+. The server now:
- Extracts only the date portion from ISO timestamps
- Validates dates before sending to Redmine
- Handles invalid date formats gracefully with warnings

**Manual Fix:**
```bash
git pull
pm2 restart redmine-gitea-sync
```

**Verification:**
```bash
# Create issue in Gitea with due date
# Watch logs - should see successful sync
pm2 logs redmine-gitea-sync

# Check Redmine - due date should appear correctly
```

---

## Connection Issues

### Issue: Cannot Connect to Redmine

**Symptoms:**
```
[ERROR] Redmine connection test failed
```

**Solutions:**

1. **Verify API URL**
   ```bash
   # Test Redmine API
   curl -H "X-Redmine-API-Key: YOUR_KEY" \
     https://redmine.example.com/issues.json
   ```

2. **Check API Key**
   - Log into Redmine
   - Go to: My Account → API access key
   - Verify key matches `REDMINE_API_KEY` in config

3. **Test Network Connectivity**
   ```bash
   # From server, test connection
   telnet redmine.example.com 443
   ping redmine.example.com
   ```

4. **Check Firewall**
   ```bash
   # Ensure outbound HTTPS is allowed
   sudo ufw status
   ```

5. **Verify SSL Certificate**
   ```bash
   # Test SSL
   openssl s_client -connect redmine.example.com:443
   ```

### Issue: Cannot Connect to Gitea

**Symptoms:**
```
[ERROR] Gitea connection test failed
```

**Solutions:**

1. **Verify API URL**
   - Should NOT include `/api/v1`
   - Correct: `https://gitea.example.com`
   - Incorrect: `https://gitea.example.com/api/v1`

2. **Check Token**
   ```bash
   # Test Gitea API
   curl -H "Authorization: token YOUR_TOKEN" \
     https://gitea.example.com/api/v1/user
   ```

3. **Verify Token Permissions**
   - Token must have: `repo`, `write:issue`
   - Regenerate token if needed

4. **Check Owner Name**
   ```bash
   # Verify owner exists
   curl -H "Authorization: token YOUR_TOKEN" \
     https://gitea.example.com/api/v1/users/YOUR_OWNER
   ```

---

## Sync Loop Issues

### Issue: Infinite Sync Loop

**Symptoms:**
- Issues constantly updating
- High CPU usage
- Logs showing rapid back-and-forth updates

**Cause:** Cache not preventing loops properly.

**Solutions:**

1. **Increase Cache TTL**
   ```javascript
   // In ecosystem.config.cjs
   env: {
     CACHE_TTL: '60000',  // 60 seconds
   }
   ```

2. **Clear Cache**
   ```bash
   curl -X POST http://localhost:3002/cache/clear
   ```

3. **Verify Webhook Configuration**
   - Ensure webhooks are not triggering on every change
   - Check webhook logs in Redmine/Gitea

4. **Check for Multiple Instances**
   ```bash
   pm2 list
   # Should show only one instance
   ```

### Issue: Updates Not Syncing

**Symptoms:**
- Changes made but not appearing on other platform
- No errors in logs

**Solutions:**

1. **Check Cache**
   ```bash
   # Cache might be blocking legitimate updates
   curl -X POST http://localhost:3002/cache/clear
   ```

2. **Verify Webhooks Are Firing**
   ```bash
   # Watch logs
   pm2 logs redmine-gitea-sync

   # Make a change in Redmine
   # Should see: "Processing Redmine issue #X"
   ```

3. **Test Webhooks Manually**
   ```bash
   # Test Redmine webhook
   curl -X POST http://localhost:3002/webhook/redmine \
     -H "Content-Type: application/json" \
     -d @test-payload.json

   # Test Gitea webhook
   curl -X POST http://localhost:3002/webhook/gitea \
     -H "Content-Type: application/json" \
     -d @gitea-payload.json
   ```

4. **Check Network Connectivity**
   - Ensure Redmine/Gitea can reach sync server
   - Test from Redmine/Gitea servers:
   ```bash
   curl http://sync-server:3002/health
   ```

---

## Performance Issues

### Issue: High Memory Usage

**Symptoms:**
```bash
pm2 list
# Shows high memory usage (>300MB)
```

**Solutions:**

1. **Increase Memory Limit**
   ```javascript
   // In ecosystem.config.cjs
   max_memory_restart: '500M',
   ```

2. **Check for Memory Leaks**
   ```bash
   pm2 monit
   # Watch memory over time
   ```

3. **Reduce Cache Size**
   ```javascript
   env: {
     CACHE_TTL: '15000',  // Reduce to 15 seconds
   }
   ```

4. **Restart Regularly**
   ```javascript
   // In ecosystem.config.cjs
   cron_restart: '0 3 * * *',  // Daily at 3 AM
   ```

### Issue: Slow Sync Performance

**Symptoms:**
- Sync takes several seconds
- Timeouts occurring

**Solutions:**

1. **Increase Timeouts**
   ```javascript
   env: {
     REDMINE_TIMEOUT: '60000',
     GITEA_TIMEOUT: '60000',
   }
   ```

2. **Reduce Retry Attempts**
   ```javascript
   env: {
     RETRY_ATTEMPTS: '2',
     RETRY_DELAY: '500',
   }
   ```

3. **Check Network Latency**
   ```bash
   # Test latency
   ping redmine.example.com
   ping gitea.example.com
   ```

4. **Disable Heavy Features**
   ```javascript
   env: {
     SYNC_LABELS: 'false',
     SYNC_MILESTONES: 'false',
     SYNC_CUSTOM_FIELDS: 'false',
   }
   ```

---

## PM2 Issues

### Issue: Process Keeps Restarting

**Symptoms:**
```bash
pm2 list
# Shows high restart count
```

**Solutions:**

1. **Check Error Logs**
   ```bash
   pm2 logs redmine-gitea-sync --err --lines 50
   ```

2. **Increase Restart Delay**
   ```javascript
   // In ecosystem.config.cjs
   restart_delay: 5000,
   max_restarts: 5,
   ```

3. **Verify Configuration**
   ```bash
   # Test configuration
   node server.js
   # Should start without errors
   ```

### Issue: PM2 Not Starting on Boot

**Symptoms:**
- Server boots but sync server not running

**Solutions:**

1. **Regenerate Startup Script**
   ```bash
   pm2 unstartup
   pm2 startup
   pm2 save
   ```

2. **Check Systemd Service**
   ```bash
   systemctl status pm2-yourusername
   journalctl -u pm2-yourusername -n 50
   ```

3. **Verify PM2 Save**
   ```bash
   pm2 save
   ls ~/.pm2/dump.pm2
   ```

### Issue: Logs Not Appearing

**Symptoms:**
```bash
pm2 logs redmine-gitea-sync
# Shows no output
```

**Solutions:**

1. **Check Log Directory**
   ```bash
   ls -la ./logs/
   # Should show error.log and out.log
   ```

2. **Create Logs Directory**
   ```bash
   mkdir -p logs
   chmod 755 logs
   pm2 restart redmine-gitea-sync
   ```

3. **Check Permissions**
   ```bash
   # Ensure PM2 user can write to logs
   chown -R $USER:$USER logs/
   ```

4. **Verify PM2 Config**
   ```javascript
   // In ecosystem.config.cjs
   error_file: './logs/error.log',
   out_file: './logs/out.log',
   ```

---

## Webhook Issues

### Issue: Webhook Not Triggering

**Symptoms:**
- No logs when creating/updating issues
- Changes not syncing

**Solutions:**

1. **Verify Webhook URL**
   - Redmine: `http://your-server:3002/webhook/redmine`
   - Gitea: `http://your-server:3002/webhook/gitea`

2. **Check Webhook Status**
   - In Gitea: Repository → Settings → Webhooks
   - Look for delivery history and error messages

3. **Test Webhook Delivery**
   ```bash
   # From Redmine/Gitea server
   curl -X POST http://sync-server:3002/webhook/redmine \
     -H "Content-Type: application/json" \
     -d '{"payload":{"issue":{"id":1,"subject":"Test"}}}'
   ```

4. **Check Firewall**
   ```bash
   # Ensure port 3002 is open
   sudo ufw status
   sudo ufw allow 3002/tcp
   ```

5. **Use Reverse Proxy**
   - Configure Nginx to forward webhooks
   - Use HTTPS for production

---

## Project Mapping Issues

### Issue: Project Not Found

**Symptoms:**
```
[WARN] No Gitea repo mapping found for Redmine project xyz
[ERROR] Redmine project xyz not found
```

**Solutions:**

1. **Configure Project Mapping**
   ```javascript
   // In ecosystem.config.cjs
   env: {
     PROJECT_MAPPING: '{"GiteaRepo":"redmine-project-id"}',
   }
   ```

2. **Verify Project Identifiers**
   ```bash
   # Check Redmine project identifier
   curl -H "X-Redmine-API-Key: KEY" \
     https://redmine.example.com/projects/PROJECT_ID.json

   # Check Gitea repository name
   curl -H "Authorization: token TOKEN" \
     https://gitea.example.com/api/v1/repos/OWNER/REPO
   ```

3. **Use Automatic Mapping**
   - Leave `PROJECT_MAPPING` empty
   - Server will auto-convert names

---

## Debug Mode

### Enable Maximum Verbosity

```javascript
// In ecosystem.config.cjs
env: {
  NODE_ENV: 'development',
  LOG_LEVEL: 'debug',
  LOG_VERBOSE: 'true',
}
```

Restart:
```bash
pm2 restart redmine-gitea-sync --update-env
```

Or use development environment:
```bash
pm2 start ecosystem.config.cjs --env development
```

### View Detailed Logs

```bash
# Real-time logs with timestamps
pm2 logs redmine-gitea-sync --timestamp

# Last 200 lines
pm2 logs redmine-gitea-sync --lines 200

# Only errors
pm2 logs redmine-gitea-sync --err

# Follow and filter
pm2 logs redmine-gitea-sync | grep ERROR
```

---

## Getting Help

### Collect Debug Information

```bash
# 1. Check server status
curl http://localhost:3002/status

# 2. Collect logs
pm2 logs redmine-gitea-sync --lines 100 > debug.log

# 3. Check configuration
pm2 show redmine-gitea-sync

# 4. Test connectivity
curl -H "X-Redmine-API-Key: KEY" https://redmine.example.com/issues.json
curl -H "Authorization: token TOKEN" https://gitea.example.com/api/v1/user

# 5. Check process status
pm2 list
pm2 monit
```

### Report an Issue

When reporting issues, include:

1. Server status output
2. Relevant logs (last 50-100 lines)
3. Configuration (with sensitive data removed)
4. Steps to reproduce
5. Expected vs actual behavior

### Additional Resources

- [README.md](README.md) - Full documentation
- [PM2_GUIDE.md](PM2_GUIDE.md) - PM2 management
- [API_DOCUMENTATION.md](API_DOCUMENTATION.md) - API reference
- [DEPLOYMENT.md](DEPLOYMENT.md) - Deployment guide