From bb51054a3568cf84cbc43d44b1c7a6048a00a45c Mon Sep 17 00:00:00 2001 From: Thomas Scott Date: Wed, 17 Dec 2025 11:45:53 +0000 Subject: [PATCH] Intiial Commit. Signed-off-by: Thomas Scott --- README.md | 465 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 465 insertions(+) create mode 100644 README.md diff --git a/README.md b/README.md new file mode 100644 index 0000000..25f7b37 --- /dev/null +++ b/README.md @@ -0,0 +1,465 @@ +# Redmine-Gitea Sync Server + +A comprehensive bidirectional synchronization server between Redmine and Gitea issue tracking systems. This server maintains real-time synchronization of issues, comments, labels, milestones, and metadata between both platforms. + +**Recommended Process Manager:** PM2 for production deployments with auto-restart, monitoring, and clustering capabilities. + +## Features + +### Core Synchronization +- **Bidirectional sync**: Issues created or updated in either system automatically sync to the other +- **Loop prevention**: Intelligent caching prevents infinite sync loops +- **Issue mapping**: Maintains permanent links between Redmine and Gitea issues +- **Status synchronization**: Maps status states between both systems + +### Advanced Features +- **Comments/Journals**: Preserves complete comment history +- **Labels**: Syncs Redmine trackers, priorities, and categories as Gitea labels +- **Milestones**: Syncs Redmine versions as Gitea milestones +- **Metadata**: Syncs assignees, due dates, start dates, estimated hours +- **Custom fields**: Optionally includes Redmine custom fields in descriptions +- **Retry logic**: Automatic retry with exponential backoff for failed requests +- **Comprehensive logging**: Multiple log levels with optional verbose mode + +## Architecture + +### Sync Flow + +**Redmine → Gitea:** +1. Redmine webhook triggers on issue update +2. Server checks cache to prevent loops +3. Searches for existing Gitea issue by Redmine ID +4. Creates or updates Gitea issue with all metadata +5. Syncs labels, milestones, and comments +6. Adds entry to sync cache + +**Gitea → Redmine:** +1. Gitea webhook triggers on issue update +2. Server checks cache to prevent loops +3. Searches for existing Redmine issue by title or ID +4. Creates or updates Redmine issue with all metadata +5. Syncs comments as journals +6. Adds entry to sync cache + +### Cache Strategy +- In-memory cache with configurable TTL (default: 30 seconds) +- Prevents sync loops while allowing future edits +- Separate caches for issues and comments +- Automatic cleanup on expiry + +### Error Handling +- Automatic retry with exponential backoff +- Comprehensive error logging +- Graceful degradation on non-critical failures +- Connection health checks + +## Installation + +### Prerequisites +- Node.js 18+ +- npm or yarn +- PM2 process manager (recommended) +- Redmine instance with API access +- Gitea instance with API access +- Network connectivity between server and both platforms + +### Setup + +1. Clone the repository: +```bash +git clone +cd redmine-gitea-sync +``` + +2. Install dependencies: +```bash +npm install +``` + +3. Install PM2 globally (if not already installed): +```bash +npm install -g pm2 +``` + +4. Configure environment variables: +```bash +# Edit the PM2 ecosystem config with your credentials +nano ecosystem.config.cjs +``` + +Update the `env` section with your Redmine and Gitea details. + +5. Start the server with PM2: +```bash +# Start the server +pm2 start ecosystem.config.cjs + +# Save PM2 configuration +pm2 save + +# Setup auto-start on system boot +pm2 startup +``` + +For development with auto-reload: +```bash +npm run dev +``` + +### Alternative: Start without PM2 +```bash +npm start +``` + +## Configuration + +### PM2 Configuration (Recommended) + +Edit `ecosystem.config.cjs` with your settings: + +```javascript +env: { + NODE_ENV: 'production', + PORT: 3002, + + // Redmine Configuration + REDMINE_API_URL: 'https://redmine.example.com', + REDMINE_API_KEY: 'your_redmine_api_key', + + // Gitea Configuration + GITEA_API_URL: 'https://gitea.example.com', + GITEA_TOKEN: 'your_gitea_token', + GITEA_OWNER: 'your_gitea_username', + + // Optional configurations + CACHE_TTL: '30000', + SYNC_LABELS: 'true', + SYNC_MILESTONES: 'true', + LOG_LEVEL: 'info', +} +``` + +After updating, restart PM2: +```bash +pm2 restart redmine-gitea-sync --update-env +``` + +### Alternative: .env File Configuration + +Create a `.env` file for environment-based configuration: + +```bash +cp .env.example .env +nano .env +``` + +### Required Environment Variables + +```env +# Server Configuration +PORT=3002 + +# Redmine Configuration +REDMINE_API_URL=https://redmine.example.com +REDMINE_API_KEY=your_redmine_api_key + +# Gitea Configuration +GITEA_API_URL=https://gitea.example.com +GITEA_TOKEN=your_gitea_token +GITEA_OWNER=your_gitea_username +``` + +### Optional Environment Variables + +```env +# Timeouts (milliseconds) +REDMINE_TIMEOUT=30000 +GITEA_TIMEOUT=30000 + +# Cache Configuration +CACHE_TTL=30000 + +# Sync Features (true/false) +SYNC_LABELS=true +SYNC_MILESTONES=true +SYNC_ATTACHMENTS=true +SYNC_CUSTOM_FIELDS=true + +# Retry Configuration +RETRY_ATTEMPTS=3 +RETRY_DELAY=1000 + +# Logging +LOG_LEVEL=info +LOG_VERBOSE=false + +# Project Mapping (JSON) +PROJECT_MAPPING={"AI_Smuggling":"ai-smuggling","MyRepo":"my-project"} +``` + +### Project Mapping + +The `PROJECT_MAPPING` variable allows you to define custom mappings between Gitea repository names and Redmine project identifiers: + +```json +{ + "GiteaRepoName": "redmine-project-identifier", + "AI_Smuggling": "ai-smuggling", + "WebApp": "web-application" +} +``` + +If no mapping is provided, the server will automatically convert repository names: +- Replace underscores with hyphens +- Convert to lowercase +- Remove special characters + +## Webhook Configuration + +### Redmine Webhook Setup + +1. Navigate to: Administration → Settings → Repositories +2. Add a new webhook with URL: `http://your-server:3002/webhook/redmine` +3. Select events to trigger: + - Issues created + - Issues updated +4. Save configuration + +### Gitea Webhook Setup + +1. Navigate to your repository → Settings → Webhooks +2. Add webhook with: + - Target URL: `http://your-server:3002/webhook/gitea` + - HTTP Method: POST + - POST Content Type: application/json + - Trigger on: Issues, Issue Comment +3. Save webhook + +**Security Note**: For production deployments, consider: +- Using HTTPS for webhook URLs +- Implementing webhook signature verification +- Restricting server access via firewall rules + +## API Endpoints + +### Health Check +``` +GET /health +``` +Returns server status and configuration information. + +**Response:** +```json +{ + "status": "ok", + "uptime": 12345.67, + "timestamp": "2025-12-17T01:49:47Z", + "cache": { + "size": 5, + "ttl": 30000 + }, + "config": { + "redmine": { + "url": "https://redmine.example.com", + "connected": true + }, + "gitea": { + "url": "https://gitea.example.com", + "owner": "username", + "connected": true + } + } +} +``` + +### Status Check +``` +GET /status +``` +Tests connectivity to both Redmine and Gitea. + +**Response:** +```json +{ + "server": "running", + "uptime": 12345.67, + "cache": { + "size": 5, + "ttl": 30000 + }, + "connections": { + "redmine": "connected", + "gitea": "connected" + } +} +``` + +### Clear Cache +``` +POST /cache/clear +``` +Manually clears the sync cache. + +**Response:** +```json +{ + "message": "Cache cleared", + "timestamp": "2025-12-17T01:49:47Z" +} +``` + +## Status Mapping + +### Redmine → Gitea +| Redmine Status | Gitea State | +|----------------|-------------| +| New (1) | open | +| In Progress (2) | open | +| Resolved (3) | open | +| Feedback (4) | open | +| Closed (5) | closed | +| Rejected (6) | closed | + +### Gitea → Redmine +| Gitea State | Redmine Status | +|-------------|----------------| +| open | In Progress (2) | +| closed | Closed (5) | + +## Label Mapping + +The server automatically creates and syncs labels based on Redmine metadata: + +- **Trackers**: bug, feature, support +- **Priorities**: Low Priority, Normal Priority, High Priority, Urgent, Immediate +- **Categories**: Prefixed with `category:` + +## Troubleshooting + +### Common Issues + +**Issue: Webhooks not triggering** +- Verify webhook URLs are accessible from Redmine/Gitea servers +- Check firewall rules +- Verify webhook is enabled and event types are selected +- Check server logs for incoming requests + +**Issue: Sync loops occurring** +- Verify cache TTL is sufficient (minimum 30 seconds recommended) +- Check that both webhooks are not triggering simultaneously +- Review logs for cache hit/miss patterns + +**Issue: Issues not syncing** +- Verify API credentials are correct +- Check project mapping configuration +- Ensure projects exist in both systems +- Review logs for error messages + +**Issue: Comments not appearing** +- Verify journal/comment webhooks are enabled +- Check that journal notes are not empty +- Review cache for recent sync entries + +### Debug Mode + +Enable verbose logging for detailed debugging: +```env +LOG_LEVEL=debug +LOG_VERBOSE=true +``` + +This will output: +- All API requests and responses +- Cache operations +- Detailed error stack traces +- Sync decision logic + +### Log Levels + +- `debug`: Detailed information for debugging +- `info`: General operational messages (default) +- `warn`: Warning messages for recoverable issues +- `error`: Error messages for failures + +## Performance Considerations + +### Recommendations +- Deploy server close to both Redmine and Gitea instances +- Use adequate cache TTL to prevent excessive API calls +- Monitor memory usage for large installations +- Consider rate limiting for high-traffic scenarios + +### Scalability +- The server maintains in-memory cache (suitable for single instance) +- For multi-instance deployments, consider Redis for shared cache +- Current implementation handles up to 1000 issues per repository efficiently + +## Security Best Practices + +1. **API Keys**: Store credentials securely, never commit to version control +2. **Network**: Use HTTPS for all webhook URLs +3. **Access Control**: Restrict server access to necessary IPs only +4. **Validation**: Webhook signature verification recommended for production +5. **Monitoring**: Implement logging and monitoring for suspicious activity + +## Development + +### Running Tests +```bash +npm test +``` + +### Code Structure +``` +. +├── server.js # Main server file +├── .env # Environment configuration +├── .env.example # Example configuration +├── package.json # Dependencies +└── README.md # Documentation +``` + +### Contributing +1. Fork the repository +2. Create a feature branch +3. Make your changes +4. Add tests if applicable +5. Submit a pull request + +## License + +Please see License file. + +## Support + +For issues, questions, or contributions: +- Create an issue in the repository + +## Changelog + +### Version 2.0.1 (Latest) +- Fixed label to field mapping (Gitea → Redmine) - tracker and priority now parse correctly from labels +- Fixed Redmine comments not syncing to Gitea - journals now always process on updates +- Fixed date format validation errors - proper conversion between ISO 8601 and YYYY-MM-DD formats +- Fixed label handling 422 errors - labels now set separately with proper ID resolution +- Enhanced error logging - 422 errors now show detailed validation messages +- Improved journal processing with debug logging +- See [FIX_SUMMARY.md](FIX_SUMMARY.md) for detailed information + +### Version 2.0.0 +- Complete rewrite with comprehensive API support +- Added PM2 process manager support with ecosystem config +- Added label synchronization +- Added milestone synchronization +- Enhanced error handling with retry logic +- Improved logging system with multiple levels +- Added health check and status endpoints +- Added custom field support +- Better project mapping configuration +- Complete documentation suite including PM2 management guide + +### Version 1.0.0 +- Initial release +- Basic bidirectional sync +- Loop prevention +- Comment/journal sync \ No newline at end of file