Hybrid Processing Architecture Guide¶
Overview¶
This document explains the hybrid sync/async processing architecture for handling high-volume topup transactions.
The Problem¶
With synchronous processing, each transaction takes 2-5 seconds. If we receive an "avalanche" of transactions:
| Scenario | Sync Mode | Result |
|---|---|---|
| 10 TPS burst | Requests queue up | 30+ second response times |
| 50 TPS sustained | Server overwhelmed | Timeouts, failures |
| 100 TPS spike | System crash | Lost transactions |
The Solution: Hybrid Mode¶
The system now supports three processing modes:
1. SYNC Mode (Default for low volume)¶
- Direct processing, immediate response
- Best for: < 20 concurrent requests
- Response time: 2-5 seconds
- Client gets immediate success/failure
2. ASYNC Mode (For high volume)¶
- Queue-based processing
- Client gets 202 Accepted immediately
- Transaction processed in background
- Client polls for status or receives webhook
Client → POST /topup → Queue Job → 202 Accepted (50ms)
↓
Worker processes
↓
Client polls /status/:id → Result
3. AUTO Mode (Recommended)¶
- Automatically switches between sync and async
- Based on system load metrics
- No client-side changes needed
Configuration¶
Environment Variables¶
# Processing mode: sync, async, or auto
PROCESSING_MODE=auto
# Switch to async when this many requests are active
ASYNC_THRESHOLD=20
# Switch back to sync when requests drop below this
SYNC_THRESHOLD=5
# Queue depth threshold for async switch
QUEUE_ASYNC_THRESHOLD=50
# Response time threshold (ms) - switch to async if exceeded
RESPONSE_TIME_THRESHOLD=5000
# Cooldown before switching back to sync (ms)
MODE_COOLDOWN=30000
# Queue worker concurrency
QUEUE_CONCURRENCY=10
# Queue job timeout (ms)
QUEUE_JOB_TIMEOUT=60000
# Max retry attempts
QUEUE_MAX_RETRIES=3
API Endpoints¶
Main Endpoint (Auto-switches)¶
Response when sync:
{
"success": true,
"processingMode": "sync",
"transaction": { "id": "RLR123456", "status": "SUCCESS" },
"billing": { "deducted_usd": 5.20, "balance_after_usd": 994.80 }
}
Response when async (202 Accepted):
{
"success": true,
"processingMode": "async",
"transaction": { "id": "RLR123456", "status": "QUEUED", "jobId": "123" },
"message": "Transaction queued for processing",
"statusUrl": "/api/v2/topup/status/RLR123456"
}
Force Sync¶
Force Async¶
Check Status¶
Response:
{
"success": true,
"transaction": {
"id": "RLR123456",
"status": "SUCCESS",
"phone": "5527642763",
"amount": 100,
"operatorTransactionId": "120468024",
"provider": "codigoarix",
"createdAt": "2025-12-04T01:00:00Z",
"processedAt": "2025-12-04T01:00:03Z"
},
"job": {
"state": "completed",
"progress": 100
}
}
System Status¶
Response:
{
"success": true,
"mode": {
"configured": "auto",
"current": "sync",
"lastSwitch": "2025-12-04T01:00:00Z"
},
"load": {
"activeRequests": 5,
"requestsPerMinute": 120,
"avgResponseTime": "2500ms"
},
"queue": {
"waiting": 0,
"active": 0,
"completed": 1234,
"failed": 12
}
}
Architecture Diagram¶
┌─────────────────────────────────────┐
│ HYBRID CONTROLLER │
│ │
│ Monitors: │
│ - Active requests │
│ - Queue depth │
│ - Response times │
│ │
│ Decides: SYNC or ASYNC? │
└───────────────┬─────────────────────┘
│
┌───────────────┴───────────────┐
│ │
┌────▼────┐ ┌────▼────┐
│ SYNC │ │ ASYNC │
│ MODE │ │ MODE │
└────┬────┘ └────┬────┘
│ │
│ ▼
│ ┌──────────────────┐
│ │ REDIS QUEUE │
│ │ (Bull) │
│ │ │
│ │ Jobs waiting... │
│ └────────┬─────────┘
│ │
│ ┌──────────────┼──────────────┐
│ │ │ │
│ ┌────▼────┐ ┌────▼────┐ ┌────▼────┐
│ │ Worker │ │ Worker │ │ Worker │
│ │ 1 │ │ 2 │ │ N │
│ └────┬────┘ └────┬────┘ └────┬────┘
│ │ │ │
└──────────────┼──────────────┼──────────────┘
│ │
▼ ▼
┌─────────────────────────────────┐
│ PROVIDER ROUTER │
│ - Codigo Arix │
│ - Telefonica ISO8583 │
│ - TISI │
│ - CSQ │
└─────────────────────────────────┘
Capacity Planning¶
With 10 Workers (QUEUE_CONCURRENCY=10)¶
| Avg Transaction Time | Sustainable TPS | Queue Absorbs Burst Up To |
|---|---|---|
| 2 seconds | 5 TPS | 500 TPS for 10 seconds |
| 3 seconds | 3.3 TPS | 330 TPS for 10 seconds |
| 5 seconds | 2 TPS | 200 TPS for 10 seconds |
Scaling Options¶
- Increase workers: Set
QUEUE_CONCURRENCY=20for 2x throughput - Add Redis replicas: For queue high availability
- Horizontal scaling: Multiple server instances sharing Redis queue
Client Integration¶
For Immediate Response (Sync-compatible clients)¶
const response = await fetch('/api/v2/topup', {
method: 'POST',
headers: {
'Content-Type': 'application/json',
'x-api-key': API_KEY,
'x-customer-id': CUSTOMER_ID
},
body: JSON.stringify({ phone: '5527642763', amount: 100 })
});
const result = await response.json();
if (result.processingMode === 'async') {
// Poll for result
const status = await pollForStatus(result.transaction.id);
} else {
// Immediate result
console.log(result.transaction.status);
}
For High-Volume Clients (Async-first)¶
// Submit transaction
const response = await fetch('/api/v2/topup-async', { ... });
const { transaction } = await response.json();
// Poll for completion
async function pollForStatus(txnId, maxAttempts = 30) {
for (let i = 0; i < maxAttempts; i++) {
const status = await fetch(`/api/v2/topup/status/${txnId}`);
const result = await status.json();
if (['SUCCESS', 'FAILED'].includes(result.transaction.status)) {
return result;
}
await sleep(2000); // Wait 2 seconds between polls
}
throw new Error('Transaction timeout');
}
Monitoring¶
Health Check¶
Queue Metrics (Prometheus format)¶
Real-time Dashboard¶
Troubleshooting¶
Queue not processing¶
- Check Redis connection:
REDIS_URLenv var - Check worker logs:
docker logs worker - Check queue stats:
GET /api/v2/topup/system-status
High queue depth¶
- Increase workers:
QUEUE_CONCURRENCY=20 - Check provider response times
- Consider horizontal scaling
Transactions stuck in QUEUED¶
- Check dead letter queue
- Verify provider credentials
- Check for rate limiting
Files Created¶
queue-processor-enhanced.js- Enhanced queue with multi-provider supportmiddleware/hybrid-mode.js- Auto-switching middlewareroutes/topupRoutesHybrid.js- New hybrid endpoints
Integration Steps¶
- Update
server.jsto use new components:
const enhancedQueueProcessor = require('./queue-processor-enhanced');
const { controller: hybridController } = require('./middleware/hybrid-mode');
const topupRoutesHybrid = require('./routes/topupRoutesHybrid');
// After providerRouter is initialized:
enhancedQueueProcessor.initializeDependencies({
providerRouter,
forexConverter,
routingService,
redisCache,
tlalocService
});
// Mount hybrid routes
app.use('/api/v2', topupRoutesHybrid({
pool,
topupLimiter,
tlalocService,
redisCache,
queueProcessor: enhancedQueueProcessor,
forexConverter,
routingService,
providerRouter,
alertSystem,
idempotencyMiddleware
}));
-
Set environment variables in Railway
-
Test with load:
-
Monitor mode switches in logs