Webhook Delivery

Zelta Pay uses a robust delivery system to ensure webhook events reach your endpoint. This guide covers delivery mechanisms, retry logic, and failure handling.

Delivery Process

How Delivery Works

1. Event Generation - Event is created and queued
2. Delivery Attempt - HTTP POST to your webhook URL
3. Response Handling - Based on status code, retry or acknowledge
4. Retry Logic - Exponential backoff with jitter
5. Dead Letter Queue - Failed deliveries are moved to DLQ

Delivery Timeline

• Initial delivery: Immediate (within 1 second)
• Retry attempts: Up to 6 attempts over 24 hours
• Final failure: Moved to Dead Letter Queue after 24 hours

Retry Schedule

Retry Attempts

Attempt	Base Delay	Jitter Range	Max Delay
1	5s	2.5s - 7.5s	7.5s
2	10s	5s - 15s	15s
3	20s	10s - 30s	30s
4	40s	20s - 60s	60s
5	80s	40s - 120s	120s
6	160s	80s - 240s	240s

Jitter Implementation

Jitter is added to prevent thundering herd problems:

function calculateRetryDelay(attempt) {
  const baseDelay = Math.pow(2, attempt) * 5; // 5s, 10s, 20s, 40s, 80s, 160s
  const jitter = Math.random() * baseDelay; // 0% to 100% of base delay
  return Math.min(baseDelay + jitter, 240000); // Cap at 4 minutes
}

Response Handling

Success Responses (2xx)

These status codes indicate successful delivery:

• 200 OK
• 201 Created
• 202 Accepted
• 204 No Content

Action: Event is acknowledged and removed from the queue.

Retryable Errors

These status codes trigger retry logic:

• 408 Request Timeout
• 429 Too Many Requests
• 5xx Server Errors (500, 502, 503, 504)

Action: Event is retried with exponential backoff.

Non-Retryable Errors

These status codes do not trigger retries:

• 4xx Client Errors (except 408, 429)

Action: Event is moved to Dead Letter Queue (DLQ).

Dead Letter Queue (DLQ)

What is DLQ?

The Dead Letter Queue stores events that failed to deliver after all retry attempts.

DLQ Events

Events are moved to DLQ when:

• All 6 retry attempts failed
• Non-retryable error received (4xx, except 408, 429)
• 24 hours have passed since first delivery attempt

DLQ Management

• Retention: Events stored for 30 days
• Access: View DLQ events in dashboard
• Manual Retry: Retry individual events from dashboard
• Bulk Retry: Retry all events in DLQ

Delivery Headers

Standard Headers

Every webhook delivery includes these headers:

Header	Description	Example
Content-Type	Always application/json	application/json
User-Agent	Zelta Pay webhook identifier	zeltapay-webhook/1.0
Zeltapay-Event-Id	Unique event identifier	evt_1234567890abcdef
Zeltapay-Event-Type	Event type	payment.success
Zeltapay-Timestamp	Unix timestamp	1640995200
Zeltapay-Signature	HMAC signature	t=1640995200, v1=abc123...

Delivery-Specific Headers

Header	Description	Example
Zeltapay-Delivery-Attempt	Current attempt number	1, 2, 3, etc.
Zeltapay-Delivery-Id	Unique delivery identifier	del_1234567890abcdef
Zeltapay-Retry-After	Seconds until next retry	300 (5 minutes)

Implementation Examples

Basic Webhook Handler

Node.js Handler

app.post('/webhook', express.raw({ type: 'application/json' }), (req, res) => {
  const deliveryAttempt = req.headers['zeltapay-delivery-attempt'];
  const eventId = req.headers['zeltapay-event-id'];

  console.log(`Delivery attempt ${deliveryAttempt} for event ${eventId}`);

  try {
    // Process webhook
    const payload = JSON.parse(req.body);
    processWebhookEvent(payload);

    // Success response
    res.status(200).json({ received: true });
  } catch (error) {
    console.error('Webhook processing error:', error);

    // Return 5xx for retryable errors
    if (error.message.includes('database') || error.message.includes('network')) {
      return res.status(500).json({ error: 'Internal server error' });
    }

    // Return 4xx for non-retryable errors
    return res.status(400).json({ error: 'Bad request' });
  }
});

Cloudflare Workers Handler

export default {
  async fetch(request, env, ctx) {
    if (request.method !== 'POST') {
      return new Response('Method not allowed', { status: 405 });
    }

    const url = new URL(request.url);
    if (url.pathname !== '/webhook') {
      return new Response('Not found', { status: 404 });
    }

    try {
      const deliveryAttempt = request.headers.get('zeltapay-delivery-attempt');
      const eventId = request.headers.get('zeltapay-event-id');

      console.log(`Delivery attempt ${deliveryAttempt} for event ${eventId}`);

      // Process webhook
      const payload = await request.json();
      await processWebhookEvent(payload);

      // Success response
      return new Response(JSON.stringify({ received: true }), {
        status: 200,
        headers: { 'Content-Type': 'application/json' }
      });

    } catch (error) {
      console.error('Webhook processing error:', error);

      // Return 5xx for retryable errors
      if (error.message.includes('database') || error.message.includes('network')) {
        return new Response(JSON.stringify({ error: 'Internal server error' }), {
          status: 500,
          headers: { 'Content-Type': 'application/json' }
        });
      }

      // Return 4xx for non-retryable errors
      return new Response(JSON.stringify({ error: 'Bad request' }), {
        status: 400,
        headers: { 'Content-Type': 'application/json' }
      });
    }
  }
};

async function processWebhookEvent(payload) {
  // Process the webhook event
  console.log('Processing webhook event:', payload.type);
}

Hono Handler

import { Hono } from 'hono';
import { cors } from 'hono/cors';

type Bindings = {
  WEBHOOK_SECRET: string;
};

const app = new Hono<{ Bindings: Bindings }>();

// Enable CORS
app.use('*', cors());

app.post('/webhook', async (c) => {
  try {
    const deliveryAttempt = c.req.header('zeltapay-delivery-attempt');
    const eventId = c.req.header('zeltapay-event-id');

    console.log(`Delivery attempt ${deliveryAttempt} for event ${eventId}`);

    // Process webhook
    const payload = await c.req.json();
    await processWebhookEvent(payload);

    // Success response
    return c.json({ received: true });

  } catch (error) {
    console.error('Webhook processing error:', error);

    // Return 5xx for retryable errors
    if (error.message.includes('database') || error.message.includes('network')) {
      return c.json({ error: 'Internal server error' }, 500);
    }

    // Return 4xx for non-retryable errors
    return c.json({ error: 'Bad request' }, 400);
  }
});

async function processWebhookEvent(payload: any) {
  // Process the webhook event
  console.log('Processing webhook event:', payload.type);
}

export default app;

Advanced Webhook Handler

Node.js Advanced Handler

class WebhookHandler {
  constructor() {
    this.processedEvents = new Set();
    this.retryCounts = new Map();
  }

  async handleWebhook(req, res) {
    const eventId = req.headers['zeltapay-event-id'];
    const deliveryAttempt = parseInt(req.headers['zeltapay-delivery-attempt'] || '1');

    // Check for duplicate events
    if (this.processedEvents.has(eventId)) {
      console.log(`Duplicate event ignored: ${eventId}`);
      return res.status(200).json({ received: true });
    }

    try {
      // Verify signature
      if (!this.verifySignature(req)) {
        return res.status(401).json({ error: 'Invalid signature' });
      }

      // Process event
      const payload = JSON.parse(req.body);
      await this.processEvent(payload);

      // Mark as processed
      this.processedEvents.add(eventId);

      // Success response
      res.status(200).json({ received: true });

    } catch (error) {
      console.error(`Error processing event ${eventId}:`, error);

      // Determine if error is retryable
      if (this.isRetryableError(error)) {
        // Return 5xx for retryable errors
        return res.status(500).json({ error: 'Internal server error' });
      } else {
        // Return 4xx for non-retryable errors
        return res.status(400).json({ error: 'Bad request' });
      }
    }
  }

  isRetryableError(error) {
    const retryableErrors = [
      'database connection',
      'network timeout',
      'service unavailable',
      'rate limit'
    ];

    return retryableErrors.some(keyword =>
      error.message.toLowerCase().includes(keyword)
    );
  }

  verifySignature(req) {
    // Implementation details in signature verification guide
    return true; // Placeholder
  }

  async processEvent(payload) {
    // Process the webhook event
    console.log('Processing event:', payload.type);
  }
}

Cloudflare Workers Advanced Handler

export default {
  async fetch(request, env, ctx) {
    if (request.method !== 'POST') {
      return new Response('Method not allowed', { status: 405 });
    }

    const url = new URL(request.url);
    if (url.pathname !== '/webhook') {
      return new Response('Not found', { status: 404 });
    }

    try {
      const eventId = request.headers.get('zeltapay-event-id');
      const deliveryAttempt = parseInt(request.headers.get('zeltapay-delivery-attempt') || '1');

      // Check for duplicate events using KV store
      const existingEvent = await env.WEBHOOK_EVENTS.get(eventId);
      if (existingEvent) {
        console.log(`Duplicate event ignored: ${eventId}`);
        return new Response(JSON.stringify({ received: true }), {
          status: 200,
          headers: { 'Content-Type': 'application/json' }
        });
      }

      // Verify signature
      if (!await this.verifySignature(request, env.WEBHOOK_SECRET)) {
        return new Response(JSON.stringify({ error: 'Invalid signature' }), {
          status: 401,
          headers: { 'Content-Type': 'application/json' }
        });
      }

      // Process event
      const payload = await request.json();
      await this.processEvent(payload, env);

      // Mark as processed (store in KV with 24h TTL)
      await env.WEBHOOK_EVENTS.put(eventId, 'processed', { expirationTtl: 86400 });

      // Success response
      return new Response(JSON.stringify({ received: true }), {
        status: 200,
        headers: { 'Content-Type': 'application/json' }
      });

    } catch (error) {
      console.error(`Error processing event:`, error);

      // Determine if error is retryable
      if (this.isRetryableError(error)) {
        // Return 5xx for retryable errors
        return new Response(JSON.stringify({ error: 'Internal server error' }), {
          status: 500,
          headers: { 'Content-Type': 'application/json' }
        });
      } else {
        // Return 4xx for non-retryable errors
        return new Response(JSON.stringify({ error: 'Bad request' }), {
          status: 400,
          headers: { 'Content-Type': 'application/json' }
        });
      }
    }
  },

  async verifySignature(request, secret) {
    // Implementation details in signature verification guide
    return true; // Placeholder
  },

  isRetryableError(error) {
    const retryableErrors = [
      'database connection',
      'network timeout',
      'service unavailable',
      'rate limit'
    ];

    return retryableErrors.some(keyword =>
      error.message.toLowerCase().includes(keyword)
    );
  },

  async processEvent(payload, env) {
    // Process the webhook event
    console.log('Processing event:', payload.type);
  }
};

Monitoring and Alerting

Delivery Metrics

Monitor these key metrics:

• Delivery Success Rate: Percentage of successful deliveries
• Average Delivery Time: Time from event creation to successful delivery
• Retry Rate: Percentage of events that required retries
• DLQ Rate: Percentage of events moved to Dead Letter Queue

Alerting Rules

Set up alerts for:

• High DLQ Rate: > 5% of events in DLQ
• Low Success Rate: < 95% delivery success rate
• High Retry Rate: > 20% of events require retries
• Delivery Timeout: Events taking > 5 minutes to deliver

Monitoring Implementation

Node.js Monitoring

class WebhookMonitor {
  constructor() {
    this.metrics = {
      total: 0,
      success: 0,
      failure: 0,
      retry: 0,
      dlq: 0
    };
  }

  recordDelivery(attempt, success) {
    this.metrics.total++;

    if (success) {
      this.metrics.success++;
    } else {
      this.metrics.failure++;
    }

    if (attempt > 1) {
      this.metrics.retry++;
    }

    this.logMetrics();
  }

  logMetrics() {
    const successRate = (this.metrics.success / this.metrics.total) * 100;
    const retryRate = (this.metrics.retry / this.metrics.total) * 100;
    const dlqRate = (this.metrics.dlq / this.metrics.total) * 100;

    console.log('Webhook Delivery Metrics:', {
      total: this.metrics.total,
      successRate: `${successRate.toFixed(2)}%`,
      retryRate: `${retryRate.toFixed(2)}%`,
      dlqRate: `${dlqRate.toFixed(2)}%`,
      failures: this.metrics.failure
    });
  }
}

Cloudflare Workers Monitoring

export default {
  async fetch(request, env, ctx) {
    if (request.method === 'POST' && new URL(request.url).pathname === '/webhook') {
      const deliveryAttempt = parseInt(request.headers.get('zeltapay-delivery-attempt') || '1');
      const eventId = request.headers.get('zeltapay-event-id');

      try {
        // Process webhook
        const payload = await request.json();
        await processWebhookEvent(payload);

        // Record successful delivery
        await recordDeliveryMetrics(env, eventId, deliveryAttempt, true);

        return new Response(JSON.stringify({ received: true }), {
          status: 200,
          headers: { 'Content-Type': 'application/json' }
        });

      } catch (error) {
        // Record failed delivery
        await recordDeliveryMetrics(env, eventId, deliveryAttempt, false);

        console.error('Webhook processing error:', error);
        return new Response(JSON.stringify({ error: 'Processing failed' }), {
          status: 500,
          headers: { 'Content-Type': 'application/json' }
        });
      }
    }

    return new Response('Not found', { status: 404 });
  }
};

async function recordDeliveryMetrics(env, eventId, attempt, success) {
  const metricsKey = `webhook_metrics_${new Date().toISOString().split('T')[0]}`;

  // Get existing metrics
  const existingMetrics = await env.WEBHOOK_METRICS.get(metricsKey);
  const metrics = existingMetrics ? JSON.parse(existingMetrics) : {
    total: 0,
    success: 0,
    failure: 0,
    retry: 0
  };

  // Update metrics
  metrics.total++;
  if (success) {
    metrics.success++;
  } else {
    metrics.failure++;
  }

  if (attempt > 1) {
    metrics.retry++;
  }

  // Store updated metrics
  await env.WEBHOOK_METRICS.put(metricsKey, JSON.stringify(metrics), { expirationTtl: 86400 * 7 }); // 7 days

  // Log metrics
  const successRate = (metrics.success / metrics.total) * 100;
  const retryRate = (metrics.retry / metrics.total) * 100;

  console.log('Webhook Delivery Metrics:', {
    total: metrics.total,
    successRate: `${successRate.toFixed(2)}%`,
    retryRate: `${retryRate.toFixed(2)}%`,
    failures: metrics.failure
  });
}

async function processWebhookEvent(payload) {
  // Process the webhook event
  console.log('Processing webhook event:', payload.type);
}

Best Practices

1. Respond Quickly

Keep webhook processing fast:

// Node.js - Respond immediately, process asynchronously
app.post('/webhook', (req, res) => {
  // Respond immediately
  res.status(200).json({ received: true });

  // Process asynchronously
  setImmediate(async () => {
    try {
      const payload = JSON.parse(req.body);
      await processWebhookEvent(payload);
    } catch (error) {
      console.error('Async webhook processing error:', error);
    }
  });
});

// Cloudflare Workers - Use background tasks
export default {
  async fetch(request, env, ctx) {
    if (request.method !== 'POST') {
      return new Response('Method not allowed', { status: 405 });
    }

    try {
      const payload = await request.json();

      // Respond immediately
      const response = new Response(JSON.stringify({ received: true }), {
        status: 200,
        headers: { 'Content-Type': 'application/json' }
      });

      // Process in background
      ctx.waitUntil(processWebhookEvent(payload));

      return response;
    } catch (error) {
      return new Response(JSON.stringify({ error: 'Processing failed' }), {
        status: 500,
        headers: { 'Content-Type': 'application/json' }
      });
    }
  }
};

2. Handle Errors Gracefully

// Node.js - Proper error handling
app.post('/webhook', async (req, res) => {
  try {
    // Process webhook
    await processWebhookEvent(req.body);

    res.status(200).json({ received: true });
  } catch (error) {
    console.error('Webhook processing error:', error);

    // Return appropriate status code
    if (error.message.includes('validation')) {
      return res.status(400).json({ error: 'Validation error' });
    }

    if (error.message.includes('database') || error.message.includes('network')) {
      return res.status(500).json({ error: 'Internal server error' });
    }

    return res.status(500).json({ error: 'Unknown error' });
  }
});

3. Implement Circuit Breaker

// Node.js Circuit Breaker
class CircuitBreaker {
  constructor(threshold = 5, timeout = 60000) {
    this.threshold = threshold;
    this.timeout = timeout;
    this.failures = 0;
    this.lastFailureTime = null;
    this.state = 'CLOSED'; // CLOSED, OPEN, HALF_OPEN
  }

  async execute(fn) {
    if (this.state === 'OPEN') {
      if (Date.now() - this.lastFailureTime > this.timeout) {
        this.state = 'HALF_OPEN';
      } else {
        throw new Error('Circuit breaker is OPEN');
      }
    }

    try {
      const result = await fn();
      this.onSuccess();
      return result;
    } catch (error) {
      this.onFailure();
      throw error;
    }
  }

  onSuccess() {
    this.failures = 0;
    this.state = 'CLOSED';
  }

  onFailure() {
    this.failures++;
    this.lastFailureTime = Date.now();

    if (this.failures >= this.threshold) {
      this.state = 'OPEN';
    }
  }
}

4. Use Message Queues

For high-volume webhooks, use message queues:

// Node.js with Bull Queue
const Queue = require('bull');
const webhookQueue = new Queue('webhook processing');

// Add webhook to queue
app.post('/webhook', (req, res) => {
  webhookQueue.add('process', {
    eventId: req.headers['zeltapay-event-id'],
    payload: req.body
  });

  res.status(200).json({ received: true });
});

// Process webhook from queue
webhookQueue.process('process', async (job) => {
  const { eventId, payload } = job.data;

  try {
    await processWebhookEvent(payload);
    console.log(`Event ${eventId} processed successfully`);
  } catch (error) {
    console.error(`Error processing event ${eventId}:`, error);
    throw error; // Will retry based on queue configuration
  }
});

Troubleshooting

Common Issues

Webhook not receiving events

• Check webhook URL is correct and accessible
• Verify HTTPS is enabled
• Check firewall settings
• Test with webhook ping

Events going to DLQ

• Check error handling in webhook endpoint
• Verify signature verification
• Check for timeout issues
• Review retry logic

High retry rates

• Optimize webhook processing time
• Check for intermittent failures
• Review error handling
• Consider using message queues

Debug Mode

Enable debug logging to troubleshoot:

// Node.js debug logging
function debugWebhookDelivery(req) {
  console.log('Webhook delivery debug:', {
    eventId: req.headers['zeltapay-event-id'],
    deliveryAttempt: req.headers['zeltapay-delivery-attempt'],
    deliveryId: req.headers['zeltapay-delivery-id'],
    timestamp: req.headers['zeltapay-timestamp'],
    signature: req.headers['zeltapay-signature'],
    bodyLength: req.body.length,
    timestamp: new Date().toISOString()
  });
}

// Cloudflare Workers debug logging
export default {
  async fetch(request, env, ctx) {
    if (request.method === 'POST' && new URL(request.url).pathname === '/webhook') {
      const eventId = request.headers.get('zeltapay-event-id');
      const deliveryAttempt = request.headers.get('zeltapay-delivery-attempt');
      const deliveryId = request.headers.get('zeltapay-delivery-id');
      const timestamp = request.headers.get('zeltapay-timestamp');
      const signature = request.headers.get('zeltapay-signature');

      console.log('Webhook delivery debug:', {
        eventId,
        deliveryAttempt,
        deliveryId,
        timestamp,
        signature,
        bodyLength: (await request.text()).length,
        userAgent: request.headers.get('user-agent'),
        cfRay: request.headers.get('cf-ray'),
        timestamp: new Date().toISOString()
      });
    }

    return new Response('Not found', { status: 404 });
  }
};

Next Steps

Now that you understand webhook delivery, explore these resources:

• Webhooks Guide - Complete webhook setup guide
• Signature Verification Guide - Complete signature verification implementation
• Webhook Idempotency Guide - Prevent duplicate processing
• Webhook Events API - Event types and payloads
• Use Cases - See webhooks in action