Webhook Idempotency

Idempotency ensures that processing the same webhook event multiple times produces the same result. This guide covers how to implement idempotency for webhook events.

What is Idempotency?

Idempotency means that performing the same operation multiple times has the same effect as performing it once. For webhooks, this means:

• Duplicate events should not cause duplicate processing
• Retry attempts should not create duplicate side effects
• Race conditions should not cause inconsistent state

Why Idempotency Matters

Common Scenarios

1. Network Issues: Webhook delivery fails, gets retried
2. Server Restarts: Webhook processed before restart, retried after
3. Race Conditions: Multiple webhook deliveries arrive simultaneously
4. Manual Retries: Events manually retried from dashboard

Without Idempotency

// ❌ Bad - Not idempotent
app.post('/webhook', (req, res) => {
  const payload = JSON.parse(req.body);

  if (payload.type === 'payment.success') {
    // This could run multiple times for the same payment!
    await updateOrderStatus(payload.metadata.orderId, 'paid');
    await sendConfirmationEmail(payload.customer.customerEmail);
    await processRefund(payload.metadata.orderId); // Oops!
  }

  res.status(200).json({ received: true });
});

With Idempotency

// ✅ Good - Idempotent
app.post('/webhook', (req, res) => {
  const payload = JSON.parse(req.body);
  const eventId = payload.eventId;

  if (payload.type === 'payment.success') {
    // Check if already processed
    if (await isEventProcessed(eventId)) {
      console.log(`Event ${eventId} already processed`);
      return res.status(200).json({ received: true });
    }

    // Process event atomically
    await processPaymentSuccess(payload);

    // Mark as processed
    await markEventProcessed(eventId);
  }

  res.status(200).json({ received: true });
});

Implementation Strategies

1. Event ID Tracking

Use the eventId field to track processed events:

// Node.js - In-memory tracking
class WebhookIdempotency {
  constructor() {
    this.processedEvents = new Set();
  }

  isEventProcessed(eventId) {
    return this.processedEvents.has(eventId);
  }

  markEventProcessed(eventId) {
    this.processedEvents.add(eventId);
  }

  // For production, use a database
  async isEventProcessedDB(eventId) {
    const result = await db.query(
      'SELECT id FROM processed_events WHERE event_id = ?',
      [eventId]
    );
    return result.length > 0;
  }

  async markEventProcessedDB(eventId) {
    await db.query(
      'INSERT INTO processed_events (event_id, processed_at) VALUES (?, ?)',
      [eventId, new Date()]
    );
  }
}

// Cloudflare Workers - KV store tracking
export default {
  async fetch(request, env, ctx) {
    if (request.method !== 'POST') {
      return new Response('Method not allowed', { status: 405 });
    }

    const url = new URL(request.url);
    if (url.pathname !== '/webhook') {
      return new Response('Not found', { status: 404 });
    }

    try {
      const payload = await request.json();
      const eventId = payload.eventId;

      // Check idempotency using KV store
      const existingEvent = await env.WEBHOOK_EVENTS.get(eventId);
      if (existingEvent) {
        console.log(`Duplicate event ignored: ${eventId}`);
        return new Response(JSON.stringify({ received: true }), {
          status: 200,
          headers: { 'Content-Type': 'application/json' }
        });
      }

      // Process event
      await processEvent(payload, env);

      // Mark as processed (store in KV with 24h TTL)
      await env.WEBHOOK_EVENTS.put(eventId, 'processed', { expirationTtl: 86400 });

      return new Response(JSON.stringify({ received: true }), {
        status: 200,
        headers: { 'Content-Type': 'application/json' }
      });

    } catch (error) {
      console.error('Webhook processing error:', error);
      return new Response(JSON.stringify({ error: 'Processing failed' }), {
        status: 500,
        headers: { 'Content-Type': 'application/json' }
      });
    }
  }
};

async function processEvent(payload, env) {
  switch (payload.type) {
    case 'payment.success':
      await handlePaymentSuccess(payload, env);
      break;
    case 'webhook.ping':
      console.log('Ping received:', payload.message);
      break;
    default:
      console.log('Unknown event type:', payload.type);
  }
}

// Hono with Cloudflare Workers - KV store tracking
import { Hono } from 'hono';
import { cors } from 'hono/cors';

type Bindings = {
  WEBHOOK_SECRET: string;
  WEBHOOK_EVENTS: KVNamespace;
};

const app = new Hono<{ Bindings: Bindings }>();

// Enable CORS
app.use('*', cors());

app.post('/webhook', async (c) => {
  try {
    const payload = await c.req.json();
    const eventId = payload.eventId;

    // Check idempotency using KV store
    const existingEvent = await c.env.WEBHOOK_EVENTS.get(eventId);
    if (existingEvent) {
      console.log(`Duplicate event ignored: ${eventId}`);
      return c.json({ received: true });
    }

    // Process event
    await processEvent(payload, c.env);

    // Mark as processed (store in KV with 24h TTL)
    await c.env.WEBHOOK_EVENTS.put(eventId, 'processed', { expirationTtl: 86400 });

    return c.json({ received: true });

  } catch (error) {
    console.error('Webhook processing error:', error);
    return c.json({ error: 'Processing failed' }, 500);
  }
});

async function processEvent(payload: any, env: Bindings) {
  switch (payload.type) {
    case 'payment.success':
      await handlePaymentSuccess(payload, env);
      break;
    case 'webhook.ping':
      console.log('Ping received:', payload.message);
      break;
    default:
      console.log('Unknown event type:', payload.type);
  }
}

export default app;

2. Database-Level Idempotency

Use database constraints to prevent duplicates:

// Node.js - Database idempotency
async function processPaymentSuccess(payload) {
  const eventId = payload.eventId;
  const orderId = payload.metadata.orderId;

  try {
    // Use database transaction with idempotency
    await db.transaction(async (trx) => {
      // Check if order is already processed
      const existingOrder = await trx('orders')
        .where('id', orderId)
        .first();

      if (existingOrder && existingOrder.status === 'paid') {
        console.log(`Order ${orderId} already processed`);
        return;
      }

      // Insert idempotency record (will fail if duplicate)
      await trx('webhook_idempotency').insert({
        event_id: eventId,
        event_type: 'payment.success',
        status: 'processing'
      });

      // Update order status
      await trx('orders')
        .where('id', orderId)
        .update({
          status: 'paid',
          paid_at: payload.transaction.paidAt,
          payment_id: payload.transaction.externalPaymentId
        });

      // Send confirmation email
      await sendConfirmationEmail(orderId);
    });
  } catch (error) {
    if (error.code === 'ER_DUP_ENTRY') {
      console.log(`Event ${eventId} already processed`);
      return;
    }
    throw error;
  }
}

3. Redis-Based Idempotency

Use Redis for fast idempotency checks:

// Node.js - Redis idempotency
const redis = require('redis');
const client = redis.createClient();

class RedisIdempotency {
  constructor() {
    this.client = client;
  }

  async isEventProcessed(eventId) {
    const result = await this.client.get(`webhook:${eventId}`);
    return result !== null;
  }

  async markEventProcessed(eventId, ttl = 86400) { // 24 hours
    await this.client.setex(`webhook:${eventId}`, ttl, 'processed');
  }

  async processEvent(eventId, processor) {
    // Use Redis SET with NX (only if not exists)
    const result = await this.client.set(
      `webhook:${eventId}`,
      'processing',
      'EX',
      86400,
      'NX'
    );

    if (result === 'OK') {
      try {
        await processor();
        await this.client.set(`webhook:${eventId}`, 'completed');
      } catch (error) {
        await this.client.del(`webhook:${eventId}`);
        throw error;
      }
    } else {
      console.log(`Event ${eventId} already processed`);
    }
  }
}

Complete Implementation Examples

Node.js with Express

const express = require('express');
const app = express();

class WebhookHandler {
  constructor() {
    this.processedEvents = new Set();
  }

  async handleWebhook(req, res) {
    try {
      const payload = JSON.parse(req.body);
      const eventId = payload.eventId;

      // Check idempotency
      if (this.processedEvents.has(eventId)) {
        console.log(`Duplicate event ignored: ${eventId}`);
        return res.status(200).json({ received: true });
      }

      // Process event
      await this.processEvent(payload);

      // Mark as processed
      this.processedEvents.add(eventId);

      res.status(200).json({ received: true });

    } catch (error) {
      console.error('Webhook processing error:', error);
      res.status(500).json({ error: 'Processing failed' });
    }
  }

  async processEvent(payload) {
    switch (payload.type) {
      case 'payment.success':
        await this.handlePaymentSuccess(payload);
        break;
      case 'webhook.ping':
        console.log('Ping received:', payload.message);
        break;
      default:
        console.log('Unknown event type:', payload.type);
    }
  }

  async handlePaymentSuccess(payload) {
    const orderId = payload.metadata.orderId;

    // Check if order is already processed
    const order = await this.getOrder(orderId);
    if (order && order.status === 'paid') {
      console.log(`Order ${orderId} already processed`);
      return;
    }

    // Process payment
    await this.updateOrderStatus(orderId, 'paid');
    await this.sendConfirmationEmail(payload.customer.customerEmail);
    await this.triggerFulfillment(orderId);
  }

  async getOrder(orderId) {
    // Implementation depends on your database
    return await db.orders.findByOrderId(orderId);
  }

  async updateOrderStatus(orderId, status) {
    // Implementation depends on your database
    return await db.orders.updateStatus(orderId, status);
  }

  async sendConfirmationEmail(email) {
    // Implementation depends on your email service
    return await emailService.sendConfirmation(email);
  }

  async triggerFulfillment(orderId) {
    // Implementation depends on your fulfillment system
    return await fulfillmentService.processOrder(orderId);
  }
}

const webhookHandler = new WebhookHandler();

app.post('/webhook', express.raw({ type: 'application/json' }), (req, res) => {
  webhookHandler.handleWebhook(req, res);
});

Cloudflare Workers with KV

export default {
  async fetch(request, env, ctx) {
    if (request.method !== 'POST') {
      return new Response('Method not allowed', { status: 405 });
    }

    const url = new URL(request.url);
    if (url.pathname !== '/webhook') {
      return new Response('Not found', { status: 404 });
    }

    try {
      const payload = await request.json();
      const eventId = payload.eventId;

      // Check idempotency using KV store
      const existingEvent = await env.WEBHOOK_EVENTS.get(eventId);
      if (existingEvent) {
        console.log(`Duplicate event ignored: ${eventId}`);
        return new Response(JSON.stringify({ received: true }), {
          status: 200,
          headers: { 'Content-Type': 'application/json' }
        });
      }

      // Process event
      await processEvent(payload, env);

      // Mark as processed (store in KV with 24h TTL)
      await env.WEBHOOK_EVENTS.put(eventId, 'processed', { expirationTtl: 86400 });

      return new Response(JSON.stringify({ received: true }), {
        status: 200,
        headers: { 'Content-Type': 'application/json' }
      });

    } catch (error) {
      console.error('Webhook processing error:', error);
      return new Response(JSON.stringify({ error: 'Processing failed' }), {
        status: 500,
        headers: { 'Content-Type': 'application/json' }
      });
    }
  }
};

async function processEvent(payload, env) {
  switch (payload.type) {
    case 'payment.success':
      await handlePaymentSuccess(payload, env);
      break;
    case 'webhook.ping':
      console.log('Ping received:', payload.message);
      break;
    default:
      console.log('Unknown event type:', payload.type);
  }
}

async function handlePaymentSuccess(payload, env) {
  const orderId = payload.metadata.orderId;

  // Check if order is already processed using KV
  const existingOrder = await env.ORDERS.get(orderId);
  if (existingOrder) {
    const order = JSON.parse(existingOrder);
    if (order.status === 'paid') {
      console.log(`Order ${orderId} already processed`);
      return;
    }
  }

  // Process payment
  const orderData = {
    id: orderId,
    status: 'paid',
    amount: payload.transaction.amount,
    currency: payload.transaction.currency,
    customerEmail: payload.customer.customerEmail,
    paidAt: payload.transaction.paidAt,
    paymentId: payload.transaction.externalPaymentId
  };

  // Store order in KV
  await env.ORDERS.put(orderId, JSON.stringify(orderData), { expirationTtl: 86400 * 30 }); // 30 days

  // Send confirmation email (using external service)
  await sendConfirmationEmail(payload.customer.customerEmail, orderId);

  // Trigger fulfillment (using external service)
  await triggerFulfillment(orderId);
}

async function sendConfirmationEmail(email, orderId) {
  // Implementation depends on your email service
  console.log(`Sending confirmation email to ${email} for order ${orderId}`);
}

async function triggerFulfillment(orderId) {
  // Implementation depends on your fulfillment system
  console.log(`Triggering fulfillment for order ${orderId}`);
}

Hono with Cloudflare Workers

import { Hono } from 'hono';
import { cors } from 'hono/cors';

type Bindings = {
  WEBHOOK_SECRET: string;
  WEBHOOK_EVENTS: KVNamespace;
  ORDERS: KVNamespace;
};

const app = new Hono<{ Bindings: Bindings }>();

// Enable CORS
app.use('*', cors());

app.post('/webhook', async (c) => {
  try {
    const payload = await c.req.json();
    const eventId = payload.eventId;

    // Check idempotency using KV store
    const existingEvent = await c.env.WEBHOOK_EVENTS.get(eventId);
    if (existingEvent) {
      console.log(`Duplicate event ignored: ${eventId}`);
      return c.json({ received: true });
    }

    // Process event
    await processEvent(payload, c.env);

    // Mark as processed (store in KV with 24h TTL)
    await c.env.WEBHOOK_EVENTS.put(eventId, 'processed', { expirationTtl: 86400 });

    return c.json({ received: true });

  } catch (error) {
    console.error('Webhook processing error:', error);
    return c.json({ error: 'Processing failed' }, 500);
  }
});

async function processEvent(payload: any, env: Bindings) {
  switch (payload.type) {
    case 'payment.success':
      await handlePaymentSuccess(payload, env);
      break;
    case 'webhook.ping':
      console.log('Ping received:', payload.message);
      break;
    default:
      console.log('Unknown event type:', payload.type);
  }
}

async function handlePaymentSuccess(payload: any, env: Bindings) {
  const orderId = payload.metadata.orderId;

  // Check if order is already processed using KV
  const existingOrder = await env.ORDERS.get(orderId);
  if (existingOrder) {
    const order = JSON.parse(existingOrder);
    if (order.status === 'paid') {
      console.log(`Order ${orderId} already processed`);
      return;
    }
  }

  // Process payment
  const orderData = {
    id: orderId,
    status: 'paid',
    amount: payload.transaction.amount,
    currency: payload.transaction.currency,
    customerEmail: payload.customer.customerEmail,
    paidAt: payload.transaction.paidAt,
    paymentId: payload.transaction.externalPaymentId
  };

  // Store order in KV
  await env.ORDERS.put(orderId, JSON.stringify(orderData), { expirationTtl: 86400 * 30 }); // 30 days

  // Send confirmation email (using external service)
  await sendConfirmationEmail(payload.customer.customerEmail, orderId);

  // Trigger fulfillment (using external service)
  await triggerFulfillment(orderId);
}

async function sendConfirmationEmail(email: string, orderId: string) {
  // Implementation depends on your email service
  console.log(`Sending confirmation email to ${email} for order ${orderId}`);
}

async function triggerFulfillment(orderId: string) {
  // Implementation depends on your fulfillment system
  console.log(`Triggering fulfillment for order ${orderId}`);
}

export default app;

Python with Flask

from flask import Flask, request, jsonify
import json
import time

app = Flask(__name__)

class WebhookHandler:
    def __init__(self):
        self.processed_events = set()

    def handle_webhook(self):
        try:
            payload = request.get_json()
            event_id = payload['eventId']

            # Check idempotency
            if event_id in self.processed_events:
                print(f"Duplicate event ignored: {event_id}")
                return jsonify({'received': True}), 200

            # Process event
            self.process_event(payload)

            # Mark as processed
            self.processed_events.add(event_id)

            return jsonify({'received': True}), 200

        except Exception as error:
            print(f"Webhook processing error: {error}")
            return jsonify({'error': 'Processing failed'}), 500

    def process_event(self, payload):
        event_type = payload['type']

        if event_type == 'payment.success':
            self.handle_payment_success(payload)
        elif event_type == 'webhook.ping':
            print(f"Ping received: {payload['message']}")
        else:
            print(f"Unknown event type: {event_type}")

    def handle_payment_success(self, payload):
        order_id = payload['metadata']['orderId']

        # Check if order is already processed
        order = self.get_order(order_id)
        if order and order['status'] == 'paid':
            print(f"Order {order_id} already processed")
            return

        # Process payment
        self.update_order_status(order_id, 'paid')
        self.send_confirmation_email(payload['customer']['customerEmail'])
        self.trigger_fulfillment(order_id)

    def get_order(self, order_id):
        # Implementation depends on your database
        pass

    def update_order_status(self, order_id, status):
        # Implementation depends on your database
        pass

    def send_confirmation_email(self, email):
        # Implementation depends on your email service
        pass

    def trigger_fulfillment(self, order_id):
        # Implementation depends on your fulfillment system
        pass

webhook_handler = WebhookHandler()

@app.route('/webhook', methods=['POST'])
def webhook():
    return webhook_handler.handle_webhook()

Database Schema

Idempotency Table

CREATE TABLE webhook_idempotency (
    id BIGINT PRIMARY KEY AUTO_INCREMENT,
    event_id VARCHAR(255) UNIQUE NOT NULL,
    event_type VARCHAR(100) NOT NULL,
    status ENUM('processing', 'completed', 'failed') NOT NULL,
    processed_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    payload JSON,
    INDEX idx_event_id (event_id),
    INDEX idx_processed_at (processed_at)
);

Orders Table

CREATE TABLE orders (
    id VARCHAR(255) PRIMARY KEY,
    status ENUM('pending', 'paid', 'cancelled', 'refunded') NOT NULL,
    amount INT NOT NULL,
    customer_email VARCHAR(255),
    payment_id VARCHAR(255),
    paid_at TIMESTAMP NULL,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
    INDEX idx_status (status),
    INDEX idx_paid_at (paid_at)
);

Best Practices

1. Use Atomic Operations

Ensure idempotency checks and processing are atomic:

// Node.js - Atomic operations
async function processEventAtomically(eventId, processor) {
  return await db.transaction(async (trx) => {
    // Check if already processed
    const existing = await trx('webhook_idempotency')
      .where('event_id', eventId)
      .first();

    if (existing) {
      return { alreadyProcessed: true };
    }

    // Mark as processing
    await trx('webhook_idempotency').insert({
      event_id: eventId,
      status: 'processing'
    });

    // Process event
    await processor(trx);

    // Mark as completed
    await trx('webhook_idempotency')
      .where('event_id', eventId)
      .update({ status: 'completed' });

    return { alreadyProcessed: false };
  });
}

2. Handle Partial Failures

Implement cleanup for failed processing:

// Node.js - Cleanup on failure
async function processEventWithCleanup(eventId, processor) {
  try {
    await db.transaction(async (trx) => {
      // Mark as processing
      await trx('webhook_idempotency').insert({
        event_id: eventId,
        status: 'processing'
      });

      // Process event
      await processor(trx);

      // Mark as completed
      await trx('webhook_idempotency')
        .where('event_id', eventId)
        .update({ status: 'completed' });
    });
  } catch (error) {
    // Mark as failed
    await db('webhook_idempotency')
      .where('event_id', eventId)
      .update({ status: 'failed' });

    throw error;
  }
}

3. Implement Cleanup

Clean up old idempotency records:

// Node.js - Cleanup old events
async function cleanupOldEvents() {
  const thirtyDaysAgo = new Date();
  thirtyDaysAgo.setDate(thirtyDaysAgo.getDate() - 30);

  await db('webhook_idempotency')
    .where('processed_at', '<', thirtyDaysAgo)
    .del();
}

4. Monitor Idempotency

Track idempotency metrics:

// Node.js - Idempotency monitoring
class IdempotencyMonitor {
  constructor() {
    this.metrics = {
      total: 0,
      duplicates: 0,
      processed: 0
    };
  }

  recordEvent(eventId, isDuplicate) {
    this.metrics.total++;

    if (isDuplicate) {
      this.metrics.duplicates++;
    } else {
      this.metrics.processed++;
    }

    this.logMetrics();
  }

  logMetrics() {
    const duplicateRate = (this.metrics.duplicates / this.metrics.total) * 100;

    console.log('Idempotency Metrics:', {
      total: this.metrics.total,
      duplicates: this.metrics.duplicates,
      processed: this.metrics.processed,
      duplicateRate: `${duplicateRate.toFixed(2)}%`
    });
  }
}

Testing Idempotency

Test Duplicate Events

// Node.js - Test duplicate events
async function testIdempotency() {
  const testEvent = {
    eventId: 'test-event-123',
    type: 'payment.success',
    metadata: { orderId: 'ORD-123' }
  };

  // Process first time
  const response1 = await fetch('/webhook', {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify(testEvent)
  });

  // Process second time (should be idempotent)
  const response2 = await fetch('/webhook', {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify(testEvent)
  });

  console.log('First response:', response1.status);
  console.log('Second response:', response2.status);

  // Both should succeed, but second should not process
}

Test Race Conditions

// Node.js - Test race conditions
async function testRaceCondition() {
  const testEvent = {
    eventId: 'race-test-123',
    type: 'payment.success',
    metadata: { orderId: 'ORD-456' }
  };

  // Send multiple requests simultaneously
  const promises = Array(5).fill().map(() =>
    fetch('/webhook', {
      method: 'POST',
      headers: { 'Content-Type': 'application/json' },
      body: JSON.stringify(testEvent)
    })
  );

  const responses = await Promise.all(promises);

  console.log('All responses:', responses.map(r => r.status));
  // All should succeed, but only one should process
}

Troubleshooting

Common Issues

1. Memory Leaks

• Use database instead of in-memory storage
• Implement cleanup for old events
• Set appropriate TTL for Redis keys

2. Race Conditions

• Use database transactions
• Implement proper locking
• Use atomic operations

3. Performance Issues

• Use Redis for fast lookups
• Implement proper indexing
• Consider partitioning for high volume

Debug Mode

Enable debug logging to troubleshoot:

// Node.js - Debug idempotency
function debugIdempotency(eventId, isDuplicate) {
  console.log('Idempotency debug:', {
    eventId,
    isDuplicate,
    timestamp: new Date().toISOString(),
    memoryUsage: process.memoryUsage()
  });
}

// Cloudflare Workers - Debug idempotency
export default {
  async fetch(request, env, ctx) {
    if (request.method === 'POST' && new URL(request.url).pathname === '/webhook') {
      const payload = await request.json();
      const eventId = payload.eventId;

      // Check idempotency
      const existingEvent = await env.WEBHOOK_EVENTS.get(eventId);
      const isDuplicate = !!existingEvent;

      console.log('Idempotency debug:', {
        eventId,
        isDuplicate,
        timestamp: new Date().toISOString(),
        existingEvent
      });
    }

    return new Response('Not found', { status: 404 });
  }
};

Next Steps

Now that you understand webhook idempotency, explore these resources:

• Webhooks Guide - Complete webhook setup guide
• Signature Verification Guide - Complete signature verification implementation
• Webhook Delivery Guide - Delivery mechanisms and retry logic
• Webhook Events API - Event types and payloads
• Use Cases - See webhooks in action