Integration Error Dead Letter Queue Pattern

advancedTested on Xanadu

The Integration Error Dead Letter Queue (DLQ) pattern captures failed integration messages into a persistent store, allowing for systematic retry processing and manual intervention when automated retries fail. This pattern prevents data loss during integration failures and provides visibility into problematic integrations.

This pattern works by intercepting failed integration attempts, storing the original payload and error context in a custom table, implementing exponential backoff retry logic, and providing tooling for operations teams to manually reprocess or analyze failed messages. It includes alerting mechanisms to notify administrators when messages repeatedly fail processing.

The pattern typically involves a scheduled job that processes queued messages, retry counters to prevent infinite loops, and audit trails for troubleshooting integration issues.

Chrome Extension

Sourdough: ServiceNow Monitoring and Analytics

A Chrome extension for ServiceNow Admins and Developers with essential tools, analytics, graphs and monitoring features.

Instance HealthGraphs & ChartsAPI HealthDeveloper ToolsQuick SearchInstance Switcher

Add to Chrome

Free to install. Pro $5/month after a 14-day no-card trial.
Pro requires the ServiceNow admin role. Upgrade inside the extension.

Overview

Tasks

CMDB

API

Metrics

Monitor

Internals

Instance:sourdoughdev·Version:Yokohama

Instance StateONLINE

System StatusFully Operational

Session Timeout90 minutes

Logged-In Sessions2 (20 active)

Build Nameyokohama-12-18-2024_p1

IP Address10.159.128.43

Instance HealthHealth Score: 90%

🔥 5dSourdough (Chrome Plugin)Dark Mode

Use This When

Use this pattern when building critical integrations that cannot afford to lose data during temporary outages or when processing high-volume integrations where occasional failures are expected. This is essential for systems that must guarantee message delivery or provide audit trails for compliance requirements.

Avoid When

Avoid this pattern for simple, low-volume integrations where manual reprocessing would be more effort than the business value warrants. Don't implement this for integrations where message loss is acceptable or when the downstream system provides its own reliable delivery guarantees.

Implementation

Server-side

// Dead Letter Queue Script Include
var IntegrationDLQ = Class.create();
IntegrationDLQ.prototype = {
    initialize: function() {
        this.MAX_RETRIES = 5;
        this.INITIAL_DELAY = 300; // 5 minutes in seconds
    },
    
    // Store failed message in DLQ
    queueFailedMessage: function(integrationName, payload, errorMessage, originalHeaders) {
        var dlqRecord = new GlideRecord('u_integration_dlq');
        dlqRecord.initialize();
        dlqRecord.setValue('u_integration_name', integrationName);
        dlqRecord.setValue('u_payload', JSON.stringify(payload));
        dlqRecord.setValue('u_error_message', errorMessage);
        dlqRecord.setValue('u_headers', JSON.stringify(originalHeaders || {}));
        dlqRecord.setValue('u_retry_count', 0);
        dlqRecord.setValue('u_next_retry', this._calculateNextRetry(0));
        dlqRecord.setValue('u_status', 'queued');
        var sysId = dlqRecord.insert();
        gs.info('Message queued in DLQ: ' + sysId + ' for integration: ' + integrationName);
        return sysId;
    },
    
    // Process messages ready for retry
    processRetryQueue: function() {
        var gr = new GlideRecord('u_integration_dlq');
        gr.addQuery('u_status', 'queued');
        gr.addQuery('u_retry_count', '<', this.MAX_RETRIES);
        gr.addQuery('u_next_retry', '<=', gs.nowDateTime());
        gr.query();
        
        var processed = 0;
        while (gr.next() && processed < 50) { // Limit batch size
            this._retryMessage(gr);
            processed++;
        }
        
        this._alertOnMaxRetries();
        return processed;
    },
    
    // Retry individual message
    _retryMessage: function(dlqRecord) {
        try {
            var integrationName = dlqRecord.getValue('u_integration_name');
            var payload = JSON.parse(dlqRecord.getValue('u_payload'));
            var headers = JSON.parse(dlqRecord.getValue('u_headers') || '{}');
            
            var success = this._sendMessage(integrationName, payload, headers);
            
            if (success) {
                dlqRecord.setValue('u_status', 'processed');
                dlqRecord.setValue('u_processed_on', gs.nowDateTime());
                gs.info('DLQ message processed successfully: ' + dlqRecord.getUniqueValue());
            } else {
                this._incrementRetry(dlqRecord);
            }
            dlqRecord.update();
            
        } catch (e) {
            gs.error('Error processing DLQ message: ' + e.message);
            this._incrementRetry(dlqRecord);
            dlqRecord.update();
        }
    },
    
    // Send message to target system
    _sendMessage: function(integrationName, payload, headers) {
        var rm = new RESTMessageV2();
        rm.setHttpMethod('POST');
        
        // Configure endpoint based on integration name
        var endpoint = this._getEndpointForIntegration(integrationName);
        rm.setEndpoint(endpoint);
        
        // Set headers
        for (var header in headers) {
            rm.setRequestHeader(header, headers[header]);
        }
        
        rm.setRequestBody(JSON.stringify(payload));
        var response = rm.execute();
        
        var statusCode = response.getStatusCode();
        if (statusCode >= 200 && statusCode < 300) {
            return true;
        } else {
            gs.warn('Integration retry failed with status: ' + statusCode + ', body: ' + response.getBody());
            return false;
        }
    },
    
    // Increment retry count with exponential backoff
    _incrementRetry: function(dlqRecord) {
        var retryCount = parseInt(dlqRecord.getValue('u_retry_count')) + 1;
        dlqRecord.setValue('u_retry_count', retryCount);
        
        if (retryCount >= this.MAX_RETRIES) {
            dlqRecord.setValue('u_status', 'failed');
            dlqRecord.setValue('u_failed_on', gs.nowDateTime());
        } else {
            dlqRecord.setValue('u_next_retry', this._calculateNextRetry(retryCount));
        }
    },
    
    // Calculate next retry time with exponential backoff
    _calculateNextRetry: function(retryCount) {
        var delaySeconds = this.INITIAL_DELAY * Math.pow(2, retryCount);
        var nextRetry = new GlideDateTime();
        nextRetry.addSeconds(delaySeconds);
        return nextRetry;
    },
    
    // Get endpoint configuration
    _getEndpointForIntegration: function(integrationName) {
        // This would typically look up configuration from a system property or table
        return gs.getProperty('integration.' + integrationName + '.endpoint', 'https://default.endpoint.com/api');
    },
    
    // Alert on messages that have exceeded max retries
    _alertOnMaxRetries: function() {
        var gr = new GlideRecord('u_integration_dlq');
        gr.addQuery('u_status', 'failed');
        gr.addQuery('u_alerted', false);
        gr.query();
        
        if (gr.getRowCount() > 0) {
            var event = new GlideRecord('sysevent');
            event.initialize();
            event.setValue('name', 'integration.dlq.max_retries_exceeded');
            event.setValue('parm1', gr.getRowCount());
            event.insert();
            
            // Mark as alerted
            while (gr.next()) {
                gr.setValue('u_alerted', true);
                gr.update();
            }
        }
    },
    
    // Manual reprocess method for UI actions
    manualReprocess: function(dlqSysId) {
        var dlqRecord = new GlideRecord('u_integration_dlq');
        if (dlqRecord.get(dlqSysId)) {
            dlqRecord.setValue('u_retry_count', 0);
            dlqRecord.setValue('u_status', 'queued');
            dlqRecord.setValue('u_next_retry', gs.nowDateTime());
            dlqRecord.setValue('u_alerted', false);
            dlqRecord.update();
            return true;
        }
        return false;
    },
    
    type: 'IntegrationDLQ'
};

Client-side

// UI Action client script for manual reprocessing
function reprocessDLQMessage() {
    if (!confirm('Are you sure you want to reprocess this failed integration message?')) {
        return;
    }
    
    var ga = new GlideAjax('IntegrationDLQ');
    ga.addParam('sysparm_name', 'manualReprocess');
    ga.addParam('sysparm_dlq_sys_id', g_form.getUniqueValue());
    ga.getXML(function(response) {
        var answer = response.responseXML.documentElement.getAttribute('answer');
        if (answer === 'true') {
            g_form.addInfoMessage('Message queued for reprocessing');
            g_form.reload();
        } else {
            g_form.addErrorMessage('Failed to queue message for reprocessing');
        }
    });
}

The code implements a complete DLQ system with a custom table to store failed messages, exponential backoff retry logic, and batch processing limits to prevent system overload. Key features include status tracking, automatic alerting when messages exceed maximum retries, and manual reprocessing capabilities through UI actions.

Alternatives

ServiceNow Event Management

Uses built-in event processing for simpler setup but provides less control over retry logic and payload storage. Better for notifications and simple retry scenarios but lacks the audit trail and manual intervention capabilities.

External Message Broker

Leverages dedicated systems like RabbitMQ or Apache Kafka for enterprise-grade message handling with better scalability. Requires additional infrastructure and complexity but provides superior performance and reliability for high-volume integrations.

Synchronous Retry with Circuit Breaker

Implements immediate retry logic within the original transaction using circuit breaker patterns. Simpler to implement but can block processing threads and doesn't provide persistent storage for failed messages, making it unsuitable for critical data.

Test Your Knowledge

Quick 3-question quiz — see how your ServiceNow skills stack up.

Question 1 of 3Performance

A list view on a table with millions of records is slow. Best fix?

Select an answer to continue