Outbound REST Retry Circuit Breaker Pattern

advancedTested on Xanadu

The Outbound REST Retry and Circuit Breaker pattern provides resilient integration capabilities when making external API calls from ServiceNow. This pattern combines exponential backoff retry logic with circuit breaker functionality to handle transient failures, rate limiting (429), and service unavailability (503) responses gracefully.

The circuit breaker monitors failure rates and can temporarily stop making calls to a failing service, allowing it time to recover while preventing resource exhaustion in ServiceNow. The retry mechanism implements exponential backoff with jitter to avoid thundering herd problems when multiple ServiceNow instances or threads attempt to retry simultaneously.

This pattern maintains state information about external service health and automatically transitions between closed (normal operation), open (failing fast), and half-open (testing recovery) states. It logs detailed metrics for monitoring and provides configurable thresholds for failure tolerance, timeout periods, and retry attempts.

Chrome Extension

Sourdough: ServiceNow Monitoring and Analytics

A Chrome extension for ServiceNow Admins and Developers with essential tools, analytics, graphs and monitoring features.

Instance HealthGraphs & ChartsAPI HealthDeveloper ToolsQuick SearchInstance Switcher

Add to Chrome

Free to install. Pro $5/month after a 14-day no-card trial.
Pro requires the ServiceNow admin role. Upgrade inside the extension.

Overview

Tasks

CMDB

API

Metrics

Monitor

Internals

Instance:sourdoughdev·Version:Yokohama

Instance StateONLINE

System StatusFully Operational

Session Timeout90 minutes

Logged-In Sessions2 (20 active)

Build Nameyokohama-12-18-2024_p1

IP Address10.159.128.43

Instance HealthHealth Score: 90%

🔥 5dSourdough (Chrome Plugin)Dark Mode

Use This When

Use this pattern when integrating with external APIs that may experience intermittent failures, rate limiting, or temporary outages, especially in high-volume integration scenarios. It's essential for mission-critical integrations where service availability directly impacts business operations and when you need to prevent cascade failures from propagating through your ServiceNow instance.

Avoid When

Avoid this pattern for simple, low-volume integrations where basic error handling suffices, or when integrating with highly reliable internal services that rarely fail. The added complexity and state management overhead can be counterproductive for one-time data migrations or integrations that don't require high availability guarantees.

Implementation

Server-side

var CircuitBreakerREST = Class.create();
CircuitBreakerREST.prototype = {
    initialize: function(serviceName, options) {
        this.serviceName = serviceName;
        this.maxRetries = options.maxRetries || 3;
        this.baseDelayMs = options.baseDelayMs || 1000;
        this.maxDelayMs = options.maxDelayMs || 30000;
        this.circuitTimeout = options.circuitTimeout || 60000; // 1 minute
        this.failureThreshold = options.failureThreshold || 5;
        this.successThreshold = options.successThreshold || 2;
        this.stateProperty = 'circuit_breaker.' + serviceName;
        this.metricsProperty = 'circuit_breaker.metrics.' + serviceName;
    },

    makeRequest: function(restMessage) {
        var state = this._getCircuitState();
        
        if (state.status === 'OPEN') {
            if (gs.now().getNumericValue() < state.nextAttempt) {
                gs.warn('Circuit breaker OPEN for ' + this.serviceName + ', failing fast');
                return { success: false, error: 'Circuit breaker open', httpStatus: 503 };
            }
            this._transitionToHalfOpen();
        }

        return this._executeWithRetry(restMessage);
    },

    _executeWithRetry: function(restMessage) {
        var attempt = 0;
        var lastResponse;
        
        while (attempt <= this.maxRetries) {
            try {
                gs.info('REST call attempt ' + (attempt + 1) + ' for service: ' + this.serviceName);
                var response = restMessage.execute();
                var httpStatus = response.getStatusCode();
                
                // Success case
                if (httpStatus >= 200 && httpStatus < 300) {
                    this._recordSuccess();
                    return {
                        success: true,
                        response: response,
                        httpStatus: httpStatus,
                        body: response.getBody(),
                        attempt: attempt + 1
                    };
                }
                
                // Check if we should retry
                if (this._shouldRetry(httpStatus, attempt)) {
                    this._recordFailure();
                    if (attempt < this.maxRetries) {
                        var delay = this._calculateDelay(attempt);
                        gs.info('Retrying in ' + delay + 'ms for service: ' + this.serviceName);
                        gs.sleep(delay);
                    }
                } else {
                    // Non-retryable error
                    this._recordFailure();
                    break;
                }
                
                lastResponse = {
                    success: false,
                    response: response,
                    httpStatus: httpStatus,
                    body: response.getBody(),
                    attempt: attempt + 1
                };
                
            } catch (e) {
                gs.error('REST call exception for service ' + this.serviceName + ': ' + e.message);
                this._recordFailure();
                lastResponse = {
                    success: false,
                    error: e.message,
                    httpStatus: 0,
                    attempt: attempt + 1
                };
            }
            
            attempt++;
        }
        
        return lastResponse;
    },

    _shouldRetry: function(httpStatus, attempt) {
        if (attempt >= this.maxRetries) return false;
        // Retry on rate limiting, server errors, and timeouts
        return httpStatus === 429 || httpStatus === 503 || httpStatus === 504 || httpStatus >= 500;
    },

    _calculateDelay: function(attempt) {
        // Exponential backoff with jitter
        var exponential = Math.min(this.baseDelayMs * Math.pow(2, attempt), this.maxDelayMs);
        var jitter = Math.random() * 0.1 * exponential; // 10% jitter
        return Math.floor(exponential + jitter);
    },

    _getCircuitState: function() {
        var stateStr = gs.getProperty(this.stateProperty, '{}');
        try {
            return JSON.parse(stateStr);
        } catch (e) {
            return { status: 'CLOSED', failures: 0, successes: 0, nextAttempt: 0 };
        }
    },

    _saveCircuitState: function(state) {
        gs.setProperty(this.stateProperty, JSON.stringify(state));
    },

    _recordSuccess: function() {
        var state = this._getCircuitState();
        state.successes = (state.successes || 0) + 1;
        state.failures = 0; // Reset failure count on success
        
        if (state.status === 'HALF_OPEN' && state.successes >= this.successThreshold) {
            state.status = 'CLOSED';
            state.successes = 0;
            gs.info('Circuit breaker CLOSED for service: ' + this.serviceName);
        }
        
        this._saveCircuitState(state);
    },

    _recordFailure: function() {
        var state = this._getCircuitState();
        state.failures = (state.failures || 0) + 1;
        state.successes = 0;
        
        if (state.status !== 'OPEN' && state.failures >= this.failureThreshold) {
            state.status = 'OPEN';
            state.nextAttempt = gs.now().getNumericValue() + this.circuitTimeout;
            gs.warn('Circuit breaker OPEN for service: ' + this.serviceName);
        }
        
        this._saveCircuitState(state);
    },

    _transitionToHalfOpen: function() {
        var state = this._getCircuitState();
        state.status = 'HALF_OPEN';
        state.successes = 0;
        this._saveCircuitState(state);
        gs.info('Circuit breaker HALF_OPEN for service: ' + this.serviceName);
    },

    getCircuitStatus: function() {
        return this._getCircuitState();
    },

    resetCircuit: function() {
        var state = { status: 'CLOSED', failures: 0, successes: 0, nextAttempt: 0 };
        this._saveCircuitState(state);
        gs.info('Circuit breaker manually reset for service: ' + this.serviceName);
    },

    type: 'CircuitBreakerREST'
};

// Usage example:
var restMessage = new sn_ws.RESTMessageV2('MyExternalService', 'GET');
restMessage.setStringParameterNoEscape('user_id', '12345');

var circuitBreaker = new CircuitBreakerREST('external-user-api', {
    maxRetries: 3,
    baseDelayMs: 1000,
    maxDelayMs: 30000,
    circuitTimeout: 60000,
    failureThreshold: 5,
    successThreshold: 2
});

var result = circuitBreaker.makeRequest(restMessage);
if (result.success) {
    gs.info('API call successful: ' + result.body);
} else {
    gs.error('API call failed after ' + result.attempt + ' attempts: ' + result.error);
}

The code implements a full circuit breaker pattern with three states (CLOSED, OPEN, HALF_OPEN) and exponential backoff retry logic. The circuit breaker uses ServiceNow system properties to persist state across executions and automatically transitions between states based on configurable failure and success thresholds. The retry mechanism includes jitter to prevent thundering herd problems and respects HTTP status codes to determine retry eligibility.

Alternatives

Simple Retry with MID Server Load Balancing

Use when you have multiple MID Servers and want the platform to handle failover automatically. Simpler to implement but provides less granular control over retry behavior and no circuit breaking functionality.

Asynchronous Queue with Scheduled Jobs

Better for high-volume integrations where immediate response isn't required. Provides natural backpressure handling and retry capabilities through job scheduling, but adds complexity in managing queue state and monitoring.

Event-Driven Integration with Business Rules

Ideal for real-time integrations where you can afford to lose some messages or when the external system can handle webhooks. Reduces retry complexity but requires the external system to support event callbacks for reliable delivery.

Test Your Knowledge

Quick 3-question quiz — see how your ServiceNow skills stack up.

Question 1 of 3Performance

A list view on a table with millions of records is slow. Best fix?

Select an answer to continue