ServiceNow Datadog Integration Guide

Architecture Overview

Implementation Steps

(function process(request, response) {
    var payload = request.body.data;
    var alertTitle = payload.title || 'Datadog Alert';
    var alertMessage = payload.body || 'Alert triggered from Datadog';
    
    var incident = new GlideRecord('incident');
    incident.initialize();
    incident.short_description = alertTitle;
    incident.description = alertMessage;
    incident.urgency = payload.priority === 'high' ? '1' : '3';
    incident.impact = '2';
    incident.state = '1'; // New
    incident.u_datadog_monitor_id = payload.id;
    var incidentSysId = incident.insert();
    
    response.setStatus(200);
    response.setHeader('Content-Type', 'application/json');
    response.getStreamWriter().writeString(JSON.stringify({status: 'success', incident: incidentSysId}));
})(request, response);

// Flow Script Step - Check if Datadog sync is needed
if (fd_data.trigger.current.state == '6' || fd_data.trigger.current.state == '7') {
    if (!gs.nil(fd_data.trigger.current.u_datadog_monitor_id)) {
        fd_data.datadog_monitor_id = fd_data.trigger.current.u_datadog_monitor_id.toString();
        fd_data.sync_required = true;
    } else {
        fd_data.sync_required = false;
    }
} else {
    fd_data.sync_required = false;
}

// Scheduled Script Execution - Datadog Sync Job
var incident = new GlideRecord('incident');
incident.addNotNullQuery('u_datadog_monitor_id');
incident.addQuery('state', 'IN', '1,2,3'); // Active states
incident.query();

while (incident.next()) {
    try {
        var hub = new sn_ihub.IntegrationHub();
        var response = hub.executeAction('DatadogSpoke', 'Get Monitor Status', {
            monitor_id: incident.u_datadog_monitor_id.toString()
        });
        
        if (response.haveError()) {
            gs.error('Datadog sync failed for incident ' + incident.number + ': ' + response.getErrorMessage());
            continue;
        }
        
        // Process sync logic based on monitor status
        var monitorStatus = response.getBody().status;
        // Add your sync logic here
        
    } catch (e) {
        gs.error('Exception in Datadog sync: ' + e.message);
    }
}

// Business Rule - Auto-assign Datadog incidents
if (current.u_source == 'datadog' && current.isNewRecord()) {
    var tags = current.u_datadog_tags.toString();
    
    if (tags.indexOf('service:database') > -1) {
        current.assignment_group = 'database_team_group_sys_id';
    } else if (tags.indexOf('service:web') > -1) {
        current.assignment_group = 'web_team_group_sys_id';
    } else {
        current.assignment_group = 'default_ops_group_sys_id';
    }
    
    // Set priority based on Datadog alert level
    if (tags.indexOf('alert_type:critical') > -1) {
        current.priority = '1';
    } else if (tags.indexOf('alert_type:warning') > -1) {
        current.priority = '3';
    }
}

// Test script to validate Datadog API connectivity
var request = new sn_ws.RESTMessageV2();
request.setEndpoint('https://api.datadoghq.com/api/v1/validate');
request.setHttpMethod('GET');
request.setRequestHeader('DD-API-KEY', 'your_api_key');
request.setRequestHeader('DD-APPLICATION-KEY', 'your_app_key');

var response = request.execute();
gs.info('Datadog API Test - Status: ' + response.getStatusCode());
gs.info('Response: ' + response.getBody());

if (response.getStatusCode() == 200) {
    gs.info('Datadog API connection successful');
} else {
    gs.error('Datadog API connection failed: ' + response.getErrorMessage());
}

Common Use Cases

Troubleshooting

Pro Tips

• →Implement custom incident correlation logic using Datadog alert tags to automatically link related incidents and prevent alert storms from creating duplicate tickets. Use ServiceNow's Event Management correlation rules combined with Datadog service and environment tags to group related alerts into single incidents, significantly reducing noise during widespread outages.
• →Create dashboard widgets in ServiceNow to display real-time Datadog metrics within incident forms using the Integration Hub spoke's metrics retrieval actions. This provides incident responders with immediate visibility into current system state without switching between platforms, improving resolution efficiency and context awareness.
• →Set up proactive incident prevention by configuring ServiceNow Predictive Intelligence to analyze patterns in Datadog-generated incidents and identify leading indicators of major outages. Use this data to create preventive change requests and maintenance windows before issues escalate to customer impact.
• →Leverage ServiceNow's Flow Designer variables and conditional logic to implement intelligent alert suppression during scheduled maintenance windows by cross-referencing Datadog alerts with ServiceNow change calendar records. This prevents unnecessary incident creation during planned downtime while maintaining visibility of unexpected issues.
• →Implement advanced error handling in your Integration Hub flows using Try-Catch actions and create custom alert notifications for integration failures, ensuring operations teams are immediately aware of monitoring blind spots when the Datadog-ServiceNow integration experiences issues.
• →Use ServiceNow's Performance Analytics to create comprehensive reporting dashboards combining Datadog alert volumes with ServiceNow incident metrics, providing leadership visibility into infrastructure health trends, MTTR improvements, and the business impact of monitoring investments.

Known Limitations

• —Datadog API rate limiting restricts most endpoints to 300 requests per hour per organization, which can impact real-time synchronization for high-volume alert environments requiring careful batch processing and sync scheduling. The Integration Hub Professional license includes execution limits that may constrain complex flows during peak alert periods, necessitating flow optimization and error handling strategies.
• —The ServiceNow Integration Hub Datadog spoke does not support all Datadog API endpoints, particularly newer features like Security Monitoring APIs or advanced Dashboard API operations, requiring custom REST Message records for unsupported functionality. Complex Datadog alert payloads with nested metric data may require custom parsing logic beyond the spoke's standard actions.
• —Webhook delivery reliability depends on network connectivity and ServiceNow instance availability, with no built-in queuing mechanism for failed deliveries during ServiceNow maintenance windows or outages. Personal Developer Instances (PDIs) may hibernate, breaking webhook delivery and requiring higher-tier ServiceNow environments for production integrations.
• —Real-time bidirectional synchronization can introduce latency during high-alert volumes, and the integration lacks native support for Datadog's advanced features like composite monitors, SLO alerts, or forecast monitoring without additional custom development. Large Datadog organizations with thousands of monitors may experience performance impacts when implementing comprehensive sync operations.
• —ServiceNow's Connection & Credential management requires manual API key rotation and lacks automatic key refresh capabilities, creating potential security and operational overhead for organizations with strict key rotation policies. The integration also cannot preserve all Datadog alert metadata, particularly complex metric query contexts and alert evaluation history, within standard ServiceNow incident fields.

Frequently Asked Questions