The ServiceNow-Splunk integration creates a powerful monitoring and incident management ecosystem that automatically correlates machine data with IT service management processes. Organizations use this integration to transform Splunk alerts into ServiceNow incidents, enrich Splunk dashboards with ServiceNow operational data, and provide unified visibility across infrastructure monitoring and service delivery. The integration supports bidirectional data flows enabling automatic incident creation from Splunk alerts, real-time ServiceNow data streaming to Splunk for analytics, and Event Management correlation for intelligent alert noise reduction. Primary automation patterns include webhook-triggered incident creation and scheduled data exports, with core functionality residing in the Event Management module and Integration Hub.
Prerequisites
- •ServiceNow Rome or later with Event Management plugin activated
- •Splunk Enterprise 8.0+ or Splunk Cloud with admin access
- •ServiceNow Integration Hub Professional license for spoke-based automation
- •Splunk Add-on for ServiceNow installed from Splunkbase
- •ServiceNow REST API access with evt_mgmt_integration role
- •Network connectivity between ServiceNow instance and Splunk deployment
- •ServiceNow MID Server if Splunk is on-premises behind firewall
Architecture Overview
The integration leverages the ServiceNow Integration Hub Splunk spoke for outbound communications and REST Message records for bidirectional data exchange, with authentication managed through Connection & Credential Aliases storing Splunk API tokens. Data flows bidirectionally with Splunk alerts triggering ServiceNow incident creation via REST callbacks, while ServiceNow streams incident, change, and CMDB data to Splunk through scheduled exports or real-time webhooks. A MID Server is required only when Splunk Enterprise resides on-premises behind corporate firewalls, as ServiceNow needs direct HTTPS connectivity for webhook delivery and API calls. Rate limiting considerations include Splunk's default 100 requests per minute for REST API calls and ServiceNow's Integration Hub throttling of 1000 spoke executions per hour per instance. Authentication credentials are securely stored in ServiceNow's Connection & Credential framework, with Splunk API tokens managed through encrypted credential records.
Sourdough: ServiceNow Monitoring and Analytics
A Chrome extension for ServiceNow Admins and Developers with essential tools, analytics, graphs and monitoring features.
Free to install. Pro $5/month after a 14-day no-card trial.
Pro requires the ServiceNow admin role. Upgrade inside the extension.
Implementation Steps
Install and configure Splunk Add-on for ServiceNow
Navigate to Splunkbase and download the official Splunk Add-on for ServiceNow to your Splunk deployment. Install the add-on through the Splunk Web interface under Apps > Manage Apps > Install app from file, then restart Splunk services. Configure the add-on by navigating to Apps > Splunk Add-on for ServiceNow > Configuration tab and create a new ServiceNow instance connection. Enter your ServiceNow instance URL (including https://), username, and password for a ServiceNow user with evt_mgmt_integration and rest_api_explorer roles. Test the connection to verify authentication and network connectivity before proceeding.
Create Splunk authentication token and ServiceNow credential record
In Splunk, navigate to Settings > Tokens and generate a new authentication token with appropriate permissions for REST API access and incident creation capabilities. Copy the generated token value as it will only be displayed once during creation. In ServiceNow, navigate to Connections & Credentials > Credentials and create a new Basic Auth Credentials record with Name 'Splunk_API_Credential', User name field containing any placeholder value, and Password field containing your Splunk authentication token. Set the credential to be available for Integration Hub connections and save the record.
Configure ServiceNow Connection Alias for Splunk integration
Navigate to Connections & Credentials > Connection & Credential Aliases and create a new alias record with Name 'Splunk_Connection' and Type 'HTTP(S)'. Set the Connection URL to your Splunk instance URL (e.g., https://mysplunk.company.com:8089) and associate it with the credential record created in the previous step. Configure the Connection Timeout to 30 seconds and enable the 'Multiple Credential Sets' option if you plan to use different authentication methods. Test the connection using the Test Connection button to validate network connectivity and authentication before saving the alias record.
Install ServiceNow Integration Hub Splunk spoke
Navigate to the ServiceNow Store and search for the official Splunk spoke application, then request installation through your instance administrator. Once installed, navigate to Integration Hub > Connections and create a new Splunk connection using the Connection Alias created in the previous step. Verify the connection shows as 'Active' status and test the connectivity by creating a simple flow that queries Splunk search results. Configure the connection timeout and retry settings according to your network latency requirements, typically setting timeout to 60 seconds and retry count to 3 attempts.
Configure Event Management rules for Splunk alert processing
Navigate to Event Management > Event Processing > Event Rules and create a new event rule with Name 'Splunk Alert Processing' and Condition targeting events where Source equals 'Splunk'. Configure the rule to set proper Classification (e.g., 'Availability'), assign appropriate Assignment Group, and map Splunk alert fields to ServiceNow incident fields using field mapping transformations. Enable the 'Create Incident' action and configure incident priority mapping based on Splunk alert severity levels (Critical->1-Critical, High->2-High, etc.). Set up proper correlation rules to prevent duplicate incidents from the same Splunk search by using correlation ID based on Splunk search name and triggering conditions.
var eventGr = new GlideRecord('em_event');
eventGr.addQuery('source', 'Splunk');
eventGr.addQuery('state', 'Ready');
eventGr.query();
while (eventGr.next()) {
var incident = new GlideRecord('incident');
incident.initialize();
incident.short_description = eventGr.getValue('message_key');
incident.description = eventGr.getValue('additional_info');
incident.priority = mapSplunkSeverity(eventGr.getValue('severity'));
incident.assignment_group = 'IT Operations';
incident.correlation_id = eventGr.getValue('source_instance') + '_' + eventGr.getValue('event_class');
incident.insert();
}Create Integration Hub flow for bidirectional incident synchronization
Navigate to Integration Hub > Flow Designer and create a new flow triggered by Incident table operations (Insert/Update). Add the Splunk spoke action 'Create Alert' or 'Update Search' depending on your synchronization requirements, and map ServiceNow incident fields to corresponding Splunk event fields using the Data Mapper. Configure the flow to handle error conditions by adding error handling steps that log failures to the system event log and optionally send notifications to administrators. Test the flow using the Test Flow functionality with sample incident data to verify proper field mapping and authentication to Splunk APIs.
// Flow script step for custom field mapping
(function execute(inputs, outputs) {
var incident = inputs.incident;
var splunkPayload = {
'search_name': 'ServiceNow_Incident_' + incident.number,
'description': incident.short_description.toString(),
'severity': mapServiceNowPriority(incident.priority.toString()),
'status': incident.state.toString(),
'assigned_to': incident.assigned_to.getDisplayValue(),
'correlation_id': incident.correlation_id.toString()
};
outputs.splunk_payload = JSON.stringify(splunkPayload);
})(inputs, outputs);Configure ServiceNow data export to Splunk for dashboard analytics
Navigate to System Web Services > REST Message and create a new REST Message record named 'Splunk_Data_Export' with Endpoint URL pointing to your Splunk HTTP Event Collector endpoint. Configure the HTTP method as POST and add necessary headers including 'Authorization' with your Splunk HEC token and 'Content-Type' as 'application/json'. Create a scheduled job using System Definition > Scheduled Jobs that executes every 15 minutes to export incident, change, and CMDB data to Splunk for dashboard consumption. Configure the export script to batch records and handle large datasets by implementing pagination and avoiding memory constraints during data transfer.
var restMessage = new sn_ws.RESTMessageV2('Splunk_Data_Export', 'POST');
restMessage.setStringParameter('endpoint', 'https://mysplunk.company.com:8088/services/collector/event');
restMessage.setRequestHeader('Authorization', 'Splunk ' + gs.getProperty('splunk.hec.token'));
restMessage.setRequestHeader('Content-Type', 'application/json');
var incidents = new GlideRecord('incident');
incidents.addQuery('sys_updated_on', '>=', gs.minutesAgoStart(15));
incidents.query();
var events = [];
while (incidents.next()) {
events.push({
'time': incidents.getValue('sys_updated_on'),
'sourcetype': 'servicenow:incident',
'event': {
'number': incidents.getValue('number'),
'state': incidents.getDisplayValue('state'),
'priority': incidents.getDisplayValue('priority'),
'assignment_group': incidents.getDisplayValue('assignment_group')
}
});
}
restMessage.setRequestBody(JSON.stringify({'events': events}));
var response = restMessage.execute();Test end-to-end integration and configure monitoring
Create a test Splunk search that generates an alert to verify automatic incident creation in ServiceNow through the configured Event Management rules and processing. Validate that incident field mapping is correct, assignment groups are properly set, and correlation rules prevent duplicate incident creation from repeated alerts. Test the reverse data flow by creating or updating a ServiceNow incident and confirming the data appears in your configured Splunk indexes within the expected timeframe. Set up monitoring for the integration by creating ServiceNow reports on Event Management processing failures and Integration Hub execution errors, and configure Splunk dashboards to track ServiceNow data ingestion volumes and processing times.
// Test script for validating integration connectivity
(function testSplunkIntegration() {
var restMessage = new sn_ws.RESTMessageV2();
restMessage.setEndpoint('https://mysplunk.company.com:8089/services/search/jobs/export');
restMessage.setHttpMethod('POST');
restMessage.setRequestHeader('Authorization', 'Bearer ' + gs.getProperty('splunk.api.token'));
restMessage.setRequestHeader('Content-Type', 'application/x-www-form-urlencoded');
restMessage.setRequestBody('search=search index=main | head 1&output_mode=json');
var response = restMessage.execute();
var httpStatus = response.getStatusCode();
gs.info('Splunk connectivity test - HTTP Status: ' + httpStatus);
if (httpStatus == 200) {
gs.info('Splunk integration test successful');
return true;
} else {
gs.error('Splunk integration test failed: ' + response.getBody());
return false;
}
})();Common Use Cases
Automatic incident creation from Splunk infrastructure alerts
Splunk monitoring detects application performance degradation or infrastructure failures and automatically creates prioritized incidents in ServiceNow with proper categorization and assignment. The integration maps Splunk alert severity levels to ServiceNow incident priorities and includes relevant log excerpts and system metrics in the incident description. Event Management correlation rules prevent duplicate incidents from repeated alerts for the same issue. This use case delivers immediate incident response initiation and reduces mean time to detection for critical system issues.
ServiceNow ITSM data analytics in Splunk dashboards
ServiceNow incident, change, and problem management data streams to Splunk for advanced analytics and executive dashboards showing service performance trends and operational metrics. Real-time data synchronization enables Splunk dashboards to display current incident volumes, resolution times, and SLA compliance rates across different service categories. The integration includes CMDB configuration item data to correlate service outages with infrastructure changes and capacity planning. Business stakeholders gain visibility into IT service delivery performance and can identify improvement opportunities through trend analysis.
Security incident escalation from Splunk Enterprise Security
Splunk Enterprise Security notable events automatically create security incidents in ServiceNow with appropriate urgency classification and assignment to security response teams. The integration transfers threat intelligence data, affected user accounts, and compromised systems information from Splunk to ServiceNow for comprehensive incident tracking. Security workflows in ServiceNow trigger additional Splunk searches for threat hunting and evidence collection during incident response. This creates a unified security operations workflow combining detection capabilities with structured incident management processes.
Change management correlation with system monitoring
ServiceNow change records synchronize with Splunk to create monitoring baselines during maintenance windows and correlate post-change system behavior with change implementation timelines. Splunk automatically adjusts alerting thresholds during approved change windows and escalates unexpected system behavior to ServiceNow as change-related incidents. The integration enables automatic change success validation by comparing pre-change and post-change system performance metrics collected in Splunk. Operations teams can quickly identify change-related issues and implement rollback procedures when system performance degrades outside acceptable parameters.
Proactive problem management through log analysis
Splunk machine learning algorithms identify recurring error patterns and system anomalies that automatically create problem records in ServiceNow for root cause analysis before widespread service impact occurs. The integration transfers detailed log analysis results, affected system lists, and trend data from Splunk to support problem investigation workflows in ServiceNow. Problem management teams use combined Splunk analytics and ServiceNow workflow capabilities to track resolution progress and implement permanent fixes for systemic issues. This proactive approach reduces incident volumes and improves overall service reliability by addressing underlying problems before they cause service disruptions.
Troubleshooting
Splunk alerts creating duplicate incidents in ServiceNow despite correlation rules
Check Event Management correlation settings under Event Processing > Event Rules and verify correlation ID field mapping is consistent between Splunk alert payloads and ServiceNow event processing rules. Review the em_event table for events with identical correlation IDs but different source_instance values, which can bypass correlation logic. Modify correlation rules to include additional fields like event_class and node for more precise matching, and consider implementing time-based correlation windows to group related alerts within specific timeframes.
ServiceNow Integration Hub Splunk spoke actions failing with 401 authentication errors
Navigate to Integration Hub > Connections and test your Splunk connection to verify authentication token validity and network connectivity. Check the Connection & Credential Alias configuration to ensure the credential record contains the correct Splunk authentication token in the password field. Review Splunk token permissions by logging into Splunk and verifying the token has not expired and includes necessary capabilities for REST API access and search operations. Update credential records with new tokens and test connections after any Splunk authentication changes.
Large data exports from ServiceNow to Splunk causing timeout errors and incomplete data transfer
Implement pagination in scheduled export jobs by modifying GlideRecord queries to process records in batches of 500-1000 records per execution using setLimit() and addQuery() with sys_id greater than the last processed record. Configure REST Message timeout values to 300 seconds for large payloads and implement retry logic with exponential backoff for failed HTTP requests. Consider using Splunk HTTP Event Collector (HEC) with multiple parallel threads and compress JSON payloads using gzip encoding to reduce transfer time and bandwidth requirements.
Event Management rules not triggering incident creation from Splunk alerts despite proper configuration
Navigate to Event Management > Event Processing > Processing Log to review event processing status and identify rule execution failures or condition matching issues. Verify incoming Splunk alert field names match exactly with Event Rule conditions, as field name case sensitivity can prevent rule triggering. Check the em_event table for events stuck in 'Processing' state and manually process them using the Event Management processor to identify rule logic errors. Review Event Rule order and conditions to ensure higher-priority rules are not preventing subsequent rule execution through early termination logic.
Splunk search results returning empty datasets when querying ServiceNow data indexes
Verify ServiceNow data is reaching Splunk by checking the Splunk Search & Reporting app for recent events in your configured indexes using search commands like 'index=servicenow_data | head 100'. Review ServiceNow scheduled job execution history under System Logs > Scheduled Jobs to confirm data export scripts are running successfully without errors. Check Splunk HTTP Event Collector (HEC) token configuration and verify the token is enabled and associated with the correct index for ServiceNow data ingestion, then validate HEC endpoint connectivity from ServiceNow using REST Message test functionality.
Integration Hub flows executing successfully but Splunk API calls returning 403 forbidden errors
Review Splunk authentication token permissions by logging into Splunk and navigating to Settings > Access Controls > Roles to verify the token's associated user has required capabilities for REST API access and search operations. Check Splunk app-specific permissions if using custom Splunk applications that restrict API access to specific user roles or IP address ranges. Validate that Integration Hub flows are using the correct Splunk REST API endpoints and HTTP methods, as some operations require specific URL patterns and parameter formats that differ from standard web interface access.
Pro Tips
- →Configure Event Management correlation rules with time-based windows using the 'Correlation Timeout' field to automatically close correlation groups after 24 hours, preventing indefinite accumulation of related events and improving system performance. Implement correlation ID generation using combination fields like source_instance + event_class + affected_ci to create unique identifiers that properly group related alerts while avoiding false correlation matches.
- →Use ServiceNow Transform Maps for complex Splunk alert field mapping instead of relying solely on Event Rules, as Transform Maps provide better field transformation capabilities and debugging options for data format conversion. Create custom transform scripts that handle JSON parsing, date format conversion, and null value handling for robust integration processing.
- →Implement Integration Hub error handling with custom notification flows that alert administrators when Splunk API calls fail repeatedly, including details about error codes and affected records for faster troubleshooting. Configure flow execution retry logic with exponential backoff delays to handle temporary network issues without overwhelming Splunk API rate limits.
- →Set up ServiceNow Performance Analytics dashboards to track integration KPIs including Splunk alert processing times, incident creation volumes, and Event Management rule execution statistics for ongoing integration health monitoring. Create automated reports that identify correlation rule effectiveness and highlight opportunities for alert noise reduction.
- →Configure Splunk HTTP Event Collector with dedicated indexes for ServiceNow data types (incidents, changes, CMDB) to improve search performance and enable granular data retention policies based on ServiceNow record lifecycle requirements. Implement Splunk data models for ServiceNow data to accelerate dashboard creation and enable advanced analytics capabilities.
- →Use ServiceNow REST Message response caching for frequently accessed Splunk search results to reduce API call volumes and improve Integration Hub flow performance, especially for lookup operations and reference data synchronization. Implement cache invalidation logic based on data freshness requirements and Splunk search result update timestamps.
Known Limitations
- —Splunk REST API enforces rate limiting of 100 requests per minute per authentication token, which can constrain high-volume bidirectional synchronization scenarios and require careful batch processing design. Integration Hub Professional license limits spoke executions to 1000 per hour per instance, potentially requiring execution throttling for large-scale data synchronization workflows.
- —ServiceNow Event Management correlation rules have limited support for complex logical operators and nested conditions, making it challenging to implement sophisticated alert grouping logic for multi-dimensional Splunk alert scenarios. Real-time data synchronization from ServiceNow to Splunk requires custom scheduled jobs or webhook implementations, as no native real-time streaming capability exists.
- —Large dataset exports from ServiceNow to Splunk can impact instance performance due to GlideRecord query limitations and memory constraints, requiring careful pagination and execution scheduling during off-peak hours. Splunk HTTP Event Collector has payload size limits of 1MB per request, necessitating data chunking for large ServiceNow records with extensive field content.
- —Integration Hub Splunk spoke actions do not support all Splunk REST API endpoints, particularly advanced search management and administrative functions, requiring custom REST Message implementations for full API coverage. Field mapping between Splunk alert structures and ServiceNow incident fields may require custom transformation scripts when alert formats vary significantly across different Splunk searches.
- —Cross-instance data synchronization latency can range from 5-15 minutes depending on network conditions and processing volumes, making the integration unsuitable for real-time alerting scenarios requiring sub-minute response times. Splunk Enterprise Security notable events require additional field mapping configuration as they use different schema structures compared to standard Splunk alerts.
Frequently Asked Questions
Can the ServiceNow-Splunk integration work with Splunk Cloud or only on-premises Splunk Enterprise?
The integration supports both Splunk Cloud and on-premises Splunk Enterprise deployments through REST API connectivity and HTTP Event Collector endpoints. Splunk Cloud users must ensure their ServiceNow instance can reach Splunk Cloud URLs over HTTPS and configure authentication tokens with appropriate permissions for API access. Network connectivity requirements are identical for both deployment models, though Splunk Cloud eliminates the need for MID Server deployment in most scenarios. Authentication token management and API rate limiting apply equally to both Splunk Cloud and Enterprise installations.
How does the integration handle Splunk search results that contain sensitive data or PII when creating ServiceNow incidents?
The integration processes Splunk alert data through ServiceNow Event Management field mapping rules, which can be configured to exclude or mask sensitive fields before incident creation. Implement custom transform scripts in Event Rules to sanitize log excerpts and system information, removing PII while preserving diagnostic value for incident resolution. Configure Splunk alert searches to use field filtering and data masking techniques before sending alerts to ServiceNow, ensuring compliance with data privacy requirements. Review ServiceNow field-level encryption and access controls to protect any sensitive data that must be transferred for incident investigation purposes.
What happens to existing ServiceNow incidents when the corresponding Splunk alerts are resolved or closed?
ServiceNow incidents created from Splunk alerts do not automatically close when Splunk alerts resolve, as the integration follows standard ITIL incident management processes requiring manual verification of issue resolution. Configure Integration Hub flows to monitor Splunk alert status changes and update corresponding ServiceNow incident states through correlation ID matching when alerts transition to resolved status. Implement Event Management rules that process Splunk 'alert cleared' notifications to automatically set ServiceNow incidents to 'Resolved' state while preserving audit trails. Consider implementing scheduled jobs that query Splunk for alert status updates and synchronize incident states based on configurable business rules and approval workflows.
Can multiple ServiceNow instances send data to the same Splunk deployment without data conflicts?
Multiple ServiceNow instances can safely send data to a single Splunk deployment by configuring distinct index names and source types for each instance to prevent data mixing and enable proper data segregation. Configure each ServiceNow instance with unique correlation ID prefixes and source identifiers in their Event Management rules to ensure proper incident tracking and prevent cross-instance correlation conflicts. Implement Splunk index-based access controls and search restrictions to ensure users only access data from authorized ServiceNow instances while maintaining centralized analytics capabilities. Design Splunk dashboards with instance filtering capabilities and configure data retention policies appropriate for each ServiceNow environment's compliance and operational requirements.
How does the integration perform during ServiceNow or Splunk maintenance windows or outages?
The integration includes built-in retry mechanisms and error handling that queue failed operations during maintenance windows, with automatic retry attempts when connectivity is restored. Configure Integration Hub flows with appropriate timeout values and error handling steps that log failures for manual review and reprocessing after service restoration. Splunk HTTP Event Collector provides some buffering capability for incoming data, but extended outages may result in data loss requiring manual synchronization after maintenance completion. Implement monitoring dashboards and alerts that notify administrators of integration failures during maintenance windows, and establish procedures for validating data consistency and processing queued operations after service restoration.
What Integration Hub license tier is required for full ServiceNow-Splunk integration functionality?
Integration Hub Professional license is required for accessing the Splunk spoke and creating custom flows with advanced error handling and field mapping capabilities. Standard Integration Hub license provides basic REST Message functionality but lacks the pre-built Splunk spoke actions and advanced flow design capabilities needed for comprehensive bidirectional integration. Professional license also provides higher execution limits (1000 spoke executions per hour vs 100 for Standard) necessary for high-volume data synchronization scenarios. Enterprise license tier includes additional features like advanced monitoring, performance analytics, and enhanced security controls that benefit large-scale integration deployments but is not required for basic functionality.
Can the integration synchronize ServiceNow CMDB data with Splunk for infrastructure correlation and analytics?
The integration supports CMDB data synchronization through scheduled exports that send configuration item details, relationships, and attributes to Splunk for infrastructure correlation and service mapping analytics. Configure custom REST Message scripts or Integration Hub flows to extract CI data including servers, applications, and business services with their operational status and relationship mappings. Splunk can correlate CMDB data with infrastructure monitoring data to provide service-aware alerting and impact analysis based on ServiceNow service hierarchy and dependency relationships. Implement change detection logic that identifies CMDB updates and synchronizes modified configuration items to keep Splunk infrastructure data current for accurate correlation and reporting capabilities.
Test Your Knowledge
Quick 3-question quiz — see how your ServiceNow skills stack up.
A list view on a table with millions of records is slow. Best fix?
Select an answer to continue