Import Set

What It Is

Import Set is ServiceNow's staging mechanism for external data integration, creating temporary holding tables where raw data lands before transformation into production tables. When you import data from CSV files, web services, or database connections, ServiceNow doesn't write directly to your target tables like incident or cmdb_ci_server. Instead, it creates import set tables with names like u_imp_incident_20241201_123456 where your raw data sits until Transform Maps process it into the final destination. This two-phase approach prevents data corruption, enables field mapping validation, and allows rollback of failed imports without affecting production data.

Architecturally, Import Sets live in the System Import Sets application within ServiceNow's integration layer, sitting between external data sources and the core data model. The feature spans multiple components: the sys_import_set table tracks import operations, dynamically created u_imp_* tables hold the actual imported records, and sys_transform_map defines how to move data from import tables to target tables. Import set tables inherit from sys_metadata and automatically include fields like sys_import_state to track processing status.

The underlying execution model relies on ServiceNow's scheduler and background processing engine. Import operations create scheduled jobs that read source data, populate import set tables, then trigger Transform Maps to process records in batches. Each import set table exists as a full ServiceNow table with GlideRecord access, business rules capability, and UI policies, though you'll rarely configure these directly. The platform automatically manages table creation, field definitions based on source data structure, and cleanup of processed records based on retention policies you configure.

You cannot function without Import Sets when integrating external data sources that don't perfectly match ServiceNow's table structure, when you need data validation before committing to production tables, or when importing large datasets that require transformation logic. Any scenario involving CSV uploads, database synchronization, web service consumption, or third-party application integration relies on this staging approach. Without Import Sets, you'd face direct writes to production tables with no rollback capability, no field mapping flexibility, and no error handling for malformed data. Enterprise implementations processing HR feeds, asset management updates, or customer data synchronization depend entirely on this controlled import process.

Platform administrators typically manage Import Set configuration including data source setup, table creation, and retention policies, while developers handle Transform Map scripting and complex field transformations. Integration specialists focus on scheduling, monitoring, and troubleshooting import failures. The admin-developer boundary often blurs here since Import Sets require understanding both the technical data transformation requirements and the business logic for mapping external data to ServiceNow's data model. Most organizations assign Import Set management to senior admins with scripting experience rather than pure developers.

Recent ServiceNow releases enhanced Import Set performance with improved batch processing in Vancouver and better error handling in Washington. Xanadu introduced enhanced data source connectors and simplified Transform Map debugging tools. The core Import Set functionality remains stable, but Vancouver changed how sys_import_state values get processed, requiring updates to custom scripts that check import record status. Washington added better memory management for large import operations and more granular control over concurrent import processing.

Where to Find and Configure It

Navigate to System Import Sets > Administration > Data Sources to create and configure data sources that define how external data connects to ServiceNow. Access System Import Sets > Administration > Import Sets to view import operations and their processing status. Configure Transform Maps at System Import Sets > Administration > Transform Maps to define how import set data maps to target table fields.

Find import set tables under System Definition > Tables by filtering for names starting with u_imp_ to review table structure and data. Monitor import progress through System Import Sets > Administration > Import Set Runs where you can see processing status and error details. Access System Logs > System Log > Import Sets for detailed processing logs and error troubleshooting.

How It Works Step by Step

Import Sets operate through a two-phase ETL process where external data first loads into temporary staging tables, then gets transformed into production tables through field mapping rules. The staging phase creates import set tables dynamically based on source data structure, while the transformation phase applies business logic, data validation, and field mapping to populate target tables. This separation allows rollback of failed imports, validation of data quality before production impact, and complex transformation logic that wouldn't be possible with direct table writes.

The import engine reads source data structure during initial import to create appropriate import set table schema, automatically determining field types and lengths based on sample data. ServiceNow then populates the import set table with all source records, setting sys_import_state to 'pending' for each record. Transform Maps then process records by matching source fields to target fields, executing any transformation scripts, and writing results to production tables. The platform updates sys_import_state to 'processed', 'error', or 'ignored' based on transformation results, maintaining a complete audit trail of the import operation.

The Execution Order

1. Data source connection established and source data structure analyzed
2. Import set table created dynamically with u_imp_ prefix and timestamp suffix
3. Source records loaded into import set table with sys_import_state='pending'
4. Transform Map identified and field mappings loaded
5. Transform script executes for each import record (onStart, onBefore, onAfter)
6. Field values mapped from source to target based on Transform Map configuration
7. Target record inserted or updated in production table
8. Import record sys_import_state updated to reflect processing result

// Common Transform Map script pattern for data validation and mapping
(function runTransformScript(source, map, log, target /*undefined onStart*/) {
    
    // Skip processing if required field is empty
    if (!source.u_employee_id) {
        return false; // Skip this record
    }
    
    // Transform and validate email format
    var email = source.u_email_address.toString().toLowerCase();
    if (email.indexOf('@') == -1) {
        log.error('Invalid email format for employee: ' + source.u_employee_id);
        return false;
    }
    target.email = email;
    
    // Look up department by external ID
    var deptGr = new GlideRecord('cmn_department');
    deptGr.addQuery('u_external_id', source.u_dept_code);
    deptGr.query();
    if (deptGr.next()) {
        target.department = deptGr.getUniqueValue();
    }
    
})(source, map, log, target);

Real-World Scenarios

HR Employee Data Import with Department Validation

Your HR system exports employee data weekly including department codes that must map to existing ServiceNow departments before creating user records. The import must validate email formats, skip terminated employees, and create user accounts only for active staff members.

Create a data source pointing to your HR CSV file location, then build a Transform Map targeting the sys_user table with field mappings for employee_number, first_name, last_name, and email. Set employee_number as the coalesce field to prevent duplicate user creation. In the Transform Map's onBefore script, add validation logic that queries the cmn_department table using the source department code and returns false if no matching department exists. Configure the data source to run weekly and set retention to keep import set tables for 30 days for audit purposes.

Watch for memory issues when processing large employee datasets — configure batch processing to handle 1000 records at a time maximum. Department validation failures will show in import logs but won't stop the entire import, so monitor the sys_import_state='error' records regularly. Set up email notifications for Transform Map failures since HR data quality issues need immediate attention to prevent access problems for new employees.

Asset Management System Integration with CI Relationships

Your asset management tool needs to sync server information into the CMDB, including server relationships where application servers depend on database servers. The external system provides server details plus parent-child relationship data that must create both CI records and relationship records in ServiceNow.

Create two separate Import Sets: one for server CI data targeting cmdb_ci_server and another for relationship data targeting cmdb_rel_ci. Configure the server Transform Map with coalescing on serial_number and map fields like name, ip_address, cpu_count, and ram. In the relationship Transform Map, use onBefore scripting to look up parent and child CI sys_ids based on the external asset IDs, then populate the relationship type field with the appropriate relationship from cmdb_rel_type. Schedule the server import to run first, followed by relationships to ensure all CIs exist before creating dependencies.

Relationship imports fail silently if parent or child CIs don't exist, leaving orphaned relationship records that create CMDB data quality issues. Always validate both ends of the relationship exist before creating the cmdb_rel_ci record, and consider implementing a cleanup job that removes relationships where referenced CIs have been deleted. Set different retention periods for CI and relationship import sets since relationship troubleshooting often requires historical data analysis.

Incident Import from External Monitoring Tools with Assignment Logic

Your monitoring system needs to create ServiceNow incidents automatically when critical alerts occur, with assignment routing based on affected CI location and service mapping. The external system provides alert details, affected hostname, and severity level but doesn't understand ServiceNow's assignment group structure or service relationships.

Configure a REST data source to receive monitoring alerts and create a Transform Map targeting the incident table with coalescing on a combination of u_external_alert_id and cmdb_ci to prevent duplicate incidents. Map basic fields like short_description, description, and urgency directly. In the onBefore script, implement logic that queries the CMDB to find the affected CI by hostname, determines the CI's location and responsible team, then looks up the appropriate assignment group from a mapping table you maintain. Set the incident state to New and let normal incident assignment rules handle further routing based on the populated assignment group.

Monitor for CI lookup failures that result in incidents without proper assignment — these create orphaned tickets that bypass normal SLA tracking. Consider implementing fallback assignment logic that routes to a default group when CI or assignment group lookups fail. Be careful with incident state management since external monitoring tools often send duplicate alerts during extended outages, potentially updating resolved incidents back to active status if your coalesce logic isn't restrictive enough.

The Classic Mistake

// BAD: No validation, assumes all fields exist
var target = new GlideRecord('incident');
target.initialize();
target.number = source.incident_number;
target.short_description = source.title;
target.description = source.details;
target.caller_id.setDisplayValue(source.requester_email);
target.category = source.category_code;
target.subcategory = source.subcategory_code;
target.priority = source.severity_level;
target.state = source.status;
target.assignment_group.setDisplayValue(source.team_name);
target.assigned_to.setDisplayValue(source.assigned_user);
target.business_service.setDisplayValue(source.service_name);
target.insert();

This fails because when source data is missing fields or contains invalid values, the transform continues processing but creates incomplete records with default values. Users see incidents with blank descriptions, incorrect priorities, or missing assignments, but the Import Log shows "Success" because no script errors occurred. ServiceNow processes each field independently, so a failed reference lookup doesn't stop the transform, it just leaves the field empty. The problem is non-obvious because the transform appears to work, but data quality degrades silently over time.

// GOOD: Validate source data before processing
if (!source.incident_number || !source.title || !source.requester_email) {
    action.setError('Missing required fields: number, title, or requester');
    return;
}

var target = new GlideRecord('incident');
target.initialize();
target.number = source.incident_number;
target.short_description = source.title;
target.description = source.details || 'Imported via transform map';

// Validate references before setting
var caller = new GlideRecord('sys_user');
if (caller.get('email', source.requester_email)) {
    target.caller_id = caller.sys_id;
} else {
    action.setError('Invalid caller email: ' + source.requester_email);
    return;
}

target.category = source.category_code || 'inquiry';
target.priority = source.severity_level || '4';
target.state = '1'; // Always start as New
target.insert();

When to Use This vs Alternatives

Import Sets are the right choice when you need to stage external data for validation and transformation before committing to production tables. Use them when data quality is uncertain, when you need to audit what was imported, or when business rules on target tables would interfere with the import process.

Choose Import Sets When You Need Staging

Import Sets excel when source data requires transformation, validation, or cleanup before reaching target tables. Direct REST Table API calls or Web Service Imports can't handle complex field mapping or reference resolution reliably. Import Sets also provide the only way to preview and rollback imports, making them essential for one-time data migrations or when importing from unreliable sources.

Use Direct Integration for Real-Time Data

Skip Import Sets for real-time integrations where data arrives clean and structured. IntegrationHub ETL or direct REST API calls perform better when you need immediate data availability and the source system handles validation. Import Sets add unnecessary latency when transform maps just perform simple field mapping without business logic.

Combine Both for Hybrid Approaches

Use Import Sets for initial data loads and ongoing bulk imports, while maintaining direct API integrations for real-time updates. This pattern works well for systems like HR feeds where daily employee updates come through Import Sets, but individual badge scans or status changes use direct Scripted REST APIs. You get the validation benefits of staging for complex data while maintaining responsiveness for simple updates.

Platform Interactions & Side Effects

• Business Rules on target tables execute during transform processing, which can modify imported data or trigger notifications you don't expect
• ACLs are bypassed during import processing - the import_admin role can write to any table regardless of field-level security
• Import Set tables (u_imp_*) and Transform Maps are included in Update Sets, but the actual data rows are not captured
• Each transform execution writes to sys_import_log and sys_import_set_row tables, creating audit trails that consume database space
• Transform scripts run in the global scope with elevated privileges, bypassing script security restrictions that apply to other server-side code
• Large Import Sets can cause memory issues during transform processing, especially when using Run Transform on thousands of rows simultaneously
• Workflow and Flow Designer triggers fire normally on target table inserts/updates, potentially sending duplicate notifications or creating approval records
• The sys_created_by field on imported records shows the user who ran the transform, not the original data creator
• Database views and reports that reference import set tables can break when the import set table is cleaned up or recreated
• Coalesce field matching uses database-level queries that bypass cache, potentially causing performance impact during large imports

Debugging and Troubleshooting

Failed imports typically present as data not appearing in target tables while showing "Completed" status in the Import Sets list. Users report missing records or incomplete data, but admins see successful import logs. The most common symptoms include reference field lookup failures (assignments to non-existent groups), choice field validation errors (invalid state values), and coalesce mismatches where updates create new records instead of updating existing ones.

Start debugging in System Logs > Import Log to see transform-specific errors, then check System Logs > System Log > All filtered by Source: Import for script execution errors. The sys_import_set_row table shows row-level processing status and error messages. Look for error patterns like "Invalid table" (wrong target table), "Multiple matches found" (coalesce conflicts), or "Access denied" (ACL issues in transform scripts).

Transform scripts generate specific error messages in the Import Log: "Field not found on target table" indicates schema mismatches, "setDisplayValue failed" means reference lookup problems, and "Coalesce field mismatch" shows duplicate detection issues. Enable the glide.import.debug_transforms system property to capture detailed transform execution logs, and use gs.log() statements in transform scripts to trace data transformation logic.

Diagnostic Checklist:

• Verify import set data loaded correctly by viewing the u_imp_* table contents before running transforms
• Test transform maps with a single row first using Transform single row to isolate mapping issues
• Check coalesce field values match existing records exactly - case sensitivity and extra spaces cause match failures
• Validate reference field lookups by running the same queries manually in Scripts - Background
• Review target table Business Rules that might be modifying or rejecting imported data during insert/update
• Confirm the user running the transform has import_admin and import_transformer roles
• Enable Debug Transform option and examine detailed processing logs in the Import Set record

Quick Reference

• Import Set tables are automatically dropped after 90 days by default via the glide.import_set.cleanup_age system property
• Transform scripts can access both source (import set row) and target (destination record) objects, plus action for error handling
• Maximum 10,000 rows per Import Set execution - larger datasets require chunking into multiple import sets
• Coalesce fields can use up to 3 field combinations, but performance degrades significantly with multiple coalesce fields
• The import_set_run table tracks execution history but doesn't store the actual imported data
• Transform maps process rows sequentially, not in parallel, making large imports inherently slow
• Field maps support setDisplayValue() for reference fields, but setValue() requires sys_id values
• Import Set APIs support JSON, XML, and Excel file formats, but CSV imports require the Load Data module
• Business Rules on import set tables (u_imp_*) don't fire during data loading, only during manual record updates
• The ignore_empty_key transform option prevents updates when coalesce fields contain empty values, avoiding unintended record matches