Test Suite

What It Is

A Test Suite is a container that groups multiple ATF tests into a logical collection that executes sequentially or in parallel. It solves the fundamental problem of regression testing at scale — ensuring that your customizations, workflows, and integrations continue working after platform upgrades, update set deployments, or configuration changes. Without Test Suites, you're stuck running individual tests manually or hoping that scattered validation catches breaking changes before users discover them.

Test Suites live within the Automated Test Framework application (sn_atf scope) and sit at the orchestration layer above individual test records. The sys_atf_test_suite table stores the suite definitions, while sys_atf_test_suite_test maintains the many-to-many relationships between suites and their constituent tests. The execution results populate sys_atf_test_result_suite with aggregate pass/fail data and timing metrics.

The data model connects Test Suites to the broader ServiceNow testing ecosystem through scheduled jobs (sysauto_script), update set commit hooks, and CI/CD pipeline integrations via the TestSuite REST API. Each suite execution spawns a background job that manages test sequencing, handles failures, and aggregates results. The execution environment runs in the global scope regardless of where individual tests are scoped, giving suites the ability to validate cross-application workflows and integrations.

You cannot function without Test Suites in three critical scenarios: automated regression testing during platform upgrades, continuous integration pipelines that deploy update sets, and compliance environments that require documented testing evidence. When ServiceNow releases twice yearly, Test Suites are the only practical way to validate that hundreds of customizations still work correctly. During update set deployments between instances, Test Suites catch integration breaks, workflow failures, and data validation issues that individual tests miss. For regulated industries, Test Suites provide the audit trail and repeatability that compliance frameworks demand.

Platform owners typically create and maintain the suite structure, defining which tests belong together and establishing execution schedules. Developers contribute individual tests to suites and maintain test-specific logic, while administrators configure suite execution parameters, manage scheduling, and interpret results for business stakeholders. The atf_test_admin role controls suite configuration, while atf_test_designer allows test creation and modification.

Vancouver introduced parallel test execution within suites, dramatically reducing total runtime for large test collections. The Run tests in parallel checkbox enables this behavior, though tests with dependencies or shared data should remain sequential. Xanadu added enhanced result filtering and the ability to re-run only failed tests from a suite execution, significantly improving debugging workflows. The Rollback on test failure option, available since Utah, now integrates better with update set deployment automation.

Where to Find and Configure It

Navigate to System Applications > Automated Test Framework > Tests > Test Suites for the primary configuration interface where you create new suites, add tests, and configure execution parameters. From Studio, access Tests > Test Suites within your application scope to manage application-specific test collections. For suite execution and results, go to System Applications > Automated Test Framework > Tests > Test Suite Results to review historical runs and debugging information.

The underlying data lives in sys_atf_test_suite.list for direct table access and bulk operations. Schedule automated execution through System Definition > Scheduled Script Executions using the ATFTestSuiteRunner script include. Monitor active test executions from System Scheduler > Scheduled Jobs > Running where suite runs appear as background jobs with the naming pattern ATF Test Suite: [Suite Name].

How It Works Step by Step

Test Suite execution operates through ServiceNow's scheduled job infrastructure, spawning a background worker process that manages the entire test lifecycle. The suite runner reads the test collection, validates each test's availability and permissions, then either executes tests sequentially or launches parallel execution threads based on the suite configuration. Each individual test runs in its own transaction scope with rollback capability, while the suite maintains aggregate state and timing metrics.

The execution engine handles test dependencies by evaluating prerequisite conditions before launching each test step. Failed tests trigger the suite's error handling logic, which can halt execution immediately, skip remaining tests in a dependency chain, or continue running independent tests based on the Stop on first failure setting. Throughout execution, the suite writes detailed logging to sys_atf_test_result records, capturing step-by-step results, timing data, and failure diagnostics for post-execution analysis.

The Execution Order

1. Suite execution begins with the scheduler creating a sys_trigger record and spawning a background job
2. The ATFTestSuiteRunner script include queries sys_atf_test_suite_test to build the test execution list
3. Each test's Active flag and application scope permissions are validated before execution
4. A new sys_atf_test_result_suite record is created with Running status and execution start timestamp
5. Individual tests execute via the sn_atf.ATFTestRunner API, either sequentially or in parallel worker threads
6. Each test step creates sys_atf_step_result records with pass/fail status, execution time, and error details
7. Transaction rollback occurs automatically after each test unless Rollback is disabled in the test configuration
8. Suite execution completes with aggregate results written to the suite result record and optional email notifications sent

// Execute a Test Suite programmatically
var suite = new GlideRecord('sys_atf_test_suite');
if (suite.get('name', 'Critical Path Regression')) {
    var runner = new sn_atf.ATFTestSuiteRunner();
    var result = runner.runTestSuite(suite.sys_id, {
        parallel: true,
        stopOnFailure: false,
        rollback: true
    });
    
    // Monitor execution status
    var suiteResult = new GlideRecord('sys_atf_test_result_suite');
    suiteResult.get(result.suite_result_id);
    
    gs.info('Suite execution started: ' + suiteResult.sys_id);
    gs.info('Total tests queued: ' + suiteResult.total_tests);
    gs.info('Execution status: ' + suiteResult.status);
}

Real-World Scenarios

Automated Pre-Upgrade Validation Suite

Before each ServiceNow platform upgrade, you need comprehensive validation that all critical business processes still function correctly. This suite must test incident workflows, change approvals, catalog ordering, and custom integrations to catch breaking changes before they impact production users.

Create a new Test Suite named Pre-Upgrade Validation and add your critical path tests in logical groups. Set Run tests in parallel to false since upgrade validation tests often share test data. Configure Stop on first failure to false so you get a complete failure picture. Add the platform owner's email to the Send results via email field and schedule execution for your upgrade maintenance window.

Update Set Deployment Gate with CI/CD Integration

Your development team pushes update sets to test and production instances through a CI/CD pipeline, but you need automated testing that prevents broken deployments from reaching production. The Test Suite must validate that new customizations don't break existing functionality and that integrations still work correctly.

Create separate suites for each deployment stage: Post-Deployment Smoke Test for quick validation and Full Integration Test for comprehensive coverage. Configure the smoke test suite with Run tests in parallel enabled and Stop on first failure enabled for fast feedback. Use the TestSuite REST API endpoint /api/sn_atf/testsuite/{suite_id}/run in your deployment pipeline to trigger execution after update set commit.

Daily Regression Suite for Critical Business Processes

Business stakeholders need daily confidence that core ITSM processes are functioning correctly, especially incident management, change approvals, and service catalog ordering. The Test Suite must run automatically each morning and alert the support team to any failures before business hours begin.

Build a Test Suite called Daily Business Process Check containing tests for incident creation and assignment, change approval workflows, and catalog item ordering. Enable Run tests in parallel to minimize execution time but disable Stop on first failure to get complete results. Create a scheduled script execution that runs at 6:00 AM daily, targeting your production instance. Configure email notifications to your support team distribution list with Send results via email set to On failure only.

The Classic Mistake

// BAD: Hardcoded sys_ids and environment-specific values
(function(outputs, steps, params, stepResult, assertEqual) {
    var incGR = new GlideRecord('incident');
    incGR.get('9d385017c611228701d22104cc95c371'); // Hardcoded sys_id
    assertEqual('P1 - Critical', incGR.priority.getDisplayValue());
    
    var userGR = new GlideRecord('sys_user');
    userGR.get('681ccaf9c0a8016400b98a06818d57c7'); // Another hardcoded sys_id
    assertEqual('john.doe@company.com', userGR.email.toString());
    
    var configGR = new GlideRecord('sys_properties');
    configGR.addQuery('name', 'company.api.endpoint');
    configGR.query();
    if (configGR.next()) {
        assertEqual('https://prod-api.company.com/v1', configGR.value.toString());
    }
})(outputs, steps, params, stepResult, assertEqual);

This approach creates Test Suites that pass in development but consistently fail in test and production instances, generating false negative results that erode confidence in automated testing. ServiceNow's ATF framework executes the exact script code as written, so hardcoded sys_ids that don't exist in target instances cause GlideRecord.get() calls to return false, leading to assertion failures on null or undefined values. The failure appears as a legitimate test failure rather than a configuration issue, making it non-obvious that the problem is environmental rather than functional. Teams often disable these "flaky" tests instead of fixing the root cause, defeating the purpose of regression testing.

// GOOD: Query-based approach with environment-agnostic logic
(function(outputs, steps, params, stepResult, assertEqual) {
    var incGR = new GlideRecord('incident');
    incGR.addQuery('priority', '1'); // Query by business logic
    incGR.addQuery('state', '1');
    incGR.setLimit(1);
    incGR.query();
    if (incGR.next()) {
        assertEqual('P1 - Critical', incGR.priority.getDisplayValue());
    }
    
    var userGR = new GlideRecord('sys_user');
    userGR.addQuery('user_name', 'admin'); // Known system user
    userGR.query();
    if (userGR.next()) {
        stepResult.setOutputMessage('Found admin user: ' + userGR.sys_id);
    }
})(outputs, steps, params, stepResult, assertEqual);

When to Use This vs Alternatives

Test Suites are the correct choice when you need automated regression testing that validates business processes end-to-end across multiple applications and integrations. They excel at catching breaking changes in complex workflows where individual unit tests would miss interaction failures between components.

Use Test Suites for Regression Testing

Choose Test Suites when validating that customizations don't break core ServiceNow functionality, especially before major updates or when promoting significant changes. Individual ATF Tests can't provide the comprehensive coverage needed to catch regressions across modules—you need the orchestrated execution that Test Suites provide. Manual testing becomes impractical when you have dozens of business-critical workflows that need validation after every change.

Use Manual Testing for Exploratory Work

Skip Test Suites for new feature development or ad-hoc investigations where you need human judgment and creative exploration. Test Suites validate known-good behaviors but can't discover new issues or evaluate user experience quality that requires subjective assessment. Use manual testing when the functionality is still evolving or when the test scenarios themselves need refinement.

Combine Test Suites with CI/CD Pipelines

Integrate Test Suites with Update Set deployment automation and Jenkins or similar CI/CD tools for maximum effectiveness. Test Suites alone provide validation but don't prevent bad deployments—you need the pipeline integration to automatically block promotions when tests fail. This combination creates a safety net that catches issues before they reach production while maintaining development velocity.

Platform Interactions & Side Effects

• Business Rules fire during test execution with gs.isInteractive() returning false, potentially bypassing UI-only validation rules
• ACL evaluation occurs with the test runner's security context, not the simulated user, affecting role-based testing accuracy
• Email notifications are suppressed by default unless glide.email.test.active system property is enabled, creating false positives for notification testing
• Test execution results are stored in sys_atf_test_result and sys_atf_test_result_step tables, consuming database storage over time
• Scheduled Test Suite executions run in the background and can impact instance performance during peak hours if not properly timed
• Update Set commit triggers write to sys_update_set_commit table and can automatically trigger associated Test Suite execution
• Script execution contexts in Test Suites bypass normal transaction boundaries, potentially masking database rollback issues
• Integration with external systems during testing can create real data changes unless properly mocked or configured for test environments
• Session state and caching behaviors differ in test execution, with g_session variables not persisting between test steps as expected
• Audit records in sys_audit table are created for test data modifications, potentially cluttering audit trails with test artifacts

Debugging and Troubleshooting

Test Suite failures typically manifest as red status indicators in the Automated Test Framework > Test Results module, with individual test step failures showing assertion errors or script exceptions. Users may report functional issues that the Test Suite should have caught but didn't, indicating gaps in test coverage or incorrect test logic. The most common failure pattern is tests passing in development but failing in higher environments due to data dependencies or environment-specific configurations.

Check System Logs > System Log > All for JavaScript errors and script exceptions during test execution, filtering by source=ATF to isolate test-related issues. The Test Result record itself contains detailed step-by-step execution logs with timing information and output messages that reveal where failures occur. Look for error messages like "AssertionError: Expected [value] but got [value]" for assertion failures, or "ReferenceError" and "TypeError" messages for script issues in custom test steps.

Enable the atf.runner.log.verbosity system property for detailed execution logging, and use the Script Debugger on test step scripts to step through execution when developing complex test logic. Performance issues often appear as timeouts with "Test step exceeded maximum execution time" messages, indicating either slow queries or infinite loops in test scripts. Database lock errors during parallel test execution show up as "Resource busy" messages in the application logs.

Diagnostic Checklist:

• Verify test data exists in the target instance by running individual test queries manually
• Check system properties referenced in tests match expected values across environments
• Review Business Rules and ACLs that might behave differently during automated execution
• Validate that test user accounts have appropriate roles and aren't locked/inactive
• Check for conflicting scheduled jobs or maintenance windows during test execution times
• Examine integration endpoints and external system availability for interface testing failures
• Run individual tests in isolation to identify dependencies between tests causing cascade failures

Quick Reference

• Test Suites can contain maximum 100 individual tests, with execution timing out at 60 minutes by default
• Parallel test execution is limited to 5 concurrent threads on most instances, configurable via atf.runner.max_parallel_tests
• Test results are automatically purged after 90 days unless the atf.result.cleanup.age_in_days property is modified
• Update Set commit triggers are evaluated even when tests fail, requiring manual intervention to prevent bad deployments
• Test Suite execution order is not guaranteed unless explicitly configured with Run tests sequentially checkbox
• Browser-based UI tests within Test Suites require the instance to be accessible from ServiceNow's ATF runners, failing in private networks
• Test data created during Test Suite execution is not automatically cleaned up, requiring explicit cleanup steps or manual data management
• Scheduled Test Suite executions can overlap if the previous run hasn't completed, potentially causing resource contention and false failures
• REST API calls within tests are subject to rate limiting and may fail with HTTP 429 errors during high-volume test execution
• Test Suite results include execution metadata in sys_atf_test_suite_result table with start time, duration, and pass/fail counts for reporting integration