How to Run Parallel Jobs in Same Stage and Trigger Other Jobs in GitLab CI Pipeline

In modern software development, continuous integration and delivery (CI/CD) pipelines are critical for ensuring code quality, accelerating feedback loops, and automating deployments. GitLab CI/CD, a built-in tool in GitLab, empowers teams to define, run, and monitor pipelines using a simple YAML file (.gitlab-ci.yml). One of the key optimizations for efficient pipelines is running parallel jobs within the same stage to save time, followed by triggering downstream jobs once these parallel tasks complete.

Whether you’re testing across multiple environments, building for different platforms, or splitting large tasks into smaller chunks, parallelization reduces pipeline duration. Similarly, triggering dependent jobs ensures that subsequent tasks (e.g., deployment, reporting) only run after prerequisites (e.g., tests, builds) complete successfully.

This blog will guide you through:

  • The basics of GitLab CI/CD pipelines, stages, and jobs.
  • Running parallel jobs in the same stage (two methods: multiple distinct jobs and job splitting with parallel).
  • Triggering downstream jobs after parallel execution (via stages, needs, and artifact dependencies).
  • Advanced scenarios, best practices, and troubleshooting.

Table of Contents#

  1. Understanding GitLab CI/CD Pipeline Basics
  2. Running Parallel Jobs in the Same Stage
  3. Triggering Other Jobs After Parallel Execution
  4. Advanced Scenarios
  5. Best Practices
  6. Troubleshooting Common Issues
  7. Conclusion
  8. References

1. Understanding GitLab CI/CD Pipeline Basics#

Before diving into parallelism and triggering, let’s recap core GitLab CI/CD concepts:

  • Pipeline: A collection of jobs grouped into stages, executed in order.
  • Stage: A logical group of jobs. Jobs in the same stage run in parallel by default (unless dependencies are defined). Stages run sequentially (e.g., build → test → deploy).
  • Job: A set of commands executed by a GitLab Runner. Each job belongs to a stage and can produce artifacts (e.g., build outputs, test reports) or cache dependencies.

A basic .gitlab-ci.yml structure looks like this:

stages:          # Define stages (sequential order)  
  - build  
  - test  
  - deploy  
 
build_job:       # Job in the 'build' stage  
  stage: build  
  script:  
    - echo "Building..."  
 
test_job:        # Job in the 'test' stage (runs after 'build' stage completes)  
  stage: test  
  script:  
    - echo "Testing..."  
 
deploy_job:      # Job in the 'deploy' stage (runs after 'test' stage completes)  
  stage: deploy  
  script:  
    - echo "Deploying..."

2. Running Parallel Jobs in the Same Stage#

Parallel jobs in a stage reduce pipeline time by leveraging concurrent execution. GitLab supports two primary approaches:

2.1 Multiple Distinct Jobs in One Stage#

Define multiple independent jobs in the same stage. By default, GitLab runs them in parallel (assuming runners are available).

Example: Parallel Test Jobs for Frontend and Backend
Suppose you want to test frontend and backend code simultaneously. Define two jobs in the test stage:

stages:  
  - test  
 
test_frontend:  
  stage: test  
  script:  
    - cd frontend  
    - npm install  
    - npm test  
  artifacts:  
    paths:  
      - frontend/test-report.xml  # Save test results  
 
test_backend:  
  stage: test  
  script:  
    - cd backend  
    - pip install -r requirements.txt  
    - pytest  
  artifacts:  
    paths:  
      - backend/test-report.xml  # Save test results

Outcome: test_frontend and test_backend run in parallel. The test stage completes when both jobs finish.

2.2 Splitting a Single Job into Parallel Instances (with parallel)#

Use the parallel keyword to split a single job into multiple parallel instances. This is ideal for:

  • Matrix builds (e.g., testing across OSes, language versions, or browsers).
  • Load distribution (e.g., splitting a large test suite into chunks).

2.2.1 Parallel Matrix Jobs#

Create a matrix of jobs using parallel:matrix to test combinations of variables (e.g., OS and Node.js versions):

stages:  
  - test  
 
test_matrix:  
  stage: test  
  image: node:$VERSION  
  parallel:  
    matrix:  
      - OS: [ubuntu-latest, windows-latest]  # 2 OS options  
        VERSION: [16, 18]                   # 2 Node.js versions  
  script:  
    - echo "Testing on $OS with Node.js $VERSION"  
    - npm install  
    - npm test  
  tags:  
    - $OS  # Use runners tagged for the target OS

Outcome: 4 parallel jobs are created (2 OS × 2 versions). Each job has unique OS and VERSION variables.

2.2.2 Fixed Number of Parallel Jobs#

Split a job into a fixed number of instances (e.g., 3 parallel jobs to run a large test suite):

stages:  
  - test  
 
test_parallel:  
  stage: test  
  parallel: 3  # Split into 3 parallel jobs  
  script:  
    - echo "Job instance $CI_NODE_INDEX of $CI_NODE_TOTAL"  # Built-in variables for parallel jobs  
    - npm install  
    - ./split_tests.sh $CI_NODE_INDEX $CI_NODE_TOTAL  # Custom script to split tests  
    - npm test -- --shard=$CI_NODE_INDEX/$CI_NODE_TOTAL  # Pass shard to test runner

Key Variables:

  • CI_NODE_INDEX: Index of the parallel job (1-based, starting at 1).
  • CI_NODE_TOTAL: Total number of parallel instances.

3. Triggering Other Jobs After Parallel Execution#

Once parallel jobs complete, you’ll often need to trigger downstream jobs (e.g., deploy, report generation). Here’s how:

3.1 Stage-Level Sequencing#

Stages run sequentially by default. Jobs in a later stage run only after all jobs in earlier stages complete.

Example: Test → Deploy Pipeline

stages:  
  - test    # Parallel jobs here  
  - deploy  # Runs after 'test' stage completes  
 
# Parallel test jobs (from 2.1)  
test_frontend:  
  stage: test  
  script: ...  
 
test_backend:  
  stage: test  
  script: ...  
 
deploy_prod:  
  stage: deploy  
  script:  
    - echo "Deploying after tests pass!"

Outcome: deploy_prod runs only after both test_frontend and test_backend succeed.

3.2 Job-Level Dependencies (with needs)#

Use needs to trigger a job before the entire stage completes. This is useful for "fast-tracking" critical jobs.

Example: Run a smoke test immediately after a build, even if other build jobs are running

stages:  
  - build  
  - test  
 
build_app:  
  stage: build  
  script: ./build.sh  
  artifacts:  
    paths: [app.zip]  
 
build_docs:  
  stage: build  # Runs in parallel with build_app  
  script: ./build_docs.sh  
 
smoke_test:  
  stage: test  
  needs: [build_app]  # Only wait for build_app (not build_docs)  
  script: ./smoke_test.sh app.zip  # Uses artifact from build_app

Outcome: smoke_test starts as soon as build_app finishes, even if build_docs is still running.

3.3 Aggregating Results from Parallel Jobs#

Parallel jobs often produce artifacts (e.g., test reports). Use a downstream job to aggregate these results.

Example: Combine Test Reports from Parallel Matrix Jobs

stages:  
  - test  
  - report  
 
# Parallel matrix jobs (from 2.2.1)  
test_matrix:  
  stage: test  
  parallel:  
    matrix: [OS: [ubuntu, windows], VERSION: [16, 18]]  
  script: npm test  
  artifacts:  
    paths: [test-report-$OS-$VERSION.xml]  # Unique artifact per job  
 
aggregate_reports:  
  stage: report  
  script:  
    - ./combine_reports.sh test-report-*.xml > combined-report.xml  # Merge all reports  
  artifacts:  
    paths: [combined-report.xml]

Outcome: aggregate_reports runs after all test_matrix jobs complete, merging their artifacts into a single report.

4. Advanced Scenarios#

4.1 Conditional Parallel Jobs#

Run parallel jobs only when specific conditions are met (e.g., changes to a directory, or a branch name). Use only, except, or rules.

Example: Run frontend tests only if frontend/ changes

test_frontend:  
  stage: test  
  script: ...  
  rules:  
    - changes:  
        - frontend/**/*  # Trigger only if frontend files change  
 
test_backend:  
  stage: test  
  script: ...  
  rules:  
    - changes:  
        - backend/**/*  # Trigger only if backend files change

4.2 Dynamic Parallelism#

Adjust the number of parallel jobs using a variable (e.g., more parallel jobs for the main branch). The parallel keyword accepts either an integer or a matrix hash, so you can reference a variable for the count:

test_parallel:  
  stage: test  
  parallel: $PARALLEL_JOBS  # Set via project variable or pipeline trigger  
  script: ...

Set PARALLEL_JOBS dynamically:

  • In GitLab UI: Go to Settings → CI/CD → Variables.
  • In pipeline triggers: Use a pipeline trigger token (created under Settings → CI/CD → Pipeline trigger tokens) with the pipeline triggers API. For example:
    curl --request POST --form "token=$TRIGGER_TOKEN" --form "ref=main" --form "variables[PARALLEL_JOBS]=5" "https://gitlab.com/api/v4/projects/$PROJECT_ID/trigger/pipeline".
    Note: $CI_JOB_TOKEN is used to trigger multi-project pipelines, not the pipeline triggers API endpoint shown here.

4.3 Triggering Child Pipelines from Parallel Jobs#

Use the trigger keyword to launch child pipelines from parallel jobs, enabling modular pipeline design.

Example: Parallel jobs trigger child pipelines for microservices

stages:  
  - trigger_microservices  
 
trigger_auth_service:  
  stage: trigger_microservices  
  trigger:  
    include: auth-service/.gitlab-ci.yml  # Child pipeline for auth service  
 
trigger_payment_service:  
  stage: trigger_microservices  
  trigger:  
    include: payment-service/.gitlab-ci.yml  # Child pipeline for payment service

Outcome: Both child pipelines run in parallel, each handling their microservice’s CI/CD.

5. Best Practices#

  • Name Jobs Clearly: Use descriptive names (e.g., test-frontend-ubuntu-node16 instead of test1).
  • Limit Artifacts: Only pass necessary artifacts between jobs to reduce pipeline time and storage.
  • Tag Runners Strategically: Use runner tags (e.g., gpu, windows) to ensure parallel jobs run on appropriate hardware.
  • Set Timeouts: Prevent stuck parallel jobs with timeout: 30m.
  • Test Matrix Sparingly: Avoid overloading pipelines with excessive matrix combinations (e.g., test critical OS/version pairs only).
  • Monitor with GitLab UI: Use the pipeline graph to visualize parallel job progress and identify bottlenecks.

6. Troubleshooting Common Issues#

  • Parallel Jobs Failing Due to Resource Limits:

    • Check runner logs for no available runners errors.
    • Add more runners or adjust runner concurrency (concurrent in config.toml).

  • Artifacts Not Propagating:

    • Ensure artifacts:paths is correctly defined in upstream jobs.
    • Use needs:artifacts: true in downstream jobs (default behavior).

  • Matrix Jobs Not Generating:

    • Verify parallel:matrix syntax (e.g., correct indentation, variable names).

  • Job Dependencies Failing with needs:

    • Ensure the target job exists and is in an earlier or the same stage.
    • Avoid circular dependencies (e.g., job A needs job B, which needs job A).

7. Conclusion#

Running parallel jobs in GitLab CI/CD and triggering downstream workflows are powerful techniques to optimize pipeline speed and efficiency. By leveraging multiple jobs per stage, the parallel keyword, and strategic dependencies with needs, you can reduce feedback loops and scale your CI/CD to handle complex projects.

Experiment with the examples provided, and refer to GitLab’s official documentation for deeper dives into specific features.

8. References#

2025-11