BrowserCat
Dashboard →
guide

Integrating Playwright with CI/CD: Jenkins vs GitHub Actions

Integrating Playwright with CI/CD Pipelines: Jenkins vs GitHub Actions vs GitLab CI/CD – The Ultimate Guide

Continuous integration can feel like herding cats – especially when your tests involve real web browsers. But fear not! In this comprehensive guide, we’ll explore how to integrate Microsoft Playwright end-to-end tests into popular CI/CD pipelines (including Jenkins, GitHub Actions, and GitLab CI/CD) to achieve a purr-fect automated testing workflow. Whether you’re a Jenkins kitten (beginner) or a DevOps fat cat (advanced), this article has something for everyone, from basic setup steps to advanced parallel execution strategies. We’ll maintain a playful tone (cat puns included 🐱) while preserving technical depth and authority. Plus, we’ll highlight how cloud services like BrowserCat can simplify your life by hosting browsers for you – a “buy, not build” no-brainer that can save time and reduce flakiness (Playwright in the Cloud).

By the end of this guide, you’ll know how to set up Playwright tests in each CI tool, run them in parallel for speed, handle test artifacts, and compare the strengths of Jenkins vs GitHub vs GitLab for your project’s needs. Let’s dive in and ensure your CI pipeline is the cat’s meow of automation!

Table of Contents

Introduction: Why Integrate Playwright with CI/CD?

In modern software development, Continuous Integration (CI) and Continuous Deployment (CD) pipelines are essential for delivering high-quality software quickly (Continuous Integration Tools for DevOps - Jenkins vs. GitLab CI vs. GitHub Action | Attract Group). Integrating your Playwright tests into CI/CD means every code change is automatically validated by running browser tests in a consistent environment. In other words, each time developers push code, the pipeline will “play (wright) it again” – executing your end-to-end tests to catch bugs early and ensure nothing breaks. This leads to a faster feedback loop and more confident releases.

Playwright is a powerful open-source framework for automating browsers (Chromium, Firefox, WebKit) with a single API. It’s great for end-to-end testing because it can mimic real user interactions across multiple browsers. By plugging Playwright tests into CI:

  • Automated Testing – You’ll ensure each commit is vetted by running critical UI flows, preventing regressions before they reach production (Integrating Playwright with CI/CD Pipelines - DEV Community).
  • Consistency – Tests run in a controlled, repeatable environment (no “works on my machine” issues).
  • Scalability – CI can run tests on multiple agents or in parallel, covering more browsers/devices or larger test suites quickly.
  • Faster Feedback – Developers get immediate feedback from failing tests, so issues can be fixed early in the development cycle.

In short, integrating Playwright with CI/CD turns your test suite into an automated guardian that pounces on bugs as soon as they appear. It helps maintain software quality and frees developers from running tests manually. In the next sections, we’ll explore how to set up Playwright in three popular CI tools – GitHub Actions, GitLab CI, and Jenkins – step by step.

Overview of Jenkins, GitHub Actions, and GitLab CI/CD

Before we dive into configurations, let’s briefly meet our three CI/CD tools and see how they differ. Each of these tools achieves the same goal (automating your build and test process) but with different approaches:

  • Jenkins – A popular open-source automation server that you host yourself (or on a server you manage). Jenkins uses a Jenkinsfile (written in a Groovy-based DSL) to define pipelines. It has a vast plugin ecosystem for customization. You’ll need to maintain the Jenkins server or agent nodes, but you get maximum control and flexibility in return.
  • GitHub Actions – A CI/CD platform integrated into GitHub. Workflows are defined with YAML files in your repo (under .github/workflows). GitHub provides hosted runners (Linux, Windows, macOS VMs) for free for public repos and with quotas for private ones. It’s cloud-based and tightly integrated with GitHub repositories (triggering on pushes, PRs, etc. is seamless). No server maintenance – just write a workflow file, and GitHub handles the rest.
  • GitLab CI/CD – GitLab’s built-in CI system, configured by a .gitlab-ci.yml file in your repo. If you use GitLab (self-hosted or GitLab.com), CI is integrated. It uses a runner system: shared runners are provided on GitLab.com, or you can host your own. Like GitHub Actions, it uses YAML and is cloud-hosted (for GitLab SaaS) or can run on your infrastructure via GitLab Runner. It natively integrates with GitLab repos and permissions.

Despite differences, all three support running Playwright tests in a similar fashion: install Node and dependencies, run npx playwright test, and gather results. The biggest differences are in setup and infrastructure: Jenkins = self-managed (more setup), GitHub/GitLab = managed (easier start). We’ll cover each in detail next.

Setting Up Playwright in GitHub Actions

GitHub Actions makes it easy to automate Playwright tests, as it’s tightly integrated with your GitHub repository. You define a workflow in YAML, and GitHub runs it on each push or pull request. Let’s create a basic workflow that installs dependencies and runs your Playwright tests on an Ubuntu runner.

1. Create a Workflow File

In your repository, create a folder .github/workflows (if it doesn’t exist) and add a file, e.g., playwright-ci.yml. This file will describe our CI pipeline.

2. Define the Workflow Configuration

Start by specifying when to run and on what environment. For example, we can run on every push to the main branch and on pull requests:

name: Playwright Tests

on:
  push:
    branches: [ main ]
  pull_request:

jobs:
  test:
    runs-on: ubuntu-latest
    env:
      CI: true  # signal CI mode to Playwright (often not strictly needed, but good practice)
    steps:
      - name: Checkout repository
        uses: actions/checkout@v3

      - name: Setup Node.js
        uses: actions/setup-node@v3
        with:
          node-version: 18

      - name: Install dependencies
        run: npm ci

      - name: Install Playwright browsers
        run: npx playwright install --with-deps

      - name: Run Playwright tests
        run: npx playwright test

      - name: Upload Playwright report on failure
        if: ${{ failure() }}
        uses: actions/upload-artifact@v3
        with:
          name: playwright-report
          path: playwright-report

Explanation of Workflow Steps

  • Triggers: It runs on pushes to the main branch and on any Pull Request (you can adjust triggers as needed).
  • Job environment: We use Ubuntu as the runner (ubuntu-latest), which comes with a base Node.js environment. We also set CI: true in environment variables (Playwright uses this to adjust default settings for CI, like not opening the UI reporter).
  • Steps:
    • Checkout the code (using GitHub’s standard action).
    • Set up Node.js (using an action to ensure the desired Node version is installed on the runner).
    • Install NPM dependencies (npm ci installs from lockfile).
    • Install Playwright browsers with necessary dependencies (npx playwright install --with-deps). This step ensures Chromium, Firefox, and WebKit binaries are installed, along with OS packages needed for headless browsers. (Using --with-deps is important on a fresh Linux machine so that things like font libraries and ffmpeg are present for the browsers.)
    • Run the tests via the Playwright test runner.
    • Upload the Playwright HTML report as an artifact only if tests fail (using the actions/upload-artifact module). This way, if a test fails, you can download the playwright-report from the GitHub Actions UI and see what went wrong. (You could also choose to always upload artifacts for each run.)

Parallelism

By default, Playwright will try to utilize available CPU cores to run tests in parallel. GitHub’s Ubuntu runners have 2 cores, so Playwright might run 2 tests concurrently. We’ll discuss more strategies for parallel execution in a later section, including how to use the matrix strategy to run across multiple browsers or shards. (Spoiler: GitHub Actions can split jobs via a matrix, e.g., running one job per browser – Chrome, Firefox, WebKit – to purr-allelize your test suite across browsers.)

Artifacts and Reports

Our example stores the HTML report. You can also upload screenshots, videos, or Playwright trace files if you’ve configured those in your tests. This is crucial for debugging because CI runs are headless – artifacts let you see what happened after the fact.

Finally, commit this workflow file to your repo. On the next push, you should see a GitHub Actions workflow kick off, running your Playwright tests automatically. If everything is set up correctly, you’ll get a green check (pass) or a red X (fail) on your commit depending on the test results. Congrats, you’ve integrated Playwright with GitHub Actions!

Setting Up Playwright in GitLab CI/CD

If your project is on GitLab, you can achieve a similar setup using GitLab CI/CD. GitLab CI is configured by adding a .gitlab-ci.yml file to your repository. This file defines your pipeline’s stages and jobs. Let’s set up a basic pipeline to run Playwright tests:

1. Create the .gitlab-ci.yml in Your Repo Root

2. Define Stages and a Test Job

We’ll use a single test stage here to run our Playwright tests. For consistency and ease, we can use the official Playwright Docker image provided by Microsoft, which has all browsers and dependencies pre-installed.

Here’s an example .gitlab-ci.yml configuration:

stages:
  - test

playwright_tests:
  stage: test
  image: mcr.microsoft.com/playwright:v1.50.1-noble  # Official Playwright Docker image
  variables:
    CI: "true"
  script:
    - npm ci
    - npx playwright install --with-deps
    - npx playwright test
  artifacts:
    when: always
    paths:
      - playwright-report/
    expire_in: 1 week

Explanation of the Configuration

  • Stages: We define a stage called test. Our only job will run in this stage.
  • Job playwright_tests: It uses the Playwright Docker image (replace the version tag with the latest Playwright version you are using). This image includes Node, the @playwright/test package, all three browsers, and system dependencies – saving us from manually installing browsers each run.
  • Environment Variables: We set CI: "true" as an environment variable (GitLab also sets some CI variables by default).
  • Script Steps:
    • npm ci to install dependencies (assuming your project’s package.json and lockfile are in the repo).
    • npx playwright install --with-deps to ensure browsers are installed. (In theory, since we used the Docker image, the browsers should already be installed. But running this ensures that if you pinned a specific Playwright version or if any dependency is missing, it gets addressed. It’s a harmless step on an up-to-date image.)
    • npx playwright test to execute the tests.
  • Artifacts: We tell GitLab to collect the playwright-report/ folder (which contains the HTML report if using the default HTML reporter) no matter if the test passes or fails (when: always). We set it to expire in a week (so old artifacts get cleaned up). This means after the job finishes, you can download the test report from GitLab’s UI. You can also include other output directories (e.g., screenshots, videos, traces) in the artifacts if needed.

Commit this file. GitLab will automatically detect the .gitlab-ci.yml and trigger the pipeline on the next push. If using GitLab.com with shared runners, the job will queue on their provided infrastructure. If you have your own GitLab Runner, ensure it’s online to pick up the job. Once the runner executes the job, you’ll see the pipeline result in the GitLab UI for each commit.

Setting Up Playwright in Jenkins

Jenkins gives you ultimate control over your CI environment. Unlike GitHub or GitLab, Jenkins jobs run on agents that you manage (which could be on your own servers or cloud VMs). To integrate Playwright, you’ll create a pipeline (via a Jenkinsfile) that sets up Node, installs Playwright, and runs tests, similar to the above. Let’s outline a straightforward Jenkins pipeline for Playwright:

1. Jenkins Pipeline Job Setup

Make sure you have a Jenkins instance up and running with an agent capable of running Node.js. This could be the master node or a dedicated agent. Jenkins allows you to either define the pipeline script directly in the job configuration or point to a Jenkinsfile in your source repository (Declarative Pipeline). We’ll assume you are using a Jenkinsfile in the repo for a Multibranch Pipeline or similar.

2. Write the Jenkinsfile

At the root of your project, create a file named Jenkinsfile. Here’s an example Declarative Pipeline that installs dependencies and runs Playwright tests:

pipeline {
    agent any  // runs on any available agent (ensure an agent has Node & browsers)
    stages {
        stage('Install Dependencies') {
            steps {
                sh 'npm ci'
                sh 'npx playwright install --with-deps'
            }
        }
        stage('Run Tests') {
            steps {
                sh 'npx playwright test'
            }
        }
    }
    post {
        always {
            archiveArtifacts artifacts: 'playwright-report/**', allowEmptyArchive: true
        }
        failure {
            echo 'Tests failed! Check the report for details.'
        }
    }
}

Explanation of the Jenkinsfile

  • Declarative Pipeline Syntax: The agent any means it will run on any available Jenkins agent. You could specify a particular agent or docker container as well (e.g., agent { docker { image 'mcr.microsoft.com/playwright:<version>-<tag>' } } to run inside the official Playwright Docker image).
  • Stage: Install Dependencies: Runs npm ci to install Node packages, then npx playwright install --with-deps to set up the browsers (similar to earlier steps).
  • Stage: Run Tests: Executes the Playwright test suite.
  • Post Section: Always archive the test report directory (so Jenkins stores it as a build artifact). We use a file glob playwright-report/** to grab the entire report folder. We allow empty archive just in case the folder doesn’t exist (so the step doesn’t fail if tests didn’t produce a report). On failure, we print a simple message to the console log. You could also add a success block or other notifications (e.g., send an email or Slack notification via plugin) in this post section if desired.

Save this Jenkinsfile in the repo and push it. In Jenkins, create a Pipeline job pointing to the repository (if using Multibranch, Jenkins will auto-detect the Jenkinsfile). Run the job – Jenkins will checkout the code and execute these steps.

Running Playwright Tests in Parallel

One of the key benefits of CI pipelines is the ability to speed up test execution by running tests in parallel. Playwright is designed to run tests concurrently by default – it launches multiple worker processes to run test files simultaneously, which can dramatically reduce overall test time. In this section, we’ll explore how to configure and maximize parallel runs in your CI environment, across all three tools.

Playwright’s Built-in Parallelism (Workers)

By default, Playwright Test will spawn a number of workers equal to the number of CPU cores on the machine (when running via npx playwright test). On CI machines with limited cores, it’s wise to explicitly set the number of workers to avoid oversubscription. For example, GitHub’s hosted runners have 2 cores, so running with the default might use 2 workers (which is fine). But if your CI agent had 16 cores, Playwright might try to use all 16 by default, which could overload the machine if it’s not that powerful or if other processes are running.

You can configure the number of workers in your playwright.config.ts (or .js) file:

// playwright.config.ts
import { defineConfig } from '@playwright/test';

export default defineConfig({
  workers: process.env.CI ? 4 : 1,
  // ... other configs ...
});

In this example, we use 4 workers in CI (if the CI env var is set) and 1 locally (to avoid stressing a dev machine). Adjust the number based on your CI machine’s CPU. For a 2-core machine, 2 workers might be ideal. The goal is to use parallelism but not to the point of causing resource thrashing.

Alternatively, you can set the workers via CLI argument when running tests: npx playwright test --workers=4. This might be convenient if you want to dynamically decide in the CI script.

Test Sharding Across Multiple Machines

Playwright also supports test sharding, which is running different subsets of tests on multiple machines in parallel. This is useful for very large test suites or if you want to parallelize beyond one machine’s capacity. The idea is to split tests, say into N groups, and run N separate jobs in parallel, each responsible for 1/N of the tests. Playwright’s CLI has a --shard=<current>/<total> option for this.

For example, suppose you have 100 tests and want to run 2 jobs in parallel (perhaps on two different CI runners). You could run one job with --shard=1/2 and another with --shard=2/2. Playwright will ensure each job runs a different half of the test suite.

Managing Test Artifacts and Reports

When running tests in a CI environment, you won’t have a browser window popping up that you can watch. All interactions are headless. Therefore, capturing artifacts (test reports, screenshots, logs, videos) is critical for debugging failures and reviewing test results.

Playwright HTML Reports

Playwright can generate a comprehensive HTML report of your test run. To enable this, ensure your Playwright config includes the HTML reporter. For example, in playwright.config.ts:

reporter: [['html', { outputFolder: 'playwright-report' }]]

This configuration will output an HTML report (and associated assets) into playwright-report/ directory by default. We saw in previous sections how each CI tool can preserve this folder:

  • GitHub Actions: using actions/upload-artifact to collect the report folder.
  • GitLab: specifying artifacts: paths: - playwright-report/ in the YAML.
  • Jenkins: using archiveArtifacts 'playwright-report/**' in the post section.

With the report artifact available, a developer can download it and open index.html to see a nice UI of test results, complete with logs, errors, and even screenshots if configured.

Best Practices for Playwright CI/CD Integration

Now that we’ve covered the how-tos, let’s highlight some best practices (or “cat commandments” 🐾) for successful Playwright integration in CI/CD. These tips ensure your tests run smoothly, reliably, and efficiently:

  • Prepare Your Tests for CI: Make sure your tests are robust and not flaky. Leverage Playwright’s built-in waiting mechanisms (like await expect(element).toHaveText(...) rather than arbitrary sleep calls) so that timing issues don’t cause random failures.
  • Use Environment Variables and Config: Parameterize your tests for different environments. For example, use process.env.BASE_URL in your tests or config to switch between dev/staging/prod URLs. CI pipelines can set these env vars (GitHub Actions and GitLab CI allow defining env: or using secrets). This avoids hard-coding values and makes tests portable.
  • Leverage Official Docker Images: For consistency across local and CI runs, consider using the official Playwright Docker image. This guarantees the same browser versions and dependencies.

Scaling Tests with BrowserCat (Cloud Execution)

We’ve seen how to set up Playwright on Jenkins, GitHub, or GitLab CI – which usually involves installing browsers on the CI machines or using Docker images. Maintaining the infrastructure for browser tests (ensuring the right browser versions, dealing with resource limits, etc.) can sometimes feel like fighting a catzilla monster. This is where BrowserCat comes in to simplify things by offering cloud-hosted browsers for your tests.

What is BrowserCat? It’s a cloud service that hosts a fleet of browsers so you don’t have to. With BrowserCat, you can run Playwright (or other automation) by connecting to their remote browsers with just a single line of code.

Jenkins vs GitHub Actions vs GitLab CI: Which to Choose?

We’ve now seen that all three CI/CD tools can get the job done for running Playwright tests. But how do you decide which one is best for your needs? It might feel like comparing lions, tigers, and cheetahs – all are capable hunters, but each has a different habitat and style. Let’s compare some key aspects side by side:

AspectJenkinsGitHub ActionsGitLab CI/CD
HostingSelf-hosted server/agents that you manage.Cloud-hosted (managed by GitHub) for GitHub repos; also supports self-hosted runners.Cloud-managed for GitLab.com; or self-managed GitLab Runner.
ConfigurationJenkinsfile (Groovy pipeline) in repo or job config.YAML workflow files in repo (.github/workflows).YAML pipeline file in repo (.gitlab-ci.yml).
Ease of SetupSteeper learning curve, must set up Jenkins and agents.Easy start, especially if repo is on GitHub (just add YAML).Moderate – similar to GH Actions if using GitLab SaaS.
IntegrationWorks with any source control (GitHub, Bitbucket, etc.) via webhooks.Natively integrated with GitHub repositories and pull requests.Natively integrated with GitLab repositories (and GitLab’s merge request system).
Plugins/ExtensibilityThousands of plugins for anything (reporting, notifications, etc.).Marketplace of pre-built actions; can write custom actions.Fewer plugins (GitLab CI is more built-in), but can call scripts or use Docker images for extensibility.
ParallelismHighly flexible (parallel stages, multiple agents), but manual setup.Matrix builds and parallel jobs supported out-of-the-box. Easy to configure.Parallel jobs and matrices supported via YAML (parallel: keyword).
InfrastructureYou maintain the servers (which can be good or bad). Need to ensure agents have required software.GitHub’s hosted runners come with common tooling; no maintenance on your part.GitLab shared runners similarly come pre-configured; or maintain your own runner.
CostJenkins is free open-source, but you pay for the hardware to run it.Free for public repos; free minutes quota for private (paid beyond that).Free tier with limited minutes; higher tiers for more runners or self-host your own.
When to useGreat if you need full control, custom environments, or have many projects (and don’t mind upkeep).Great for projects on GitHub that want quick CI setup and tight integration (checks on PRs, etc.). Minimal maintenance – just write YAML and go.Great if you’re on GitLab or prefer its interface. Good integration with GitLab’s features (issues, MR, container registry). Also offers on-prem (self-hosted GitLab) option for an all-in-one DevOps platform.

Conclusion

Integrating Playwright with your CI/CD pipeline is a game-changer for software quality – it ensures every code change gets a rigorous check in real browsers, all without manual intervention. In this guide, we’ve walked through how to set up Playwright tests in GitHub Actions, GitLab CI/CD, and Jenkins, comparing the approaches and highlighting best practices. From simple one-command pipelines to complex parallel executions, you have many options to tailor the setup to your team’s needs.

We kept the tone playful with some feline fun, but the takeaways are serious:

  • Automation is key to catching issues early and often.
  • Each CI tool has its strengths; choose what fits your project and expertise.
  • Don’t be afraid to run tests in parallel – your time is precious, so let the machines do more in less time.
  • Use artifacts and reports to gain visibility into headless test runs.
  • Maintain your tests and pipeline with care (flakiness is the enemy, but can be tamed).

Finally, consider leveraging cloud solutions like BrowserCat for effortless scaling. Instead of managing a herd of browser instances yourself, let a cloud service handle the heavy lifting. As we’ve seen, BrowserCat can make your pipeline paw-some by providing cloud browsers on-demand, so your tests run faster and more reliably – all with a tiny change to your code.

We hope this guide has demystified Playwright CI/CD integration and perhaps even inspired you to augment your pipeline (or should we say pipe-feline?). With the right setup in Jenkins, GitHub, or GitLab – and maybe a sprinkle of cloud magic – you’ll achieve a smooth, continuous testing process that gives you nine lives’ worth of confidence in every release.

Now go forth and automate – may your tests be green and your deployments be purr-fectly seamless! 🎉🐾

Automate Everything.

Tired of managing a fleet of fickle browsers? Sick of skipping e2e tests and paying the piper later?

Sign up now for free access to our headless browser fleet…

Get started today!