Playwright is an end-to-end testing framework that allows robust and reliable testing of web applications. One of the critical aspects of any testing framework is its reporting capabilities. Detailed test reports are essential for understanding test results, identifying issues, and determining the status of your test suite. In this blog, we will explore the default test reports in Playwright, the different types of built-in reporters, and how to customize reports to meet your specific requirements.
Default Test Report in Playwright
When you run Playwright tests without specifying a reporter, it uses the list reporter by default. This reporter provides a simple output in the console, showing the status of each test (passed, failed, skipped), along with the duration of each test. This default output is helpful for quick test runs and local development but may not be sufficient for larger projects or CI/CD pipelines where more detailed reporting is required.
Different Types of Test Reporters in Playwright
Playwright comes with several built-in as well as external reporters that cater to different reporting needs. Here’s a brief overview of each:
- List Reporter:
- When using the List Reporter, each test is listed with its name, status (such as passed, failed, or skipped), and the time it took to run. This reporter provides a full overview of the test suite in a textual format.
- Features:
- A clear and comprehensive view of each test outcome.
- Useful for diagnosing issues in specific tests.
- Output can be verbose for large test suites.
- Dot Reporter:
- The Dot Reporter is minimalistic, printing a dot (.) for every test that passes and other symbols like F for failed tests. It provides a visual cue of test progress without much detail.
- Features:
- Provides a quick overview of test progress.
- Easy to see if the suite is running successfully.
- Less useful for detailed analysis as it doesn’t provide specifics about individual test results.
- Line Reporter:
- Like the Dot Reporter but uses a line (—) for each test. This can be a more visually appealing way to represent progress than a series of dots.
- Features:
- Offers a clear indication of test progress.
- Slightly more detailed than the Dot Reporter, making it easier to track progress through a test suite.
- Provides minimal feedback on individual test status.
- JSON Reporter:
- Output test results in JSON format, which is structured and hierarchical. This format includes details like test names, statuses, durations, and error messages.
- Features:
- Ideal for integration with other tools or systems that can process JSON data.
- Enables programmatic access to test results for custom reporting or analysis.
- Allows for automated parsing and handling of test results.
- JUnit Reporter:
- Produces test results in JUnit XML format, which is a standard format used by many CI/CD systems. This XML includes information about each test case, including its status, time taken, and any error messages.
- Features:
- Compatible with many CI/CD tools, making it easy to integrate with build and deployment pipelines.
- Provides detailed information that CI tools can use to generate reports or trigger actions based on test results.
- Suitable for teams using CI systems that rely on the JUnit format for test reporting.
- HTML Reporter:
- Generates an interactive HTML report that provides a rich, detailed view of test results. This report includes test names, statuses, and durations, and often incorporates screenshots and videos of test execution.
- Features:
- Provides a visual and interactive interface for viewing test results.
- Includes multimedia elements like screenshots and videos, which are useful for debugging failed tests.
- Ideal for sharing test results with stakeholders who prefer a graphical representation over text-based reports.
Customizing Your Report for Playwright Tests
While the built-in reporters are powerful, you might need more customization to meet specific requirements. Here’s a step-by-step guide to creating and integrating a custom reporter with Playwright.
Step1: Create a Custom Reporter
First, create a new file for your custom reporter, e.g., custom-reporter.js.
const { Reporter } = require('@playwright/test/reporter');
class CustomReporter extends Reporter {
onBegin(config, suite) {
console.log(`Starting the test suite with ${suite.allTests().length} tests.`);
}
onTestBegin(test) {
console.log(`Starting test: ${test.title}`);
}
onTestEnd(test, result) {
console.log(`Test ${test.title} ended with status: ${result.status}`);
}
onEnd(result) {
console.log(`Test suite finished with status: ${result.status}`);
}
}
module.exports = CustomReporter; Step 2: Integrate the Custom Reporter
Next, integrate the custom reporter into your Playwright configuration file, e.g., playwright.config.js.
const CustomReporter = require('./custom-reporter');
module.exports = {
reporter: [[CustomReporter, {}]],
}; Step 3: Adding Custom Metadata
You can enhance your custom reporter by adding custom metadata, such as test durations and other relevant information.
class CustomReporter extends Reporter {
onTestEnd(test, result) {
const metadata = {
title: test.title,
status: result.status,
duration: result.duration,
};
console.log('Test metadata:', metadata);
}
} Step 4: Including Screenshots and Videos
Playwright allows you to capture screenshots and videos, which can be included in your custom reports for better diagnostics.
// playwright.config.js
module.exports = {
use: {
screenshot: 'on',
video: 'retain-on-failure',
},
};
In your custom reporter, you can link to these artifacts.
const fs = require('fs');
const path = require('path');
class CustomReporter extends Reporter {
onTestEnd(test, result) {
if (result.status === 'failed') {
const screenshotPath = path.join(__dirname, 'screenshots', `${test.title}.png`);
const videoPath = path.join(__dirname, 'videos', `${test.title}.webm`);
fs.writeFileSync(screenshotPath, result.attachments.find(a => a.name === 'screenshot').path);
fs.writeFileSync(videoPath, result.attachments.find(a => a.name === 'video').path);
console.log(`Screenshot and video for ${test.title} saved.`);
}
}
}
Step 5: Generating HTML Reports with Custom Data
For a more polished report, you might want to generate an HTML file. You can use basic HTML and CSS to style your report.
class CustomHTMLReporter extends Reporter {
onEnd(result) {
const htmlContent = `
<html>
<head>
<title>Test Report</title>
<style>
body { font-family: Arial, sans-serif; }
h1 { color: #343a40; }
.test { margin-bottom: 20px; }
.test .status { font-weight: bold; }
</style>
</head>
<body>
<h1>Test Report</h1>
${result.suites.map(suite => suite.tests.map(test => `
<div class="test">
<h2>${test.title}</h2>
<p class="status">${test.results[0].status}</p>
</div>
`).join('')).join('')}
</body>
</html>
`;
fs.writeFileSync(path.join(__dirname, 'custom-report.html'), htmlContent);
}
}
module.exports = CustomHTMLReporter; Step 6: Automating Report Generation
Integrate the custom report generation into your CI/CD pipeline to automatically generate and publish reports.
# .github/workflows/playwright.yml
name: Playwright Tests
on: [push]
jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- name: Setup Node.js
uses: actions/setup-node@v2
with:
node-version: '14'
- run: npm install
- run: npm test
- name: Upload Test Report
uses: actions/upload-artifact@v2
with:
name: test-report
path: custom-report.html
Customized test report sample output
Conclusion
Customizing test reports in Playwright allows you to create detailed, readable, and actionable reports that can significantly enhance your testing process. By leveraging built-in reporters, creating custom reporters, and integrating with CI/CD pipelines, you can ensure that your test results are always clear and comprehensive.
Integrating these Playwright test reports into your CI/CD workflows helps in maintaining high-quality code by making it easier to catch issues early in the development cycle. This proactive approach to test reporting contributes to faster development cycles.
Thus, by customizing your test reports, you can greatly improve your development and testing workflows, making them more effective and aligned with your project’s specific needs.
