5 of 5 free conversions left. Sign in for unlimited use.
PlayRight
A small open-source plugin for Playwright Test (JavaScript / TypeScript) and pytest-playwright (Python). When a test fails it tells you what kind of failure it looks like, shows the evidence behind that guess and suggests fixes with code you can paste. It is a transparent rule engine (32 weighted rules and 22 fix recipes in one shared data file), not a trained model, and it runs entirely on your machine.
The six categories
| Category | Meaning |
|---|---|
| locator | The element could not be found or identified: the selector no longer matches the page. |
| timing | The element or state exists but was not ready in time: a race, a slow step or a hard-coded wait. |
| data | The test depends on data or state that changed: dynamic values, duplicates, expired logins, leftovers from another test. |
| app-bug | The application behaved differently from what the test expects, consistently: probably a real defect. |
| environment | The test could not run properly: server down, network, certificates, browsers not installed. |
| unknown | Not enough evidence to classify this failure. |
Install and configure
Playwright Test (JavaScript / TypeScript)
npm install --save-dev playright-reporter
// playwright.config.ts
import { defineConfig } from '@playwright/test';
export default defineConfig({
retries: 1, // retries let PlayRight tell intermittent from consistent failures
reporter: [
['list'],
['playright-reporter', { outputDir: 'playright-report', language: 'javascript' }],
],
});
pytest-playwright (Python)
pip install tf2cdk # registers the pytest plugin; it stays inactive until you ask for it pytest --playright # triage failures, Python fix snippets pytest --playright --playright-language javascript pytest --playright --playright-dir out/triage pytest --playright --playright-ai # optional note from a local model
Both write playright-report.json and playright-report.md and keep .playright/history.json (runs and failures per test). Add that folder to .gitignore or cache it in CI. The pytest plugin does not support pytest-xdist.
How it decides
- It reads the error message and call log, the stack, the failing source line and the captured output, plus facts such as "failed on every retry" or "passed on a retry".
- Each rule is a set of regular-expression conditions with a weight. Every rule that matches adds its weight to its category.
- The highest-scoring category wins. Below a score of 4 the answer is unknown. Confidence is
top / (top + second + 3). - The titles of the matching rules are shown as evidence, close alternatives are listed, and the fixes attached to the winning rules are rendered with your locator, timeout and URL filled in.
The JavaScript reporter and the Python engine read the same rules file and are tested against the same cases, so they give the same answer.
Honest limits
- It reads text. It does not open your trace, screenshots or the page, so it cannot see why an element is missing.
- A verdict is a lead, not a proof. "Locator" and "app-bug" can look alike when an element is missing because the app broke.
- Fix snippets are patterns: check them against your page before using them, and never raise timeouts to hide a real bug.
- Unusual failures end up as unknown on purpose rather than being guessed.
Privacy
The plugins make no network calls. The optional AI note talks only to a model server on your own machine or private network (Ollama or vLLM), is switched off by default and is labelled unverified; it never changes the rule verdict. On this website pasted output is analyzed in memory and not stored.
About the PlayRight Playwright Plugin
PlayRight is a small open-source plugin for Playwright Test and pytest-playwright. When a test fails it classifies the failure as locator, timing, data, app-bug, environment or unknown, lists the rules that fired as evidence and suggests fixes with JavaScript or Python code. It is a transparent rule engine with weighted regular expressions in one shared data file, not a trained model, and the plugin makes no network calls.
How to use the PlayRight Playwright Plugin
- Install the reporter for Playwright Test (npm) or the pytest plugin that ships with tf2cdk (pip) and add it to your config or pytest command line.
- Run your tests with retries enabled and, for pytest, with the --playright option so failures are triaged.
- Read the verdict under each failure or in playright-report.md, then apply the suggested fix after checking it against your page.
Frequently asked questions
What does PlayRight decide?
For each failed test it picks the most likely cause among locator, timing, data, app-bug and environment, or says unknown when no rule scores high enough. It shows the matching rules, a confidence number and close alternatives, so you can see why.
Is it machine learning?
No. It is a rule engine: regular expressions over the error message, call log, stack, source line and output, each with a weight. An optional note from a local language model can be added, but it is labelled unverified and never changes the rule verdict.
Does it work with pytest-playwright?
Yes. The pytest plugin is inactive until you run pytest --playright, then it prints the triage under the failures and writes playright-report.json and playright-report.md. It does not support pytest-xdist.
How does it tell flaky from consistent failures?
It uses the retry results of the current run (passed on a retry, failed on every attempt, same message each time) and a small history file with the number of runs and failures per test.
Does PlayRight send my test output anywhere?
The text you submit is processed in memory to produce the result and is not saved. Like any web server, Code Converter logs basic visit data (page, IP address and browser) but never the content you paste.