Merge gate
The merge gate puts a check named WeHarness evaluation on each pull request of a project’s GitHub repository. The check comes from an evaluation of the project’s approved cases (see Evaluations) on the pull request’s head commit. The code under test runs only in the evaluation’s isolated sandbox.
Turn it on
Section titled “Turn it on”An owner or admin turns the gate on for a project in the web app: Evaluations → Merge gate. A token or a coding host cannot turn it on.
After that, opening, reopening or pushing to a pull request starts one evaluation of its head commit. A newer push replaces the older commit’s evaluation, and the older check says so.
- Credits come from the workspace, recorded against the person who turned the gate on. The pull request’s author is never charged. If that person stops being an owner or admin, the gate pauses until someone turns it on again.
- Forks and outside authors never start an evaluation on their own. A maintainer starts it by adding the label
weharness:evaluate, or from the web app. - A daily cap limits the evaluations a project starts on its own each day. A maintainer’s start from the web app is not capped.
What the check says
Section titled “What the check says”| Conclusion | When |
|---|---|
| Success | Every attempt of every case passed. |
| Failure | A critical or mandatory case failed or regressed against the base branch, and the gate is set to block. |
| Neutral | Flaky or inconclusive results, or a failure when the gate only warns. Also when WeHarness could not run the cases: an outage, no credits, a paused gate, the daily cap, or no approved runner or case. The check says why. |
The gate never blocks a merge because WeHarness is down. Held-out cases appear only as counts.
Make it required
Section titled “Make it required”To stop a merge while the check fails, make it a required status check in GitHub:
- In the repository, open Settings → Rules → Rulesets (or Settings → Branches → Branch protection rules).
- Add or edit the rule for your main branch, and turn on Require status checks to pass.
- Add WeHarness evaluation to the required checks. It appears in the list once the check has run on one pull request.
As every non-blocking outcome is neutral, a required check stops only a real failure. A neutral check still lets the pull request merge.
Running the cases in your own CI
Section titled “Running the cases in your own CI”The cases run in WeHarness’s sandbox, not in your CI. A GitHub Action that runs them in your own pipeline needs its own credential, a workspace CI token, which is not available yet. If you need it, write to info@weharness.io.