6.2 Slack Real-Time Failure Alerts & Test Run Digests

How to configure real-time Slack channel notifications when test suites complete or quality gates fail.

Alerting Lives on the Suite, Not a Global Toggle

Slack notifications are configured per test suite rather than as one organization-wide switch, because different suites usually deserve different attention – a nightly smoke suite failing at 2am might need to page someone immediately, while a slower-moving regression suite that only runs before a release is fine surfacing in a digest nobody has to react to in real time. Each suite carries its own alerting settings: a target Slack channel, a toggle for whether only failures should post or every run should post (the digest behavior this guide’s title refers to), and a threshold for how many consecutive failures need to happen before Slack hears about it at all.

Why a Consecutive-Failures Threshold Exists

A single failing run isn’t necessarily worth a page – it might be exactly the kind of one-off flake the Flake Governance system is already tracking toward auto-quarantine, and a channel that pages on every isolated blip trains people to ignore it. Setting the threshold above 1 means Slack only fires once a case has actually failed multiple runs in a row – by which point it’s a real, not-yet-quarantined problem rather than noise. A quarantined test failing doesn’t count toward the threshold either, for the same reason it doesn’t fail the CI build: it’s already flagged, and someone is already on it.

Configuring a Suite’s Slack Channel

• 1. Open the suite’s own Settings tab and find its Alerts panel.
• 2. Paste the Slack Incoming Webhook URL for the channel you want notified, or select a previously connected channel.
• 3. Choose whether to alert on every run (a digest, useful for suites people want a heartbeat from) or on failures only.
• 4. Set the consecutive-failures threshold – 1 to alert immediately, higher to require a real streak before Slack hears about it.
• 5. Save – the setting takes effect starting with the suite’s next run.

What Actually Lands in the Channel

• The message includes the run’s pass/fail counts, a direct link back into the Run Report, and the names of the specific cases that failed, so the first thing anyone sees in Slack is enough to decide whether it’s worth clicking through.
• A suite currently posts to one channel. If different environments need different routing – say, staging failures to a QA channel and production drift alerts to an on-call channel – give each environment its own suite or schedule rather than trying to route one suite’s alerts two ways.