Re-scanning after a fix: how Veracly tracks compliance regression

The reason continuous monitoring exists is that the vast majority of compliance bugs are introduced after the original audit, not during it. A marketing team ships a new analytics pixel; a partner agency drops in a Hotjar tag for a campaign; a redesign re-orders the cookie banner with a different reject-all path. None of those changes touched the page that was originally audited, and all of them break compliance.

Veracly’s scheduled re-scan plus score-drop alert is the answer. Here is how the mechanic actually works, stated narrowly, because the useful version of this article is the one that tells you what the alert does not watch.

What the re-scan covers

The same scope as your initial scan: every page the crawler reaches inside your configured page budget (fifty on Starter, two-fifty on Growth, five hundred on Pro, a thousand on Agency). Discovery is breadth-first from your homepage, following internal links to a depth of two, on the same hostname only.

We do not seed the crawl from your sitemap. Veracly fetches and honours robots.txt, and it parses the Sitemap: directive it finds there, then deliberately ignores it. A sitemap is a claim about what a site wants indexed, not a map of what a visitor can reach. A new page enters the scan set once something inside the crawl depth links to it.

Every scan produces a full report, same structure, same rule set, same PDF. The history lives on the site detail page as an attestation timeline: one row per completed scan, with the per-jurisdiction scores on that date and a link to the signed verification record for that report. That is the surface for reading the trend, and you can publish it as a link for a regulator or a buyer.

What the timeline is not is a diff. Veracly does not compute a finding-level comparison between two scans, so working out which issues appeared or disappeared is still a two-window read of the two reports. We would rather say that than sell you a diff view that does not exist.

What triggers a score-drop alert

One signal, and it is narrower than most monitoring copy implies. After each scan we take the average score across every jurisdiction evaluated for that site and compare it with the average from the previous completed scan of the same site. If the new average is lower by at least your alert threshold, the report goes out flagged as a score drop.

• The comparison is on the average, not on any one card. A ten-point fall on the GDPR card, with the other four cards holding steady, moves a five-jurisdiction average by two points and does not fire.
• The default threshold is 10 points. Settable anywhere from 1 to 100 under Account → Settings. It is an organization-level setting, applied to every site you monitor, not a per-site one.
• The first scan of a site never fires. There is no prior scan to compare against, so the report is delivered as an ordinary report-ready.

And the thing that does not trigger an alert: a new critical finding. The alerting path reads scores; it never reads the finding list. A brand-new pre-consent pixel that costs you eight points against a ten-point threshold arrives as a normal report-ready email, and you find it by opening the report. If net-new criticals are what you care about most, lower the threshold to three or four and accept the extra noise, that is the lever that exists.

Delivery is by email to the organization owner, always. Slack is additional and optional on Growth, Pro, and Agency: paste an incoming-webhook URL under Account → Settings. Both go out the moment the report finishes rendering. There is no alert banner in the dashboard.

What the alert email actually contains

It is the report-ready email with a red block at the top. In full:

• A subject line naming the domain and the fact that the score dropped.
• The drop block: previous average, new average, and the delta.
• The new average score and the number of jurisdictions it spans.
• A download button for the PDF.
• A line on how long that link stays live, seven days for a paid presigned URL, sixty days for a free scan.

It does not list the findings that moved, does not name the affected URLs, and does not link to a diff. Everything diagnostic is in the PDF. The email is a notification, not a report, and we are not going to push finding text into an inbox we do not control.

The Slack message carries the same numbers, the new average, the per-jurisdiction scores, and the drop, plus a download button.

Investigating a regression

One workflow that works:

1. Download the new report from the alert email and open the per-jurisdiction scorecard. That tells you which card moved.
2. Open the attestation timeline for the site under Account → Sites → [domain]. It lists the per-jurisdiction scores for every completed scan, so you can see whether the card fell in one step or drifted over several runs.
3. Open the previous report alongside the new one. There is no automated diff, so isolating what is new is a side-by-side read of the two Top Priorities tables. It takes a few minutes and it is the only way to get there today.
4. Look at the affected URLs. If they are clustered on one page or one path, the regression came from a change on that page. If they are spread across many pages, the regression came from a global change (a layout component, a CDN-loaded widget, a CMP configuration change).
5. Check the deploy log for the same window. Most regressions are traceable to a specific deploy or tag-manager change since the previous scan.
6. File one ticket per new finding. The remediation appendix in the new report has the fix snippet, same as the first scan.
7. Hit Scan now once the fix lands. The button is on the site detail page; do not wait for the next scheduled scan.

The regressions this actually catches

Worth naming the list concretely, because it is narrower than the word “compliance” suggests:

• A new tracking pixel firing before consent. The most common real regression, and the one the scanner handles best: a marketer adds a tag through Tag Manager, nobody checks the consent gate, and the next scan records the request firing pre-consent, with a DevTools recipe you can reproduce yourself.
• A redesign that drops a contrast ratio, a form label, or an accessible name. These are axe-core violations and they move the score on the next run.
• A cookie-banner rebuild that loses reject-all parity, or loses the reject control altogether. Note the limit: parity is judged from the rendered DOM, not by clicking Reject and re-measuring. Veracly never clicks a consent banner, because a click would itself be a consent gesture and would pollute the pre-consent reading.
• A policy page that moved or started returning a 404. The privacy policy, accessibility statement, and imprint checks look for a reachable page on your own hostname; a relocated page reads as a missing one.

And the regressions it will not catch, which matter just as much when you decide how much weight to put on the score: a redesign that introduces a keyboard trap (Veracly never exercises the keyboard, and SC 2.1.2 is one of the twenty-eight criteria we mark manual and score-neutral), a cookie policy that has gone stale against the tags you actually run (there is no declared-versus-observed comparison), anything behind a login, and anything on a page the crawler cannot reach. Those need a person.

Tuning the threshold

If the default 10-point threshold is too noisy for your site, or too quiet, it is a single number under Account → Settings, valid from 1 to 100, applied across every site in the organization. There is no per-site override today.

Tune it after two or three normal-state scans rather than at signup. Large sites with a lot of third-party widgets do wobble by a point or two between runs, and averaging across jurisdictions damps that further, which is part of why the default sits at 10 rather than at 3.

See also: Reading your first Veracly report · The top 10 issues Veracly finds and how to fix them