An 11.5% acceptance lift still needs a secret-leak replay

GitHub’s 13 October technical report describes a two-stage detector for generic secrets. Regular expressions generate high-recall candidates; a quantized ModernBERT classifier then reads the value, file path, secret type and nearby text to separate likely credentials from placeholders, checksums and test fixtures.

GitHub reports that, on the same public traffic, the new pipeline produced 13% higher precision and 19% higher recall than its previous LLM pipeline. In a customer A/B experiment, repository-owner-weighted acceptance rose 11.5%. Median post-push scan execution time fell 86%, and the compact classifier cost more than 98% less to serve. The company says weighting acceptance by repository owner reduces distortion from customers who resolve alerts in bulk.

Those are stronger details than a single benchmark headline, but they remain vendor evaluation. The public article does not disclose raw counts, confidence intervals, threshold-specific confusion matrices or subgroup results by language and repository type. “Acceptance” records how users resolved a finding; it is not independently adjudicated ground truth. A leaked credential that users never see sits outside that denominator.

The method also uses product-derived examples, repository candidates, human annotation, weak labels and synthetic hard negatives. GitHub says examples sharing a repository, candidate value or example were grouped before splitting, which reduces obvious leakage. It does not establish transfer to an organisation’s private monorepos, generated fixtures, unusual configuration formats or domain-specific secrets.

Reproduce the decision boundary on your loss function

Before expanding coverage or enabling push blocking, build a sealed replay set from rotated real secrets, harmless lookalikes and previously missed incidents. Keep the values unusable and restrict access. Stratify by repository type, language, generated code, documentation, tests and deployment configuration. Do not tune on the final replay.

Measure precision and recall at the actual threshold, then add operational outcomes: confirmed usable credentials prevented, true secrets suppressed, median block latency, override rate and time to rotate after a miss. Review false negatives separately because one suppressed production credential can outweigh many avoided false alarms.

Run the current and proposed pipelines on identical commits without exposing developers to which system produced the alert. Predeclare a stop rule for any protected secret family whose recall worsens or any repository class with an unexplained gap. If the classifier passes, roll out by cohort and retain deterministic high-confidence detectors as a separate layer.

The training lesson is equally narrow. Teams need people who can design a hard-negative set, interpret threshold trade-offs and investigate misses; they do not need a generic lesson that “smaller models are better.” Record the control under AI data security, with the blind replay and rollback owner attached.