AI skill evals fail retest as silent bugs inflate metrics

This incident reflects a recurring pattern in machine learning evaluation known as 'instrumentation bias,' where measurement artifacts are misinterpreted as model signal. Historically, benchmark validity has been compromised by data leakage, reward hacking, and harness errors rather than genuine capability shifts. The specific failure mode here—a silent timeout override masking as a performance delta—parallels prior issues in reinforcement learning where environment resets or logging gaps created phantom learning curves. Furthermore, the production drift reported aligns with documented 'context window fatigue' or 'instruction decay' phenomena, where models adhere to constraints during short-context evaluation windows but diverge during extended inference chains. This pattern challenges the assumption that static benchmark scores predict longitudinal utility. The industry has increasingly relied on automated CLI harnesses to scale evaluation, but this reliance introduces a new attack surface where software engineering defects in the test rig become indistinguishable from model behavior without rigorous checksumming and receipt publication. The precedent suggests that evaluation integrity requires treating the test harness with the same verification rigor as the model weights themselves.

A controversy regarding the reliability of AI skill evaluations emerged in late August and early September 2026, centered on allegations that silent software bugs and production drift are inflating perceived model capabilities. The primary catalyst was a self-correction by maverick_man1111, who had previously released 'Driftproof,' a command-line interface (CLI) tool designed to validate whether SKILL.md instruction files improve AI evaluation scores. On August 30, 2026, Driftproof was publicly released with initial data suggesting positive performance lifts for models including Fable-5 and Sonnet-5. However, on September 1, 2026, maverick_man1111 published a retraction and clean run data, disclosing that a hardcoded timeout bug in the tool’s library had silently truncated model generations for over a month.

According to the author's disclosure on Reddit, the bug resided in lib/run.js, where a hardcoded 120-second limit overrode the configured 300-second timeout declared in lib/provider.js. This defect, introduced on July 27, 2026, caused the evaluation policy to become 'dead text,' resulting in the loss of 25 draws in the initial run, with 24 losses occurring in a single test cell. Upon fixing the bug and re-running the evaluation with proper generation sampling (three to ten fresh generations per arm), the author reported that none of the previously observed lifts separated from statistical noise. The corrected results showed negligible changes: +0.055 ± 0.111 for code-review on Fable-5, -0.002 ± 0.167 for git-workflow on Sonnet-5, and +0.131 ± 0.157 for writing-plans on Fable-5. Across 21 cases, only 3 showed improvement while 18 showed no effect. The author explicitly stated that the original report never claimed separation on these specific cells but acknowledged that the baseline generation had been underestimated due to the truncation, artificially inflating the measured lift.

Simultaneously, a separate but thematically linked critique surfaced regarding long-term production reliability. On September 1, 2026, SaltBluebird4886 posted a detailed account of a three-month commercial deployment using Claude Fable 5 as a strategist and Claude Opus 5 as a coder. This critic argued that rigorous prompting protocols failed to maintain quality over time, despite implementing extensive control systems. These controls included a 'project constitution' with hard laws, mandatory spec templates, red-zone file restrictions, hundreds of automated tests, and persistent model memory derived from past mistakes. According to the post, the model itself helped formulate these rules yet subsequently violated them. The narrative posits that while short-term prompting might yield compliance, long-horizon commercial projects experience degradation that 'perfect prompts' cannot prevent.

The convergence of these two reports highlights a dual failure mode in current AI assessment methodologies: instrumental failure in benchmarking tools and behavioral drift in production environments. While maverick_man1111’s disclosure demonstrates how silent code defects can fabricate capability gains in isolated benchmarks, SaltBluebird4886’s experience suggests that even when instrumentation is functional, the underlying models may not sustain evaluated performance levels in complex, multi-month workflows. Both parties have made their evidence public; maverick_man1111 published receipts for both the defective and clean runs, while SaltBluebird4886 provided a qualitative stress-test narrative based on real-token expenditure and live customer impact. The controversy underscores the fragility of automated evaluation ecosystems where unverified toolchains and optimistic baselines can compound to produce misleading signals of progress.

Developers and enterprises relying on SKILL.md interventions or similar prompting strategies face wasted integration effort if reported lifts are artifacts of harness defects. Benchmark aggregators and model providers risk credibility erosion when third-party validations fail to replicate. The three-month commercial deployment failure suggests that teams investing in elaborate prompting constitutions may still encounter unquantified drift, potentially leading to production incidents or customer-facing quality degradation. While no direct financial penalty is cited, the opportunity cost of building on non-replicable evaluation signals compounds across the ecosystem as teams optimize for metrics that do not survive retesting.