Astra and Fable criticized for reusing outdated alignment evals

The tension between rapid model scaling and evaluation latency is a persistent structural pattern in AI safety research. Historically, benchmark saturation precedes methodological crises, where static tests cease to discriminate between capable models and instead measure test-set memorization or proxy optimization. This cycle has repeated across domains from natural language understanding to code generation, typically triggering a 'benchmark retirement' phase followed by the introduction of dynamic or adversarial evaluation frameworks. The current dispute mirrors prior controversies where leaderboard dominance was decoupled from real-world utility due to eval drift. In alignment specifically, the transition from behavioral compliance checks to mechanistic interpretability or adversarial robustness testing often lags behind capability jumps by 12-18 months. This lag creates windows where safety claims rely on proxies that researchers consider insufficient for the new capability frontier. The pattern suggests that evaluation obsolescence is not merely a company-specific failure but an endemic feature of exponential capability growth outpacing measurement science.

On September 13, 2026, a controversy emerged within the AI safety and evaluation community regarding the alignment testing methodologies employed by Astra and Fable. The dispute centers on allegations that both companies continue to rely on outdated evaluation frameworks originally established in early 2025, despite significant advancements in model capabilities since that time. According to a post on Hacker News by user Levitating, both Astra and Fable are accused of 'hacking on simple variants' of these older benchmarks rather than developing robust assessments for current-generation risks [1]. This criticism suggests that the safety evaluations currently used to validate models like GPT-6 Astra and Claude-5-Fable may be functionally obsolete, potentially creating a false sense of security regarding their alignment.

The core allegation is that the benchmarks in question were designed for an earlier generation of models and fail to capture the nuanced failure modes or dangerous capabilities of systems deployed in late 2026. User Levitating’s critique draws a direct parallel to 'cheating in chess,' implying that optimizing for these static tests does not correlate with genuine safety or alignment but rather represents a form of metric gaming [1]. This perspective posits that high scores on these legacy evals are artifacts of overfitting rather than evidence of reliable behavior. The timing of this critique coincides with broader industry turbulence regarding benchmark validity; separate reports indicate that Artificial Analysis recently replaced its own intelligence index after discovering similar optimization issues, suggesting a systemic challenge in maintaining evaluation integrity as models improve rapidly [2].

Despite the specificity of these allegations, neither Astra nor Fable has issued a public response as of the latest available information. Their silence leaves several key factual questions unresolved, including whether they have internally updated their evaluation suites without public disclosure, or if they genuinely rely on the 2025 variants criticized by researchers. The lack of rebuttal also makes it difficult to determine if the companies dispute the characterization of their methods as 'hacking' or if they acknowledge the limitations but face technical constraints in developing next-generation evals. In the absence of official statements, the narrative remains driven entirely by external critics and independent observers who argue that the gap between benchmark performance and actual safety assurance has widened dangerously.

The controversy highlights a recurring tension in AI development: the lag between model capability and evaluation methodology. Critics argue that reusing 2025-era tests for 2026-era models is akin to using a high school math test to evaluate a PhD candidate; it measures basic competence but fails to assess advanced reasoning or novel risks. The comparison to chess cheating underscores the adversarial nature of modern evaluation, where models (or their developers) may exploit known patterns in test sets to achieve superficially impressive results. While the specific technical deficiencies of the 2025 benchmarks are not detailed in the available sources, the implication is that they lack the dynamic or adversarial components necessary to stress-test contemporary foundation models effectively.

This incident occurs against a backdrop of increasing skepticism toward proprietary benchmarks. The recent adjustment of the Artificial Analysis Intelligence Index, which saw leadership changes after replacing the τ³ benchmark, demonstrates how volatile performance rankings can be when evaluation criteria shift [2]. Such volatility reinforces critic arguments that static benchmarks are insufficient for tracking true progress or safety. For Astra and Fable, the stakes involve not just reputational damage but the fundamental credibility of their safety claims. If their alignment assurances rest on tests that the research community deems obsolete, stakeholders may question the reliability of their deployments until more rigorous validation methods are demonstrated or disclosed.

AI safety researchers lose confidence in published alignment metrics, complicating peer review and collaborative safety standards. Downstream enterprise customers deploying Astra or Fable models may operate under false safety assumptions if 2025 evals miss 2026-era failure modes. The broader ecosystem faces increased uncertainty about which safety claims are trustworthy, potentially slowing adoption or triggering premature regulatory intervention based on incomplete evidence. Magnitude is currently limited to reputational and epistemic harm; no financial penalties, user counts, or revenue figures are cited in available sources. The primary risk is cascading loss of trust in alignment evaluation as a discipline if multiple leading labs are simultaneously found to be gaming obsolete benchmarks.