Study finds LLM benchmark scores vary 8.4 points daily

Temporal instability in machine learning systems has historically been documented as concept drift or data drift, where model performance degrades as input distributions shift over time. In traditional MLOps, this is managed through continuous retraining and monitoring pipelines. However, in the context of proprietary LLM APIs, users lack visibility into backend updates, quantization changes, or routing adjustments that providers may deploy without notice. Previous studies on benchmark saturation have focused on metric gaming or contamination, where models memorize test sets, rather than temporal variance in live inference. The pattern observed here differs from contamination because it manifests as oscillation rather than monotonic improvement or sudden collapse. This aligns with emerging concerns about 'silent updates' in SaaS AI products, where versioning is opaque. The precedent for open-source evaluation infrastructure exists in projects like LM Evaluation Harness, but those typically prioritize breadth of tasks over longitudinal density. This case extends the pattern by treating time itself as a primary dimension of evaluation, shifting focus from 'how smart is the model' to 'how stable is the service'.

On August 29, 2026, Ionut Vi published a temporal variance analysis of Large Language Model (LLM) API performance, presenting findings derived from 31,352 hourly benchmark scores collected across multiple providers and model families. The analysis, which utilized the open-source AIStupidLevel infrastructure, reported a significant discrepancy between short-term and long-term performance stability. According to the published data, within-day variation in composite benchmark scores averaged 2.8 points, whereas between-day variation reached 8.4 points. This finding suggests that model performance behind production APIs is subject to substantial daily fluctuations that exceed ordinary stochastic noise observed within shorter time windows.

The methodology employed for this analysis involved repeated measurements using consistent tasks and scoring parameters. Ionut Vi states that the evaluation pipeline tested models across coding, deep reasoning, tool calling, and high-frequency canary tasks. To mitigate the influence of individual generation anomalies, tasks were executed five times with results aggregated. Coding responses were validated through execution rather than solely through model-based judgment, and tool-calling tests required models to construct valid arguments and complete workflows within isolated Docker environments. Prompts, scoring logic, and API parameters were maintained consistently wherever supported by the provider. The resulting dataset covered 49 distinct model identifiers and utilized a normalized 0-100 composite score.

Ionut Vi, who identifies as neutral regarding specific model vendors but advocates for sustained monitoring over point-in-time evaluation, argues that these daily variations offer materially stronger signals for drift detection than hourly changes. The core assertion is that traditional static leaderboards, which typically measure performance at a single moment, may fail to capture this temporal instability. By distinguishing between 2.8-point hourly variance and 8.4-point daily variance, the analysis posits that reliance on static benchmarks could lead to inaccurate assessments of model reliability in production settings. The author explicitly frames the research question as investigating how stable models are over time and how sustained performance changes can be separated from ordinary variation.

The AIStupidLevel Project, identified here as the defender of the methodological validity, provides the MIT-licensed infrastructure used to generate these claims. The project’s frontend and backend code are openly available, allowing third parties to validate the data collection and scoring processes. This transparency serves as the primary defense against potential skepticism regarding the magnitude of the reported variance. By open-sourcing the evaluation system, the project aims to establish continuous monitoring standards rather than merely criticizing existing evaluation paradigms. The disclosure that Ionut Vi developed AIStupidLevel establishes a direct link between the analyst and the tooling, positioning the work as both a technical contribution and an empirical study.

The controversy centers on whether an 8.4-point daily swing represents genuine model drift, backend optimization cycles, or uncontrolled variables in the testing environment. While the analysis controls for prompts and scoring logic, it cannot control for undocumented server-side changes by API providers. Critics might argue that without access to provider deployment logs, attributing this variance to model instability rather than testing artifacts remains speculative. However, the sheer volume of data points—over 31,000 hourly scores—and the execution-based validation of coding tasks provide a robust empirical basis for the claim that performance is non-stationary. The distinction between within-day and between-day variance is critical; if the variance were purely stochastic, one would expect similar magnitudes across both time scales. The threefold difference observed implies a systematic, temporally correlated factor driving performance changes on a daily cadence.

This analysis challenges the prevailing assumption in the AI industry that model capabilities are fixed properties once deployed. If confirmed by independent replication, the findings necessitate a shift toward continuous evaluation frameworks. Stakeholders selecting models for safety-critical or high-reliability applications may need to account for this temporal uncertainty. The AIStupidLevel Project’s provision of open-source tooling lowers the barrier for such replication, potentially transforming this single analysis into a broader industry standard for temporal stability testing. Until then, the 8.4-point figure stands as a specific, quantified challenge to the reliability of static benchmarking practices.

Enterprises and developers selecting LLMs via static leaderboards face material risk of deploying models whose real-world performance deviates up to 8.4 points daily from evaluated baselines. This affects safety-critical applications, automated coding pipelines, and any system assuming deterministic model behavior. The 31,352-score dataset spans 49 models across multiple providers, indicating the issue is systemic rather than vendor-specific. Organizations without continuous monitoring may experience unexplained failures or quality degradation. Conversely, vendors demonstrating temporal stability gain competitive advantage. The open-source AIStupidLevel infrastructure enables low-cost adoption of temporal evaluation, potentially reshaping procurement criteria away from peak benchmark scores toward stability metrics.