Clean Engineering, Unstable Measurement: A Preregistered Reliability Failure of Black-Box LLM Observers on Shared Endpoints

Executive Summary

The rapid ascent of LLMs and AI agents has placed immense pressure on robust evaluation methodologies. We rely on LLMs to judge model outputs, gate training data, and determine leaderboard rankings, implicitly assuming they act as stable, consistent measurement instruments. A groundbreaking preregistered study, “Clean Engineering, Unstable Measurement: A Preregistered Reliability Failure of Black-Box LLM Observers on Shared Endpoints,” unequivocally shatters this assumption. The paper reveals a profound and systemic instability in how commercial black-box LLMs respond to identical requests, undermining the very foundation of current evaluation practices in Machine Learning. This isn’t a minor bug; it’s a fundamental challenge to our ability to reliably gauge progress or even reproduce results, demanding immediate attention from researchers and practitioners alike.

Technical Deep Dive

The core premise of modern LLM evaluation is that sending the same prompt to the same model name today should yield the same, or at least highly correlated, results tomorrow. This study rigorously audited this rarely stated but critical assumption through two preregistered campaigns, with all thresholds fixed in advance. The methodology was straightforward yet devastatingly effective: repeatedly querying commercial LLMs on shared endpoints and measuring the consistency of their responses, specifically their ranking judgments.

The findings are stark. Across an audit of 52,988 request attempts, the reliability metrics fell dramatically short of required targets.

• Same-window repeat rankings (sending the same request multiple times in quick succession) achieved a Spearman correlation of merely 0.400 against a required 0.90. This is akin to a measuring tape that changes its length mid-measurement.
• Byte-identical next-day replays (sending the exact same request 24 hours later) fared only slightly better at 0.78 against a required 0.99. Imagine a laboratory scale that gives significantly different readings for the same item day after day, despite being “calibrated.”

Crucially, this instability was not due to execution errors; the execution record was at ceiling, indicating the requests were processed successfully. The paper identifies three primary mechanisms driving this alarming gap:

1. Label-to-meaning mapping bias: The way LLMs interpret and map labels (e.g., a numerical score) to underlying meaning can be highly unstable, biasing readouts as strongly as the actual signal.
2. Candidate gaps below noise floor: The differences between candidates being judged were often seven orders of magnitude smaller than the instrument’s own internal noise floor, rendering the LLM effectively blind to subtle but critical distinctions.
3. Byte-identical input returning different rankings: Even when the input was precisely the same, the LLM returned different ranking decisions, a fundamental noise issue that exact-permutation readouts only exacerbate.

Further preregistered follow-ups explored potential mitigations. Waiting longer did not help. Switching providers (across four major LLM providers) revealed a shared floor of instability, with medians ranging from 0.74 to 0.88, none predicted by exposed metadata. Self-hosting on batch-invariant kernels offered only temporary relief, lasting only while the server was quiet. This points to shared serving infrastructure and non-deterministic execution as key culprits.

The authors distil their evidence into a three-level snapshot-identity ladder, eight design rules for robust evaluation, and a reporting checklist. They suggest that even a small pilot (2% of their study’s call volume) would have exposed these unreachable reliability gates before committing to large-scale, flawed evaluations.

Real-World Applications

The implications of this “Unstable Measurement” are profound and far-reaching for anyone working with LLMs and building AI agents:

• Training Data Gating: If LLM judges are used to filter or curate training data, the instability means that “good” data might be inconsistently admitted or rejected, leading to less reliable downstream models. This directly impacts the quality of future Machine Learning models.
• Leaderboards and Benchmarking: Current LLM leaderboards, which often rely on LLM-as-a-judge evaluations, are fundamentally compromised. The observed unreliability means rankings could be arbitrary or change day-to-day, making objective comparisons impossible. We might be celebrating illusory progress.
• RLHF/RLAIF Pipelines: Reinforcement Learning from Human Feedback (RLHF) and AI Feedback (RLAIF) critically depend on consistent reward signals. If the LLM judge providing that signal is unstable, the entire learning process becomes noisy and inefficient, potentially leading to misaligned or suboptimal AI agents.
• Model Safety and Alignment: Evaluating model safety and alignment is paramount. If the instruments used to detect harmful outputs or misaligned behaviors are unreliable, our ability to identify and mitigate risks is severely hampered. A seemingly safe model today could be judged unsafe tomorrow, not because its behavior changed, but because the measurement instrument wavered.
• Agent Decision-Making: For advanced AI agents that use LLMs for self-reflection, planning, or assessing their own performance, this instability introduces a dangerous level of unpredictability. An agent might misinterpret its environment or performance, leading to poor decisions or unexpected behaviors.
• Reproducibility Crisis: The lack of measurement stability directly contributes to a reproducibility crisis in LLM research and development, making it difficult for others to verify claims or build upon prior work.

Future Outlook

This paper isn’t just a critique; it’s a call to action. In the next 2-3 years, we must shift from a naive assumption of LLM judge stability to a rigorous, instrument-validation-first approach.

• Robust Evaluation Protocols: The industry needs to develop and standardize robust evaluation protocols that explicitly account for and measure instrument reliability before any gates are frozen. This might involve multi-shot averaging, confidence-interval-based judgments, or even ensemble-based judging, though the paper indicates that current substitutions didn’t fully repair the issue.
• Transparency and Control: A push for greater transparency from LLM providers regarding model versions, serving infrastructure, and execution determinism will become critical. The inability to predict stability from metadata suggests a need for deeper insights into the underlying “Clean Engineering” practices.
• Deterministic Serving: There will be increasing demand for more deterministic serving infrastructure, especially for critical evaluation tasks. This could mean dedicated, isolated endpoints or local, batch-invariant deployments for sensitive measurements.
• New Measurement Paradigms: The field may need to move beyond simple ranking or scoring by black-box LLMs towards more explainable, auditable, and inherently stable evaluation methods. This could involve combining LLM judgments with symbolic reasoning, human consensus, or highly structured, low-variance evaluation tasks.
• Engineering for Stability: Researchers and engineers will need to design LLMs and AI agents with internal mechanisms that are less susceptible to external measurement noise, or that can compensate for it. This includes developing more robust internal representations and decision-making processes that are less sensitive to subtle input variations.

The future of intelligent systems, particularly the reliable deployment and alignment of AI agents, hinges on our ability to measure them accurately. This research highlights that the “model name on a shared endpoint” is not a frozen instrument, and accepting this reality is the first step towards truly clean engineering in a complex, unstable measurement landscape.

Key Takeaways

• Fundamental Instability: Black-box LLM judges, used for critical evaluation tasks, exhibit profound and systemic unreliability, failing to meet basic instrument validation requirements.
• Widespread Impact: This instability compromises LLM leaderboards, training data curation, RLHF pipelines, model safety evaluations, and the decision-making of AI agents.
• Three Core Mechanisms: Instability stems from biased label-to-meaning mapping, candidate differences falling below the LLM’s noise floor, and non-deterministic responses to byte-identical inputs.
• Provider Agnostic: The issue is prevalent across multiple major LLM providers and is not easily mitigated by waiting or simple metric substitutions.
• Call for Rigor: The paper advocates for measuring the evaluation instrument’s reliability before freezing any gates on it, proposing an identity ladder, design rules, and a reporting checklist.
• Urgent Shift Needed: The community must pivot towards more transparent, deterministic, and rigorously validated evaluation protocols to ensure the integrity and progress of Machine Learning and AI agent development.