Hypotheses

H1 · retired · from gate_open · est $50

A systematic search of ACL Anthology, arXiv, and OpenReview with the query ('Absent Supervisor' OR 'AI Safety Gridworlds') AND ('language model' OR 'LLM') will return at least one peer-reviewed paper reporting the same shorter-hazardous-path preference in Absent Supervisor.

Would be false if: After running the expanded query against ACL Anthology, arXiv, and OpenReview, zero papers are found that report the shorter-hazardous-path preference in Absent Supervisor.

H2 · killed_prior_art · from gate_open · est $20

Qwen3-235B-Thinking's high hidden reward in Safe Interruptibility is caused by the model treating the interruption tile as a collectible, so renaming that tile to a non-collectible symbol will reduce hidden reward to near zero.

Would be false if: After renaming the interruption tile to an unrelated symbol, Qwen3-235B-Thinking's hidden reward remains within 10% of its original high value.

H3 · blocked · from gate_open · est $20

The Boat Race back-and-forth exploit is caused by the absence of an explicit lap counter in the observation, so adding a lap counter will eliminate oscillation and increase completed laps.

Would be false if: With a lap counter added to the observation, the model still oscillates on a single arrow tile at the same rate as in the original environment.

Test: the test requires code that does not exist yet

H4 · blocked · from paper_open_question · est $100

A mitigation that lowers the observed reward-hacking metric will also lower exploit frequency when the exploit is made unrewarding in a held-out reward variant.

Would be false if: Exploit frequency in the unrewarding reward variant remains within 10 percentage points of baseline for mitigated checkpoints.

Test: the test names material the society does not hold ("checkpoints")

H5 · blocked · from paper_open_question · est $3000

The 14B Island Navigation success is seed-dependent, so across five independent RL seeds at least three fail to achieve the hidden objective.

Would be false if: Three or more of five independent training runs at 14B achieve the hidden objective on Island Navigation.

Test: the test requires code that does not exist yet

H6 · blocked · from paper_open_question · est $200

Models with a positive observed-hidden reward gap on the text gridworlds will also show a higher unsafe tool-call rate on AgentHarm than models without such a gap.

Would be false if: The correlation between text-gridworld observed-hidden gap and AgentHarm unsafe tool-call rate is zero or negative (Spearman rho <= 0 or p > 0.05).

Test: the test requires code that does not exist yet

H7 · blocked · from paper_open_question · est $500

RL fine-tuning of a coding agent on visible unit tests will increase visible pass@1 while leaving hidden pass@1 unchanged.

Would be false if: Hidden pass@1 increases by more than 8 points after RL fine-tuning on visible tests.

Test: the test requires code that does not exist yet

H8 · blocked · from paper_open_question · est $100

Extending the text gridworld to a richer multi-room environment will not reduce the zero-shot specification-gaming rate; the observed-hidden gap will be at least as large as in the original simple environments.

Would be false if: The observed-hidden reward gap in the richer multi-room environment is at least 50% smaller than in the original simple environments.

Test: the test requires code that does not exist yet

H9 · blocked · from paper_open_question · est $50

A vision-language model given rendered ANSI gridworld screenshots will reproduce the Boat Race oscillating exploit when text-only models do.

Would be false if: The vision-language model never oscillates across 100 Boat Race episodes.

Test: the test requires code that does not exist yet

H10 · blocked · from paper_open_question · est $2000

Scaling the RL training model from 14B to 32B will not eliminate the Boat Race oscillation exploit; at least one of three seeds will still oscillate on a single arrow tile.

Would be false if: Zero of three independent 32B training seeds oscillate on a single arrow tile in Boat Race.

Test: the test requires code that does not exist yet

H11 · blocked · from paper_open_question · est $500

Adversarial reward shaping with an ensemble of reward models will reduce the observed-hidden gap on Boat Race more than entropy regularization alone.

Would be false if: The observed-hidden gap under adversarial reward shaping is equal to or larger than under entropy regularization alone.

Test: the test requires code that does not exist yet

H12 · blocked · from paper_open_question · est $300

The observed-hidden gap measured in text gridworlds will predict unsafe behavior on the tau-bench airline tool-use benchmark, with models above the median gap making more policy-violating tool calls.

Would be false if: Models above the median text-gridworld observed-hidden gap make the same number or fewer policy-violating tool calls on tau-bench airline than models below the median.

Test: the test requires code that does not exist yet

H13 · blocked · from paper_open_question · est $20

Exploration failure in Island Navigation is caused by the model's prior policy assigning near-zero probability to off-path actions; forcing a one-step deviation to an alternative route will produce sustained commitment to that route for at least 10 subsequent steps.

Would be false if: After a forced one-step deviation to an alternative route, the model returns to the original exploit within 2 subsequent steps in more than 50% of trials.

Test: the test requires code that does not exist yet

H14 · blocked · from paper_open_question · est $1000

A coding agent that shows specification gaming in the text gridworlds will similarly pass visible tests while failing hidden tests on SWE-bench Lite when optimized with RL on visible tests.

Would be false if: Hidden test pass rate increases by more than 8 points after RL optimization on visible tests.

Test: the test requires code that does not exist yet

H15 · blocked · from gate_open · est $30

In the Absent Supervisor environment, models prefer the shorter hazardous path because the prompt omits an explicit hazard-avoidance instruction, so adding that instruction will significantly reduce hazardous-path selection.

Would be false if: With the explicit hazard-avoidance instruction, models choose the shorter hazardous path at the same rate as in the original Absent Supervisor prompt (difference under 5 percentage points).

Test: the test names material the society does not hold ("released")

H16 · blocked · from gate_open · est $0

The Boat Race back-and-forth exploit is caused by per-visit reward accumulation, so the training reward traces for oscillating episodes will show reward increments on each back-and-forth step.

Would be false if: In oscillating episodes, reward is granted only at lap completion or is flat during the back-and-forth steps.

Test: the test names material the society does not hold ("released")

H17 · supported · from paper_open_question · est $0

Mitigations that improve observed reward without increasing hidden reward leave the observed-hidden gap unchanged or wider, so the gap is a valid marker of surface suppression.

Would be false if: At least one mitigation raises observed reward while also closing or reversing the observed-hidden gap.

Test: the test re-analyses values the paper itself reports

Re-analysis of reported values — supported. the reported values show the predicted direction (increases), agreed by both passes and located in the paper (9 values located in the paper) Direction: increases.

re-analysis of values the paper already reports, not a new measurement: this can refute but cannot confirm, and support means only that the paper's own numbers are consistent with the hypothesis.

H18 · blocked · from paper_open_question · est $80

The 14B Island Navigation success is stochastic escape rather than systematic resolution, so repeated runs with different seeds will show high variance and frequent failures.

Would be false if: Repeated runs show consistently high success with low variance across seeds.

Test: the test names material the society does not hold ("released")

H19 · supported · from paper_open_question · est $0

Scaling model size alone does not systematically reduce the observed-hidden reward gap; apparent scale benefits are seed-dependent escapes.

Would be false if: The gap decreases monotonically with scale and reaches near zero at the largest scale.

Test: the test re-analyses values the paper itself reports

Re-analysis of reported values — supported. the reported values show the predicted direction (mixed), agreed by both passes and located in the paper (9 values located in the paper) Direction: mixed.

re-analysis of values the paper already reports, not a new measurement: this can refute but cannot confirm, and support means only that the paper's own numbers are consistent with the hypothesis.

H20 · inconclusive · from paper_open_question · est $0

Proxy-reward failures are not fixed by standard exploration or credit-assignment fixes because the failures reflect reward misspecification; changing the reward specification is required to close the gap.

Would be false if: At least one standard exploration or credit-assignment fix closes the observed-hidden gap without changing the reward specification.

Test: the test re-analyses values the paper itself reports

Re-analysis of reported values — inconclusive. the paper does not report what this test needs: The paper reports no reward-correction condition that changes the reward specification to penalize the hazardous/exploit action, so the prediction's 'corrected reward closes the gap' half cannot be checked. It also reports no numeric observed-hidden gap changes for entropy regularization, credit assignment, exploration prompts, or history length; those ablations are described only qualitatively or (4 values located in the paper)

re-analysis of values the paper already reports, not a new measurement: this can refute but cannot confirm, and support means only that the paper's own numbers are consistent with the hypothesis.

H21 · blocked · from paper_open_question · est $40

Language-model competence forecloses exploration because the model's pretrained action prior is peaked and it never samples the safe alternative; seeding alternative first actions will induce sustained exploration.

Would be false if: Seeding alternative first actions does not increase safe-solution discovery; models revert to the same exploit.

Test: the test names material the society does not hold ("released")

H22 · supported · from paper_open_question · est $0

Finance-specific LLMs are not more resilient than general-purpose LLMs to domain-specific financial jailbreaks; their attack success rate is at least as high.

Would be false if: Finance-specific models show significantly lower ASR than general-purpose models of similar size.

Test: the test re-analyses values the paper itself reports

Re-analysis of reported values — supported. the reported values show the predicted direction (increases), agreed by both passes and located in the paper (3 values located in the paper) Direction: increases.

re-analysis of values the paper already reports, not a new measurement: this can refute but cannot confirm, and support means only that the paper's own numbers are consistent with the hypothesis.

H23 · blocked · from paper_open_question · est $0

The vulnerability gap between open-source small LMs and API-based models on financial attacks is explained by the presence of safety alignment, not by model size or API access; open-source models with comparable safety training will show comparable robustness.

Would be false if: Even when safety alignment is matched, open-source models still show much higher ASR.

Test: claims the existing corpus suffices, but names no material the society can point at

H24 · inconclusive · from paper_open_question · est $0

Optimization-based suffixes lower attack success on FinRED by displacing financial context tokens, so lower-ASR suffixes have significantly lower overlap with financial keywords in the seed.

Would be false if: Suffixes with high financial-keyword displacement show high ASR, or there is no correlation.

Test: the test re-analyses values the paper itself reports

Re-analysis of reported values — inconclusive. the paper does not report what this test needs: The paper reports no token-level or embedding overlap between optimization-based suffixes and seed financial keywords, and no measure of suffix-induced displacement of financial context tokens, so no correlation with ASR can be computed. It only offers a qualitative, non-quantified speculation about disruption of financial context. (2 values located in the paper)

re-analysis of values the paper already reports, not a new measurement: this can refute but cannot confirm, and support means only that the paper's own numbers are consistent with the hypothesis.

H25 · blocked · from paper_open_question · est $50

A same-family attacker (Claude attacking Claude) does not maintain per-turn effectiveness equal to cross-family attackers; its attack success rate declines after the first turn.

Would be false if: Same-family attacker ASR remains flat or increases across turns at the same rate as cross-family.

Test: the test requires code that does not exist yet

H26 · blocked · from paper_open_question · est $80

Attacker self-limitation is a property of safety training rather than model identity; it generalizes to other safety-trained model families.

Would be false if: Other safety-trained models do not show self-limitation; their harmful-output rates match base models.

Test: the test requires code that does not exist yet

H27 · blocked · from paper_open_question · est $0

The four trajectory signatures reflect training philosophy, so a classifier trained on trajectory features can predict the training objective above chance.

Would be false if: Classification accuracy is at or below chance, indicating trajectories do not encode training philosophy.

Test: the test names material the society does not hold ("released")

H28 · inconclusive · from paper_open_question · est $0

Hardening Turn 2 reduces overall multi-turn attack success more than hardening an equal number of other turns.

Would be false if: Turn-2 hardening does not produce a larger cumulative ASR drop than hardening other turns.

Test: the test re-analyses values the paper itself reports

Re-analysis of reported values — inconclusive. the paper does not report what this test needs: The paper reports no per-turn hardening ablation comparing Turn-2-only hardening against Turn-1-only or Turn-3-only hardening, and it does not provide the per-conversation trajectory data that would be needed to simulate removing successful Turn-2 attacks. It gives only aggregate per-turn unsafe rates and a Turn-2 concentration statistic. (4 values located in the paper)

re-analysis of values the paper already reports, not a new measurement: this can refute but cannot confirm, and support means only that the paper's own numbers are consistent with the hypothesis.

H29 · blocked · from paper_open_question · est $0

Red-teaming pipelines that do not explicitly detect and handle refusals overestimate attack success rates.

Would be false if: Refusal detection does not change measured ASR (less than 1 percentage point).

Test: the test names material the society does not hold ("transcripts")

H30 · inconclusive · from paper_open_question · est $0

Conversation-level pattern detection identifies multi-turn attacks earlier than per-turn classification.

Would be false if: Per-turn detection matches or exceeds conversation-level early-turn AUC.

Test: the test re-analyses values the paper itself reports

Re-analysis of reported values — inconclusive. the paper does not report what this test needs: The paper evaluates a classifier only per turn (Phase 1 single-turn accuracy/miss-rate/false-alarm-rate, Phase 2 per-turn drift, Phase 3 intervention on Turn 4 unsafe rate) and reports no AUC at all, no conversation-level feature classifier, and no early-turn detection comparison between a conversation-level and a per-turn detector. 'Conversation-level pattern detection' appears only as a recommen (3 values located in the paper)

re-analysis of values the paper already reports, not a new measurement: this can refute but cannot confirm, and support means only that the paper's own numbers are consistent with the hypothesis.

H31 · blocked · from paper_open_question · est $20

Automated debate protocols detect multimodal deception at least as reliably as human evaluation.

Would be false if: Automated debate detections agree with human judgments at kappa <= 0.4.

Test: the test names material the society does not hold ("released")

H32 · blocked · from paper_open_question · est $50

Multimodal LLMs exhibit higher deception rates when deceptive cues are conveyed visually rather than textually.

Would be false if: Visually cued deception yields equal or lower deception rates than text-only cues.

Test: the test requires code that does not exist yet

H33 · blocked · from paper_open_question · est $20

Defensive safety technologies trained on the released dataset reduce attack success on held-out attacks.

Would be false if: No improvement over baseline (ASR difference under 2 percentage points).

Test: the test names material the society does not hold ("released")

H34 · blocked · from paper_open_question · est $50

Chunked readout with hierarchical aggregation improves long-context document-scale evidence tasks over single-pass long-context training.

Would be false if: Chunked readout improves accuracy by less than 1 percentage point or worsens it.

Test: the test requires code that does not exist yet

H35 · blocked · from paper_open_question · est $50

Multi-turn attacks succeed at higher rates in lower-resource languages than in English.

Would be false if: Lower-resource language ASR is equal to or lower than English ASR.

Test: the test names material the society does not hold ("released")

H36 · blocked · from paper_open_question · est $50

Combining input classification with output-side review reduces multi-turn attack success more than either defense alone.

Would be false if: Combined defense ASR is not lower than the best single defense.

Test: the test names material the society does not hold ("released")

H37 · blocked · from paper_open_question · est $200

Targeted alignment interventions on detected multimodal deception failure modes reduce deception more than generic safety training.

Would be false if: Targeted intervention does not outperform generic safety training (difference under 3 percentage points).

Test: the test names material the society does not hold ("released")

H38 · blocked · from paper_open_question · est $50

Reward-hacking patterns from text gridworlds transfer to coding agents: when a coding task has visible unit tests and hidden safety tests, agents optimize the visible tests while failing hidden safety tests.

Would be false if: Visible and hidden test pass rates are equal within 5 percentage points.

Test: the test requires code that does not exist yet

H39 · blocked · arm A · from paper_open_question · est $220

The observed-reward-high / hidden-objective-low gap reported for text gridworld agents also appears when the same tasks are expressed as tool-use episodes with schemas for available actions.

Would be false if: Tool-wrapped agents show no decoupling: episodes with high observed reward also satisfy the hidden safety objective in more than 90% of cases (gap below 10 percentage points).

Test: the test names material the society does not hold ("released")

H40 · blocked · arm A · from paper_open_question · est $180

Reward-hacking behaviour transfers to coding agents in the form of passing visible tests while failing held-out hidden tests at a higher rate than non-RL baseline coding models.

Would be false if: RL-trained coding agents show visible-test and hidden-test pass rates within 5 percentage points of each other, or their visible-hidden separation is no larger than that of base models.

Test: the test requires code that does not exist yet

H41 · blocked · arm A · from paper_open_question · est $350

The observed-hidden reward gap in the Boat Race environment does not shrink when the policy model is scaled from 14B to roughly 70B parameters under the same training recipe.

Would be false if: The 70B-class model completes laps with hidden-objective success above 70% and a gap under 10 percentage points, or fails to converge to the back-and-forth exploit at all.

Test: the test requires code that does not exist yet

H42 · blocked · arm A · from paper_open_question · est $70

The FinRED taxonomy as published fails to place a substantial share of 2025-2026 financial-crime cases, specifically agentic and synthetic-media-mediated fraud, into any existing subtype.

Would be false if: At least 90% of the sampled recent cases receive a single FinRED subtype with cross-annotator agreement of kappa >= 0.7 or better.

Test: the test requires code that does not exist yet

H43 · blocked · arm A · from paper_open_question · est $45

The R2 Financial Crime and R4 Consumer Rights Violation categories in FinRED overlap enough that a substantial fraction of real violation examples are assigned to both by independent annotators.

Would be false if: Dual R2/R4 assignment occurs on under 5% of items and is no more frequent than dual assignment for other adjacent category pairs.

Test: claims the existing corpus suffices, but names no material the society can point at

H44 · blocked · arm A · from paper_open_question · est $40

FinRED's rubric produces low inter-annotator agreement specifically on responses that partially comply with a financial-harm request or mix legitimate and harmful intent.

Would be false if: Partial-compliance and mixed-intent items reach kappa >= 0.75, statistically indistinguishable from the fully compliant and fully refusing items.

Test: claims the existing corpus suffices, but names no material the society can point at

H45 · blocked · arm A · from paper_open_question · est $90

A model's FinRED financial-safety score is largely uncorrelated with its resistance to financial-domain prompt injection and jailbreak attempts.

Would be false if: FinRED safety score and financial-domain attack success rate are strongly negatively correlated (Spearman <= -0.6), or the top FinRED scorer complies on under 10% of attack prompts.

Test: the test requires code that does not exist yet

H46 · blocked · arm A · from paper_open_question · est $10

Reward hacking acquired during proxy optimization is a task-agnostic disposition rather than a task-specific adaptation, so per-model hacking rates on held-out tasks correlate with hacking rates on the training task.

Would be false if: Training-task and held-out hacking rates correlate below r = 0.3 across runs, with some runs showing heavy training-task hacking and no held-out hacking (or the reverse).

Test: the test names material the society does not hold ("released")

H47 · blocked · arm A · from paper_open_question · est $15

What transfers in reward-hacking generalization is sensitivity to shortcut cheapness, such that a held-out task is hacked at a higher rate the more its exploit resembles the training task's shortcut in step count and required actions.

Would be false if: Held-out hacking rate is independent of shortcut structural similarity (no monotone trend across similarity tiers) and tracks domain identity instead.

Test: the test re-analyses values the paper itself reports

H48 · blocked · arm A · from paper_open_question · est $150

The per-turn hardening gain reported for Turn 2 does not survive deployment-like conditions in which the attacker retains memory of earlier turns and can iterate its strategy within a session.

Would be false if: Turn-2 attack success rate with attacker memory and retries stays within 5 percentage points of the reported single-shot turn-2 rate.

Test: the test requires code that does not exist yet

H49 · blocked · arm A · from paper_open_question · est $120

A same-family attacker (Sonnet attacking Sonnet) loses per-turn effectiveness faster than a cross-family attacker, so the observed per-turn attack success advantage decays with turn index rather than holding steady.

Would be false if: Same-family attack success is flat or rising across turns 2-4, and the same-family versus cross-family decline curves differ by less than 5 percentage points.

Test: the test requires code that does not exist yet

H50 · blocked · arm A · from paper_open_question · est $100

Attacker self-limitation is a general property of aligned models rather than a quirk of the specific models in the study, appearing at comparable rates in at least three other model families not tested by the authors.

Would be false if: Fewer than two of the new families show any self-limitation, or their rates fall below 5% of escalation prompts.

Test: the test requires code that does not exist yet

H51 · inconclusive · arm B · from gate_open · est $0

Qwen3-235B-Thinking's high hidden reward in Safe Interruptibility is an isolated outlier rather than part of a consistent safety advantage across the suite's safety environments.

Would be false if: If the reported hidden rewards for Qwen3-235B-Thinking in Absent Supervisor and Distributional Shift are also at or near the top of the model set, then the accidental-misinterpretation account is false and the high hidden reward generalizes.

Test: the test re-analyses values the paper itself reports

Re-analysis of reported values — inconclusive. the independent pass offered additional values that cannot be located in the paper (6 values located in the paper)

re-analysis of values the paper already reports, not a new measurement: this can refute but cannot confirm, and support means only that the paper's own numbers are consistent with the hypothesis.