Abstract When a world model (WM) serves as a reinforcement-learning environment, the policy moves toward whatever the WM judges a success, so any systematic tendency of the WM to draw failures as successes gets amplified. On RoboTwin adjust_bottle , with the action-conditioned video model BWM as the environment and π0.5 as the policy, this work asks under what conditions in-WM RL gains are reliable. The base BWM draws 82.5% of simulator failures as successes, a gap that pixel metrics barely register. Fine-tuning the WM on the policy’s own failure rollouts brings fake success down to 26%; a success-only control does not improve at all. Inside the fine-tuned WM, a PPO recipe that fixes three defects (truncation bootstrapping, where the critic attaches, and the value learning rate) beats SFT significantly on all five training seeds once evaluated back in the simulator: mean gain +9.7 percentage points, standard deviation 4.0. The original recipe had a standard deviation of 14.3 and a worst seed at −19.0. Closed-loop replay shows that the degraded seed under the original recipe was exploiting the WM’s fake successes, while the corrected recipe stays at the SFT mismatch level. Ablations locate the remaining error in failures whose outcome depends on action details from a shared initial state: the WM draws 29–40% of them as successes even after training on them, adding on-policy data does not fix this, and these are exactly the states that RL gradients come from. Introduction In a WM used as an RL environment, the policy emits an action chunk, the WM renders the corresponding video, a reward model reads the video and judges success, and the policy updates on that reward. A WM used as a policy evaluator can tolerate errors as long as they hit all policies about equally, because the ranking survives. An RL environment cannot: the policy actively searches for actions that earn reward. If the WM consistently gets a class of actions wrong, for instance drawing a grasp as successful whenever the gripper closes, the policy will take that class of actions more and more often. A WM therefore has to meet a stricter bar as an RL environment than as an evaluator: the outcome of every rollout must be drawn correctly. Existing in-WM RL work mostly reports single-run training gains, and work on repairing WMs mostly validates prediction quality or ranking. Two questions remain open: whether a repaired WM still judges correctly on the trajectories an updated policy visits, and whether in-WM RL gains reproduce across training seeds. This work answers both on a single task. Contributions: Measuring outcome fidelity. The same action sequence drives both the simulator and the WM, and one frozen reward reads both sets of frames, giving the WM’s fake-success rate. Closed-loop replay then measures whether an updated policy is exploiting WM errors. Failure-data fine-tuning. Fine-tuning the WM on the policy’s own failure rollouts cuts fake success from 82.5% to 26%, and the effect comes from the failure data itself. Reproducible in-WM PPO. After three recipe defects are fixed, all five training seeds beat SFT significantly, with mismatch at the SFT level. Locating the remaining error. Ablations show that the WM’s outcome judgments on action forks from a shared initial state remain unreliable and do not respond to more on-policy data. This needs solving before in-WM RL can scale further. 2 Related Work Training VLAs inside world models. WoVR, WMPO and World-Gymnast post-train VLAs with RL inside action-conditioned video models and report single-run gains of +5 to +29 percentage points. The setup here is closest to WoVR, and the multi-frame history training window follows its recipe. The difference is in validation: every recipe here is trained with five seeds, and closed-loop replay checks whether the policy is exploiting WM errors. Optimism bias and failure data. MiraBench documents the tendency of video WMs to draw failures as successes; none of the 12 mode