Adapting World Models from
VLM Plans for Robotic Tool Use

Robotic tool use requires semantic reasoning about how a tool can serve a goal and physical prediction of its interactions with the scene. Vision-language models (VLMs) can propose plausible strategies, but their plans often fail to account for task-specific geometry and contact dynamics. We present a framework that autonomously adapts an action-conditioned visual world model from the observed outcomes of VLM-proposed executions. The VLM generates gripper-centric plans through an anchor-offset representation, and the world model predicts their consequences for evaluation by a separately trained reward model. Only the world model is updated during interaction, using transitions from both successful and unsuccessful attempts. The adapted model then supports selection among new VLM proposals through imagined rollouts. Under this setting, the approach improves over direct VLM plan execution across simulated and real-world tool use tasks and outperforms the evaluated policy-learning baselines. These results suggest that learning task-specific dynamics from VLM-generated interaction data can improve tool use plan selection with a limited additional execution budget.