Panasonic HD Develops a World Model for Mobile Work Assistance

Rather than reproducing the post-operation screen as an image, this technology predicts operation-related changes in natural language, such as "pressing the button opens the settings screen" or "an input field appears." As a result, without being affected by information unrelated to the operation-such as the time, notifications, or screen layout-the AI agent can evaluate predicted state changes for multiple candidate actions and select the action most likely to achieve the goal.

MobileWorld, developed as part of this research, is a dataset based on human interaction trajectories on Android devices. It includes operation details, screens before and after each operation, descriptions of screen changes, and questions and answers about subsequent states. The dataset contains approximately 942,000 examples describing screen changes and approximately 543,000 question-and-answer examples. A vision-language model was used to generate and automatically filter the data, with a subset also verified manually. Qwen3-VL-8B-Instruct ⁽*²⁾, further trained using MobileWorld, was evaluated on MobileWorldBench. When its predicted text was scored for accuracy, completeness, and relevance, it achieved the highest score among the methods compared. The world model was then integrated into an AI agent and evaluated on AndroidWorld ⁽*³⁾. In this evaluation, the world model predicts the future state for each candidate action, and the AI agent uses those predictions to select the action it will actually execute. The task success rate improved substantially from 46.9% for the baseline without the world model to 54.3% with the world model integrated.

By applying this technology to work that spans multiple applications-including equipment management, parts inventory, and work records-we aim to reduce workloads at worksites through AI agents that leverage the knowledge and judgment of experienced personnel. Going forward, in addition to smartphones, we will examine its applicability to websites and PC applications and advance research and development toward AI technology that supports a broad range of internal operations.