In this blog we will demonstrate how to run local agents enabled by VLMs (Vision-Language Models) hosted by the Lemonade framework in the domain of robot control. These models allowed us to run an interactive robotic arm manipulation simulation entirely locally, showing the effectiveness of Lemonade for Physical AI. Through combining the RAI framework for robot control, O3DE for simulation, ROS 2 Jazzy for the robotics interface, and the Lemonade SDK for local AI model deployment on the GPU, we were able to perform tasks such as picking, placing, and sorting different objects. All code is available in the AMD Ryzers repository . For this demonstration we will set up RAI and Lemonade through Ryzers, pull the Gemma-4-E2B-it-GGUF model, and launch an O3DE robot arm simulation. From there, you can give the arm natural-language commands, such as asking it to pick up objects, place them in different locations, and sort them across the table as the model executes each task in real time. Agentic Reasoning vs. End-to-End Policies One of our previous blogs explored running local AI models for manipulation through an end-to-end policy, fine-tuning a Vision Language Action model (VLA) on demonstration datasets to map camera input directly to motor commands. Although VLAs show a lot of promise, as end-to-end trained systems they lack interpretability and deterministic behavior. RAI instead takes an agentic approach, preserving the reasoning capability of language models while maintaining functionality: by exposing existing perception, planning, and control methods as tools, it constrains the policy outputs to valid and safe actions; there’s no task-specific policy. To achieve this, we place a general-purpose VLM in the loop as a reasoning agent that receives the scene through the camera feed, reasons over the available tools, and delegates motion planning to MoveIt . The trade-offs are per-step inference latency, since each action requires model reasoning, and a hard requirement for vision-capable models, since the agent must see the scene to identify objects and reason about their positions before acting. In exchange, there is no need for data collection, fine-tuning, or training infrastructure. Any model compatible with Lemonade can immediately start working, every decision is interpretable as an explicit tool call, and the agent can approach new tasks without extra training. RAI RAI was created by Robotec.ai as an open-source agentic AI framework. Built on top of ROS 2, it acts as a bridge between multimodal AI models and robotic systems, enabling robots to perform more complex tasks, interpret human language, and make autonomous decisions. The integration with ROS 2 is significant, as it is an industry-ready middleware already used in production, backed by an extensive community. Further, RAI supports local models, which enables us to plug in Lemonade directly and benefit from out-of-the-box manipulation with minimal edits. AMD Lemonade Lemonade is a local inference server that serves a variety of models, including LLMs, VLMs, speech-to-text, and text-to-speech, on your hardware through an OpenAI-compatible API, letting any supporting tool point at a model running on the machine instead of the cloud. With its ROCm backend it offloads the models onto the Radeon iGPU, keeping inference fast, private, and entirely on-device. Since RAI reads its endpoint from a standard OpenAI configuration, swapping in Lemonade is a one-line change; with that, Lemonade is now available for Physical AI. The Importance of Locally Hosted Models for Physical AI Running AI inference locally on robotic systems offers several advantages over hosting the compute in the cloud. Because the model runs on the machine itself, there is no round-trip to a remote server, cutting latency and removing any dependency on network connectivity. This keeps the robot responsive even offline, lets the control loop react in real time, and ensures that sensor data never leaves the