VLA models unify perception, language understanding, and robot control in a single architecture — enabling robots to follow natural language instructions and act in the physical world.
Select a modality to highlight its path through the VLA architecture.
How much the action output attends to each modality.
How robots learn to see, understand language, and act — all at once.
A Vision-Language-Action (VLA) model is a neural network that simultaneously processes camera images, natural language instructions, and robot joint states to produce a sequence of robot actions. The word “action” here is literal — the model outputs motor commands that move a robot arm, open a gripper, or navigate a mobile platform.
The unifying insight is that perception (what do I see?), understanding (what am I being asked to do?), and control (what joint angles should I set?) can all be learned jointly from data, rather than being hand-engineered as separate subsystems.
Traditional robot controllers decompose the problem into stages: a perception module detects objects, a task planner produces a symbolic plan, and a controller executes each step. Each stage requires domain expertise, hand-crafted representations, and brittle rule sets. Adding a new object or task often requires re-engineering several stages.
A learned VLA policy collapses this pipeline into a single model trained end-to-end on demonstrations. Given a camera frame, a language instruction like “pick up the red cup and place it in the bowl,” and the robot’s current joint angles, the policy directly outputs the next action. No explicit task planner. No hand-coded object models. The representations are learned entirely from data.
Every VLA model processes three streams of information:
These three streams are fused — either via cross-attention or by concatenating token sequences — and decoded into an action sequence.
The pi0.5 model (from Physical Intelligence, implemented in LeRobot v0.4.4) uses a large language model (LLM) as the backbone for processing vision and language tokens. On top of this backbone, a flow-matching diffusion head generates the action sequence.
Flow matching is a generative approach: instead of predicting a single action deterministically, the model learns to transform noise into an action distribution. This captures the multimodality of expert demonstrations — there are often multiple valid ways to accomplish a task.
VLAExplain’s attention analysis for pi0.5 supports ActionCode Attribution (how a predicted action token attends to vision, language, state, and future action tokens) and Cross-Modal Attention (how text attends to image/state, how image attends to text/state, etc.) across all layers and attention heads.
UnifoLM-VLA (from Unitree Robotics) uses a Diffusion Transformer (DiT) as the action expert network. The DiT has an alternating layer structure: even layers use cross-attention, letting the action tokens attend to vision patches and task text descriptions. Odd layers use self-attention, computing correlations within the state, future token, and action token sequences.
This design separates “gathering information from the scene” (cross-attention in even layers) from “reasoning about how state, future predictions, and actions relate to each other” (self-attention in odd layers). VLAExplain’s action attribution analysis for UnifoLM-VLA exposes both attention modes, and also computes module-level attention intensity (total intensity reflecting overall distribution; mean intensity eliminating dimensional differences).
ActionCode attribution is the key interpretability concept behind VLAExplain. For a given predicted action at inference step t, the tool computes how much attention each action token places on each input modality: image patches, text tokens, robot state tokens, and future action tokens.
This answers the question: when the robot decided to move its gripper left by 2 cm, which pixels was it “looking at”? Which words in the instruction were most relevant? How much did it rely on its current joint state?
The visualisation overlays these attention weights as a heatmap on the raw camera frame, making it possible to verify that the model is attending to the correct object — or to diagnose failures when it is not.
Both pi0.5 and UnifoLM-VLA are evaluated on the LIBERO benchmark — a simulation suite of tabletop manipulation tasks designed to test generalisation across task suites, object layouts, and goal specifications. LIBERO provides reproducible evaluation: the same environment, the same initial conditions, the same success criterion for every run.
VLAExplain collects attention data during LIBERO inference runs: at each action step it saves the raw RGB frame, the attention weights from all transformer layers and heads, and the language input IDs and state values. This data is then loaded into the Gradio-based visualisation app for interactive exploration.
VLA models are beginning to transfer from simulation to hardware. Physical Intelligence’s pi0 family has demonstrated real-robot manipulation; Unitree’s UnifoLM-VLA is designed for humanoid and quadruped platforms. The appeal is instruction following: telling a robot “put the banana on the plate” rather than programming a specialised pick-and-place routine.
Limitations remain: data hunger (thousands of demonstrations per task), distribution shift (sim-to-real gaps), inference latency (large transformers are slow on embedded hardware), and lack of interpretability (which is exactly what VLAExplain addresses). VLAExplain currently operates in simulation and has not been tested on real hardware, but the attention data format is hardware-agnostic.
Say the answer out loud before you reveal it — recalling it is what makes it stick, and rereading it is not.
What are the three input modalities in a VLA model?
How does a VLA policy differ from a classical robot controller?
What distinguishes pi0.5 from UnifoLM-VLA architecturally?
What does ActionCode attribution tell you about a VLA model's decision?
VLA = (image + language instruction + robot state) → action. Three modalities fused via cross-attention. pi0.5: LLM backbone + flow-matching head. UnifoLM-VLA: DiT with even-layer cross-attention (vision/text) and odd-layer self-attention (state/future/action). ActionCode attribution reveals per-token attention to each modality. Evaluated on LIBERO simulation.