WIZARD: Adapting robot policies by generating the weights they need.
~70 ms
one forward pass generates the full task adapter
1
visual demonstration — no action labels
40%
held-out LIBERO-Spatial · 19% for 1-demo fine-tuning
37%
held-out Franka tasks · 17% for fine-tuning
Key takeaways
WIZARD generates the LoRA adapter that fine-tuning would have learned — from one instruction and one visual demonstration, in a single ~70 ms forward pass, with no action labels and no target-task gradients.
It leads the evaluated baselines across all held-out LIBERO and MetaWorld settings, and beats one-demonstration fine-tuning on a physical Franka (37% vs 17%).
Adaptation becomes an inference problem whose cost is amortised across tasks — with clear limits on unseen object semantics and long-horizon composition.
Imagine a robot working at a flexible manufacturing station. For weeks, it has been handling the same set of components reliably. Then the line changes: a part arrives in a new position, a new object variant is introduced, or the desired sequence of operations changes.
The robot may already have most of the capabilities it needs. It can perceive the workspace, understand an instruction, reach, grasp and move objects. But that does not mean its existing policy will execute the new task reliably.
Today, the standard response is to collect demonstrations for the new task, record the robot actions associated with them, fine-tune the policy, validate the updated model, and only then deploy it. For a single task this can be manageable. Repeated across hundreds of task variants, robot cells or deployment environments, it becomes an adaptation pipeline of its own.
This was the problem that motivated WIZARD. Instead of asking how to fine-tune a Vision-Language-Action model more efficiently, we asked a different question: can we infer the weights that fine-tuning would have produced?
WIZARD — Weight-space Inference for Zero-gradient Adaptation from Robotic Demonstration — explores that idea. Given a language instruction and a visual demonstration of a previously held-out task, it generates a task-specific adapter for a frozen robot policy in a single forward pass, without using the demonstration’s robot-action labels and without gradient updates on the target task.
That changes adaptation from something the robot has to optimise into something a learned system can predict.
Held-out tasks WIZARD has never trained on: new positions, new goals, unseen object categories. What the source distribution already encoded transfers; what it didn’t doesn’t.
~1.5 min
Why existing approaches still leave an adaptation problem
Large Vision-Language-Action models, or VLAs, have substantially changed what a single robot policy can represent. Models such as OpenVLA and π0.5 combine visual observations, language and robot actions within large pretrained policies, allowing one model to cover many tasks rather than training a separate controller from scratch for every behaviour.
But broad capability and reliable task adaptation are not the same thing.
Fine-tuning: effective, but repeated for every task
The most direct solution is still supervised fine-tuning. Collect demonstrations of the new task, including the actions executed by the robot, and update the policy using gradient descent.
Fine-tuning has a major advantage: it works directly on the behaviour we care about. Given enough representative data and optimisation, a task-specific model can often become very strong.
But adaptation still amounts to a new training problem every time.
Someone has to collect action-labelled demonstrations. Those demonstrations have to be processed and stored. Optimisation has to run. Hyperparameters and checkpoints have to be managed. The resulting policy has to be evaluated before deployment.
Parameter-efficient methods such as Low-Rank Adaptation, or LoRA, dramatically reduce the number of trainable parameters, but they do not remove this loop . They make the optimisation smaller; they do not make it disappear.
Meta-learning: learning how to adapt
Meta-training builds a repository of task experts. At test time, WIZARD predicts the adaptation weights for an unseen task directly.
Meta-learning attacks the problem from another direction. Instead of learning only a task, the model learns something about the adaptation process itself.
The canonical example is Model-Agnostic Meta-Learning, or MAML (Finn et al., 2017). MAML learns an initial set of parameters from which a new task can be reached with relatively few gradient updates.
Conceptually, this is already closer to what we want: use experience across previous tasks to make future adaptation faster.
But classical optimisation-based meta-learning still performs optimisation on the new task. For modern VLAs, this means we still need a differentiable target signal, an adaptation loop and enough computation to backpropagate through a large policy or its adapters.
WIZARD takes the amortisation idea one step further. Instead of meta-learning an initialization from which adaptation is faster, we ask whether the adaptation result itself can be generated.
In-context adaptation: changing behaviour without changing weights
Foundation models suggest another possibility: keep the weights fixed and describe the new task through context.
This is extremely powerful in language models. A few examples in a prompt can change how a model solves a problem without any parameter update. Robotics has increasingly explored related ideas through multimodal prompting and demonstration-conditioned policies, including systems such as VIMA.
But physical action generation is a stricter problem. The output has to be a sequence of actions that remains correct as those actions change the world.
Prompting also cannot guarantee that the fixed policy already contains a sufficiently accessible representation of the required behaviour. If the new task falls outside the behaviours that the fixed policy can reliably invoke from context, adding more context does not itself create a new task-specific parameterisation.
This leads to the space WIZARD explores: preserve the deployment-time efficiency of inference, but use that inference to modify the policy itself.
~0.5 min
From optimisation to generation
Most adaptation algorithms implicitly treat the parameters for a new task as the solution to an optimisation problem.
Start with parameters W. Collect target-task examples. Compute a loss. Follow its gradients. Repeat until the resulting W + ΔW produces the desired behaviour.
WIZARD asks whether ΔW can instead become the output of another model.
There is a useful analogy with amortised inference. Suppose solving a problem numerically requires running an iterative optimiser for every new input. If we solve enough examples, we may eventually be able to train a network that maps each input directly to a good solution.
WIZARD applies the same principle to robot adaptation.
During training, we solve many conventional adaptation problems. For each source task, we train an expert adapter. These experts show us where different tasks live in the policy’s weight space.
The meta-network then learns the mapping:
task evidence → task-specific parameters
At deployment, it no longer has to rediscover those parameters through gradient descent. A visual demonstration and language instruction specify the task; the generator predicts an adapter.
WIZARD does not eliminate optimisation from the entire learning system. Source experts and the generator still have to be trained offline. What it attempts to eliminate is dedicated target-task optimization at deployment.
That is what makes the reframing interesting: adaptation becomes an inference problem whose cost can be amortised across tasks.
~0.5 min
Why generate LoRA adapters instead of an entire robot policy?
Generating the parameters of a modern VLA directly would be an enormous prediction problem. The π0.5 model used in our experiments has billions of parameters, and most of them do not need to change for every new task.
LoRA gives us a much smaller target. Rather than modifying every weight directly, it adds a structured low-rank update to each matrix: W’ = W + BA, where A and B are much smaller matrices whose product has low rank. For a large pretrained network, a compact update is often enough to steer behaviour toward a new task.
For WIZARD this is doubly useful. LoRA gives each task a compact representation in weight space — we ask the meta-network to generate low-rank factors, not an entire VLA. And the base model stays frozen, so every generated adapter is expressed relative to the same underlying policy.
In our π0.5 implementation we use rank-16 LoRA adapters across the model’s linear modules. The resulting adapter contains 46.1 million parameters — about 1.27% of the full model, generated in roughly 70 milliseconds on an NVIDIA A6000. Attempting to generate the full π0.5 parameter set directly exceeded the available 48 GB of memory in our setup.
LoRA, in other words, is what makes weight generation tractable in the first place.
~2 min
How WIZARD learns to generate a robot adapter
The core of WIZARD is a meta-network trained not to predict robot actions, but to predict the parameters of another network.
That creates a different learning problem from standard imitation learning.
Source-task experts
We begin with a collection of source manipulation tasks.
For every source task, we freeze the π0.5 backbone and train a task-specific rank-16 LoRA adapter using conventional action-labelled demonstrations. These adapters are our source experts.
This stage uses the information a standard supervised robotics pipeline would normally require: observations, robot state and actions.
Once an expert exists, its LoRA parameters become a training target. We can now ask what evidence about the task would have been sufficient to predict those parameters. For the generator, we deliberately remove the robot actions.
Representing the task
WIZARD receives two pieces of evidence: the natural-language instruction and the visual observations from a demonstration.
Both are passed through the frozen representation pathway of the VLA itself. This produces a multimodal representation of what the task asks for and what performing it looks like.
One design choice proved particularly important: we use the robot model’s own multimodal representation rather than a generic language embedding. When we replaced it with task representations derived from Gemma or Qwen, different tasks became much less separable in representation space and adapter-generation performance fell to zero in our LIBERO-Spatial ablation.
The signal has to encode behaviourally relevant differences between tasks, not simply linguistic similarity.
Language and vision also need to be used together. Vision alone reaches 25% on LIBERO-Spatial; language and vision together reach 40%; removing visual conditioning entirely causes performance to collapse to zero. Language tells us what the intended task is. The visual demonstration grounds that instruction in a particular physical arrangement and behaviour.
Structuring the weights
The output side presents a different problem.
A VLA is not one homogeneous matrix. Its LoRA parameters belong to many different parts of the architecture — vision, language, multimodal processing, action-related projections — with different shapes and numerical scales.
Simply flattening everything into one enormous vector discards this structure. Instead, WIZARD serialises the adapter into a structured representation where functionally related parameters stay grouped. This gives the meta-network something closer to a structured weight map than an unstructured list of millions of numbers.
Different parameter groups also have very different numerical scales. We normalise them independently and ask the generator to predict both the normalised values and the statistics required to restore their original scale. In our ablations, predicting approximately the right pattern of weights was not enough — their magnitude and direction both had to be modelled explicitly before performance moved off zero.
The generator
The resulting meta-network, HyperConvVLA, is a convolutional weight generator conditioned on the aligned VLA representation.
Training uses three objectives: a reconstruction loss encouraging the generated adapter to match the source expert parameter by parameter; a scale loss encouraging correct parameter statistics; and a cosine loss encouraging the generated update to point in a similar direction in weight space.
Loss ablation · LIBERO-Spatial average success
Training objective
Avg. success
Reconstruction only
0%
Reconstruction + cosine
0%
Reconstruction + scale
27%
Reconstruction + scale + cosine
33%
Components of the training objective removed in turn: performance stays at zero until the scale of the parameters is modelled explicitly.
The result is a learned mapping from a representation of a manipulation task to the parameter update associated with that task.
Adaptation becomes an inference problem whose cost can be amortised across tasks.
What happens when the robot sees a new task?
Deployment is much simpler than training.
A new target task arrives. WIZARD receives its instruction and one visual demonstration — without the demonstration’s robot actions. The frozen VLA converts the instruction and video into a task representation. The meta-network processes that representation once and predicts the structured rank-16 LoRA adapter, together with its scale statistics. The adapter is reconstructed and inserted into the frozen π0.5 backbone. The resulting specialised policy can then execute the target task.
No target-task backward pass. No gradient updates. One forward pass through the meta-network.
There is also an interesting negative result here. We tested whether giving WIZARD more demonstrations would improve performance by averaging representations from 1, 3, 5 or 10 examples. On LIBERO-Spatial, the result stayed essentially unchanged across all four. The current model therefore does not yet exploit additional demonstrations — a limitation we return to later.
~2.5 min
What happens on tasks WIZARD has never trained on?
The central experimental question is not whether WIZARD can reproduce adapters for tasks it has already seen. It is whether the mapping learned across source tasks can produce useful parameters for held-out tasks.
Holding out entire LIBERO task distributions
LIBERO is a manipulation benchmark organised into four complementary suites.
LIBERO-Spatial changes where objects are located. LIBERO-Object changes which objects must be manipulated. LIBERO-Goal varies the desired goal and behaviour. LIBERO-10 contains longer, multi-stage tasks.
Each suite contains ten tasks.
Our primary experiment uses a deliberately strong distribution shift. When evaluating one suite, we remove all ten of its tasks from WIZARD training and use only the other three suites as source tasks.
For example, when LIBERO-Spatial is the target distribution, WIZARD never sees any LIBERO-Spatial task while learning its task-to-weight mapping.
Each generated policy is then evaluated from 50 predefined initial states per target task.
On LIBERO-Spatial, the frozen pretrained π0.5 achieves 0%. Nearest-neighbour retrieval — simply taking the adapter from the most similar source task — reaches 2%. Fine-tuning π0.5 with one action-labelled target demonstration reaches 19%, while the LoRA-Gen weight-generation baseline reaches 28%.
WIZARD reaches 40%.
LIBERO-Goal follows the same pattern: 14% for one-demonstration fine-tuning, 17% for LoRA-Gen and 22% for WIZARD.
The result that is just as important is where this approach does not work well.
On LIBERO-Object, WIZARD reaches only 3%, compared with 1% for both fine-tuning and LoRA-Gen. In this held-out setting, the target suite introduces object categories absent from WIZARD’s source tasks. Generating parameters for unseen object semantics is substantially harder than transferring spatial relations or goal structures.
LIBERO-10 exposes a different weakness. These tasks require multiple behaviours to be sequenced over a longer horizon. When we evaluate their constituent parts separately, WIZARD improves over the baselines, but complete long-horizon behaviour remains far from solved.
Each suite is held out entirely — all ten of its tasks removed from training. 50 initial states per target task. LIBERO-10 is reported per part (A/B). LoRA-Gen was not evaluated on the LIBERO-10 parts.
That pattern is revealing. WIZARD can often recover useful task-specific behaviour, but semantic extrapolation and temporal composition remain much harder than generating a variation of a behaviour represented across the source distribution.
The result also survives a change of benchmark: on MetaWorld ML45, trained on 45 source tasks and evaluated on five held-out targets with different manipulation dynamics, WIZARD reaches 66% average success against 58% for one-demonstration fine-tuning and 60% for LoRA-Gen. A smaller improvement than on LIBERO-Spatial, but the benefit transfers.
Franka · pick the cup — held-out task, one demonstration, no action labelsFranka · pick the banana — held-out task, one demonstration, no action labels
Watch the approach and grasp: WIZARD never receives the target demonstration's action labels and performs no target-task gradient updates.
This experiment is worth describing carefully. It is not a pure simulation-to-real transfer test. Both WIZARD and the fine-tuning baseline begin from the same π0.5 checkpoint already fine-tuned on DROID real-world robot data. The experiment asks whether WIZARD’s adaptation mechanism remains useful once both approaches share the same real-world grounding.
We use five tabletop tasks: pick the banana, pick the apple, pick the marker, pick the cup, and move the apple close to the cup.
For each evaluation, one task is held out. The remaining four are source tasks. A single demonstration of the target task is collected by teleoperation. Fine-tuning receives the corresponding action labels; WIZARD receives the same demonstration without those labels.
Each resulting policy is evaluated over 30 physical rollouts.
Fine-tuned π0.5 averages 17% success. WIZARD reaches 37%, improving on the baseline on all five target tasks.
The largest gain is on pick the cup: 30% with one-demonstration fine-tuning and 63% with WIZARD.
Both methods start from the same π0.5 checkpoint already fine-tuned on DROID. One teleoperated demonstration per held-out task: fine-tuning receives its action labels, WIZARD does not.
The robot experiment is still small, and 37% success is clearly not deployment-ready reliability. But it tells us something important: the generated weights are not merely exploiting quirks of a simulator’s dynamics or rendering. They can produce measurable behavioural improvements on a physical robot under a held-out-task protocol.
~1 min
Where’s the gap
Transfer is strongest when the source distribution already contains related structure.
The experiments expose several limitations that we think are as informative as the positive results.
The first is semantic extrapolation. LIBERO-Object is the clearest example. When the target tasks depend on object categories that are not represented in the source distribution, performance remains very low. A weight generator cannot yet reliably invent expertise that is too far outside the behaviours encoded by its training tasks.
The second is long-horizon composition. WIZARD generates one adapter intended to specialise the policy for an entire task. That works better for relatively coherent behaviours than for tasks requiring multiple distinct skills to be selected and sequenced over time. LIBERO-10 suggests that recovering the constituent behaviours is easier than composing them into a complete trajectory.
The third is how task evidence is aggregated. Giving the current model more demonstrations does not improve average performance because we simply average their representations. Richer mechanisms could use several demonstrations to identify invariant task structure, distinguish useful from uninformative frames or represent uncertainty over the intended behaviour.
The fourth is the cost that has been moved offline. WIZARD does not make training free. It requires task-specific source experts, and these expert adapters supervise the generator. Each source expert takes roughly an hour to train and the meta-network roughly two hours on our setup, although source experts can be trained independently and in parallel. A central scalability question is therefore how performance grows with the number and diversity of source tasks.
Finally, our physical evaluation covers five tasks and starts from an already real-world-grounded checkpoint. Much broader testing is required before making claims about arbitrary real-world deployment, cross-embodiment transfer or adaptation to entirely new physical domains.
These limitations point to the same underlying problem: generating parameters is only as powerful as the representation of the task and the structure learned across previous adaptations.
~1.5 min
Why this matters for physical AI
We see three broader implications.
1. The economics of deployment change when adaptation becomes inference
Consider a robot deployed across many production cells.
Under conventional adaptation, every new variant can trigger a small machine-learning project: collect robot trajectories, annotate or preserve the action signal, launch training, evaluate checkpoints, validate the behaviour and distribute a new adapter.
WIZARD does not remove the need for validation — physical systems should not deploy unknown behaviour blindly — but it changes the expensive part of the loop .
The target evidence can be a demonstration used to describe the task rather than a labelled dataset used to optimise it.
The difference becomes more important as the number of deployment conditions grows.
Our data-efficiency experiment gives a concrete example. On one LIBERO-Spatial task, WIZARD reaches 90% success from a single demonstration without action labels. Fine-tuning π0.5 requires approximately 25 action-labelled demonstrations to reach the same performance.
This does not mean one unlabelled demonstration will replace 25 demonstrations universally. It shows that cross-task experience can sometimes substitute for target-task supervision when adaptation itself is learned.
2. Adaptation can absorb part of the variability of the physical world
Physical systems face a combinatorial deployment problem.
The same nominal task can change because of object identity, pose, geometry, workspace arrangement, goal specification, robot platform or environment.
It is unlikely that one static policy will behave perfectly under every combination.
One response is to continually add all of these cases to one ever-larger training distribution. Another is to give the deployed system a mechanism for specialising itself.
WIZARD explores the latter.
The important distinction is that the generated adapter is not retrieved from a library of known tasks. Nearest-neighbour retrieval performs poorly in our held-out LIBERO experiments. The generator is learning a mapping that can place a new task at a new point in parameter space.
The current results also tell us where this idea needs to become stronger. Spatial and goal changes transfer considerably better than unseen object semantics. A useful adaptive physical-AI system will need task representations and parameter generators capable of extrapolating across all of these axes.
3. Neural-network weights can become outputs, not only learned state
The most interesting implication for us is methodological.
Deep-learning systems normally treat their parameters as the endpoint of training. Data goes in, optimisation happens, weights come out — and we then regard those weights as fixed until another training run.
Weight-space learning makes that boundary less rigid.
A network’s parameters can themselves become a structured prediction target.
That opens a different hierarchy of learning systems. One model acts in the environment. Another model observes a problem and modifies the first. Eventually, adaptation could involve generating, composing, routing or revising small parameter modules as the situation evolves.
WIZARD is an early version of this idea: one task representation generates one static LoRA adapter.
But the broader research question is larger: can a general physical-AI system learn not only what action to take, but what internal parameters it should use to solve the problem it is facing?
Generated weights are useful even when gradient-based adaptation remains available.
On LIBERO-Spatial Task 10, a target expert reaches 96% success. Starting subsequent fine-tuning from a WIZARD-generated adapter reaches that level after 70 optimisation steps, compared with 90 steps from a standard LoRA initialization.
Generation and optimisation do not have to be competing paradigms. Generation can provide the initial hypothesis. Optimisation can refine it when labelled target experience becomes available.
~0.5 min
What comes next
WIZARD turns task adaptation into a prediction problem, but the current formulation is deliberately simple: infer one task representation, generate one adapter and use that adapter for the whole rollout.
The failure modes suggest the next steps.
Better task representations could improve extrapolation to genuinely new semantics. Better aggregation could allow the model to learn more from several demonstrations instead of averaging them. Scaling meta-training across a much larger and more diverse set of source behaviours could reveal whether a richer weight-space structure emerges as the task distribution grows.
Long-horizon behaviour raises an even more fundamental question. A complex task may not correspond to one fixed point in weight space. It may require different specialisations at different stages of execution.
That suggests moving from task-level weight generation toward dynamic and compositional weight adaptation: generating reusable parameter modules and selecting or composing them as the robot’s current subproblem changes.
The long-term goal is not simply faster fine-tuning.
It is to build policies for which adaptation is itself a learned capability — systems that can observe a new problem, infer how they need to change, and reorganise their own behaviour accordingly.
We use Google Analytics to understand how the site is used. Nothing is collected until you accept, and the site works the same without it. Your choice is remembered on this device for 30 days.
