Jitendra MALIK

Three data sources are popular for robot learning today (1) Humans tele-operating robots (2) Humans acting as humans doing tasks (3) Robots acting in the "safe space" of a simulator. Happy to share this work from UC Berkeley which shows the power of (2) combined with (3). Visual imitation followed by trial and error is a good recipe for robots as it is for human children. Whither tele-op?

Very cool result, Congrats! There is an insightful quote from philosopher Merleau-Ponty, " We know not through our intellect but through our experience" . The essence of robot learning is with reinforcement learning, as it is with humans.

3D/4D reconstruction is one of the great achievements of computer vision. With roots in photogrammetry, multi-view geometry was much studied (e.g. Hartley & Zisserman). In the last decade, it was successfully combined with deep learning. Shape priors which come from recognition got incorporated (e.g. for humans, in HMR) and recent systems like VGGT and SAM 3D can do amazing stuff. This progress is directly relevant for robotics, which (duh) operates in the 3D world. Many robotics papers in the learning era weren't exploiting 3D structure, which IMHO is just wasting valuable signal. But there has been a strong real2sim2real tradition for years e.g. here are two projects from my group (and there are many others, I am not claiming exclusivity). I am glad that people have rediscovered this capability with Astra. But please do give credit to human researchers, not just AI models. zhec.github.io/rhoi/ ,
videomimic.net

Toru raises fundamental concerns with which I agree. It is amazing to see the progress in VLMs such as Astra, building on the work of generations of scientists. But Astra doesn't cite what it is building on (I have co-authored a couple of in-hand rotation papers which might have been used in the Astra pen-spinning demo). I fully acknowledge that VLMs make the result of previous research much more accessible to the general public, just like encyclopedias and Google Search did earlier, and that is a good thing. However, in the past, society had mechanisms like patent disclosures and paper citations as ways of doing credit assignment. Should we not accuse these AI models of plagiarism if they don't cite their sources? Somewhat relatedly, I note the open letter by 25 Fields Medalists in response to the Navier Stokes result complaining about the disruption caused to the mathematics research process. Some people interpreted it as turf protection, but the point was more subtle. If certain kinds of creative work stops because incentives are disrupted, it is like farmers eating their seed corn.

In the old days we used to talk about "graduate student descent" or "researcher gradient descent" as referring to the phase in research where one tries various techniques, changes parameters in the hope of getting to some desired output. Now we can get AI agents to do it. The classic CV paradigm of analysis by synthesis becomes considerably more powerful this way.

An open letter signed by 24 Fields Medalists is worth paying attention. You may want to peruse "A Severe Misalignment of AI in Mathematics" mathandai.org

in view of OpenAI's claim about Navier Stokes, it is important that we hear alternative points of view on how it happened. From Tristan Buckmaster, Prof. at NYU. cims.nyu.edu/~tristanb/stat…

I am seeing claims around LLMs, specifically Astra having made serious progress on robotics. The tasks that are demonstrated are simple pick and place tasks with parallel jaw grippers. LLMs can do planning, and the impressive demos in these tasks primarily show that. But robotics is also about dexterity and dynamics - which is why we need high frequency controllers/policies that can deal with torques and forces. So here is a simple challenge. Can you prompt an LLM to output the high frequency control commands for a legged robot in varying terrain e.g. ashish-kmr.github.io RSS 2021, CoRL 2022 (this is by now 5 year old technology, so I am not picking a particularly hard task). I am not questioning the usefulness of LLMs for high level planning or in agentically assisting a robotics researcher (we use them all the time!).

Scientific terms should have precision. If we use the terms VLM, VLA, WAM in an indiscriminate fashion, as is becoming common in robotics, we are not helping clarity in communication. Let's keep the historical origins of these terms in mind. VLMs arose as multimodal extensions of LLMs-the training was for tasks like VQA (VIsual Question Answering). These capture the static semantics of the scene behind an image. No dynamics. World Models (e.g. @ylecun , Ha & Schmidhuber 2018) on the other hand are primarily dynamics models, which go back to control theory -1960 (Bellman, Kalman etc.) This makes them natural for robotics planning / policies- I am in a state s, what action a should I perform to get to state s'. In classical control, these models were written down a priori by modeling the physics of the system; today we think of them as learned neural networks trained from temporal data e.g. video, robot trajectories. But the concept is the same. We shouldn't mix this concept with VLMs.

Nice idea to train to predict future sensory inputs along with action sequences. @ir413 Ilija Radosavovic et al (NeurIPS 2024) showed this for humanoid locomotion but the idea is perfectly general proceedings.neurips.cc/paper_files/pa… PS: Calling these models VLAs is a stretch.

I concur, and the point is broader than just for Biology. Science does not advance by "pure thinking" alone. We need to do experiments, and interpret the results of those experiments, which in turn raises new questions which are resolved by new experiments.

Highly performant open weights frontier models such as Kimi are a competitive threat to OpenAI & Anthropic, but probably for everyone else these are a win. Hope more US entities will release top quality open weights models as well. Government regulation of AI models to prevent public harm could be done equally for proprietary or open weight models. Classic arguments in favor of open source software apply to AI models too.

Novel approach to training tactile policies..

In computer vision, the field advanced significantly due to openly available datasets and models; hope this will be true in robotics as well.

We can convert human videos to robot hand-object interaction trajectories in 4D. Enjoy!
Paper: arxiv.org/abs/2606.19333
Website: do-as-i-do.com
Code: github.com/malik-group/do…
Authors:@bhawna_paliwal_,@HarithejaE,@willjhliang, @pabbeel , @notmahi , @JitendraMalikCV

I want to offer some unsolicited advice to computer vision researchers jumping into robotics. Don't focus too much on VLMs, VLAs etc. That's fine, but the real action is at the sensorimotor level. Most of the open problems in robotics are in manipulation, which is about hand-object interaction, and contacts and forces are central. Proprioception and tactile sensing are as important as vision. Don't get seduced by cherry-picked demos. You can't do robotics without doing robotics.

SAM 3D: 3Dfy Anything in Images received an honorable mention for best paper at CVPR 2026. Even more satisfying, it has enabled learning human-object-interaction trajectories from video for training robots (I am at ICRA and had numerous conversations on this!). You can read the paper at openaccess.thecvf.com/content/CVPR20…

We have seen some impressive robot manipulation policies recently. This is great, but for these results to be convincing (and practical) we should insist on generalization across 1. Object location (within some range) 2. Different instances of an object category 3. Background clutter. Authors should present experiments which demonstrate the range of variation which can be handled. Far too often the policy doesn't even generalize across all instances of an object category! Legged locomotion policies were convincing only when they worked across different terrain as in ashish-kmr.github.io/rma-legged-rob… We need to do the same for manipulanda.

I see every week on X an announcement or demo which implies that robotic manipulation has been solved. The only reason I don't believe it is because manipulation had already been solved last week by somebody else! So may I propose the "5 year old paired comparison test" ? At the next conference let's set up a number of tables to which you can bring your robot hardware. Next to it we will have another table where there will be a 5 year old child. In parallel we will try 100 different manipulation tasks that a neutral person has chosen- we could start with "pick up anything" - only household objects (e.g. as might be found in a typical American home) will be used, and we compare the performance of your robot with that of the 5 year old. Can you pick up a coin? Or a book? Or untwist a bottle top? Or insert any plug into a matching socket? Rotate one face of a Rubik's cube? Until your robot can do all the "open world manipulation" that a 5 year old kid can, some humility is in order.

In robotics manipulation, we see many cherry picked demos but no standardized benchmarks. I suggest using STT (Success weighted by normalized inverse Task Time), analogous to SPL from navigation, replacing length by time to do task relative to a human e.g. for "Pick up anything" on random household objects. arxiv.org/pdf/1807.06757