Mustafa Mukadam

Presenting PTLD: an approach to learning tactile dexterous policies without ever simulating the tactile sensor.

Tactile is essential for performing highly dexterous manipulation. However, collecting tactile observations reliably has been a fundamental bottleneck: (1) teleoperating a multi fingered hand for dynamic tasks is challenging, making sim-to-real imperative, (2) one can’t realistically simulate tactile today

Policies trained on real robot data via imitation can be surprisingly capable. But for domains like dexterous manipulation, they are often not quite good enough: they move slowly, miss grasps, make unreliable contact, and fail under small perturbations.

Can we improve them without any additional data collection on the real robot?

In SCORE, we show that we can improve real-world diffusion/flow policies cheaply by using simulation to simply learn how to steer them on deployment. This leads to large gains in real-world success and speed across a variety of tasks, without requiring additional real-world experience:

weirdlabuw.github.io/score/

🧵 (1/10)

Has visual fidelity outpaced dynamic fidelity in tactile simulation for sim-to-real transfer of policies?

Introducing HydroShear 🏄‍♂️ : a hydroelastic tactile shear simulation for training zero-shot sim-to-real tactile policies in contact-rich tasks where fingertip force and shear matter most!

Webpage: hydroshear.github.io

Robot videos in 1x 🧵👇

Most world models for robot manipulation learn physics from pixels. But pixels don’t see it all.

Can we ground these models in the "feeling" of contact to disambiguate visually identical states?

Visuo-Tactile World Models (VT-WM): robot imagination in a shared space👇

Dexterous hands vary widely—so do tactile modalities. 🖐️🌈
Our vision on tactile human-to-robot transfer:
🔓 Not tied to specific hardware
♻️ Reuse human tactile demos across embodiments

Presenting TactAlign, a cross-sensor tactile alignment for cross-embodiment policy transfer.

How can we leverage the full spectrum of touch for more robust robot manipulation? 🦾Meet Sparsh-X, a backbone for multisensory touch representation across image, audio, motion, and pressure.
🧵1/6 👇

Just open-sourced Geometric Retargeting (GeoRT) — the kinematic retargeting module behind DexterityGen.

Includes tools for importing custom hands.
Give it a try: github.com/facebookresear…

A software by @berkeley_ai and @AIatMeta. More coming soon.

Robots need touch for human-like hands to reach the goal of general manipulation. However, approaches today don’t use tactile sensing or use specific architectures per tactile task.

Can 1 model improve many tactile tasks?
🌟Introducing Sparsh-skin: tinyurl.com/y935wz5c

1/6

Meet Vulcan, our first robot with a sense of touch. 🤖 🖐️
This innovative system combines physical AI with novel hardware solutions—like computer vision, tactile sensing, and machine learning—to reimagine how we complete orders for our customers.

With the ability to pick and stow approximately 75% of all various item types we store at our fulfillment centers, and at speeds comparable to our frontline employees, Vulcan can assist by handling inventory in harder-to-reach areas—making our fulfillment centers both safer and more efficient.