Przejdź do treści
Global sound
Daily Paper Cast

Daily Paper Cast

DreamX: Video Diffusion Model is a Universal Latent Action Learner

· 21 min · Odcinek 2476

Notatki do odcinka

🤗 Upvotes: 39 | cs.CV, cs.AI, cs.RO

Authors:

Jisoo Kim, Chaehyun Kim, Junsu Kim, Jiho Park, Seongchan Kim, Eunbeen Hong, Soowon Son, Fu-En Yang, Seungryong Kim

Title:

DreamX: Video Diffusion Model is a Universal Latent Action Learner

Arxiv:

http://arxiv.org/abs/2611.05377v1

Abstract:

Learning a universal action representation from video requires modeling dynamics that are consistent across different executions of the same action yet invariant to changes in embodiment and appearance. Existing latent action models infer actions from visual transitions, but their representations often entangle the underlying action with irrelevant visual variations, limiting transfer across embodiments. We introduce DreamX, which repurposes a pretrained video diffusion model as a universal latent action learner. Our approach extracts compact latent actions from the video model's rich spatiotemporal features, compressing each latent video frame by $\boldsymbol{300\times}$ ($\boldsymbol{1{,}200 \rightarrow 4}$ tokens). In addition, by leveraging the video model's generative prior, these latent actions capture action-relevant dynamics disentangled from visual appearance. The resulting latent space serves as a shared action interface across embodiments (human and robot), environments, and action parameterizations. On LARY-Bench, DreamX achieves state-of-the-art performance among latent action models. Moreover, actions encoded in one setting remain effective in another: the latent space supports any-to-any video transfer and zero-shot action transfer across diverse embodiments and environments. Furthermore, this shared interface also enables policy learning in a compact latent action space, reducing the computational cost of video-diffusion-based policies.

Strona: share.transistor.fm

Na żywo, ostatnia godzina: 13 853 utworów na 5 522 stacjach w 126 krajach

Odkrywaj świat przez radio

Więcej sposobów na odkrywanie (9)

Usłyszałeś coś fajnego? Zapisz to albo otwórz od razu w Spotify.

★ Artist of the week The fastest riser on the radio this week
Your song here? From € 9 per week →
KISS FM 100.6Słuchaj dalej