Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generation

[00:00] Intro

[00:28] Limitation of existing unified models

[00:57] Janus's decoupled visual encoding solution

[01:18] Advantages of decoupling

[02:03] Janus architecture

[02:50] Three-stage training

[03:41] Ablation studies

[04:23] Extensions for Janus

[05:10] Performance gains

[05:47] Current limitations

[06:31] Impact of simplicity and extensibility

[07:10] Qualitative results

[08:18] Potential applications

[08:52] Key takeaways

Abstract: In this paper, we introduce Janus, an autoregressive framework that unifies multimodal understanding and generation. Prior research often…