MODUS: Decoder-only Any-to-Any Modeling of Diverse Modalities
MODUS is a single decoder-only model trained jointly on many modalities (RGB, depth, surface normals, segmentation, detection, edges, captions, and learned features such as DINO / CLIP / ImageBind). Given any of them as input, it can generate any of the others. This demo shows three ways to use it.
🌐 Project page · 📄 Paper EPFL · Apple · University of Copenhagen · CUHK · University of Geneva · Lambda AI
Any-to-Any Generation
Pick one input modality (an image or a caption) and generate any set of target modalities from it. Each target is generated on its own, conditioned only on the input and not on the other targets.
💡 Tip: clicking an example shows its precomputed outputs for every modality instantly (no GPU used). Selecting several output modalities at once (a few at a time) runs them in a single GPU session — far cheaper on your quota than one at a time. Fewer diffusion steps (in Advanced) = faster and less quota.
Chained Prediction
Generate modalities one after another, where each one is conditioned on the input and on every modality generated before it (for example RGB, then depth, then surface normals). Because of this self-conditioning the outputs are mutually consistent, unlike generating each one independently. Pick a condition, an intermediate (bridge) modality, and a final target, or use a quick-pick chain below.
Input Features: ViT vs VAE
MODUS can represent an input image with two kinds of features: ViT features (from a semantic encoder) and VAE features (from a reconstruction encoder). The same RGB→target is run three times, feeding the input as ViT features only, VAE features only, or both, so you can see how each representation changes the generated output.