ArXiv Preprint - Mirasol3B: A Multimodal Autoregressive model for time-aligned and contextual modalities

In this episode we discuss Mirasol3B: A Multimodal Autoregressive model for time-aligned and contextual modalities by AJ Piergiovanni, Isaac Noble, Dahun Kim, Michael S. Ryoo, Victor Gomes, Anelia Angelova. The paper presents Mirasol3B, a multimodal model that handles the disparate natures of video, audio, and text modalities through separate autoregressive components, dividing the process according to the modalities' distinct characteristics. It introduces a Combiner mechanism to manage large volumes of audio and video data by partitioning input sequences into snippets and learning compact representations that capture temporal dependencies. This innovative approach achieves superior performance on multimodal benchmarks while maintaining computational efficiency compared to larger models.