# mPLUG-Owl3: Towards Long Image-Sequence Understanding in Multi-Modal Large Language Models > Research note on mPLUG-Owl3, a multimodal LLM with hyper attention for long image-sequence understanding and multi-image reasoning. - Authors: Hunter Heidenreich - Published: 2024-08-08 - Updated: 2026-03-04 - Paper: mPLUG-Owl3 Paper - Code: mPLUG-Owl3 GitHub - Model: mPLUG-Owl3-1B (HuggingFace) - Model: mPLUG-Owl3-2B (HuggingFace) - Model: mPLUG-Owl3-7B (HuggingFace) - Source: Bevaya Labs, --- ## What kind of paper is this? - This paper introduces a new model, building upon the past-line of mPLUG models. - I'd say it's a relatively small idea plus evaluation, but they present an intersting architectural variation that seems to enable their scaling up to long image sequence lengths. ## What is the motivation? - The motivation is to **improve long image-sequence understanding** in multimodal LLMs. - This has massive downstream impact, enabling in-context learning with long sequences of images and text, and also applications to video understanding and other multimodal tasks. ## What is the novelty? - The most relevant related work is the original [mPLUG-Owl](2023_04_mplug_owl.md) model and all subsequent mPLUG models. - This paper contributes by showing massive gains on scenarios where one wants to have a very long interleaved sequence of images and text, like video understanding or long-form image-text understanding. ## What experiments were performed? - They show marginal or borderline improvements (or at least no degradation) in performance on standard VQA and image-text tasks. - Where they really gain is on multi-image tasks and benchmarks - Where they stand out light-years above other models is when they design a new benchmark which they call Distractor Resistance that requires a model to perform reasoning on hundreds of images in a sequence. - Here, they demonstrate an ability to perform this task, albeit with degrading performance as the number of images increases. - Most models fail to perform this task at all, so this is a significant step forward. ## Dataset ## Methodology Highlights - One of the key things the authors do is allow for continual cross-attention between the language and vision outputs - They call this hyper attention - It just allows visual features to update language ones; not bidirectional - It isn't applied in every block; it's a sparsely added layer - The same layer norm is applied to both text and visual input to encourage similar semantic spaces - Modality specific KQV matrices are used to allow these components to specialize - This is reminiscient of past mPLUG-Owl family models - They only let text refer to image features that occur before the text in the sequence, unlike other models that just attend to all images provided - Adaptive gating (gating that is input dependent on the text features) is used to provide a final merging of the text representation update with the current representation ## Training Details - Stage 1: Pre-training - This stage freezes most of the weights - Only a linear projection, vision KV projection matrices, and the adaptive gate parameters are trained - Sequence length is kept shorter (768) and resolution is kept small (384^2) - 20k updates, batch size of 2048 - Stage 2: Multi-image training - This stage focuses on training models that can reason over multiple images in an input - Now the linear projection and the full language model are updated - Sequence length expanded to 4k and full dynamic resolution enabled - 3k updates, batch size of 1024 - Stage 3: Self-Supervised Finetuning - This stage focuses on making sure the model can follow instructions provided - Again, full LLM and linear projection are updated - Same sequence length and resolution as stage 2 - 11k updates, batch size of 1024