Papers
arxiv:2406.16860

Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs

Published on Jun 24
Β· Submitted by ellisbrown on Jun 25
#1 Paper of the day
Authors:
,
,
,

Abstract

We introduce Cambrian-1, a family of multimodal LLMs (MLLMs) designed with a vision-centric approach. While stronger language models can enhance multimodal capabilities, the design choices for vision components are often insufficiently explored and disconnected from visual representation learning research. This gap hinders accurate sensory grounding in real-world scenarios. Our study uses LLMs and visual instruction tuning as an interface to evaluate various visual representations, offering new insights into different models and architectures -- self-supervised, strongly supervised, or combinations thereof -- based on experiments with over 20 vision encoders. We critically examine existing MLLM benchmarks, addressing the difficulties involved in consolidating and interpreting results from various tasks, and introduce a new vision-centric benchmark, CV-Bench. To further improve visual grounding, we propose the Spatial Vision Aggregator (SVA), a dynamic and spatially-aware connector that integrates high-resolution vision features with LLMs while reducing the number of tokens. Additionally, we discuss the curation of high-quality visual instruction-tuning data from publicly available sources, emphasizing the importance of data source balancing and distribution ratio. Collectively, Cambrian-1 not only achieves state-of-the-art performance but also serves as a comprehensive, open cookbook for instruction-tuned MLLMs. We provide model weights, code, supporting tools, datasets, and detailed instruction-tuning and evaluation recipes. We hope our release will inspire and accelerate advancements in multimodal systems and visual representation learning.

Community

Paper author Paper submitter

πŸͺΌ Paper: https://arxiv.org/pdf/2406.16860
🌎 Website: https://cambrian-mllm.github.io
πŸ’» Code: cambrian-mllm/cambrian
πŸ” Checkpoints: Cambrian-1
πŸ“Š Data: Cambrian-10M
🏁 Benchmark: https://huggingface.co/datasets/nyu-visionx/CV-Bench

Congrats on your work! Excited to see that it's fully open source! ✨

Top scorer for Youssef Al-Nasri in the 2022 World Cup

congrats! can't wait to read it!

Sign up or log in to comment

Models citing this paper 4

Datasets citing this paper 3

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2406.16860 in a Space README.md to link it from this page.

Collections including this paper 17