Archive

probably a good study on how to fine tune a small model

🚀 How can we make sense of visual document at a large scale for retrieval? With our new paper “ColPali”, we propose to leverage VLMs to construct efficient multi-vector embeddings in the visual space for document retrieval: https://t.co/KIdzcIABcL 📝🔍 (1/10)

From Apple

In this work, we discuss building performant Multimodal Large Language Models (MLLMs). In particular, we study the importance of various architecture components and data choices. Through careful and comprehensive ablations of the image encoder, the vision language connector, and various pre-training data choices, we identified several crucial design lessons. For example, we demonstrate that for large-scale multimodal pre-training using a careful mix of image-caption, interleaved image-text, and text-only data is crucial for achieving state-of-the-art (SOTA) few-shot results across multiple benchmarks, compared to other published pre-training results. Further, we show that the image encoder together with image resolution and the image token count has substantial impact, while the vision-language connector design is of comparatively negligible importance. By scaling up the presented recipe, we build MM1, a family of multimodal models up to 30B parameters, consisting of both dense models and mixture-of-experts (MoE) variants, that are SOTA in pre-training metrics and achieve competitive performance after supervised fine-tuning on a range of established multimodal benchmarks. Thanks to large-scale pre-training, MM1 enjoys appealing properties such as enhanced in-context learning, and multi-image reasoning, enabling few-shot chain-of-thought prompting.

When writing and talking, people sometimes pause to think. Although reasoning-focused works have often framed reasoning as a method of answering questions or completing agentic tasks, reasoning is implicit in almost all written text. For example, this applies to the steps not stated between the lines of a proof or to the theory of mind underlying a conversation. In the Self-Taught Reasoner (STaR, Zelikman et al. 2022), useful thinking is learned by inferring rationales from few-shot examples in question-answering and learning from those that lead to a correct answer. This is a highly constrained setting -- ideally, a language model could instead learn to infer unstated rationales in arbitrary text. We present Quiet-STaR, a generalization of STaR in which LMs learn to generate rationales at each token to explain future text, improving their predictions. We address key challenges, including 1) the computational cost of generating continuations, 2) the fact that the LM does not initially know how to generate or use internal thoughts, and 3) the need to predict beyond individual next tokens. To resolve these, we propose a tokenwise parallel sampling algorithm, using learnable tokens indicating a thought's start and end, and an extended teacher-forcing technique. Encouragingly, generated rationales disproportionately help model difficult-to-predict tokens and improve the LM's ability to directly answer difficult questions. In particular, after continued pretraining of an LM on a corpus of internet text with Quiet-STaR, we find zero-shot improvements on GSM8K (5.9%→10.9%) and CommonsenseQA (36.3%→47.2%) and observe a perplexity improvement of difficult tokens in natural text. Crucially, these improvements require no fine-tuning on these tasks. Quiet-STaR marks a step towards LMs that can learn to reason in a more general and scalable way.

I will take short digression from jokes about COBOL and point out a pair of delightful works from Merigoux et al.:

A Modern Compiler for the French Tax Code:

The Catala Language:

The Traveling Salesman Problem approximation ratio hasn't improved from 3/2 since 1976, via MST + Minimum-Weight Perfect Matching + Eulerian Shortcuts.

The 3/2 barrier has been broken after 44 years! The abstract is one sentence, the paper is 85 pages 🤣

You: "🧐Ah mon cheri I have framed this as a very clever convex optimization problem. I'll derive recoverability bounds and include some non-reproducible tables in my paper"

The guy she said not to worry about "Gon' train this 175 billion parameter model"

This preprint paper about Multicore OCaml describes results that I have believed to be impossible for the past 15 years. This is an incredible accomplishment and a huge milestone for OCaml.