Meta-Transformer: A Unified Framework for Multimodal Learning -- Unified Multimodal Learning. Meta-Transformer utilizes the same backbone to encode natural languages, images, point clouds, audio, video, infrared, hyperspectral, X-ray, time-series, tabular, and graph data. It reveals the potential of transformer architectures for universal perception.
Multimodal learning involves utilizing data from various modalities to improve model capacity. Despite the years of development in this field, it remains challenging to devise a unified framework for processing natural language, 2D images, 3D point clouds, and audio spectrograms due to crucial gaps among these different modalities. This study proposes a novel approach that demonstrates a network with frozen parameters can encode the data from the aforementioned four modalities and achieve favorable performance, resulting in a unified framework called Meta Transformer. Using this framework, the raw input data from various modalities are converted to a shared token space, allowing a subsequent encoder with frozen parameters to extract high-level semantic features of the input data. Composed of three main components: a unified data tokenizer, a modality-shared encoder, and task-specific heads for downstream tasks ...
Tiny LLM - LLM Serving in a Week -- uses MLX (on MacOS) to ... implement an LLM serving project from scratch -- with only matrix manipulations APIs, so that I can understand what it takes to load those LLM model parameters and do the math magic to generate text.
Lil'LogHi, this is Lilian. I’m documenting my learning notes in this blog. Other than writing a ML blog, I’m leading Applied Research at OpenAI on the side.
Microsoft LMOps -- a research initiative on fundamental research and technology for building AI products w/ foundation models, especially on the general technology for enabling AI capabilities w/ LLMs and Generative AI models.
Sparks of Artificial General Intelligence: Early experiments with GPT-4. (2023) PDF -- (Bubeck et al. 2023)
Supervised Learning - precise mathematical condition under which learning should succeed, which is - Low training error + more training data than "degrees of freedom" = low test error
Prompt Engineering | Lil'Log -- "This post only focuses on prompt engineering for autoregressive language models, so nothing with Cloze tests, image generation or multimodality models."
Controllable Neural Text Generation | Lil'Log -- "How to steer a powerful unconditioned language model? In this post, we will delve into several approaches for controlled content generation with an unconditioned langage model. For example, if we plan to use LM to generate reading materials for kids, we would like to guide the output stories to be safe, educational and easily understood by children."
How is LLaMa.cpp possible?how can we run llama.cpp on local machines when the expectation is that large models need expensive GPUS (eg: A100) to run
Introducing Code Llama, a state-of-the-art large language model for codingCode Llama is a code-specialized version of Llama 2 that was created by further training Llama 2 on its code-specific datasets, sampling more data from that same dataset for longer. Essentially, Code Llama features enhanced coding capabilities, built on top of Llama 2. It can generate code, and natural language about code, from both code and natural language prompts (e.g., “Write me a function that outputs the fibonacci sequence.”) It can also be used for code completion and debugging. It supports many of the most popular languages being used today, including Python, C++, Java, PHP, Typescript (Javascript), C#, and Bash.
gptscript-ai/gptscript: Build AI assistants that interact with your systems "GPTScript is a framework that allows Large Language Models (LLMs) to operate and interact with various systems. These systems can range from local executables to complex applications with OpenAPI schemas, SDK libraries, or any RAG-based solutions. GPTScript is designed to easily integrate any system, whether local or remote, with your LLM using just a few lines of prompts."
Fine-tuning has one huge advantage though: it is far more effective at guiding a model's behavior than prompting, so you can often get away with a much smaller model. That gets you faster responses and lower inference costs. A fine-tuned Llama 7B model is 50x cheaper than GPT-3.5 on a per-token basis, and for many use cases can produce results that are as good or better!
For example, classifying the 2M recipes at https://huggingface.co/datasets/corbt/all-recipes with GPT-4 would cost $23k. Even with GPT-3.5 it would cost over $1k. The model we fine-tuned performs similarly to GPT-4 and costs just $19 to run over the entire dataset.
OpenBMB/ToolBench: An open platform for training, serving, and evaluating large language model for tool learning.
Haystack | HaystackOpen-source LLM framework to build production-ready applications.
Use the latest LLMs: hosted models by OpenAI or Cohere, open-source LLMs, or other pre-trained models
All tooling in one place: preprocessing, pipelines, agents & tools, prompts, evaluation and finetuning
Choose your favorite database: Elasticsearch, OpenSearch, Weaviate, Pinecone, Qdrant, Milvus and more
Scale to millions of documents: use Haystack's proven retrieval architecture
Compare it to LangChainAI
GPT4AllA free-to-use, locally running, privacy-aware chatbot. No GPU or internet required.
an AI proxy that lets you use a variety of providers (OpenAI, Anthropic, LLaMa2, Mistral, and others) behind a single interface w/ caching & API key management.
MLC LLMMachine Learning Compilation for Large Language Models (MLC LLM) is a high-performance universal deployment solution that allows native deployment of any large language models with native APIs with compiler acceleration. The mission of this project is to enable everyone to develop, optimize and deploy AI models natively on everyone’s devices with ML compilation techniques.
Shoggoth is a peer-to-peer, anonymous network for publishing and distributing open-source code, Machine Learning models, datasets, and research papers.
modal-labs/quillmanA complete chat app that transcribes audio in real-time, streams back a response from a language model, and synthesizes this response as natural-sounding speech. This repo is meant to serve as a starting point for your own language model-based apps, as well as a playground for experimentation.
"phi-3-mini, a 3.8 billion parameter language model trained on 3.3 trillion tokens, whose overall performance, as measured by both academic benchmarks and internal testing, rivals that of models such as Mixtral 8x7B and GPT-3.5 (e.g., phi-3-mini achieves 69% on MMLU and 8.38 on MT-bench), despite being small enough to be deployed on a phone." (Abdin et al. 2024) (No code, or model was announced with the paper.)
(@liu2024kan)
(Lin et al. 2024) "Unstructured data formats account for over 80% of the data currently stored, and extracting value from such formats remains a considerable challenge.... ZenDB efficiently extracts semantic hierarchical structures from such templatized documents, and introduces a novel query engine that leverages these structures for accurate and cost-effective query execution. Users can impose a schema on their documents, and query it, all via SQL. Extensive experiments on three real-world document collections demonstrate ZenDB's benefits, achieving up to 30% cost savings compared to LLM-based baselines, while maintaining or improving accuracy, and surpassing RAG-based baselines by up to 61% in precision and 80% in recall, at a marginally higher cost."
"To make deliberate progress towards more intelligent and more human-like artificial systems, we need to be following an appropriate feedback signal: we need to be able to define and evaluate intelligence in a way that enables comparisons between two systems, as well as comparisons with humans."
"We then articulate a new formal definition of intelligence based on Algorithmic Information Theory, describing intelligence as skill-acquisition efficiency and highlighting the concepts of scope, generalization difficulty, priors, and experience, as critical pieces to be accounted for in characterizing intelligent systems."
JAX is a Python library which augments numpy and Python code with function transformations which make it trivial to perform operations common in machine learning programs. Concretely, this makes it simple to write standard Python/numpy code and immediately be able to
Compute the derivative of a function via a successor to autograd
Just-in-time compile a function to run efficiently on an accelerator via XLA
Automagically vectorize a function, so that e.g. you can process a “batch” of data in parallel. learn more ... "You don't know Jax" (2019)
The little book of deep learning [pdf] by François Fleuret - professor of computer science at the University of Geneva, Switzerland; Apr 2023. #phoneformatted. See also 100-page-book
udlbook/Understanding Deep Learning by Simon J.D. Prince. Has 68 Jupyter Notebooks, which can be downloaded from github. Plots are drawn with -- Inkscape with the texText plugin to do the equations. Free and highly recommended. I draw the axes and ticks by hand because it looks nicer than anything from PyPlot etc.2023-11-15¶
"collect more data" vs "tune hyper parameters" - systematic approach to making this decision is the distinction between a beginner and a experienced person.
