Collections

134

Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer

Paper • 1701.06538 • Published Jan 23, 2017 • 4
Sparse Networks from Scratch: Faster Training without Losing Performance

Paper • 1907.04840 • Published Jul 10, 2019 • 3
ZeRO: Memory Optimizations Toward Training Trillion Parameter Models

Paper • 1910.02054 • Published Oct 4, 2019 • 4
A Mixture of h-1 Heads is Better than h Heads

Paper • 2005.06537 • Published May 13, 2020 • 2

-

STaR: Bootstrapping Reasoning With Reasoning

Paper • 2203.14465 • Published Mar 28, 2022 • 2
DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models

Paper • 2401.06066 • Published Jan 11 • 42
DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model

Paper • 2405.04434 • Published May 7 • 13
Prompt Cache: Modular Attention Reuse for Low-Latency Inference

Paper • 2311.04934 • Published Nov 7, 2023 • 28

1

Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer

Sparse Networks from Scratch: Faster Training without Losing Performance

ZeRO: Memory Optimizations Toward Training Trillion Parameter Models

A Mixture of h-1 Heads is Better than h Heads

STaR: Bootstrapping Reasoning With Reasoning

DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models

DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model

Prompt Cache: Modular Attention Reuse for Low-Latency Inference

deepseek-ai/deepseek-moe-16b-base

deepseek-ai/deepseek-moe-16b-chat

DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models

Robust Mixture-of-Expert Training for Convolutional Neural Networks

Experts Weights Averaging: A New General Training Scheme for Vision Transformers

ConstitutionalExperts: Training a Mixture of Principle-based Prompts

Mixture-of-LoRAs: An Efficient Multitask Tuning for Large Language Models

Non-asymptotic oracle inequalities for the Lasso in high-dimensional mixture of experts

MoE-LLaVA: Mixture of Experts for Large Vision-Language Models

MoE-Mamba: Efficient Selective State Space Models with Mixture of Experts

MoE-Infinity: Activation-Aware Expert Offloading for Efficient MoE Serving

Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Models

Self-Rewarding Language Models

Tuning Language Models by Proxy

DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models

AtP*: An efficient and scalable method for localizing LLM behaviour to components

The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits

MobiLlama: Towards Accurate and Lightweight Fully Transparent GPT

LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens

BlackMamba: Mixture of Experts for State-Space Models

OpenMoE: An Early Effort on Open Mixture-of-Experts Language Models

MoE-LLaVA: Mixture of Experts for Large Vision-Language Models

DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models

Simple linear attention language models balance the recall-throughput tradeoff

Linear Transformers with Learnable Kernel Functions are Better In-Context Models

Repeat After Me: Transformers are Better than State Space Models at Copying

Zoology: Measuring and Improving Recall in Efficient Language Models

BlackMamba: Mixture of Experts for State-Space Models

OpenMoE: An Early Effort on Open Mixture-of-Experts Language Models

DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models