2025
Intrinsic Fingerprint of LLMs
[Published: 07-04] View
The paper proposes a robust fingerprinting method based on the inter-layer standard-deviation distribution pattern of attention parameter matrices (Q/K/V/O), used to detect lineage relationships between large language models (LLMs) — for example, whether one model was derived from another through continued training, fine-tuning, or an upgrade. It claims the method reveals that Huawei’s Pangu Pro MoE model may have been derived from the Qwen-2.5 14B model without authorization.
DeepSeek-R1
[Published: 01-22] View
The paper introduces the DeepSeek-R1 family of models, which stimulate reasoning ability in large language models through pure reinforcement learning (RL): DeepSeek-R1-Zero applies RL directly to the base model (no supervised fine-tuning), causing reflection, long chain-of-thought reasoning, and other abilities to emerge on their own, lifting AIME math competition accuracy from 15.6% to 71%; DeepSeek-R1 adds cold-start data and multi-stage training (RL + SFT) to solve problems such as language mixing, matching the reasoning performance of OpenAI-o1-1217 (AIME 79.8%); and distilled small models transfer R1’s reasoning ability to the Qwen/Llama series (1.5B–70B), where the 7B model surpasses GPT-4o and the 32B model approaches o1-mini. The models and data are open-sourced, advancing the development of efficient reasoning models.
2024
DeepSeek-V2
[Published: 05-07] View
The paper introduces DeepSeek-V2, the strongest open-source MoE large language model, which through the innovative MLA (Multi-head Latent Attention) and DeepSeekMoE (fine-grained Mixture-of-Experts) architectures achieves high performance, economical training (42.5% cost savings), and efficient inference (93.3% KV cache reduction, 5.76x throughput improvement) — 236 billion total parameters, 21 billion activated per token, with 128K context support.
DeepSeek-VL
[Published: 03-08] View
DeepSeek-VL is an open-source vision-language large model (1.3B/7B) focused on real-world applications. Its innovations include: data construction, integrating real multimodal data such as web screenshots, PDFs, and charts, and building an instruction fine-tuning dataset based on user-scenario classification; a hybrid vision encoder, fusing SigLIP (semantics) with SAM-B (detail) to process 1024×1024 high-resolution images with 576 tokens; and a training strategy, proposing a “modality warmup” technique that dynamically adjusts the ratio of language to multimodal data (ending at 7:3) to solve language-ability degradation during multimodal training. Final results: it surpasses same-scale models on 8 multimodal benchmarks such as MMB/SEED, with language ability approaching pure text models (e.g. HellaSwag 68.4 vs 68.5).
2021
Latent Diffusion Models
[Published: 12-20] View
The paper proposes Latent Diffusion Models (LDM), which substantially reduce the computational cost of high-resolution image synthesis by training the diffusion model in the low-dimensional latent space of a pretrained autoencoder. Key innovations include: spatial compression with semantic preservation, using the autoencoder to separate perceptual compression from generative learning and avoid the high overhead of pixel-level diffusion; a cross-attention conditioning mechanism, introducing multimodal controls (such as text and layout) into image generation and supporting tasks like text-to-image; efficient high-resolution synthesis, achieving SOTA results on datasets such as ImageNet and CelebA-HQ (e.g. FID=5.11) while training more than 2.7x faster and improving inference efficiency markedly; and generality, being applicable to tasks such as inpainting, super-resolution, and layout generation, with the model open-sourced.
2017
Attention Is All You Need
[Published: 06-12] View
The paper proposes the Transformer architecture, based entirely on attention mechanisms and discarding the traditional recurrent neural network (RNN) and convolutional neural network (CNN), achieving better results, higher parallelism, and shorter training time on machine translation tasks, and setting new SOTA results on WMT 2014 English-German/English-French translation (BLEU 28.4/41.8).
2015
Deep Residual Learning for Image Recognition
[Published: 12-10] View
The paper proposes the deep residual learning framework (ResNet), which by introducing shortcut connections with identity mappings solves the degradation problem in deep neural network training (where training error actually increases as the network gets deeper), making it possible to train networks with more than 100 layers; its 152-layer residual network won the ILSVRC 2015 image classification championship with a 3.57% Top-5 error rate, and significantly improved performance on tasks such as detection and segmentation.
