From the model card
Fields below are copied from the tags and counters on the HuggingFace repository distilbert/distilgpt2 at our last fetch. They are set by the uploader, not verified by us; rows with no tag are omitted. How this page is made.
- Publisher (HF namespace)
- distilbert
- Pipeline tag
- text-generation
- Library
- Transformers
- Framework tags
- PyTorch, TensorFlow, JAX, Rust (candle)
- Weight formats
- TFLite, Core ML, safetensors
- License tag
apache-2.0— read the license file in the repo before relying on it- Language tags
- English (en)
- Papers cited
- arXiv:1910.01108, arXiv:2201.08542, arXiv:2203.12574, arXiv:1910.09700, arXiv:1503.02531
- Datasets declared
- openwebtext
- Downloads (HF counter at last fetch)
- 1,940,717
- Likes (HF counter at last fetch)
- 639
- Model card
- https://huggingface.co/distilbert/distilgpt2
Use cases
- Text autocomplete in lightweight applications
- Baseline for evaluating distillation techniques
- On-device text generation where GPT-2 is too slow
- Teaching and experimenting with autoregressive generation
Pros
- Faster inference than GPT-2 small with minor quality trade-off
- Apache-2.0 licensed
- Maintained by HuggingFace with stable API support
- Well-documented distillation methodology in published research
Cons
- GPT-2 quality is low by 2024 standards — both GPT-2 and DistilGPT2 produce incoherent long-form text
- English-only
- No instruction tuning — free-form next-token prediction
- Largely superseded by TinyLlama and SmolLM at comparable sizes
Tags
transformerspytorchtfjaxtfliterustcoremlsafetensorsgpt2text-generationexbertendataset:openwebtextarxiv:1910.01108arxiv:2201.08542arxiv:2203.12574arxiv:1910.09700arxiv:1503.02531license:apache-2.0model-index