From the model card
Fields below are copied from the tags and counters on the HuggingFace repository EleutherAI/gpt-neox-20b at our last fetch. They are set by the uploader, not verified by us; rows with no tag are omitted. How this page is made.
- Publisher (HF namespace)
- EleutherAI
- Pipeline tag
- text-generation
- Library
- Transformers
- Framework tags
- PyTorch
- Weight formats
- safetensors
- License tag
apache-2.0— read the license file in the repo before relying on it- Language tags
- English (en)
- Papers cited
- arXiv:2204.06745, arXiv:2101.00027, arXiv:2201.07311, arXiv:2104.09864
- Datasets declared
- EleutherAI/pile
- Downloads (HF counter at last fetch)
- 746,824
- Likes (HF counter at last fetch)
- 585
- Model card
- https://huggingface.co/EleutherAI/gpt-neox-20b
Use cases
- Open-weights LLM research and ablation studies
- Text generation and completion on English corpora
- Fine-tuning base for domain-specific text generation
- Benchmarking historical open LLM progress
- Low-level analysis of large autoregressive LM behavior
Pros
- Apache-2.0 licensed with fully open weights and training details
- Rotary position embeddings enable better extrapolation than learned positional encodings
- Trained on the Pile — well-documented open training corpus
- Historical reference point for open LLM research reproducibility
Cons
- 20B parameters require significant GPU memory (40GB+ in fp32) for inference
- No instruction tuning — generates completions, not chat responses
- Significantly outperformed by newer open models (Llama 3, Qwen) on all benchmarks
- Pile dataset includes web data with known quality and bias issues
Tags
transformerspytorchsafetensorsgpt_neoxtext-generationcausal-lmendataset:EleutherAI/pilearxiv:2204.06745arxiv:2101.00027arxiv:2201.07311arxiv:2104.09864license:apache-2.0text-generation-inferenceendpoints_compatibleregion:usdeploy:azure