Small language model
Type of artificial intelligence model
Small language models (SLM) or compact language models are artificial intelligence language models designed for human natural language processing including language and text generation. Small language models typically have less than forty billion parameters. This make them feasible to train or host entirely on consumer electronics such as personal computers, laptops, or smart devices.
In contrast, a large language model typically contains hundreds of billions of training parameters, with some models exceeding a trillion parameters. This enables the model to encode vast amounts of information, improving the generalizability and accuracy of its outputs. However, training and hosting such models demands enormous computational resources, rendering it infeasible for an individual to do so using a single computer and graphics processing unit.
Most contemporary small language models use the same architecture as a large language model, but with a smaller parameter count and sometimes lower arithmetic precision. Similar to LLMs, SLMs are optimized via knowledge distillation, pruning, and quantization.
01Models
Notable families with some models under ten billion parameters include:
- Qwen by Alibaba Cloud
- Gemma by Google DeepMind
- Granite by IBM
- Nemotron by Nvidia
- Muse Glimmer by Meta Superintelligence Labs
- Mistral Small by Mistral AI
- SmolLM by Hugging Face family of SMLs, including SmolLM2 models with 135 million, 360 million, and 1.7 billion parameters.
02Language model with small pre-training dataset
Traditional AI language systems need enormous computers and vast amounts of data.
Pre-training matters, even tiny models show significant performance improvements when pre-trained performance increases with larger pre-training datasets. Classification accuracy improves when pre-training and test datasets share similar tokens.
Shallow architectures can replicate deep model performance through collaborative learning.
Sources and credits
This article is adapted from the Wikipedia article “Small language model”, written by its contributors and licensed under CC BY-SA 4.0. Fathomly has changed the layout, removed citation markers, navigation and maintenance notices, and adjusted punctuation. This adapted version is shared under the same license. For references, see the original article.
Fathomly is not affiliated with or endorsed by the Wikimedia Foundation. Spotted a problem? Tell us.
