The Computer Language Benchmarks Game
Free software project
The Computer Language Benchmarks Game (formerly called The Great Computer Language Shootout) is a free software project for comparing how a given subset of simple algorithms can be implemented in various popular programming languages.
The project consists of:
- A set of very simple algorithmic problems (thirteen in total)
- Various implementations to the above problems in various programming languages
- A set of unit tests to verify that the submitted implementations solve the problem statement
- A framework for running and timing the implementations
- A website to facilitate the interactive comparison of the results
01Supported languages
02Metrics
The following aspects of each given implementation are measured:
- overall user runtime
- peak memory allocation
- gzipped size of the solution's source code
- sum of total CPU time over all threads
- individual CPU utilization
It is common to see multiple solutions in the same programming language for the same problem. This highlights that within the constraints of a given language, a solution can be given which is either of high abstraction, is memory efficient, is fast, or can be parallelized better.
03Benchmark programs
It was a design choice from the start to only include very simple toy problems, each providing a different kind of programming challenge. This provides users of the Benchmark Game the opportunity to scrutinize the various implementations.
04History
The project was known as The Great Computer Language Shootout until 2007.
A port for Windows was maintained separately between 2002 and 2003.
The sources have been archived on GitLab.
There are also older forks on GitHub.
The project is continuously evolving. The list of supported programming languages is updated approximately once per year, following market trends. Users can also submit improved solutions to any of the problems or suggest testing methodology refinement.
05Caveats
The developers themselves highlight the fact that those doing research should exercise caution when using such microbenchmarks:
[...] the JavaScript benchmarks are fleetingly small, and behave in ways that are significantly different than the real applications. We have documented numerous differences in behavior, and we conclude from these measured differences that results based on the benchmarks may mislead JavaScript engine implementers. Furthermore, we observe interesting behaviors in real JavaScript applications that the benchmarks fail to exhibit, suggesting that previously unexplored optimization strategies may be productive in practice.
06Impact
The benchmark results have uncovered various compiler issues. Sometimes a given compiler failed to process unusual, but otherwise grammatically valid constructs. At other times, runtime performance was shown to be below expectations, which prompted compiler developers to revise their optimization capabilities.
Various research articles have been based on the benchmarks, its results and its methodology.
Sources and credits
This article is adapted from the Wikipedia article “The Computer Language Benchmarks Game”, written by its contributors and licensed under CC BY-SA 4.0. Fathomly has changed the layout, removed citation markers, navigation and maintenance notices, and adjusted punctuation. This adapted version is shared under the same license. For references, see the original article.
Fathomly is not affiliated with or endorsed by the Wikimedia Foundation. Spotted a problem? Tell us.
