TensorFloat-32
Numbering format in Nvidia hardware
TensorFloat-32 (TF32) is a numeric floating point format designed for Tensor Core running on certain Nvidia GPUs. It was first implemented in the Ampere architecture. TensorFloat-32 combines the 8-bit exponent size of IEEE single precision with the 10-bit mantissa size of half precision for a total of 19 bits per number. It is comparable to the bfloat16 format, which uses a 7-bit mantissa.
01Format
The binary format is:
- 1 sign bit
- 8 exponent bits
- 10 significand bits (also called mantissa, or precision bits)
The 19-significant-bit format fits within a double word (32 bits), and while it lacks precision compared with a normal 32-bit IEEE 754 floating-point number, it provides much faster computation, up to 8 times on a A100 (compared to a V100 using FP32).
Stored in the same space as FP32, it is not a distinct storage format, but a specification for reduced-precision FP32 multiply-accumulate operations. FP32 inputs are rounded to TF32, multiplied to produce a 21-bit product (including the implicit msbit, this is an 11×11→22-bit multiply), and summed into a standard FP32 accumulator.
Sources and credits
This article is adapted from the Wikipedia article “TensorFloat-32”, written by its contributors and licensed under CC BY-SA 4.0. Fathomly has changed the layout, removed citation markers, navigation and maintenance notices, and adjusted punctuation. This adapted version is shared under the same license. For references, see the original article.
Fathomly is not affiliated with or endorsed by the Wikimedia Foundation. Spotted a problem? Tell us.