Chapter 5. About tensor types and model variants

Each tensor type is a compression variant of the same base model, which means that the same model can have multiple variants with different trade-offs in performance. All models include tensor type and size on their details page in the model catalog.

A validated model can have multiple variants, each with a different tensor type. For example, the Llama-3-8B model can have the following variants:

Each model variant has its own performance artifacts: recommended hardware, throughput, and latency can vary significantly between them.

Pareto-optimal filtering takes model variants into account when recommending configurations in the catalog. For example, an INT4 variant can achieve the same requests per second with less hardware than an FP16 variant.