Red Hat Summit Red Hat SummitSupportConsoleDéveloppeursCommencer un essaiContact

Ce contenu n'est pas disponible dans la langue sélectionnée.

Chapter 1. About Speculators

Speculators is a unified library for building, training, and storing speculative decoding algorithms for large language model (LLM) inference, including frameworks such as vLLM.

Important

Speculators is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.

For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.

Speculative decoding is an optimization technique that improves inference performance for the LLM you are trying to serve. Red Hat AI Inference Server supports Eagle 3, a speculative decoding algorithm that uses a small, single-layer draft model and a full-sized 'verifier' model, which is the LLM you are serving. The Eagle 3 speculator model auto-regressively predicts several tokens, and then the verifier model processes these tokens in parallel. As the verifier model can accept multiple tokens per forward pass, effective throughput increases. When the verifier model rejects a token, it samples a corrected token from its own distribution, ensuring the output matches what it would produce alone.

Speculative decoding provides the following advantages:

  • Latency decreases through parallel token validation.
  • Eagle 3 speculator models require minimal processing due to their small size.
  • Output quality matches what the verifier model would produce alone.
Red Hat logoGithubredditYoutubeTwitter

Apprendre

Essayez, achetez et vendez

Communautés

À propos de Red Hat

Nous proposons des solutions renforcées qui facilitent le travail des entreprises sur plusieurs plates-formes et environnements, du centre de données central à la périphérie du réseau.

Rendre l’open source plus inclusif

Red Hat s'engage à remplacer le langage problématique dans notre code, notre documentation et nos propriétés Web. Pour plus de détails, consultez le Blog Red Hat.

À propos de la documentation Red Hat

Nous aidons les utilisateurs de Red Hat à innover et à atteindre leurs objectifs grâce à nos produits et services avec un contenu auquel ils peuvent faire confiance. Découvrez nos récentes mises à jour.

Legal Notice

Theme

© 2026 Red HatRetour au début