Red Hat Summit Red Hat SummitSupportConsoleDevelopersStart a trialContact

Chapter 1. About Speculators

Speculators is a unified library for building, training, and storing speculative decoding algorithms for large language model (LLM) inference, including frameworks such as vLLM.

Important

Speculators is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.

For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.

Speculative decoding is an optimization technique that improves inference performance for the LLM you are trying to serve. Red Hat AI Inference Server supports Eagle 3, a speculative decoding algorithm that uses a small, single-layer draft model and a full-sized 'verifier' model, which is the LLM you are serving. The Eagle 3 speculator model auto-regressively predicts several tokens, and then the verifier model processes these tokens in parallel. As the verifier model can accept multiple tokens per forward pass, effective throughput increases. When the verifier model rejects a token, it samples a corrected token from its own distribution, ensuring the output matches what it would produce alone.

Speculative decoding provides the following advantages:

  • Latency decreases through parallel token validation.
  • Eagle 3 speculator models require minimal processing due to their small size.
  • Output quality matches what the verifier model would produce alone.
Red Hat logoGithubredditYoutubeTwitter

Learn

Try, buy, & sell

Communities

About Red Hat Documentation

We help Red Hat users innovate and achieve their goals with our products and services with content they can trust. Explore our recent updates.

Making open source more inclusive

Red Hat is committed to replacing problematic language in our code, documentation, and web properties. For more details, see the Red Hat Blog.

About Red Hat

We deliver hardened solutions that make it easier for enterprises to work across platforms and environments, from the core datacenter to the network edge.

Theme

© 2026 Red HatBack to top