Chapter 6. Improve model accuracy with inference-time scaling

Procedure

1. Install ITS Hub

   pip install its-hub[lm]

2. Verify the installation:

   python -c "from its_hub import OpenAICompatibleLanguageModel, SelfConsistency, BestOfN, LLMJudge; print('OK')"

3. Connect to your model by creating a language model instance that points to your endpoint.

   In the following example, replace the endpoint, api_key, and model_name placeholder values with your actual endpoint details. If your endpoint does not require authentication, as is common with a local vLLM, set api_key="NO_API_KEY".

   from its_hub import OpenAICompatibleLanguageModel
   
   lm = OpenAICompatibleLanguageModel(
       endpoint="https://<your-endpoint>/v1",
       api_key="<your-api-key>",
       model_name="<your-model-name>",
   )

4. Use the Self-Consistency algorithm to generate multiple responses to the same prompt and select the most common answer. It is similar to asking the same question several times and going with the majority answer.

   • Basic usage

     In the following example, the budget parameter controls how many responses are generated. A budget of 5 means the model generates 5 responses and votes on the answer.

     from its_hub import SelfConsistency
     
     sc = SelfConsistency()
     result = sc.infer(lm, "What is the capital of France?", budget=5)
     print(result)

   • Extract a specific part of the response for voting

     By default, Self-Consistency compares the entire response text. Optionally, you can use a projection function to extract a specific part of the response, such as the final answer, as shown in the following example:

     def extract_last_line(response):
         return response.strip().split("\n")[-1]
     
     sc = SelfConsistency(consistency_space_projection_func=extract_last_line)
     result = sc.infer(lm, "What is 2 + 2?", budget=5)
     print(result)

   • Get the full result

     By default, infer() returns only the winning response as a dictionary. To see all responses and vote counts, set return\_response\_only=False:

     result = sc.infer(lm, "What is the capital of France?", budget=5, return_response_only=False)
     
     print(f"Selected response index: {result.selected_index}")
     print(f"Vote counts: {result.response_counts}")
     print(f"Total responses: {len(result.responses)}")
     print(f"Winning response: {result.the_one}")

5. Use the Best-of-N algorithm to generate multiple candidate responses, score each one, and return the highest-scoring response. ITS Hub includes an inline implementation of an outcome reward model (LLMJudge).

   • Basic usage. In the following example, a budget of 4 means that the model generates 4 candidate responses. The LLM judge scores each one and returns the best.

     from its_hub import BestOfN, LLMJudge
     
     judge = LLMJudge(lm=lm)
     bon = BestOfN(orm=judge)
     
     result = bon.infer(
         lm,
         "Explain how a CPU works to someone with no technical background",
         budget=4,
     )
     print(result)

   • Get the full result:

     result = bon.infer(
         lm,
         "Explain how a CPU works to someone with no technical background",
         budget=4,
         return_response_only=False,
     )
     
     print(f"Scores: {result.scores}")
     print(f"Selected index: {result.selected_index}")
     print(f"Best response: {result.the_one}")

6. Clean up resources:

   • To release HTTP connections, run the following command to close the language model instance:

     import asyncio
     asyncio.run(lm.close())

   • To handle cleanup automatically, use an async context manager:

     async with OpenAICompatibleLanguageModel(
         endpoint="https://<your-endpoint>/v1",
         api_key="<your-api-key>",
         model_name="<your-model-name>",
     ) as lm:
         result = await sc.ainfer(lm, "What is the capital of France?", budget=5)