Red Hat Summit 红帽全球峰会支持控制台（Console）开发人员开始试用联系人

5.2. Chatting with the model

Once you serve your model, you can now chat with the model.

重要

The model you are chatting with must match the model you are serving. With the default config.yaml file, the granite-7b-redhat-lab model is the default for serving and chatting.

Prerequisites

  • You installed RHEL AI with the bootable container image.
  • You initialized InstructLab.
  • You downloaded your preferred Granite LLMs.
  • You are serving a model.
  • You have root user access on your machine.

Procedure

  1. Since you are serving the model in one terminal window, you must open another terminal to chat with the model.

  2. To chat with the default model, run the following command: 

    $ ilab model chat

  3. To chat with a specific model run the following command: 

    $ ilab model chat --model <model-path>

    Example command

    $ ilab model chat --model ~/.cache/instructlab/models/granite-8b-code-instruct

Example output of the chatbot

$ ilab model chat
╭────────────────────────────────────────────────────────────────────────────────────────────────────────────────── system ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ Welcome to InstructLab Chat w/ GRANITE-8B-CODE-INSTRUCT (type /h for help)                                                                                                                                                                    │
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
>>>                                                                                                                                                                                                                        [S][default]

+ Type exit to leave the chatbot.

By default, the ilab CLI does not use authentication. If you want to expose your server to the internet, you can create a API key that connects to your server with the following procedures.

Prerequisites

  • You installed the Red Hat Enterprise Linux AI image on bare metal.
  • You initialized InstructLab
  • You downloaded your preferred Granite LLMs.
  • You have root user access on your machine.

Procedure

  1. Create a API key that is held in $VLLM_API_KEY parameter by running the following command: 

    $ export VLLM_API_KEY=$(python -c 'import secrets; print(secrets.token_urlsafe())')

  2. You can view the API key by running the following command: 

    $ echo $VLLM_API_KEY

  3. Update the config.yaml by running the following command: 

    $ ilab config edit

  4. Add the following parameters to the vllm_args section of your config.yaml file. 

    serve:
    vllm:
        vllm_args:
        - --api-key
        - <api-key-string>

    where

    <api-key-string>
    Specify your API key string.

  5. You can verify that the server is using API key authentication by running the following command: 

    $ ilab model chat

    Then, seeing the following error that shows an unauthorized user. 

    openai.AuthenticationError: Error code: 401 - {'error': 'Unauthorized'}

  6. Verify that your API key is working by running the following command: 

    $ ilab model chat -m granite-7b-redhat-lab --endpoint-url https://inference.rhelai.com/v1 --api-key $VLLM_API_KEY

    Example output

    $ ilab model chat
╭────────────────────────────────────────────────────────────────────────────────────────────────────────────────── system ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ Welcome to InstructLab Chat w/ GRANITE-7B-LAB (type /h for help)                                                                                                                                                                    │
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
>>>                                                                                                                                                                                                                        [S][default]

Red Hat logoGithubredditYoutubeTwitter

学习

尝试、购买和销售

社区

關於紅帽

我们提供强化的解决方案，使企业能够更轻松地跨平台和环境（从核心数据中心到网络边缘）工作。

让开源更具包容性

红帽致力于替换我们的代码、文档和 Web 属性中存在问题的语言。欲了解更多详情，请参阅红帽博客.

关于红帽文档

通过我们的产品和服务，以及可以信赖的内容，帮助红帽用户创新并实现他们的目标。 了解我们当前的更新.

Legal Notice

Theme

© 2026 Red Hat返回顶部