Red Hat Summit이 콘텐츠는 선택한 언어로 제공되지 않습니다.
Chapter 4. Serving and chatting with your new model
You must deploy the model to your machine by serving the model. This deploys the model and makes the model available for interacting and chatting.
4.1. Serving the new model
To interact with your new model, you must activate the model in a machine through serving. The ilab model serve command starts a vLLM server that allows you to chat with the model.
Prerequisites
- You installed RHEL AI with the bootable container image.
- You initialized InstructLab.
- You customized your taxonomy tree, ran synthetic data generation, trained, and evaluated your new model.
- You need root user access on your machine.
Procedure
You can serve the model by running the following command:
$ ilab model serve --model-path <path-to-best-performed-checkpoint>
where:
- <path-to-best-performed-checkpoint>
Specify the full path to the checkpoint you built after training. Your new model is the best performed checkpoint with its file path displayed after training.
Example command:
$ ilab model serve --model-path ~/.local/share/instructlab/phased/phase2/checkpoints/hf_format/samples_1945/
ImportantEnsure you have a slash
/at the end of your model path.Example output of the
ilab model servecommand$ ilab model serve --model-path ~/.local/share/instructlab/phased/phase2/checkpoints/hf_format/<checkpoint> INFO 2024-03-02 02:21:11,352 lab.py:201 Using model /home/example-user/.local/share/instructlab/checkpoints/hf_format/checkpoint_1945 with -1 gpu-layers and 4096 max context size. Starting server process After application startup complete see http://127.0.0.1:8000/docs for API. Press CTRL+C to shut down the server.
4.2. Chatting with the new model
You can chat with your model that has been trained with your data.
Prerequisites
- You installed RHEL AI with the bootable container image.
- You initialized InstructLab.
- You customized your taxonomy tree, ran synthetic data generated, trained and evaluated your new model.
- You served your checkpoint model.
- You need root user access on your machine.
Procedure
- Since you are serving the model in one terminal window, you must open a new terminal window to chat with the model.
To chat with your new model, run the following command:
$ ilab model chat --model <path-to-best-performed-checkpoint-file>
where:
- <path-to-best-performed-checkpoint-file>
Specify the new model checkpoint file you built after training. Your new model is the best performed checkpoint with its file path displayed after training.
Example command:
$ ilab model chat --model ~/.local/share/instructlab/phased/phase2/checkpoints/hf_format/samples_1945
Example output of the InstructLab chatbot
$ ilab model chat ╭────────────────────────────────────────────────────────────────────────────────────────────────────────────────── system ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ Welcome to InstructLab Chat w/ CHECKPOINT_1945 (type /h for help) │ ╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯ >>> [S][default]
Type
exitto leave the chatbot.
