Chapter 5. Serving and chatting with the models

Procedure.

1. Create a directory for your systemd user service by running the following command:

   $ mkdir -p $HOME/.config/systemd/user

2. Create your systemd service file with the following example configurations:

   $ cat << EOF > $HOME/.config/systemd/user/ilab-serve.service
   [Unit]
   Description=ilab model serve service
   
   [Install]
   WantedBy=multi-user.target default.target 

   1

   
   
   [Service]
   ExecStart=ilab model serve --model-family granite
   Restart=always
   EOF

   1

   Specifies to start by default on boot.

3. Reload the systemd manager configuration by running the following command:

   $ systemctl --user daemon-reload

4. Start the ilab model serve systemd service by running the following command:

   $ systemctl --user start ilab-serve.service

5. You can check that the service is running with the following command:

   $ systemctl --user status ilab-serve.service

6. You can check the service logs by running the following command:

   $ journalctl --user-unit ilab-serve.service

7. To allow the service to start on boot, run the following command:

   $ sudo loginctl enable-linger

8. Optional: There are a few optional commands you can run for maintaining your systemd service.

   • You can stop the ilab-serve system service by running the following command:

     $ systemctl --user stop ilab-serve.service

   • You can prevent the service from starting on boot by removing the "WantedBy=multi-user.target default.target" from the $HOME/.config/systemd/user/ilab-serve.service file.

Procedure

1. Create a directory for your certificate file and key by running the following command:

   $ mkdir -p `pwd`/nginx/ssl/

2. Create an OpenSSL configuration file with the proper configurations by running the following command:

   $ cat > openssl.cnf <<EOL
   [ req ]
   default_bits = 2048
   distinguished_name = <req-distinguished-name> 

   1

   
   x509_extensions = v3_req
   prompt = no
   
   [ req_distinguished_name ]
   C  = US
   ST = California
   L  = San Francisco
   O  = My Company
   OU = My Division
   CN = rhelai.redhat.com
   
   [ v3_req ]
   subjectAltName = <alt-names> 

   2

   
   basicConstraints = critical, CA:true
   subjectKeyIdentifier = hash
   authorityKeyIdentifier = keyid:always,issuer
   
   [ alt_names ]
   DNS.1 = rhelai.redhat.com 

   3

   
   DNS.2 = www.rhelai.redhat.com 

   4

   1

   Specify the distinguished name for your requirements.

   2

   Specify the alternate name for your requirements.

   3 4

   Specify the server common name for RHEL AI. In the example, the server name is rhelai.redhat.com.

3. Generate a self signed certificate with a Subject Alternative Name (SAN) enabled with the following commands:

   $ openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout `pwd`/nginx/ssl/rhelai.redhat.com.key -out `pwd`/nginx/ssl/rhelai.redhat.com.crt -config openssl.cnf

   $ openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout

4. Create the Nginx Configuration file and add it to the `pwd/nginx/conf.d` by running the following command:

   mkdir -p `pwd`/nginx/conf.d
   
   echo 'server {
       listen 8443 ssl;
       server_name <rhelai.redhat.com> 

   1

   
   
       ssl_certificate /etc/nginx/ssl/rhelai.redhat.com.crt;
       ssl_certificate_key /etc/nginx/ssl/rhelai.redhat.com.key;
   
       location / {
           proxy_pass http://127.0.0.1:8000;
           proxy_set_header Host $host;
           proxy_set_header X-Real-IP $remote_addr;
           proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
           proxy_set_header X-Forwarded-Proto $scheme;
       }
   }
   ' > `pwd`/nginx/conf.d/rhelai.redhat.com.conf

   1

   Specify the name of your server. In the example, the server name is rhelai.redhat.com

5. Run the Nginx container with the new configurations by running the following command:

   $ podman run --net host -v `pwd`/nginx/conf.d:/etc/nginx/conf.d:ro,Z -v `pwd`/nginx/ssl:/etc/nginx/ssl:ro,Z nginx

   If you want to use port 443, you must run the podman run command as a root user..

6. You can now connect to a serving ilab machine using a secure endpoint URL. Example command:

   $ ilab model chat -m /instructlab/instructlab/granite-7b-redhat-lab --endpoint-url

7. You can also connect to the serving RHEL AI machine with the following command:

   $ curl --location 'https://rhelai.redhat.com:8443/v1' \
   --header 'Content-Type: application/json' \
   --header 'Authorization: Bearer <api-key>' \
   --data '{
     "model": "/var/home/cloud-user/.cache/instructlab/models/granite-7b-redhat-lab",
     "messages": [
         {
             "role": "system",
             "content": "You are a helpful assistant."
         },
         {
             "role": "user",
             "content": "Hello!"
         }
     ]
   }' | jq .

   where

   <api-key>

   Specify your API key. You can create your own API key by following the procedure in "Creating an API key for chatting with a model".

8. Optional: You can also get the server certificate and append it to the Certifi CA Bundle

   1. Get the server certificate by running the following command:

      $ openssl s_client -connect rhelai.redhat.com:8443 </dev/null 2>/dev/null | openssl x509 -outform PEM > server.crt

   2. Copy the certificate to you system’s trusted CA storage directory and update the CA trust store with the following commands:

      $ sudo cp server.crt /etc/pki/ca-trust/source/anchors/

      $ sudo update-ca-trust

   3. You can append your certificate to the Certifi CA bundle by running the following command:

      $ cat server.crt >> $(python -m certifi)

   4. You can now run ilab model chat with a self-signed certificate. Example command:

      $ ilab model chat -m /instructlab/instructlab/granite-7b-redhat-lab --endpoint-url https://rhelai.redhat.com:8443/v1

Procedure

1. Create a API key that is held in $VLLM_API_KEY parameter by running the following command:

   $ export VLLM_API_KEY=$(python -c 'import secrets; print(secrets.token_urlsafe())')

2. You can view the API key by running the following command:

   $ echo $VLLM_API_KEY

3. Update the config.yaml by running the following command:

   $ ilab config edit

4. Add the following parameters to the vllm_args section of your config.yaml file.

   serve:
       vllm:
           vllm_args:
           - --api-key
           - <api-key-string>

   where

   <api-key-string>

   Specify your API key string.

5. You can verify that the server is using API key authentication by running the following command:

   $ ilab model chat

   Then, seeing the following error that shows an unauthorized user.

   openai.AuthenticationError: Error code: 401 - {'error': 'Unauthorized'}

6. Verify that your API key is working by running the following command:

   $ ilab model chat -m granite-7b-redhat-lab --endpoint-url https://inference.rhelai.com/v1 --api-key $VLLM_API_KEY

   Example output

   $ ilab model chat
   ╭────────────────────────────────────────────────────────────────────────────────────────────────────────────────── system ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
   │ Welcome to InstructLab Chat w/ GRANITE-7B-LAB (type /h for help)                                                                                                                                                                    │
   ╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
   >>>                                                                                                                                                                                                                        [S][default]