Chapter 1. Managing model-serving runtimes

Procedure

1. From the OpenShift AI dashboard, click Settings Model resources and operations Serving runtimes.

   The Serving runtimes page opens and shows the model-serving runtimes that are already installed and enabled.

2. To add a custom runtime, choose one of the following options:

   • To start with an existing runtime (for example, vLLM NVIDIA GPU ServingRuntime for KServe), click the action menu (⋮) next to the existing runtime and then click Duplicate.
   • To add a new custom runtime, click Add serving runtime.
3. In the Select the model serving platforms this runtime supports list, select Single-model serving platform.
4. In the Select the API protocol this runtime supports list, select REST or gRPC.

5. Optional: If you started a new runtime (rather than duplicating an existing one), add your code by choosing one of the following options:

   • Upload a YAML file

     1. Click Upload files.

     2. In the file browser, select a YAML file on your computer.

        The embedded YAML editor opens and shows the contents of the file that you uploaded.

   • Enter YAML code directly in the editor

     1. Click Start from scratch.
     2. Enter or paste YAML code directly in the embedded editor.
   Note

   In many cases, creating a custom runtime will require adding new or custom parameters to the env section of the ServingRuntime specification.

6. Click Add.

   The Serving runtimes page opens and shows the updated list of runtimes that are installed. Observe that the custom runtime that you added is automatically enabled. The API protocol that you specified when creating the runtime is shown.

7. Optional: To edit your custom runtime, click the action menu (⋮) and select Edit.