Chapter 3. Deploying a RAG stack in a project

1. In the OpenShift AI dashboard, navigate to the project details page and click the Deployments tab.
2. In the Model serving platform tile, click Select model.

3. Click the Deploy model button.

   The Deploy model dialog opens.

4. Configure the deployment properties for your model:

   1. In the Model deployment name field, enter a unique name for your deployment.
   2. In the Serving runtime field, select vLLM NVIDIA GPU serving runtime for KServe from the drop-down list.
   3. In the Deployment mode field, select KServe RawDeployment from the drop-down list.
   4. Set Number of model server replicas to deploy to 1.

   5. In the Model server size field, select Custom from the drop-down list.

      • Set CPUs requested to 1 core.
      • Set Memory requested to 10 GiB.
      • Set CPU limit to 2 core.
      • Set Memory limit to 14 GiB.
      • Set Accelerator to NVIDIA GPUs.
      • Set Accelerator count to 1.
   6. From the Connection type, select a relevant data connection from the drop-down list.

5. In the Additional serving runtime arguments field, specify the following recommended arguments:

   --dtype=half
   --max-model-len=20000
   --gpu-memory-utilization=0.95
   --enable-chunked-prefill
   --enable-auto-tool-choice
   --tool-call-parser=llama3_json
   --chat-template=/app/data/template/tool_chat_template_llama3.2_json.jinja

   1. Click Deploy.

      Note

      Model deployment can take several minutes, especially for the first model that is deployed on the cluster. Initial deployment may take more than 10 minutes while the relevant images download.

Verification

1. Verify that the kserve-controller-manager and odh-model-controller pods are running:

   1. Open a new terminal window.
   2. Log in to your OpenShift cluster from the CLI:
   3. In the upper-right corner of the OpenShift web console, click your user name and select Copy login command.
   4. After you have logged in, click Display token.

   5. Copy the Log in with this token command and paste it in the OpenShift CLI (oc).

      $ oc login --token=<token> --server=<openshift_cluster_url>

   6. Enter the following command to verify that the kserve-controller-manager and odh-model-controller pods are running:

      $ oc get pods -n redhat-ods-applications | grep -E 'kserve-controller-manager|odh-model-controller'

   7. Confirm that you see output similar to the following example:

      kserve-controller-manager-7c865c9c9f-xyz12   1/1     Running   0          4m21s
      odh-model-controller-7b7d5fd9cc-wxy34        1/1     Running   0          3m55s

   8. If you do not see either of the kserve-controller-manager and odh-model-controller pods, there could be a problem with your deployment. In addition, if the pods appear in the list, but their Status is not set to Running, check the pod logs for errors:

      $ oc logs <pod-name> -n redhat-ods-applications

   9. Check the status of the inference service:

      $ oc get inferenceservice -n llamastack
      $ oc get pods -n <project name> | grep llama

      • The deployment automatically creates the following resources:

        • A ServingRuntime resource.
        • An InferenceService resource, a Deployment, a pod, and a service pointing to the pod.

      • Verify that the server is running. For example:

        $ oc logs llama-32-3b-instruct-predictor-77f6574f76-8nl4r  -n <project name>

        Check for output similar to the following example log:

        INFO     2025-05-15 11:23:52,750 __main__:498 server: Listening on ['::', '0.0.0.0']:8321
        INFO:     Started server process [1]
        INFO:     Waiting for application startup.
        INFO     2025-05-15 11:23:52,765 __main__:151 server: Starting up
        INFO:     Application startup complete.
        INFO:     Uvicorn running on http://['::', '0.0.0.0']:8321 (Press CTRL+C to quit)

      • The deployed model displays in the Deployments tab on the project details page for the project it was deployed under.

2. If you see a ConvertTritonGPUToLLVM error in the pod logs when querying the /v1/chat/completions API, and the vLLM server restarts or returns a 500 Internal Server error, apply the following workaround:

   Before deploying the model, remove the --enable-chunked-prefill argument from the Additional serving runtime arguments field in the deployment dialog.

   The error is displayed similar to the following:

   /opt/vllm/lib64/python3.12/site-packages/vllm/attention/ops/prefix_prefill.py:36:0: error: Failures have been detected while processing an MLIR pass pipeline
   /opt/vllm/lib64/python3.12/site-packages/vllm/attention/ops/prefix_prefill.py:36:0: note: Pipeline failed while executing [`ConvertTritonGPUToLLVM` on 'builtin.module' operation]: reproducer generated at `std::errs, please share the reproducer above with Triton project.`
   INFO:     10.129.2.8:0 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error

Procedure

1. Open a new terminal window.

   1. Log in to your OpenShift cluster from the CLI:
   2. In the upper-right corner of the OpenShift web console, click your user name and select Copy login command.
   3. After you have logged in, click Display token.

   4. Copy the Log in with this token command and paste it in the OpenShift CLI (oc).

      $ oc login --token=<token> --server=<openshift_cluster_url>

2. If you enabled Require token authentication during model deployment, retrieve your token:

   $ export MODEL_TOKEN=$(oc get secret default-name-llama-32-3b-instruct-sa -n <project name> --template={{ .data.token }} | base64 -d)

3. Obtain your model endpoint URL:

   • If you enabled Make deployed models available through an external route during model deployment, click Endpoint details on the Deployments page in the OpenShift AI dashboard to obtain your model endpoint URL.

   • In addition, if you did not enable Require token authentication during model deployment, you can also enter the following command to retrieve the endpoint URL:

     $ export MODEL_ENDPOINT="https://$(oc get route llama-32-3b-instruct -n <project name> --template={{ .spec.host }})"

4. Test the endpoint with a sample chat completion request:

   • If you did not enable Require token authentication during model deployment, enter a chat completion request. For example:

     $ curl -X POST $MODEL_ENDPOINT/v1/chat/completions \
      -H "Content-Type: application/json" \
      -d '{
      "model": "llama-32-3b-instruct",
      "messages": [
        {
          "role": "user",
          "content": "Hello"
        }
      ]
     }'

   • If you enabled Require token authentication during model deployment, include a token in your request. For example:

     curl -s -k $MODEL_ENDPOINT/v1/chat/completions \
     --header "Authorization: Bearer $MODEL_TOKEN" \
     --header 'Content-Type: application/json' \
     -d '{
       "model": "llama-32-3b-instruct",
       "messages": [
         {
           "role": "user",
           "content": "can you tell me a funny joke?"
         }
       ]
     }' | jq .

     Note

     The -k flag disables SSL verification and should only be used in test environments or with self-signed certificates.

Procedure

1. In the OpenShift console, click the Quick Create ( ) icon and then click the Import YAML option.
2. Verify that your project is the selected project.

3. In the Import YAML editor, paste the following manifest and click Create:

   apiVersion: v1
   kind: Secret
   metadata:
     name: milvus-secret
   type: Opaque
   stringData:
     root-password: "MyStr0ngP@ssw0rd"
   ---
   kind: PersistentVolumeClaim
   apiVersion: v1
   metadata:
     name: milvus-pvc
   spec:
     accessModes:
       - ReadWriteOnce
     resources:
       requests:
         storage: 20Gi
     volumeMode: Filesystem
   ---
   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: etcd-deployment
     labels:
       app: etcd
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: etcd
     strategy:
       type: Recreate
     template:
       metadata:
         labels:
           app: etcd
       spec:
         containers:
           - name: etcd
             image: quay.io/coreos/etcd:v3.5.5
             command:
               - etcd
               - --advertise-client-urls=http://127.0.0.1:2379
               - --listen-client-urls=http://0.0.0.0:2379
               - --data-dir=/etcd
             ports:
               - containerPort: 2379
             volumeMounts:
               - name: etcd-data
                 mountPath: /etcd
             env:
               - name: ETCD_AUTO_COMPACTION_MODE
                 value: revision
               - name: ETCD_AUTO_COMPACTION_RETENTION
                 value: "1000"
               - name: ETCD_QUOTA_BACKEND_BYTES
                 value: "4294967296"
               - name: ETCD_SNAPSHOT_COUNT
                 value: "50000"
         volumes:
           - name: etcd-data
             emptyDir: {}
         restartPolicy: Always
   ---
   apiVersion: v1
   kind: Service
   metadata:
     name: etcd-service
   spec:
     ports:
       - port: 2379
         targetPort: 2379
     selector:
       app: etcd
   ---
   apiVersion: apps/v1
   kind: Deployment
   metadata:
     labels:
       app: milvus-standalone
     name: milvus-standalone
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: milvus-standalone
     strategy:
       type: Recreate
     template:
       metadata:
         labels:
           app: milvus-standalone
       spec:
         containers:
           - name: milvus-standalone
             image: milvusdb/milvus:v2.6.0
             args: ["milvus", "run", "standalone"]
             env:
               - name: DEPLOY_MODE
                 value: standalone
               - name: ETCD_ENDPOINTS
                 value: etcd-service:2379
               - name: COMMON_STORAGETYPE
                 value: local
               - name: MILVUS_ROOT_PASSWORD
                 valueFrom:
                   secretKeyRef:
                     name: milvus-secret
                     key: root-password
             livenessProbe:
               exec:
                 command: ["curl", "-f", "http://localhost:9091/healthz"]
               initialDelaySeconds: 90
               periodSeconds: 30
               timeoutSeconds: 20
               failureThreshold: 5
             ports:
               - containerPort: 19530
                 protocol: TCP
               - containerPort: 9091
                 protocol: TCP
             volumeMounts:
               - name: milvus-data
                 mountPath: /var/lib/milvus
         restartPolicy: Always
         volumes:
           - name: milvus-data
             persistentVolumeClaim:
               claimName: milvus-pvc
   ---
   apiVersion: v1
   kind: Service
   metadata:
     name: milvus-service
   spec:
     selector:
       app: milvus-standalone
     ports:
       - name: grpc
         port: 19530
         targetPort: 19530
       - name: http
         port: 9091
         targetPort: 9091

   Note
   • Use the gRPC port (19530) for the MILVUS_ENDPOINT setting in Llama Stack.
   • The HTTP port (9091) is reserved for health checks.
   • If you deploy Milvus in a different namespace, use the fully qualified service name in your Llama Stack configuration. For example: http://milvus-service.<namespace>.svc.cluster.local:19530

Verification

1. In the OpenShift web console, click Workloads Deployments.
2. Verify that both etcd-deployment and milvus-standalone show a status of 1 of 1 pods available.
3. Click Pods in the navigation panel and confirm that pods for both deployments are Running.
4. Click the milvus-standalone pod name, then select the Logs tab.

5. Verify that Milvus reports a healthy startup with output similar to:

   Milvus Standalone is ready to serve ...
   Listening on 0.0.0.0:19530 (gRPC)

6. Click Networking Services and confirm that the milvus-service and etcd-service resources exist and are exposed on ports 19530 and 2379, respectively.

7. (Optional) Click Pods milvus-standalone Terminal and run the following health check:

   curl http://localhost:9091/healthz

   A response of {"status": "healthy"} confirms that Milvus is running correctly.

Procedure

1. Open a new terminal window and log in to your OpenShift cluster from the CLI:

   In the upper-right corner of the OpenShift web console, click your user name and select Copy login command. After you have logged in, click Display token. Copy the Log in with this token command and paste it in the OpenShift CLI (oc).

   $ oc login --token=<token> --server=<openshift_cluster_url>

2. Create a secret that contains the inference model environment variables:

   export INFERENCE_MODEL="llama-3-2-3b"
   export VLLM_URL="https://llama-32-3b-instruct-predictor:8443/v1"
   export VLLM_TLS_VERIFY="false"   # Use "true" in production
   export VLLM_API_TOKEN="<token identifier>"
   
   oc create secret generic llama-stack-inference-model-secret \
     --from-literal=INFERENCE_MODEL="$INFERENCE_MODEL" \
     --from-literal=VLLM_URL="$VLLM_URL" \
     --from-literal=VLLM_TLS_VERIFY="$VLLM_TLS_VERIFY" \
     --from-literal=VLLM_API_TOKEN="$VLLM_API_TOKEN"

3. Choose one of the following deployment examples:

Procedure

1. In a new notebook cell, install the client:

   %pip install llama_stack_client

2. Import LlamaStackClient and create a client instance:

   from llama_stack_client import LlamaStackClient
   client = LlamaStackClient(base_url="<your deployment endpoint>")

3. List the available models:

   # Fetch all registered models
   models = client.models.list()

4. Verify that the list includes your Llama model and an embedding model. For example:

   [Model(identifier='llama-32-3b-instruct', metadata={}, api_model_type='llm', provider_id='vllm-inference', provider_resource_id='llama-32-3b-instruct', type='model', model_type='llm'),
    Model(identifier='ibm-granite/granite-embedding-125m-english', metadata={'embedding_dimension': 768.0}, api_model_type='embedding', provider_id='sentence-transformers', provider_resource_id='ibm-granite/granite-embedding-125m-english', type='model', model_type='embedding')]

5. Select one LLM and one embedding model:

   model_id = next(m.identifier for m in models if m.model_type == "llm")
   embedding_model = next(m for m in models if m.model_type == "embedding")
   embedding_model_id = embedding_model.identifier
   embedding_dimension = int(embedding_model.metadata["embedding_dimension"])

6. (Optional) Create a vector store (choose one). Skip this step if you already have one.

   Example 3.1. Option 1: Inline Milvus Lite (embedded)

   vector_store_name = "my_inline_db"
   vector_store = client.vector_stores.create(
       name=vector_store_name,
       extra_body={
           "embedding_model": embedding_model_id,
           "embedding_dimension": embedding_dimension,
           "provider_id": "milvus",   # inline Milvus Lite
       },
   )
   vector_store_id = vector_store.id
   print(f"Registered inline Milvus Lite DB: {vector_store_id}")

   Note

   Use inline Milvus Lite for development and small datasets. Persistence and scale are limited compared to remote Milvus.

1. If you already have a vector store, set its identifier:

   # For an existing vector store:
   # vector_store_id = "<your existing vector store ID>"

2. Define raw text to ingest:

   raw_text = """Llama Stack can embed raw text into a vector store for retrieval.
   This example ingests a small passage for demonstration."""

3. Ingest raw text by using the Vector Store Files API:

   items = [
       {
           "id": "raw_text_001",
           "text": raw_text,
           "mime_type": "text/plain",
           "metadata": {"source": "example_passage"},
       }
   ]
   result = client.vector_stores.files.create(
       vector_store_id=vector_store_id,
       items=items,
       chunk_size_in_tokens=100,
   )
   print("Text ingestion result:", result)

4. Ingest an HTML source:

   html_item = [
       {
           "id": "doc_html_001",
           "text": "https://www.paulgraham.com/greatwork.html",
           "mime_type": "text/html",
           "metadata": {"note": "Example URL"},
       }
   ]
   result = client.vector_stores.files.create(
       vector_store_id=vector_store_id,
       items=html_item,
       chunk_size_in_tokens=50,
   )
   print("HTML ingestion result:", result)

Procedure

1. In a new notebook cell, install the client:

   %pip install -q llama_stack_client

2. In a new notebook cell, import Agent, AgentEventLogger, and LlamaStackClient:

   from llama_stack_client import Agent, AgentEventLogger, LlamaStackClient

3. Create a client instance by setting your deployment endpoint:

   client = LlamaStackClient(base_url="<your deployment endpoint>")

4. List available models:

   models = client.models.list()

5. Select an LLM (and, if needed below, capture an embedding model for store registration):

   model_id = next(m.identifier for m in models if m.model_type == "llm")
   
   embedding = next((m for m in models if m.model_type == "embedding"), None)
   if embedding:
       embedding_model_id = embedding.identifier
       embedding_dimension = int(embedding.metadata.get("embedding_dimension", 768))

6. If you do not already have a vector store ID, register a vector store (choose one):

   Example 3.4. Option 1: Inline Milvus Lite (embedded)

   vector_store_name = "my_inline_db"
   vector_store = client.vector_stores.create(
       name=vector_store_name,
       extra_body={
           "embedding_model": embedding_model_id,
           "embedding_dimension": embedding_dimension,
           "provider_id": "milvus",   # inline Milvus Lite
       },
   )
   vector_store_id = vector_store.id
   print(f"Registered inline Milvus Lite DB: {vector_store_id}")

   Note

   Use inline Milvus Lite for development and small datasets. Persistence and scale are limited compared to remote Milvus.

1. If you already have a vector store, set its identifier:

   # For an existing store:
   # vector_store_id = "<your existing vector store ID>"

2. Query the ingested content by using the OpenAI-compatible Responses API with file search:

   query = "What benefits do the ingested passages provide for retrieval?"
   
   response = client.responses.create(
       model=model_id,
       input=query,
       tools=[
           {
               "type": "file_search",
               "vector_store_ids": [vector_store_id],
           }
       ],
   )
   print("Responses API result:", getattr(response, "output_text", response))

3. Query the ingested content by using the high-level Agent API:

   agent = Agent(
       client,
       model=model_id,
       instructions="You are a helpful assistant.",
       tools=[
           {
               "name": "builtin::rag/knowledge_search",
               "args": {"vector_store_ids": [vector_store_id]},
           }
       ],
   )
   
   prompt = "How do you do great work?"
   print("Prompt>", prompt)
   
   session_id = agent.create_session("rag_session")
   stream = agent.create_turn(
       messages=[{"role": "user", "content": prompt}],
       session_id=session_id,
       stream=True,
   )
   
   for log in AgentEventLogger().log(stream):
       log.print()

Procedure

1. In a new notebook cell, install the client:

   %pip install -q llama_stack_client

2. In a new notebook cell, import Agent, AgentEventLogger, and LlamaStackClient:

   from llama_stack_client import Agent, AgentEventLogger, LlamaStackClient

3. In a new notebook cell, assign your deployment endpoint to the base_url parameter to create a LlamaStackClient instance:

   client = LlamaStackClient(base_url="<your deployment endpoint>")

4. List the available models:

   models = client.models.list()

5. Select the first LLM and the first embedding model:

   model_id = next(m.identifier for m in models if m.model_type == "llm")
   embedding_model = next(m for m in models if m.model_type == "embedding")
   embedding_model_id = embedding_model.identifier
   embedding_dimension = int(embedding_model.metadata.get("embedding_dimension", 768))

6. Register a vector store (choose one option). Skip this step if your pipeline registers the store automatically.

   Example 3.7. Option 1: Inline Milvus Lite (embedded)

   vector_store_name = "my_inline_db"
   vector_store = client.vector_stores.create(
       name=vector_store_name,
       extra_body={
           "embedding_model": embedding_model_id,
           "embedding_dimension": embedding_dimension,
           "provider_id": "milvus",   # inline Milvus Lite
       },
   )
   vector_store_id = vector_store.id
   print(f"Registered inline Milvus Lite DB: {vector_store_id}")

   Note

   Inline Milvus Lite is best for development. Data durability and scale are limited compared to remote Milvus.

1. In the OpenShift web console, import the YAML file containing your Docling pipeline into your project, as described in Importing a pipeline.

2. Create a pipeline run to execute your Docling pipeline, as described in Executing a pipeline run. The pipeline run inserts your PDF documents into the vector store. If you run the Docling pipeline from the RAG demo samples repository, you can optionally customize the following parameters before starting the pipeline run:

   • base_url: The base URL to fetch PDF files from.
   • pdf_filenames: A comma-separated list of PDF filenames to download and convert.
   • num_workers: The number of parallel workers.
   • vector_store_id: The vector store identifier.
   • service_url: The Milvus service URL (only for remote Milvus).
   • embed_model_id: The embedding model to use.
   • max_tokens: The maximum tokens for each chunk.
   • use_gpu: Enable or disable GPU acceleration.

Verification

1. In your Jupyter notebook, query the LLM with a question that relates to the ingested content. For example:

   from llama_stack_client import Agent, AgentEventLogger
   import uuid
   
   rag_agent = Agent(
       client,
       model=model_id,
       instructions="You are a helpful assistant",
       tools=[
           {
               "name": "builtin::rag/knowledge_search",
               "args": {"vector_store_ids": [vector_store_id]},
           }
       ],
   )
   
   prompt = "What can you tell me about the birth of word processing?"
   print("prompt>", prompt)
   
   session_id = rag_agent.create_session(session_name=f"s{uuid.uuid4().hex}")
   
   response = rag_agent.create_turn(
       messages=[{"role": "user", "content": prompt}],
       session_id=session_id,
       stream=True,
   )
   
   for log in AgentEventLogger().log(response):
       log.print()

2. Query chunks from the vector store:

   query_result = client.vector_io.query(
       vector_store_id=vector_store_id,
       query="what do you know about?",
   )
   print(query_result)