Chapter 3. Deploying a RAG stack in a data science project

Procedure

1. Log in to the OpenShift web console as a cluster administrator.
2. In the Administrator perspective, click Operators Installed Operators.
3. Click the Red Hat OpenShift AI Operator to open its details.
4. Click the Data Science Cluster tab.
5. On the DataScienceClusters page, click the default-dsc object.

6. Click the YAML tab.

   An embedded YAML editor opens, displaying the configuration for the DataScienceCluster custom resource.

7. In the YAML editor, locate the spec.components section. If the llamastackoperator field does not exist, add it. Then, set the managementState field to Managed:

   spec:
     components:
       llamastackoperator:
         managementState: Managed

   Copy to Clipboard Toggle word wrap
8. Click Save to apply your changes.

1. In the OpenShift web console, click Workloads Pods.
2. From the Project list, select the redhat-ods-applications namespace.
3. Confirm that a pod with the label app.kubernetes.io/name=llama-stack-operator appears and has a status of Running.

1. In the OpenShift AI dashboard, navigate to the project details page and click the Models tab.
2. In the Single-model serving platform tile, click Select single-model.

3. Click the Deploy model button.

   The Deploy model dialog opens.

4. Configure the deployment properties for your model:

   1. In the Model deployment name field, enter a unique name for your deployment.
   2. In the Serving runtime field, select vLLM NVIDIA GPU serving runtime for KServe from the drop-down list.
   3. In the Deployment mode field, select Standard from the drop-down list.
   4. Set Number of model server replicas to deploy to 1.

   5. In the Model server size field, select Custom from the drop-down list.

      • Set CPUs requested to 1 core.
      • Set Memory requested to 10 GiB.
      • Set CPU limit to 2 core.
      • Set Memory limit to 14 GiB.
      • Set Accelerator to NVIDIA GPUs.
      • Set Accelerator count to 1.
   6. From the Connection type, select a relevant data connection from the drop-down list.

5. In the Additional serving runtime arguments field, specify the following recommended arguments:

   --dtype=half
   --max-model-len=20000
   --gpu-memory-utilization=0.95
   --enable-chunked-prefill
   --enable-auto-tool-choice
   --tool-call-parser=llama3_json
   --chat-template=/app/data/template/tool_chat_template_llama3.2_json.jinja

   Copy to Clipboard Toggle word wrap

   1. Click Deploy.

      Note

      Model deployment can take several minutes, especially for the first model that is deployed on the cluster. Initial deployment may take more than 10 minutes while the relevant images download.

Verification

1. Verify that the kserve-controller-manager and odh-model-controller pods are running:

   1. Open a new terminal window.
   2. Log in to your OpenShift cluster from the CLI:
   3. In the upper-right corner of the OpenShift web console, click your user name and select Copy login command.
   4. After you have logged in, click Display token.

   5. Copy the Log in with this token command and paste it in the OpenShift command-line interface (CLI).

      $ oc login --token=<token> --server=<openshift_cluster_url>

      Copy to Clipboard Toggle word wrap

   6. Enter the following command to verify that the kserve-controller-manager and odh-model-controller pods are running:

      $ oc get pods -n redhat-ods-applications | grep -E 'kserve-controller-manager|odh-model-controller'

      Copy to Clipboard Toggle word wrap

   7. Confirm that you see output similar to the following example:

      kserve-controller-manager-7c865c9c9f-xyz12   1/1     Running   0          4m21s
      odh-model-controller-7b7d5fd9cc-wxy34        1/1     Running   0          3m55s

      Copy to Clipboard Toggle word wrap

   8. If you do not see either of the kserve-controller-manager and odh-model-controller pods, there could be a problem with your deployment. In addition, if the pods appear in the list, but their Status is not set to Running, check the pod logs for errors:

      $ oc logs <pod-name> -n redhat-ods-applications

      Copy to Clipboard Toggle word wrap

   9. Check the status of the inference service:

      $ oc get inferenceservice -n llamastack
      $ oc get pods -n <data science project name> | grep llama

      Copy to Clipboard Toggle word wrap

      • The deployment automatically creates the following resources:

        • A ServingRuntime resource.
        • An InferenceService resource, a Deployment, a pod, and a service pointing to the pod.

      • Verify that the server is running. For example:

        $ oc logs llama-32-3b-instruct-predictor-77f6574f76-8nl4r  -n <data science project name>

        Copy to Clipboard Toggle word wrap

        Check for output similar to the following example log:

        INFO     2025-05-15 11:23:52,750 __main__:498 server: Listening on ['::', '0.0.0.0']:8321
        INFO:     Started server process [1]
        INFO:     Waiting for application startup.
        INFO     2025-05-15 11:23:52,765 __main__:151 server: Starting up
        INFO:     Application startup complete.
        INFO:     Uvicorn running on http://['::', '0.0.0.0']:8321 (Press CTRL+C to quit)

        Copy to Clipboard Toggle word wrap
      • The deployed model displays in the Models tab on the Data Science project details page for the project it was deployed under.

2. If you see a ConvertTritonGPUToLLVM error in the pod logs when querying the /v1/chat/completions API, and the vLLM server restarts or returns a 500 Internal Server error, apply the following workaround:

   Before deploying the model, remove the --enable-chunked-prefill argument from the Additional serving runtime arguments field in the deployment dialog.

   The error appears similar to the following:

   /opt/vllm/lib64/python3.12/site-packages/vllm/attention/ops/prefix_prefill.py:36:0: error: Failures have been detected while processing an MLIR pass pipeline
   /opt/vllm/lib64/python3.12/site-packages/vllm/attention/ops/prefix_prefill.py:36:0: note: Pipeline failed while executing [`ConvertTritonGPUToLLVM` on 'builtin.module' operation]: reproducer generated at `std::errs, please share the reproducer above with Triton project.`
   INFO:     10.129.2.8:0 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error

   Copy to Clipboard Toggle word wrap

Procedure

1. Open a new terminal window.

   1. Log in to your OpenShift cluster from the CLI:
   2. In the upper-right corner of the OpenShift web console, click your user name and select Copy login command.
   3. After you have logged in, click Display token.

   4. Copy the Log in with this token command and paste it in the OpenShift command-line interface (CLI).

      $ oc login --token=<token> --server=<openshift_cluster_url>

      Copy to Clipboard Toggle word wrap

2. If you enabled Require token authentication during model deployment, retrieve your token:

   $ export MODEL_TOKEN=$(oc get secret default-name-llama-32-3b-instruct-sa -n <project name> --template={{ .data.token }} | base64 -d)

   Copy to Clipboard Toggle word wrap

3. Obtain your model endpoint URL:

   • If you enabled Make deployed models available through an external route during model deployment, click Endpoint details on the Model deployments page in the OpenShift AI dashboard to obtain your model endpoint URL.

   • In addition, if you did not enable Require token authentication during model deployment, you can also enter the following command to retrieve the endpoint URL:

     $ export MODEL_ENDPOINT="https://$(oc get route llama-32-3b-instruct -n <project name> --template={{ .spec.host }})"

     Copy to Clipboard Toggle word wrap

4. Test the endpoint with a sample chat completion request:

   • If you did not enable Require token authentication during model deployment, enter a chat completion request. For example:

     $ curl -X POST $MODEL_ENDPOINT/v1/chat/completions \
      -H "Content-Type: application/json" \
      -d '{
      "model": "llama-32-3b-instruct",
      "messages": [
        {
          "role": "user",
          "content": "Hello"
        }
      ]
     }'

     Copy to Clipboard Toggle word wrap

   • If you enabled Require token authentication during model deployment, include a token in your request. For example:

     curl -s -k $MODEL_ENDPOINT/v1/chat/completions \
     --header "Authorization: Bearer $MODEL_TOKEN" \
     --header 'Content-Type: application/json' \
     -d '{
       "model": "llama-32-3b-instruct",
       "messages": [
         {
           "role": "user",
           "content": "can you tell me a funny joke?"
         }
       ]
     }' | jq .

     Copy to Clipboard Toggle word wrap
     Note

     The -k flag disables SSL verification and should only be used in test environments or with self-signed certificates.

Procedure

1. Open a new terminal window.

   1. Log in to your OpenShift cluster from the CLI:
   2. In the upper-right corner of the OpenShift web console, click your user name and select Copy login command.
   3. After you have logged in, click Display token.

   4. Copy the Log in with this token command and paste it in the OpenShift command-line interface (CLI).

      $ oc login --token=<token> --server=<openshift_cluster_url>

      Copy to Clipboard Toggle word wrap

2. Create a secret containing the inference model environment variables:

   export INFERENCE_MODEL="llama-3-2-3b"
   export VLLM_URL="https://llama-32-3b-instruct-predictor:8443/v1"
   export VLLM_TLS_VERIFY="false" # Use "true" in production!
   export VLLM_API_TOKEN="<token identifier>"
   
   oc create secret generic llama-stack-inference-model-secret \
     --from-literal INFERENCE_MODEL="$INFERENCE_MODEL" \
     --from-literal VLLM_URL="$VLLM_URL" \
     --from-literal VLLM_TLS_VERIFY="$VLLM_TLS_VERIFY" \
     --from-literal VLLM_API_TOKEN="$VLLM_API_TOKEN"

   Copy to Clipboard Toggle word wrap
3. Log in to the OpenShift web console.
4. From the left-hand navigation, select Administrator view.
5. Click the Quick Create ( ) icon and then click the Import YAML option.

6. In the YAML editor that appears, create a custom resource definition (CRD) similar to the following example:

   apiVersion: llamastack.io/v1alpha1
   kind: LlamaStackDistribution
   metadata:
     name: lsd-llama-milvus
   spec:
     replicas: 1
     server:
       containerSpec:
         resources:
           requests:
             cpu: "250m"
             memory: "500Mi"
           limits:
             cpu: "2"
             memory: "12Gi"
         env:
           - name: INFERENCE_MODEL
             valueFrom:
               secretKeyRef:
                 key: INFERENCE_MODEL
                 name: llama-stack-inference-model-secret
           - name: VLLM_URL
             valueFrom:
               secretKeyRef:
                 key: VLLM_URL
                 name: llama-stack-inference-model-secret
           - name: VLLM_TLS_VERIFY
             valueFrom:
               secretKeyRef:
                 key: VLLM_TLS_VERIFY
                 name: llama-stack-inference-model-secret
           - name: VLLM_API_TOKEN
             valueFrom:
               secretKeyRef:
                 key: VLLM_API_TOKEN
                 name: llama-stack-inference-model-secret
           - name: MILVUS_DB_PATH
             value: ~/.llama/milvus.db
           - name: FMS_ORCHESTRATOR_URL
             value: "http://localhost"
         name: llama-stack
         port: 8321
       distribution:
         name: rh-dev

   Copy to Clipboard Toggle word wrap
   Note

   The rh-dev value is an internal image reference. When you create the LlamaStackDistribution custom resource, the OpenShift AI Operator automatically resolves rh-dev to the container image in the appropriate registry. This internal image reference allows the underlying image to update without requiring changes to your custom resource.

7. Click Create.

Procedure

1. In a new notebook cell, install the llama_stack client package:

   %pip install llama_stack

   Copy to Clipboard Toggle word wrap

2. In a new notebook cell, import RAGDocument and LlamaStackClient:

   from llama_stack_client import RAGDocument, LlamaStackClient

   Copy to Clipboard Toggle word wrap

3. In a new notebook cell, assign your deployment endpoint to the base_url parameter to create a LlamaStackClient instance:

   client = LlamaStackClient(base_url="<your deployment endpoint>")

   Copy to Clipboard Toggle word wrap

4. List the available models:

   # Fetch all registered models
   models = client.models.list()

   Copy to Clipboard Toggle word wrap

5. Verify that the list of registered models includes your Llama model and an embedding model. Here is an example of a list of registered models:

   [Model(identifier='llama-32-3b-instruct', metadata={}, api_model_type='llm', provider_id='vllm-inference', provider_resource_id='llama-32-3b-instruct', type='model', model_type='llm'),
    Model(identifier='ibm-granite/granite-embedding-125m-english', metadata={'embedding_dimension': 768.0}, api_model_type='embedding', provider_id='sentence-transformers', provider_resource_id='ibm-granite/granite-embedding-125m-english', type='model', model_type='embedding')]

   Copy to Clipboard Toggle word wrap

6. Select the first LLM and the first embedding model:

   model_id = next(m.identifier for m in models if m.model_type == "llm")
   
   embedding_model = next(m for m in models if m.model_type == "embedding")
   embedding_model_id = embedding_model.identifier
   embedding_dimension = embedding_model.metadata["embedding_dimension"]

   Copy to Clipboard Toggle word wrap

7. In a new notebook cell, define the following parameters:

   • vector_db_id: a unique name that identifies your vector database, for example, my_milvus_db.

   • provider_id: the connector key that your Llama Stack gateway has enabled. For the Milvus vector database, this connector key is "milvus". You can also list the available connectors:

     print(client.vector_dbs.list_providers()) # lists available connectors
     
     vector_db_id = "<your vector database ID>"
     provider_id  = "<your provider ID>"

     Copy to Clipboard Toggle word wrap

8. In a new notebook cell, register or confirm your vector database to store embeddings:

   _ = client.vector_dbs.register(
   vector_db_id=vector_db_id,
   embedding_model=embedding_model_id,
   embedding_dimension=embedding_dimension,
   provider_id=provider_id,
   )
   print(f"Registered vector DB: {vector_db_id}")

   Copy to Clipboard Toggle word wrap
   Important

   If you skip this step, and as a result, you do not register your vector database with your vector database ID, an error occurs if you attempt to ingest text into your vector database.

9. In a new notebook cell, define the raw text that you want to ingest into the vector store:

   # Example raw text passage
   raw_text = """
   LlamaStack can embed raw text into a vector store for retrieval.
   This example ingests a small passage for demonstration.
   """

   Copy to Clipboard Toggle word wrap

10. In a new notebook cell, create a RAGDocument object to contain the raw text:

    document = RAGDocument(
    document_id="raw_text_001",
    content=raw_text,
    mime_type="text/plain",
    metadata={"source": "example_passage"},
    )

    Copy to Clipboard Toggle word wrap

11. In a new notebook cell, ingest the raw text:

    client.tool_runtime.rag_tool.insert(
        documents=[document],
        vector_db_id=vector_db_id,
        chunk_size_in_tokens=100,
    )
    print("Raw text ingested successfully")

    Copy to Clipboard Toggle word wrap

12. In a new notebook cell, create a RAGDocument from an HTML source and ingest it into the vector store:

    source = "https://www.paulgraham.com/greatwork.html"
    print("rag_tool> Ingesting document:", source)
    
    document = RAGDocument(
        document_id="document_1",
        content=source,
        mime_type="text/html",
        metadata={},
    )

    Copy to Clipboard Toggle word wrap

13. In a new notebook cell, ingest the content into the vector store:

    client.tool_runtime.rag_tool.insert(
        documents=[document],
        vector_db_id=vector_db_id,
        chunk_size_in_tokens=50,
    )
    print("Raw text ingested successfully")

    Copy to Clipboard Toggle word wrap

Procedure

1. In a new notebook cell, install the llama_stack client package:

   %pip install llama_stack_client

   Copy to Clipboard Toggle word wrap

2. In a new notebook cell, import Agent, AgentEventLogger, and LlamaStackClient:

   from llama_stack_client import Agent, AgentEventLogger, LlamaStackClient

   Copy to Clipboard Toggle word wrap

3. In a new notebook cell, assign your deployment endpoint to the base_url parameter to create a LlamaStackClient instance. For example:

   client = LlamaStackClient(base_url="http://lsd-llama-milvus-service:8321/")

   Copy to Clipboard Toggle word wrap

4. In a new notebook cell, list the available models:

   models = client.models.list()

   Copy to Clipboard Toggle word wrap

5. Verify that the list of registered models includes your Llama model and an embedding model. Here is an example of a list of registered models:

   [Model(identifier='llama-32-3b-instruct', metadata={}, api_model_type='llm', provider_id='vllm-inference', provider_resource_id='llama-32-3b-instruct', type='model', model_type='llm'),
    Model(identifier='ibm-granite/granite-embedding-125m-english', metadata={'embedding_dimension': 768.0}, api_model_type='embedding', provider_id='sentence-transformers', provider_resource_id='ibm-granite/granite-embedding-125m-english', type='model', model_type='embedding')]

   Copy to Clipboard Toggle word wrap

6. In a new notebook cell, select the first LLM in your list of registered models:

   model_id = next(m.identifier for m in models if m.model_type == "llm")

   Copy to Clipboard Toggle word wrap

7. In a new notebook cell, define the vector_db_id, which is a unique name that identifies your vector database, for example, my_milvus_db. If you do not know your vector database ID, contact an administrator.

   vector_db_id = "<your vector database ID>"

   Copy to Clipboard Toggle word wrap

8. In a new notebook cell, query the ingested content using the low-level RAG tool:

   # Example RAG query for one-off lookups
   query = "What benefits do the ingested passages provide for retrieval?"
   result = client.tool_runtime.rag_tool.query(
       vector_db_ids=[vector_db_id],
       content=query,
   )
   print("Low-level query result:", result)

   Copy to Clipboard Toggle word wrap

9. In a new notebook cell, query the ingested content by using the high-level Agent API:

   # Create an Agent for conversational RAG queries
   agent = Agent(
       client,
       model=model_id,
       instructions="You are a helpful assistant.",
       tools=[
           {
               "name": "builtin::rag/knowledge_search",
               "args": {"vector_db_ids": [vector_db_id]},
           }
       ],
   )
   
   prompt = "How do you do great work?"
   print("Prompt>", prompt)
   
   # Create a session and run a streaming turn
   session_id = agent.create_session("rag_session")
   response = agent.create_turn(
       messages=[{"role": "user", "content": prompt}],
       session_id=session_id,
       stream=True,
   )
   
   # Log and print the agent's response
   for log in AgentEventLogger().log(response):
       log.print()

   Copy to Clipboard Toggle word wrap

Procedure

1. In a new notebook cell, install the llama_stack client package:

   %pip install llama_stack_client

   Copy to Clipboard Toggle word wrap

2. In a new notebook cell, import Agent, AgentEventLogger, and LlamaStackClient:

   from llama_stack_client import Agent, AgentEventLogger, LlamaStackClient

   Copy to Clipboard Toggle word wrap

3. In a new notebook cell, assign your deployment endpoint to the base_url parameter to create a LlamaStackClient instance:

   client = LlamaStackClient(base_url="<your deployment endpoint>")

   Copy to Clipboard Toggle word wrap

4. List the available models:

   models = client.models.list()

   Copy to Clipboard Toggle word wrap

5. Select the first LLM and the first embedding model:

   model_id = next(m.identifier for m in models if m.model_type == "llm")
   embedding_model = next(m for m in models if m.model_type == "embedding")
   embedding_model_id = embedding_model.identifier
   embedding_dimension = embedding_model.metadata["embedding_dimension"]

   Copy to Clipboard Toggle word wrap

6. In a new notebook cell, define the following parameters:

   • vector_db_id: a unique name that identifies your vector database, for example, my_milvus_db.

   • provider_id: the connector key that your Llama Stack gateway has enabled. For the Milvus vector database, this connector key is "milvus". You can also list the available connectors:

     print(client.vector_dbs.list_providers()) # lists available connectors
     
     vector_db_id = "<your vector database ID>"
     provider_id  = "<your provider ID>"

     Copy to Clipboard Toggle word wrap
     Important

     If you are using the sample Docling pipeline from the RAG demo repository, the pipeline registers the database automatically and you can skip this step. However, if you are using your own pipeline, you must register the database yourself.

7. In a new notebook cell, register or confirm your vector database to store embeddings:

   _ = client.vector_dbs.register(
   vector_db_id=vector_db_id,
   embedding_model=embedding_model_id,
   embedding_dimension=embedding_dimension,
   provider_id=provider_id,
   )
   print(f"Registered vector DB: {vector_db_id}")

   Copy to Clipboard Toggle word wrap
8. In the OpenShift web console, import the YAML file containing your docling pipeline into your data science project, as described in Importing a data science pipeline.

9. Create a pipeline run to execute your Docling pipeline, as described in Executing a pipeline run. The pipeline run inserts your PDF documents into the vector database. If you run the Docling pipeline from the RAG demo samples repository, you can optionally customize the following parameters before starting the pipeline run:

   • base_url: The base URL to fetch PDF files from.
   • pdf_filenames: A comma-separated list of PDF filenames to download and convert.
   • num_workers: The number of parallel workers.
   • vector_db_id: The Milvus vector database ID.
   • service_url: The Milvus service URL.
   • embed_model_id: The embedding model to use.
   • max_tokens: The maximum tokens for each chunk.
   • use_gpu: Enable or disable GPU acceleration.

Verification

1. In your Jupyter notebook, query the LLM with a question that relates to the ingested content. For example:

   from llama_stack_client import Agent, AgentEventLogger
   import uuid
   
   rag_agent = Agent(
       client,
       model=model_id,
       instructions="You are a helpful assistant",
       tools=[
           {
               "name": "builtin::rag/knowledge_search",
               "args": {"vector_db_ids": [vector_db_id]},
           }
       ],
   )
   
   prompt = "What can you tell me about the birth of word processing?"
   print("prompt>", prompt)
   
   session_id = rag_agent.create_session(session_name=f"s{uuid.uuid4().hex}")
   
   response = rag_agent.create_turn(
       messages=[{"role": "user", "content": prompt}],
       session_id=session_id,
       stream=True,
   )
   
   for log in AgentEventLogger().log(response):
       log.print()

   Copy to Clipboard Toggle word wrap

2. Query chunks from the vector database:

   query_result = client.vector_io.query(
       vector_db_id=vector_db_id,
       query="what do you know about?",
   )
   print(query_result)

   Copy to Clipboard Toggle word wrap