Chapter 4. Llama Stack application examples

1. In the OpenShift AI dashboard, navigate to the project details page and click the Deployments tab.
2. In the Model serving platform tile, click Select model.

3. Click the Deploy model button.

   The Deploy model dialog opens.

4. Configure the deployment properties for your model:

   1. In the Model deployment name field, enter a unique name for your deployment.
   2. In the Serving runtime field, select vLLM NVIDIA GPU serving runtime for KServe from the drop-down list.
   3. In the Deployment mode field, select KServe RawDeployment from the drop-down list.
   4. Set Number of model server replicas to deploy to 1.

   5. In the Model server size field, select Custom from the drop-down list.

      • Set CPUs requested to 1 core.
      • Set Memory requested to 10 GiB.
      • Set CPU limit to 2 core.
      • Set Memory limit to 14 GiB.
      • Set Accelerator to NVIDIA GPUs.
      • Set Accelerator count to 1.
   6. From the Connection type, select a relevant data connection from the drop-down list.

5. In the Additional serving runtime arguments field, specify the following recommended arguments:

   --dtype=half
   --max-model-len=20000
   --gpu-memory-utilization=0.95
   --enable-chunked-prefill
   --enable-auto-tool-choice
   --tool-call-parser=llama3_json
   --chat-template=/app/data/template/tool_chat_template_llama3.2_json.jinja

   1. Click Deploy.

      Note

      Model deployment can take several minutes, especially for the first model that is deployed on the cluster. Initial deployment may take more than 10 minutes while the relevant images download.

Verification

1. Verify that the kserve-controller-manager and odh-model-controller pods are running:

   1. Open a new terminal window.
   2. Log in to your OpenShift cluster from the CLI:
   3. In the upper-right corner of the OpenShift web console, click your user name and select Copy login command.
   4. After you have logged in, click Display token.

   5. Copy the Log in with this token command and paste it in the OpenShift CLI (oc).

      $ oc login --token=<token> --server=<openshift_cluster_url>

   6. Enter the following command to verify that the kserve-controller-manager and odh-model-controller pods are running:

      $ oc get pods -n redhat-ods-applications | grep -E 'kserve-controller-manager|odh-model-controller'

   7. Confirm that you see output similar to the following example:

      kserve-controller-manager-7c865c9c9f-xyz12   1/1     Running   0          4m21s
      odh-model-controller-7b7d5fd9cc-wxy34        1/1     Running   0          3m55s

   8. If you do not see either of the kserve-controller-manager and odh-model-controller pods, there could be a problem with your deployment. In addition, if the pods appear in the list, but their Status is not set to Running, check the pod logs for errors:

      $ oc logs <pod-name> -n redhat-ods-applications

   9. Check the status of the inference service:

      $ oc get inferenceservice -n llamastack
      $ oc get pods -n <project name> | grep llama

      • The deployment automatically creates the following resources:

        • A ServingRuntime resource.
        • An InferenceService resource, a Deployment, a pod, and a service pointing to the pod.

      • Verify that the server is running. For example:

        $ oc logs llama-32-3b-instruct-predictor-77f6574f76-8nl4r  -n <project name>

        Check for output similar to the following example log:

        INFO     2025-05-15 11:23:52,750 __main__:498 server: Listening on ['::', '0.0.0.0']:8321
        INFO:     Started server process [1]
        INFO:     Waiting for application startup.
        INFO     2025-05-15 11:23:52,765 __main__:151 server: Starting up
        INFO:     Application startup complete.
        INFO:     Uvicorn running on http://['::', '0.0.0.0']:8321 (Press CTRL+C to quit)

      • The deployed model displays in the Deployments tab on the project details page for the project it was deployed under.

2. If you see a ConvertTritonGPUToLLVM error in the pod logs when querying the /v1/chat/completions API, and the vLLM server restarts or returns a 500 Internal Server error, apply the following workaround:

   Before deploying the model, remove the --enable-chunked-prefill argument from the Additional serving runtime arguments field in the deployment dialog.

   The error is displayed similar to the following:

   /opt/vllm/lib64/python3.12/site-packages/vllm/attention/ops/prefix_prefill.py:36:0: error: Failures have been detected while processing an MLIR pass pipeline
   /opt/vllm/lib64/python3.12/site-packages/vllm/attention/ops/prefix_prefill.py:36:0: note: Pipeline failed while executing [`ConvertTritonGPUToLLVM` on 'builtin.module' operation]: reproducer generated at `std::errs, please share the reproducer above with Triton project.`
   INFO:     10.129.2.8:0 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error

Procedure

1. Open a new terminal window.

   1. Log in to your OpenShift cluster from the CLI:
   2. In the upper-right corner of the OpenShift web console, click your user name and select Copy login command.
   3. After you have logged in, click Display token.

   4. Copy the Log in with this token command and paste it in the OpenShift CLI (oc).

      $ oc login --token=<token> --server=<openshift_cluster_url>

2. If you enabled Require token authentication during model deployment, retrieve your token:

   $ export MODEL_TOKEN=$(oc get secret default-name-llama-32-3b-instruct-sa -n <project name> --template={{ .data.token }} | base64 -d)

3. Obtain your model endpoint URL:

   • If you enabled Make deployed models available through an external route during model deployment, click Endpoint details on the Deployments page in the OpenShift AI dashboard to obtain your model endpoint URL.

   • In addition, if you did not enable Require token authentication during model deployment, you can also enter the following command to retrieve the endpoint URL:

     $ export MODEL_ENDPOINT="https://$(oc get route llama-32-3b-instruct -n <project name> --template={{ .spec.host }})"

4. Test the endpoint with a sample chat completion request:

   • If you did not enable Require token authentication during model deployment, enter a chat completion request. For example:

     $ curl -X POST $MODEL_ENDPOINT/v1/chat/completions \
      -H "Content-Type: application/json" \
      -d '{
      "model": "llama-32-3b-instruct",
      "messages": [
        {
          "role": "user",
          "content": "Hello"
        }
      ]
     }'

   • If you enabled Require token authentication during model deployment, include a token in your request. For example:

     curl -s -k $MODEL_ENDPOINT/v1/chat/completions \
     --header "Authorization: Bearer $MODEL_TOKEN" \
     --header 'Content-Type: application/json' \
     -d '{
       "model": "llama-32-3b-instruct",
       "messages": [
         {
           "role": "user",
           "content": "can you tell me a funny joke?"
         }
       ]
     }' | jq .

     Note

     The -k flag disables SSL verification and should only be used in test environments or with self-signed certificates.

Procedure

1. In the OpenShift console, click the Quick Create ( ) icon and then click the Import YAML option.
2. Verify that your project is the selected project.

3. In the Import YAML editor, paste the following manifest and click Create:

   apiVersion: v1
   kind: Secret
   metadata:
     name: milvus-secret
   type: Opaque
   stringData:
     root-password: "MyStr0ngP@ssw0rd"
   ---
   kind: PersistentVolumeClaim
   apiVersion: v1
   metadata:
     name: milvus-pvc
   spec:
     accessModes:
       - ReadWriteOnce
     resources:
       requests:
         storage: 20Gi
     volumeMode: Filesystem
   ---
   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: etcd-deployment
     labels:
       app: etcd
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: etcd
     strategy:
       type: Recreate
     template:
       metadata:
         labels:
           app: etcd
       spec:
         containers:
           - name: etcd
             image: quay.io/coreos/etcd:v3.5.5
             command:
               - etcd
               - --advertise-client-urls=http://127.0.0.1:2379
               - --listen-client-urls=http://0.0.0.0:2379
               - --data-dir=/etcd
             ports:
               - containerPort: 2379
             volumeMounts:
               - name: etcd-data
                 mountPath: /etcd
             env:
               - name: ETCD_AUTO_COMPACTION_MODE
                 value: revision
               - name: ETCD_AUTO_COMPACTION_RETENTION
                 value: "1000"
               - name: ETCD_QUOTA_BACKEND_BYTES
                 value: "4294967296"
               - name: ETCD_SNAPSHOT_COUNT
                 value: "50000"
         volumes:
           - name: etcd-data
             emptyDir: {}
         restartPolicy: Always
   ---
   apiVersion: v1
   kind: Service
   metadata:
     name: etcd-service
   spec:
     ports:
       - port: 2379
         targetPort: 2379
     selector:
       app: etcd
   ---
   apiVersion: apps/v1
   kind: Deployment
   metadata:
     labels:
       app: milvus-standalone
     name: milvus-standalone
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: milvus-standalone
     strategy:
       type: Recreate
     template:
       metadata:
         labels:
           app: milvus-standalone
       spec:
         containers:
           - name: milvus-standalone
             image: milvusdb/milvus:v2.6.0
             args: ["milvus", "run", "standalone"]
             env:
               - name: DEPLOY_MODE
                 value: standalone
               - name: ETCD_ENDPOINTS
                 value: etcd-service:2379
               - name: COMMON_STORAGETYPE
                 value: local
               - name: MILVUS_ROOT_PASSWORD
                 valueFrom:
                   secretKeyRef:
                     name: milvus-secret
                     key: root-password
             livenessProbe:
               exec:
                 command: ["curl", "-f", "http://localhost:9091/healthz"]
               initialDelaySeconds: 90
               periodSeconds: 30
               timeoutSeconds: 20
               failureThreshold: 5
             ports:
               - containerPort: 19530
                 protocol: TCP
               - containerPort: 9091
                 protocol: TCP
             volumeMounts:
               - name: milvus-data
                 mountPath: /var/lib/milvus
         restartPolicy: Always
         volumes:
           - name: milvus-data
             persistentVolumeClaim:
               claimName: milvus-pvc
   ---
   apiVersion: v1
   kind: Service
   metadata:
     name: milvus-service
   spec:
     selector:
       app: milvus-standalone
     ports:
       - name: grpc
         port: 19530
         targetPort: 19530
       - name: http
         port: 9091
         targetPort: 9091

   Note
   • Use the gRPC port (19530) for the MILVUS_ENDPOINT setting in Llama Stack.
   • The HTTP port (9091) is reserved for health checks.
   • If you deploy Milvus in a different namespace, use the fully qualified service name in your Llama Stack configuration. For example: http://milvus-service.<namespace>.svc.cluster.local:19530

Verification

1. In the OpenShift web console, click Workloads Deployments.
2. Verify that both etcd-deployment and milvus-standalone show a status of 1 of 1 pods available.
3. Click Pods in the navigation panel and confirm that pods for both deployments are Running.
4. Click the milvus-standalone pod name, then select the Logs tab.

5. Verify that Milvus reports a healthy startup with output similar to:

   Milvus Standalone is ready to serve ...
   Listening on 0.0.0.0:19530 (gRPC)

6. Click Networking Services and confirm that the milvus-service and etcd-service resources exist and are exposed on ports 19530 and 2379, respectively.

7. (Optional) Click Pods milvus-standalone Terminal and run the following health check:

   curl http://localhost:9091/healthz

   A response of {"status": "healthy"} confirms that Milvus is running correctly.

Procedure

1. Open a new terminal window and log in to your OpenShift cluster from the CLI:

   In the upper-right corner of the OpenShift web console, click your user name and select Copy login command. After you have logged in, click Display token. Copy the Log in with this token command and paste it in the OpenShift CLI (oc).

   $ oc login --token=<token> --server=<openshift_cluster_url>

2. Create a secret that contains the inference model and the remote embeddings environment variables:

   # Remote LLM
   export INFERENCE_MODEL="llama-3-2-3b"
   export VLLM_URL="https://llama-32-3b-instruct-predictor:8443/v1"
   export VLLM_TLS_VERIFY="false"   # Use "true" in production
   export VLLM_API_TOKEN="<token identifier>"
   export VLLM_MAX_TOKENS=16384
   
   # Remote embedding configuration
   export EMBEDDING_MODEL="nomic-embed-text-v1-5"
   export EMBEDDING_PROVIDER_MODEL_ID="nomic-embed-text-v1-5"
   export VLLM_EMBEDDING_URL="<embedding-endpoint>/v1"
   export VLLM_EMBEDDING_API_TOKEN="<embedding-token>"
   export VLLM_EMBEDDING_MAX_TOKENS=8192
   export VLLM_EMBEDDING_TLS_VERIFY="true"
   
   oc create secret generic llama-stack-secret -n <project-name> \
     --from-literal=INFERENCE_MODEL="$INFERENCE_MODEL" \
     --from-literal=VLLM_URL="$VLLM_URL" \
     --from-literal=VLLM_TLS_VERIFY="$VLLM_TLS_VERIFY" \
     --from-literal=VLLM_API_TOKEN="$VLLM_API_TOKEN" \
     --from-literal=VLLM_MAX_TOKENS="$VLLM_MAX_TOKENS" \
     --from-literal=EMBEDDING_MODEL="$EMBEDDING_MODEL" \
     --from-literal=EMBEDDING_PROVIDER_MODEL_ID="$EMBEDDING_PROVIDER_MODEL_ID" \
     --from-literal=VLLM_EMBEDDING_URL="$VLLM_EMBEDDING_URL" \
     --from-literal=VLLM_EMBEDDING_TLS_VERIFY="$VLLM_EMBEDDING_TLS_VERIFY" \
     --from-literal=VLLM_EMBEDDING_API_TOKEN="$VLLM_EMBEDDING_API_TOKEN" \
     --from-literal=VLLM_EMBEDDING_MAX_TOKENS="$VLLM_EMBEDDING_MAX_TOKENS"

3. Choose one of the following deployment examples:

Procedure

1. In a new notebook cell, install the client:

   %pip install llama_stack_client

2. Install the requests library if it is not already available:

   %pip install requests

3. Import LlamaStackClient and create a client instance:

   from llama_stack_client import LlamaStackClient
   
   # Use the Llama Stack service or route URL that is reachable from the workbench.
   # Do not append /v1 when using llama_stack_client.
   client = LlamaStackClient(base_url="<llama-stack-base-url>")

4. List the available models:

   models = client.models.list()

5. Verify that the list includes:

   • At least one LLM model.

   • At least one embedding model (remote or inline).

     [Model(identifier='llama-32-3b-instruct', model_type='llm', provider_id='vllm-inference'),
      Model(identifier='nomic-embed-text-v1-5', model_type='embedding', metadata={'embedding_dimension': 768})]

6. Select one LLM and one embedding model:

   model_id = next(m.identifier for m in models if m.model_type == "llm")
   
   embedding_model = next(m for m in models if m.model_type == "embedding")
   embedding_model_id = embedding_model.identifier
   embedding_dimension = int(embedding_model.metadata["embedding_dimension"])

7. (Optional) Create a vector store. Skip this step if you already have one.

   Note

   Provider IDs can differ between interfaces. In the Python SDK, you typically use the provider name directly (for example, provider_id: "pgvector"). In some CLI tools and examples, remote providers might use a prefixed identifier (for example, --vector-db-provider-id remote-pgvector). Use the provider ID format that matches the interface you are using.

1. If you already have a vector store, set its identifier:

   # vector_store_id = "<existing-vector-store-id>"

2. Download a PDF, upload it to Llama Stack, and add it to your vector store:

   import requests
   
   pdf_url = "https://www.federalreserve.gov/aboutthefed/files/quarterly-report-20250822.pdf"
   filename = "quarterly-report-20250822.pdf"
   
   response = requests.get(pdf_url)
   response.raise_for_status()
   
   with open(filename, "wb") as f:
       f.write(response.content)
   
   with open(filename, "rb") as f:
       file_info = client.files.create(
           file=(filename, f),
           purpose="assistants",
       )
   
   vector_store_file = client.vector_stores.files.create(
       vector_store_id=vector_store_id,
       file_id=file_info.id,
       chunking_strategy={
           "type": "static",
           "static": {
               "max_chunk_size_tokens": 800,
               "chunk_overlap_tokens": 400,
           },
       },
   )
   
   print(vector_store_file)

Procedure

1. In a new notebook cell, install the client:

   %pip install -q llama_stack_client

2. Import LlamaStackClient:

   from llama_stack_client import LlamaStackClient

3. Create a client instance:

   # Use the Llama Stack service or route URL that is reachable from the workbench.
   # Do not append /v1 when using llama_stack_client.
   client = LlamaStackClient(base_url="<llama-stack-base-url>")

4. List available models:

   models = client.models.list()

5. Select an LLM. If you plan to register a new vector store, also capture an embedding model:

   model_id = next(m.identifier for m in models if m.model_type == "llm")
   
   embedding = next((m for m in models if m.model_type == "embedding"), None)
   if embedding:
       embedding_model_id = embedding.identifier
       embedding_dimension = int(embedding.metadata.get("embedding_dimension", 768))

6. If you do not already have a vector store ID, register a vector store (choose one):

   Example 4.5. Option 1: Inline Milvus (embedded)

   vector_store = client.vector_stores.create(
       name="my_inline_milvus",
       extra_body={
           "embedding_model": embedding_model_id,
           "embedding_dimension": embedding_dimension,
           "provider_id": "milvus",
       },
   )
   vector_store_id = vector_store.id

   Note

   Inline Milvus is suitable for development and small datasets. In OpenShift AI 3.2 and later, metadata persistence uses PostgreSQL by default.

1. If you already have a vector store, set its identifier:

   # vector_store_id = "<existing-vector-store-id>"

2. Query without using a vector store:

   system_instructions = """You are a precise and reliable AI assistant.
   Use retrieved context when it is available.
   If nothing relevant is found, say so clearly."""
   
   query = "How do you do great work?"
   
   response = client.responses.create(
       model=model_id,
       input=query,
       instructions=system_instructions,
   )
   
   print(response.output_text)

3. Query by using the Responses API with file search:

   response = client.responses.create(
       model=model_id,
       input=query,
       instructions=system_instructions,
       tools=[
           {
               "type": "file_search",
               "vector_store_ids": [vector_store_id],
           }
       ],
   )
   
   print(response.output_text)

Procedure

1. In a new notebook cell, install the client:

   %pip install -q llama_stack_client

2. In a new notebook cell, import LlamaStackClient:

   from llama_stack_client import LlamaStackClient

3. In a new notebook cell, assign your deployment endpoint to the base_url parameter to create a LlamaStackClient instance:

   client = LlamaStackClient(base_url="http://<llama-stack-service>:8321")

   Note

   LlamaStackClient requires the service root without the /v1 path suffix. For example, use http://llama-stack-service:8321.

   The /v1 suffix is required only when you use OpenAI-compatible SDKs or send raw HTTP requests to the OpenAI-compatible API surface.

4. List the available models:

   models = client.models.list()

5. Select the first LLM and the first embedding model:

   model_id = next(m.identifier for m in models if m.model_type == "llm")
   embedding_model = next(m for m in models if m.model_type == "embedding")
   embedding_model_id = embedding_model.identifier
   embedding_dimension = int(embedding_model.metadata.get("embedding_dimension", 768))

6. Register a vector store (choose one option). Skip this step if your pipeline registers the store automatically.

1. In the OpenShift web console, import the YAML file containing your Docling pipeline into your project, as described in Importing a pipeline.

2. Create a pipeline run to execute your Docling pipeline, as described in Executing a pipeline run. The pipeline run inserts your PDF documents into the vector store. If you run the Docling pipeline from the RAG demo samples repository, you can optionally customize the following parameters before starting the pipeline run:

   • base_url: The base URL to fetch PDF files from.
   • pdf_filenames: A comma-separated list of PDF filenames to download and convert.
   • num_workers: The number of parallel workers.
   • vector_store_id: The vector store identifier.
   • service_url: The Milvus service URL (only for remote Milvus).
   • embed_model_id: The embedding model to use.
   • max_tokens: The maximum tokens for each chunk.
   • use_gpu: Enable or disable GPU acceleration.

Verification

1. In your Jupyter notebook, query the LLM with a question that relates to the ingested content:

   system_instructions = """You are a precise and reliable AI assistant.
   Use retrieved context when it is available.
   If nothing relevant is found in the available files, say so clearly."""
   
   prompt = "What can you tell me about the birth of word processing?"
   
   # Query using the Responses API with file search
   response = client.responses.create(
       model=model_id,
       input=prompt,
       instructions=system_instructions,
       tools=[
           {
               "type": "file_search",
               "vector_store_ids": [vector_store_id],
           }
       ],
   )
   
   print("Answer (with vector stores):")
   print(response.output_text)

2. Query chunks from the vector store:

   query_result = client.vector_io.query(
       vector_store_id=vector_store_id,
       query="word processing",
   )
   print(query_result)

   • The pipeline run completes successfully in your project.
   • Document embeddings are stored in the vector store and are available for retrieval.
   • No errors or warnings appear in the pipeline logs or your notebook output.

Procedure

1. Optional: Start the Llama Stack server and enable a vector database provider If you have not already started Llama Stack with a vector database provider enabled, start the server by using a configuration similar to one of the following examples:

   • Using inline Milvus:

     MILVUS_URL=milvus uv run llama stack run run.yaml

   • Using remote PostgreSQL with pgvector:

     ENABLE_PGVECTOR=true PGVECTOR_DB=pgvector PGVECTOR_USER=<user> PGVECTOR_PASSWORD=<password> uv run llama stack run run.yaml

2. Optional: Register additional embedding models The default supported embedding models are granite-embedding-30m and granite-embedding-125m, served by the sentence-transformers framework. If you want to benchmark additional embedding models, register them with Llama Stack before running the benchmark script.

   For example, register an embedding model by using the Llama Stack client:

   llama-stack-client models register all-MiniLM-L6-v2 \
     --provider-id sentence-transformers \
     --provider-model-id all-minilm:latest \
     --metadata {"embedding_dimension": 384} \
     --model-type embedding

   Note

   Any embedding models specified in the --embedding-models option must be registered before running the benchmark script.

3. Run the beir_benchmarks.py benchmarking script.

   • Enter the following command to use the configuration from run.yaml and the default dataset scifact with inline Milvus:

     MILVUS_URL=milvus uv run python beir_benchmarks.py

   • Enter the following command to run the benchmark by using remote PostgreSQL with pgvector:

     ENABLE_PGVECTOR=true PGVECTOR_DB=pgvector uv run python beir_benchmarks.py \
       --vector-db-provider-id pgvector

   • Alternatively, enter the following command to connect to a custom Llama Stack server:

     LLAMA_STACK_URL="http://localhost:8321" MILVUS_URL=milvus uv run python beir_benchmarks.py

4. Use environment variables and command line options to modify the benchmark run. For example, set the environment variable for the vector database provider before executing the script.

   • Enter the following command to use a larger batch size for document ingestion:

     MILVUS_URL=milvus uv run python beir_benchmarks.py --batch-size 300

   • Enter the following command to benchmark multiple datasets, for example, scifact and scidocs:

     MILVUS_URL=milvus uv run python beir_benchmarks.py \
       --dataset-names scifact scidocs

   • Enter the following command to compare embedding models, for example, granite-embedding-30m and all-MiniLM-L6-v2:

     MILVUS_URL=milvus uv run python beir_benchmarks.py \
       --embedding-models granite-embedding-30m all-MiniLM-L6-v2

     Note

     Ensure that all-MiniLM-L6-v2 is registered with Llama Stack before running this command. See step 2 for registration instructions.

   • Enter the following command to use a custom BEIR compatible dataset:

     MILVUS_URL=milvus uv run python beir_benchmarks.py \
       --dataset-names my-dataset \
       --custom-datasets-urls https://example.com/my-beir-dataset.zip

   • Enter the following command to change the vector database provider:

     # Use remote PostgreSQL with pgvector
     ENABLE_PGVECTOR=true PGVECTOR_DB=llama-stack PGVECTOR_USER=<user> PGVECTOR_PASSWORD=<password> uv run python beir_benchmarks.py \
       --vector-db-provider-id pgvector

1. Locate the results directory. All output files are saved to the following path:

   <path-to>/rag/benchmarks/beir-benchmarks/results

2. Examine the output. Compare your results with the sample output structure. The report includes performance metrics such as map@cut_k and ndcg@cut_k for each dataset and embedding model pair. The script also calculates a statistical significance test called a p value.

   Example output for scifact and map_cut_10:

   scifact map_cut_10
    granite-embedding-125m : 0.6879
    granite-embedding-30m  : 0.6578
    p_value                : 0.0150
   
    p_value < 0.05 indicates a statistically significant difference.
    The granite-embedding-125m model performs better for this dataset and metric.

3. Interpret the results. A p value below 0.05 indicates that the performance difference between models is statistically significant. The model with the higher metric value performs better. Use these results to identify which embedding model performs best for your dataset.

Procedure

1. Log in to the OpenShift web console.
2. From the Project list, select the project where you want to deploy Qdrant.
3. Click Import YAML.

4. Paste the following YAML:

   Important

   This example deploys a standalone Qdrant service for development or evaluation. For production deployments, review and adapt the configuration to meet your organization’s security, availability, backup, and lifecycle requirements.

   apiVersion: v1
   kind: Secret
   metadata:
     name: <qdrant_credentials_secret>
   type: Opaque
   stringData:
     QDRANT_API_KEY: "<api_key>"
   
   ---
   apiVersion: v1
   kind: PersistentVolumeClaim
   metadata:
     name: <qdrant_pvc>
   spec:
     accessModes:
     - ReadWriteOnce
     resources:
       requests:
         storage: <storage_size>
   
   ---
   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: <qdrant_deployment>
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: <qdrant_app_label>
     template:
       metadata:
         labels:
           app: <qdrant_app_label>
       spec:
         containers:
         - name: qdrant
           image: qdrant/qdrant:v1.12.0
           ports:
           - name: http
             containerPort: 6333
           - name: grpc
             containerPort: 6334
           env:
           - name: QDRANT__SERVICE__API_KEY
             valueFrom:
               secretKeyRef:
                 name: <qdrant_credentials_secret>
                 key: QDRANT_API_KEY
           volumeMounts:
           - name: qdrant-storage
             mountPath: /qdrant/storage
           - name: qdrant-storage
             mountPath: /qdrant/snapshots
             subPath: snapshots
           readinessProbe:
             httpGet:
               path: /readyz
               port: 6333
             initialDelaySeconds: 5
             periodSeconds: 10
           livenessProbe:
             httpGet:
               path: /healthz
               port: 6333
             initialDelaySeconds: 10
             periodSeconds: 20
         volumes:
         - name: qdrant-storage
           persistentVolumeClaim:
             claimName: <qdrant_pvc>
   
   ---
   apiVersion: v1
   kind: Service
   metadata:
     name: <qdrant_service>
   spec:
     selector:
       app: <qdrant_app_label>
     ports:
     - name: http
       port: 6333
       targetPort: 6333
     - name: grpc
       port: 6334
       targetPort: 6334
     type: ClusterIP

   Note

   If your Qdrant instance does not require authentication, remove the Secret and the QDRANT__SERVICE__API_KEY environment variable from the Deployment configuration.

5. Replace the placeholder values as follows:

   • <qdrant_credentials_secret>: A name for the Secret that stores the Qdrant API key, for example qdrant-credentials.
   • <api_key>: An API key for authenticating with Qdrant. If authentication is not required, remove the Secret and the QDRANT__SERVICE__API_KEY environment variable from the Deployment.
   • <qdrant_pvc>: A name for the PersistentVolumeClaim, for example qdrant-pvc.
   • <storage_size>: The storage capacity to request, for example 10Gi.
   • <qdrant_deployment>: A name for the Deployment, for example qdrant.
   • <qdrant_app_label>: A label for the application, for example qdrant.
   • <qdrant_service>: A name for the Service, for example qdrant-service.
6. Click Create.

Procedure

1. In the OpenShift web console, switch to the Administrator perspective.

2. Create a Secret that stores the Qdrant connection details used by Llama Stack. This Secret must contain the URL of the Qdrant service and, if required, the API key.

   Note

   If you deployed Qdrant by using the procedure in Deploying a Qdrant vector database, create this Secret separately for the Llama Stack configuration. The Secret created during the Qdrant deployment does not contain the QDRANT_URL value required by the Llama Stack provider.

   1. From the Project list, select the project where the LlamaStackDistribution resource is deployed.
   2. Click Workloads Secrets.
   3. Click Create From YAML.

   4. Paste the following YAML:

      apiVersion: v1
      kind: Secret
      metadata:
        name: qdrant-connection
      type: Opaque
      stringData:
        QDRANT_URL: "<qdrant_url>"
        QDRANT_API_KEY: "<api_key>"

   5. Replace the placeholder values as follows:

      • <qdrant_url>: The full URL to the Qdrant service, for example http://qdrant-service:6333. For in-cluster deployments, use the Service name and port. For external deployments, use the external URL.
      • <api_key>: The API key for authenticating with Qdrant. If authentication is not enabled for your Qdrant instance, remove the QDRANT_API_KEY entry from both the Secret and the env section in the LlamaStackDistribution configuration.
   6. Click Create.

3. Update your LlamaStackDistribution custom resource to reference the Secret and supply the required environment variables.

   1. Click Operators Installed Operators.
   2. Select the Llama Stack Operator.
   3. Click the LlamaStackDistribution tab.
   4. Select your LlamaStackDistribution resource.
   5. Click YAML.

   6. Update the resource to include the following fields.

      Note

      The environment variable names and configuration fields used by the Qdrant provider can vary depending on the Llama Stack version included with OpenShift AI. Before applying this configuration, verify that the variables and fields match the supported versions listed in Supported Configurations for 3.x.

      apiVersion: llamastack.io/v1alpha1
      kind: LlamaStackDistribution
      metadata:
        name: llamastack
      spec:
        server:
          containerSpec:
            env:
              - name: ENABLE_QDRANT
                value: "true"
              - name: QDRANT_URL
                valueFrom:
                  secretKeyRef:
                    name: qdrant-connection
                    key: QDRANT_URL
              - name: QDRANT_API_KEY
                valueFrom:
                  secretKeyRef:
                    name: qdrant-connection
                    key: QDRANT_API_KEY

   7. Click Save.

Procedure

1. Determine how you will access the Llama Stack API.

   You can access the API from within the cluster or from outside the cluster.

   • In-cluster access: Run the curl commands from a pod in the same project, or from a workstation that has network access to the Llama Stack Service.

   • External access: Expose the Llama Stack Service by creating a Route, and then use the Route URL from your local workstation.

     For this procedure, set LLAMA_STACK_URL to the service or route root URL without the /v1 suffix. The example commands append /v1 as part of the endpoint path.

     For more information about API compatibility and base URL requirements, see

     OpenAI compatibility for RAG APIs in Llama Stack.

     Example base URL for in-cluster access

     LLAMA_STACK_URL="http://llamastack-service:8321"

     Example base URL for external access through a Route

     LLAMA_STACK_URL="https://llamastack-route.example.com"

2. Create a vector store and capture its ID.

   CREATE_RESPONSE=$(curl -s -X POST "${LLAMA_STACK_URL}/v1/vector_stores" \
     -H "Content-Type: application/json" \
     -d '{
       "name": "my-rag-store",
       "embedding_model": "sentence-transformers/ibm-granite/granite-embedding-125m-english",
       "embedding_dimension": 768,
       "provider_id": "qdrant-remote"
     }')
   
   VECTOR_STORE_ID=$(echo "$CREATE_RESPONSE" | jq -r '.id')
   echo "Vector store ID: ${VECTOR_STORE_ID}"

   Ensure that the VECTOR_STORE_ID variable contains a valid value before continuing.