Chapter 8. Process batch inference jobs with Distributed Inference with llm-d

Procedure

1. Verify that the batch gateway component is enabled in the DataScienceCluster resource:

   $ oc get datasciencecluster default-dsc -o yaml

   Expected output:

   spec:
     components:
       aigateway:
         managementState: Managed
         batchGateway:
           managementState: Managed

   If the batch gateway component is not enabled, edit the DataScienceCluster resource:

   $ oc edit datasciencecluster default-dsc

   Add or update the following configuration:

   spec:
     components:
       aigateway:
         managementState: Managed
         batchGateway:
           managementState: Managed

   Save the changes and wait for the DataScienceCluster to reconcile. The operator creates the batch gateway namespace and deploys the Batch Gateway Operator.

2. Create an internal ClusterIP gateway for batch processor inference routing.

   The batch processor routes inference requests through this internal gateway to enforce model-level authorization while bypassing per-user token rate limits on the external gateway.

   Save the following YAML to a file named batch-internal-gateway.yaml:

   apiVersion: gateway.networking.k8s.io/v1
   kind: Gateway
   metadata:
     name: batch-internal-gateway
     namespace: openshift-ingress
     annotations:
       networking.istio.io/service-type: ClusterIP
   spec:
     gatewayClassName: openshift-default
     listeners:
     - name: http
       port: 80
       protocol: HTTP
       allowedRoutes:
         namespaces:
           from: Selector
           selector:
             matchLabels:
               llm-d.ai/gateway-route: "true"

   Apply the Gateway resource:

   $ oc apply -f batch-internal-gateway.yaml

   Important

   The networking.istio.io/service-type: ClusterIP annotation ensures the gateway is not externally accessible. The internal gateway uses HTTP only because traffic is cluster-internal.

3. Create an HTTPRoute to route batch processor inference requests to the model server through the internal gateway.

   The batch processor rewrites the request path to match the model server’s API and includes the user’s original authentication token.

   Save the following YAML to a file named batch-llm-route.yaml:

   apiVersion: gateway.networking.k8s.io/v1
   kind: HTTPRoute
   metadata:
     name: batch-llm-route
     namespace: <llmisvc_namespace>
   spec:
     parentRefs:
     - name: batch-internal-gateway
       namespace: openshift-ingress
     rules:
     - matches:
       - path:
           type: PathPrefix
           value: /<llmisvc_namespace>/<llmisvc_name>/v1/chat/completions
       filters:
       - type: URLRewrite
         urlRewrite:
           path:
             type: ReplacePrefixMatch
             replacePrefixMatch: /v1/chat/completions
       backendRefs:
       - group: inference.networking.k8s.io
         kind: InferencePool
         name: <inference_pool_name>
         port: 80
     - matches:
       - path:
           type: PathPrefix
           value: /<llmisvc_namespace>/<llmisvc_name>/v1/completions
       filters:
       - type: URLRewrite
         urlRewrite:
           path:
             type: ReplacePrefixMatch
             replacePrefixMatch: /v1/completions
       backendRefs:
       - group: inference.networking.k8s.io
         kind: InferencePool
         name: <inference_pool_name>
         port: 80
     - matches:
       - path:
           type: PathPrefix
           value: /<llmisvc_namespace>/<llmisvc_name>/v1/embeddings
       filters:
       - type: URLRewrite
         urlRewrite:
           path:
             type: ReplacePrefixMatch
             replacePrefixMatch: /v1/embeddings
       backendRefs:
       - group: inference.networking.k8s.io
         kind: InferencePool
         name: <inference_pool_name>
         port: 80
     - matches:
       - path:
           type: PathPrefix
           value: /<llmisvc_namespace>/<llmisvc_name>/v1/responses
       filters:
       - type: URLRewrite
         urlRewrite:
           path:
             type: ReplacePrefixMatch
             replacePrefixMatch: /v1/responses
       backendRefs:
       - group: inference.networking.k8s.io
         kind: InferencePool
         name: <inference_pool_name>
         port: 80
     - matches:
       - path:
           type: PathPrefix
           value: /<llmisvc_namespace>/<llmisvc_name>/v1/moderations
       filters:
       - type: URLRewrite
         urlRewrite:
           path:
             type: ReplacePrefixMatch
             replacePrefixMatch: /v1/moderations
       backendRefs:
       - group: inference.networking.k8s.io
         kind: InferencePool
         name: <inference_pool_name>
         port: 80

   where:

   <llmisvc_namespace>

   Specifies the namespace where the LLMInferenceService is deployed.

   <llmisvc_name>

   Specifies the name of the LLMInferenceService resource.

   <inference_pool_name>

   Specifies the name of the InferencePool resource created by the LLMInferenceService.

4. Apply the HTTPRoute resource:

   $ oc apply -f batch-llm-route.yaml

5. Create an AuthPolicy for the batch-llm-route to enforce model-level authorization:

   Save the following YAML to a file named batch-llm-authpolicy.yaml:

   apiVersion: kuadrant.io/v1
   kind: AuthPolicy
   metadata:
     name: batch-llm-authpolicy
     namespace: <llmisvc_namespace>
   spec:
     targetRef:
       group: gateway.networking.k8s.io
       kind: HTTPRoute
       name: batch-llm-route
     rules:
       authentication:
         "kubernetes-token":
           kubernetesTokenReview:
             audiences:
             - "https://kubernetes.default.svc"
       authorization:
         "llmisvc-access":
           kubernetesSubjectAccessReview:
             user:
               selector: auth.identity.user.username
             resourceAttributes:
               group: serving.kserve.io
               resource: llminferenceservices
               name: <llmisvc_name>
               namespace: <llmisvc_namespace>
               verb: get
       response:
         success:
           headers:
             "X-MaaS-Username":
               plain:
                 selector: auth.identity.user.username

   Apply the AuthPolicy resource:

   $ oc apply -f batch-llm-authpolicy.yaml

   The AuthPolicy validates the user’s Kubernetes token and checks whether the user has permission to get the specific LLMInferenceService resource using SubjectAccessReview.

6. Create a Kubernetes Secret with connection details for Redis, PostgreSQL, and S3-compatible storage.

   Save the following YAML to a file named batch-gateway-secrets.yaml:

   apiVersion: v1
   kind: Secret
   metadata:
     name: batch-gateway-secrets
     namespace: <batch_gateway_namespace>
   type: Opaque
   stringData:
     redis-url: "redis://<redis_host>:<redis_port>/0"
     postgresql-url: "postgresql://<db_user>:<db_password>@<db_host>:<db_port>/<db_name>?sslmode=disable"
     s3-secret-access-key: "<s3_secret_key>"

   where:

   <batch_gateway_namespace>

   Specifies the namespace for the batch gateway deployment.

   <redis_host>

   Specifies the hostname or IP address of the Redis or Valkey instance.

   <redis_port>

   Specifies the port number of the Redis or Valkey instance (default: 6379).

   <db_user>

   Specifies the PostgreSQL username.

   <db_password>

   Specifies the PostgreSQL password.

   <db_host>

   Specifies the hostname or IP address of the PostgreSQL instance.

   <db_port>

   Specifies the port number of the PostgreSQL instance (default: 5432).

   <db_name>

   Specifies the name of the PostgreSQL database for batch metadata.

   <s3_secret_key>

   Specifies the secret access key for S3-compatible storage.

   Note

   S3 configuration settings (region, endpoint, access key ID, and bucket name) are non-sensitive and are configured in the LLMBatchGateway custom resource, not in this secret. Only the S3 secret access key belongs in the secret.

   Apply the Secret:

   $ oc apply -f batch-gateway-secrets.yaml

7. Deploy infrastructure dependencies for the batch gateway.

   The batch gateway requires Redis or Valkey for job queuing, PostgreSQL for metadata storage, and S3-compatible storage for batch files. Deploy these services before creating the LLMBatchGateway custom resource.

   For production deployments, use managed services or deploy these components with appropriate persistence, backups, and high availability. For development or testing environments, you can deploy minimal instances using Helm charts or Kubernetes manifests.

   Note

   For example deployment manifests for Redis, PostgreSQL, and MinIO, see the batch gateway RHOAI deployment guide.

8. Create an LLMBatchGateway custom resource to deploy the batch gateway components.

   Save the following YAML to a file named llm-batch-gateway.yaml:

   apiVersion: llm-d.ai/v1alpha1
   kind: LLMBatchGateway
   metadata:
     name: batch-gateway
     namespace: <batch_gateway_namespace>
   spec:
     secretRef:
       name: batch-gateway-secrets
     dbBackend: postgresql
     fileStorage:
       s3:
         region: <s3_region>
         endpoint: <s3_endpoint>
         accessKeyId: <s3_access_key_id>
         prefix: <s3_bucket_name>
         usePathStyle: true
         autoCreateBucket: true
     apiServer:
       replicas: 1
     processor:
       replicas: 1
       globalInferenceGateway:
         url: http://batch-internal-gateway.openshift-ingress.svc.cluster.local/<llmisvc_namespace>/<llmisvc_name>/v1
         requestTimeout: 5m
         maxRetries: 3
         initialBackoff: 1s
         maxBackoff: 60s
       config:
         inferenceObjective: batch-workload
     gc:
       interval: 30m
     tls:
       enabled: true
       certManager:
         issuerName: selfsigned-issuer
         issuerKind: ClusterIssuer
         dnsNames:
         - batch-gateway-apiserver
         - batch-gateway-apiserver.<batch_gateway_namespace>.svc.cluster.local
         - localhost

   where:

   <batch_gateway_namespace>

   Specifies the namespace where the batch gateway is deployed.

   <s3_region>

   Specifies the S3 region (for example, us-east-1).

   <s3_endpoint>

   Specifies the S3 endpoint URL. For MinIO deployed in-cluster, use http://minio.<batch_gateway_namespace>.svc.cluster.local:9000.

   <s3_access_key_id>

   Specifies the S3 access key ID.

   <s3_bucket_name>

   Specifies the S3 bucket name for batch input and output files.

   <llmisvc_namespace>

   Specifies the namespace of the LLMInferenceService.

   <llmisvc_name>

   Specifies the name of the LLMInferenceService resource.

   Apply the LLMBatchGateway resource:

   $ oc apply -f llm-batch-gateway.yaml

   The Batch Gateway Operator deploys the batch-gateway-apiserver, batch-processor, and garbage-collector components. Component images are pinned by the operator from its deployment configuration.

   Note

   To use filesystem storage instead of S3, replace the fileStorage.s3 section with:

   fileStorage:
     fs:
       basePath: /tmp/batch-gateway
       claimName: <your-pvc-name>

   The PersistentVolumeClaim must have ReadWriteMany access mode.

9. Create an HTTPRoute to route external batch API traffic to the batch-gateway-apiserver.

   Save the following YAML to a file named batch-api-route.yaml:

   apiVersion: gateway.networking.k8s.io/v1
   kind: HTTPRoute
   metadata:
     name: batch-api-route
     namespace: <batch_gateway_namespace>
   spec:
     parentRefs:
     - name: <external_gateway_name>
       namespace: <gateway_namespace>
     rules:
     - matches:
       - path:
           type: PathPrefix
           value: /v1/batches
       - path:
           type: PathPrefix
           value: /v1/files
       backendRefs:
       - name: batch-gateway-apiserver
         port: 8000

   where:

   <external_gateway_name>

   Specifies the name of the external Kubernetes Gateway resource that receives incoming batch API requests.

   <gateway_namespace>

   Specifies the namespace of the external gateway (typically openshift-ingress).

   Apply the HTTPRoute resource:

   $ oc apply -f batch-api-route.yaml

   Apply an AuthPolicy for the batch API routes.

   Save the following YAML to a file named batch-api-authpolicy.yaml:

   apiVersion: kuadrant.io/v1
   kind: AuthPolicy
   metadata:
     name: batch-api-authpolicy
     namespace: <batch_gateway_namespace>
   spec:
     targetRef:
       group: gateway.networking.k8s.io
       kind: HTTPRoute
       name: batch-api-route
     rules:
       authentication:
         "kubernetes-token":
           kubernetesTokenReview:
             audiences:
             - "https://kubernetes.default.svc"
       response:
         success:
           headers:
             "X-MaaS-Username":
               plain:
                 selector: auth.identity.user.username

   Apply the AuthPolicy resource:

   $ oc apply -f batch-api-authpolicy.yaml

   The batch API AuthPolicy performs authentication only. Model-level authorization is enforced by the batch-llm-authpolicy on the internal gateway when the processor forwards inference requests.

10. Apply a RateLimitPolicy to enforce per-user rate limits on the batch API.

    Save the following YAML to a file named batch-rate-limit-policy.yaml:

    apiVersion: kuadrant.io/v1
    kind: RateLimitPolicy
    metadata:
      name: batch-rate-limit
      namespace: <batch_gateway_namespace>
    spec:
      targetRef:
        group: gateway.networking.k8s.io
        kind: HTTPRoute
        name: batch-api-route
      limits:
        "per-user":
          rates:
          - limit: 20
            window: 1m
          counters:
          - auth.identity.user.username

    Apply the RateLimitPolicy resource:

    $ oc apply -f batch-rate-limit-policy.yaml

11. Create an InferenceObjective at priority -1 for batch workloads.

    Save the following YAML to a file named batch-inference-objective.yaml:

    apiVersion: llm-d.ai/v1alpha2
    kind: InferenceObjective
    metadata:
      name: batch-workload
      namespace: <llmisvc_namespace>
    spec:
      priority: -1
      poolRef:
        group: inference.networking.k8s.io
        kind: InferencePool
        name: <inference_pool_name>

    where:

    <llmisvc_namespace>

    Specifies the namespace where the InferencePool is deployed.

    <inference_pool_name>

    Specifies the name of the InferencePool resource for the target model.

    Apply the InferenceObjective resource:

    $ oc apply -f batch-inference-objective.yaml

    The negative priority value (priority: -1) means batch requests run at lower priority than interactive workloads and are the first to be dropped when the system reaches saturation, protecting real-time inference SLOs.

Verification

1. Verify that the batch gateway pods are running:

   $ oc get pods -n <batch_gateway_namespace>

   The output shows pods for the batch-gateway-apiserver, batch-gateway-processor, and batch-gateway-gc components in Running status.

2. Submit a test batch job to verify the subsystem is functioning:

   1. Create a test JSONL input file:

      $ cat > /tmp/test-batch.jsonl << 'EOF'
      {"custom_id": "test-1", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "<model_name>", "messages": [{"role": "user", "content": "Hello"}], "max_tokens": 10}}
      EOF

   2. Upload the file:

      $ curl -s -X POST "https://<batch_gateway_url>/v1/files" \
        -H "Authorization: Bearer <auth_token>" \
        -F "file=@/tmp/test-batch.jsonl" \
        -F "purpose=batch"

   3. Create a batch job by using the returned file_id:

      $ curl -s -X POST "https://<batch_gateway_url>/v1/batches" \
        -H "Authorization: Bearer <auth_token>" \
        -H "Content-Type: application/json" \
        -d '{
          "input_file_id": "<file_id>",
          "endpoint": "/v1/chat/completions",
          "completion_window": "24h"
        }'

   4. Poll the batch status until it reaches completed:

      $ curl -s "https://<batch_gateway_url>/v1/batches/<batch_id>" \
        -H "Authorization: Bearer <auth_token>"

      A successful test batch job transitions through validating, in_progress, finalizing, and completed states.

Procedure

1. Prepare a JSONL input file with one inference request per line.

   Each line must be a valid JSON object with the following fields:

   {"custom_id": "request-1", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "<model_name>", "messages": [{"role": "user", "content": "Summarize the benefits of container orchestration."}], "max_tokens": 256}}
   {"custom_id": "request-2", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "<model_name>", "messages": [{"role": "user", "content": "Explain Kubernetes networking."}], "max_tokens": 256}}

   where:

   custom_id

   Specifies a unique identifier for correlating this request with its result in the output file.

   method

   Specifies the HTTP method. Must be POST.

   url

   Specifies the inference endpoint path. Supported values are /v1/chat/completions, /v1/completions, /v1/embeddings, /v1/responses, and /v1/moderations.

   body

   Specifies the request body, following the same schema as the corresponding synchronous inference endpoint.

   Note

   This example shows chat completion requests. You can use this same workflow for different use cases such as embedding generation, offline evaluations, or dataset processing by adjusting the content and structure of your JSONL file. For example, use /v1/embeddings for embedding generation or construct evaluation-specific prompts for offline model evaluation.

   Save this file locally, for example as batch-input.jsonl.

2. Upload the input file to the batch gateway:

   $ curl -s -X POST "https://<batch_gateway_url>/v1/files" \
     -H "Authorization: Bearer <auth_token>" \
     -F "file=@batch-input.jsonl" \
     -F "purpose=batch"

   where:

   <batch_gateway_url>

   Specifies the URL of the batch-gateway-apiserver endpoint.

   <auth_token>

   Specifies your Kubernetes ServiceAccount token or user token.

   The response includes a file_id that you use to create the batch job:

   {
     "id": "file-abc123",
     "object": "file",
     "purpose": "batch",
     "filename": "batch-input.jsonl",
     "bytes": 512,
     "created_at": 1719878400
   }

   Note

   For additional file operations such as listing files, retrieving file metadata, or deleting files, see Section 8.4, “Batch inference API reference”.

3. Create a batch job by using the uploaded file:

   $ curl -s -X POST "https://<batch_gateway_url>/v1/batches" \
     -H "Authorization: Bearer <auth_token>" \
     -H "Content-Type: application/json" \
     -d '{
       "input_file_id": "<file_id>",
       "endpoint": "/v1/chat/completions",
       "completion_window": "24h"
     }'

   where:

   <file_id>

   Specifies the id value from the file upload response.

   The response includes a batch_id and shows the initial status as validating:

   {
     "id": "batch-xyz789",
     "object": "batch",
     "endpoint": "/v1/chat/completions",
     "input_file_id": "file-abc123",
     "status": "validating",
     "created_at": 1719878400
   }

   Note

   This example shows required parameters only. For optional parameters such as custom metadata or batch listing and filtering options, see Section 8.4, “Batch inference API reference”.

4. Poll the batch job status until it reaches completed:

   $ curl -s "https://<batch_gateway_url>/v1/batches/<batch_id>" \
     -H "Authorization: Bearer <auth_token>"

   where:

   <batch_id>

   Specifies the id value from the batch creation response.

   The batch job transitions through the following states: validating in_progress finalizing completed. A completed batch response includes output_file_id and request counts:

   {
     "id": "batch-xyz789",
     "object": "batch",
     "status": "completed",
     "input_file_id": "file-abc123",
     "output_file_id": "file-out456",
     "request_counts": {
       "total": 2,
       "completed": 2,
       "failed": 0
     }
   }

5. Retrieve the batch results by downloading the output file:

   $ curl -s "https://<batch_gateway_url>/v1/files/<output_file_id>/content" \
     -H "Authorization: Bearer <auth_token>"

   where:

   <output_file_id>

   Specifies the output_file_id value from the completed batch response.

   The output is a JSONL file with one result per line, correlated by custom_id:

   {"id": "response-1", "custom_id": "request-1", "response": {"status_code": 200, "body": {"choices": [{"message": {"role": "assistant", "content": "Container orchestration provides..."}}]}}}
   {"id": "response-2", "custom_id": "request-2", "response": {"status_code": 200, "body": {"choices": [{"message": {"role": "assistant", "content": "Kubernetes networking..."}}]}}}

6. Optional: Cancel a batch job:

   $ curl -s -X POST "https://<batch_gateway_url>/v1/batches/<batch_id>/cancel" \
     -H "Authorization: Bearer <auth_token>"

   You can cancel a batch job in the validating or in_progress state. Jobs still queued in the validating state are cancelled immediately and transition directly to cancelled. Jobs already being processed in the in_progress state transition through cancelling to cancelled as the processor winds down. Requests that were already completed before cancellation are included in the output file.