Home
Products
Red Hat OpenShift AI Self-Managed
3.5
Deploy models using Distributed Inference with llm-d
Chapter 4. Enabling authentication and authorization for an LLM inference service

Chapter 4. Enabling authentication and authorization for an LLM inference service

In OpenShift AI 3.0 and later, authentication and authorization are automatically enabled for LLMInferenceService resources when Red Hat Connectivity Link is configured. You can use the security.opendatahub.io/enable-auth: "true" annotation to explicitly enable authentication, such as re-enabling it after it was previously disabled.

Prerequisites

You have configured Red Hat Connectivity Link for Distributed Inference with llm-d as described in Configuring authentication for Distributed Inference with llm-d using Red Hat Connectivity Link.
You have created an LLMInferenceService resource as described in Enabling Distributed Inference with llm-d.
You have access to the OpenShift CLI (oc).

Procedure

By default, authentication is enabled automatically. To explicitly enable authentication or to re-enable it after disabling, annotate your LLMInferenceService resource:

apiVersion: serving.kserve.io/v1alpha1
kind: LLMInferenceService
metadata:
  name: sample-llm-inference-service
  annotations:
    security.opendatahub.io/enable-auth: "true"
spec:
  replicas: 2
  model:
    uri: hf://RedHatAI/Qwen3-8B-FP8-dynamic
    name: RedHatAI/Qwen3-8B-FP8-dynamic
  router:
    route: {}
    gateway: {}
    scheduler: {}
    template:
      containers:
      - name: main
        resources:
          limits:
            cpu: '4'
            memory: 32Gi
            nvidia.com/gpu: "1"
          requests:
            cpu: '2'
            memory: 16Gi
            nvidia.com/gpu: "1"

Apply the configuration:

oc apply -f <llm-inference-service-file>.yaml

Verification

Confirm that the LLMInferenceService resource has the annotation:

oc get llminferenceservice sample-llm-inference-service -o jsonpath='{.metadata.annotations.security\.opendatahub\.io/enable-auth}'

The command returns true.

Verify that the inference service is protected by attempting to access it without authentication:
```
curl -v https://<inference-endpoint-url>/v1/models
```
The request returns a 401 Unauthorized response, confirming that unauthenticated requests are rejected.

4.1. Making authenticated inference requests to Distributed Inference with llm-d
Copy link

When you enable authentication for an LLMInferenceService, you must include a valid JSON web token (JWT) in your inference requests. You can generate a token from a ServiceAccount or use an OIDC token from an identity provider.

Prerequisites

You have enabled authentication for your LLMInferenceService as described in Enabling authentication and authorization for an LLM inference service.
You have access to the OpenShift CLI (oc).
You have the inference endpoint URL for your deployed model.

Procedure

Create a ServiceAccount with permissions to access the LLMInferenceService:
```
oc create serviceaccount llm-user -n <namespace>
```

Create a Role that grants permission to get the LLMInferenceService:

oc apply -f - <<EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: llm-inference-viewer
  namespace: <namespace>
rules:
- apiGroups: ["serving.kserve.io"]
  resources: ["llminferenceservices"]
  verbs: ["get"]
  resourceNames: ["<llm-inference-service-name>"]
EOF

Bind the Role to the ServiceAccount:

oc create rolebinding llm-user-binding \
  --role=llm-inference-viewer \
  --serviceaccount=<namespace>:llm-user \
  -n <namespace>

Generate a JWT token from the ServiceAccount:
```
TOKEN=$(oc create token llm-user -n <namespace> --duration=1h)
```
The --duration flag sets the token validity period. Adjust this value based on your workload requirements. For production use, consider using a shorter duration or integrating with an OIDC provider.

Verification

Verify that the authenticated request was successful:

curl -v https://<inference-endpoint-url>/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer ${TOKEN}" \
  -d '{
    "model": "<model-name>",
    "messages": [{
      "role": "user",
      "content": "What is Red Hat OpenShift AI?"
    }]
  }'

A successful response indicates that authentication is working correctly.

Verify that requests without authentication are rejected:

curl -v https://<inference-endpoint-url>/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "<model-name>",
    "messages": [{
      "role": "user",
      "content": "What is Red Hat OpenShift AI?"
    }]
  }'

The request returns a 401 Unauthorized response with a message indicating that authentication is required.

Verify that requests with an invalid or expired token are rejected:

curl -v https://<inference-endpoint-url>/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer invalid-token" \
  -d '{
    "model": "<model-name>",
    "messages": [{
      "role": "user",
      "content": "What is Red Hat OpenShift AI?"
    }]
  }'

The request returns a 401 Unauthorized response, confirming that invalid tokens are rejected.

Chapter 4. Enabling authentication and authorization for an LLM inference service

4.1. Making authenticated inference requests to Distributed Inference with llm-d
Copy link

Learn

Try, buy, & sell

Communities

About Red Hat

Making open source more inclusive

About Red Hat Documentation

Theme

Red Hat legal and privacy links

Red Hat legal and privacy links

Chapter 4. Enabling authentication and authorization for an LLM inference service

4.1. Making authenticated inference requests to Distributed Inference with llm-dCopy linkLink copied to clipboard!

Learn

Try, buy, & sell

Communities

About Red Hat

Making open source more inclusive

About Red Hat Documentation

Theme

Red Hat legal and privacy links

Red Hat legal and privacy links

4.1. Making authenticated inference requests to Distributed Inference with llm-d
Copy link