Chapter 9. Configure token-based rate limiting with TokenRateLimitPolicy

1. On an incoming request, the gateway evaluates the matching rules and predicates from the TokenRateLimitPolicy resources.
2. If the request matches, the gateway prepares the necessary rate limit descriptors and monitors the response.
3. After receiving the response, the gateway extracts the usage.total_tokens field from the JSON response body.
4. The gateway then sends a RateLimitRequest to Limitador, including the actual token count as a hits_addend.
5. Limitador tracks the cumulative token usage and responds to the gateway with OK or OVER_LIMIT.

Procedure

1. Create a TokenRateLimitPolicy resource. This example defines two limits: one for free users on a 10,000 tokens per day request limit, and one for pro users with a 100,000 tokens per day request limit.

   apiVersion: kuadrant.io/v1alpha1
   kind: TokenRateLimitPolicy
   metadata:
     name: llm-protection
   spec:
     targetRef:
       group: gateway.networking.k8s.io
       kind: Gateway
       name: ai-gateway
     limits:
       free-users:
         rates:
           - limit: 10000 # 10k tokens per day for free tier
             window: 24h
         when:
           - predicate: request.path == "/v1/chat/completions" # Inference traffic only
           - predicate: |
               auth.identity.groups.split(",").exists(g, g == "free")
         counters:
           - expression: auth.identity.userid
       pro-users:
         rates:
           - limit: 100000 # 200 tokens per minute for pro users
             window: 24h
         when:
           - predicate: request.path == "/v1/chat/completions" # Inference traffic only
           - predicate: |
               auth.identity.groups.split(",").exists(g, g == "pro")
         counters:
           - expression: auth.identity.userid

   Copy to Clipboard Toggle word wrap

2. Apply the policy:

   oc apply -f your-tokenratelimitpolicy.yaml -n my-api-namespace

   Copy to Clipboard Toggle word wrap

3. Check the status of the policy to ensure it has been accepted and enforced on the target HTTPRoute. Look for conditions with type: Accepted and type: Enforced with status: "True".

   oc get tokenratelimitpolicy llm-protection -n my-api-namespace -o jsonpath='{.status.conditions}'

   Copy to Clipboard Toggle word wrap

4. Send requests to your API endpoint, including the required authentication details.

   curl -H "Authorization: <auth-details>" \
        -d '{"model": "gpt-4", "messages": [{"role": "user", "content": "Hello"}]}' \
        <your-api-endpoint>

   Copy to Clipboard Toggle word wrap