Chapter 2. Use models-as-a-service

Procedure

1. In the OpenShift AI dashboard, click Gen AI studio AI asset endpoints.
2. On the Models tab, locate a model published to Models-as-a-Service (MaaS) and click View in the Endpoints column.

3. In the Endpoints dialog, view your subscription assignment from the Subscription dropdown.

   If you belong to multiple subscriptions that include this model, you can select which subscription to use for the model. The selected subscription’s token limits apply to your inference requests.

   Tip

   If you frequently exceed token limits or need access to additional models, contact your administrator to request a subscription update.

Procedure

1. In the OpenShift AI dashboard, click Gen AI studio AI asset endpoints.
2. On the Models tab, locate the model you want to access and click View in the Endpoints column.
3. In the Endpoints dialog, verify that the model displays a Model as a Service badge at the top.
4. Copy the external API endpoint URL and store it securely. This URL is required for API calls to the model.
5. Under the Authentication section, select your subscription from the Subscription dropdown.
6. Click Generate API key.

7. Copy the generated API key immediately and store it securely.

   Important

   The temporary API key is displayed only once and expires after 1 hour. Temporary API keys generated from the Endpoints dialog are scoped to the selected subscription. To create API keys with custom expiration dates, navigate to Gen AI studio API keys in the OpenShift AI dashboard.

8. Click Close.

Procedure

1. Set your API key as an environment variable:

   $ export MAAS_API_KEY="<your_api_key>"

   The command uses the following placeholder:

   <your_api_key>

   Specifies the API key you generated or created, with the sk-oai- prefix.

2. Set the external API endpoint URL as an environment variable:

   $ export MAAS_URL="<external_api_endpoint>"

   The command uses the following placeholder:

   <external_api_endpoint>

   Specifies the external API endpoint URL you copied from the Endpoints dialog.

3. List available models using the /v1/models endpoint:

   $ curl -X GET "${MAAS_URL}/v1/models" \
     -H "Authorization: Bearer ${MAAS_API_KEY}"

   Example response in OpenAI-compatible format:

   {
     "object": "list",
     "data": [
       {
         "id": "facebook-opt-125m",
         "object": "model",
         "created": 1234567890,
         "owned_by": "llm",
         "ready": true
       },
       {
         "id": "llama-2-7b-chat",
         "object": "model",
         "created": 1234567890,
         "owned_by": "llm",
         "ready": true
       }
     ]
   }

4. Call a model using the chat completions endpoint:

   $ curl -X POST "${MAAS_URL}/llm/<model_name>/v1/chat/completions" \
     -H "Authorization: Bearer ${MAAS_API_KEY}" \
     -H "Content-Type: application/json" \
     -d { "model": "<model_name>", "messages": [ { "role": "user", "content": "Explain quantum computing in simple terms." } ], "max_tokens": 150, "temperature": 0.7 }

   where:

   <model_name>

   Specifies the model deployment name from the Models as a service page, such as facebook-opt-125m or llama-2-7b-chat.

   Example response:

   {
     "id": "chatcmpl-abc123",
     "object": "chat.completion",
     "created": 1234567890,
     "model": "llama-2-7b-chat",
     "choices": [
       {
         "index": 0,
         "message": {
           "role": "assistant",
           "content": "Quantum computing is a type of computing that uses quantum mechanics..."
         },
         "finish_reason": "stop"
       }
     ],
     "usage": {
       "prompt_tokens": 12,
       "completion_tokens": 45,
       "total_tokens": 57
     }
   }

Procedure

1. Create a new Jupyter notebook in your OpenShift AI workbench.

2. In the first code cell, configure the Models-as-a-Service (MaaS) gateway URL and API key:

   DEMO_MAAS_BASE = "https://maas.example.com"
   DEMO_API_KEY = "your-api-key-here"

   The code uses the following variables:

   DEMO_MAAS_BASE

   Specifies the MaaS gateway base URL. Extract this from the external API endpoint URL you copied from the Endpoints dialog. For example, if the endpoint is \https://maas.apps.cluster.example.com/llm/facebook-opt-125m, use \https://maas.apps.cluster.example.com. The specific model URLs are retrieved from the API response in the next step.

   DEMO_API_KEY

   Specifies your API key. Use the temporary or permanent key you generated earlier.

3. In a new code cell, add the setup code to configure the HTTP client:

   import json
   import os
   import ssl
   import urllib.error
   import urllib.request
   from typing import Any, Dict, Optional
   
   _mb = globals().get("DEMO_MAAS_BASE", "")
   if isinstance(_mb, str) and _mb.strip():
       MAAS_BASE = _mb.strip().rstrip("/")
   else:
       MAAS_BASE = os.environ.get("MAAS_BASE", "https://maas.YOUR_DOMAIN_HERE").strip().rstrip("/")
   
   _ak = globals().get("DEMO_API_KEY", "")
   if isinstance(_ak, str) and _ak.strip():
       API_KEY = _ak.strip()
   else:
       API_KEY = (os.environ.get("MAAS_API_KEY") or os.environ.get("API_KEY") or "").strip()
   
   VERIFY_TLS = os.environ.get("VERIFY_TLS", "").lower() in ("1", "true", "yes")
   MODELS_URL = f"{MAAS_BASE}/maas-api/v1/models"
   
   if not API_KEY:
       raise SystemExit(
           "Set DEMO_API_KEY in the quick-swap cell or MAAS_API_KEY / API_KEY in the environment."
       )
   
   
   def http_json(
       method: str,
       url: str,
       *,
       token: Optional[str] = None,
       data: Optional[Dict[str, Any]] = None,
   ):
       """Minimal JSON HTTP helper (stdlib only)."""
       headers = {"Content-Type": "application/json", "Accept": "application/json"}
       if token:
           headers["Authorization"] = f"Bearer {token}"
       body = None
       if data is not None:
           body = json.dumps(data).encode("utf-8")
       ctx = ssl.create_default_context()
       if not VERIFY_TLS:
           ctx.check_hostname = False
           ctx.verify_mode = ssl.CERT_NONE
       req = urllib.request.Request(url, data=body, headers=headers, method=method)
       try:
           with urllib.request.urlopen(req, context=ctx, timeout=120) as resp:
               raw = resp.read().decode("utf-8")
               return resp.status, json.loads(raw) if raw else {}
       except urllib.error.HTTPError as e:
           err_body = e.read().decode("utf-8", errors="replace")
           try:
               parsed = json.loads(err_body) if err_body else {}
           except json.JSONDecodeError:
               parsed = {"_raw": err_body}
           raise RuntimeError(f"HTTP {e.code}: {parsed}") from None
   
   
   print("MAAS_BASE :", MAAS_BASE)
   print("MODELS_URL:", MODELS_URL)
   print("VERIFY_TLS:", VERIFY_TLS)
   print("API key set:", bool(API_KEY))

   This cell defines the http_json helper function and prints the configuration summary.

4. In a new code cell, list the available models:

   _, models_body = http_json("GET", MODELS_URL, token=API_KEY)
   
   data = models_body.get("data") or []
   if not data:
       raise SystemExit("No models in response; deploy a model and check subscription binding.")
   
   first = data[0]
   MODEL_NAME = first.get("id") or first.get("name")
   MODEL_URL = (first.get("url") or "").rstrip("/")
   if not MODEL_NAME or not MODEL_URL:
       raise RuntimeError(f"Could not parse model id/url from: {first}")
   
   print(f"MODEL_NAME: {MODEL_NAME}")
   print(f"MODEL_URL: {MODEL_URL}")
   print("\nAll models:")
   print(json.dumps(models_body, indent=2))

   This cell retrieves the list of models you can access and stores the first model’s name and URL for testing.

5. In a new code cell, make a single inference request:

   COMPLETIONS_URL = f"{MODEL_URL}/v1/completions"
   inference_payload = {
       "model": MODEL_NAME,
       "prompt": "Hello from the notebook demo.",
       "max_tokens": 50,
   }
   
   status, completion = http_json("POST", COMPLETIONS_URL, token=API_KEY, data=inference_payload)
   
   usage = completion.get("usage") or {}
   choices = completion.get("choices") or []
   choice0 = choices[0] if choices and isinstance(choices[0], dict) else {}
   completion_text = choice0.get("text") or ""
   
   print(f"HTTP status: {status}")
   print(f"\nTokens used:")
   print(f"  prompt_tokens: {usage.get('prompt_tokens', '—')}")
   print(f"  completion_tokens: {usage.get('completion_tokens', '—')}")
   print(f"  total_tokens: {usage.get('total_tokens', '—')}")
   print(f"\nCompletion text:")
   print(completion_text if completion_text else "(empty)")

   This cell sends a completion request and displays the response with token usage information.

6. Optional: In a new code cell, test the rate limiting behavior:

   import time
   
   # Configuration
   SLEEP_BETWEEN_REQUESTS_SEC = 1.0
   CONSECUTIVE_429_TO_STOP = 3
   MAX_REQUESTS = 64
   
   inference_payload = {
       "model": MODEL_NAME,
       "prompt": "Rate limit probe.",
       "max_tokens": 50,
   }
   
   ctx = ssl.create_default_context()
   if not VERIFY_TLS:
       ctx.check_hostname = False
       ctx.verify_mode = ssl.CERT_NONE
   
   consecutive_429 = 0
   last_code = None
   for i in range(1, MAX_REQUESTS + 1):
       body = json.dumps(inference_payload).encode("utf-8")
       req = urllib.request.Request(
           COMPLETIONS_URL,
           data=body,
           headers={
               "Authorization": f"Bearer {API_KEY}",
               "Content-Type": "application/json",
           },
           method="POST",
       )
       try:
           with urllib.request.urlopen(req, context=ctx, timeout=120) as resp:
               last_code = resp.status
       except urllib.error.HTTPError as e:
           last_code = e.code
   
       print(f"{i:3d}  HTTP {last_code}", end="")
       if last_code == 429:
           consecutive_429 += 1
           print(f"  (429 streak {consecutive_429}/{CONSECUTIVE_429_TO_STOP})")
           if consecutive_429 >= CONSECUTIVE_429_TO_STOP:
               print(f"Stopping: {CONSECUTIVE_429_TO_STOP} consecutive 429 responses.")
               break
       else:
           consecutive_429 = 0
           print()
   
       if SLEEP_BETWEEN_REQUESTS_SEC > 0 and i < MAX_REQUESTS:
           time.sleep(SLEEP_BETWEEN_REQUESTS_SEC)
   else:
       if consecutive_429 < CONSECUTIVE_429_TO_STOP:
           print(f"Stopped: reached MAX_REQUESTS={MAX_REQUESTS} without {CONSECUTIVE_429_TO_STOP} consecutive 429s.")

   This cell sends repeated requests to test rate limiting. The loop stops after receiving 3 consecutive HTTP 429 responses or reaching the maximum request count.

Procedure

1. From the OpenShift AI dashboard, click Gen AI studio AI asset endpoints.
2. Locate the MaaS model that you want to test.

3. In the Actions column for that model, click the action menu (⋮) and then select Try in playground.

   The playground interface opens in a new browser tab.

4. In the Configure panel, verify your MaaS settings:

   1. From the Model dropdown, verify that the correct model is selected.

      MaaS models are displayed in the format <endpoint-name>/<model-id>. For example, maas-vllm-inference-1/facebook/opt-125m.

   2. From the Subscription dropdown, select the subscription to use for testing.

      If you belong to multiple subscriptions, select the subscription you want to use for testing.

5. Test the model by entering a prompt in the message field and then clicking Send.

   The model response appears in the chat interface.

6. Optional: Adjust model parameters, configure system prompts, or use additional playground features.

   For information about playground features such as temperature settings, streaming responses, system prompts, model comparison, RAG integration, and code export, see Experimenting with models in the gen AI playground.