1.11. About querying your AI model

Procedure

1. Get the IP address of the MicroShift node and assign it to the IP variable as the following example command shows:

   $ IP=$(oc get nodes -o json | jq -r '.items[0].status.addresses[0].address')

2. Identify the name of the route you created by running the following command:

   $ oc get route -n ai-test <route_name> -o yaml 

   1

   1

   Replace <route_name> with the actual name of your route.

3. Extract and assign the HOST value of the route to the DOMAIN variable by running the following command:

   DOMAIN=$(oc get route -n ai-test <route_name> -o=jsonpath="{ .status.ingress[0].host }") 

   1

   1

   Replace <route_name> with the actual name of your route.

4. Enable data transfer from the route to the MicroShift IP address by running the following command:

   $ curl -i "${DOMAIN}/v2/models/ovms-resnet50/ready" --connect-to "${DOMAIN}::${IP}:" 

   1

   1

   Instead of using the --connect-to "${DOMAIN}::${IP}:" flag, you can also use real DNS, or add the IP address and the domain to the /etc/hosts file.

   Example output

   HTTP/1.1 200 OK
   content-type: application/json
   date: Wed, 12 Mar 2025 16:01:32 GMT
   content-length: 0
   set-cookie: 56bb4b6df4f80f0b59f56aa0a5a91c1a=4af1408b4a1c40925456f73033d4a7d1; path=/; HttpOnly

5. Query the model metadata by running the following command:

   $ curl "${DOMAIN}/v2/models/ovms-resnet50" --connect-to "${DOMAIN}::${IP}:"

   Example output

   {"name":"ovms-resnet50","versions":["1"],"platform":"OpenVINO","inputs":[{"name":"0","datatype":"FP32","shape":[1,224,224,3]}],"outputs":[{"name":"1463","datatype":"FP32","shape":[1,1000]}]

Procedure

1. Download an image of a bee from the OpenVINO Model Server examples by running the following command:

   $ curl -O https://raw.githubusercontent.com/openvinotoolkit/model_server/main/demos/common/static/images/bee.jpeg

2. Create the request data by running the following script:

   IMAGE=./bee.jpeg
   REQ=./request.json
   
   # Add an inference header
   echo -n '{"inputs" : [{"name": "0", "shape": [1], "datatype": "BYTES"}]}' > "${REQ}"
   # Get the size of the inference header 

   1

   
   HEADER_LEN="$(stat -c %s "${REQ}")"
   # Add size of the data (image) in binary format (4 bytes, little endian) 

   2

   
   printf "%08X" $(stat --format=%s "${IMAGE}") | sed 's/\(..\)/\1\n/g' | tac | tr -d '\n' | xxd -r -p >> "${REQ}"
   # Add the data, that is, append the image to the request file
   cat "${IMAGE}" >> "${REQ}"

   1

   The inference header size must be passed to OpenVINO Model Server later in the form of an HTTP header.

   2

   The OpenVINO Model Server requires 4 bytes in little endian byte order.

Verification

1. To determine the model’s prediction, get the index of the highest element in the .outputs[0].data to determine the model’s predicted value by using the following Python script:

   import json
   with open('response.json') as f:
       response = json.load(f)
   data = response["outputs"][0]["data"]
   argmax = data.index(max(data))
   print(argmax)

   Example output

   309 

   1

   1

   In this example, the element labeled 309 is the model’s response.

2. Validate the output against resnet’s input data, for example:

   ../../../../demos/common/static/images/bee.jpeg 309