Inference serving geospatial foundation models

Procedure

1. Open a terminal on your server host, and log in to registry.redhat.io:

   $ podman login registry.redhat.io

   Copy to Clipboard Toggle word wrap

2. Pull the AI Inference Server NVIDIA CUDA container image:

   $ podman pull registry.redhat.io/rhaiis/vllm-cuda-rhel9:3.3.0

   Copy to Clipboard Toggle word wrap

3. If your system has SELinux enabled, configure SELinux to allow device access:

   $ sudo setsebool -P container_use_devices 1

   Copy to Clipboard Toggle word wrap

4. Create a volume and mount it into the container. Adjust the container permissions so that the container can use it.

   $ mkdir -p rhaiis-cache

   Copy to Clipboard Toggle word wrap

   $ chmod g+rwX rhaiis-cache

   Copy to Clipboard Toggle word wrap

5. Add your HF_TOKEN Hugging Face token to the private.env file. Source the private.env file.

   $ echo "export HF_TOKEN=<your_HF_token>" > private.env

   Copy to Clipboard Toggle word wrap

   $ source private.env

   Copy to Clipboard Toggle word wrap

6. Start the AI Inference Server container image.

   1. For NVIDIA CUDA accelerators, if the host system has multiple GPUs and uses NVSwitch, then start NVIDIA Fabric Manager. To detect if your system is using NVSwitch, first check if files are present in /proc/driver/nvidia-nvswitch/devices/, and then start NVIDIA Fabric Manager. Starting NVIDIA Fabric Manager requires root privileges.

      $ ls /proc/driver/nvidia-nvswitch/devices/

      Copy to Clipboard Toggle word wrap

      PCI device addresses for each available AI accelerator are returned.

      $ systemctl start nvidia-fabricmanager

      Copy to Clipboard Toggle word wrap
      Important

      NVIDIA Fabric Manager is only required on systems with multiple GPUs that use NVSwitch. For more information, see NVIDIA Server Architectures.

   2. Check that the AI Inference Server container can access NVIDIA GPUs on the host by running the following command:

      $ podman run --rm -it \
      --security-opt=label=disable \
      --device nvidia.com/gpu=all \
      nvcr.io/nvidia/cuda:12.4.1-base-ubi9 \
      nvidia-smi

      Copy to Clipboard Toggle word wrap

      All available AI accelerators are returned.

   3. Start the container with the TerraTorch backend and the Prithvi geospatial model.

      $ podman run --rm -it \
        --device nvidia.com/gpu=all \
        --security-opt=label=disable \
        --shm-size=4g \
        -p 8000:8000 \
        --userns=keep-id:uid=1001 \
        --env "HUGGING_FACE_HUB_TOKEN=$HF_TOKEN" \
        --env "HF_HUB_OFFLINE=0" \
        -v ./rhaiis-cache:/opt/app-root/src/.cache:Z \
        registry.redhat.io/rhaiis/vllm-cuda-rhel9:3.3.0 \
        --model ibm-nasa-geospatial/Prithvi-EO-2.0-300M-TL-Sen1Floods11 \
        --skip-tokenizer-init \
        --enforce-eager \
        --io-processor-plugin terratorch_segmentation \
        --enable-mm-embeds

      Copy to Clipboard Toggle word wrap

      For detailed information about TerraTorch server arguments and configuration options, see TerraTorch configuration options.

7. In a separate tab in your terminal, send an inference request with your geospatial data.

   $ curl -X POST http://localhost:8000/pooling \
     -H "Content-Type: application/json" \
     -d '{
       "model": "ibm-nasa-geospatial/Prithvi-EO-2.0-300M-TL-Sen1Floods11",
       "data": {
         "data": "https://<your_sample_geospatial_image>.tiff",
         "data_format": "url",
         "image_format": "tiff",
         "out_data_format": "b64_json"
       },
       "priority": 0
     }'

   Copy to Clipboard Toggle word wrap

   Example output

   {
     "request_id": "pool-98f71fcf667df37b",
     "created_at": 1770725528,
     "data": {
       "data_format": "b64_json",
       "data": "<BASE64_ENCODED_TIFF_DATA>",
       "request_id": "pool-98f71fcf667df37b"
     }
   }

   Copy to Clipboard Toggle word wrap

   The model returns a JSON response containing base64-encoded prediction data. Decode the data.data field to retrieve the output GeoTIFF file containing segmentation results.