Red Hat SummitChapter 4. Deploying and testing a model
4.1. Preparing a model for deployment
After you train a model, you can deploy it by using the OpenShift AI model serving capabilities.
To prepare a model for deployment, you must complete the following tasks:
- Move the model from your workbench to your S3-compatible object storage. Use the connection that you created in the Storing data with connections section and upload the model from a notebook.
- Convert the model to the portable ONNX format. ONNX allows you to transfer models between frameworks with minimal preparation and without the need for rewriting the models.
Prerequisites
You created the
My Storageconnection and have added it to your workbench.
Procedure
-
In your JupyterLab environment, open the
2_save_model.ipynbfile. - Follow the instructions in the notebook to make the model accessible in storage and save it in the portable ONNX format.
Verification
When you have completed the notebook instructions, the models/fraud/1/model.onnx file is in your object storage and it is ready for your model server to use.
Next step
4.2. Deploying a model
Now that the model is accessible in storage and saved in the portable ONNX format, you can use an OpenShift AI model server to deploy it as an API.
OpenShift AI offers two options for model serving:
- Single-model serving - Each model in the project is deployed on its own model server. This platform works well for large models or models that need dedicated resources.
- Multi-model serving - All models in the project are deployed on the same model server. This platform is suitable for sharing resources amongst deployed models. Multi-model serving is the only option offered in the Red Hat Developer Sandbox environment.
For this tutorial, since you are only deploying only one model, you can select either serving type. The steps for deploying the fraud detection model depend on the type of model serving platform you select:
4.2.1. Deploying a model on a single-model server
OpenShift AI single-model servers host only one model. You create a new model server and deploy your model to it.
Prerequisites
-
A user with
adminprivileges has enabled the single-model serving platform on your OpenShift cluster.
Procedure
In the OpenShift AI dashboard, navigate to the project details page and click the Models tab.
NoteDepending on how model serving has been configured on your cluster, you might see only one model serving platform option.
- In the Single-model serving platform tile, click Select single-model.
In the form, provide the following values:
-
For Model deployment name, type
fraud. -
For Serving runtime, select
OpenVINO Model Server. -
For Model framework (name - version), select
onnx-1. -
For Deployment mode, select
Standard. -
For Existing connection, select
My Storage. -
Type the path that leads to the version folder that contains your model file:
models/fraud Leave the other fields with the default settings.
-
For Model deployment name, type
- Click Deploy.
Verification
Notice the loading symbol under the Status section. The symbol changes to a green checkmark when the deployment completes successfully.
Next step
4.2.2. Deploying a model on a multi-model server
OpenShift AI multi-model servers can host several models at once. You create a new model server and deploy your model to it.
Prerequisites
-
A user with
adminprivileges has enabled the multi-model serving platform on your OpenShift cluster.
Procedure
In the OpenShift AI dashboard, navigate to the project details page and click the Models tab.
NoteDepending on how model serving has been configured on your cluster, you might see only one model serving platform option.
- In the Multi-model serving platform tile, click Select multi-model.
In the form, provide the following values:
-
For Model server name, type a name, for example
Model Server. -
For Serving runtime, select
OpenVINO Model Server. Leave the other fields with the default settings.
-
For Model server name, type a name, for example
- Click Add.
In the Models and model servers list, next to the new model server, click Deploy model.
In the form, provide the following values:
-
For Model deployment name, type
fraud. -
For Model framework (name - version), select
onnx-1. -
For Existing connection, select
My Storage. -
Type the path that leads to the version folder that contains your model file:
models/fraud Leave the other fields with the default settings.
-
For Model deployment name, type
- Click Deploy.
Verification
Notice the loading symbol under the Status section. The symbol changes to a green checkmark when the deployment completes successfully.
Next step
4.3. Testing the model API
Now that you’ve deployed the model, you can test its API endpoints.
Procedure
- In the OpenShift AI dashboard, navigate to the project details page and click the Models tab.
Take note of the model’s Inference endpoint URL. You need this information when you test the model API.
If the Inference endpoint field contains an Internal endpoint details link, click the link to open a text box that shows the URL details, and then take note of the restUrl value.
Return to the JupyterLab environment and try out your new endpoint.
If you deployed your model with the multi-model serving platform, follow the directions in
3_rest_requests_multi_model.ipynbto try a REST API call and4_grpc_requests_multi_model.ipynbto try a gRPC API call.If you deployed your model with the single-model serving platform, follow the directions in
5_rest_requests_single_model.ipynbto try a REST API call.
Next step
(Optional) Running a data science pipeline generated from Python code
