SupportConsoleDevelopersStart a trialContact

Chapter 1. Overview of the Kubeflow Spark Operator (KSO)

The Kubeflow Spark Operator allows you to run Spark data processing applications in a distributed environment on Red Hat OpenShift AI. You can create custom resources (CRs) for specifying, running and surfacing Spark applications.

KSO for Apache Spark currently supports the following:

  • Spark versions 4.0.1 and above.
  • When a SparkApplication custom resource is created, the operator creates and runs a spark-submit job that starts Spark driver and executor pods.
  • Native Cron support for running scheduled applications.
  • Customization of Spark pods, including: mounting ConfigMaps/volumes and setting pod affinity.
  • Automatic application restart with custom policies.
  • Collecting and exporting application metrics and driver metrics to Prometheus.

1.1. Kubeflow Spark Operator (KSO) architecture
Copy link

The Spark operator consists of the following:

  • SparkApplication Controller: A job that watches Create, Update and Delete events in SparkApplication resources.
  • Submission Runner: When a SparkApplication custom resource is created, the operator creates and runs a spark-submit job that starts Spark driver and executor pods.
  • Spark Pod Monitor: Observes the Driver and Executor pods and updates the .status field of the SparkApplication CR.
  • Mutating Admission Webhook: A component that intercepts pod creation requests and injects ConfigMap mounts or Volumes into the Driver and Executor pods before they are scheduled.

1.2. IBM Power (ppc64le) considerations
Copy link

When deploying the Kubeflow Spark Operator on IBM Power (ppc64le) systems, ensure that both the Spark runtime image used by SparkApplication resources and the Spark Operator image are built for the target architecture.

The Spark Operator controller manages SparkApplication resources and launches Spark driver and executor pods using the image specified in the SparkApplication custom resource. If architecture-compatible images are not available, Spark runtime and Spark Operator images must be built from source for ppc64le.

For more information about building architecture-specific images, refer to the upstream Apache Spark and Kubeflow Spark Operator repositories:

Red Hat logoGithubredditYoutubeTwitter

Learn

Try, buy, & sell

Communities

About Red Hat

We deliver hardened solutions that make it easier for enterprises to work across platforms and environments, from the core datacenter to the network edge.

Making open source more inclusive

Red Hat is committed to replacing problematic language in our code, documentation, and web properties. For more details, see the Red Hat Blog.

About Red Hat Documentation

We help Red Hat users innovate and achieve their goals with our products and services with content they can trust. Explore our recent updates.

Legal Notice

Theme

© 2026 Red HatBack to top