Chapter 3. Using the Kubeflow Spark Operator

Procedure

1. Create a ContainerFile called spark-image with the following parameters:

   FROM apache/spark:4.0.1
   
   LABEL description="Apache Spark image compatible with OpenShift restricted-v2 SCC"
   
   USER root
   
   RUN chgrp -R 0 /opt/spark && \
       chmod -R g=u /opt/spark && \
       mkdir -p /opt/spark/work-dir /opt/spark/logs && \
       chgrp -R 0 /opt/spark/work-dir /opt/spark/logs && \
       chmod -R 775 /opt/spark/work-dir /opt/spark/logs
   
   RUN chmod 1777 /tmp
   
   ENV HOME=/home/spark
   
   RUN mkdir -p /home/spark && \
       chgrp -R 0 /home/spark && \
       chmod -R g=u /home/spark && \
       chmod 775 /home/spark

2. Build and tag the image for your registry with the following command:

   $ podman build -t quay.io//spark-image:4.0.1 -f spark-image

3. Push your spark-image to the registry:

   $ podman push quay.io//spark-image:4.0.1

4. The SparkApplication custom resource definition (CRD) allows you to define Spark applications using YAML manifests.

   $ oc apply -f sparkapplication-example.yaml

   Example SparkApplication custom resource (CR)

   apiVersion: sparkoperator.k8s.io/v1beta2
   kind: SparkApplication
   metadata:
     name: ${APP_NAME}
     namespace: ${APP_NAMESPACE}
   spec:
     type: Scala
     mode: cluster
     image: <your-custom-spark-image>
     imagePullPolicy: IfNotPresent
     mainClass: org.apache.spark.examples.SparkPi
     mainApplicationFile: local:///opt/spark/examples/jars/spark-examples.jar
     arguments:
       - "1000"
     sparkVersion: "4.0.1"
     restartPolicy:
       type: Never
     driver:
       cores: 1
       coreLimit: "1200m"
       memory: "512m"
       labels:
         version: 4.0.1
       serviceAccount: spark-operator-spark
       securityContext: {}
     executor:
       cores: 1
       instances: 1
       memory: "512m"
       labels:
         version: 4.0.1
       securityContext: {}

   Expand

   Table 3.1. SparkApplication CR reference

   Field	Description
   namespace	The namespace to run the Spark application, the default is the redhat-ods-applications namespace. If you want to run a Spark application on a custom namespace, you need to adjust the role binding to match the roles of the default namespace. For more information
   type	The language of the application. For example, Python, Scala, etc.
   mode	Deployment mode, example fields include, cluster or client. In cluster mode, the driver runs in a pod.
   image	The container image to use for the driver and executors. This image must include the Spark v4.0.1 runtime.
   mainApplicationFile	The entry point path. For example, local:///app/scripts/run_spark_job.py.
   sparkVersion	The version of Spark to use, the version included must match the image.
   restartPolicy	Handling of failures. Example fields include: Never, OnFailure, Always.
   driver/executor	Resource requests (cores, memory), labels, service accounts, and security contexts.
   volumes/volumeMounts	PVCs for input and output data.
   serviceAccount	The spark-operator-spark service account is created automatically by the Spark Operator installation and includes the necessary RBAC permissions for Spark drivers to create and manage executor pods.

   Show more