Chapter 4. Run Spark workloads from OpenShift AI workbenches

Procedure

1. Create the ServiceAccount in the target namespace.

   In the following command, replace _<target_namespace>_ with the name of your namespace:

   $ cat <<EOF | oc apply -f -
   apiVersion: v1
   kind: ServiceAccount
   metadata:
     name: spark-operator-spark
     namespace: <target_namespace>
   EOF

2. Create the Role with the required permissions in the target namespace.

   The Spark driver needs permissions to create and manage executor pods, ConfigMaps, PersistentVolumeClaims, and Services:

   $ cat <<EOF | oc apply -f -
   apiVersion: rbac.authorization.k8s.io/v1
   kind: Role
   metadata:
     name: spark-role
     namespace: <target_namespace>
   rules:
   - apiGroups:
     - ""
     resources:
     - pods
     - configmaps
     - persistentvolumeclaims
     - services
     verbs:
     - get
     - list
     - watch
     - create
     - update
     - patch
     - delete
     - deletecollection
   EOF

3. Create the RoleBinding to connect the ServiceAccount to the Role:

   $ cat <<EOF | oc apply -f -
   apiVersion: rbac.authorization.k8s.io/v1
   kind: RoleBinding
   metadata:
     name: spark-role-binding
     namespace: <target_namespace>
   subjects:
   - kind: ServiceAccount
     name: spark-operator-spark
   roleRef:
     kind: Role
     name: spark-role
     apiGroup: rbac.authorization.k8s.io
   EOF

4. Create the NetworkPolicy to allow Spark internal communication and workbench connectivity:

   $ cat <<EOF | oc apply -f -
   apiVersion: networking.k8s.io/v1
   kind: NetworkPolicy
   metadata:
     name: spark-operator-allow-internal
     namespace: <target_namespace>
   spec:
     podSelector:
       matchLabels:
         sparkoperator.k8s.io/launched-by-spark-operator: "true"
     policyTypes:
       - Ingress
     ingress:
       - ports:
           - port: 7078
             protocol: TCP
           - port: 7079
             protocol: TCP
           - port: 4040
             protocol: TCP
         from:
           - podSelector: {}
           - namespaceSelector:
               matchLabels:
                 network.openshift.io/policy-group: ingress
       - ports:
           - port: 15002
             protocol: TCP
         from:
           - podSelector: {}
           - namespaceSelector:
               matchLabels:
                 opendatahub.io/dashboard: "true"
             podSelector:
               matchLabels:
                 opendatahub.io/workbenches: "true"
   EOF

   • Ports 7078 and 7079 enable remote procedure call (RPC) and block manager communication between the Spark driver and executor pods.
   • Port 4040 enables the Spark UI.
   • Port 15002 enables gRPC connectivity from workbench pods in namespaces managed by the OpenShift AI dashboard.

5. Deploy the SparkConnect custom resource in the target namespace.

   Create a YAML file with the metadata.namespace field set to your target namespace and apply it:

   $ oc apply -f spark-connect-server.yaml

   For the complete SparkConnect custom resource example, see SparkConnect custom resource and infrastructure reference.

Verification

1. Verify that the server pod is running in the target namespace:

   $ oc get pods -n <target_namespace> -l sparkoperator.k8s.io/launched-by-spark-operator=true

   The output shows a pod with a Running status.

2. Verify that the auto-created Service exists:

   $ oc get svc -n <target_namespace> -l sparkoperator.k8s.io/connect-name=spark-connect

   The output shows a ClusterIP Service exposing port 15002.

3. Verify that the RBAC resources are created correctly:

   $ oc get serviceaccount spark-operator-spark -n <target_namespace>
   $ oc get role spark-role -n <target_namespace>
   $ oc get rolebinding spark-role-binding -n <target_namespace>

4. Verify that the NetworkPolicy was created in the target namespace:

   $ oc get networkpolicy spark-operator-allow-internal -n <target_namespace>

Procedure

1. Create a YAML file for the SparkConnect resource. The following example shows a basic SparkConnect configuration:

   apiVersion: sparkoperator.k8s.io/v1alpha1
   kind: SparkConnect
   metadata:
     name: spark-connect
     namespace: <your_namespace>
   spec:
     sparkVersion: 4.0.1
     server:
       template:
         spec:
           containers:
           - name: spark-kubernetes-driver
             image: quay.io/opendatahub/data-processing:Spark-v4.0.1
             imagePullPolicy: Always
             resources:
               requests:
                 cpu: 1
                 memory: 1Gi
               limits:
                 cpu: 1
                 memory: 1Gi
           serviceAccountName: spark-operator-spark
           securityContext: {}
     executor:
       instances: 2
       cores: 1
       memory: 512m
       template:
         spec:
           containers:
           - name: spark-kubernetes-executor
             image: quay.io/opendatahub/data-processing:Spark-v4.0.1
             imagePullPolicy: Always
           securityContext: {}

   Note

   The securityContext: {} field at the pod level is intentionally empty. On OpenShift, the restricted-v2 Security Context Constraint (SCC) automatically enforces security restrictions such as runAsNonRoot, capability dropping, and seccomp profiles. Do not add explicit security context fields to the SparkConnect custom resource.

2. Apply the SparkConnect custom resource:

   $ oc apply -f spark-connect-server.yaml

3. Verify that the Spark Connect server pod is running:

   $ oc get pods -n <your_namespace> -l sparkoperator.k8s.io/launched-by-spark-operator=true

   The output shows a pod named spark-connect-server with a Running status:

   NAME                     READY   STATUS    RESTARTS   AGE
   spark-connect-server     1/1     Running   0          30s

4. Verify that the Operator has created the ClusterIP Service automatically:

   $ oc get svc -n <your_namespace> -l sparkoperator.k8s.io/connect-name=spark-connect

   The output shows the auto-created Service exposing ports for Spark internal communication and gRPC connectivity:

   NAME                     TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                                   AGE
   spark-connect-server     ClusterIP   172.30.x.x      <none>        7078/TCP,7079/TCP,4040/TCP,15002/TCP      30s

   Note

   The Operator creates this Service automatically. You do not need to create a Service manually. The Service DNS name follows the convention _<connect_name>_-server._<namespace>_.svc.cluster.local. For example, the DNS name is spark-connect-server.<your_namespace>.svc.cluster.local.

5. Record the Service DNS name for use in the workbench connection string:

   spark-connect-server.<namespace>.svc.cluster.local

   Workbench users use this DNS name with port 15002 to construct the sc:// connection URL in their workbench notebooks.

Procedure

1. In your Jupyter notebook, install PySpark and the required dependencies by running the following command in a notebook cell:

   !pip install -q pyspark==4.0.1 pandas pyarrow grpcio grpcio-status zstandard

   Important

   The PySpark version must match the Spark version used in the Spark Connect server container image. A version mismatch causes connection failures. After the installation completes, restart the notebook kernel so that the new packages take effect.

2. Construct the Spark Connect URL by using the Service DNS name and port 15002.

   The URL follows this format:

   sc://<connect_name>-server.<namespace>.svc.cluster.local:15002

   For a SparkConnect resource named spark-connect deployed in {dbd-config-default-namespace}, the URL is:

   sc://spark-connect-server.redhat-ods-applications.svc.cluster.local:15002

3. Create a SparkSession by using the sc:// connection URL.

   Run the following code in a notebook cell:

   from pyspark.sql import SparkSession
   
   spark = SparkSession.builder.remote(
       "sc://spark-connect-server.redhat-ods-applications.svc.cluster.local:15002"
   ).getOrCreate()

   Note

   Replace the Service DNS name and namespace if your Spark Connect server is deployed in a different namespace.

4. Run a PySpark DataFrame workload to verify the connection.

   The following example creates a DataFrame, applies a filter, and displays the results:

   data = [("Alice", 34), ("Bob", 45), ("Claire", 66)]
   df = spark.createDataFrame(data, ["Name", "Age"])
   df.filter(df.Age > 40).show()

   The expected output is as follows:

   +------+---+
   |  Name|Age|
   +------+---+
   |   Bob| 45|
   |Claire| 66|
   +------+---+

1. Check the Spark Connect server pod status and logs.

   In the following commands, replace _<namespace>_ with the namespace where the SparkConnect resource is deployed:

   $ oc get pods -n <namespace> -l sparkoperator.k8s.io/launched-by-spark-operator=true

   If the pod is not in Running status, inspect the pod events for startup failures:

   $ oc describe pod spark-connect-server -n <namespace>

   Check the server logs for error messages:

   $ oc logs spark-connect-server -n <namespace>

2. Verify that the ClusterIP Service exists and has the correct selector labels.

   $ oc get svc -n <namespace> -l sparkoperator.k8s.io/connect-name=spark-connect

   If no Service is returned, verify that the SparkConnect custom resource has been applied correctly. The Operator creates the Service automatically during reconciliation.

   Inspect the Service details to confirm that the selector labels and port match:

   $ oc describe svc spark-connect-server -n <namespace>

   The Service should select pods with labels sparkoperator.k8s.io/launched-by-spark-operator: true, sparkoperator.k8s.io/connect-name: _<connect_name>_, spark-role: connect-server, and spark-version: _<spark_version>_. The Service exposes ports 7078, 7079, 4040, and 15002.

3. Verify that the Service DNS name resolves from inside the workbench pod.

   Find the workbench pod name:

   $ oc get pods --all-namespaces -l opendatahub.io/workbenches=true

   Test DNS resolution from inside the workbench pod. In the following command, replace _<workbench_pod>_ and _<workbench_namespace>_ with your workbench pod name and namespace:

   $ oc exec -n <workbench_namespace> <workbench_pod> -- nslookup spark-connect-server.<namespace>.svc.cluster.local

   If DNS resolution fails, verify that the Service exists in the expected namespace.

4. Verify that the NetworkPolicy allows traffic on port 15002 from your workbench.

   Check that the workbench pod has the required label:

   $ oc get pod <workbench_pod> -n <workbench_namespace> --show-labels | grep opendatahub.io/workbenches

   Check that the workbench namespace has the required label:

   $ oc get namespace <workbench_namespace> --show-labels | grep opendatahub.io/dashboard

   If either label is missing, the spark-operator-allow-internal NetworkPolicy blocks the gRPC traffic. The workbench pod must have the opendatahub.io/workbenches: "true" label, and the workbench namespace must have the opendatahub.io/dashboard: "true" label.

5. Test gRPC connectivity from the workbench pod.

   Open a remote shell on the workbench pod and attempt a PySpark connection:

   $ oc rsh -n <workbench_namespace> <workbench_pod>

   Inside the shell, run a basic connection test:

   python -c "from pyspark.sql import SparkSession; spark = SparkSession.builder.remote('sc://spark-connect-server.<namespace>.svc.cluster.local:15002').getOrCreate(); print(spark.version)"

   If this command succeeds but the notebook cell fails, restart the notebook kernel to reload the installed packages.