Prevent workspace failures during node scaling

Prevent workspace failures during node scaling by configuring OpenShift Dev Spaces startup timeouts and pod annotations to work with the cluster autoscaler.

Before you begin

  • You have an active oc session with administrative permissions to the OpenShift cluster. See Getting started with the CLI.
  • You have the cluster autoscaler enabled on the OpenShift cluster.

About this task

When the autoscaler adds a new node, workspace startup can take longer than usual until node provisioning is complete. When the autoscaler removes a node, workspace pods should not be evicted because eviction can cause interruptions and loss of unsaved data.

Procedure

  1. Set the startup timeout and event handling in the CheCluster Custom Resource to handle autoscaler node additions: 
    spec:
  devEnvironments:
    startTimeoutSeconds: 600
    ignoredUnrecoverableEvents:
      - FailedScheduling

    where:

    startTimeoutSeconds
    Set to at least 600 seconds to allow time for a new node to be provisioned during workspace startup.
    ignoredUnrecoverableEvents
    Ignore the FailedScheduling event to allow workspace startup to continue when a new node is provisioned. This setting is enabled by default.
  2. Add the safe-to-evict annotation to the CheCluster Custom Resource to prevent workspace pod eviction when the autoscaler removes a node: 
    spec:
  devEnvironments:
    workspacesPodAnnotations:
      cluster-autoscaler.kubernetes.io/safe-to-evict: "false"

Results

  • Start a workspace and verify that the workspace pod contains the cluster-autoscaler.kubernetes.io/safe-to-evict: "false" annotation: 
    $ oc get pod <workspace_pod_name> -o jsonpath='{.metadata.annotations.cluster-autoscaler\.kubernetes\.io/safe-to-evict}'
false