Chapter 17. Deploying distributed units manually on single-node OpenShift

Procedure

• The base64-encoded content below contains the CPU set that the management workloads are constrained to. This content must be adjusted to match the set specified in the performanceprofile and must be accurate for the number of cores on the cluster.

  apiVersion: machineconfiguration.openshift.io/v1
  kind: MachineConfig
  metadata:
    labels:
      machineconfiguration.openshift.io/role: master
    name: 02-master-workload-partitioning
  spec:
    config:
      ignition:
        version: 3.2.0
      storage:
        files:
        - contents:
            source: data:text/plain;charset=utf-8;base64,W2NyaW8ucnVudGltZS53b3JrbG9hZHMubWFuYWdlbWVudF0KYWN0aXZhdGlvbl9hbm5vdGF0aW9uID0gInRhcmdldC53b3JrbG9hZC5vcGVuc2hpZnQuaW8vbWFuYWdlbWVudCIKYW5ub3RhdGlvbl9wcmVmaXggPSAicmVzb3VyY2VzLndvcmtsb2FkLm9wZW5zaGlmdC5pbyIKW2NyaW8ucnVudGltZS53b3JrbG9hZHMubWFuYWdlbWVudC5yZXNvdXJjZXNdCmNwdXNoYXJlcyA9IDAKQ1BVcyA9ICIwLTEsIDUyLTUzIgo=
          mode: 420
          overwrite: true
          path: /etc/crio/crio.conf.d/01-workload-partitioning
          user:
            name: root
        - contents:
            source: data:text/plain;charset=utf-8;base64,ewogICJtYW5hZ2VtZW50IjogewogICAgImNwdXNldCI6ICIwLTEsNTItNTMiCiAgfQp9Cg==
          mode: 420
          overwrite: true
          path: /etc/kubernetes/openshift-workload-pinning
          user:
            name: root

• The contents of /etc/crio/crio.conf.d/01-workload-partitioning should look like this:

  [crio.runtime.workloads.management]
  activation_annotation = "target.workload.openshift.io/management"
  annotation_prefix = "resources.workload.openshift.io"
  [crio.runtime.workloads.management.resources]
  cpushares = 0
  cpuset = "0-1, 52-53" 1

  1

  The cpuset value varies based on the installation.

  If Hyper-Threading is enabled, specify both threads for each core. The cpuset value must match the reserved CPUs that you define in the spec.cpu.reserved field in the performance profile.

• The contents of /etc/kubernetes/openshift-workload-pinning should look like this:

  {
    "management": {
      "cpuset": "0-1,52-53" 1
    }
  }

  1

  The cpuset must match the cpuset value in /etc/crio/crio.conf.d/01-workload-partitioning.

Procedure

• No configuration changes are needed. Use the provided settings:

  apiVersion: machineconfiguration.openshift.io/v1
  kind: MachineConfig
  metadata:
    labels:
      machineconfiguration.openshift.io/role: master
    name: load-sctp-module
  spec:
    config:
      ignition:
        version: 2.2.0
      storage:
        files:
          - contents:
              source: data:,
              verification: {}
            filesystem: root
              mode: 420
              path: /etc/modprobe.d/sctp-blacklist.conf
          - contents:
              source: data:text/plain;charset=utf-8,sctp
            filesystem: root
              mode: 420
              path: /etc/modules-load.d/sctp-load.conf

Procedure

• No configuration changes are needed. Use the provided settings:

  Local Storage Operator

  apiVersion: v1
  kind: Namespace
  metadata:
    annotations:
      workload.openshift.io/allowed: management
    name: openshift-local-storage
  ---
  apiVersion: operators.coreos.com/v1
  kind: OperatorGroup
  metadata:
    name: openshift-local-storage
    namespace: openshift-local-storage
  spec:
    targetNamespaces:
      - openshift-local-storage

  Logging Operator

  apiVersion: v1
  kind: Namespace
  metadata:
    annotations:
      workload.openshift.io/allowed: management
    name: openshift-logging
  ---
  apiVersion: operators.coreos.com/v1
  kind: OperatorGroup
  metadata:
    name: cluster-logging
    namespace: openshift-logging
  spec:
    targetNamespaces:
      - openshift-logging

  Performance Addon Operator

  apiVersion: v1
  kind: Namespace
  metadata:
    annotations:
      workload.openshift.io/allowed: management
    labels:
      openshift.io/cluster-monitoring: "true"
    name: openshift-performance-addon-operator
  spec: {}
  ---
  apiVersion: operators.coreos.com/v1
  kind: OperatorGroup
  metadata:
    name: performance-addon-operator
    namespace: openshift-performance-addon-operator

  PTP Operator

  apiVersion: v1
  kind: Namespace
  metadata:
    annotations:
      workload.openshift.io/allowed: management
    labels:
      openshift.io/cluster-monitoring: "true"
    name: openshift-ptp
  ---
  apiVersion: operators.coreos.com/v1
  kind: OperatorGroup
  metadata:
    name: ptp-operators
    namespace: openshift-ptp
  spec:
    targetNamespaces:
      - openshift-ptp

  SRIOV Network Operator

  apiVersion: v1
  kind: Namespace
  metadata:
    annotations:
      workload.openshift.io/allowed: management
      name: openshift-sriov-network-operator
  ---
  apiVersion: operators.coreos.com/v1
  kind: OperatorGroup
  metadata:
    name: sriov-network-operators
    namespace: openshift-sriov-network-operator
  spec:
    targetNamespaces:
      - openshift-sriov-network-operator

Procedure

• Use the following example to configure the subscription:

  apiVersion: operators.coreos.com/v1alpha1
  kind: Subscription
  metadata:
    name: cluster-logging
    namespace: openshift-logging
  spec:
    channel: "stable" 1
    name: cluster-logging
    source: redhat-operators
    sourceNamespace: openshift-marketplace
    installPlanApproval: Manual 2
  ---
  apiVersion: operators.coreos.com/v1alpha1
  kind: Subscription
  metadata:
    name: local-storage-operator
    namespace: openshift-local-storage
  spec:
    channel: "stable" 3
    installPlanApproval: Automatic
    name: local-storage-operator
    source: redhat-operators
    sourceNamespace: openshift-marketplace
    installPlanApproval: Manual
  ---
  apiVersion: operators.coreos.com/v1alpha1
  kind: Subscription
  metadata:
    name: performance-addon-operator
    namespace: openshift-performance-addon-operator
  spec:
    channel: "4.10" 4
    name: performance-addon-operator
    source: performance-addon-operator
    sourceNamespace: openshift-marketplace
    installPlanApproval: Manual
  ---
  apiVersion: operators.coreos.com/v1alpha1
  kind: Subscription
  metadata:
      name: ptp-operator-subscription
      namespace: openshift-ptp
  spec:
    channel: "stable" 5
    name: ptp-operator
    source: redhat-operators
    sourceNamespace: openshift-marketplace
    installPlanApproval: Manual
  ---
  apiVersion: operators.coreos.com/v1alpha1
  kind: Subscription
  metadata:
    name: sriov-network-operator-subscription
    namespace: openshift-sriov-network-operator
  spec:
    channel: "stable" 6
    name: sriov-network-operator
    source: redhat-operators
    sourceNamespace: openshift-marketplace
    installPlanApproval: Manual

  1

  Specify the channel to get the cluster-logging Operator.

  2

  Specify Manual or Automatic. In Automatic mode, the Operator automatically updates to the latest versions in the channel as they become available in the registry. In Manual mode, new Operator versions are installed only after they are explicitly approved.

  3

  Specify the channel to get the local-storage-operator Operator.

  4

  Specify the channel to get the performance-addon-operator Operator.

  5

  Specify the channel to get the ptp-operator Operator.

  6

  Specify the channel to get the sriov-network-operator Operator.

Procedure

• Edit the ClusterLogging custom resource (CR) in the openshift-logging project:

  apiVersion: logging.openshift.io/v1
  kind: ClusterLogging 1
   metadata:
    name: instance
    namespace: openshift-logging
  spec:
    collection:
      logs:
        fluentd: {}
        type: fluentd
    curation:
      type: "curator"
      curator:
        schedule: "30 3 * * *"
      managementState: Managed
  ---
  apiVersion: logging.openshift.io/v1
  kind: ClusterLogForwarder 2
  metadata:
    name: instance
    namespace: openshift-logging
  spec:
    inputs:
      - infrastructure: {}
    outputs:
      - name: kafka-open
        type: kafka
        url: tcp://10.46.55.190:9092/test    3
    pipelines:
      - inputRefs:
        - audit
        name: audit-logs
        outputRefs:
        - kafka-open
      - inputRefs:
        - infrastructure
        name: infrastructure-logs
        outputRefs:
        - kafka-open

  1

  Updates the existing instance or creates the instance if it does not exist.

  2

  Updates the existing instance or creates the instance if it does not exist.

  3

  Specifies the destination of the kafka server.

Procedure

• Configure the performance addons using the following example:

  Recommended performance profile configuration

  apiVersion: performance.openshift.io/v2
  kind: PerformanceProfile
  metadata:
    name: openshift-node-performance-profile 1
  spec:
    additionalKernelArgs:
      - "idle=poll"
      - "rcupdate.rcu_normal_after_boot=0"
    cpu:
      isolated: 2-51,54-103 2
      reserved: 0-1,52-53   3
    hugepages:
      defaultHugepagesSize: 1G
      pages:
        - count: 32 4
          size: 1G  5
          node: 0 6
    machineConfigPoolSelector:
      pools.operator.machineconfiguration.openshift.io/master: ""
    net:
      userLevelNetworking: true 7
    nodeSelector:
      node-role.kubernetes.io/master: ''
    numa:
      topologyPolicy: "restricted"
    realTimeKernel:
      enabled: true    8

  1

  Ensure that the value for name matches that specified in the spec.profile.data field of TunedPerformancePatch.yaml and the status.configuration.source.name field of validatorCRs/informDuValidator.yaml.

  2

  Set the isolated CPUs. Ensure all of the Hyper-Threading pairs match.

  3

  Set the reserved CPUs. When workload partitioning is enabled, system processes, kernel threads, and system container threads are restricted to these CPUs. All CPUs that are not isolated should be reserved.

  4

  Set the number of huge pages.

  5

  Set the huge page size.

  6

  Set node to the NUMA node where the hugepages are allocated.

  7

  Set userLevelNetworking to true to isolate the CPUs from networking interrupts.

  8

  Set enabled to true to install the real-time Linux kernel.

Procedure

• Configure PTP using the following example:

  apiVersion: ptp.openshift.io/v1
  kind: PtpConfig
  metadata:
    name: du-ptp-slave
    namespace: openshift-ptp
  spec:
    profile:
      - interface: ens5f0     1
        name: slave
        phc2sysOpts: -a -r -n 24
        ptp4lConf: |
          [global]
          #
          # Default Data Set
          #
          twoStepFlag 1
          slaveOnly 0
          priority1 128
          priority2 128
          domainNumber 24
          #utc_offset 37
          clockClass 248
          clockAccuracy 0xFE
          offsetScaledLogVariance 0xFFFF
          free_running 0
          freq_est_interval 1
          dscp_event 0
          dscp_general 0
          dataset_comparison ieee1588
          G.8275.defaultDS.localPriority 128
          #
          # Port Data Set
          #
          logAnnounceInterval -3
          logSyncInterval -4
          logMinDelayReqInterval -4
          logMinPdelayReqInterval -4
          announceReceiptTimeout 3
          syncReceiptTimeout 0
          delayAsymmetry 0
          fault_reset_interval 4
          neighborPropDelayThresh 20000000
          masterOnly 0
          G.8275.portDS.localPriority 128
          #
          # Run time options
          #
          assume_two_step 0
          logging_level 6
          path_trace_enabled 0
          follow_up_info 0
          hybrid_e2e 0
          inhibit_multicast_service 0
          net_sync_monitor 0
          tc_spanning_tree 0
          tx_timestamp_timeout 50
          unicast_listen 0
          unicast_master_table 0
          unicast_req_duration 3600
          use_syslog 1
          verbose 0
          summary_interval 0
          kernel_leap 1
          check_fup_sync 0
          #
          # Servo Options
          #
          pi_proportional_const 0.0
          pi_integral_const 0.0
          pi_proportional_scale 0.0
          pi_proportional_exponent -0.3
          pi_proportional_norm_max 0.7
          pi_integral_scale 0.0
          pi_integral_exponent 0.4
          pi_integral_norm_max 0.3
          step_threshold 2.0
          first_step_threshold 0.00002
          max_frequency 900000000
          clock_servo pi
          sanity_freq_limit 200000000
          ntpshm_segment 0
          #
          # Transport options
          #
          transportSpecific 0x0
          ptp_dst_mac 01:1B:19:00:00:00
          p2p_dst_mac 01:80:C2:00:00:0E
          udp_ttl 1
          udp6_scope 0x0E
          uds_address /var/run/ptp4l
          #
          # Default interface options
          #
          clock_type OC
          network_transport UDPv4
          delay_mechanism E2E
          time_stamping hardware
          tsproc_mode filter
          delay_filter moving_median
          delay_filter_length 10
          egressLatency 0
          ingressLatency 0
          boundary_clock_jbod 0
          #
          # Clock description
          #
          productDescription ;;
          revisionData ;;
          manufacturerIdentity 00:00:00
          userDescription ;
          timeSource 0xA0
        ptp4lOpts: -2 -s --summary_interval -4
  recommend:
    - match:
        - nodeLabel: node-role.kubernetes.io/master
      priority: 4
      profile: slave

Procedure

• No configuration changes are needed. Use the provided settings:

  apiVersion: machineconfiguration.openshift.io/v1
  kind: MachineConfig
  metadata:
    labels:
      machineconfiguration.openshift.io/role: master
    name: disable-chronyd
  spec:
    config:
      systemd:
        units:
          - contents: |
              [Unit]
              Description=NTP client/server
              Documentation=man:chronyd(8) man:chrony.conf(5)
              After=ntpdate.service sntp.service ntpd.service
              Conflicts=ntpd.service systemd-timesyncd.service
              ConditionCapability=CAP_SYS_TIME
              [Service]
              Type=forking
              PIDFile=/run/chrony/chronyd.pid
              EnvironmentFile=-/etc/sysconfig/chronyd
              ExecStart=/usr/sbin/chronyd $OPTIONS
              ExecStartPost=/usr/libexec/chrony-helper update-daemon
              PrivateTmp=yes
              ProtectHome=yes
              ProtectSystem=full
              [Install]
              WantedBy=multi-user.target
            enabled: false
            name: chronyd.service
      ignition:
        version: 2.2.0

Procedure

• Use the following configuration to configure SRIOV on a single node distributed unit (DU). Note that the first custom resource (CR) is required. The following CRs are examples.

  apiVersion: sriovnetwork.openshift.io/v1
  kind: SriovOperatorConfig
  metadata:
    name: default
    namespace: openshift-sriov-network-operator
  spec:
    configDaemonNodeSelector:
      node-role.kubernetes.io/master: ""
    disableDrain: true
    enableInjector: true
    enableOperatorWebhook: true
  ---
  apiVersion: sriovnetwork.openshift.io/v1
  kind: SriovNetwork
  metadata:
    name: sriov-nw-du-mh
    namespace: openshift-sriov-network-operator
  spec:
    networkNamespace: openshift-sriov-network-operator
    resourceName: du_mh
    vlan: 150 1
  ---
  apiVersion: sriovnetwork.openshift.io/v1
  kind: SriovNetworkNodePolicy
  metadata:
    name: sriov-nnp-du-mh
    namespace: openshift-sriov-network-operator
  spec:
    deviceType: vfio-pci 2
    isRdma: false
    nicSelector:
      pfNames:
        - ens7f0 3
    nodeSelector:
      node-role.kubernetes.io/master: ""
    numVfs: 8 4
    priority: 10
    resourceName: du_mh
  ---
  apiVersion: sriovnetwork.openshift.io/v1
  kind: SriovNetwork
  metadata:
    name: sriov-nw-du-fh
    namespace: openshift-sriov-network-operator
  spec:
    networkNamespace: openshift-sriov-network-operator
    resourceName: du_fh
    vlan: 140 5
  ---
  apiVersion: sriovnetwork.openshift.io/v1
  kind: SriovNetworkNodePolicy
  metadata:
    name: sriov-nnp-du-fh
    namespace: openshift-sriov-network-operator
  spec:
    deviceType: netdevice 6
    isRdma: true
    nicSelector:
      pfNames:
        - ens5f0 7
    nodeSelector:
      node-role.kubernetes.io/master: ""
    numVfs: 8 8
    priority: 10
    resourceName: du_fh

Procedure

• You can disable the Operator using the following configuration file. No configuration changes are needed. Use the provided settings:

  apiVersion: operator.openshift.io/v1
  kind: Console
  metadata:
    annotations:
      include.release.openshift.io/ibm-cloud-managed: "false"
      include.release.openshift.io/self-managed-high-availability: "false"
      include.release.openshift.io/single-node-developer: "false"
      release.openshift.io/create-only: "true"
    name: cluster
  spec:
    logLevel: Normal
    managementState: Removed
    operatorLogLevel: Normal

• Enable workload partitioning.
• Other MachineConfig objects – There is a set of MachineConfig custom resources (CRs) included by default. You can choose to include these additional MachineConfig CRs that are unique to their environment. It is recommended, but not required, to apply these CRs during installation in order to minimize the number of reboots that can occur during post-install configuration.

• After OpenShift Container Platform is installed on the cluster, use the following command to apply the CRs you configured for the distributed units (DUs):