Chapter 25. Configuring for VMware vSphere

Procedure

1. Create a VM folder and move OpenShift Container Platform Node VMs to this folder.

2. Set the disk.EnableUUID parameter to true for each Node VM. This setting ensures that the VMware vSphere’s Virtual Machine Disk (VMDK) always presents a consistent UUID to the VM, allowing the disk to be mounted properly.

   Every VM node that will be participating in the cluster must have the disk.EnableUUID parameter set to true. To set this value, follow the steps for either the vSphere console or govc CLI tool:

   1. From the vSphere HTML Client navigate to VM properties VM Options Advanced Configuration Parameters disk.enableUUID=TRUE

   2. Or using the govc CLI, find the Node VM paths:

      $govc ls /datacenter/vm/<vm-folder-name>

      1. Set disk.EnableUUID to true for all VMs:

         $govc vm.change -e="disk.enableUUID=1" -vm='VM Path'

Procedure

1. Edit the master configuration file at /etc/origin/master/master-config.yaml by default on all masters and update the contents of the apiServerArguments and controllerArguments sections:

   kubernetesMasterConfig:
     ...
     apiServerArguments:
       cloud-provider:
         - "vsphere"
       cloud-config:
         - "/etc/origin/cloudprovider/vsphere.conf"
     controllerArguments:
       cloud-provider:
         - "vsphere"
       cloud-config:
         - "/etc/origin/cloudprovider/vsphere.conf"

   Important

   When triggering a containerized installation, only the /etc/origin and /var/lib/origin directories are mounted to the master and node container. Therefore, master-config.yaml must be in /etc/origin/master rather than /etc/.

2. When you configure OpenShift Container Platform for vSphere using Ansible, the /etc/origin/cloudprovider/vsphere.conf file is created automatically. Because you are manually configuring OpenShift Container Platform for vSphere, you must create the file. Before you create the file, decide if you want multiple vCenter zones or not.

   The cluster installation process configures single-zone or single vCenter by default. However, deploying OpenShift Container Platform in vSphere on different zones can be helpful to avoid single-point-of-failures, but creates the need for shared storage across zones. If an OpenShift Container Platform node host goes down in zone "A" and the pods should be moved to zone "B". See Running in multiple zones in the Kubernetes documentation for more information.

   • To configure a single vCenter server, use the following format for the /etc/origin/cloudprovider/vsphere.conf file:

     [Global] 1
             user = "myusername" 2
             password = "mypassword" 3
             port = "443" 4
             insecure-flag = "1" 5
             datacenters = "mydatacenter" 6
     
     [VirtualCenter "10.10.0.2"] 7
             user = "myvCenterusername"
             password = "password"
     
     [Workspace] 8
             server = "10.10.0.2" 9
             datacenter = "mydatacenter"
             folder = "path/to/vms" 10
             default-datastore = "shared-datastore" 11
             resourcepool-path = "myresourcepoolpath" 12
     
     [Disk]
             scsicontrollertype = pvscsi 13
     
     [Network]
             public-network = "VM Network" 14

     1

     Any properties set in the [Global] section are used for all specified vcenters unless overriden by the settings in the individual [VirtualCenter] sections.

     2

     vCenter username for the vSphere cloud provider.

     3

     vCenter password for the specified user.

     4

     Optional. Port number for the vCenter server. Defaults to port 443.

     5

     Set to 1 if the vCenter uses a self-signed certificate.

     6

     Name of the data center on which Node VMs are deployed.

     7

     Override specific [Global] properties for this Virtual Center. Possible setting scan be [Port], [user], [insecure-flag], [datacenters]. Any settings not specified are pulled from the [Global] section.

     8

     Set any properties used for various vSphere Cloud Provider functionality. For example, dynamic provisioning, Storage Profile Based Volume provisioning, and others.

     9

     IP Address or FQDN for the vCenter server.

     10

     Path to the VM directory for node VMs.

     11

     Set to the name of the datastore to use for provisioning volumes using the storage classes or dynamic provisioning. Prior to OpenShift Container Platform 3.9, if the datastore was located in a storage directory or is a member of a datastore cluster, the full path was required.

     12

     Optional. Set to the path to the resource pool where dummy VMs for Storage Profile Based volume provisioning must be created.

     13

     Type of SCSI controller the VMDK will be attached to the VM as.

     14

     Set to the network port group for vSphere to access the node, which is called VM Network by default. This is the node host’s ExternalIP that is registered with Kubernetes.

   • To configure a multiple vCenter servers, use the following format for the /etc/origin/cloudprovider/vsphere.conf file:

     [Global] 1
             user = "myusername" 2
             password = "mypassword" 3
             port = "443" 4
             insecure-flag = "1" 5
             datacenters = "us-east, us-west" 6
     
     [VirtualCenter "10.10.0.2"] 7
             user = "myvCenterusername"
             password = "password"
     
     [VirtualCenter "10.10.0.3"]
             port = "448"
             insecure-flag = "0"
     
     [Workspace] 8
             server = "10.10.0.2" 9
             datacenter = "mydatacenter"
             folder = "path/to/vms" 10
             default-datastore = "shared-datastore" 11
             resourcepool-path = "myresourcepoolpath" 12
     
     [Disk]
             scsicontrollertype = pvscsi 13
     
     [Network]
             public-network = "VM Network" 14

     1

     Any properties set in the [Global] section are used for all specified vcenters unless overriden by the settings in the individual [VirtualCenter] sections.

     2

     vCenter username for the vSphere cloud provider.

     3

     vCenter password for the specified user.

     4

     Optional. Port number for the vCenter server. Defaults to port 443.

     5

     Set to 1 if the vCenter uses a self-signed certificate.

     6

     Name of the data centers on which Node VMs are deployed.

     7

     Override specific [Global] properties for this Virtual Center. Possible setting scan be [Port], [user], [insecure-flag], [datacenters]. Any settings not specified are pulled from the [Global] section.

     8

     Set any properties used for various vSphere Cloud Provider functionality. For example, dynamic provisioning, Storage Profile Based Volume provisioning, and others.

     9

     IP Address or FQDN for the vCenter server where the Cloud Provider communicates.

     10

     Path to the VM directory for node VMs.

     11

     Set to the name of the datastore to use for provisioning volumes using the storage classes or dynamic provisioning. Prior to OpenShift Container Platform 3.9, if the datastore was located in a storage directory or is a member of a datastore cluster, the full path was required.

     12

     Optional. Set to the path to the resource pool where dummy VMs for Storage Profile Based volume provisioning must be created.

     13

     Type of SCSI controller the VMDK will be attached to the VM as.

     14

     Set to the network port group for vSphere to access the node, which is called VM Network by default. This is the node host’s ExternalIP that is registered with Kubernetes.

3. Restart the OpenShift Container Platform host services:

   # master-restart api
   # master-restart controllers
   # systemctl restart atomic-openshift-node

1. Edit the appropriate node configuration map and update the contents of the kubeletArguments section:

   kubeletArguments:
     cloud-provider:
       - "vsphere"
     cloud-config:
       - "/etc/origin/cloudprovider/vsphere.conf"

   Important

   The nodeName must match the VM name in vSphere in order for the cloud provider integration to work properly. The name must also be RFC1123 compliant.

2. Restart the OpenShift Container Platform services on all nodes.

   # systemctl restart atomic-openshift-node

1. Log in to the CLI as a cluster administrator.

2. Check and back up existing node labels:

   $ oc describe node <node_name> | grep -Poz '(?s)Labels.*\n.*(?=Taints)'

3. Delete the nodes:

   $ oc delete node <node_name>

4. On each node host, restart the OpenShift Container Platform service.

   # systemctl restart atomic-openshift-node

5. Add back any labels on each node that you previously had.

1. Define a PV object definition, for example vsphere-pv.yaml:

   apiVersion: v1
   kind: PersistentVolume
   metadata:
     name: pv0001 1
   spec:
     capacity:
       storage: 2Gi 2
     accessModes:
       - ReadWriteOnce
     persistentVolumeReclaimPolicy: Retain
     vsphereVolume: 3
       volumePath: "[datastore1] volumes/myDisk" 4
       fsType: ext4 5

   1

   The name of the volume. This must be how it is identified by PV claims or from pods.

   2

   The amount of storage allocated to this volume.

   3

   The volume type being used. This example uses vsphereVolume. The label is used to mount a vSphere VMDK volume into pods. The contents of a volume are preserved when it is unmounted. The volume type supports VMFS and VSAN datastore.

   4

   The existing VMDK volume to use. You must enclose the datastore name in square brackets ([]) in the volume definition, as shown.

   5

   The file system type to mount. For example, ext4, xfs, or other file-systems.

   Important

   Changing the value of the fsType parameter after the volume is formatted and provisioned can result in data loss and pod failure.

2. Create the PV:

   $ oc create -f vsphere-pv.yaml
     persistentvolume "pv0001" created

3. Verify that the PV was created:

   $ oc get pv
   NAME    LABELS  CAPACITY  ACCESSMODES   STATUS    CLAIM   REASON  AGE
   pv0001  <none>  2Gi       RWO           Available                 2s