Chapter 22. Configuring for VMware vSphere

Procedure

1. Edit the master configuration file at /etc/origin/master/master-config.yaml by default on all masters and update the contents of the apiServerArguments and controllerArguments sections:

   kubernetesMasterConfig:
     ...
     apiServerArguments:
       cloud-provider:
         - "vsphere"
       cloud-config:
         - "/etc/origin/cloudprovider/vsphere.conf"
     controllerArguments:
       cloud-provider:
         - "vsphere"
       cloud-config:
         - "/etc/origin/cloudprovider/vsphere.conf"

   Important

   When triggering a containerized installation, only the /etc/origin and /var/lib/origin directories are mounted to the master and node container. Therefore, master-config.yaml must be in /etc/origin/master rather than /etc/.

2. When you configure OpenShift Container Platform for vSphere using Ansible, the /etc/origin/cloudprovider/vsphere.conf file is created automatically. Because you are manually configuring OpenShift Container Platform for vSphere, you must create the file. Before you create the file, decide if you want multiple vCenter zones or not.

   The cluster installation process configures single-zone or single vCenter by default. However, deploying OpenShift Container Platform in vSphere on different zones can be helpful to avoid single-point-of-failures, but creates the need for shared storage across zones. If an OpenShift Container Platform node host goes down in zone "A" and the pods should be moved to zone "B". See Multiple zone limitations in the Kubernetes documentation for more information.

   • To configure a single vCenter server, use the following format for the /etc/origin/cloudprovider/vsphere.conf file:

     [Global] 1
             user = "myusername" 2
             password = "mypassword" 3
             port = "443" 4
             insecure-flag = "1" 5
             datacenters = "mydatacenter" 6
     
     [VirtualCenter "10.10.0.2"] 7
             user = "myvCenterusername"
             password = "password"
     
     [Workspace] 8
             server = "10.10.0.2" 9
             datacenter = "mydatacenter"
             folder = "path/to/vms" 10
             default-datastore = "shared-datastore" 11
             resourcepool-path = "myresourcepoolpath" 12
     
     [Disk]
             scsicontrollertype = pvscsi 13
     
     [Network]
             public-network = "VM Network" 14

     1

     Any properties set in the [Global] section are used for all specified vcenters unless overriden by the settings in the individual [VirtualCenter] sections.

     2

     vCenter username for the vSphere cloud provider.

     3

     vCenter password for the specified user.

     4

     Optional. Port number for the vCenter server. Defaults to port 443.

     5

     Set to 1 if the vCenter uses a self-signed certificate.

     6

     Name of the data center on which Node VMs are deployed.

     7

     Override specific [Global] properties for this Virtual Center. Possible setting scan be [Port], [user], [insecure-flag], [datacenters]. Any settings not specified are pulled from the [Global] section.

     8

     Set any properties used for various vSphere Cloud Provider functionality. For example, dynamic provisioning, Storage Profile Based Volume provisioning, and others.

     9

     IP Address or FQDN for the vCenter server.

     10

     Path to the VM directory for node VMs.

     11

     Set to the name of the datastore to use for provisioning volumes using the storage classes or dynamic provisioning. Prior to OpenShift Container Platform 3.9, if the datastore was located in a storage directory or is a member of a datastore cluster, the full path was required.

     12

     Optional. Set to the path to the resource pool where dummy VMs for Storage Profile Based volume provisioning must be created.

     13

     Type of SCSI controller the VMDK will be attached to the VM as.

     14

     Set to the network port group for vSphere to access the node, which is called VM Network by default. This is the node host’s ExternalIP that is registered with Kubernetes.

   • To configure a multiple vCenter servers, use the following format for the /etc/origin/cloudprovider/vsphere.conf file:

     [Global] 1
             user = "myusername" 2
             password = "mypassword" 3
             port = "443" 4
             insecure-flag = "1" 5
             datacenters = "us-east, us-west" 6
     
     [VirtualCenter "10.10.0.2"] 7
             user = "myvCenterusername"
             password = "password"
     
     [VirtualCenter "10.10.0.3"]
             port = "448"
             insecure-flag = "0"
     
     [Workspace] 8
             server = "10.10.0.2" 9
             datacenter = "mydatacenter"
             folder = "path/to/vms" 10
             default-datastore = "shared-datastore" 11
             resourcepool-path = "myresourcepoolpath" 12
     
     [Disk]
             scsicontrollertype = pvscsi 13
     
     [Network]
             public-network = "VM Network" 14

     1

     Any properties set in the [Global] section are used for all specified vcenters unless overriden by the settings in the individual [VirtualCenter] sections.

     2

     vCenter username for the vSphere cloud provider.

     3

     vCenter password for the specified user.

     4

     Optional. Port number for the vCenter server. Defaults to port 443.

     5

     Set to 1 if the vCenter uses a self-signed certificate.

     6

     Name of the data centers on which Node VMs are deployed.

     7

     Override specific [Global] properties for this Virtual Center. Possible setting scan be [Port], [user], [insecure-flag], [datacenters]. Any settings not specified are pulled from the [Global] section.

     8

     Set any properties used for various vSphere Cloud Provider functionality. For example, dynamic provisioning, Storage Profile Based Volume provisioning, and others.

     9

     IP Address or FQDN for the vCenter server where the Cloud Provider communicates.

     10

     Path to the VM directory for node VMs.

     11

     Set to the name of the datastore to use for provisioning volumes using the storage classes or dynamic provisioning. Prior to OpenShift Container Platform 3.9, if the datastore was located in a storage directory or is a member of a datastore cluster, the full path was required.

     12

     Optional. Set to the path to the resource pool where dummy VMs for Storage Profile Based volume provisioning must be created.

     13

     Type of SCSI controller the VMDK will be attached to the VM as.

     14

     Set to the network port group for vSphere to access the node, which is called VM Network by default. This is the node host’s ExternalIP that is registered with Kubernetes.

     Important

     This ensures that the VMDK always presents a consistent UUID to the VM, allowing the disk to be mounted properly.

     For every virtual machine node that will be participating in the cluster: VM properties VM Options Advanced Configuration Parameters disk.enableUUID=TRUE

     Alternatively, the GOVC tool can be used:

     1. Set up the GOVC environment:

        export GOVC_URL='vCenter IP OR FQDN'
        export GOVC_USERNAME='vCenter User'
        export GOVC_PASSWORD='vCenter Password'
        export GOVC_INSECURE=1

3. Find the Node VM paths:

   govc ls /datacenter/vm/<vm-folder-name>

   1. Set disk.EnableUUID to true for all VMs:

      govc vm.change -e="disk.enableUUID=1" -vm='VM Path'

      Note

      If OpenShift Container Platform node VMs are created from a template VM, then disk.EnableUUID=1 can be set on the template VM. VMs cloned from this template inherit this property.

4. Restart the OpenShift Container Platform host services:

   # master-restart api
   # master-restart controllers
   # systemctl restart atomic-openshift-node

1. Edit the appropriate node configuration map and update the contents of the kubeletArguments section:

   kubeletArguments:
     cloud-provider:
       - "vsphere"
     cloud-config:
       - "/etc/origin/cloudprovider/vsphere.conf"

   Important

   The nodeName must match the VM name in vSphere in order for the cloud provider integration to work properly. The name must also be RFC1123 compliant.

2. Restart the OpenShift Container Platform services on all nodes.

   # systemctl restart atomic-openshift-node

1. Log in to the CLI as a cluster administrator.

2. Check and back up existing node labels:

   $ oc describe node <node_name> | grep -Poz '(?s)Labels.*\n.*(?=Taints)'

3. Delete the nodes:

   $ oc delete node <node_name>

4. On each node host, restart the OpenShift Container Platform service.

   # systemctl restart atomic-openshift-node

5. Add back any labels on each node that you previously had.

1. OpenShift Container Platform creates the following storageclass when you use the vsphere-volume provisioner and if you use the openshift_cloudprovider_kind=vsphere and openshift_vsphere_* variables in the Ansible inventory. Otherwise, you can create it manually:

   $ oc get --export storageclass vsphere-standard -o yaml
   kind: StorageClass
   apiVersion: storage.k8s.io/v1
   metadata:
     name: "vsphere-standard" 1
   provisioner: kubernetes.io/vsphere-volume 2
   parameters:
       diskformat: zeroedthick 3
       datastore: "ose3-vmware" 4
   reclaimPolicy: Delete

   1

   The name of the storage class.

   2

   The type of storage provisioner: vsphere-volume

   3

   The type of disk: zeroedthick, thin.

   4

   The source datastore where the disk will be created.

2. After you request a PV and using the storageclass shown in the previous step, OpenShift Container Platform creates VMDK disks in the vSphere infrastructure. To verify that the disks were created:

   $ ls /vmfs/volumes/ose3-vmware/kubevols | grep kubernetes
   kubernetes-dynamic-pvc-790615e8-a22a-11e8-bc85-0050568e2982.vmdk

Procedure

1. Create a new configuration file from the vSphere volume:

     kind: PersistentVolumeClaim
     apiVersion: v1
     metadata:
       name: vsphere-registry-storage
       annotations:
         volume.beta.kubernetes.io/storage-class: vsphere-standard
     spec:
       accessModes:
         - ReadWriteOnce
       resources:
         requests:
           storage: 30Gi

2. Create the file in OpenShift Container Platform:

   $ oc create -f pvc-registry.yaml

3. Update the volume configuration to use the new PVC:

   $ oc volume dc docker-registry --add --name=registry-storage -t \
   pvc --claim-name=vsphere-registry-storage --overwrite

4. Redeploy the registry to read the updated configuration:

   $ oc rollout latest docker-registry -n default

5. Verify the volume has been assigned:

   $ oc volume dc docker-registry -n default