Windows Container Support for OpenShift
Red Hat OpenShift for Windows Containers Guide
Abstract
Chapter 1. Red Hat OpenShift support for Windows Containers overview
Red Hat OpenShift support for Windows Containers is a feature providing the ability to run Windows compute nodes in an OpenShift Container Platform cluster. This is possible by using the Red Hat Windows Machine Config Operator (WMCO) to install and manage Windows nodes. With a Red Hat subscription, you can get support for running Windows workloads in OpenShift Container Platform. For more information, see the release notes.
For workloads including both Linux and Windows, OpenShift Container Platform allows you to deploy Windows workloads running on Windows Server containers while also providing traditional Linux workloads hosted on Red Hat Enterprise Linux CoreOS (RHCOS) or Red Hat Enterprise Linux (RHEL). For more information, see getting started with Windows container workloads.
You need the WMCO to run Windows workloads in your cluster. The WMCO orchestrates the process of deploying and managing Windows workloads on a cluster. For more information, see how to enable Windows container workloads.
You can create a Windows MachineSet object to create infrastructure Windows machine sets and related machines so that you can move supported Windows workloads to the new Windows machines. You can create a Windows MachineSet object on multiple platforms.
You can schedule Windows workloads to Windows compute nodes.
You can perform Windows Machine Config Operator upgrades to ensure that your Windows nodes have the latest updates.
You can remove a Windows node by deleting a specific machine.
You can disable Windows container workloads by performing the following:
- Uninstalling the Windows Machine Config Operator
- Deleting the Windows Machine Config Operator namespace
Chapter 2. Windows Container Support for Red Hat OpenShift release notes
2.1. About Windows Container Support for Red Hat OpenShift
Red Hat OpenShift support for Windows Containers enables running Windows compute nodes in an OpenShift Container Platform cluster. Running Windows workloads is possible by using the Red Hat Windows Machine Config Operator (WMCO) to install and manage Windows nodes. With Windows nodes available, you can run Windows container workloads in OpenShift Container Platform.
The release notes for Red Hat OpenShift support for Windows Containers tracks the development of the WMCO, which provides all Windows container workload capabilities in OpenShift Container Platform.
2.2. Getting support
Windows Container Support for Red Hat OpenShift is provided and available as an optional, installable component. Windows Container Support for Red Hat OpenShift is not part of the OpenShift Container Platform subscription. It requires an additional Red Hat subscription and is supported according to the Scope of coverage and Service level agreements.
You must have a separate subscription to receive support for Windows Container Support for Red Hat OpenShift. Without this additional Red Hat subscription, deploying Windows container workloads in production clusters is not supported. You can request support through the Red Hat Customer Portal.
For more information, see the Red Hat OpenShift Container Platform Life Cycle Policy document for Red Hat OpenShift support for Windows Containers.
If you do not have this additional Red Hat subscription, you can use the Community Windows Machine Config Operator, which is a distribution that lacks official support.
2.3. Windows Machine Config Operator prerequisites
The following information details the supported cloud provider versions, Windows Server versions, and networking configurations for the Windows Machine Config Operator. See the vSphere documentation for any information that is relevant to only that platform.
2.3.1. Supported cloud providers based on OpenShift Container Platform and WMCO versions
| Cloud provider | Supported OpenShift Container Platform version | Supported WMCO version |
|---|---|---|
| Amazon Web Services (AWS) | 4.6+ | WMCO 1.0+ |
| Microsoft Azure | 4.6+ | WMCO 1.0+ |
| VMware vSphere | 4.7+ | WMCO 2.0+ |
2.3.2. Supported Windows Server versions
The following table lists the supported Windows Server version based on the applicable cloud provider. Any unlisted Windows Server version is not supported and will cause errors. To prevent these errors, only use the appropriate version according to the cloud provider in use.
| Cloud provider | Supported Windows Server version |
|---|---|
| Amazon Web Services (AWS) | Windows Server Long-Term Servicing Channel (LTSC): Windows Server 2019 |
| Microsoft Azure | Windows Server Long-Term Servicing Channel (LTSC): Windows Server 2019 |
| VMware vSphere | Windows Server Semi-Annual Channel (SAC): Windows Server 20H2 |
2.3.3. Supported networking
Hybrid networking with OVN-Kubernetes is the only supported networking configuration. See the additional resources below for more information on this functionality. The following tables outline the type of networking configuration and Windows Server versions to use based on your cloud provider. You must specify the network configuration when you install the cluster. Be aware that OpenShift SDN networking is the default network for OpenShift Container Platform clusters. However, OpenShift SDN is not supported by WMCO.
| Cloud provider | Supported networking |
|---|---|
| Amazon Web Services (AWS) | Hybrid networking with OVN-Kubernetes |
| Microsoft Azure | Hybrid networking with OVN-Kubernetes |
| VMware vSphere | Hybrid networking with OVN-Kubernetes with a custom VXLAN port |
| Hybrid networking with OVN-Kubernetes | Supported Windows Server version |
|---|---|
| Default VXLAN port | Windows Server Long-Term Servicing Channel (LTSC): Windows Server 2019 |
| Custom VXLAN port | Windows Server Semi-Annual Channel (SAC): Windows Server 20H2 |
2.3.4. Supported installation method
The installer-provisioned infrastructure installation method is the only supported installation method. This is consistent across all supported cloud providers. User-provisioned infrastructure installation method is not supported.
2.4. Release notes for Red Hat Windows Machine Config Operator 2.0.5
Issued: 2022-05-02
The WMCO 2.0.5 is now available with bug fixes. The components of the WMCO were released in link:RHSA-2022:93074.
2.5. Release notes for Red Hat Windows Machine Config Operator 2.0.4
Issued: 2022-02-16
The WMCO 2.0.4 is now available with bug fixes. The components of the WMCO were released in RHBA-2022:0241.
2.5.1. Bug fixes
-
For clusters installed on VMware vSphere, the WMCO ignored the
Deletingphase notification event, leaving incorrect node information in thewindows-exportermetrics endpoint. This resulted in an invalid mapping for the Prometheus metrics endpoint. This bug has been fixed; the WMCO now recognizes theDeletingphase notification event and maps the Prometheus metrics endpoint appropriately. (BZ#1995340)
2.6. Release notes for Red Hat Windows Machine Config Operator 2.0.3
Issued: 2021-07-28
The WMCO 2.0.3 is now available with bug fixes. The components of the WMCO were released in RHBA-2021:2926.
2.6.1. Bug fixes
- This WMCO release fixes a bug that prevented users from upgrading to WMCO 3.0.0. Users should upgrade to WMCO 2.0.3 before upgrading to OpenShift Container Platform 4.8, which only supports WMCO 3.0.0. (BZ#1985349)
2.7. Release notes for Red Hat Windows Machine Config Operator 2.0.2
Issued: 2021-07-08
The WMCO 2.0.2 is now available with bug fixes. The components of the WMCO were released in RHBA-2021:2671.
Users who are running a version of WMCO prior to 2.0.3 should first upgrade to WMCO 2.0.3 prior to upgrading to WMCO 3.0.0. (BZ#1983153)
2.7.1. Bug fixes
-
OpenShift Container Platform 4.8 enables the
BoundServiceAccountTokenVolumeoption by default. This option attaches the projected volumes to all of the pods. In addition, OpenShift Container Platform 4.8 adds theRunAsUseroption to theSecurityContext. This combination results in Windows pods being stuck in theContainerCreatingstatus. To work around this issue, you should upgrade to WMCO 2.0.2 before upgrading your cluster to OpenShift Container Platform 4.8. (BZ#1975553)
2.8. Release notes for Red Hat Windows Machine Config Operator 2.0.1
Issued: 2021-06-23
The WMCO 2.0.1 is now available with bug fixes. The components of the WMCO were released in RHSA-2021:2130.
2.8.1. New features and improvements
This release adds the following new features and improvements.
2.8.1.1. Increased image pull time-out duration
Image pull time-out has been increased to 30 minutes.
2.8.1.2. Autoscaling for Windows instances
Cluster autoscaling is now supported for Windows instances. You can complete the following actions for Windows nodes:
- Define and deploy a cluster autoscaler.
- Create a Windows node using a Windows machine set.
- Define and deploy a machine autoscaler, referencing a Windows machine set.
2.8.2. Bug fixes
- Previously, when using the Windows kube-proxy component on an AWS installation, when you created a LoadBalancer service, packets would be misrouted and reached an unintended destination. Now, packets are no longer wrongly routed to unintended destinations. (BZ#1946538)
-
Previously, Windows nodes were not reporting some key node-level metrics via telemetry monitoring. The
windows_exporterreports various metrics aswindows_*instead of thenode_exporterequivalent ofnode_*. Now, the telemetry results cover all of the expected metrics. (BZ#1955319) -
Previously, when the WMCO configured Windows instances, if the hybrid-overlay or kube-proxy components failed, the node might report itself as
Ready. Now, the error is detected and the node reports itself asNotReady. (BZ#1956412) - Previously, the kube-proxy service would terminate unexpectedly after the load balancer is created if you created the load balancer after the Windows pods begin running. Now, the kube-proxy service does not crash when recreating the load balancer service. (BZ#1939968)
2.8.3. RHSA-2021:2130 - Windows Container support for OpenShift Container Platform security update
As part of the previously noted bug fix (BZ#1946538), an update for Windows kube-proxy is now available for Red Hat Windows Machine Config Operator 2.0.1. Details of the update are documented in the RHSA-2021:2130 advisory.
2.9. Release notes for Red Hat Windows Machine Config Operator 2.0.0
This release of the WMCO provides bug fixes and enhancements for running Windows compute nodes in an OpenShift Container Platform cluster. The components of the WMCO 2.0.0 were released in RHBA-2021:0440.
Running Windows container workloads is not supported for clusters in a restricted network or disconnected environment.
Version 2.x of the WMCO is only compatible with OpenShift Container Platform 4.7.
2.9.1. New features and improvements
This release adds the following new features and improvements.
2.9.1.1. Support for clusters running on VMware vSphere
You can now run Windows nodes on a cluster installed on VMware vSphere version 6.5, 6.7, or 7.0. You can create a Windows MachineSet object on vSphere to host Windows Server compute nodes. For more information, see Creating a Windows MachineSet object on vSphere.
2.9.1.2. Enhanced Windows node monitoring
Windows nodes are now fully integrated with most of the monitoring capabilities provided by the web console. However, it is not possible to view workload graphs for pods running on Windows nodes in this release.
2.10. Known issues
-
When you create Windows pods with
RunAsUserNameset in its "SecurityContext" with a projected volume associated with these pods, the file ownership permissions for the projected entities are ignored, resulting in incorrectly configured ownership permissions. - The filesystem graphs available in the web console do not display for Windows nodes. This is caused by changes in the filesystem queries. This will be fixed in a future release of WMCO. (BZ#1930347)
- The Prometheus windows_exporter used by the WMCO currently collects metrics through HTTP, so it is considered unsafe. You must ensure that only trusted users can retrieve metrics from the endpoint. The windows_exporter feature recently added support for HTTPS configuration, but this configuration has not been implemented for WMCO. Support for HTTPS configuration in the WMCO will be added in a future release.
When the
RunAsUserpermission is set in the security context of a Linux-based pod, the projected files have the correct permissions set, including container user ownership. However, when the Windows equivalentRunAsUsernamepermission is set in a Windows pod, the kubelet is prevented from setting correct ownership on the files in the projected volume. This problem can get exacerbated when used in conjunction with a hostPath volume where best practices are not followed. For example, giving a pod access to theC:\var\lib\kubelet\pods\folder results in that pod being able to access service account tokens from other pods.By default, the projected files will have the following ownership, as shown in this example Windows projected volume file:
Path : Microsoft.PowerShell.Core\FileSystem::C:\var\run\secrets\kubernetes.io\serviceaccount\..2021_08_31_22_22_18.318230061\ca.crt Owner : BUILTIN\Administrators Group : NT AUTHORITY\SYSTEM Access : NT AUTHORITY\SYSTEM Allow FullControl BUILTIN\Administrators Allow FullControl BUILTIN\Users Allow ReadAndExecute, Synchronize Audit : Sddl : O:BAG:SYD:AI(A;ID;FA;;;SY)(A;ID;FA;;;BA)(A;ID;0x1200a9;;;BU)This indicates all administrator users, such as someone with the
ContainerAdministratorrole, have read, write, and execute access, while non-administrator users have read and execute access.ImportantOpenShift Container Platform applies the
RunAsUsersecurity context to all pods irrespective of its operating system. This means Windows pods automatically have theRunAsUserpermission applied to its security context.In addition, if a Windows pod is created with a projected volume with the default
RunAsUserpermission set, the pod remains in theContainerCreatingphase.To handle these issues, OpenShift Container Platform forces the file permission handling in projected service account volumes set in the security context of the pod to not be honored for projected volumes on Windows. Note that this behavior for Windows pods is how file permission handling used to work for all pod types prior to OpenShift Container Platform 4.7. (BZ#1971745)
2.11. Known limitations
Note the following limitations when working with Windows nodes managed by the WMCO (Windows nodes):
The following OpenShift Container Platform features are not supported on Windows nodes:
- Red Hat OpenShift Developer CLI (odo)
- Image builds
- OpenShift Pipelines
- OpenShift Service Mesh
- OpenShift monitoring of user-defined projects
- OpenShift Serverless
- Horizontal Pod Autoscaling
- Vertical Pod Autoscaling
The following Red Hat features are not supported on Windows nodes:
- Windows nodes do not support pulling container images from private registries. You can use images from public registries or pre-pull the images.
- Windows nodes do not support workloads created by using deployment configs. You can use a deployment or other method to deploy workloads.
- Windows nodes are not supported in clusters that use a cluster-wide proxy. This is because the WMCO is not able to route traffic through the proxy connection for the workloads.
- Windows nodes are not supported in clusters that are in a disconnected environment.
- Red Hat OpenShift support for Windows Containers supports only in-tree storage drivers for all cloud providers.
Kubernetes has identified the following node feature limitations :
- Huge pages are not supported for Windows containers.
- Privileged containers are not supported for Windows containers.
- Pod termination grace periods require the containerd container runtime to be installed on the Windows node.
- Kubernetes has identified several API compatibility issues.
Chapter 3. Understanding Windows container workloads
Red Hat OpenShift support for Windows Containers provides built-in support for running Microsoft Windows Server containers on OpenShift Container Platform. For those that administer heterogeneous environments with a mix of Linux and Windows workloads, OpenShift Container Platform allows you to deploy Windows workloads running on Windows Server containers while also providing traditional Linux workloads hosted on Red Hat Enterprise Linux CoreOS (RHCOS) or Red Hat Enterprise Linux (RHEL).
Multi-tenancy for clusters that have Windows nodes is not supported. Hostile multi-tenant usage introduces security concerns in all Kubernetes environments. Additional security features like pod security policies, or more fine-grained role-based access control (RBAC) for nodes, make exploits more difficult. However, if you choose to run hostile multi-tenant workloads, a hypervisor is the only security option you should use. The security domain for Kubernetes encompasses the entire cluster, not an individual node. For these types of hostile multi-tenant workloads, you should use physically isolated clusters.
Windows Server Containers provide resource isolation using a shared kernel but are not intended to be used in hostile multitenancy scenarios. Scenarios that involve hostile multitenancy should use Hyper-V Isolated Containers to strongly isolate tenants.
3.1. Windows Machine Config Operator prerequisites
The following information details the supported cloud provider versions, Windows Server versions, and networking configurations for the Windows Machine Config Operator. See the vSphere documentation for any information that is relevant to only that platform.
3.1.1. Supported cloud providers based on OpenShift Container Platform and WMCO versions
| Cloud provider | Supported OpenShift Container Platform version | Supported WMCO version |
|---|---|---|
| Amazon Web Services (AWS) | 4.6+ | WMCO 1.0+ |
| Microsoft Azure | 4.6+ | WMCO 1.0+ |
| VMware vSphere | 4.7+ | WMCO 2.0+ |
3.1.2. Supported Windows Server versions
The following table lists the supported Windows Server version based on the applicable cloud provider. Any unlisted Windows Server version is not supported and will cause errors. To prevent these errors, only use the appropriate version according to the cloud provider in use.
| Cloud provider | Supported Windows Server version |
|---|---|
| Amazon Web Services (AWS) | Windows Server Long-Term Servicing Channel (LTSC): Windows Server 2019 |
| Microsoft Azure | Windows Server Long-Term Servicing Channel (LTSC): Windows Server 2019 |
| VMware vSphere | Windows Server Semi-Annual Channel (SAC): Windows Server 20H2 |
3.1.3. Supported networking
Hybrid networking with OVN-Kubernetes is the only supported networking configuration. See the additional resources below for more information on this functionality. The following tables outline the type of networking configuration and Windows Server versions to use based on your cloud provider. You must specify the network configuration when you install the cluster. Be aware that OpenShift SDN networking is the default network for OpenShift Container Platform clusters. However, OpenShift SDN is not supported by WMCO.
| Cloud provider | Supported networking |
|---|---|
| Amazon Web Services (AWS) | Hybrid networking with OVN-Kubernetes |
| Microsoft Azure | Hybrid networking with OVN-Kubernetes |
| VMware vSphere | Hybrid networking with OVN-Kubernetes with a custom VXLAN port |
| Hybrid networking with OVN-Kubernetes | Supported Windows Server version |
|---|---|
| Default VXLAN port | Windows Server Long-Term Servicing Channel (LTSC): Windows Server 2019 |
| Custom VXLAN port | Windows Server Semi-Annual Channel (SAC): Windows Server 20H2 |
3.1.4. Supported installation method
The installer-provisioned infrastructure installation method is the only supported installation method. This is consistent across all supported cloud providers. User-provisioned infrastructure installation method is not supported.
Additional resources
3.2. Windows workload management
To run Windows workloads in your cluster, you must first install the Windows Machine Config Operator (WMCO). The WMCO is a Linux-based Operator that runs on Linux-based control plane and compute nodes. The WMCO orchestrates the process of deploying and managing Windows workloads on a cluster.
Figure 3.1. WMCO design
Before deploying Windows workloads, you must create a Windows compute node and have it join the cluster. The Windows node hosts the Windows workloads in a cluster, and can run alongside other Linux-based compute nodes. You can create a Windows compute node by creating a Windows machine set to host Windows Server compute machines. You must apply a Windows-specific label to the machine set that specifies a Windows OS image that has the Docker-formatted container runtime add-on enabled.
Currently, the Docker-formatted container runtime is used in Windows nodes. Kubernetes is deprecating Docker as a container runtime; you can reference the Kubernetes documentation for more information in Docker deprecation. Containerd will be the new supported container runtime for Windows nodes in a future release of Kubernetes.
The WMCO watches for machines with the Windows label. After a Windows machine set is detected and its respective machines are provisioned, the WMCO configures the underlying Windows virtual machine (VM) so that it can join the cluster as a compute node.
Figure 3.2. Mixed Windows and Linux workloads
The WMCO expects a predetermined secret in its namespace containing a private key that is used to interact with the Windows instance. WMCO checks for this secret during boot up time and creates a user data secret which you must reference in the Windows MachineSet object that you created. Then the WMCO populates the user data secret with a public key that corresponds to the private key. With this data in place, the cluster can connect to the Windows VM using an SSH connection.
After the cluster establishes a connection with the Windows VM, you can manage the Windows node using similar practices as you would a Linux-based node.
The OpenShift Container Platform web console provides most of the same monitoring capabilities for Windows nodes that are available for Linux nodes. However, the ability to monitor workload graphs for pods running on Windows nodes is not available at this time.
Scheduling Windows workloads to a Windows node can be done with typical pod scheduling practices like taints, tolerations, and node selectors; alternatively, you can differentiate your Windows workloads from Linux workloads and other Windows-versioned workloads by using a RuntimeClass object.
3.3. Windows node services
The following Windows-specific services are installed on each Windows node:
| Service | Description |
|---|---|
| kubelet | Registers the Windows node and manages its status. |
| Container Network Interface (CNI) plug-ins | Exposes networking for Windows nodes. |
| Windows Machine Config Bootstrapper (WMCB) | Configures the kubelet and CNI plug-ins. |
| Exports Prometheus metrics from Windows nodes | |
| hybrid-overlay | Creates the OpenShift Container Platform Host Network Service (HNS). |
| kube-proxy | Maintains network rules on nodes allowing outside communication. |
3.4. Known limitations
Note the following limitations when working with Windows nodes managed by the WMCO (Windows nodes):
The following OpenShift Container Platform features are not supported on Windows nodes:
- Red Hat OpenShift Developer CLI (odo)
- Image builds
- OpenShift Pipelines
- OpenShift Service Mesh
- OpenShift monitoring of user-defined projects
- OpenShift Serverless
- Horizontal Pod Autoscaling
- Vertical Pod Autoscaling
The following Red Hat features are not supported on Windows nodes:
- Windows nodes do not support pulling container images from private registries. You can use images from public registries or pre-pull the images.
- Windows nodes do not support workloads created by using deployment configs. You can use a deployment or other method to deploy workloads.
- Windows nodes are not supported in clusters that use a cluster-wide proxy. This is because the WMCO is not able to route traffic through the proxy connection for the workloads.
- Windows nodes are not supported in clusters that are in a disconnected environment.
- Red Hat OpenShift support for Windows Containers supports only in-tree storage drivers for all cloud providers.
Kubernetes has identified the following node feature limitations :
- Huge pages are not supported for Windows containers.
- Privileged containers are not supported for Windows containers.
- Pod termination grace periods require the containerd container runtime to be installed on the Windows node.
- Kubernetes has identified several API compatibility issues.
Chapter 4. Enabling Windows container workloads
Before adding Windows workloads to your cluster, you must install the Windows Machine Config Operator (WMCO), which is available in the OpenShift Container Platform OperatorHub. The WMCO orchestrates the process of deploying and managing Windows workloads on a cluster.
Dual NIC is not supported on WMCO-managed Windows instances.
Prerequisites
-
You have access to an OpenShift Container Platform cluster using an account with
cluster-adminpermissions. -
You have installed the OpenShift CLI (
oc). - You have installed your cluster using installer-provisioned infrastructure. Clusters installed with user-provisioned infrastructure are not supported for Windows container workloads.
- You have configured hybrid networking with OVN-Kubernetes for your cluster. This must be completed during the installation of your cluster. For more information, see Configuring hybrid networking.
- You are running an OpenShift Container Platform cluster version 4.6.8 or later.
The WMCO is not supported in clusters that use a cluster-wide proxy because the WMCO is not able to route traffic through the proxy connection for the workloads.
Additional resources
- For the comprehensive prerequisites for the Windows Machine Config Operator, see Understanding Windows container workloads.
4.1. Installing the Windows Machine Config Operator
You can install the Windows Machine Config Operator using either the web console or OpenShift CLI (oc).
4.1.1. Installing the Windows Machine Config Operator using the web console
You can use the OpenShift Container Platform web console to install the Windows Machine Config Operator (WMCO).
Dual NIC is not supported on WMCO-managed Windows instances.
Procedure
- From the Administrator perspective in the OpenShift Container Platform web console, navigate to the Operators → OperatorHub page.
-
Use the Filter by keyword box to search for
Windows Machine Config Operatorin the catalog. Click the Windows Machine Config Operator tile. - Review the information about the Operator and click Install.
On the Install Operator page:
- Select the stable channel as the Update Channel. The stable channel enables the latest stable release of the WMCO to be installed.
- The Installation Mode is preconfigured because the WMCO must be available in a single namespace only.
-
Choose the Installed Namespace for the WMCO. The default Operator recommended namespace is
openshift-windows-machine-config-operator. - Click the Enable Operator recommended cluster monitoring on the Namespace checkbox to enable cluster monitoring for the WMCO.
Select an Approval Strategy.
- The Automatic strategy allows Operator Lifecycle Manager (OLM) to automatically update the Operator when a new version is available.
- The Manual strategy requires a user with appropriate credentials to approve the Operator update.
Click Install. The WMCO is now listed on the Installed Operators page.
NoteThe WMCO is installed automatically into the namespace you defined, like
openshift-windows-machine-config-operator.- Verify that the Status shows Succeeded to confirm successful installation of the WMCO.
4.1.2. Installing the Windows Machine Config Operator using the CLI
You can use the OpenShift CLI (oc) to install the Windows Machine Config Operator (WMCO).
Dual NIC is not supported on WMCO-managed Windows instances.
Procedure
Create a namespace for the WMCO.
Create a
Namespaceobject YAML file for the WMCO. For example,wmco-namespace.yaml:apiVersion: v1 kind: Namespace metadata: name: openshift-windows-machine-config-operator 1 labels: openshift.io/cluster-monitoring: "true" 2
Create the namespace:
$ oc create -f <file-name>.yaml
For example:
$ oc create -f wmco-namespace.yaml
Create the Operator group for the WMCO.
Create an
OperatorGroupobject YAML file. For example,wmco-og.yaml:apiVersion: operators.coreos.com/v1 kind: OperatorGroup metadata: name: windows-machine-config-operator namespace: openshift-windows-machine-config-operator spec: targetNamespaces: - openshift-windows-machine-config-operator
Create the Operator group:
$ oc create -f <file-name>.yaml
For example:
$ oc create -f wmco-og.yaml
Subscribe the namespace to the WMCO.
Create a
Subscriptionobject YAML file. For example,wmco-sub.yaml:apiVersion: operators.coreos.com/v1alpha1 kind: Subscription metadata: name: windows-machine-config-operator namespace: openshift-windows-machine-config-operator spec: channel: "stable" 1 installPlanApproval: "Automatic" 2 name: "windows-machine-config-operator" source: "redhat-operators" 3 sourceNamespace: "openshift-marketplace" 4
- 1
- Specify
stableas the channel. - 2
- Set an approval strategy. You can set
AutomaticorManual. - 3
- Specify the
redhat-operatorscatalog source, which contains thewindows-machine-config-operatorpackage manifests. If your OpenShift Container Platform is installed on a restricted network, also known as a disconnected cluster, specify the name of theCatalogSourceobject you created when you configured the Operator LifeCycle Manager (OLM). - 4
- Namespace of the catalog source. Use
openshift-marketplacefor the default OperatorHub catalog sources.
Create the subscription:
$ oc create -f <file-name>.yaml
For example:
$ oc create -f wmco-sub.yaml
The WMCO is now installed to the
openshift-windows-machine-config-operator.
Verify the WMCO installation:
$ oc get csv -n openshift-windows-machine-config-operator
Example output
NAME DISPLAY VERSION REPLACES PHASE windows-machine-config-operator.2.0.0 Windows Machine Config Operator 2.0.0 Succeeded
4.2. Configuring a secret for the Windows Machine Config Operator
To run the Windows Machine Config Operator (WMCO), you must create a secret in the WMCO namespace containing a private key. This is required to allow the WMCO to communicate with the Windows virtual machine (VM).
Prerequisites
- You installed the Windows Machine Config Operator (WMCO) using Operator Lifecycle Manager (OLM).
- You created a PEM-encoded file containing an RSA key.
Procedure
Define the secret required to access the Windows VMs:
$ oc create secret generic cloud-private-key --from-file=private-key.pem=${HOME}/.ssh/<key> \ -n openshift-windows-machine-config-operator 1
- 1
- You must create the private key in the WMCO namespace, like
openshift-windows-machine-config-operator.
It is recommended to use a different private key than the one used when installing the cluster.
4.3. Additional resources
Chapter 5. Creating Windows MachineSet objects
5.1. Creating a Windows MachineSet object on AWS
You can create a Windows MachineSet object to serve a specific purpose in your OpenShift Container Platform cluster on Amazon Web Services (AWS). For example, you might create infrastructure Windows machine sets and related machines so that you can move supporting Windows workloads to the new Windows machines.
Prerequisites
- You installed the Windows Machine Config Operator (WMCO) using Operator Lifecycle Manager (OLM).
You are using a supported Windows Server as the operating system image with the Docker-formatted container runtime add-on enabled.
Use the following
awscommand to query valid AMI images:$ aws ec2 describe-images --region <aws region name> --filters "Name=name,Values=Windows_Server-2019*English*Full*Containers*" "Name=is-public,Values=true" --query "reverse(sort_by(Images, &CreationDate))[*].{name: Name, id: ImageId}" --output tableImportantCurrently, the Docker-formatted container runtime is used in Windows nodes. Kubernetes is deprecating Docker as a container runtime; you can reference the Kubernetes documentation for more information in Docker deprecation. Containerd will be the new supported container runtime for Windows nodes in a future release of Kubernetes.
5.1.1. Machine API overview
The Machine API is a combination of primary resources that are based on the upstream Cluster API project and custom OpenShift Container Platform resources.
For OpenShift Container Platform 4.7 clusters, the Machine API performs all node host provisioning management actions after the cluster installation finishes. Because of this system, OpenShift Container Platform 4.7 offers an elastic, dynamic provisioning method on top of public or private cloud infrastructure.
The two primary resources are:
- Machines
-
A fundamental unit that describes the host for a node. A machine has a
providerSpecspecification, which describes the types of compute nodes that are offered for different cloud platforms. For example, a machine type for a worker node on Amazon Web Services (AWS) might define a specific machine type and required metadata. - Machine sets
MachineSetresources are groups of machines. Machine sets are to machines as replica sets are to pods. If you need more machines or must scale them down, you change the replicas field on the machine set to meet your compute need.WarningControl plane machines cannot be managed by machine sets.
The following custom resources add more capabilities to your cluster:
- Machine autoscaler
-
The
MachineAutoscalerresource automatically scales machines in a cloud. You can set the minimum and maximum scaling boundaries for nodes in a specified machine set, and the machine autoscaler maintains that range of nodes. TheMachineAutoscalerobject takes effect after aClusterAutoscalerobject exists. BothClusterAutoscalerandMachineAutoscalerresources are made available by theClusterAutoscalerOperatorobject. - Cluster autoscaler
- This resource is based on the upstream cluster autoscaler project. In the OpenShift Container Platform implementation, it is integrated with the Machine API by extending the machine set API. You can set cluster-wide scaling limits for resources such as cores, nodes, memory, GPU, and so on. You can set the priority so that the cluster prioritizes pods so that new nodes are not brought online for less important pods. You can also set the scaling policy so that you can scale up nodes but not scale them down.
- Machine health check
-
The
MachineHealthCheckresource detects when a machine is unhealthy, deletes it, and, on supported platforms, makes a new machine.
In OpenShift Container Platform version 3.11, you could not roll out a multi-zone architecture easily because the cluster did not manage machine provisioning. Beginning with OpenShift Container Platform version 4.1, this process is easier. Each machine set is scoped to a single zone, so the installation program sends out machine sets across availability zones on your behalf. And then because your compute is dynamic, and in the face of a zone failure, you always have a zone for when you must rebalance your machines. The autoscaler provides best-effort balancing over the life of a cluster.
5.1.2. Sample YAML for a Windows MachineSet object on AWS
This sample YAML defines a Windows MachineSet object running on Amazon Web Services (AWS) that the Windows Machine Config Operator (WMCO) can react upon.
apiVersion: machine.openshift.io/v1beta1
kind: MachineSet
metadata:
labels:
machine.openshift.io/cluster-api-cluster: <infrastructure_id> 1
name: <infrastructure_id>-windows-worker-<zone> 2
namespace: openshift-machine-api
spec:
replicas: 1
selector:
matchLabels:
machine.openshift.io/cluster-api-cluster: <infrastructure_id> 3
machine.openshift.io/cluster-api-machineset: <infrastructure_id>-windows-worker-<zone> 4
template:
metadata:
labels:
machine.openshift.io/cluster-api-cluster: <infrastructure_id> 5
machine.openshift.io/cluster-api-machine-role: worker
machine.openshift.io/cluster-api-machine-type: worker
machine.openshift.io/cluster-api-machineset: <infrastructure_id>-windows-worker-<zone> 6
machine.openshift.io/os-id: Windows 7
spec:
metadata:
labels:
node-role.kubernetes.io/worker: "" 8
providerSpec:
value:
ami:
id: <windows_container_ami> 9
apiVersion: awsproviderconfig.openshift.io/v1beta1
blockDevices:
- ebs:
iops: 0
volumeSize: 120
volumeType: gp2
credentialsSecret:
name: aws-cloud-credentials
deviceIndex: 0
iamInstanceProfile:
id: <infrastructure_id>-worker-profile 10
instanceType: m5a.large
kind: AWSMachineProviderConfig
placement:
availabilityZone: <zone> 11
region: <region> 12
securityGroups:
- filters:
- name: tag:Name
values:
- <infrastructure_id>-worker-sg 13
subnet:
filters:
- name: tag:Name
values:
- <infrastructure_id>-private-<zone> 14
tags:
- name: kubernetes.io/cluster/<infrastructure_id> 15
value: owned
userDataSecret:
name: windows-user-data 16
namespace: openshift-machine-api- 1 3 5 10 13 14 15
- Specify the infrastructure ID that is based on the cluster ID that you set when you provisioned the cluster. You can obtain the infrastructure ID by running the following command:
$ oc get -o jsonpath='{.status.infrastructureName}{"\n"}' infrastructure cluster - 2 4 6
- Specify the infrastructure ID, worker label, and zone.
- 7
- Configure the machine set as a Windows machine.
- 8
- Configure the Windows node as a compute machine.
- 9
- Specify the AMI ID of a Windows image with a container runtime installed. You must use Windows Server 2019.
- 11
- Specify the AWS zone, like
us-east-1a. - 12
- Specify the AWS region, like
us-east-1. - 16
- Created by the WMCO when it is configuring the first Windows machine. After that, the
windows-user-datais available for all subsequent machine sets to consume.
5.1.3. Creating a machine set
In addition to the ones created by the installation program, you can create your own machine sets to dynamically manage the machine compute resources for specific workloads of your choice.
Prerequisites
- Deploy an OpenShift Container Platform cluster.
-
Install the OpenShift CLI (
oc). -
Log in to
ocas a user withcluster-adminpermission.
Procedure
Create a new YAML file that contains the machine set custom resource (CR) sample and is named
<file_name>.yaml.Ensure that you set the
<clusterID>and<role>parameter values.If you are not sure which value to set for a specific field, you can check an existing machine set from your cluster:
$ oc get machinesets -n openshift-machine-api
Example output
NAME DESIRED CURRENT READY AVAILABLE AGE agl030519-vplxk-worker-us-east-1a 1 1 1 1 55m agl030519-vplxk-worker-us-east-1b 1 1 1 1 55m agl030519-vplxk-worker-us-east-1c 1 1 1 1 55m agl030519-vplxk-worker-us-east-1d 0 0 55m agl030519-vplxk-worker-us-east-1e 0 0 55m agl030519-vplxk-worker-us-east-1f 0 0 55m
Check values of a specific machine set:
$ oc get machineset <machineset_name> -n \ openshift-machine-api -o yamlExample output
... template: metadata: labels: machine.openshift.io/cluster-api-cluster: agl030519-vplxk 1 machine.openshift.io/cluster-api-machine-role: worker 2 machine.openshift.io/cluster-api-machine-type: worker machine.openshift.io/cluster-api-machineset: agl030519-vplxk-worker-us-east-1a
Create the new
MachineSetCR:$ oc create -f <file_name>.yaml
View the list of machine sets:
$ oc get machineset -n openshift-machine-api
Example output
NAME DESIRED CURRENT READY AVAILABLE AGE agl030519-vplxk-windows-worker-us-east-1a 1 1 1 1 11m agl030519-vplxk-worker-us-east-1a 1 1 1 1 55m agl030519-vplxk-worker-us-east-1b 1 1 1 1 55m agl030519-vplxk-worker-us-east-1c 1 1 1 1 55m agl030519-vplxk-worker-us-east-1d 0 0 55m agl030519-vplxk-worker-us-east-1e 0 0 55m agl030519-vplxk-worker-us-east-1f 0 0 55m
When the new machine set is available, the
DESIREDandCURRENTvalues match. If the machine set is not available, wait a few minutes and run the command again.
5.1.4. Additional resources
- For more information on managing machine sets, see the Machine management section.
5.2. Creating a Windows MachineSet object on Azure
You can create a Windows MachineSet object to serve a specific purpose in your OpenShift Container Platform cluster on Microsoft Azure. For example, you might create infrastructure Windows machine sets and related machines so that you can move supporting Windows workloads to the new Windows machines.
Prerequisites
- You installed the Windows Machine Config Operator (WMCO) using Operator Lifecycle Manager (OLM).
- You are using a supported Windows Server as the operating system image with the Docker-formatted container runtime add-on enabled.
Currently, the Docker-formatted container runtime is used in Windows nodes. Kubernetes is deprecating Docker as a container runtime; you can reference the Kubernetes documentation for more information in Docker deprecation. Containerd will be the new supported container runtime for Windows nodes in a future release of Kubernetes.
5.2.1. Machine API overview
The Machine API is a combination of primary resources that are based on the upstream Cluster API project and custom OpenShift Container Platform resources.
For OpenShift Container Platform 4.7 clusters, the Machine API performs all node host provisioning management actions after the cluster installation finishes. Because of this system, OpenShift Container Platform 4.7 offers an elastic, dynamic provisioning method on top of public or private cloud infrastructure.
The two primary resources are:
- Machines
-
A fundamental unit that describes the host for a node. A machine has a
providerSpecspecification, which describes the types of compute nodes that are offered for different cloud platforms. For example, a machine type for a worker node on Amazon Web Services (AWS) might define a specific machine type and required metadata. - Machine sets
MachineSetresources are groups of machines. Machine sets are to machines as replica sets are to pods. If you need more machines or must scale them down, you change the replicas field on the machine set to meet your compute need.WarningControl plane machines cannot be managed by machine sets.
The following custom resources add more capabilities to your cluster:
- Machine autoscaler
-
The
MachineAutoscalerresource automatically scales machines in a cloud. You can set the minimum and maximum scaling boundaries for nodes in a specified machine set, and the machine autoscaler maintains that range of nodes. TheMachineAutoscalerobject takes effect after aClusterAutoscalerobject exists. BothClusterAutoscalerandMachineAutoscalerresources are made available by theClusterAutoscalerOperatorobject. - Cluster autoscaler
- This resource is based on the upstream cluster autoscaler project. In the OpenShift Container Platform implementation, it is integrated with the Machine API by extending the machine set API. You can set cluster-wide scaling limits for resources such as cores, nodes, memory, GPU, and so on. You can set the priority so that the cluster prioritizes pods so that new nodes are not brought online for less important pods. You can also set the scaling policy so that you can scale up nodes but not scale them down.
- Machine health check
-
The
MachineHealthCheckresource detects when a machine is unhealthy, deletes it, and, on supported platforms, makes a new machine.
In OpenShift Container Platform version 3.11, you could not roll out a multi-zone architecture easily because the cluster did not manage machine provisioning. Beginning with OpenShift Container Platform version 4.1, this process is easier. Each machine set is scoped to a single zone, so the installation program sends out machine sets across availability zones on your behalf. And then because your compute is dynamic, and in the face of a zone failure, you always have a zone for when you must rebalance your machines. The autoscaler provides best-effort balancing over the life of a cluster.
5.2.2. Sample YAML for a Windows MachineSet object on Azure
This sample YAML defines a Windows MachineSet object running on Microsoft Azure that the Windows Machine Config Operator (WMCO) can react upon.
apiVersion: machine.openshift.io/v1beta1
kind: MachineSet
metadata:
labels:
machine.openshift.io/cluster-api-cluster: <infrastructure_id> 1
name: <windows_machine_set_name> 2
namespace: openshift-machine-api
spec:
replicas: 1
selector:
matchLabels:
machine.openshift.io/cluster-api-cluster: <infrastructure_id> 3
machine.openshift.io/cluster-api-machineset: <windows_machine_set_name> 4
template:
metadata:
labels:
machine.openshift.io/cluster-api-cluster: <infrastructure_id> 5
machine.openshift.io/cluster-api-machine-role: worker
machine.openshift.io/cluster-api-machine-type: worker
machine.openshift.io/cluster-api-machineset: <windows_machine_set_name> 6
machine.openshift.io/os-id: Windows 7
spec:
metadata:
labels:
node-role.kubernetes.io/worker: "" 8
providerSpec:
value:
apiVersion: azureproviderconfig.openshift.io/v1beta1
credentialsSecret:
name: azure-cloud-credentials
namespace: openshift-machine-api
image: 9
offer: WindowsServer
publisher: MicrosoftWindowsServer
resourceID: ""
sku: 2019-Datacenter-with-Containers
version: latest
kind: AzureMachineProviderSpec
location: <location> 10
managedIdentity: <infrastructure_id>-identity 11
networkResourceGroup: <infrastructure_id>-rg 12
osDisk:
diskSizeGB: 128
managedDisk:
storageAccountType: Premium_LRS
osType: Windows
publicIP: false
resourceGroup: <infrastructure_id>-rg 13
subnet: <infrastructure_id>-worker-subnet
userDataSecret:
name: windows-user-data 14
namespace: openshift-machine-api
vmSize: Standard_D2s_v3
vnet: <infrastructure_id>-vnet 15
zone: "<zone>" 16- 1 3 5 11 12 13 15
- Specify the infrastructure ID that is based on the cluster ID that you set when you provisioned the cluster. You can obtain the infrastructure ID by running the following command:
$ oc get -o jsonpath='{.status.infrastructureName}{"\n"}' infrastructure cluster - 2 4 6
- Specify the Windows machine set name. Windows machine names on Azure cannot be more than 15 characters long. Therefore, the machine set name cannot be more than 9 characters long, due to the way machine names are generated from it.
- 7
- Configure the machine set as a Windows machine.
- 8
- Configure the Windows node as a compute machine.
- 9
- Specify a
WindowsServerimage offering that defines the2019-Datacenter-with-ContainersSKU. - 10
- Specify the Azure region, like
centralus. - 14
- Created by the WMCO when it is configuring the first Windows machine. After that, the
windows-user-datais available for all subsequent machine sets to consume. - 16
- Specify the zone within your region to place machines on. Be sure that your region supports the zone that you specify.
5.2.3. Creating a machine set
In addition to the ones created by the installation program, you can create your own machine sets to dynamically manage the machine compute resources for specific workloads of your choice.
Prerequisites
- Deploy an OpenShift Container Platform cluster.
-
Install the OpenShift CLI (
oc). -
Log in to
ocas a user withcluster-adminpermission.
Procedure
Create a new YAML file that contains the machine set custom resource (CR) sample and is named
<file_name>.yaml.Ensure that you set the
<clusterID>and<role>parameter values.If you are not sure which value to set for a specific field, you can check an existing machine set from your cluster:
$ oc get machinesets -n openshift-machine-api
Example output
NAME DESIRED CURRENT READY AVAILABLE AGE agl030519-vplxk-worker-us-east-1a 1 1 1 1 55m agl030519-vplxk-worker-us-east-1b 1 1 1 1 55m agl030519-vplxk-worker-us-east-1c 1 1 1 1 55m agl030519-vplxk-worker-us-east-1d 0 0 55m agl030519-vplxk-worker-us-east-1e 0 0 55m agl030519-vplxk-worker-us-east-1f 0 0 55m
Check values of a specific machine set:
$ oc get machineset <machineset_name> -n \ openshift-machine-api -o yamlExample output
... template: metadata: labels: machine.openshift.io/cluster-api-cluster: agl030519-vplxk 1 machine.openshift.io/cluster-api-machine-role: worker 2 machine.openshift.io/cluster-api-machine-type: worker machine.openshift.io/cluster-api-machineset: agl030519-vplxk-worker-us-east-1a
Create the new
MachineSetCR:$ oc create -f <file_name>.yaml
View the list of machine sets:
$ oc get machineset -n openshift-machine-api
Example output
NAME DESIRED CURRENT READY AVAILABLE AGE agl030519-vplxk-windows-worker-us-east-1a 1 1 1 1 11m agl030519-vplxk-worker-us-east-1a 1 1 1 1 55m agl030519-vplxk-worker-us-east-1b 1 1 1 1 55m agl030519-vplxk-worker-us-east-1c 1 1 1 1 55m agl030519-vplxk-worker-us-east-1d 0 0 55m agl030519-vplxk-worker-us-east-1e 0 0 55m agl030519-vplxk-worker-us-east-1f 0 0 55m
When the new machine set is available, the
DESIREDandCURRENTvalues match. If the machine set is not available, wait a few minutes and run the command again.
5.2.4. Additional resources
- For more information on managing machine sets, see the Machine management section.
5.3. Creating a Windows MachineSet object on vSphere
You can create a Windows MachineSet object to serve a specific purpose in your OpenShift Container Platform cluster on VMware vSphere. For example, you might create infrastructure Windows machine sets and related machines so that you can move supporting Windows workloads to the new Windows machines.
Prerequisites
- You installed the Windows Machine Config Operator (WMCO) using Operator Lifecycle Manager (OLM).
- You are using a supported Windows Server as the operating system image with the Docker-formatted container runtime add-on enabled.
Currently, the Docker-formatted container runtime is used in Windows nodes. Kubernetes is deprecating Docker as a container runtime; you can reference the Kubernetes documentation for more information on Docker deprecation. Containerd will be the new supported container runtime for Windows nodes in a future release of Kubernetes.
5.3.1. Machine API overview
The Machine API is a combination of primary resources that are based on the upstream Cluster API project and custom OpenShift Container Platform resources.
For OpenShift Container Platform 4.7 clusters, the Machine API performs all node host provisioning management actions after the cluster installation finishes. Because of this system, OpenShift Container Platform 4.7 offers an elastic, dynamic provisioning method on top of public or private cloud infrastructure.
The two primary resources are:
- Machines
-
A fundamental unit that describes the host for a node. A machine has a
providerSpecspecification, which describes the types of compute nodes that are offered for different cloud platforms. For example, a machine type for a worker node on Amazon Web Services (AWS) might define a specific machine type and required metadata. - Machine sets
MachineSetresources are groups of machines. Machine sets are to machines as replica sets are to pods. If you need more machines or must scale them down, you change the replicas field on the machine set to meet your compute need.WarningControl plane machines cannot be managed by machine sets.
The following custom resources add more capabilities to your cluster:
- Machine autoscaler
-
The
MachineAutoscalerresource automatically scales machines in a cloud. You can set the minimum and maximum scaling boundaries for nodes in a specified machine set, and the machine autoscaler maintains that range of nodes. TheMachineAutoscalerobject takes effect after aClusterAutoscalerobject exists. BothClusterAutoscalerandMachineAutoscalerresources are made available by theClusterAutoscalerOperatorobject. - Cluster autoscaler
- This resource is based on the upstream cluster autoscaler project. In the OpenShift Container Platform implementation, it is integrated with the Machine API by extending the machine set API. You can set cluster-wide scaling limits for resources such as cores, nodes, memory, GPU, and so on. You can set the priority so that the cluster prioritizes pods so that new nodes are not brought online for less important pods. You can also set the scaling policy so that you can scale up nodes but not scale them down.
- Machine health check
-
The
MachineHealthCheckresource detects when a machine is unhealthy, deletes it, and, on supported platforms, makes a new machine.
In OpenShift Container Platform version 3.11, you could not roll out a multi-zone architecture easily because the cluster did not manage machine provisioning. Beginning with OpenShift Container Platform version 4.1, this process is easier. Each machine set is scoped to a single zone, so the installation program sends out machine sets across availability zones on your behalf. And then because your compute is dynamic, and in the face of a zone failure, you always have a zone for when you must rebalance your machines. The autoscaler provides best-effort balancing over the life of a cluster.
5.3.2. Preparing your vSphere environment for Windows container workloads
You must prepare your vSphere environment for Windows container workloads by creating the vSphere Windows VM golden image and enabling communication with the internal API server for the WMCO.
5.3.2.1. Creating the vSphere Windows VM golden image
Create a vSphere Windows virtual machine (VM) golden image.
Prerequisites
- You have created a private/public key pair, which is used to configure key-based authentication in the OpenSSH server. The private key must also be configured in the Windows Machine Config Operator (WMCO) namespace. This is required to allow the WMCO to communicate with the Windows VM. See the "Configuring a secret for the Windows Machine Config Operator" section for more details.
You must use Microsoft PowerShell commands in several cases when creating your Windows VM. PowerShell commands in this guide are distinguished by the PS C:\> prefix.
Procedure
- Create a new VM in the vSphere client using the Windows Server Semi-Annual Channel (SAC): Windows Server 20H2 ISO image that includes the Microsoft patch KB4565351. This patch is required to set the VXLAN UDP port, which is required for clusters installed on vSphere. See the VMware documentation for more information.
- Install and configure VMware Tools version 11.0.6 or greater on the Windows VM. See the VMware Tools documentation for more information.
After installing VMware Tools on the Windows VM, verify the following:
The
C:\ProgramData\VMware\VMware Tools\tools.conffile exists with the following entry:exclude-nics=
If the
tools.conffile does not exist, create it with theexclude-nicsoption uncommented and set as an empty value.This entry ensures the cloned vNIC generated on the Windows VM by the hybrid-overlay is not ignored.
The Windows VM has a valid IP address in vCenter:
C:\> ipconfig
The VMTools Windows service is running:
PS C:\> Get-Service -Name VMTools | Select Status, StartType
- Install and configure the OpenSSH Server on the Windows VM. See Microsoft’s documentation on installing OpenSSH for more details.
Set up SSH access for an administrative user. See Microsoft’s documentation on the Administrative user to do this.
ImportantThe public key used in the instructions must correspond to the private key you create later in the WMCO namespace that holds your secret. See the "Configuring a secret for the Windows Machine Config Operator" section for more details.
-
Install the
dockercontainer runtime on your Windows VM following the Microsoft documentation. You must create a new firewall rule in the Windows VM that allows incoming connections for container logs. Run the following PowerShell command to create the firewall rule on TCP port 10250:
PS C:\> New-NetFirewallRule -DisplayName "ContainerLogsPort" -LocalPort 10250 -Enabled True -Direction Inbound -Protocol TCP -Action Allow -EdgeTraversalPolicy Allow
- Clone the Windows VM so it is a reusable image. Follow the VMware documentation on how to clone an existing virtual machine for more details.
In the cloned Windows VM, run the Windows Sysprep tool:
C:\> C:\Windows\System32\Sysprep\sysprep.exe /generalize /oobe /shutdown /unattend:<path_to_unattend.xml> 1- 1
- Specify the path to your
unattend.xmlfile.
NoteThere is a limit on how many times you can run the
sysprepcommand on a Windows image. Consult Microsoft’s documentation for more information.An example
unattend.xmlis provided, which maintains all the changes needed for the WMCO. You must modify this example; it cannot be used directly.Example 5.1. Example
unattend.xml<?xml version="1.0" encoding="UTF-8"?> <unattend xmlns="urn:schemas-microsoft-com:unattend"> <settings pass="specialize"> <component xmlns:wcm="http://schemas.microsoft.com/WMIConfig/2002/State" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" name="Microsoft-Windows-International-Core" processorArchitecture="amd64" publicKeyToken="31bf3856ad364e35" language="neutral" versionScope="nonSxS"> <InputLocale>0409:00000409</InputLocale> <SystemLocale>en-US</SystemLocale> <UILanguage>en-US</UILanguage> <UILanguageFallback>en-US</UILanguageFallback> <UserLocale>en-US</UserLocale> </component> <component xmlns:wcm="http://schemas.microsoft.com/WMIConfig/2002/State" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" name="Microsoft-Windows-Security-SPP-UX" processorArchitecture="amd64" publicKeyToken="31bf3856ad364e35" language="neutral" versionScope="nonSxS"> <SkipAutoActivation>true</SkipAutoActivation> </component> <component xmlns:wcm="http://schemas.microsoft.com/WMIConfig/2002/State" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" name="Microsoft-Windows-SQMApi" processorArchitecture="amd64" publicKeyToken="31bf3856ad364e35" language="neutral" versionScope="nonSxS"> <CEIPEnabled>0</CEIPEnabled> </component> <component xmlns:wcm="http://schemas.microsoft.com/WMIConfig/2002/State" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" name="Microsoft-Windows-Shell-Setup" processorArchitecture="amd64" publicKeyToken="31bf3856ad364e35" language="neutral" versionScope="nonSxS"> <ComputerName>winhost</ComputerName> 1 </component> </settings> <settings pass="oobeSystem"> <component xmlns:wcm="http://schemas.microsoft.com/WMIConfig/2002/State" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" name="Microsoft-Windows-Shell-Setup" processorArchitecture="amd64" publicKeyToken="31bf3856ad364e35" language="neutral" versionScope="nonSxS"> <AutoLogon> <Enabled>false</Enabled> 2 </AutoLogon> <OOBE> <HideEULAPage>true</HideEULAPage> <HideLocalAccountScreen>true</HideLocalAccountScreen> <HideOEMRegistrationScreen>true</HideOEMRegistrationScreen> <HideOnlineAccountScreens>true</HideOnlineAccountScreens> <HideWirelessSetupInOOBE>true</HideWirelessSetupInOOBE> <NetworkLocation>Work</NetworkLocation> <ProtectYourPC>1</ProtectYourPC> <SkipMachineOOBE>true</SkipMachineOOBE> <SkipUserOOBE>true</SkipUserOOBE> </OOBE> <RegisteredOrganization>Organization</RegisteredOrganization> <RegisteredOwner>Owner</RegisteredOwner> <DisableAutoDaylightTimeSet>false</DisableAutoDaylightTimeSet> <TimeZone>Eastern Standard Time</TimeZone> <UserAccounts> <AdministratorPassword> <Value>MyPassword</Value> 3 <PlainText>true</PlainText> </AdministratorPassword> </UserAccounts> </component> </settings> </unattend>- 1
- Specify the
ComputerName, which must follow the Kubernetes' names specification. These specifications also apply to Guest OS customization performed on the resulting template while creating new VMs. - 2
- Disable the automatic logon to avoid the security issue of leaving an open terminal with Administrator privileges at boot. This is the default value and must not be changed.
- 3
- Replace the
MyPasswordplaceholder with the password for the Administrator account. This prevents the built-in Administrator account from having a blank password by default. Follow Microsoft’s best practices for choosing a password.
After the Sysprep tool has completed, the Windows VM will power off. You must not use or power on this VM anymore.
- Convert the Windows VM to a template in vCenter.
5.3.2.1.1. Additional resources
5.3.2.2. Enabling communication with the internal API server for the WMCO on vSphere
The Windows Machine Config Operator (WMCO) downloads the Ignition config files from the internal API server endpoint. You must enable communication with the internal API server so that your Windows virtual machine (VM) can download the Ignition config files, and the kubelet on the configured VM can only communicate with the internal API server.
Prerequisites
- You have installed a cluster on vSphere.
Procedure
-
Add a new DNS entry for
api-int.<cluster_name>.<base_domain>that points to the external API server URLapi.<cluster_name>.<base_domain>. This can be a CNAME or an additional A record.
The external API endpoint was already created as part of the initial cluster installation on vSphere.
5.3.3. Sample YAML for a Windows MachineSet object on vSphere
This sample YAML defines a Windows MachineSet object running on VMware vSphere that the Windows Machine Config Operator (WMCO) can react upon.
apiVersion: machine.openshift.io/v1beta1
kind: MachineSet
metadata:
labels:
machine.openshift.io/cluster-api-cluster: <infrastructure_id> 1
name: <windows_machine_set_name> 2
namespace: openshift-machine-api
spec:
replicas: 1
selector:
matchLabels:
machine.openshift.io/cluster-api-cluster: <infrastructure_id> 3
machine.openshift.io/cluster-api-machineset: <windows_machine_set_name> 4
template:
metadata:
labels:
machine.openshift.io/cluster-api-cluster: <infrastructure_id> 5
machine.openshift.io/cluster-api-machine-role: worker
machine.openshift.io/cluster-api-machine-type: worker
machine.openshift.io/cluster-api-machineset: <windows_machine_set_name> 6
machine.openshift.io/os-id: Windows 7
spec:
metadata:
labels:
node-role.kubernetes.io/worker: "" 8
providerSpec:
value:
apiVersion: vsphereprovider.openshift.io/v1beta1
credentialsSecret:
name: vsphere-cloud-credentials
diskGiB: 128 9
kind: VSphereMachineProviderSpec
memoryMiB: 16384
network:
devices:
- networkName: "<vm_network_name>" 10
numCPUs: 4
numCoresPerSocket: 1
snapshot: ""
template: <windows_vm_template_name> 11
userDataSecret:
name: windows-user-data 12
workspace:
datacenter: <vcenter_datacenter_name> 13
datastore: <vcenter_datastore_name> 14
folder: <vcenter_vm_folder_path> 15
resourcePool: <vsphere_resource_pool> 16
server: <vcenter_server_ip> 17- 1 3 5
- Specify the infrastructure ID that is based on the cluster ID that you set when you provisioned the cluster. You can obtain the infrastructure ID by running the following command:
$ oc get -o jsonpath='{.status.infrastructureName}{"\n"}' infrastructure cluster - 2 4 6
- Specify the Windows machine set name. The machine set name cannot be more than 9 characters long, due to the way machine names are generated in vSphere.
- 7
- Configure the machine set as a Windows machine.
- 8
- Configure the Windows node as a compute machine.
- 9
- Specify the size of the vSphere Virtual Machine Disk (VMDK).Note
This parameter does not set the size of the Windows partition. You can resize the Windows partition by using the
unattend.xmlfile or by creating the vSphere Windows virtual machine (VM) golden image with the required disk size. - 10
- Specify the vSphere VM network to deploy the machine set to. This VM network must be where other Linux compute machines reside in the cluster.
- 11
- Specify the full path of the Windows vSphere VM template to use, such as
golden-images/windows-server-template. The name must be unique.ImportantDo not specify the original VM template. The VM template must remain off and must be cloned for new Windows machines. Starting the VM template configures the VM template as a VM on the platform, which prevents it from being used as a template that machine sets can apply configurations to.
- 12
- The
windows-user-datais created by the WMCO when the first Windows machine is configured. After that, thewindows-user-datais available for all subsequent machine sets to consume. - 13
- Specify the vCenter Datacenter to deploy the machine set on.
- 14
- Specify the vCenter Datastore to deploy the machine set on.
- 15
- Specify the path to the vSphere VM folder in vCenter, such as
/dc1/vm/user-inst-5ddjd. - 16
- Optional: Specify the vSphere resource pool for your Windows VMs.
- 17
- Specify the vCenter server IP or fully qualified domain name.
5.3.4. Creating a machine set
In addition to the ones created by the installation program, you can create your own machine sets to dynamically manage the machine compute resources for specific workloads of your choice.
Prerequisites
- Deploy an OpenShift Container Platform cluster.
-
Install the OpenShift CLI (
oc). -
Log in to
ocas a user withcluster-adminpermission.
Procedure
Create a new YAML file that contains the machine set custom resource (CR) sample and is named
<file_name>.yaml.Ensure that you set the
<clusterID>and<role>parameter values.If you are not sure which value to set for a specific field, you can check an existing machine set from your cluster:
$ oc get machinesets -n openshift-machine-api
Example output
NAME DESIRED CURRENT READY AVAILABLE AGE agl030519-vplxk-worker-us-east-1a 1 1 1 1 55m agl030519-vplxk-worker-us-east-1b 1 1 1 1 55m agl030519-vplxk-worker-us-east-1c 1 1 1 1 55m agl030519-vplxk-worker-us-east-1d 0 0 55m agl030519-vplxk-worker-us-east-1e 0 0 55m agl030519-vplxk-worker-us-east-1f 0 0 55m
Check values of a specific machine set:
$ oc get machineset <machineset_name> -n \ openshift-machine-api -o yamlExample output
... template: metadata: labels: machine.openshift.io/cluster-api-cluster: agl030519-vplxk 1 machine.openshift.io/cluster-api-machine-role: worker 2 machine.openshift.io/cluster-api-machine-type: worker machine.openshift.io/cluster-api-machineset: agl030519-vplxk-worker-us-east-1a
Create the new
MachineSetCR:$ oc create -f <file_name>.yaml
View the list of machine sets:
$ oc get machineset -n openshift-machine-api
Example output
NAME DESIRED CURRENT READY AVAILABLE AGE agl030519-vplxk-windows-worker-us-east-1a 1 1 1 1 11m agl030519-vplxk-worker-us-east-1a 1 1 1 1 55m agl030519-vplxk-worker-us-east-1b 1 1 1 1 55m agl030519-vplxk-worker-us-east-1c 1 1 1 1 55m agl030519-vplxk-worker-us-east-1d 0 0 55m agl030519-vplxk-worker-us-east-1e 0 0 55m agl030519-vplxk-worker-us-east-1f 0 0 55m
When the new machine set is available, the
DESIREDandCURRENTvalues match. If the machine set is not available, wait a few minutes and run the command again.
5.3.5. Additional resources
- For more information on managing machine sets, see the Machine management section.
Chapter 6. Scheduling Windows container workloads
You can schedule Windows workloads to Windows compute nodes.
The WMCO is not supported in clusters that use a cluster-wide proxy because the WMCO is not able to route traffic through the proxy connection for the workloads.
Prerequisites
- You installed the Windows Machine Config Operator (WMCO) using Operator Lifecycle Manager (OLM).
- You are using a Windows container as the OS image with the Docker-formatted container runtime add-on enabled.
- You have created a Windows machine set.
Currently, the Docker-formatted container runtime is used in Windows nodes. Kubernetes is deprecating Docker as a container runtime; you can reference the Kubernetes documentation for more information in Docker deprecation. Containerd will be the new supported container runtime for Windows nodes in a future release of Kubernetes.
6.1. Windows pod placement
Before deploying your Windows workloads to the cluster, you must configure your Windows node scheduling so pods are assigned correctly. Since you have a machine hosting your Windows node, it is managed the same as a Linux-based node. Likewise, scheduling a Windows pod to the appropriate Windows node is completed similarly, using mechanisms like taints, tolerations, and node selectors.
With multiple operating systems, and the ability to run multiple Windows OS variants in the same cluster, you must map your Windows pods to a base Windows OS variant by using a RuntimeClass object. For example, if you have multiple Windows nodes running on different Windows Server container versions, the cluster could schedule your Windows pods to an incompatible Windows OS variant. You must have RuntimeClass objects configured for each Windows OS variant on your cluster. Using a RuntimeClass object is also recommended if you have only one Windows OS variant available in your cluster.
For more information, see Microsoft’s documentation on Host and container version compatibility.
The container base image must be the same Windows OS version and build number that is running on the node where the conainer is to be scheduled.
Also, if you upgrade the Windows nodes from one version to another, for example going from 20H2 to 2022, you must upgrade your container base image to match the new version. For more information, see Windows container version compatibility.
Additional resources
6.2. Creating a RuntimeClass object to encapsulate scheduling mechanisms
Using a RuntimeClass object simplifies the use of scheduling mechanisms like taints and tolerations; you deploy a runtime class that encapsulates your taints and tolerations and then apply it to your pods to schedule them to the appropriate node. Creating a runtime class is also necessary in clusters that support multiple operating system variants.
Procedure
Create a
RuntimeClassobject YAML file. For example,runtime-class.yaml:apiVersion: node.k8s.io/v1beta1 kind: RuntimeClass metadata: name: <runtime_class_name> 1 handler: 'docker' scheduling: nodeSelector: 2 kubernetes.io/os: 'windows' kubernetes.io/arch: 'amd64' node.kubernetes.io/windows-build: '10.0.17763' tolerations: 3 - effect: NoSchedule key: os operator: Equal value: "Windows"
- 1
- Specify the
RuntimeClassobject name, which is defined in the pods you want to be managed by this runtime class. - 2
- Specify labels that must be present on nodes that support this runtime class. Pods using this runtime class can only be scheduled to a node matched by this selector. The node selector of the runtime class is merged with the existing node selector of the pod. Any conflicts prevent the pod from being scheduled to the node.
- 3
- Specify tolerations to append to pods, excluding duplicates, running with this runtime class during admission. This combines the set of nodes tolerated by the pod and the runtime class.
Create the
RuntimeClassobject:$ oc create -f <file-name>.yaml
For example:
$ oc create -f runtime-class.yaml
Apply the
RuntimeClassobject to your pod to ensure it is scheduled to the appropriate operating system variant:apiVersion: v1 kind: Pod metadata: name: my-windows-pod spec: runtimeClassName: <runtime_class_name> 1 ...- 1
- Specify the runtime class to manage the scheduling of your pod.
6.3. Sample Windows container workload deployment
You can deploy Windows container workloads to your cluster once you have a Windows compute node available.
This sample deployment is provided for reference only.
Example Service object
apiVersion: v1
kind: Service
metadata:
name: win-webserver
labels:
app: win-webserver
spec:
ports:
# the port that this service should serve on
- port: 80
targetPort: 80
selector:
app: win-webserver
type: LoadBalancer
Example Deployment object
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: win-webserver
name: win-webserver
spec:
selector:
matchLabels:
app: win-webserver
replicas: 1
template:
metadata:
labels:
app: win-webserver
name: win-webserver
spec:
tolerations:
- key: "os"
value: "Windows"
Effect: "NoSchedule"
containers:
- name: windowswebserver
image: mcr.microsoft.com/windows/servercore:ltsc2019
imagePullPolicy: IfNotPresent
command:
- powershell.exe
- -command
- $listener = New-Object System.Net.HttpListener; $listener.Prefixes.Add('http://*:80/'); $listener.Start();Write-Host('Listening at http://*:80/'); while ($listener.IsListening) { $context = $listener.GetContext(); $response = $context.Response; $content='<html><body><H1>Red Hat OpenShift + Windows Container Workloads</H1></body></html>'; $buffer = [System.Text.Encoding]::UTF8.GetBytes($content); $response.ContentLength64 = $buffer.Length; $response.OutputStream.Write($buffer, 0, $buffer.Length); $response.Close(); };
securityContext:
runAsNonRoot: false
windowsOptions:
runAsUserName: "ContainerAdministrator"
nodeSelector:
kubernetes.io/os: windows
When using the mcr.microsoft.com/powershell:<tag> container image, you must define the command as pwsh.exe. If you are using the mcr.microsoft.com/windows/servercore:<tag> container image, you must define the command as powershell.exe. For more information, see Microsoft’s documentation.
6.4. Scaling a machine set manually
To add or remove an instance of a machine in a machine set, you can manually scale the machine set.
This guidance is relevant to fully automated, installer-provisioned infrastructure installations. Customized, user-provisioned infrastructure installations do not have machine sets.
Prerequisites
-
Install an OpenShift Container Platform cluster and the
occommand line. -
Log in to
ocas a user withcluster-adminpermission.
Procedure
View the machine sets that are in the cluster:
$ oc get machinesets -n openshift-machine-api
The machine sets are listed in the form of
<clusterid>-worker-<aws-region-az>.View the machines that are in the cluster:
$ oc get machine -n openshift-machine-api
Set the annotation on the machine that you want to delete:
$ oc annotate machine/<machine_name> -n openshift-machine-api machine.openshift.io/cluster-api-delete-machine="true"
Cordon and drain the node that you want to delete:
$ oc adm cordon <node_name> $ oc adm drain <node_name>
Scale the machine set:
$ oc scale --replicas=2 machineset <machineset> -n openshift-machine-api
Or:
$ oc edit machineset <machineset> -n openshift-machine-api
You can scale the machine set up or down. It takes several minutes for the new machines to be available.
Verification
Verify the deletion of the intended machine:
$ oc get machines
Chapter 7. Windows node upgrades
You can ensure your Windows nodes have the latest updates by upgrading the Windows Machine Config Operator (WMCO).
7.1. Windows Machine Config Operator upgrades
When a new version of the Windows Machine Config Operator (WMCO) is released that is compatible with the current cluster version, the Operator is upgraded based on the upgrade channel and subscription approval strategy it was installed with when using the Operator Lifecycle Manager (OLM). The WMCO upgrade results in the Kubernetes components in the Windows machine being upgraded.
If you are upgrading to a new version of the WMCO and want to use cluster monitoring, you must have the openshift.io/cluster-monitoring=true label present in the WMCO namespace. If you add the label to a pre-existing WMCO namespace, and there are already Windows nodes configured, restart the WMCO pod to allow monitoring graphs to display.
For a non-disruptive upgrade, the WMCO terminates the Windows machines configured by the previous version of the WMCO and recreates them using the current version. This is done by deleting the Machine object, which results in the drain and deletion of the Windows node. To facilitate an upgrade, the WMCO adds a version annotation to all the configured nodes. During an upgrade, a mismatch in version annotation results in the deletion and recreation of a Windows machine. To have minimal service disruptions during an upgrade, the WMCO only updates one Windows machine at a time.
The WMCO is only responsible for updating Kubernetes components, not for Windows operating system updates. You provide the Windows image when creating the VMs; therefore, you are responsible for providing an updated image. You can provide an updated Windows image by changing the image configuration in the MachineSet spec.
For more information on Operator upgrades using the Operator Lifecycle Manager (OLM), see Upgrading installed Operators.
Chapter 8. Removing Windows nodes
You can remove a Windows node by deleting its host Windows machine.
8.1. Deleting a specific machine
You can delete a specific machine.
You cannot delete a control plane machine.
Prerequisites
- Install an OpenShift Container Platform cluster.
-
Install the OpenShift CLI (
oc). -
Log in to
ocas a user withcluster-adminpermission.
Procedure
View the machines that are in the cluster and identify the one to delete:
$ oc get machine -n openshift-machine-api
The command output contains a list of machines in the
<clusterid>-worker-<cloud_region>format.Delete the machine:
$ oc delete machine <machine> -n openshift-machine-api
ImportantBy default, the machine controller tries to drain the node that is backed by the machine until it succeeds. In some situations, such as with a misconfigured pod disruption budget, the drain operation might not be able to succeed in preventing the machine from being deleted. You can skip draining the node by annotating "machine.openshift.io/exclude-node-draining" in a specific machine. If the machine being deleted belongs to a machine set, a new machine is immediately created to satisfy the specified number of replicas.
Chapter 9. Disabling Windows container workloads
You can disable the capability to run Windows container workloads by uninstalling the Windows Machine Config Operator (WMCO) and deleting the namespace that was added by default when you installed the WMCO.
9.1. Uninstalling the Windows Machine Config Operator
You can uninstall the Windows Machine Config Operator (WMCO) from your cluster.
Prerequisites
-
Delete the Windows
Machineobjects hosting your Windows workloads.
Procedure
-
From the Operators → OperatorHub page, use the Filter by keyword box to search for
Red Hat Windows Machine Config Operator. - Click the Red Hat Windows Machine Config Operator tile. The Operator tile indicates it is installed.
- In the Windows Machine Config Operator descriptor page, click Uninstall.
9.2. Deleting the Windows Machine Config Operator namespace
You can delete the namespace that was generated for the Windows Machine Config Operator (WMCO) by default.
Prerequisites
- The WMCO is removed from your cluster.
Procedure
Remove all Windows workloads that were created in the
openshift-windows-machine-config-operatornamespace:$ oc delete --all pods --namespace=openshift-windows-machine-config-operator
Verify that all pods in the
openshift-windows-machine-config-operatornamespace are deleted or are reporting a terminating state:$ oc get pods --namespace openshift-windows-machine-config-operator
Delete the
openshift-windows-machine-config-operatornamespace:$ oc delete namespace openshift-windows-machine-config-operator
Additional resources
Legal Notice
Copyright © 2024 Red Hat, Inc.
OpenShift documentation is licensed under the Apache License 2.0 (https://www.apache.org/licenses/LICENSE-2.0).
Modified versions must remove all Red Hat trademarks.
Portions adapted from https://github.com/kubernetes-incubator/service-catalog/ with modifications by Red Hat.
Red Hat, Red Hat Enterprise Linux, the Red Hat logo, the Shadowman logo, JBoss, OpenShift, Fedora, the Infinity logo, and RHCE are trademarks of Red Hat, Inc., registered in the United States and other countries.
Linux® is the registered trademark of Linus Torvalds in the United States and other countries.
Java® is a registered trademark of Oracle and/or its affiliates.
XFS® is a trademark of Silicon Graphics International Corp. or its subsidiaries in the United States and/or other countries.
MySQL® is a registered trademark of MySQL AB in the United States, the European Union and other countries.
Node.js® is an official trademark of Joyent. Red Hat Software Collections is not formally related to or endorsed by the official Joyent Node.js open source or commercial project.
The OpenStack® Word Mark and OpenStack logo are either registered trademarks/service marks or trademarks/service marks of the OpenStack Foundation, in the United States and other countries and are used with the OpenStack Foundation’s permission. We are not affiliated with, endorsed or sponsored by the OpenStack Foundation, or the OpenStack community.
All other trademarks are the property of their respective owners.
