Chapter 1. Add-ons overview
Red Hat Advanced Cluster Management for Kubernetes add-ons can improve some areas of performance and add functionality to enhance your applications. The following sections provide a summary of the add-ons that are available for Red Hat Advanced Cluster Management:
1.1. Submariner multicluster networking and service discovery
Submariner is an open source tool that can be used with Red Hat Advanced Cluster Management for Kubernetes to provide direct networking and service discovery between two or more managed clusters in your environment, either on-premises or in the cloud. Submariner is compatible with Multi-Cluster Services API (Kubernetes Enhancements Proposal #1645). For more information about Submariner, see the Submariner site.
Make sure to see the Red Hat Advanced Cluster Management support matrix for more details about the support levels of infrastructure providers, including which providers support automated console deployments or require manual deployment.
See the following topics to learn more about how to use Submariner:
1.1.1. Deploying Submariner on disconnected clusters
Deploying Submariner on disconnected clusters can help with security concerns by reducing the risk of external attacks on clusters. To deploy Submariner with Red Hat Advanced Cluster Management for Kubernetes on disconnected clusters, you must first complete the steps outlined in Install in disconnected network environments.
1.1.1.1. Configuring Submariner on disconnected clusters
After following the steps outlined in Install in disconnected network environments, you must configure Submariner during the installation to support deployment on disconnected clusters. See the following topics:
1.1.1.1.1. Mirroring images in the local registry
Make sure to mirror the Submariner Operator bundle
image in the local registry before deploying Submariner on disconnected clusters.
Note: If you are using Red Hat Advanced Cluster Management 2.7.2 or older, you must also mirror the nettest-rhel8
image.
1.1.1.1.2. Customizing catalogSource names
By default, submariner-addon
searches for a catalogSource
with the name redhat-operators
. When using a catalogSource
with a different name, you must update the value of the SubmarinerConfig.Spec.subscriptionConfig.Source
parameter in the SubmarinerConfig
associated with your managed cluster with the custom name of the catalogSource
.
1.1.1.1.3. Enabling airGappedDeployment in SubmarinerConfig
When installing submariner-addon
on a managed cluster from the Red Hat Advanced Cluster Management for Kubernetes console, you can select the Disconnected cluster option so that Submariner does not make API queries to external servers.
If you are installing Submariner by using the APIs, you must set the airGappedDeployment
parameter to true
in the SubmarinerConfig
associated with your managed cluster.
1.1.2. Configuring Submariner
Red Hat Advanced Cluster Management for Kubernetes provides Submariner as an add-on for your hub cluster. You can find more information about Submariner in the Submariner open source project documentation.
1.1.2.1. Prerequisites
Ensure that you have the following prerequisites before using Submariner:
-
A credential to access the hub cluster with
cluster-admin
permissions. - IP connectivity must be configured between the gateway nodes. When connecting two clusters, at least one of the clusters must be accessible to the gateway node using its public or private IP address designated to the gateway node. See Submariner NAT Traversal for more information.
- If you are using OVN Kubernetes, clusters must be at Red Hat OpenShift Container Platform version 4.11 or later.
- If your Red Hat OpenShift Container Platform clusters use OpenShift SDN CNI, the firewall configuration across all nodes in each of the managed clusters must allow 4800/UDP in both directions.
- The firewall configuration must allow 4500/UDP and 4490/UDP on the gateway nodes for establishing tunnels between the managed clusters.
If the gateway nodes are directly reachable over their private IPs without any NAT in between, make sure that the firewall configuration allows the ESP protocol on the gateway nodes.
Note: This is configured automatically when your clusters are deployed in an Amazon Web Services, Google Cloud Platform, Microsoft Azure, or Red Hat OpenStack environment, but must be configured manually for clusters on other environments and for the firewalls that protect private clouds.
The
managedcluster
name must follow the DNS label standard as defined in RFC 1123 and meet the following requirements:- Contain 63 characters or fewer
- Contain only lowercase alphanumeric characters or '-'
- Start with an alphanumeric character
- End with an alphanumeric character
1.1.2.2. Submariner ports table
View the following table to see which Submariner ports you need to enable:
Name | Default value | Customizable | Optional or required |
---|---|---|---|
IPsec NATT | 4500/UDP | Yes | Required |
VXLAN | 4800/UDP | No | Required |
NAT discovery port | 4490/UDP | No | Required |
See the Submariner upstream prerequisites documentation for more detailed information about the prerequisites.
1.1.2.3. Globalnet
Globalnet is a feature included with the Submariner add-on which supports connectivity between clusters with overlapping CIDRs. Globalnet is a cluster set wide configuration, and can be selected when the first managed cluster is added to the cluster set. When Globalnet is enabled, each managed cluster is allocated a global CIDR from the virtual Global Private Network. The global CIDR is used for supporting inter-cluster communication.
If there is a chance that your clusters running Submariner might have overlapping CIDRs, consider enabling Globalnet. When using the console, the ClusterAdmin
can enable Globalnet for a cluster set by selecting the option Enable Globalnet when enabling the Submariner add-on for clusters in the cluster set. After you enable Globalnet, you cannot disable it without removing Submariner.
When using the Red Hat Advanced Cluster Management APIs, the ClusterAdmin
can enable Globalnet by creating a submariner-broker
object in the <ManagedClusterSet>-broker
namespace.
The ClusterAdmin
role has the required permissions to create this object in the broker namespace. The ManagedClusterSetAdmin
role, which is sometimes created to act as a proxy administrator for the cluster set, does not have the required permissions. To provide the required permissions, the ClusterAdmin
must associate the role permissions for the access-to-brokers-submariner-crd
to the ManagedClusterSetAdmin
user.
Complete the following steps to create the submariner-broker
object:
Retrieve the
<broker-namespace>
by running the following command:oc get ManagedClusterSet <cluster-set-name> -o jsonpath="{.metadata.annotations['cluster\.open-cluster-management\.io/submariner-broker-ns']}"
Create a
submariner-broker
object that specifies the Globalnet configuration by creating a YAML file namedsubmariner-broker
. Add content that resembles the following lines to the YAML file:apiVersion: submariner.io/v1alpha1 kind: Broker metadata: name: submariner-broker namespace: <broker-namespace> spec: globalnetEnabled: <true-or-false>
Replace
broker-namespace
with the name of your broker namespace.Replace
true-or-false
withtrue
to enable Globalnet.Note: The
metadata
name
parameter must besubmariner-broker
.Apply the file to your YAML file by entering the following command:
oc apply -f submariner-broker.yaml
For more information about Globalnet, see Globalnet controller in the Submariner documentation.
1.1.3. Installing the subctl command utility
The subctl
utility is shipped in a container image. Complete the following steps to install the subctl
utility locally:
Log in to the registry by running the following command and entering your credentials when prompted:
oc registry login --registry registry.redhat.io
Download the
subctl
container and extract a compressed version of thesubctl
binary to/tmp
by entering the following command:oc image extract registry.redhat.io/rhacm2/subctl-rhel8:v0.14 --path="/dist/subctl-v0.14*-linux-amd64.tar.xz":/tmp/ --confirm
Decompress the
subctl
utility by entering the following command:tar -C /tmp/ -xf /tmp/subctl-v0.14*-linux-amd64.tar.xz
Install the
subctl
utility by entering the following command:install -m744 /tmp/subctl-v0.14*/subctl-v0.14*-linux-amd64 /$HOME/.local/bin/subctl
1.1.3.1. Using the subctl commands
After adding the utility to your path, view the following table for a brief description of the available commands:
Creates a | |
Removes the | |
Provides information about Submariner resources. | |
Verifies connectivity, service discovery, and other Submariner features when Submariner is configured across a pair of clusters. | |
Benchmarks throughput and latency across a pair of clusters that are enabled with Submariner or within a single cluster. | |
Runs checks to identify issues that prevent the Submariner deployment from working correctly. | |
Collects information from the clusters to help troubleshoot a Submariner deployment. | |
Displays the version details of the |
For more information about the subctl
utility and its commands, see subctl
in the Submariner documentation.
1.1.4. Deploying Submariner by using the console
Before you deploy Submariner with Red Hat Advanced Cluster Management for Kubernetes, you must prepare the clusters on the hosting environment. You can use the SubmarinerConfig
API or the Red Hat Advanced Cluster Management for Kubernetes console to automatically prepare Red Hat OpenShift Container Platform clusters on the following providers:
- Amazon Web Services
- Google Cloud Platform
- Red Hat OpenStack Platform
- Microsoft Azure
- VMware vSphere
Note: Only non-NSX deployments are supported on VMware vSphere.
To deploy Submariner on other providers, follow the instructions in Deploying Submariner manually.
Complete the following steps to deploy Submariner with the Red Hat Advanced Cluster Management for Kubernetes console:
Required access: Cluster administrator
- From the console, select Infrastructure > Clusters.
- On the Clusters page, select the Cluster sets tab. The clusters that you want enable with Submariner must be in the same cluster set.
- If the clusters on which you want to deploy Submariner are already in the same cluster set, skip to step 5.
If the clusters on which you want to deploy Submariner are not in the same cluster set, create a cluster set for them by completing the following steps:
- Select Create cluster set.
- Name the cluster set, and select Create.
- Select Manage resource assignments to assign clusters to the cluster set.
- Select the managed clusters that you want to connect with Submariner to add them to the cluster set.
- Select Review to view and confirm the clusters that you selected.
- Select Save to save the cluster set, and view the resulting cluster set page.
- On the cluster set page, select the Submariner add-ons tab.
- Select Install Submariner add-ons.
- Select the clusters on which you want to deploy Submariner.
See the fields in the following table and enter the required information in the Install Submariner add-ons editor:
Field Notes AWS Access Key ID
Only visible when you import an AWS cluster.
AWS Secret Access Key
Only visible when you import an AWS cluster.
Base domain resource group name
Only visible when you import an Azure cluster.
Client ID
Only visible when you import an Azure cluster.
Client secret
Only visible when you import an Azure cluster.
Subscription ID
Only visible when you import an Azure cluster.
Tenant ID
Only visible when you import an Azure cluster.
Google Cloud Platform service account JSON key
Only visible when you import a Google Cloud Platform cluster.
Instance type
The instance type of the gateway node that is created on the managed cluster.
IPsec NAT-T port
The default value for the IPsec NAT traversal port is port
4500
. If your managed cluster environment is VMware vSphere, ensure that this port is opened on your firewall.Gateway count
The number of gateway nodes to be deployed on the managed cluster. For AWS, GCP, Azure, and OpenStack clusters, dedicated Gateway nodes are deployed. For VWware clusters, existing worker nodes are tagged as gateway nodes. The default value is
1
. If the value is greater than 1, the Submariner gateway High Availability (HA) is automatically enabled.Cable driver
The Submariner gateway cable engine component that maintains the cross-cluster tunnels. The default value is
Libreswan IPsec
.Disconnected cluster
If enabled, tells Submariner to not access any external servers for public IP resolution.
Globalnet CIDR
Only visible when the Globalnet configuration is selected on the cluster set. The Globalnet CIDR to be used for the managed cluster. If left blank, a CIDR is allocated from the cluster set pool.
- Select Next at the end of the editor to move to the editor for the next cluster, and complete the editor for each of the remaining clusters that you selected.
- Verify your configuration for each managed cluster.
Click Install to deploy Submariner on the selected managed clusters.
It might take several minutes for the installation and configuration to complete. You can check the Submariner status in the list on the Submariner add-ons tab:
-
Connection status
indicates how many Submariner connections are established on the managed cluster. -
Agent status
indicates whether Submariner is successfully deployed on the managed cluster. The console might report a status ofDegraded
until it is installed and configured. -
Gateway nodes labeled
indicates the number of gateway nodes on the managed cluster.
-
Submariner is now deployed on the selected clusters.
1.1.5. Deploying Submariner manually
Before you deploy Submariner with Red Hat Advanced Cluster Management for Kubernetes, you must prepare the clusters on the hosting environment for the connection. See Deploying Submariner by using the console to learn how to automatically deploy Submariner on supported clusters by using the console.
If your cluster is hosted on a provider that does not support automatic Submariner deployment, see the following sections to prepare the infrastructure manually. Each provider has unique steps for preparation, so make sure to select the correct provider.
1.1.5.1. Preparing bare metal for Submariner
To prepare bare metal clusters for deploying Submariner, complete the following steps:
- Ensure that the firewall allows inbound/outbound traffic for external clients on the 4500/UDP and 4490/UDP ports for the Gateway nodes. Also, if the cluster is deployed with OpenShiftSDN CNI, allow inbound/outbound UDP/4800 traffic within the local cluster nodes.
Customize and apply YAML content that is similar to the following example:
apiVersion: submarineraddon.open-cluster-management.io/v1alpha1 kind: SubmarinerConfig metadata: name: submariner namespace: <managed-cluster-namespace> spec: gatewayConfig: gateways: 1
Replace
managed-cluster-namespace
with the name of your managed cluster. The name of theSubmarinerConfig
must besubmariner
, as shown in the example.This configuration labels one of the worker nodes as the Submariner gateway on your bare metal cluster.
By default, Submariner uses IP security (IPsec) to establish the secure tunnels between the clusters on the gateway nodes. You can either use the default IPsec NATT port, or you can specify a different port that you configured. When you run this procedure without specifying an IPsec NATT port, 4500/UDP is used for the connections.
- Identify the Gateway node configured by Submariner and enable firewall configurations to allow the IPsec NATT (UDP/4500) and NatDiscovery (UDP/4490) ports for external traffic.
See Customizing Submariner deployments for information about the customization options.
1.1.5.2. Preparing Microsoft Azure Red Hat OpenShift for Submariner by using the console (Technology Preview)
The Microsoft Azure Red Hat OpenShift service combines various tools and resources to help simplify the process of building container-based applications. To prepare Azure Red Hat OpenShift clusters for deploying Submariner by using the console, complete the following steps:
- Download the Python wheel and CLI extension.
From the Azure CLI, run the following command to install the extension:
az extension add --upgrade -s <path-to-extension>
Replace
path-to-extension
with the path to where you downloaded the.whl
extension file.Run the following command to verify that the CLI extension is being used:
az extension list
If the extension is being used, the output might resemble the following example:
"experimental": false, "extensionType": "whl", "name": "aro", "path": "<path-to-extension>", "preview": true, "version": "1.0.x"
From the Azure CLI, register the preview feature by running the following command:
az feature registration create --namespace Microsoft.RedHatOpenShift --name AdminKubeconfig
Retrieve the administrator
kubeconfig
by running the following command:az aro get-admin-kubeconfig -g <resource group> -n <cluster resource name>
Note: The
az aro
command saves thekubeconfig
to the local directory and uses the namekubeconfig
. To use it, set the environment variableKUBECONFIG
to match the path of the file. See the following example:export KUBECONFIG=<path-to-kubeconfig> oc get nodes
- Import your Azure Red Hat OpenShift cluster to your cluster list by selecting Infrastructure > Clusters > Import cluster from the Red Hat Advanced Cluster Management console.
Select the
Kubeconfig
Import mode and enter the content from yourkubeconfig
file in the Kubeconfig window. Follow the instructions in the console to complete the import.You can verify that your Azure Red Hat OpenShift cluster was imported successfully by navigating to Infrastructure > Clusters.
- Navigate to Infrastructure > Clusters > Cluster sets and select the name of the cluster set that you want to add. Then, click the Submariner add-ons tab.
- Click the Install Submariner add-ons button and set your Azure Red Hat OpenShift cluster as your Target clusters. Follow the instructions in the console to complete the install.
-
Navigate to Infrastructure > Clusters > Cluster sets > Submariner add-ons to verify that your Azure Red Hat OpenShift cluster Connection status is
Healthy
.
1.1.5.2.1. Preparing Microsoft Azure Red Hat OpenShift for Submariner by using the API (Technology Preview)
To prepare Azure Red Hat OpenShift clusters for deploying Submariner by using the API, customize and apply YAML content that is similar to the following example:
apiVersion: submarineraddon.open-cluster-management.io/v1alpha1 kind: SubmarinerConfig metadata: name: submariner namespace: <managed-cluster-namespace> spec: loadBalancerEnable: true
Replace managed-cluster-namespace
with the name of your managed cluster.
The name of the SubmarinerConfig
must be submariner
, as shown in the example.
This configuration labels one of the worker nodes as the Submariner gateway on your Azure Red Hat OpenShift cluster.
By default, Submariner uses IP security (IPsec) to establish the secure tunnels between the clusters on the gateway nodes. You can either use the default IPsec NATT port, or you can specify a different port that you configured. When you run this procedure without specifying an IPsec NATT port, port 4500/UDP is used for the connections.
See Customizing Submariner deployments for information about the customization options.
1.1.5.3. Preparing Red Hat OpenShift Service on AWS for Submariner by using the console (Technology Preview)
Red Hat OpenShift Service on AWS provides a stable and flexible platform for application development and modernization. To prepare OpenShift Service on AWS clusters for deploying Submariner, complete the following steps:
Create a new node to run Submariner gateway by running the following command:
rosa create machinepool --cluster=<cluster_name> --name=sm-gw-mp --replicas=<number of Submariner gateway > --labels='submariner.io/gateway=true'
Log in to OpenShift Service on AWS by running the following commands:
rosa login oc login <rosa-cluster-url>:6443 --username cluster-admin --password <password>
Create a
kubeconfig
for your OpenShift Service on AWS cluster by running the following command:oc config view --flatten=true > rosa_kube/kubeconfig
- Import your OpenShift Service on AWS cluster to your cluster list by selecting Infrastructure > Clusters > Import cluster from the Red Hat Advanced Cluster Management console.
Select the
Kubeconfig
Import mode and enter the content from yourkubeconfig
file in the Kubeconfig window. Follow the instructions in the console to complete the import.You can verify that your OpenShift Service on AWS cluster was imported successfully by navigating to Infrastructure > Clusters.
- Navigate to Infrastructure > Clusters > Cluster sets and select the name of the cluster set that you want to add. Then, click the Submariner add-ons tab.
- Click the Install Submariner add-ons button and set your OpenShift Service on AWS cluster as your Target clusters. Follow the instructions in the console to complete the installation.
-
Navigate to Infrastructure > Clusters > Cluster sets > Submariner add-ons to verify that your OpenShift Service on AWS cluster Connection status is
Healthy
.
1.1.5.3.1. Preparing Red Hat OpenShift Service on AWS for Submariner by using the API (Technology Preview)
To prepare OpenShift Service on AWS clusters for deploying Submariner by using the API, complete the following steps:
Create a new node to run Submariner gateway by running the following command:
rosa create machinepool --cluster=<cluster_name> --name=sm-gw-mp --replicas=<number of Submariner gateway > --labels='submariner.io/gateway=true'
Customize and apply YAML content that is similar to the following example:
apiVersion: submarineraddon.open-cluster-management.io/v1alpha1 kind: SubmarinerConfig metadata: name: submariner namespace: <managed-cluster-namespace> spec: loadBalancerEnable: true
Replace
managed-cluster-namespace
with the name of your managed cluster.The name of the
SubmarinerConfig
must besubmariner
, as shown in the example.By default, Submariner uses IP security (IPsec) to establish the secure tunnels between the clusters on the gateway nodes. You can either use the default IPsec NATT port, or you can specify a different port that you configured. When you run this procedure without specifying an IPsec NATT port, port 4500/UDP is used for the connections.
See Customizing Submariner deployments for information about the customization options.
1.1.5.4. Deploy Submariner with the ManagedClusterAddOn API
After manually preparing your selected hosting environment, you can deploy Submariner with the ManagedClusterAddOn
API by completing the following steps:
Create a
ManagedClusterSet
resource on the hub cluster by using the instructions provided in the Creating a ManagedClusterSet documentation. Make sure your entry for theManagedClusterSet
resembles the following content:apiVersion: cluster.open-cluster-management.io/v1beta2 kind: ManagedClusterSet metadata: name: <managed-cluster-set-name>
Replace
managed-cluster-set-name
with a name for theManagedClusterSet
that you are creating.Important: The maximum character length of a Kubernetes namespace is 63 characters. The maximum character length you can use for the
<managed-cluster-set-name>
is 56 characters. If the character length of<managed-cluster-set-name>
exceeds 56 characters, the<managed-cluster-set-name>
is cut off from the head.After the
ManagedClusterSet
is created, thesubmariner-addon
creates a namespace called<managed-cluster-set-name>-broker
and deploys the Submariner broker to it.Create the
Broker
configuration on the hub cluster in the<managed-cluster-set-name>-broker
namespace by customizing and applying YAML content that is similar to the following example:apiVersion: submariner.io/v1alpha1 kind: Broker metadata: name: submariner-broker namespace: <managed-cluster-set-name>-broker labels: cluster.open-cluster-management.io/backup: submariner spec: globalnetEnabled: <true-or-false>
Replace
managed-cluster-set-name
with the name of the managed cluster.Set the value of
globalnetEnabled
totrue
if you want to enable Submariner Globalnet in theManagedClusterSet
.Add one managed cluster to the
ManagedClusterSet
by running the following command:oc label managedclusters <managed-cluster-name> "cluster.open-cluster-management.io/clusterset=<managed-cluster-set-name>" --overwrite
Replace
<managed-cluster-name>
with the name of the managed cluster that you want to add to theManagedClusterSet
.Replace
<managed-cluster-set-name>
with the name of theManagedClusterSet
to which you want to add the managed cluster.Customize and apply YAML content that is similar to the following example:
apiVersion: submarineraddon.open-cluster-management.io/v1alpha1 kind: SubmarinerConfig metadata: name: submariner namespace: <managed-cluster-namespace> spec:{}
Replace
managed-cluster-namespace
with the namespace of your managed cluster.Note: The name of the
SubmarinerConfig
must besubmariner
, as shown in the example.Deploy Submariner on the managed cluster by customizing and applying YAML content that is similar to the following example:
apiVersion: addon.open-cluster-management.io/v1alpha1 kind: ManagedClusterAddOn metadata: name: submariner namespace: <managed-cluster-name> spec: installNamespace: submariner-operator
Replace
managed-cluster-name
with the name of the managed cluster that you want to use with Submariner.The
installNamespace
field in the spec of theManagedClusterAddOn
is the namespace on the managed cluster where it installs Submariner. Currently, Submariner must be installed in thesubmariner-operator
namespace.After the
ManagedClusterAddOn
is created, thesubmariner-addon
deploys Submariner to thesubmariner-operator
namespace on the managed cluster. You can view the deployment status of Submariner from the status of thisManagedClusterAddOn
.Note: The name of
ManagedClusterAddOn
must besubmariner
.- Repeat steps three, four, and five for all of the managed clusters that you want to enable Submariner on.
After Submariner is deployed on the managed cluster, you can verify the Submariner deployment status by checking the status of submariner
ManagedClusterAddOn
by running the following command:oc -n <managed-cluster-name> get managedclusteraddons submariner -oyaml
Replace
managed-cluster-name
with the name of the managed cluster.In the status of the Submariner
ManagedClusterAddOn
, three conditions indicate the deployment status of Submariner:-
SubmarinerGatewayNodesLabeled
condition indicates whether there are labeled Submariner gateway nodes on the managed cluster. -
SubmarinerAgentDegraded
condition indicates whether the Submariner is successfully deployed on the managed cluster. -
SubmarinerConnectionDegraded
condition indicates how many connections are established on the managed cluster with Submariner.
-
1.1.6. Customizing Submariner deployments
You can customize some of the settings of your Submariner deployments, including your Network Address Translation-Traversal (NATT) port, number of gateway nodes, and instance type of your gateway nodes. These customizations are consistent across all of the providers.
1.1.6.1. NATT port
If you want to customize your NATT port, customize and apply the following YAML content for your provider environment:
apiVersion: submarineraddon.open-cluster-management.io/v1alpha1 kind: SubmarinerConfig metadata: name: submariner namespace: <managed-cluster-namespace> spec: credentialsSecret: name: <managed-cluster-name>-<provider>-creds IPSecNATTPort: <NATTPort>
-
Replace
managed-cluster-namespace
with the namespace of your managed cluster. Replace
managed-cluster-name
with the name of your managed cluster-
AWS: Replace
provider
withaws
. The value of<managed-cluster-name>-aws-creds
is your AWS credential secret name, which you can find in the cluster namespace of your hub cluster. -
GCP: Replace
provider
withgcp
. The value of<managed-cluster-name>-gcp-creds
is your Google Cloud Platform credential secret name, which you can find in the cluster namespace of your hub cluster. -
OpenStack: Replace
provider
withosp
. The value of<managed-cluster-name>-osp-creds
is your Red Hat OpenStack Platform credential secret name, which you can find in the cluster namespace of your hub cluster. -
Azure: Replace
provider
withazure
. The value of<managed-cluster-name>-azure-creds
is your Microsoft Azure credential secret name, which you can find in the cluster namespace of your hub cluster.
-
AWS: Replace
-
Replace
managed-cluster-namespace
with the namespace of your managed cluster. -
Replace
managed-cluster-name
with the name of your managed cluster. The value ofmanaged-cluster-name-gcp-creds
is your Google Cloud Platform credential secret name, which you can find in the cluster namespace of your hub cluster. -
Replace
NATTPort
with the NATT port that you want to use.
Note: The name of the SubmarinerConfig
must be submariner
, as shown in the example.
1.1.6.2. Number of gateway nodes
If you want to customize the number of your gateway nodes, customize and apply YAML content that is similar to the following example:
apiVersion: submarineraddon.open-cluster-management.io/v1alpha1 kind: SubmarinerConfig metadata: name: submariner namespace: <managed-cluster-namespace> spec: credentialsSecret: name: <managed-cluster-name>-<provider>-creds gatewayConfig: gateways: <gateways>
-
Replace
managed-cluster-namespace
with the namespace of your managed cluster. Replace
managed-cluster-name
with the name of your managed cluster.-
AWS: Replace
provider
withaws
. The value of<managed-cluster-name>-aws-creds
is your AWS credential secret name, which you can find in the cluster namespace of your hub cluster. -
GCP: Replace
provider
withgcp
. The value of<managed-cluster-name>-gcp-creds
is your Google Cloud Platform credential secret name, which you can find in the cluster namespace of your hub cluster. -
OpenStack: Replace
provider
withosp
. The value of<managed-cluster-name>-osp-creds
is your Red Hat OpenStack Platform credential secret name, which you can find in the cluster namespace of your hub cluster. -
Azure: Replace
provider
withazure
. The value of<managed-cluster-name>-azure-creds
is your Microsoft Azure credential secret name, which you can find in the cluster namespace of your hub cluster.
-
AWS: Replace
-
Replace
gateways
with the number of gateways that you want to use. If the value is greater than 1, the Submariner gateway automatically enables high availability.
Note: The name of the SubmarinerConfig
must be submariner
, as shown in the example.
1.1.6.3. Instance types of gateway nodes
If you want to customize the instance type of your gateway node, customize and apply YAML content that is similar to the following example:
apiVersion: submarineraddon.open-cluster-management.io/v1alpha1 kind: SubmarinerConfig metadata: name: submariner namespace: <managed-cluster-namespace> spec: credentialsSecret: name: <managed-cluster-name>-<provider>-creds gatewayConfig: instanceType: <instance-type>
-
Replace
managed-cluster-namespace
with the namespace of your managed cluster. Replace
managed-cluster-name
with the name of your managed cluster.-
AWS: Replace
provider
withaws
. The value of<managed-cluster-name>-aws-creds
is your AWS credential secret name, which you can find in the cluster namespace of your hub cluster. -
GCP: Replace
provider
withgcp
. The value of<managed-cluster-name>-gcp-creds
is your Google Cloud Platform credential secret name, which you can find in the cluster namespace of your hub cluster. -
OpenStack: Replace
provider
withosp
. The value of<managed-cluster-name>-osp-creds
is your Red Hat OpenStack Platform credential secret name, which you can find in the cluster namespace of your hub cluster. -
Azure: Replace
provider
withazure
. The value of<managed-cluster-name>-azure-creds
is your Microsoft Azure credential secret name, which you can find in the cluster namespace of your hub cluster.
-
AWS: Replace
-
Replace
instance-type
with the AWS instance type that you want to use.
Note: The name of the SubmarinerConfig
must be submariner
, as shown in the example.
1.1.6.4. Cable driver
The Submariner Gateway Engine component creates secure tunnels to other clusters. The cable driver component maintains the tunnels by using a pluggable architecture in the Gateway Engine component. You can use the Libreswan or VXLAN implementations for the cableDriver
configuration of the cable engine component. See the following example:
apiVersion: submarineraddon.open-cluster-management.io/v1alpha1 kind: SubmarinerConfig metadata: name: submariner namespace: <managed-cluster-namespace> spec: cableDriver: vxlan credentialsSecret: name: <managed-cluster-name>-<provider>-creds
Best practice: Do not use the VXLAN cable driver on public networks. The VXLAN cable driver is unencrypted. Only use VXLAN to avoid unnecessary double encryption on private networks. For example, some on-premise environments might handle the tunnel’s encryption with a dedicated line-level hardware device.
1.1.7. Managing service discovery for Submariner
After Submariner is deployed into the same environment as your managed clusters, the routes are configured for secure IP routing between the pod and services across the clusters in the managed cluster set.
1.1.7.1. Enabling service discovery for Submariner
To make a service from a cluster visible and discoverable to other clusters in the managed cluster set, you must create a ServiceExport
object. After a service is exported with a ServiceExport
object, you can access the service by the following format: <service>.<namespace>.svc.clusterset.local
. If multiple clusters export a service with the same name, and from the same namespace, they are recognized by other clusters as a single logical service.
This example uses the nginx
service in the default
namespace, but you can discover any Kubernetes ClusterIP
service or headless service:
Apply an instance of the
nginx
service on a managed cluster that is in theManagedClusterSet
by entering the following commands:oc -n default create deployment nginx --image=nginxinc/nginx-unprivileged:stable-alpine oc -n default expose deployment nginx --port=8080
Export the service by creating a
ServiceExport
entry by entering a command with thesubctl
tool that is similar to the following command:subctl export service --namespace <service-namespace> <service-name>
Replace
service-namespace
with the name of the namespace where the service is located. In this example, it isdefault
.Replace
service-name
with the name of the service that you are exporting. In this example, it isnginx
.See
export
in the Submariner documentation for more information about other available flags.Run the following command from a different managed cluster to confirm that it can access the
nginx
service:oc -n default run --generator=run-pod/v1 tmp-shell --rm -i --tty --image quay.io/submariner/nettest -- /bin/bash curl nginx.default.svc.clusterset.local:8080
The nginx
service discovery is now configured for Submariner.
1.1.7.2. Disabling service discovery for Submariner
To disable a service from being exported to other clusters, enter a command similar to the following example for nginx
:
subctl unexport service --namespace <service-namespace> <service-name>
Replace service-namespace
with the name of the namespace where the service is located.
Replace service-name
with the name of the service that you are exporting.
See unexport
in the Submariner documentation for more information about other available flags.
The service is no longer available for discovery by clusters.
1.1.8. Uninstalling Submariner
You can uninstall the Submariner components from your clusters using the Red Hat Advanced Cluster Management for Kubernetes console or the command-line. For Submariner versions earlier than 0.12, additional steps are needed to completely remove all data plane components. The Submariner uninstall is idempotent, so you can repeat steps without any issues.
1.1.8.1. Uninstalling Submariner by using the console
To uninstall Submariner from a cluster by using the console, complete the following steps:
- From the console navigation, select Infrastructure > Clusters, and select the Cluster sets tab.
- Select the cluster set that contains the clusters from which you want to remove the Submariner components.
- Select the Submariner Add-ons tab to view the clusters in the cluster set that have Submariner deployed.
- In the Actions menu for the cluster that you want to uninstall Submariner, select Uninstall Add-on.
- In the Actions menu for the cluster that you want to uninstall Submariner, select Delete cluster sets.
Repeat those steps for other clusters from which you are removing Submariner.
Tip: You can remove the Submariner add-on from multiple clusters in the same cluster set by selecting multiple clusters and clicking Actions. Select Uninstall Submariner add-ons.
If the version of Submariner that you are removing is earlier than version 0.12, continue with Uninstalling Submariner manually. If the Submariner version is 0.12, or later, Submariner is removed.
Important: Verify that all of the cloud resources are removed from the cloud provider to avoid additional charges by your cloud provider. See Verifying Submariner resource removal for more information.
1.1.8.2. Uninstalling Submariner by using the CLI
To uninstall Submariner by using the command line, complete the following steps:
Remove the Submariner deployment for the cluster by running the following command:
oc -n <managed-cluster-namespace> delete managedclusteraddon submariner
Replace
managed-cluster-namespace
with the namespace of your managed cluster.Remove the cloud resources of the cluster by running the following command:
oc -n <managed-cluster-namespace> delete submarinerconfig submariner
Replace
managed-cluster-namespace
with the namespace of your managed cluster.Delete the cluster set to remove the broker details by running the following command:
oc delete managedclusterset <managedclusterset>
Replace
managedclusterset
with the name of your managed cluster set.
If the version of Submariner that you are removing is earlier than version 0.12, continue with Uninstalling Submariner manually. If the Submariner version is 0.12, or later, Submariner is removed.
Important: Verify that all of the cloud resources are removed from the cloud provider to avoid additional charges by your cloud provider. See Verifying Submariner resource removal for more information.
1.1.8.3. Uninstalling Submariner manually
When uninstalling versions of Submariner that are earlier than version 0.12, complete steps 5-8 in the Manual Uninstall section in the Submariner documentation.
After completing those steps, your Submariner components are removed from the cluster.
Important: Verify that all of the cloud resources are removed from the cloud provider to avoid additional charges by your cloud provider. See Verifying Submariner resource removal for more information.
1.1.8.4. Verifying Submariner resource removal
After uninstalling Submariner, verify that all of the Submariner resources are removed from your clusters. If they remain on your clusters, some resources continue to accrue charges from infrastructure providers. Ensure that you have no additional Submariner resourceson your cluster by completing the following steps:
Run the following command to list any Submariner resources that remain on the cluster:
oc get cluster <CLUSTER_NAME> grep submariner
Replace
CLUSTER_NAME
with the name of your cluster.Remove any resources on the list by entering the following command:
oc delete resource <RESOURCE_NAME> cluster <CLUSTER_NAME>
Replace
RESOURCE_NAME
with the name of the Submariner resource that you want to remove.- Repeat steps 1-2 for each of the clusters until your search does not identify any resources.
The Submariner resources are removed from your cluster.
1.2. VolSync persistent volume replication service
VolSync is a Kubernetes operator that enables asynchronous replication of persistent volumes within a cluster, or across clusters with storage types that are not otherwise compatible for replication. It uses the Container Storage Interface (CSI) to overcome the compatibility limitation. After deploying the VolSync operator in your environment, you can leverage it to create and maintain copies of your persistent data. VolSync can only replicate persistent volume claims on Red Hat OpenShift Container Platform clusters that are at version 4.8 or later.
Important: VolSync only supports replicating persistent volume claims with the volumeMode
of Filesystem
. If you do not select the volumeMode
, it defaults to Filesystem
.
1.2.1. Replicating persistent volumes with VolSync
You can use three supported methods to replicate persistent volumes with VolSync, which depend on the number of synchronization locations that you have: Rsync, restic, or Rclone.
1.2.1.1. Prerequisites
Before installing VolSync on your clusters, you must have the following requirements:
- A configured Red Hat OpenShift Container Platform environment running a Red Hat Advanced Cluster Management version 2.4, or later, hub cluster
- At least two configured clusters that are managed by the same Red Hat Advanced Cluster Management hub cluster
-
Network connectivity between the clusters that you are configuring with VolSync. If the clusters are not on the same network, you can configure the Submariner multicluster networking and service discovery and use the
ClusterIP
value forServiceType
to network the clusters, or use a load balancer with theLoadBalancer
value forServiceType
. - The storage driver that you use for your source persistent volume must be CSI-compatible and able to support snapshots.
1.2.1.2. Installing VolSync on the managed clusters
To enable VolSync to replicate the persistent volume claim on one cluster to the persistent volume claim of another cluster, you must install VolSync on both the source and the target managed clusters.
VolSync does not create its own namespace, so it is in the same namespace as other OpenShift Container Platform all-namespace operators. Any changes that you make to the operator settings for VolSync also affects the other operators in the same namespace, such as if you change to manual approval for channel updates.
You can use either of two methods to install VolSync on two clusters in your environment. You can either add a label to each of the managed clusters in the hub cluster, or you can manually create and apply a ManagedClusterAddOn
, as they are described in the following sections:
1.2.1.2.1. Installing VolSync using labels
To install VolSync on the managed cluster by adding a label.
Complete the following steps from the Red Hat Advanced Cluster Management console:
-
Select one of the managed clusters from the
Clusters
page in the hub cluster console to view its details. In the Labels field, add the following label:
addons.open-cluster-management.io/volsync=true
The VolSync service pod is installed on the managed cluster.
- Add the same label the other managed cluster.
Run the following command on each managed cluster to confirm that the VolSync operator is installed:
oc get csv -n openshift-operators
There is an operator listed for VolSync when it is installed.
-
Select one of the managed clusters from the
Complete the following steps from the command-line interface:
- Start a command-line session on the hub cluster.
Enter the following command to add the label to the first cluster:
oc label managedcluster <managed-cluster-1> "addons.open-cluster-management.io/volsync"="true"
Replace
managed-cluster-1
with the name of one of your managed clusters.Enter the following command to add the label to the second cluster:
oc label managedcluster <managed-cluster-2> "addons.open-cluster-management.io/volsync"="true"
Replace
managed-cluster-2
with the name of your other managed cluster.A
ManagedClusterAddOn
resource should be created automatically on your hub cluster in the namespace of each corresponding managed cluster.
1.2.1.2.2. Installing VolSync using a ManagedClusterAddOn
To install VolSync on the managed cluster by adding a ManagedClusterAddOn
manually, complete the following steps:
On the hub cluster, create a YAML file called
volsync-mcao.yaml
that contains content that is similar to the following example:apiVersion: addon.open-cluster-management.io/v1alpha1 kind: ManagedClusterAddOn metadata: name: volsync namespace: <managed-cluster-1-namespace> spec: {}
Replace
managed-cluster-1-namespace
with the namespace of one of your managed clusters. This namespace is the same as the name of the managed cluster.Note: The name must be
volsync
.Apply the file to your configuration by entering a command similar to the following example:
oc apply -f volsync-mcao.yaml
Repeat the procedure for the other managed cluster.
A
ManagedClusterAddOn
resource should be created automatically on your hub cluster in the namespace of each corresponding managed cluster.
1.2.1.3. Configuring an Rsync replication
You can create a 1:1 asynchronous replication of persistent volumes by using an Rsync replication. You can use Rsync-based replication for disaster recovery or sending data to a remote site.
The following example shows how to configure by using the Rsync method. For additional information about Rsync, see Usage in the VolSync documentation.
1.2.1.3.1. Configuring Rsync replication across managed clusters
For Rsync-based replication, configure custom resources on the source and destination clusters. The custom resources use the address
value to connect the source to the destination, and the sshKeys
to ensure that the transferred data is secure.
Note: You must copy the values for address
and sshKeys
from the destination to the source, so configure the destination before you configure the source.
This example provides the steps to configure an Rsync replication from a persistent volume claim on the source
cluster in the source-ns
namespace to a persistent volume claim on a destination
cluster in the destination-ns
namespace. You can replace those values with other values, if necessary.
Configure your destination cluster.
Run the following command on the destination cluster to create the namespace:
oc create ns <destination-ns>
Replace
destination-ns
with a name for the namespace that will contain your destination persistent volume claim.Copy the following YAML content to create a new file called
replication_destination.yaml
:apiVersion: volsync.backube/v1alpha1 kind: ReplicationDestination metadata: name: <destination> namespace: <destination-ns> spec: rsync: serviceType: LoadBalancer copyMethod: Snapshot capacity: 2Gi accessModes: [ReadWriteOnce] storageClassName: gp2-csi volumeSnapshotClassName: csi-aws-vsc
Note: The
capacity
value should match the capacity of the persistent volume claim that is being replicated.Replace
destination
with the name of your replication destination CR.Replace
destination-ns
with the name of the namespace where your destination is located.For this example, the
ServiceType
value ofLoadBalancer
is used. The load balancer service is created by the source cluster to enable your source managed cluster to transfer information to a different destination managed cluster. You can useClusterIP
as the service type if your source and destinations are on the same cluster, or if you have Submariner network service configured. Note the address and the name of the secret to refer to when you configure the source cluster.The
storageClassName
andvolumeSnapshotClassName
are optional parameters. Specify the values for your environment, particularly if you are using a storage class and volume snapshot class name that are different than the default values for your environment.Run the following command on the destination cluster to create the
replicationdestination
resource:oc create -n <destination-ns> -f replication_destination.yaml
Replace
destination-ns
with the name of the namespace where your destination is located.After the
replicationdestination
resource is created, following parameters and values are added to the resource:Parameter Value .status.rsync.address
IP address of the destination cluster that is used to enable the source and destination clusters to communicate.
.status.rsync.sshKeys
Name of the SSH key file that enables secure data transfer from the source cluster to the destination cluster.
Run the following command to copy the value of
.status.rsync.address
to use on the source cluster:ADDRESS=`oc get replicationdestination <destination> -n <destination-ns> --template={{.status.rsync.address}}` echo $ADDRESS
Replace
destination
with the name of your replication destination custom resource.Replace
destination-ns
with the name of the namespace where your destination is located.The output should appear similar to the following output, which is for an Amazon Web Services environment:
a831264645yhrjrjyer6f9e4a02eb2-5592c0b3d94dd376.elb.us-east-1.amazonaws.com
Run the following command to copy the name of the secret and the contents of the secret that are provided as the value of
.status.rsync.sshKeys
.SSHKEYS=`oc get replicationdestination <destination> -n <destination-ns> --template={{.status.rsync.sshKeys}}` echo $SSHKEYS
Replace
destination
with the name of your replication destination custom resource.Replace
destination-ns
with the name of the namespace where your destination is located.You will have to enter it on the source cluster when you configure the source. The output should be the name of your SSH keys secret file, which might resemble the following name:
volsync-rsync-dst-src-destination-name
Identify the source persistent volume claim that you want to replicate.
Note: The source persistent volume claim must be on a CSI storage class.
Create the
ReplicationSource
items.Copy the following YAML content to create a new file called
replication_source.yaml
on the source cluster:apiVersion: volsync.backube/v1alpha1 kind: ReplicationSource metadata: name: <source> namespace: <source-ns> spec: sourcePVC: <persistent_volume_claim> trigger: schedule: "*/3 * * * *" #/* rsync: sshKeys: <mysshkeys> address: <my.host.com> copyMethod: Snapshot storageClassName: gp2-csi volumeSnapshotClassName: gp2-csi
Replace
source
with the name for your replication source custom resource. See step 3-vi of this procedure for instructions on how to replace this automatically.Replace
source-ns
with the namespace of the persistent volume claim where your source is located. See step 3-vi of this procedure for instructions on how to replace this automatically.Replace
persistent_volume_claim
with the name of your source persistent volume claim.Replace
mysshkeys
with the keys that you copied from the.status.rsync.sshKeys
field of theReplicationDestination
when you configured it.Replace
my.host.com
with the host address that you copied from the.status.rsync.address
field of theReplicationDestination
when you configured it.If your storage driver supports cloning, using
Clone
as the value forcopyMethod
might be a more streamlined process for the replication.StorageClassName
andvolumeSnapshotClassName
are optional parameters. If you are using a storage class and volume snapshot class name that are different than the defaults for your environment, specify those values.You can now set up the synchronization method of the persistent volume.
Copy the SSH secret from the destination cluster by entering the following command against the destination cluster:
oc get secret -n <destination-ns> $SSHKEYS -o yaml > /tmp/secret.yaml
Replace
destination-ns
with the namespace of the persistent volume claim where your destination is located.Open the secret file in the
vi
editor by entering the following command:vi /tmp/secret.yaml
In the open secret file on the destination cluster, make the following changes:
-
Change the namespace to the namespace of your source cluster. For this example, it is
source-ns
. -
Remove the owner references (
.metadata.ownerReferences
).
-
Change the namespace to the namespace of your source cluster. For this example, it is
On the source cluster, create the secret file by entering the following command on the source cluster:
oc create -f /tmp/secret.yaml
On the source cluster, modify the
replication_source.yaml
file by replacing the value of theaddress
andsshKeys
in theReplicationSource
object with the values that you noted from the destination cluster by entering the following commands:sed -i "s/<my.host.com>/$ADDRESS/g" replication_source.yaml sed -i "s/<mysshkeys>/$SSHKEYS/g" replication_source.yaml oc create -n <source> -f replication_source.yaml
Replace
my.host.com
with the host address that you copied from the.status.rsync.address
field of theReplicationDestination
when you configured it.Replace
mysshkeys
with the keys that you copied from the.status.rsync.sshKeys
field of theReplicationDestination
when you configured it.Replace
source
with the name of the persistent volume claim where your source is located.Note: You must create the file in the same namespace as the persistent volume claim that you want to replicate.
Verify that the replication completed by running the following command on the
ReplicationSource
object:oc describe ReplicationSource -n <source-ns> <source>
Replace
source-ns
with the namespace of the persistent volume claim where your source is located.Replace
source
with the name of your replication source custom resource.If the replication was successful, the output should be similar to the following example:
Status: Conditions: Last Transition Time: 2021-10-14T20:48:00Z Message: Synchronization in-progress Reason: SyncInProgress Status: True Type: Synchronizing Last Transition Time: 2021-10-14T20:41:41Z Message: Reconcile complete Reason: ReconcileComplete Status: True Type: Reconciled Last Sync Duration: 5m20.764642395s Last Sync Time: 2021-10-14T20:47:01Z Next Sync Time: 2021-10-14T20:48:00Z
If the
Last Sync Time
has no time listed, then the replication is not complete.
You have a replica of your original persistent volume claim.
1.2.1.4. Configuring a restic backup
A restic-based backup copies a restic-based backup copy of the persistent volume to a location that is specified in your restic-config.yaml
secret file. A restic backup does not synchronize data between the clusters, but provides data backup.
Complete the following steps to configure a restic-based backup:
Specify a repository where your backup images are stored by creating a secret that resembles the following YAML content:
apiVersion: v1 kind: Secret metadata: name: restic-config type: Opaque stringData: RESTIC_REPOSITORY: <my-restic-repository> RESTIC_PASSWORD: <my-restic-password> AWS_ACCESS_KEY_ID: access AWS_SECRET_ACCESS_KEY: password
Replace
my-restic-repository
with the location of the S3 bucket repository where you want to store your backup files.Replace
my-restic-password
with the encryption key that is required to access the repository.Replace
access
andpassword
with the credentials for your provider, if required. Refer to Preparing a new repository for more information.If you need to prepare a new repository, see Preparing a new repository for the procedure. If you use that procedure, skip the step that requires you to run the
restic init
command to initialize the repository. VolSync automatically initializes the repository during the first backup.Important: When backing up multiple persistent volume claims to the same S3 bucket, the path to the bucket must be unique for each persistent volume claim. Each persistent volume claim is backed up with a separate
ReplicationSource
, and each requires a separate restic-config secret.By sharing the same S3 bucket, each
ReplicationSource
has write access to the entire S3 bucket.Configure your backup policy by creating a
ReplicationSource
object that resembles the following YAML content:apiVersion: volsync.backube/v1alpha1 kind: ReplicationSource metadata: name: mydata-backup spec: sourcePVC: <source> trigger: schedule: "*/30 * * * *" #\* restic: pruneIntervalDays: 14 repository: <restic-config> retain: hourly: 6 daily: 5 weekly: 4 monthly: 2 yearly: 1 copyMethod: Clone # The StorageClass to use when creating the PiT copy (same as source PVC if omitted) #storageClassName: my-sc-name # The VSC to use if the copy method is Snapshot (default if omitted) #volumeSnapshotClassName: my-vsc-name
Replace
source
with the persistent volume claim that you are backing up.Replace the value for
schedule
with how often to run the backup. This example has the schedule for every 30 minutes. See Scheduling your synchronization for more information.Replace the value of
PruneIntervalDays
to the number of days that elapse between instances of repacking the data to save space. The prune operation can generate significant I/O traffic while it is running.Replace
restic-config
with the name of the secret that you created in step 1.Set the values for
retain
to your retention policy for the backed up images.Best practice: Use
Clone
for the value ofCopyMethod
to ensure that a point-in-time image is saved.For additional information about the backup options, see Backup options in the VolSync documentation.
Note: Restic movers run without root permissions by default. If you want to run restic movers as root, run the following command to add the elevated permissions annotation to your namespace.
oc annotate namespace <namespace> volsync.backube/privileged-movers=true
Replace <namespace>
with the name of your namespace.
1.2.1.4.1. Restoring a restic backup
You can restore the copied data from a restic backup into a new persistent volume claim. Best practice: Restore only one backup into a new persistent volume claim. To restore the restic backup, complete the following steps:
Create a new persistent volume claim to contain the new data similar to the following example:
kind: PersistentVolumeClaim apiVersion: v1 metadata: name: <pvc-name> spec: accessModes: - ReadWriteOnce resources: requests: storage: 3Gi
Replace
pvc-name
with the name of the new persistent volume claim.Create a
ReplicationDestination
custom resource that resembles the following example to specify where to restore the data:apiVersion: volsync.backube/v1alpha1 kind: ReplicationDestination metadata: name: <destination> spec: trigger: manual: restore-once restic: repository: <restic-repo> destinationPVC: <pvc-name> copyMethod: Direct
Replace
destination
with the name of your replication destination CR.Replace
restic-repo
with the path to your repository where the source is stored.Replace
pvc-name
with the name of the new persistent volume claim where you want to restore the data. Use an existing persistent volume claim for this, rather than provisioning a new one.
The restore process only needs to be completed once, and this example restores the most recent backup. For more information about restore options, see Restore options in the VolSync documentation.
1.2.1.5. Configuring an Rclone replication
An Rclone backup copies a single persistent volume to multiple locations by using Rclone through an intermediate object storage location, like AWS S3. It can be helpful when distributing data to multiple locations.
Complete the following steps to configure an Rclone replication:
Create a
ReplicationSource
custom resource that resembles the following example:apiVersion: volsync.backube/v1alpha1 kind: ReplicationSource metadata: name: <source> namespace: <source-ns> spec: sourcePVC: <source-pvc> trigger: schedule: "*/6 * * * *" #\* rclone: rcloneConfigSection: <intermediate-s3-bucket> rcloneDestPath: <destination-bucket> rcloneConfig: <rclone-secret> copyMethod: Snapshot storageClassName: <my-sc-name> volumeSnapshotClassName: <my-vsc>
Replace
source-pvc
with the name for your replication source custom resource.Replace
source-ns
with the namespace of the persistent volume claim where your source is located.Replace
source
with the persistent volume claim that you are replicating.Replace the value of
schedule
with how often to run the replication. This example has the schedule for every 6 minutes. This value must be within quotation marks. See Scheduling your synchronization for more information.Replace
intermediate-s3-bucket
with the path to the configuration section of the Rclone configuration file.Replace
destination-bucket
with the path to the object bucket where you want your replicated files copied.Replace
rclone-secret
with the name of the secret that contains your Rclone configuration information.Set the value for
copyMethod
asClone
,Direct
, orSnapshot
. This value specifies whether the point-in-time copy is generated, and if so, what method is used for generating it.Replace
my-sc-name
with the name of the storage class that you want to use for your point-in-time copy. If not specified, the storage class of the source volume is used.Replace
my-vsc
with the name of theVolumeSnapshotClass
to use, if you specifiedSnapshot
as yourcopyMethod
. This is not required for other types ofcopyMethod
.Create a
ReplicationDestination
custom resource that resembles the following example:apiVersion: volsync.backube/v1alpha1 kind: ReplicationDestination metadata: name: database-destination namespace: dest spec: trigger: schedule: "3,9,15,21,27,33,39,45,51,57 * * * *" #/* rclone: rcloneConfigSection: <intermediate-s3-bucket> rcloneDestPath: <destination-bucket> rcloneConfig: <rclone-secret> copyMethod: Snapshot accessModes: [ReadWriteOnce] capacity: 10Gi storageClassName: <my-sc> volumeSnapshotClassName: <my-vsc>
Replace the value for
schedule
with how often to move the replication to the destination. The schedules for the source and destination must be offset to allow the data to finish replicating before it is pulled from the destination. This example has the schedule for every 6 minutes, offset by 3 minutes. This value must be within quotation marks. See Scheduling your synchronization for more information.Replace
intermediate-s3-bucket
with the path to the configuration section of the Rclone configuration file.Replace
destination-bucket
with the path to the object bucket where you want your replicated files copied.Replace
rclone-secret
with the name of the secret that contains your Rclone configuration information.Set the value for
copyMethod
asClone
,Direct
, orSnapshot
. This value specifies whether the point-in-time copy is generated, and if so, which method is used for generating it.The value for
accessModes
specifies the access modes for the persistent volume claim. Valid values areReadWriteOnce
orReadWriteMany
.The
capacity
specifies the size of the destination volume, which must be large enough to contain the incoming data.Replace
my-sc
with the name of the storage class that you want to use as the destination for your point-in-time copy. If not specified, the system storage class is used.Replace
my-vsc
with the name of theVolumeSnapshotClass
to use, if you specifiedSnapshot
as yourcopyMethod
. This is not required for other types ofcopyMethod
. If not included, the system defaultVolumeSnapshotClass
is used.
Note: Rclone movers run without root permissions by default. If you want to run Rclone movers as root, run the following command to add the elevated permissions annotation to your namespace.
oc annotate namespace <namespace> volsync.backube/privileged-movers=true
Replace <namespace>
with the name of your namespace.
1.2.2. Converting a replicated image to a usable persistent volume claim
You might need to use the replicated image to recover data, or create a new instance of a persistent volume claim. The copy of the image must be converted to a persistent volume claim before it can be used. To convert a replicated image to a persistent volume claim, complete the following steps:
When the replication is complete, identify the latest snapshot from the
ReplicationDestination
object by entering the following command:$ kubectl get replicationdestination <destination> -n <destination-ns> --template={{.status.latestImage.name}}
Note the value of the latest snapshot for when you create your persistent volume claim.
Replace
destination
with the name of your replication destination.Replace
destination-ns
with the namespace of your destination.Create a
pvc.yaml
file that resembles the following example:apiVersion: v1 kind: PersistentVolumeClaim metadata: name: <pvc-name> namespace: <destination-ns> spec: accessModes: - ReadWriteOnce dataSource: kind: VolumeSnapshot apiGroup: snapshot.storage.k8s.io name: <snapshot_to_replace> resources: requests: storage: 2Gi
Replace
pvc-name
with a name for your new persistent volume claim.Replace
destination-ns
with the namespace where the persistent volume claim is located.Replace
snapshot_to_replace
with theVolumeSnapshot
name that you found in the previous step.Best practice: You can update
resources.requests.storage
with a different value when the value is at least the same size as the initial source persistent volume claim.Validate that your persistent volume claim is running in the environment by entering the following command:
$ kubectl get pvc -n <destination-ns>
Your original backup image is running as the main persistent volume claim.
1.2.3. Scheduling your synchronization
Select from three options when determining how you start your replications: always running, on a schedule, or manually. Scheduling your replications is an option that is often selected.
The Schedule option runs replications at scheduled times. A schedule is defined by a cronspec
, so the schedule can be configured as intervals of time or as specific times. The order of the schedule values are:
"minute (0-59) hour (0-23) day-of-month (1-31) month (1-12) day-of-week (0-6)"
The replication starts when the scheduled time occurs. Your setting for this replication option might resemble the following content:
spec: trigger: schedule: "*/6 * * * *"
After enabling one of these methods, your synchronization schedule runs according to the method that you configured.
See the VolSync documentation for additional information and options.
1.3. Enabling klusterlet add-ons on clusters for cluster management
After you install Red Hat Advanced Cluster Management for Kubernetes and then create or import clusters with multicluster engine operator you can enable the klusterlet add-ons for those managed clusters. The klusterlet add-ons are not enabled by default if you created or imported clusters unless you create or import with the Red Hat Advanced Cluster Management console. See the following available klusterlet add-ons:
- application-manager
- cert-policy-controller
- config-policy-controller
- iam-policy-controller
- governance-policy-framework
- search-collector
Complete the following steps to enable the klusterlet add-ons for the managed clusters after Red Hat Advanced Cluster Management is installed:
Create a YAML file that is similar to the following
KlusterletAddonConfig
, with thespec
value that represents the add-ons:apiVersion: agent.open-cluster-management.io/v1 kind: KlusterletAddonConfig metadata: name: <cluster_name> namespace: <cluster_name> spec: applicationManager: enabled: true certPolicyController: enabled: true iamPolicyController: enabled: true policyController: enabled: true searchCollector: enabled: true
Note: The
policy-controller
add-on is divided into two add-ons: Thegovernance-policy-framework
and theconfig-policy-controller
. As a result, thepolicyController
controls thegovernance-policy-framework
and theconfig-policy-controller
managedClusterAddons
.-
Save the file as
klusterlet-addon-config.yaml
. Apply the YAML by running the following command on the hub cluster:
oc apply -f klusterlet-addon-config.yaml
To verify whether the enabled
managedClusterAddons
are created after theKlusterletAddonConfig
is created, run the following command:oc get managedclusteraddons -n <cluster namespace>
1.4. Enabling cluster-wide proxy on existing cluster add-ons
You can configure the KlusterletAddonConfig
in the cluster namespace to add the proxy environment variables to all the klusterlet add-on pods of the managed Red Hat OpenShift Container Platform clusters. Complete the following steps to configure the KlusterletAddonConfig
to add the three environment variables to the pods of the klusterlet add-ons:
Edit the
KlusterletAddonConfig
file that is in the namespace of the cluster that needs the proxy. You can use the console to find the resource, or you can edit from the terminal with the following command:oc -n <my-cluster-name> edit klusterletaddonconfig <my-cluster-name>
Note: If you are working with only one cluster, you do not need
<my-cluster-name>
at the end of your command. See the following command:oc -n <my-cluster-name> edit klusterletaddonconfig
Edit the
.spec.proxyConfig
section of the file so it resembles the following example. Thespec.proxyConfig
is an optional section:spec proxyConfig: httpProxy: "<proxy_not_secure>" httpsProxy: "<proxy_secure>" noProxy: "<no_proxy>"
Replace
proxy_not_secure
with the address of the proxy server forhttp
requests. For example, usehttp://192.168.123.145:3128
.Replace
proxy_secure
with the address of the proxy server forhttps
requests. For example, usehttps://192.168.123.145:3128
.Replace
no_proxy
with a comma delimited list of IP addresses, hostnames, and domain names where traffic will not be routed through the proxy. For example, use.cluster.local,.svc,10.128.0.0/14,example.com
.If the OpenShift Container Platform cluster is created with cluster wide proxy configured on the hub cluster, the cluster wide proxy configuration values are added to the pods of the klusterlet add-ons as environment variables when the following conditions are met:
-
The
.spec.policyController.proxyPolicy
in theaddon
section is enabled and set toOCPGlobalProxy
. The
.spec.applicationManager.proxyPolocy
is enabled and set toCustomProxy
.Note: The default value of
proxyPolicy
in theaddon
section isDisabled
.See the following examples of
proxyPolicy
entries:apiVersion: agent.open-cluster-management.io/v1 kind: KlusterletAddonConfig metadata: name: clusterName namespace: clusterName spec: proxyConfig: httpProxy: http://pxuser:12345@10.0.81.15:3128 httpsProxy: http://pxuser:12345@10.0.81.15:3128 noProxy: .cluster.local,.svc,10.128.0.0/14, example.com applicationManager: enabled: true proxyPolicy: CustomProxy policyController: enabled: true proxyPolicy: OCPGlobalProxy searchCollector: enabled: true proxyPolicy: Disabled certPolicyController: enabled: true proxyPolicy: Disabled iamPolicyController: enabled: true proxyPolicy: Disabled
-
The
Important: Global proxy settings do not impact alert forwarding. To set up alert forwarding for Red Hat Advanced Cluster Management hub clusters with a cluster-wide proxy, see Forwarding alerts for more details.