Add-ons

Retrieve the <broker-namespace> by running the following command:

oc get ManagedClusterSet <cluster-set-name> -o jsonpath="{.metadata.annotations['cluster\.open-cluster-management\.io/submariner-broker-ns']}"

oc get ManagedClusterSet <cluster-set-name> -o jsonpath="{.metadata.annotations['cluster\.open-cluster-management\.io/submariner-broker-ns']}"

Create a submariner-broker object that specifies the Globalnet configuration by creating a YAML file named submariner-broker. Add content that resembles the following lines to the YAML file:

apiVersion: submariner.io/v1alpha1
kind: Broker
metadata:
  name: submariner-broker
  namespace: <broker-namespace>
spec:
  globalnetEnabled: <true-or-false>

apiVersion: submariner.io/v1alpha1
kind: Broker
metadata:
  name: submariner-broker
  namespace: <broker-namespace>
spec:
  globalnetEnabled: <true-or-false>

Replace broker-namespace with the name of your broker namespace.

Replace true-or-false with true to enable Globalnet.

Note: The metadata name parameter must be submariner-broker.

Apply the file to your YAML file by entering the following command:

oc apply -f submariner-broker.yaml

oc apply -f submariner-broker.yaml

Customize and apply YAML content that is similar to the following example:

apiVersion: submarineraddon.open-cluster-management.io/v1alpha1
kind: SubmarinerConfig
metadata:
    name: submariner
    namespace: <managed-cluster-namespace>
spec:
    gatewayConfig:
      gateways: 1

apiVersion: submarineraddon.open-cluster-management.io/v1alpha1
kind: SubmarinerConfig
metadata:
    name: submariner
    namespace: <managed-cluster-namespace>
spec:
    gatewayConfig:
      gateways: 1

Replace managed-cluster-namespace with the name of your managed cluster. The name of the SubmarinerConfig must be submariner, as shown in the example.

This configuration labels one of the worker nodes as the Submariner gateway on your bare metal cluster.

By default, Submariner uses IP security (IPsec) to establish the secure tunnels between the clusters on the gateway nodes. You can either use the default IPsec NATT port, or you can specify a different port that you configured. When you run this procedure without specifying an IPsec NATT port, 4500/UDP is used for the connections.

From the Azure CLI, run the following command to install the extension:

az extension add --upgrade -s <path-to-extension>

az extension add --upgrade -s <path-to-extension>

Replace path-to-extension with the path to where you downloaded the .whl extension file.

Run the following command to verify that the CLI extension is being used:

az extension list

az extension list

If the extension is being used, the output might resemble the following example:

"experimental": false,
"extensionType": "whl",
"name": "aro",
"path": "<path-to-extension>",
"preview": true,
"version": "1.0.x"

"experimental": false,
"extensionType": "whl",
"name": "aro",
"path": "<path-to-extension>",
"preview": true,
"version": "1.0.x"

From the Azure CLI, register the preview feature by running the following command:

az feature registration create --namespace Microsoft.RedHatOpenShift --name AdminKubeconfig

az feature registration create --namespace Microsoft.RedHatOpenShift --name AdminKubeconfig

Retrieve the administrator kubeconfig by running the following command:

az aro get-admin-kubeconfig -g <resource group> -n <cluster resource name>

az aro get-admin-kubeconfig -g <resource group> -n <cluster resource name>

Note: The az aro command saves the kubeconfig to the local directory and uses the name kubeconfig. To use it, set the environment variable KUBECONFIG to match the path of the file. See the following example:

export KUBECONFIG=<path-to-kubeconfig>
oc get nodes

export KUBECONFIG=<path-to-kubeconfig>
oc get nodes

Select the Kubeconfig Import mode and enter the content from your kubeconfig file in the Kubeconfig window. Follow the instructions in the console to complete the import.

You can verify that your Azure Red Hat OpenShift cluster was imported successfully by navigating to Infrastructure > Clusters.

Create a new node to run Submariner gateway by running the following command:

rosa create machinepool --cluster=<cluster_name> --name=sm-gw-mp --replicas=<number of Submariner gateway > --labels='submariner.io/gateway=true'

rosa create machinepool --cluster=<cluster_name> --name=sm-gw-mp --replicas=<number of Submariner gateway > --labels='submariner.io/gateway=true'

Log in to OpenShift Service on AWS by running the following commands:

rosa login
oc login <rosa-cluster-url>:6443 --username cluster-admin --password <password>

rosa login
oc login <rosa-cluster-url>:6443 --username cluster-admin --password <password>

Create a kubeconfig for your OpenShift Service on AWS cluster by running the following command:

oc config view --flatten=true > rosa_kube/kubeconfig

oc config view --flatten=true > rosa_kube/kubeconfig

Select the Kubeconfig Import mode and enter the content from your kubeconfig file in the Kubeconfig window. Follow the instructions in the console to complete the import.

You can verify that your OpenShift Service on AWS cluster was imported successfully by navigating to Infrastructure > Clusters.

Create a ManagedClusterSet resource on the hub cluster by using the instructions provided in the Creating a ManagedClusterSet documentation. Make sure your entry for the ManagedClusterSet resembles the following content:

apiVersion: cluster.open-cluster-management.io/v1beta2
kind: ManagedClusterSet
metadata:
  name: <managed-cluster-set-name>

apiVersion: cluster.open-cluster-management.io/v1beta2
kind: ManagedClusterSet
metadata:
  name: <managed-cluster-set-name>

Replace managed-cluster-set-name with a name for the ManagedClusterSet that you are creating.

Important: The maximum character length of a Kubernetes namespace is 63 characters. The maximum character length you can use for the <managed-cluster-set-name> is 56 characters. If the character length of <managed-cluster-set-name> exceeds 56 characters, the <managed-cluster-set-name> is cut off from the head.

After the ManagedClusterSet is created, the submariner-addon creates a namespace called <managed-cluster-set-name>-broker and deploys the Submariner broker to it.

Create the Broker configuration on the hub cluster in the <managed-cluster-set-name>-broker namespace by customizing and applying YAML content that is similar to the following example:

apiVersion: submariner.io/v1alpha1
kind: Broker
metadata:
     name: submariner-broker
     namespace: <managed-cluster-set-name>-broker
     labels:
         cluster.open-cluster-management.io/backup: submariner
spec:
     globalnetEnabled: <true-or-false>

apiVersion: submariner.io/v1alpha1
kind: Broker
metadata:
     name: submariner-broker
     namespace: <managed-cluster-set-name>-broker
     labels:
         cluster.open-cluster-management.io/backup: submariner
spec:
     globalnetEnabled: <true-or-false>

Replace managed-cluster-set-name with the name of the managed cluster.

Set the value of globalnetEnabled to true if you want to enable Submariner Globalnet in the ManagedClusterSet.

Add one managed cluster to the ManagedClusterSet by running the following command:

oc label managedclusters <managed-cluster-name> "cluster.open-cluster-management.io/clusterset=<managed-cluster-set-name>" --overwrite

oc label managedclusters <managed-cluster-name> "cluster.open-cluster-management.io/clusterset=<managed-cluster-set-name>" --overwrite

Replace <managed-cluster-name> with the name of the managed cluster that you want to add to the ManagedClusterSet.

Replace <managed-cluster-set-name> with the name of the ManagedClusterSet to which you want to add the managed cluster.

Customize and apply YAML content that is similar to the following example:

apiVersion: submarineraddon.open-cluster-management.io/v1alpha1
kind: SubmarinerConfig
metadata:
    name: submariner
    namespace: <managed-cluster-namespace>
spec:{}

apiVersion: submarineraddon.open-cluster-management.io/v1alpha1
kind: SubmarinerConfig
metadata:
    name: submariner
    namespace: <managed-cluster-namespace>
spec:{}

Replace managed-cluster-namespace with the namespace of your managed cluster.

Note: The name of the SubmarinerConfig must be submariner, as shown in the example.

Deploy Submariner on the managed cluster by customizing and applying YAML content that is similar to the following example:

apiVersion: addon.open-cluster-management.io/v1alpha1
kind: ManagedClusterAddOn
metadata:
     name: submariner
     namespace: <managed-cluster-name>
spec:
     installNamespace: submariner-operator

apiVersion: addon.open-cluster-management.io/v1alpha1
kind: ManagedClusterAddOn
metadata:
     name: submariner
     namespace: <managed-cluster-name>
spec:
     installNamespace: submariner-operator

Replace managed-cluster-name with the name of the managed cluster that you want to use with Submariner.

The installNamespace field in the spec of the ManagedClusterAddOn is the namespace on the managed cluster where it installs Submariner. Currently, Submariner must be installed in the submariner-operator namespace.

After the ManagedClusterAddOn is created, the submariner-addon deploys Submariner to the submariner-operator namespace on the managed cluster. You can view the deployment status of Submariner from the status of this ManagedClusterAddOn.

Note: The name of ManagedClusterAddOn must be submariner.

After Submariner is deployed on the managed cluster, you can verify the Submariner deployment status by checking the status of submariner ManagedClusterAddOn by running the following command:

oc -n <managed-cluster-name> get managedclusteraddons submariner -oyaml

oc -n <managed-cluster-name> get managedclusteraddons submariner -oyaml

Replace managed-cluster-name with the name of the managed cluster.

In the status of the Submariner ManagedClusterAddOn, three conditions indicate the deployment status of Submariner:

Apply an instance of the nginx service on a managed cluster that is in the ManagedClusterSet by entering the following commands:

oc -n default create deployment nginx --image=nginxinc/nginx-unprivileged:stable-alpine
oc -n default expose deployment nginx --port=8080

oc -n default create deployment nginx --image=nginxinc/nginx-unprivileged:stable-alpine
oc -n default expose deployment nginx --port=8080

Export the service by creating a ServiceExport entry by entering a command with the subctl tool that is similar to the following command:

subctl export service --namespace <service-namespace> <service-name>

subctl export service --namespace <service-namespace> <service-name>

Replace service-namespace with the name of the namespace where the service is located. In this example, it is default.

Replace service-name with the name of the service that you are exporting. In this example, it is nginx.

See export in the Submariner documentation for more information about other available flags.

Run the following command from a different managed cluster to confirm that it can access the nginx service:

oc -n default run --generator=run-pod/v1 tmp-shell --rm -i --tty --image quay.io/submariner/nettest -- /bin/bash curl nginx.default.svc.clusterset.local:8080

oc -n default run --generator=run-pod/v1 tmp-shell --rm -i --tty --image quay.io/submariner/nettest -- /bin/bash curl nginx.default.svc.clusterset.local:8080

Repeat those steps for other clusters from which you are removing Submariner.

Tip: You can remove the Submariner add-on from multiple clusters in the same cluster set by selecting multiple clusters and clicking Actions. Select Uninstall Submariner add-ons.

Remove the Submariner deployment for the cluster by running the following command:

oc -n <managed-cluster-namespace> delete managedclusteraddon submariner

oc -n <managed-cluster-namespace> delete managedclusteraddon submariner

Replace managed-cluster-namespace with the namespace of your managed cluster.

Remove the cloud resources of the cluster by running the following command:

oc -n <managed-cluster-namespace> delete submarinerconfig submariner

oc -n <managed-cluster-namespace> delete submarinerconfig submariner

Replace managed-cluster-namespace with the namespace of your managed cluster.

Delete the cluster set to remove the broker details by running the following command:

oc delete managedclusterset <managedclusterset>

oc delete managedclusterset <managedclusterset>

Replace managedclusterset with the name of your managed cluster set.

Run the following command to list any Submariner resources that remain on the cluster:

oc get cluster <CLUSTER_NAME> grep submariner

oc get cluster <CLUSTER_NAME> grep submariner

Replace CLUSTER_NAME with the name of your cluster.

Remove any resources on the list by entering the following command:

oc delete resource <RESOURCE_NAME> cluster <CLUSTER_NAME>

oc delete resource <RESOURCE_NAME> cluster <CLUSTER_NAME>

Replace RESOURCE_NAME with the name of the Submariner resource that you want to remove.

Specify a repository where your backup images are stored by creating a secret that resembles the following YAML content:

apiVersion: v1
kind: Secret
metadata:
  name: restic-config
type: Opaque
stringData:
  RESTIC_REPOSITORY: <my-restic-repository>
  RESTIC_PASSWORD: <my-restic-password>
  AWS_ACCESS_KEY_ID: access
  AWS_SECRET_ACCESS_KEY: password

apiVersion: v1
kind: Secret
metadata:
  name: restic-config
type: Opaque
stringData:
  RESTIC_REPOSITORY: <my-restic-repository>
  RESTIC_PASSWORD: <my-restic-password>
  AWS_ACCESS_KEY_ID: access
  AWS_SECRET_ACCESS_KEY: password

Replace my-restic-repository with the location of the S3 bucket repository where you want to store your backup files.

Replace my-restic-password with the encryption key that is required to access the repository.

Replace access and password with the credentials for your provider, if required. Refer to Preparing a new repository for more information.

If you need to prepare a new repository, see Preparing a new repository for the procedure. If you use that procedure, skip the step that requires you to run the restic init command to initialize the repository. VolSync automatically initializes the repository during the first backup.

Important: When backing up multiple persistent volume claims to the same S3 bucket, the path to the bucket must be unique for each persistent volume claim. Each persistent volume claim is backed up with a separate ReplicationSource, and each requires a separate restic-config secret.

By sharing the same S3 bucket, each ReplicationSource has write access to the entire S3 bucket.

Configure your backup policy by creating a ReplicationSource object that resembles the following YAML content:

apiVersion: volsync.backube/v1alpha1
kind: ReplicationSource
metadata:
  name: mydata-backup
spec:
  sourcePVC: <source>
  trigger:
    schedule: "*/30 * * * *" #\*
  restic:
    pruneIntervalDays: 14
    repository: <restic-config>
    retain:
      hourly: 6
      daily: 5
      weekly: 4
      monthly: 2
      yearly: 1
    copyMethod: Clone
  # The StorageClass to use when creating the PiT copy (same as source PVC if omitted)
  #storageClassName: my-sc-name
  # The VSC to use if the copy method is Snapshot (default if omitted)
  #volumeSnapshotClassName: my-vsc-name

apiVersion: volsync.backube/v1alpha1
kind: ReplicationSource
metadata:
  name: mydata-backup
spec:
  sourcePVC: <source>
  trigger:
    schedule: "*/30 * * * *" #\*
  restic:
    pruneIntervalDays: 14
    repository: <restic-config>
    retain:
      hourly: 6
      daily: 5
      weekly: 4
      monthly: 2
      yearly: 1
    copyMethod: Clone
  # The StorageClass to use when creating the PiT copy (same as source PVC if omitted)
  #storageClassName: my-sc-name
  # The VSC to use if the copy method is Snapshot (default if omitted)
  #volumeSnapshotClassName: my-vsc-name

Replace source with the persistent volume claim that you are backing up.

Replace the value for schedule with how often to run the backup. This example has the schedule for every 30 minutes. See Scheduling your synchronization for more information.

Replace the value of PruneIntervalDays to the number of days that elapse between instances of repacking the data to save space. The prune operation can generate significant I/O traffic while it is running.

Replace restic-config with the name of the secret that you created in step 1.

Set the values for retain to your retention policy for the backed up images.

Best practice: Use Clone for the value of CopyMethod to ensure that a point-in-time image is saved.

For additional information about the backup options, see Backup options in the VolSync documentation.

Create a ReplicationSource custom resource that resembles the following example:

apiVersion: volsync.backube/v1alpha1
kind: ReplicationSource
metadata:
  name: <source>
  namespace: <source-ns>
spec:
  sourcePVC: <source-pvc>
  trigger:
    schedule: "*/6 * * * *" #\*
  rclone:
    rcloneConfigSection: <intermediate-s3-bucket>
    rcloneDestPath: <destination-bucket>
    rcloneConfig: <rclone-secret>
    copyMethod: Snapshot
    storageClassName: <my-sc-name>
    volumeSnapshotClassName: <my-vsc>

apiVersion: volsync.backube/v1alpha1
kind: ReplicationSource
metadata:
  name: <source>
  namespace: <source-ns>
spec:
  sourcePVC: <source-pvc>
  trigger:
    schedule: "*/6 * * * *" #\*
  rclone:
    rcloneConfigSection: <intermediate-s3-bucket>
    rcloneDestPath: <destination-bucket>
    rcloneConfig: <rclone-secret>
    copyMethod: Snapshot
    storageClassName: <my-sc-name>
    volumeSnapshotClassName: <my-vsc>

Replace source-pvc with the name for your replication source custom resource.

Replace source-ns with the namespace of the persistent volume claim where your source is located.

Replace source with the persistent volume claim that you are replicating.

Replace the value of schedule with how often to run the replication. This example has the schedule for every 6 minutes. This value must be within quotation marks. See Scheduling your synchronization for more information.

Replace intermediate-s3-bucket with the path to the configuration section of the Rclone configuration file.

Replace destination-bucket with the path to the object bucket where you want your replicated files copied.

Replace rclone-secret with the name of the secret that contains your Rclone configuration information.

Set the value for copyMethod as Clone, Direct, or Snapshot. This value specifies whether the point-in-time copy is generated, and if so, what method is used for generating it.

Replace my-sc-name with the name of the storage class that you want to use for your point-in-time copy. If not specified, the storage class of the source volume is used.

Replace my-vsc with the name of the VolumeSnapshotClass to use, if you specified Snapshot as your copyMethod. This is not required for other types of copyMethod.

Create a ReplicationDestination custom resource that resembles the following example:

apiVersion: volsync.backube/v1alpha1
kind: ReplicationDestination
metadata:
  name: database-destination
  namespace: dest
spec:
  trigger:
    schedule: "3,9,15,21,27,33,39,45,51,57 * * * *" #/*
  rclone:
    rcloneConfigSection: <intermediate-s3-bucket>
    rcloneDestPath: <destination-bucket>
    rcloneConfig: <rclone-secret>
    copyMethod: Snapshot
    accessModes: [ReadWriteOnce]
    capacity: 10Gi
    storageClassName: <my-sc>
    volumeSnapshotClassName: <my-vsc>

apiVersion: volsync.backube/v1alpha1
kind: ReplicationDestination
metadata:
  name: database-destination
  namespace: dest
spec:
  trigger:
    schedule: "3,9,15,21,27,33,39,45,51,57 * * * *" #/*
  rclone:
    rcloneConfigSection: <intermediate-s3-bucket>
    rcloneDestPath: <destination-bucket>
    rcloneConfig: <rclone-secret>
    copyMethod: Snapshot
    accessModes: [ReadWriteOnce]
    capacity: 10Gi
    storageClassName: <my-sc>
    volumeSnapshotClassName: <my-vsc>

Replace the value for schedule with how often to move the replication to the destination. The schedules for the source and destination must be offset to allow the data to finish replicating before it is pulled from the destination. This example has the schedule for every 6 minutes, offset by 3 minutes. This value must be within quotation marks. See Scheduling your synchronization for more information.

Replace intermediate-s3-bucket with the path to the configuration section of the Rclone configuration file.

Replace destination-bucket with the path to the object bucket where you want your replicated files copied.

Replace rclone-secret with the name of the secret that contains your Rclone configuration information.

Set the value for copyMethod as Clone, Direct, or Snapshot. This value specifies whether the point-in-time copy is generated, and if so, which method is used for generating it.

The value for accessModes specifies the access modes for the persistent volume claim. Valid values are ReadWriteOnce or ReadWriteMany.

The capacity specifies the size of the destination volume, which must be large enough to contain the incoming data.

Replace my-sc with the name of the storage class that you want to use as the destination for your point-in-time copy. If not specified, the system storage class is used.

Replace my-vsc with the name of the VolumeSnapshotClass to use, if you specified Snapshot as your copyMethod. This is not required for other types of copyMethod. If not included, the system default VolumeSnapshotClass is used.