Networking

Open and edit the SubmarinerConfig file by running the following command. Replace values where needed:

oc edit submarinerconfig -n <managed-cluster-ns> submariner

Add the following annotations in the metadata field. Replace values where needed:

Note: All the IDs you need to add are alphanumeric.

annotations:
  submariner.io/control-plane-sg-id: <control-plane-group-id> 1
  submariner.io/subnet-id-list: <subnet-id-list> 2
  submariner.io/vpc-id: <custom-vpc-id> 3
  submariner.io/worker-sg-id: <worker-security-group-id> 4

Customize and apply YAML content that is similar to the following example:

apiVersion: submarineraddon.open-cluster-management.io/v1alpha1
kind: SubmarinerConfig
metadata:
    name: submariner
    namespace: <managed-cluster-namespace>
spec:
    gatewayConfig:
      gateways: 1

Replace managed-cluster-namespace with the name of your managed cluster. The name of the SubmarinerConfig must be submariner, as shown in the example.

This configuration labels one of the worker nodes as the Submariner gateway on your bare metal cluster.

By default, Submariner uses IP security (IPsec) to establish the secure tunnels between the clusters on the gateway nodes. You can either use the default IPsec NATT port, or you can specify a different port that you configured. When you run this procedure without specifying an IPsec NATT port, 4500/UDP is used for the connections.

From the Azure CLI, run the following command to install the extension:

az extension add --upgrade -s <path-to-extension>

Replace path-to-extension with the path to where you downloaded the .whl extension file.

Run the following command to verify that the CLI extension is being used:

az extension list

If the extension is being used, the output might resemble the following example:

"experimental": false,
"extensionType": "whl",
"name": "aro",
"path": "<path-to-extension>",
"preview": true,
"version": "1.0.x"

From the Azure CLI, register the preview feature by running the following command:

az feature registration create --namespace Microsoft.RedHatOpenShift --name AdminKubeconfig

Retrieve the administrator kubeconfig by running the following command:

az aro get-admin-kubeconfig -g <resource group> -n <cluster resource name>

Note: The az aro command saves the kubeconfig to the local directory and uses the name kubeconfig. To use it, set the environment variable KUBECONFIG to match the path of the file. See the following example:

export KUBECONFIG=<path-to-kubeconfig>
oc get nodes

Log in to OpenShift Service on AWS by running the following commands:

rosa login
oc login <rosa-cluster-url>:6443 --username cluster-admin --password <password>

Create a kubeconfig for your OpenShift Service on AWS cluster by running the following command:

oc config view --flatten=true > rosa_kube/kubeconfig

Create a ManagedClusterSet resource on the hub cluster by using the instructions provided in the Creating a ManagedClusterSet documentation. Make sure your entry for the ManagedClusterSet resembles the following content:

apiVersion: cluster.open-cluster-management.io/v1beta2
kind: ManagedClusterSet
metadata:
  name: <managed-cluster-set-name>

Replace managed-cluster-set-name with a name for the ManagedClusterSet that you are creating.

Important: The maximum character length of a Kubernetes namespace is 63 characters. The maximum character length you can use for the <managed-cluster-set-name> is 56 characters. If the character length of <managed-cluster-set-name> exceeds 56 characters, the <managed-cluster-set-name> is cut off from the head.

After the ManagedClusterSet is created, the submariner-addon creates a namespace called <managed-cluster-set-name>-broker and deploys the Submariner broker to it.

Create the Broker configuration on the hub cluster in the <managed-cluster-set-name>-broker namespace by customizing and applying YAML content that is similar to the following example:

apiVersion: submariner.io/v1alpha1
kind: Broker
metadata:
     name: submariner-broker
     namespace: <managed-cluster-set-name>-broker
     labels:
         cluster.open-cluster-management.io/backup: submariner
spec:
     globalnetEnabled: <true-or-false>

Replace managed-cluster-set-name with the name of the managed cluster.

Set the value of globalnetEnabled to true if you want to enable Submariner Globalnet in the ManagedClusterSet.

Add one managed cluster to the ManagedClusterSet by running the following command:

oc label managedclusters <managed-cluster-name> "cluster.open-cluster-management.io/clusterset=<managed-cluster-set-name>" --overwrite

Replace <managed-cluster-name> with the name of the managed cluster that you want to add to the ManagedClusterSet.

Replace <managed-cluster-set-name> with the name of the ManagedClusterSet to which you want to add the managed cluster.

Customize and apply YAML content that is similar to the following example:

apiVersion: submarineraddon.open-cluster-management.io/v1alpha1
kind: SubmarinerConfig
metadata:
    name: submariner
    namespace: <managed-cluster-namespace>
spec:{}

Replace managed-cluster-namespace with the namespace of your managed cluster.

Note: The name of the SubmarinerConfig must be submariner, as shown in the example.

Deploy Submariner on the managed cluster by customizing and applying YAML content that is similar to the following example:

apiVersion: addon.open-cluster-management.io/v1alpha1
kind: ManagedClusterAddOn
metadata:
     name: submariner
     namespace: <managed-cluster-name>
spec:
     installNamespace: submariner-operator

Replace managed-cluster-name with the name of the managed cluster that you want to use with Submariner.

The installNamespace field in the spec of the ManagedClusterAddOn is the namespace on the managed cluster where it installs Submariner. Currently, Submariner must be installed in the submariner-operator namespace.

After the ManagedClusterAddOn is created, the submariner-addon deploys Submariner to the submariner-operator namespace on the managed cluster. You can view the deployment status of Submariner from the status of this ManagedClusterAddOn.

Note: The name of ManagedClusterAddOn must be submariner.

After Submariner is deployed on the managed cluster, you can verify the Submariner deployment status by checking the status of submariner ManagedClusterAddOn by running the following command:

oc -n <managed-cluster-name> get managedclusteraddons submariner -oyaml

Replace managed-cluster-name with the name of the managed cluster.

In the status of the Submariner ManagedClusterAddOn, three conditions indicate the deployment status of Submariner:

Apply an instance of the nginx service on a managed cluster that is in the ManagedClusterSet by entering the following commands:

oc -n default create deployment nginx --image=nginxinc/nginx-unprivileged:stable-alpine
oc -n default expose deployment nginx --port=8080

Export the service by creating a ServiceExport entry by entering a command with the subctl tool that is similar to the following command:

subctl export service --namespace <service-namespace> <service-name>

Replace service-namespace with the name of the namespace where the service is located. In this example, it is default.

Replace service-name with the name of the service that you are exporting. In this example, it is nginx.

See export in the Submariner documentation for more information about other available flags.

Run the following command from a different managed cluster to confirm that it can access the nginx service:

oc -n default run --generator=run-pod/v1 tmp-shell --rm -i --tty --image quay.io/submariner/nettest -- /bin/bash curl nginx.default.svc.clusterset.local:8080

Repeat those steps for other clusters from which you are removing Submariner.

Tip: You can remove the Submariner add-on from multiple clusters in the same cluster set by selecting multiple clusters and clicking Actions. Select Uninstall Submariner add-ons.

Remove the Submariner deployment for the cluster by running the following command:

oc -n <managed-cluster-namespace> delete managedclusteraddon submariner

Replace managed-cluster-namespace with the namespace of your managed cluster.

Remove the cloud resources of the cluster by running the following command:

oc -n <managed-cluster-namespace> delete submarinerconfig submariner

Replace managed-cluster-namespace with the namespace of your managed cluster.

Delete the cluster set to remove the broker details by running the following command:

oc delete managedclusterset <managedclusterset>

Replace managedclusterset with the name of your managed cluster set.

Run the following command to list any Submariner resources that remain on the cluster:

oc get cluster <CLUSTER_NAME> grep submariner

Replace CLUSTER_NAME with the name of your cluster.

Remove any resources on the list by entering the following command:

oc delete resource <RESOURCE_NAME> cluster <CLUSTER_NAME>

Replace RESOURCE_NAME with the name of the Submariner resource that you want to remove.