Chapter 1. multicluster global hub

The multicluster global hub operator contains the components of multicluster global hub. The operator deploys all of the required components for global multicluster management. The components include multicluster-global-hub-manager, multicluster-global-hub-grafana, and provided versions of Kafka and PostgreSQL in the multicluster global hub cluster and multicluster-global-hub-agent in the managed hub clusters.

The multicluster global hub manager is used to persist the data into the postgreSQL database. The data is from Kafka transport. The manager also posts the data to the Kafka transport, so it can be synchronized with the data on the managed hub clusters.

The multicluster global hub agent runs on the managed hub clusters. It synchronizes the data between the multicluster global hub cluster and the managed hub clusters. For example, the agent synchronizes the information of the managed clusters from the managed hub clusters to the multicluster global hub cluster and synchronizes the policy or application from the multicluster global hub cluster to the managed hub clusters.

Grafana runs on the multicluster global hub cluster as the main service for multicluster global hub visualizations. The PostgreSQL data collected by the Global Hub Manager is its default DataSource. By exposing the service using the route called multicluster-global-hub-grafana, you can access the multicluster global hub Grafana dashboards by accessing the console.

See the following networking requirements:

See the following supported components:

You must meet the following requirements before you install multicluster global hub in a disconnected environment:

Installing multicluster global hub in a disconnected environment involves the use of a local mirror image registry. At this point, it is assumed that you have set up a mirror registry during the OpenShift Container Platform cluster installation.

Complete the following procedures to provision the mirror registry for multicluster global hub:

In order to have your cluster obtain container images for the multicluster global hub operator from your local mirror registry, rather than from the internet-hosted registries, you must configure an ImageContentSourcePolicy resource on your disconnected cluster to redirect image references to your mirror registry. The ImageContentSourcePolicy only support the image mirror with image digest.

If you mirrored your catalog using the oc adm catalog mirror command, the needed image content source policy configuration is in the imageContentSourcePolicy.yaml file inside the manifests- directory that is created by that command.

If you used the oc-mirror plug-in to mirror your catalog instead, the imageContentSourcePolicy.yaml file is within the oc-mirror-workspace/results- directory created by the oc-mirror plug-in.

In either case, you can apply the policies to your disconnected command using an oc apply or oc replace command such as oc replace -f ./<path>/imageContentSourcePolicy.yaml

The required image content source policy statements can vary based on how you created your mirror registry, but are similar to this example:

apiVersion: operator.openshift.io/v1alpha1
kind: ImageContentSourcePolicy
metadata:
  labels:
    operators.openshift.org/catalog: "true"
  name: global-hub-operator-icsp
spec:
  repositoryDigestMirrors:
  - mirrors:
    - myregistry.example.com:5000/multicluster-globalhub
    source: registry.redhat.io/multicluster-globalhub
  - mirrors:
    - myregistry.example.com:5000/openshift4
    source: registry.redhat.io/openshift4
  - mirrors:
    - myregistry.example.com:5000/redhat
    source: registry.redhat.io/redhat
  - mirrors:
    - myregistry.example.com:5000/rhel9
    source: registry.redhat.io/rhel9
  - mirrors:
    - myregistry.example.com:5000/amq-streams
    source: registry.redhat.io/amq-streams

You can configure different image registries for different managed hubs with the ManagedClusterImageRegistry. See Importing a cluster that has a ManagedClusterImageRegistry to use the ManagedClusterImageRegistry API to replace the agent image.

By completing the previous step, a label and an annotation are added to the selected ManagedCluster. This means that the agent image in the cluster are replaced with the mirror image.

oc get pods -n multicluster-global-hub
NAME                                                READY   STATUS    RESTARTS   AGE
multicluster-global-hub-operator-687584cb7c-fnftj   1/1     Running   0          2m12s

If you have your own instance of Kafka, you can use it as the transport for multicluster global hub. Complete the following steps to integrate an instance of Kafka:

If you have your own PostgreSQL relational database, you can use it as the storage for multicluster global hub.

The minimum required storage size is 20GB. This amount can store 3 managed hubs with 250 managed clusters and 50 policies per managed hub for 18 months. You need to create a secret named multicluster-global-hub-storage in the multicluster-global-hub namespace. The secret must contain the following fields:

The host must be accessible from the multicluster global hub cluster. If your PostgreSQL database is in a Kubernetes cluster, you can consider using the service type with nodePort or LoadBalancer to expose the database. For more information, see Accessing the provisioned postgres database for troubleshooting.

Using an existing Grafana instance might work with multicluster global hub if you are relying on your own Grafana to get metrics from multiple sources, such as Prometheus, from different clusters and if you aggregate the metrics yourself. To get multicluster global hub data into your own Grafana, you need to configure the data source and import the dashboards.

See User authentication in the STRIMZI documentation for more information about how to extract the ca.crt certificate from the secret.

See User authentication in the STRIMZI documentation for the steps to extract the user.crt certificate from the secret.

To import an existing hub cluster as a managed hub cluster, complete the following steps:

Note: If you upgrade multicluster global hub, you must manually add the label global-hub.open-cluster-management.io/deploy-mode=default to all the managed hub clusters.

When you import an existing managed hub cluster in the hosted mode, you only get support for imported managed hub clusters that use a kubeconfig file. Red Hat Advanced Cluster Management for Kubernetes uses this kubeconfig file to generate the auto-import-secret which connects to your managed hub cluster. In the hosted mode, multicluster global hub does not support backup and restore.

Import your managed hub cluster by setting the label global-hub.open-cluster-management.io/deploy-mode=hosted to managedcluster.

With this label, multicluster global hub does the following actions:

After accessing the global hub Grafana data, you can monitor the policies that were configured through the hub cluster environments that are managed.

From the multicluster global hub dashboard, you can identify the compliance status of the policies of the system over a selected time range. The policy compliance status is updated daily, so the dashboard does not display the status of the current day until the following day.

To navigate the multicluster global hub dashboards, you can observe and filter the policy data by grouping them by policy or by cluster.

If you prefer to examine the policy data by using the policy grouping, start from the and the dashboard called Global Hub - Policy Group Compliancy Overview.

This dashboard allows you to filter the policy data based on standard, category, and control. After selecting a specific point in time on the graph, you are directed to the Global Hub - Offending Policies dashboard. The Global Hub - Offending Policies dashboard lists the non-compliant or unknown policies at that time. After selecting a target policy, you can view related events and see what has changed by accessing the Global Hub - What’s Changed / Policies dashboard.

Similarly, if you want to examine the policy data by cluster grouping, begin by using the Global Hub - Cluster Group Compliancy Overview dashboard. The navigation flow is identical to the policy grouping flow, but you select filters that are related to the cluster, such as managed cluster labels and values. Instead of viewing policy events for all clusters, after reaching the Global Hub - What’s Changed / Clusters dashboard, you can view policy events related to an individual cluster.

To understand the health and performance of your Kafka deployment and Postgres database, collect their metrics. When you check the metrics, you can identify issues before they become critical and make informed decisions about resource allocation and capacity planning. If you do not collect and check the metrics, you might have limited visibility into the behavior of your Kafka deployment, making troubleshooting more difficult.

You can check the dashboards and their metrics in the multicluster global hub Grafana. In the Strimzi folder, you can view the following dashboards:

To understand the health and performance of your Postgres config, collect and check the Postgres metrics in the Grafana dashboard. In the Postgres folder, you can view the following dashboard:

To filter your managed clusters and to view their details and events, use the Grafana dashboards. With the Grafana dashboard, you can filter your managed clusters based on: hub, labels and name. Additionally, you can view the distribution of your managed clusters based on: hub, status, cloud, and version.

In the Cluster folder, you can view the following dashboard:

If the default Grafana alert rules do not provide useful information, you can delete the Grafana alert rule by including a deleteRules section in the multicluster-global-hub-custom-alerting config map. See Customize Grafana alerting resources for more information about the multicluster-global-hub-custom-alerting config map.

To delete all of the default alerts, the deleteRules configuration section should resemble the following example:

    deleteRules:
      - orgId: 1
        uid: globalhub_suspicious_policy_change
      - orgId: 1
        uid: globalhub_cluster_compliance_status_change_frequently
      - orgId: 1
        uid: globalhub_high_number_of_policy_events
      - orgId: 1
        uid: globalhub_data_retention_job
      - orgId: 1
        uid: globalhub_local_compliance_job

The multicluster global hub supports creating custom Grafana alerts. Complete the following steps to customize your Grafana alerts:

apiVersion: v1
kind: Secret
metadata:
  name: multicluster-global-hub-custom-grafana-config
  namespace: multicluster-global-hub
type: Opaque
stringData:
  grafana.ini: |
    [smtp]
    enabled = true
    host = smtp.google.com:465
    user = <example@google.com>
    password = <password>
    ;cert_file =
    ;key_file =
    skip_verify = true
    from_address = <example@google.com>
    from_name = Grafana
    ;ehlo_identity = dashboard.example.com 

apiVersion: v1
data:
  alerting.yaml: |
    contactPoints:
      - orgId: 1
        name: globalhub_policy
        receivers:
          - uid: globalhub_policy_alert_email
            type: email
            settings:
              addresses: <example@redhat.com>
              singleEmail: false
          - uid: globalhub_policy_alert_slack
            type: slack
            settings:
              url: <Slack-webhook-URL>
              title: |
                {{ template "globalhub.policy.title" . }}
              text: |
                {{ template "globalhub.policy.message" . }}
    policies:
      - orgId: 1
        receiver: globalhub_policy
        group_by: ['grafana_folder', 'alertname']
        matchers:
          - grafana_folder = Policy
        repeat_interval: 1d
    deleteRules:
      - orgId: 1
        uid: [Alert Rule Uid]
    muteTimes:
      - orgId: 1
        name: mti_1
        time_intervals:
          - times:
              - start_time: '06:00'
                end_time: '23:59'
                location: 'UTC'
            weekdays: ['monday:wednesday', 'saturday', 'sunday']
            months: ['1:3', 'may:august', 'december']
            years: ['2020:2022', '2030']
            days_of_month: ['1:5', '-3:-1']
kind: ConfigMap
metadata:
  name: multicluster-global-hub-custom-alerting
  namespace: multicluster-global-hub

If you need to restore your multicluster global hub cluster, see prepare the new hub cluster. Install the multicluster global hub operator but do not create the multicluster global hub custom resource (CR) because the CR is automatically restored.

You need to enable multicluster global hub.

To enable global search, add the global-search-preview=true annotation to the search-v2-operator resource by running the following command:

oc annotate search search-v2-operator -n open-cluster-management global-search-preview=true

The search operator is updated with the following status condition:

status:
  conditions:
    - lastTransitionTime: '2024-05-31T19:49:37Z'
      message: None
      reason: None
      status: 'True'
      type: GlobalSearchReady

With multicluster global hub, you can manage large fleets of clusters by implementing the event-driven architecture of multicluster global hub. multicluster global hub connects itself to your managed hub clusters. With the multicluster global hub event-based design, you can enable multicluster global hub to communicate, organize, synchronize, and transfer resources and cluster states among all you hub clusters.

The manage cluster migration process requires coordination between a source and target Red Hat Advanced Cluster Management hub cluster. The multicluster-global-hub-agent performs the migrations tasks on the source and target hub clusters. The multicluster-global-hub-manager controls the flow of the migration between the source and target hub clusters and manages the ManagedClusterMigration resources.

During the managed cluster migration, the source and target hub clusters go through different phases. These phases and their conditions help track the status changes of the hub clusters throughout the migration process. See the following table for an outline of each phase and its condition:

See the following table for the supported versions and the corresponding multicluster global hub source, and target hub versions:

To prepare your multicluster global hub environment for migration, you can create a Brownfield environment by deploying the multicluster global hub control plane directly into the source hub cluster. Then, you can import the target hub cluster into the multicluster global hub environment that is in the Hosted mode.

Complete the following steps:

After you configure your multicluster global hub migration environment, migrate your managed clusters. Complete the following steps to migrate the cluster1 sample and the associated resources from hub1 to hub2 hub clusters:

You can create a ManagedClusterMigration resource in the multicluster global hub migration environment by directly selecting your managed clusters by name or by using a Placement resource.

To migrate your specific managed clusters by name, complete the following steps:

To migrate your managed clusters with the Placement resource, complete the following steps:

During the migration process, you can track the statuses of your managed clusters by using the multicluster global hub ConfigMap or the migration custom resource.

multicluster global hub automatically creates a ConfigMap which gives you a detailed cluster list, telling you if the cluster migration fails or succeeds. The migration custom resources gives you the status of the entire migration process.

Track the migration status of your managed clusters by completing the following steps:

status:
  conditions:
    - type: ResourceValidated
      status: "True"
      message: Migration resources have been validated
    - type: ResourceInitialized
      status: "True"
      message: All source and target hubs have been initialized
    - type: ResourceDeployed
      status: "True"
      message: Resources have been successfully deployed to the target hub cluster
    - type: ClusterRegistered
      status: "True"
      message: All migrated clusters have been successfully registered
    - type: ResourceCleaned
      status: "True"
      message: Resources have been successfully cleaned up from the hub clusters
  phase: Completed

To learn more about importing hub clusters, see the following resources:

When the multicluster global hub upgrade to version 1.4.0 removes your historical data, recover your data by completing the following steps:

After multicluster global hub upgrades to version 1.4.0, it switches to the new built-in Postgres instance. The global hub operator does not automatically delete resources associated with the legacy Postgres instance. To delete the legacy Postgres resources, complete the following steps: