Chapter 1. multicluster global hub

The multicluster global hub operator contains the components of multicluster global hub. The operator deploys all of the required components for global multicluster management. The components include multicluster-global-hub-manager, multicluster-global-hub-grafana, and provided versions of Kafka and PostgreSQL in the multicluster global hub cluster and multicluster-global-hub-agent in the managed hub clusters.

The operator also leverages the manifestwork custom resource to deploy the Red Hat Advanced Cluster Management for Kubernetes operator on the managed cluster. After the Red Hat Advanced Cluster Management operator is deployed on the managed cluster, the managed cluster becomes a standard Red Hat Advanced Cluster Management cluster. This hub cluster is now a managed hub cluster.

The multicluster global hub manager is used to persist the data into the postgreSQL database. The data is from Kafka transport. The manager also posts the data to the Kafka transport, so it can be synchronized with the data on the managed hub clusters.

The multicluster global hub agent runs on the managed hub clusters. It synchronizes the data between the multicluster global hub cluster and the managed hub clusters. For example, the agent synchronizes the information of the managed clusters from the managed hub clusters to the multicluster global hub cluster and synchronizes the policy or application from the multicluster global hub cluster to the managed hub clusters.

Grafana runs on the multicluster global hub cluster as the main service for multicluster global hub visualizations. The PostgreSQL data collected by the Global Hub Manager is its default DataSource. By exposing the service using the route called multicluster-global-hub-grafana, you can access the multicluster global hub Grafana dashboards by accessing the console.

To install Global hub, you need the following requirements:

Required access: Cluster administrator

OpenShift Container Platform Dedicated environment required access: You must have cluster-admin permissions. By default dedicated-admin role does not have the required permissions to create namespaces in the OpenShift Container Platform Dedicated environment.

See the following networking requirements:

Learn about supported platforms and components.

You must meet the following requirements before you install multicluster global hub in a disconnected environment:

Installing multicluster global hub in a disconnected environment involves the use of a local mirror image registry. At this point, it is assumed that you have set up a mirror registry during the OpenShift Container Platform cluster installation.

Complete the following procedures to provision the mirror registry for multicluster global hub:

In order to have your cluster obtain container images for the multicluster global hub operator from your local mirror registry, rather than from the internet-hosted registries, you must configure an ImageContentSourcePolicy resource on your disconnected cluster to redirect image references to your mirror registry. The ImageContentSourcePolicy only support the image mirror with image digest.

If you mirrored your catalog using the oc adm catalog mirror command, the needed image content source policy configuration is in the imageContentSourcePolicy.yaml file inside the manifests- directory that is created by that command.

If you used the oc-mirror plug-in to mirror your catalog instead, the imageContentSourcePolicy.yaml file is within the oc-mirror-workspace/results- directory created by the oc-mirror plug-in.

In either case, you can apply the policies to your disconnected command using an oc apply or oc replace command such as oc replace -f ./<path>/imageContentSourcePolicy.yaml

The required image content source policy statements can vary based on how you created your mirror registry, but are similar to this example:

apiVersion: operator.openshift.io/v1alpha1
kind: ImageContentSourcePolicy
metadata:
  labels:
    operators.openshift.org/catalog: "true"
  name: global-hub-operator-icsp
spec:
  repositoryDigestMirrors:
  - mirrors:
    - myregistry.example.com:5000/multicluster-globalhub
    source: registry.redhat.io/multicluster-globalhub
  - mirrors:
    - myregistry.example.com:5000/openshift4
    source: registry.redhat.io/openshift4
  - mirrors:
    - myregistry.example.com:5000/redhat
    source: registry.redhat.io/redhat

You can configure different image registries for different managed hubs with the ManagedClusterImageRegistry. See Importing a cluster that has a ManagedClusterImageRegistry to use the ManagedClusterImageRegistry API to replace the agent image.

By completing the previous step, a label and an annotation are added to the selected ManagedCluster. This means that the agent image in the cluster are replaced with the mirror image.

oc get pods -n multicluster-global-hub
NAME                                                READY   STATUS    RESTARTS   AGE
multicluster-global-hub-operator-687584cb7c-fnftj   1/1     Running   0          2m12s

If you have your own instance of Kafka, you can use it as the transport for multicluster global hub. Kafka 3.3 is the tested version. Complete the following steps to integrate an instance of Kafka:

If you have your own PostgreSQL relational database, you can use it as the storage for multicluster global hub. PostgreSQL 13 is the tested version.

The minimum required storage size is 20GB. This amount can store 3 managed hubs with 250 managed clusters and 50 policies per managed hub for 18 months. You need to create a secret named multicluster-global-hub-storage in the multicluster-global-hub namespace. The secret must contain the following fields:

oc create secret generic multicluster-global-hub-storage -n multicluster-global-hub \
    --from-literal=database_uri=<postgresql-uri> \
    --from-literal=database_uri_with_readonlyuser=<postgresql-uri-with-readonlyuser> \
    --from-file=ca.crt=<CA-for-postgres-server>

The host must be accessible from the multicluster global hub cluster. If your PostgreSQL database is in a Kubernetes cluster, you can consider using the service type with nodePort or LoadBalancer to expose the database. For more information, see Accessing the provisioned postgres database for troubleshooting.

Using an existing Grafana instance might work with multicluster global hub if you are relying on your own Grafana to get metrics from multiple sources, such as Prometheus, from different clusters and if you aggregate the metrics yourself. To get multicluster global hub data into your own Grafana, you need to configure the data source and import the dashboards.

See User authentication in the STRIMZI documentation for more information about how to extract the ca.crt certificate from the secret.

See User authentication in the STRIMZI documentation for the steps to extract the user.crt certificate from the secret.

After accessing the global hub Grafana data, you can monitor the policies that were configured through the hub cluster environments that are managed.

From the multicluster global hub dashboard, you can identify the compliance status of the policies of the system over a selected time range. The policy compliance status is updated daily, so the dashboard does not display the status of the current day until the following day.

To navigate the multicluster global hub dashboards, you can observe and filter the policy data by grouping them by policy or by cluster.

If you prefer to examine the policy data by using the policy grouping, start from the and the dashboard called Global Hub - Policy Group Compliancy Overview.

This dashboard allows you to filter the policy data based on standard, category, and control. After selecting a specific point in time on the graph, you are directed to the Global Hub - Offending Policies dashboard. The Global Hub - Offending Policies dashboard lists the non-compliant or unknown policies at that time. After selecting a target policy, you can view related events and see what has changed by accessing the Global Hub - What’s Changed / Policies dashboard.

Similarly, if you want to examine the policy data by cluster grouping, begin by using the Global Hub - Cluster Group Compliancy Overview dashboard. The navigation flow is identical to the policy grouping flow, but you select filters that are related to the cluster, such as managed cluster labels and values. Instead of viewing policy events for all clusters, after reaching the Global Hub - What’s Changed / Clusters dashboard, you can view policy events related to an individual cluster.

If the default Grafana alert rules do not provide useful information, you can delete the Grafana alert rule by including a deleteRules section in the multicluster-global-hub-custom-alerting config map. See Customize Grafana alerting resources for more information about the multicluster-global-hub-custom-alerting config map.

To delete all of the default alerts, the deleteRules configuration section should resemble the following example:

    deleteRules:
      - orgId: 1
        uid: globalhub_suspicious_policy_change
      - orgId: 1
        uid: globalhub_cluster_compliance_status_change_frequently
      - orgId: 1
        uid: globalhub_high_number_of_policy_events
      - orgId: 1
        uid: globalhub_data_retention_job
      - orgId: 1
        uid: globalhub_local_compliance_job

The multicluster global hub supports creating custom Grafana alerts. Complete the following steps to customize your Grafana alerts:

apiVersion: v1
kind: Secret
metadata:
  name: multicluster-global-hub-custom-grafana-config
  namespace: multicluster-global-hub
type: Opaque
stringData:
  grafana.ini: |
    [smtp]
    enabled = true
    host = smtp.google.com:465
    user = <example@google.com>
    password = <password>
    ;cert_file =
    ;key_file =
    skip_verify = true
    from_address = <example@google.com>
    from_name = Grafana
    ;ehlo_identity = dashboard.example.com 1

apiVersion: v1
data:
  alerting.yaml: |
    contactPoints:
      - orgId: 1
        name: globalhub_policy
        receivers:
          - uid: globalhub_policy_alert_email
            type: email
            settings:
              addresses: <example@redhat.com>
              singleEmail: false
          - uid: globalhub_policy_alert_slack
            type: slack
            settings:
              url: <Slack-webhook-URL>
              title: |
                {{ template "globalhub.policy.title" . }}
              text: |
                {{ template "globalhub.policy.message" . }}
    policies:
      - orgId: 1
        receiver: globalhub_policy
        group_by: ['grafana_folder', 'alertname']
        matchers:
          - grafana_folder = Policy
        repeat_interval: 1d
    deleteRules:
      - orgId: 1
        uid: [Alert Rule Uid]
    muteTimes:
      - orgId: 1
        name: mti_1
        time_intervals:
          - times:
              - start_time: '06:00'
                end_time: '23:59'
                location: 'UTC'
            weekdays: ['monday:wednesday', 'saturday', 'sunday']
            months: ['1:3', 'may:august', 'december']
            years: ['2020:2022', '2030']
            days_of_month: ['1:5', '-3:-1']
kind: ConfigMap
metadata:
  name: multicluster-global-hub-custom-alerting
  namespace: multicluster-global-hub

If you need to restore your multicluster global hub cluster, see prepare the new hub cluster. Install the multicluster global hub operator but do not create the multicluster global hub custom resource (CR) because the CR is automatically restored.