Declarative cluster configuration
Configuring an OpenShift cluster with cluster configurations by using OpenShift GitOps and creating and synchronizing applications in the default and code mode by using the GitOps CLI
Abstract
Chapter 1. Configuring an OpenShift cluster by deploying an application with cluster configurations Copy linkLink copied to clipboard!
With Red Hat OpenShift GitOps, you can configure Argo CD to recursively sync the content of a Git directory with an application that contains custom configurations for your cluster.
1.1. Prerequisites Copy linkLink copied to clipboard!
- You have logged in to the OpenShift Container Platform cluster as an administrator.
- You have installed the Red Hat OpenShift GitOps Operator on your OpenShift Container Platform cluster.
1.2. Using an Argo CD instance to manage cluster-scoped resources Copy linkLink copied to clipboard!
To manage cluster-scoped resources, update the existing Subscription
object for the Red Hat OpenShift GitOps Operator and add the namespace of the Argo CD instance to the ARGOCD_CLUSTER_CONFIG_NAMESPACES
environment variable in the spec
section.
Procedure
- In the Administrator perspective of the web console, navigate to Operators → Installed Operators → Red Hat OpenShift GitOps → Subscription.
- Click the Actions drop-down menu then click Edit Subscription.
On the openshift-gitops-operator Subscription details page, under the YAML tab, edit the
Subscription
YAML file by adding the namespace of the Argo CD instance to theARGOCD_CLUSTER_CONFIG_NAMESPACES
environment variable in thespec
section:Copy to Clipboard Copied! Toggle word wrap Toggle overflow To verify that the Argo instance is configured with a cluster role to manage cluster-scoped resources, perform the following steps:
- Navigate to User Management → Roles and from the Filter drop-down menu select Cluster-wide Roles.
Search for the
argocd-application-controller
by using the Search by name field.The Roles page displays the created cluster role.
TipAlternatively, in the OpenShift CLI, run the following command:
oc auth can-i create oauth -n openshift-gitops --as system:serviceaccount:openshift-gitops:openshift-gitops-argocd-application-controller
oc auth can-i create oauth -n openshift-gitops --as system:serviceaccount:openshift-gitops:openshift-gitops-argocd-application-controller
Copy to Clipboard Copied! Toggle word wrap Toggle overflow The output
yes
verifies that the Argo instance is configured with a cluster role to manage cluster-scoped resources. Else, check your configurations and take necessary steps as required.
1.3. Default permissions of an Argo CD instance Copy linkLink copied to clipboard!
By default Argo CD instance has the following permissions:
-
Argo CD instance has the
admin
privileges to manage resources only in the namespace where it is deployed. For instance, an Argo CD instance deployed in the foo namespace has theadmin
privileges to manage resources only for that namespace. Argo CD has the following cluster-scoped permissions because Argo CD requires cluster-wide
read
privileges on resources to function appropriately:Copy to Clipboard Copied! Toggle word wrap Toggle overflow
You can edit the cluster roles used by the
argocd-server
andargocd-application-controller
components where Argo CD is running such that thewrite
privileges are limited to only the namespaces and resources that you wish Argo CD to manage.oc edit clusterrole argocd-server oc edit clusterrole argocd-application-controller
$ oc edit clusterrole argocd-server $ oc edit clusterrole argocd-application-controller
Copy to Clipboard Copied! Toggle word wrap Toggle overflow
1.4. Running the Argo CD instance at the cluster-level Copy linkLink copied to clipboard!
The default Argo CD instance and the accompanying controllers, installed by the Red Hat OpenShift GitOps Operator, can now run on the infrastructure nodes of the cluster by setting a simple configuration toggle.
Procedure
Label the existing nodes:
oc label node <node-name> node-role.kubernetes.io/infra=""
$ oc label node <node-name> node-role.kubernetes.io/infra=""
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Optional: If required, you can also apply taints and isolate the workloads on infrastructure nodes and prevent other workloads from scheduling on these nodes:
oc adm taint nodes -l node-role.kubernetes.io/infra \ infra=reserved:NoSchedule infra=reserved:NoExecute
$ oc adm taint nodes -l node-role.kubernetes.io/infra \ infra=reserved:NoSchedule infra=reserved:NoExecute
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Add the
runOnInfra
toggle in theGitOpsService
custom resource:Copy to Clipboard Copied! Toggle word wrap Toggle overflow Optional: If taints have been added to the nodes, then add
tolerations
to theGitOpsService
custom resource, for example:Copy to Clipboard Copied! Toggle word wrap Toggle overflow -
Verify that the workloads in the
openshift-gitops
namespace are now scheduled on the infrastructure nodes by viewing Pods → Pod details for any pod in the console UI.
Any nodeSelectors
and tolerations
manually added to the default Argo CD custom resource are overwritten by the toggle and tolerations
in the GitOpsService
custom resource.
1.5. Creating an application by using the Argo CD dashboard Copy linkLink copied to clipboard!
Argo CD provides a dashboard which allows you to create applications.
This sample workflow walks you through the process of configuring Argo CD to recursively sync the content of the cluster
directory to the cluster-configs
application. The directory defines the OpenShift Container Platform web console cluster configurations that add a link to the Red Hat Developer Blog - Kubernetes under the
menu in the web console, and defines a namespace
spring-petclinic
on the cluster.
Prerequisites
- You have logged in to the OpenShift Container Platform cluster as an administrator.
- You have installed the Red Hat OpenShift GitOps Operator on your OpenShift Container Platform cluster.
- You have logged in to Argo CD instance.
Procedure
- In the Argo CD dashboard, click NEW APP to add a new Argo CD application.
For this workflow, create a cluster-configs application with the following configurations:
- Application Name
-
cluster-configs
- Project
-
default
- Sync Policy
-
Manual
- Repository URL
-
https://github.com/redhat-developer/openshift-gitops-getting-started
- Revision
-
HEAD
- Path
-
cluster
- Destination
-
https://kubernetes.default.svc
- Namespace
-
spring-petclinic
- Directory Recurse
-
checked
- Click CREATE to create your application.
- Open the Administrator perspective of the web console and expand Administration → Namespaces.
-
Search for and select the namespace, then enter
argocd.argoproj.io/managed-by=openshift-gitops
in the Label field so that the Argo CD instance in theopenshift-gitops
namespace can manage your namespace.
1.6. Creating an application by using the oc tool Copy linkLink copied to clipboard!
You can create Argo CD applications in your terminal by using the oc
tool.
Prerequisites
- You have installed the Red Hat OpenShift GitOps Operator on your OpenShift Container Platform cluster.
- You have logged in to an Argo CD instance.
Procedure
Download the sample application:
git clone git@github.com:redhat-developer/openshift-gitops-getting-started.git
$ git clone git@github.com:redhat-developer/openshift-gitops-getting-started.git
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Create the application:
oc create -f openshift-gitops-getting-started/argo/app.yaml
$ oc create -f openshift-gitops-getting-started/argo/app.yaml
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Run the
oc get
command to review the created application:oc get application -n openshift-gitops
$ oc get application -n openshift-gitops
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Add a label to the namespace your application is deployed in so that the Argo CD instance in the
openshift-gitops
namespace can manage it:oc label namespace spring-petclinic argocd.argoproj.io/managed-by=openshift-gitops
$ oc label namespace spring-petclinic argocd.argoproj.io/managed-by=openshift-gitops
Copy to Clipboard Copied! Toggle word wrap Toggle overflow
1.7. Creating an application in the default mode by using the GitOps CLI Copy linkLink copied to clipboard!
You can create applications in the default mode by using the GitOps argocd
CLI.
This sample workflow walks you through the process of configuring Argo CD to recursively sync the content of the cluster
directory to the cluster-configs
application. The directory defines the OpenShift Container Platform cluster configurations and the spring-petclinic
namespace on the cluster.
Prerequisites
- You have installed the Red Hat OpenShift GitOps Operator on your OpenShift Container Platform cluster.
-
You have installed the OpenShift CLI (
oc
). -
You have installed the Red Hat OpenShift GitOps
argocd
CLI. - You have logged in to Argo CD instance.
Procedure
Get the
admin
account password for the Argo CD server:ADMIN_PASSWD=$(oc get secret openshift-gitops-cluster -n openshift-gitops -o jsonpath='{.data.admin\.password}' | base64 -d)
$ ADMIN_PASSWD=$(oc get secret openshift-gitops-cluster -n openshift-gitops -o jsonpath='{.data.admin\.password}' | base64 -d)
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Get the Argo CD server URL:
SERVER_URL=$(oc get routes openshift-gitops-server -n openshift-gitops -o jsonpath='{.status.ingress[0].host}')
$ SERVER_URL=$(oc get routes openshift-gitops-server -n openshift-gitops -o jsonpath='{.status.ingress[0].host}')
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Log in to the Argo CD server by using the
admin
account password and enclosing it in single quotes:ImportantEnclosing the password in single quotes ensures that special characters, such as
$
, are not misinterpreted by the shell. Always use single quotes to enclose the literal value of the password.argocd login --username admin --password ${ADMIN_PASSWD} ${SERVER_URL}
$ argocd login --username admin --password ${ADMIN_PASSWD} ${SERVER_URL}
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Example
argocd login --username admin --password '<password>' openshift-gitops.openshift-gitops.apps-crc.testing
$ argocd login --username admin --password '<password>' openshift-gitops.openshift-gitops.apps-crc.testing
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Verify that you are able to run
argocd
commands in the default mode by listing all applications:argocd app list
$ argocd app list
Copy to Clipboard Copied! Toggle word wrap Toggle overflow If the configuration is correct, then existing applications will be listed with the following header:
Sample output
NAME CLUSTER NAMESPACE PROJECT STATUS HEALTH SYNCPOLICY CONDITIONS REPO PATH TARGET
NAME CLUSTER NAMESPACE PROJECT STATUS HEALTH SYNCPOLICY CONDITIONS REPO PATH TARGET
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Create an application in the default mode:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Label the
spring-petclinic
destination namespace to be managed by theopenshif-gitops
Argo CD instance:oc label ns spring-petclinic "argocd.argoproj.io/managed-by=openshift-gitops"
$ oc label ns spring-petclinic "argocd.argoproj.io/managed-by=openshift-gitops"
Copy to Clipboard Copied! Toggle word wrap Toggle overflow List the available applications to confirm that the application is created successfully:
argocd app list
$ argocd app list
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Even though the
cluster-configs
Argo CD application has theHealthy
status, it is not automatically synced due to itsnone
sync policy, causing it to remain in theOutOfSync
status.
1.8. Creating an application in core mode by using the GitOps CLI Copy linkLink copied to clipboard!
You can create applications in core
mode by using the GitOps argocd
CLI.
This sample workflow walks you through the process of configuring Argo CD to recursively sync the content of the cluster
directory to the cluster-configs
application. The directory defines the OpenShift Container Platform cluster configurations and the spring-petclinic
namespace on the cluster.
Prerequisites
- You have installed the Red Hat OpenShift GitOps Operator on your OpenShift Container Platform cluster.
-
You have installed the OpenShift CLI (
oc
). -
You have installed the Red Hat OpenShift GitOps
argocd
CLI.
Procedure
Log in to the OpenShift Container Platform cluster by using the
oc
CLI tool:oc login -u <username> -p <password> <server_url>
$ oc login -u <username> -p <password> <server_url>
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Example
oc login -u kubeadmin -p '<password>' https://api.crc.testing:6443
$ oc login -u kubeadmin -p '<password>' https://api.crc.testing:6443
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Check whether the context is set correctly in the
kubeconfig
file:oc config current-context
$ oc config current-context
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Set the default namespace of the current context to
openshift-gitops
:oc config set-context --current --namespace openshift-gitops
$ oc config set-context --current --namespace openshift-gitops
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Set the following environment variable to override the Argo CD component names:
export ARGOCD_REPO_SERVER_NAME=openshift-gitops-repo-server
$ export ARGOCD_REPO_SERVER_NAME=openshift-gitops-repo-server
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Verify that you are able to run
argocd
commands incore
mode by listing all applications:argocd app list --core
$ argocd app list --core
Copy to Clipboard Copied! Toggle word wrap Toggle overflow If the configuration is correct, then existing applications will be listed with the following header:
Sample output
NAME CLUSTER NAMESPACE PROJECT STATUS HEALTH SYNCPOLICY CONDITIONS REPO PATH TARGET
NAME CLUSTER NAMESPACE PROJECT STATUS HEALTH SYNCPOLICY CONDITIONS REPO PATH TARGET
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Create an application in
core
mode:Copy to Clipboard Copied! Toggle word wrap Toggle overflow Label the
spring-petclinic
destination namespace to be managed by theopenshif-gitops
Argo CD instance:oc label ns spring-petclinic "argocd.argoproj.io/managed-by=openshift-gitops"
$ oc label ns spring-petclinic "argocd.argoproj.io/managed-by=openshift-gitops"
Copy to Clipboard Copied! Toggle word wrap Toggle overflow List the available applications to confirm that the application is created successfully:
argocd app list --core
$ argocd app list --core
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Even though the
cluster-configs
Argo CD application has theHealthy
status, it is not automatically synced due to itsnone
sync policy, causing it to remain in theOutOfSync
status.
1.9. Synchronizing your application with your Git repository Copy linkLink copied to clipboard!
You can synchronize your application with your Git repository by modifying the synchronization policy for Argo CD. The policy modification automatically applies the changes in your cluster configurations from your Git repository to the cluster.
Procedure
- In the Argo CD dashboard, notice that the cluster-configs Argo CD application has the statuses Missing and OutOfSync. Because the application was configured with a manual sync policy, Argo CD does not sync it automatically.
- Click SYNC on the cluster-configs tile, review the changes, and then click SYNCHRONIZE. Argo CD will detect any changes in the Git repository automatically. If the configurations are changed, Argo CD will change the status of the cluster-configs to OutOfSync. You can modify the synchronization policy for Argo CD to automatically apply changes from your Git repository to the cluster.
- Notice that the cluster-configs Argo CD application now has the statuses Healthy and Synced. Click the cluster-configs tile to check the details of the synchronized resources and their status on the cluster.
-
Navigate to the OpenShift Container Platform web console and click
to verify that a link to the Red Hat Developer Blog - Kubernetes is now present there.
Navigate to the Project page and search for the
spring-petclinic
namespace to verify that it has been added to the cluster.Your cluster configurations have been successfully synchronized to the cluster.
1.10. Synchronizing an application in the default mode by using the GitOps CLI Copy linkLink copied to clipboard!
You can synchronize applications in the default mode by using the GitOps argocd
CLI.
This sample workflow walks you through the process of configuring Argo CD to recursively sync the content of the cluster
directory to the cluster-configs
application. The directory defines the OpenShift Container Platform cluster configurations and the spring-petclinic
namespace on the cluster.
Prerequisites
- You have installed the Red Hat OpenShift GitOps Operator on your OpenShift Container Platform cluster.
- You have logged in to Argo CD instance.
-
You have installed the OpenShift CLI (
oc
). -
You have installed the Red Hat OpenShift GitOps
argocd
CLI.
Procedure
Get the
admin
account password for the Argo CD server:ADMIN_PASSWD=$(oc get secret openshift-gitops-cluster -n openshift-gitops -o jsonpath='{.data.admin\.password}' | base64 -d)
$ ADMIN_PASSWD=$(oc get secret openshift-gitops-cluster -n openshift-gitops -o jsonpath='{.data.admin\.password}' | base64 -d)
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Get the Argo CD server URL:
SERVER_URL=$(oc get routes openshift-gitops-server -n openshift-gitops -o jsonpath='{.status.ingress[0].host}')
$ SERVER_URL=$(oc get routes openshift-gitops-server -n openshift-gitops -o jsonpath='{.status.ingress[0].host}')
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Log in to the Argo CD server by using the
admin
account password and enclosing it in single quotes:ImportantEnclosing the password in single quotes ensures that special characters, such as
$
, are not misinterpreted by the shell. Always use single quotes to enclose the literal value of the password.argocd login --username admin --password ${ADMIN_PASSWD} ${SERVER_URL}
$ argocd login --username admin --password ${ADMIN_PASSWD} ${SERVER_URL}
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Example
argocd login --username admin --password '<password>' openshift-gitops.openshift-gitops.apps-crc.testing
$ argocd login --username admin --password '<password>' openshift-gitops.openshift-gitops.apps-crc.testing
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Because the application is configured with the
none
sync policy, you must manually trigger the sync operation:argocd app sync openshift-gitops/app-cluster-configs
$ argocd app sync openshift-gitops/app-cluster-configs
Copy to Clipboard Copied! Toggle word wrap Toggle overflow List the application to confirm that the application has the
Healthy
andSynced
statuses:argocd app list
$ argocd app list
Copy to Clipboard Copied! Toggle word wrap Toggle overflow
1.11. Synchronizing an application in core mode by using the GitOps CLI Copy linkLink copied to clipboard!
You can synchronize applications in core
mode by using the GitOps argocd
CLI.
This sample workflow walks you through the process of configuring Argo CD to recursively sync the content of the cluster
directory to the cluster-configs
application. The directory defines the OpenShift Container Platform cluster configurations and the spring-petclinic
namespace on the cluster.
Prerequisites
- You have installed the Red Hat OpenShift GitOps Operator on your OpenShift Container Platform cluster.
-
You have installed the OpenShift CLI (
oc
). -
You have installed the Red Hat OpenShift GitOps
argocd
CLI.
Procedure
Log in to the OpenShift Container Platform cluster by using the
oc
CLI tool:oc login -u <username> -p <password> <server_url>
$ oc login -u <username> -p <password> <server_url>
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Example
oc login -u kubeadmin -p '<password>' https://api.crc.testing:6443
$ oc login -u kubeadmin -p '<password>' https://api.crc.testing:6443
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Check whether the context is set correctly in the
kubeconfig
file:oc config current-context
$ oc config current-context
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Set the default namespace of the current context to
openshift-gitops
:oc config set-context --current --namespace openshift-gitops
$ oc config set-context --current --namespace openshift-gitops
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Set the following environment variable to override the Argo CD component names:
export ARGOCD_REPO_SERVER_NAME=openshift-gitops-repo-server
$ export ARGOCD_REPO_SERVER_NAME=openshift-gitops-repo-server
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Because the application is configured with the
none
sync policy, you must manually trigger the sync operation:argocd app sync --core openshift-gitops/app-cluster-configs
$ argocd app sync --core openshift-gitops/app-cluster-configs
Copy to Clipboard Copied! Toggle word wrap Toggle overflow List the application to confirm that the application has the
Healthy
andSynced
statuses:argocd app list --core
$ argocd app list --core
Copy to Clipboard Copied! Toggle word wrap Toggle overflow
1.12. In-built permissions for cluster configuration Copy linkLink copied to clipboard!
By default, the Argo CD instance has permissions to manage specific cluster-scoped resources such as cluster Operators, optional OLM Operators and user management.
Argo CD does not have cluster-admin permissions.
Permissions for the Argo CD instance:
Resources | Descriptions |
---|---|
Resource Groups | Configure the user or administrator |
| Optional Operators managed by OLM |
| Groups, Users and their permissions |
| Control plane Operators managed by CVO used to configure cluster-wide build configuration, registry configuration and scheduler policies |
| Storage |
| Console customization |
1.13. Adding permissions for cluster configuration Copy linkLink copied to clipboard!
You can grant permissions for an Argo CD instance to manage cluster configuration. Create a cluster role with additional permissions and then create a new cluster role binding to associate the cluster role with a service account.
Prerequisites
-
You have access to an OpenShift Container Platform cluster with
cluster-admin
privileges and are logged in to the web console. - You have installed the Red Hat OpenShift GitOps Operator on your OpenShift Container Platform cluster.
Procedure
In the web console, select User Management → Roles → Create Role. Use the following
ClusterRole
YAML template to add rules to specify the additional permissions.Copy to Clipboard Copied! Toggle word wrap Toggle overflow - Click Create to add the cluster role.
- To create the cluster role binding, select User Management → Role Bindings → Create Binding.
- Select All Projects from the Project drop-down.
- Click Create binding.
- Select Binding type as Cluster-wide role binding (ClusterRoleBinding).
- Enter a unique value for the RoleBinding name.
- Select the newly created cluster role or an existing cluster role from the drop down list.
Select the Subject as ServiceAccount and the provide the Subject namespace and name.
-
Subject namespace:
openshift-gitops
-
Subject name:
openshift-gitops-argocd-application-controller
-
Subject namespace:
Click Create. The YAML file for the
ClusterRoleBinding
object is as follows:Copy to Clipboard Copied! Toggle word wrap Toggle overflow
1.14. Installing OLM Operators using Red Hat OpenShift GitOps Copy linkLink copied to clipboard!
Red Hat OpenShift GitOps with cluster configurations manages specific cluster-scoped resources and takes care of installing cluster Operators or any namespace-scoped OLM Operators.
Consider a case where as a cluster administrator, you have to install an OLM Operator such as Tekton. You use the OpenShift Container Platform web console to manually install a Tekton Operator or the OpenShift CLI to manually install a Tekton subscription and Tekton Operator group on your cluster.
Red Hat OpenShift GitOps places your Kubernetes resources in your Git repository. As a cluster administrator, use Red Hat OpenShift GitOps to manage and automate the installation of other OLM Operators without any manual procedures. For example, after you place the Tekton subscription in your Git repository by using Red Hat OpenShift GitOps, the Red Hat OpenShift GitOps automatically takes this Tekton subscription from your Git repository and installs the Tekton Operator on your cluster.
1.14.1. Installing cluster-scoped Operators Copy linkLink copied to clipboard!
Operator Lifecycle Manager (OLM) uses a default global-operators
Operator group in the openshift-operators
namespace for cluster-scoped Operators. Hence you do not have to manage the OperatorGroup
resource in your Gitops repository. However, for namespace-scoped Operators, you must manage the OperatorGroup
resource in that namespace.
To install cluster-scoped Operators, create and place the Subscription
resource of the required Operator in your Git repository.
Example: Grafana Operator subscription
1.14.2. Installing namepace-scoped Operators Copy linkLink copied to clipboard!
To install namespace-scoped Operators, create and place the Subscription
and OperatorGroup
resources of the required Operator in your Git repository.
Example: Ansible Automation Platform Resource Operator
When deploying multiple Operators using Red Hat OpenShift GitOps, you must create only a single Operator group in the corresponding namespace. If more than one Operator group exists in a single namespace, any CSV created in that namespace transition to a failure
state with the TooManyOperatorGroups
reason. After the number of Operator groups in their corresponding namespaces reaches one, all the previous failure
state CSVs transition to pending
state. You must manually approve the pending install plan to complete the Operator installation.
Chapter 2. Sharding clusters across Argo CD Application Controller replicas Copy linkLink copied to clipboard!
You can shard clusters across multiple Argo CD Application Controller replicas if the controller is managing too many clusters and uses too much memory.
2.1. Enabling the round-robin sharding algorithm Copy linkLink copied to clipboard!
The round-robin
sharding algorithm is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.
For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.
By default, the Argo CD Application Controller uses the non-uniform legacy
hash-based sharding algorithm to assign clusters to shards. This can result in uneven cluster distribution. You can enable the round-robin
sharding algorithm to achieve more equal cluster distribution across all shards.
Using the round-robin
sharding algorithm in Red Hat OpenShift GitOps provides the following benefits:
- Ensure more balanced workload distribution
- Prevent shards from being overloaded or underutilized
- Optimize the efficiency of computing resources
- Reduce the risk of bottlenecks
- Improve overall performance and reliability of the Argo CD system
The introduction of alternative sharding algorithms allows for further customization based on specific use cases. You can select the algorithm that best aligns with your deployment needs, which results in greater flexibility and adaptability in diverse operational scenarios.
To leverage the benefits of alternative sharding algorithms in GitOps, it is crucial to enable sharding during deployment.
2.1.1. Enabling the round-robin sharding algorithm in the web console Copy linkLink copied to clipboard!
You can enable the round-robin
sharding algorithm by using the OpenShift Container Platform web console.
Prerequisites
- You have installed the Red Hat OpenShift GitOps Operator on your OpenShift Container Platform cluster.
- You have access to the OpenShift Container Platform web console.
-
You have access to the cluster with
cluster-admin
privileges.
Procedure
- In the Administrator perspective of the web console, go to Operators → Installed Operators.
- Click Red Hat OpenShift GitOps from the installed operators and go to the Argo CD tab.
-
Click the Argo CD instance where you want to enable the
round-robin
sharding algorithm, for example,openshift-gitops
. Click the YAML tab and edit the YAML file as shown in the following example:
Example Argo CD instance with round-robin sharding algorithm enabled
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Click Save.
A success notification alert,
openshift-gitops has been updated to version <version>
, appears.NoteIf you edit the default
openshift-gitops
instance, the Managed resource dialog box is displayed. Click Save again to confirm the changes.Verify that the sharding is enabled with
round-robin
as the sharding algorithm by performing the following steps:- Go to Workloads → StatefulSets.
- Select the namespace where you installed the Argo CD instance from the Project drop-down list.
- Click <instance_name>-application-controller, for example, openshift-gitops-application-controller, and go to the Pods tab.
- Observe the number of created application controller pods. It should correspond with the number of set replicas.
Click on the controller pod you want to examine and go to the Logs tab to view the pod logs.
Example controller pod logs snippet
time="2023-12-13T09:05:34Z" level=info msg="ArgoCD Application Controller is starting" built="2023-12-01T19:21:49Z" commit=a3vd5c3df52943a6fff6c0rg181fth3248976299 namespace=openshift-gitops version=v2.9.2+c5ea5c4 time="2023-12-13T09:05:34Z" level=info msg="Processing clusters from shard 1" time="2023-12-13T09:05:34Z" level=info msg="Using filter function: round-robin" time="2023-12-13T09:05:34Z" level=info msg="Using filter function: round-robin" time="2023-12-13T09:05:34Z" level=info msg="appResyncPeriod=3m0s, appHardResyncPeriod=0s"
time="2023-12-13T09:05:34Z" level=info msg="ArgoCD Application Controller is starting" built="2023-12-01T19:21:49Z" commit=a3vd5c3df52943a6fff6c0rg181fth3248976299 namespace=openshift-gitops version=v2.9.2+c5ea5c4 time="2023-12-13T09:05:34Z" level=info msg="Processing clusters from shard 1" time="2023-12-13T09:05:34Z" level=info msg="Using filter function: round-robin"
1 time="2023-12-13T09:05:34Z" level=info msg="Using filter function: round-robin" time="2023-12-13T09:05:34Z" level=info msg="appResyncPeriod=3m0s, appHardResyncPeriod=0s"
Copy to Clipboard Copied! Toggle word wrap Toggle overflow - 1
- Look for the
"Using filter function: round-robin"
message.
In the log Search field, search for
processed by shard
to verify that the cluster distribution across shards is even, as shown in the following example.ImportantEnsure that you set the log level to
debug
to observe these logs.Example controller pod logs snippet
Copy to Clipboard Copied! Toggle word wrap Toggle overflow NoteIf the number of clusters "C" is a multiple of the number of shard replicas "R", then each shard must have the same number of assigned clusters "N", which is equal to "C" divided by "R". The previous example shows 3 clusters and 3 replicas; therefore, each shard has 1 cluster assigned.
2.1.2. Enabling the round-robin sharding algorithm by using the CLI Copy linkLink copied to clipboard!
You can enable the round-robin
sharding algorithm by using the command-line interface.
Prerequisites
- You have installed the Red Hat OpenShift GitOps Operator on your OpenShift Container Platform cluster.
-
You have access to the cluster with
cluster-admin
privileges.
Procedure
Enable sharding and set the number of replicas to the wanted value by running the following command:
oc patch argocd <argocd_instance> -n <namespace> --patch='{"spec":{"controller":{"sharding":{"enabled":true,"replicas":<value>}}}}' --type=merge
$ oc patch argocd <argocd_instance> -n <namespace> --patch='{"spec":{"controller":{"sharding":{"enabled":true,"replicas":<value>}}}}' --type=merge
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Example output
argocd.argoproj.io/<argocd_instance> patched
argocd.argoproj.io/<argocd_instance> patched
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Configure the sharding algorithm to
round-robin
by running the following command:oc patch argocd <argocd_instance> -n <namespace> --patch='{"spec":{"controller":{"env":[{"name":"ARGOCD_CONTROLLER_SHARDING_ALGORITHM","value":"round-robin"}]}}}' --type=merge
$ oc patch argocd <argocd_instance> -n <namespace> --patch='{"spec":{"controller":{"env":[{"name":"ARGOCD_CONTROLLER_SHARDING_ALGORITHM","value":"round-robin"}]}}}' --type=merge
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Example output
argocd.argoproj.io/<argocd_instance> patched
argocd.argoproj.io/<argocd_instance> patched
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Verify that the number of Argo CD Application Controller pods corresponds with the number of set replicas by running the following command:
oc get pods -l app.kubernetes.io/name=<argocd_instance>-application-controller -n <namespace>
$ oc get pods -l app.kubernetes.io/name=<argocd_instance>-application-controller -n <namespace>
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Example output
NAME READY STATUS RESTARTS AGE <argocd_instance>-application-controller-0 1/1 Running 0 11s <argocd_instance>-application-controller-1 1/1 Running 0 32s <argocd_instance>-application-controller-2 1/1 Running 0 22s
NAME READY STATUS RESTARTS AGE <argocd_instance>-application-controller-0 1/1 Running 0 11s <argocd_instance>-application-controller-1 1/1 Running 0 32s <argocd_instance>-application-controller-2 1/1 Running 0 22s
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Verify that the sharding is enabled with
round-robin
as the sharding algorithm by running the following command:oc logs <argocd_application_controller_pod> -n <namespace>
$ oc logs <argocd_application_controller_pod> -n <namespace>
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Example output snippet
time="2023-12-13T09:05:34Z" level=info msg="ArgoCD Application Controller is starting" built="2023-12-01T19:21:49Z" commit=a3vd5c3df52943a6fff6c0rg181fth3248976299 namespace=<namespace> version=v2.9.2+c5ea5c4 time="2023-12-13T09:05:34Z" level=info msg="Processing clusters from shard 1" time="2023-12-13T09:05:34Z" level=info msg="Using filter function: round-robin" time="2023-12-13T09:05:34Z" level=info msg="Using filter function: round-robin" time="2023-12-13T09:05:34Z" level=info msg="appResyncPeriod=3m0s, appHardResyncPeriod=0s"
time="2023-12-13T09:05:34Z" level=info msg="ArgoCD Application Controller is starting" built="2023-12-01T19:21:49Z" commit=a3vd5c3df52943a6fff6c0rg181fth3248976299 namespace=<namespace> version=v2.9.2+c5ea5c4 time="2023-12-13T09:05:34Z" level=info msg="Processing clusters from shard 1" time="2023-12-13T09:05:34Z" level=info msg="Using filter function: round-robin"
1 time="2023-12-13T09:05:34Z" level=info msg="Using filter function: round-robin" time="2023-12-13T09:05:34Z" level=info msg="appResyncPeriod=3m0s, appHardResyncPeriod=0s"
Copy to Clipboard Copied! Toggle word wrap Toggle overflow - 1
- Look for the
"Using filter function: round-robin"
message.
Verify that the cluster distribution across shards is even by performing the following steps:
Set the log level to
debug
by running the following command:oc patch argocd <argocd_instance> -n <namespace> --patch='{"spec":{"controller":{"logLevel":"debug"}}}' --type=merge
$ oc patch argocd <argocd_instance> -n <namespace> --patch='{"spec":{"controller":{"logLevel":"debug"}}}' --type=merge
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Example output
argocd.argoproj.io/<argocd_instance> patched
argocd.argoproj.io/<argocd_instance> patched
Copy to Clipboard Copied! Toggle word wrap Toggle overflow View the logs and search for
processed by shard
to observe to which shard each cluster is attached by running the following command:oc logs <argocd_application_controller_pod> -n <namespace> | grep "processed by shard"
$ oc logs <argocd_application_controller_pod> -n <namespace> | grep "processed by shard"
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Example output snippet
time="2023-12-13T09:05:34Z" level=debug msg="Cluster with id= will be processed by shard 0" time="2023-12-13T09:05:34Z" level=debug msg="Cluster with id=068d8b26-6rhi-4w23-jrf6-wjjfyw833n23 will be processed by shard 1" time="2023-12-13T09:05:34Z" level=debug msg="Cluster with id=836d8b53-96k4-f68r-8wq0-sh72j22kl90w will be processed by shard 2"
time="2023-12-13T09:05:34Z" level=debug msg="Cluster with id= will be processed by shard 0"
1 time="2023-12-13T09:05:34Z" level=debug msg="Cluster with id=068d8b26-6rhi-4w23-jrf6-wjjfyw833n23 will be processed by shard 1"
2 time="2023-12-13T09:05:34Z" level=debug msg="Cluster with id=836d8b53-96k4-f68r-8wq0-sh72j22kl90w will be processed by shard 2"
3 Copy to Clipboard Copied! Toggle word wrap Toggle overflow NoteIf the number of clusters "C" is a multiple of the number of shard replicas "R", then each shard must have the same number of assigned clusters "N", which is equal to "C" divided by "R". The previous example shows 3 clusters and 3 replicas; therefore, each shard has 1 cluster assigned.
2.2. Enabling dynamic scaling of shards of the Argo CD Application Controller Copy linkLink copied to clipboard!
Dynamic scaling of shards is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.
For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.
By default, the Argo CD Application Controller assigns clusters to shards indefinitely. If you are using the round-robin
sharding algorithm, this static assignment can result in uneven distribution of shards, particularly when replicas are added or removed. You can enable dynamic scaling of shards to automatically adjust the number of shards based on the number of clusters managed by the Argo CD Application Controller at a given time. This ensures that shards are well-balanced and optimizes the use of compute resources.
After you enable dynamic scaling, you cannot manually modify the shard count. The system automatically adjusts the number of shards based on the number of clusters managed by the Argo CD Application Controller at a given time.
2.2.1. Enabling dynamic scaling of shards in the web console Copy linkLink copied to clipboard!
You can enable dynamic scaling of shards by using the OpenShift Container Platform web console.
Prerequisites
-
You have access to the cluster with
cluster-admin
privileges. - You have access to the OpenShift Container Platform web console.
- You have installed the Red Hat OpenShift GitOps Operator on your OpenShift Container Platform cluster.
Procedure
- In the Administator perspective of the OpenShift Container Platform web console, go to Operators → Installed Operators.
- From the the list of Installed Operators, select the Red Hat OpenShift GitOps Operator, and then click the ArgoCD tab.
-
Select the Argo CD instance name for which you want to enable dynamic scaling of shards, for example,
openshift-gitops
. Click the YAML tab, and then edit and configure the
spec.controller.sharding
properties as follows:Example Argo CD YAML file with dynamic scaling enabled
Copy to Clipboard Copied! Toggle word wrap Toggle overflow - 1
- Set
dynamicScalingEnabled
totrue
to enable dynamic scaling. - 2
- Set
minShards
to the minimum number of shards that you want to have. The value must be set to1
or greater. - 3
- Set
maxShards
to the maximum number of shards that you want to have. The value must be greater than the value ofminShards
. - 4
- Set
clustersPerShard
to the number of clusters that you want to have per shard. The value must be set to1
or greater.
Click Save.
A success notification alert,
openshift-gitops has been updated to version <version>
, appears.NoteIf you edit the default
openshift-gitops
instance, the Managed resource dialog box is displayed. Click Save again to confirm the changes.
Verification
Verify that sharding is enabled by checking the number of pods in the namespace:
- Go to Workloads → StatefulSets.
-
Select the namespace where the Argo CD instance is deployed from the Project drop-down list, for example,
openshift-gitops
. -
Click the name of the
StatefulSet
object that has the name of the Argo CD instance, for exampleopenshift-gitops-apllication-controller
. -
Click the Pods tab, and then verify that the number of pods is equal to or greater than the value of
minShards
that you have set in the Argo CDYAML
file.
2.2.2. Enabling dynamic scaling of shards by using the CLI Copy linkLink copied to clipboard!
You can enable dynamic scaling of shards by using the OpenShift CLI (oc
).
Prerequisites
- You have installed the Red Hat OpenShift GitOps Operator on your OpenShift Container Platform cluster.
-
You have access to the cluster with
cluster-admin
privileges.
Procedure
-
Log in to the cluster by using the
oc
tool as a user withcluster-admin
privileges. Enable dynamic scaling by running the following command:
oc patch argocd <argocd_instance> -n <namespace> --type=merge --patch='{"spec":{"controller":{"sharding":{"dynamicScalingEnabled":true,"minShards":<value>,"maxShards":<value>,"clustersPerShard":<value>}}}}'
$ oc patch argocd <argocd_instance> -n <namespace> --type=merge --patch='{"spec":{"controller":{"sharding":{"dynamicScalingEnabled":true,"minShards":<value>,"maxShards":<value>,"clustersPerShard":<value>}}}}'
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Example command
oc patch argocd openshift-gitops -n openshift-gitops --type=merge --patch='{"spec":{"controller":{"sharding":{"dynamicScalingEnabled":true,"minShards":1,"maxShards":3,"clustersPerShard":1}}}}'
$ oc patch argocd openshift-gitops -n openshift-gitops --type=merge --patch='{"spec":{"controller":{"sharding":{"dynamicScalingEnabled":true,"minShards":1,"maxShards":3,"clustersPerShard":1}}}}'
1 Copy to Clipboard Copied! Toggle word wrap Toggle overflow - 1
- The example command enables dynamic scaling for the
openshift-gitops
Argo CD instance in theopenshift-gitops
namespace, and sets the minimum number of shards to1
, the maximum number of shards to3
, and the number of clusters per shard to1
. The values ofminShard
andclustersPerShard
must be set to1
or greater. The value ofmaxShard
must be equal to or greater than the value ofminShard
.
Example output
argocd.argoproj.io/openshift-gitops patched
argocd.argoproj.io/openshift-gitops patched
Copy to Clipboard Copied! Toggle word wrap Toggle overflow
Verification
Check the
spec.controller.sharding
properties of the Argo CD instance:oc get argocd <argocd_instance> -n <namespace> -o jsonpath='{.spec.controller.sharding}'
$ oc get argocd <argocd_instance> -n <namespace> -o jsonpath='{.spec.controller.sharding}'
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Example command
oc get argocd openshift-gitops -n openshift-gitops -o jsonpath='{.spec.controller.sharding}'
$ oc get argocd openshift-gitops -n openshift-gitops -o jsonpath='{.spec.controller.sharding}'
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Example output when dynamic scaling of shards is enabled
{"dynamicScalingEnabled":true,"minShards":1,"maxShards":3,"clustersPerShard":1}
{"dynamicScalingEnabled":true,"minShards":1,"maxShards":3,"clustersPerShard":1}
Copy to Clipboard Copied! Toggle word wrap Toggle overflow -
Optional: Verify that dynamic scaling is enabled by checking the configured
spec.controller.sharding
properties in the configurationYAML
file of the Argo CD instance in the OpenShift Container Platform web console. Check the number of Argo CD Application Controller pods:
oc get pods -n <namespace> -l app.kubernetes.io/name=<argocd_instance>-application-controller
$ oc get pods -n <namespace> -l app.kubernetes.io/name=<argocd_instance>-application-controller
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Example command
oc get pods -n openshift-gitops -l app.kubernetes.io/name=openshift-gitops-application-controller
$ oc get pods -n openshift-gitops -l app.kubernetes.io/name=openshift-gitops-application-controller
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Example output
NAME READY STATUS RESTARTS AGE openshift-gitops-application-controller-0 1/1 Running 0 2m
NAME READY STATUS RESTARTS AGE openshift-gitops-application-controller-0 1/1 Running 0 2m
1 Copy to Clipboard Copied! Toggle word wrap Toggle overflow - 1
- The number of Argo CD Application Controller pods must be equal to or greater than the value of
minShard
.