16.3. Using an External Git Repository

Overview

Figure 16.2, “External Git Repository Architecture” shows an overview of the Fabric architecture when the fabric is configured to use an external Git repository.

Figure 16.2. External Git Repository Architecture

External git repository architecture

When you configure a fabric with an external Git repository (which must be done at fabric creation time), the external Git repository becomes the primary Git repository for all of the containers in the Fabric. All of the Fabric servers in the ensemble maintain their own copy of the Git repository (under their respective data/ directories), but this local copy is kept up-to-date by regularly polling the external Git repository for updates. If a change is detected in the external Git repository, every Fabric server will do a git pull to update it's local copy of the Git repository.

It is also possible for an administrator to clone a local copy of the external Git repository. Using standard git commands, the administrator can now edit the configuration files in the local copy and push the changes to the external Git repository. As soon as those changes are received by the external Git repository, the Fabric servers will detect that an update has occurred and pull the latest configuration.

Preparing an external Git repository

When setting up this type of Fabric architecture, the first step is to prepare an external Git repository. When setting up this repository, you should pay attention to the following points:

The Git repository must be initialized. For example, if you were creating a new Git repository on your local file system, you would initialize it using the command git init. If you are using a Git server to host your repository (for example, Gerrit, GitLab, or GitHub), the Git repository is usually initialized automatically, after you create it.
You must ensure that all of your Fabric servers are able to access the external Git repository. For example, if your Git server uses a HTTP based protocol to access the repository, you are generally required to have username/password credentials for the HTTP BASIC authentication protocol.

Authentication

In this architecture, authentication is handled by the external Git repository (and the Git server that hosts it). The most common cases are:

HTTP URL—in this case, the Git server is likely to use HTTP with TLS (HTTPS), to verify the server identity, and HTTP BASIC authentication, to verify the client identity. When creating the fabric (with the fabric:create command), you need to specify the following additional options in this case:
- --external-git-url ExternalGitHttpUrl
- --external-git-user ExternalGitUser
- --external-git-password ExternalGitPass
File URL—in this case, no authentication is required. You can specify the Git URL either in the form /path/to/repo (recommended) or file:///path/to/repo (slower). If the Fabric servers are deployed on separate hosts, you must make sure that they all have access to the specified directory (for example, through a Network File Server). When creating the fabric (with the fabric:create command), you need to specify the following additional options in this case:
- --external-git-url ExternalGitFileUrl

Creating a fabric with an external Git repository

Typically, to create a fabric with an external Git repository, you would enter a console command like the following:

fabric:create --new-user admin --new-user-password AdminPass --new-user-role Administrator \
  --zookeeper-password ZooPass --global-resolver manualip \
  --resolver manualip --manual-ip StaticIPAddress --wait-for-provisioning \
  --external-git-url ExternalGitHttpUrl \
  --external-git-user ExternalGitUser --external-git-password ExternalGitPass

Note the following points:

A new user is created with username, admin, password, AdminPass, and role, Administrator. You can use these JAAS credentials to log on to any of the containers in the fabric.
The Zookeeper password is set to ZooPass (the only time you are prompted to enter the Zookeeper password is when joining a container to the fabric).
The resolver policy for the root container is set to manualip (using the --resolver option) and the global resolver policy (which becomes the default resolver policy for containers created in this fabric) is also set to manualip. This enables you to specify the root container's IP address, StaticIPAddress, explicitly. It is essential that you assign a static IP address to the Fabric server host (for demonstrations and tests on a single machine, you can use the loopback address, 127.0.0.1).
The Git URL, ExternalGitHttpUrl, is specified through the --external-git-url option.
Assuming that you use a HTTP Git URL with BASIC authentication enabled, you will also need to specify credentials, using the --external-git-user and --external-git-password options.

What happens if the external Git repository fails?

Because the external Git repository is the primary Git repository, which is used to synchronize configuration data with the other Fabric servers, it is technically a single point of failure. The effect of a failure of the external Git repository is not as serious as you might think, however. It does not lead to a failure of the Fabric servers. In case of an external Git repository failure (or a loss of connectivity) the Fabric servers continue to operate with the configuration data they have already cached in their local copies of the Git repository. As soon as the external Git repository comes back on line, they will re-synchronize their configuration data.

Synchronizing Git Repository

Each container maintains local copy of Git repository (a clone in git terminology) that is synchronized with single remote Git repository (git remote) (either entirely external, or one managed by one of Fabric containers itself - one is chosen from list of available Git repositories running in master-slaves configuration). All containers synchronize their local copies with central in two ways:

by doing periodical git pull operation (in case of external git repository)
after triggering event sent via Zookeeper registry - typically after invocation of fabric:profile-edit command or after an update to a profile via hawtio application

In ideal state, all containers maintain local copies that are identical to remote (reference) version. But in some scenarios, the synchronization state may not be correct, for example, networking operation failure. Red Hat Fuse introduces two new diagnostic commands that help in such situations:

fabric:git-summary
fabric:git-synchronize

fabric:git-summary

fabric:git-summary command retrieves status of git repository and its branches or versions. The command asks all (or selected) containers to send an information about the state of each version tracked in their local Git repository.The syntax is as follows:

fabric:git-summary [options] [container]

where, container refers to the container names.

The Options for fabric:git-summary command are as follows:

Option	Description
-a, --all	Send command to all containers
-t, --timeout	Timeout used when waiting for response(s). (defaults to 5000)
--help	Display the help message for the command

fabric:git-summary examples

Following example shows output of fabric:git-summary --all

JBossFuse:karaf@root> fabric:git-summary --all
Git master is: root
Scheduled git-summary command to 3 containers. Awaiting response(s).

=== Summary for master Git repository (container: root) ===
    Git master URL: <current container>
[version]  [SHA1]                                    [timestamp]          [message]                                

1.0        f1fd4e9ae0e6e0374ce70646a536ededb817910c  2017-07-11 19:28:29  Update configurations for profile: fabric

master     65aefd723bac7b6ed8c8f41fa788e2fe419e1026  2017-07-11 19:28:29  Create profile: fabric-ensemble-0000-1   

=== Summary for local Git repository (container: child1) ===
    Git master URL: http://example.com:8181/git/fabric/

[version]  [SHA1]                                    [timestamp]          [message]                                

1.0        f1fd4e9ae0e6e0374ce70646a536ededb817910c  2017-07-11 19:28:29  Update configurations for profile: fabric

master     65aefd723bac7b6ed8c8f41fa788e2fe419e1026  2017-07-11 19:28:29  Create profile: fabric-ensemble-0000-1   

=== Summary for local Git repository (container: child2) ===
    Git master URL: http://example.com:8181/git/fabric/

[version]  [SHA1]                                    [timestamp]          [message]                                

1.0        f1fd4e9ae0e6e0374ce70646a536ededb817910c  2017-07-11 19:28:29  Update configurations for profile: fabric

master     65aefd723bac7b6ed8c8f41fa788e2fe419e1026  2017-07-11 19:28:29  Create profile: fabric-ensemble-0000-1   

=== Summary for local Git repository (container: root) ===
    Git master URL: http://example.com:8181/git/fabric/

[version]  [SHA1]                                    [timestamp]          [message]                                

1.0        f1fd4e9ae0e6e0374ce70646a536ededb817910c  2017-07-11 19:28:29  Update configurations for profile: fabric

master     65aefd723bac7b6ed8c8f41fa788e2fe419e1026  2017-07-11 19:28:29  Create profile: fabric-ensemble-0000-1

Firstly, this command prints information about current master Git repository (root in this case). It also prints the configured git master URL for each container. For git master summary, it prints Git master URL: <current container>. For each container it prints Git master URL: http://example.com:8181/git/fabric/. This allows you to verify that all containers are configured correctly with git master server.

Next it shows that the command is sent (technically - via Zookeeper registry) to 3 containers asynchronously. There is at most one summary for master Git repository (zero if master Git repository can't be found) showing table with all found versions and recent Git commits (SHA1, timestamp and message to make identification of change easier). Finally, there is a list of all containers' local Git repository status (including local repository of container which is at the same time Git master).

In ideal situation, all containers should show the same state of all versions. Following example shows output of fabric:git-summary command if there is one container with desynchronized local repository:

JBossFuse:karaf@root> fabric:git-summary child1

The list of container names: [child1]

Git master is: root

Scheduled git-summary command to 1 containers. Awaiting response(s).
  
Containers that require synchronization: [child1]

Please use "fabric:git-synchronize" command

=== Summary for master Git repository (container: root) ===

[version]  [SHA1]                                    [timestamp]          [message]                                

1.0        f1fd4e9ae0e6e0374ce70646a536ededb817910c  2017-07-11 19:28:29  Update configurations for profile: fabric

master     65aefd723bac7b6ed8c8f41fa788e2fe419e1026  2017-07-11 19:28:29  Create profile: fabric-ensemble-0000-1   

=== Summary for local Git repository (container: child1) ===

[version]  [SHA1]                                    [timestamp]          [message]                             

1.0        9343f17dc9e642e456da366c6a005ca202d12485  2017-07-11 19:44:54  Externally made commit                

master     65aefd723bac7b6ed8c8f41fa788e2fe419e1026  2017-07-11 19:28:29  Create profile: fabric-ensemble-0000-1

Summary for child1 container shows different commit for version 1.0. Here, we can use fabric:git-synchronize command which is explained in the next section.

fabric:git-synchronize

fabric:git-synchronize command schedules synchronization of local git repositories for selected or all containers. This is needed when output of fabric:git-summary command shows that something is not right with git repositories. Using this command we can order selected or all containers to fetch current state of git branches (fabric versions) from central git repository and synchronize local git repositories. The syntax is as follows:

fabric:git-synchronize [options] [container]

where, container refers to the container names.

The Options for fabric:git-synchronize command are as follows:

Option	Description
-r, --random-delay	Delay synchronization attempt by random number of seconds. By default each container will delay synchronization by 0-15 seconds to protect against too many pull operations (defaults to 15).
-a, --all	Send command to all containers
-t, --timeout	Timeout used when waiting for response(s) (defaults to 5000)
-p, --allow-push	Whether containers are allowed to push local Git repository state to central Git repository. This option is used when local git repository of container has newer changes than in central git repository. By default, these changes will be removed and local git repository will be synchronized to what is currently available in central repository (this is normal situation, because git repository should be altered only using commands like fabric:profile-edit). If some local repository was changed manually, this option may be used to synchronize from local to central git repository
--help	Display the help message for the command

Example of fabric:git-synchronize

Following example shows output of fabric:git-synchronize:

JBossFuse:karaf@root> fabric:git-synchronize child1

The list of container names: [child1]

Scheduled git-synchronize command to 1 containers

Since it's assumed that execution of git repository synchronization may take some time, this command doesn't produce more output.

`io.fabric8.datastore PID Configuration`

In order to configure git repository functionality in fabric environment, io.fabric8.datastore PID configuration from default profile may be used. It uses following options:

gitRemotePollInterval: This options is used to configure polling interval in case of external git repository (in milliseconds).
gitRemoteUrl, gitRemoteUser and gitRemotePassword: These options are used to configure external (not managed by fabric) git repository to store profile data.
gitTimeout: This option is used during remote git operations (fetch, pull, push). The timeout is in seconds.
gitAllowRemoteUpdate: This option is analogous to fabric:git-synchronize --allow-push. By default, in all versions of fabric, local changes were tried to be pushed to central git repository if it was possible (local changes were newer than central ones and git merge operation was successful). By default this option is true, but it can be changed as well. For normal operations this should be true.
gitRandomFetchDelay: This option is analogous to fabric:git-synchronize --random-delay. Value of 0 means containers will fetch remote changes immediately after fabric:profile-edit. When there are tens and hundreds of containers, setting this option may prevent DoS-like behavior observed on central git repository server (which by default is one of Fuse fabric containers).

External Git repository tutorial

The following tutorial explains how to create a fabric, which synchronizes its configuration with an external Git repository:

Create a new (empty) Git repository, which you can use as the external Git repository. Typically, you would create the Git repository in a hosting service, such as GitLab, Gerrit, or GitHub. Make a note of the new repository's HTTP URL, ExternalGitHttpUrl, and make sure that it is possible to access the external Git repository from the hosts where you will be deploying your Fabric servers.
(Optional) Prepare the container for a cold start. Delete the following directories:
```
InstallDir/data
InstallDir/instances
```
Important
Performing a cold start completely wipes the current state of the root container, including all of the deployed bundles, and features, and most of the stored data. Do not perform this operation on a production system.
Start up the container, by entering the following command:
```
./bin/amq
```
Create a new fabric. At the container prompt, enter the following console command:
```
fabric:create --new-user admin --new-user-password AdminPass --new-user-role Administrator \
  --zookeeper-password ZooPass --global-resolver manualip \
  --resolver manualip --manual-ip 127.0.0.1 --wait-for-provisioning \
  --external-git-url ExternalGitHttpUrl \
  --external-git-user ExternalGitUser --external-git-password ExternalGitPass
```
You need to substitute your own values for AdminPass and ZooPass. The ExternalGitHttpUrl is the HTTP URL of the external Git repository you created earlier and the ExternalGitUser value and the ExternalGitPass value are the username/password credentials required to access the external Git repository (using HTTP BASIC authentication).
This sample command uses the --manual-ip option to assign the loopback address, 127.0.0.1, to the root container. If your host has a static IP address and hostname assigned to it, however, it would be better to use the assigned hostname here instead.
You need to wait a minute or two for this command to complete.
After your fabric has been created, navigate to the contents of the external Git repository in your browser (assuming that your Git server supports this functionality). The external repository should now be populated with the default configuration of your fabric, with two branches available: master and 1.0. The 1.0 branch is the branch that is initially used by your fabric.
Create a local clone of the external Git repository, which you can then use to push or pull profile configurations. Open a new command prompt and, in a convenient location on the file system, enter the following command:
```
git clone -b 1.0 ExternalGitHttpUrl
```
This git command will prompt you to enter the username and password credentials for the external Git repository.
This command clones the Fabric Git repository and checks out the 1.0 branch. You should now be able to see the profile configuration files under the fabric/profiles subdirectory.
You can now use regular git commands to configure your Fabric profiles. Simply edit the files in your local Git repository, add the changes, commit, and then push the changes to the external Git repository (working in the 1.0 branch). Shortly after the changes are pushed to the external Git repository, the containers in your Fabric ensemble (the Fabric servers) will poll the repository, pull the changes, and redeploy any changed profiles..

Choose the git cluster member to be the git master

When you create a fabric cluster, a list of containers run the fabric-git-server bundle, which is part of the fabric-git-server feature. However, only the first node has services and url data in the cluster member registration as child3 in the example below:

JBossFuse:karaf@root> zk:get /fabric/registry/clusters/git/00000000139
{"id":"fabric-repo","container":"child3","uuid":"84be3c4e-0529-4402-98e7-319a5b1de6fa","services":["${zk:child3/http}/git/fabric/"],"url":"${zk:child3/http}/git/fabric/"}
JBossFuse:karaf@root> zk:get /fabric/registry/clusters/git/00000000140
{"id":"fabric-repo","container":"child2","uuid":"b38a2f49-ac06-49e8-bdaf-ee16e9e0867a","services":null,"url":null}
JBossFuse:karaf@root> zk:get /fabric/registry/clusters/git/00000000141
{"id":"fabric-repo","container":"child1","uuid":"15e604fe-5c49-4fb5-abb4-2496c7aeaa79","services":null,"url":null}
JBossFuse:karaf@root> zk:get /fabric/registry/clusters/git/00000000142
{"id":"fabric-repo","container":"root","uuid":"4bc24f5b-21bb-4d7b-b358-f9ea93f9ee26","services":null,"url":null}

Change the container to be the git master by using the fabric:git-master command:
Note
This command is only available from R13 and later.
```
fabric:git-master root
Changing Git master to new cluster member: root
```

Check the logs to confirm that the container you specified is the new git master.

2019-04-03 15:31:37,969 | ... | Container became git master. Registering git servlet.
2019-04-03 15:31:37,969 | ... | Cloning master root repo into /data/servers/jboss-fuse-6.3.0.redhat-388/data/git/servlet/fabric
2019-04-03 15:31:40,274 | ... | Cloning master root repository finished
2019-04-03 15:31:40,274 | ... | Registering /git servlet in http service
2019-04-03 15:31:40,482 | ... | Registering git JMX endpoint
2019-04-03 15:31:40,484 | ... | Updating state for ZK Group for path /fabric/registry/clusters/git/00000000142: {"id":"fabric-repo","container":"root","uuid":"4bc24f5b-21bb-4d7b-b358-f9ea93f9ee26","services":["${zk:root/http}/git/fabric/"],"url":"${zk:root/http}/git/fabric/"}
...
2019-04-03 15:31:40,491 | ... | Remote url change from: http://everfree.forest:8184/git/fabric/ to: http://everfree.forest:8181/git/fabric/