Deploying AMQ Interconnect on OpenShift

Procedure

1. If you do not have an existing certificate authority (CA) certificate for inter-router connections, create one.

   These commands create a self-signed CA certificate for inter-router connections:

   # Create a new directory for the inter-router certificates.
   $ mkdir internal-certs
   
   # Create a private key for the CA.
   $ openssl genrsa -out internal-certs/ca-key.pem 2048
   
   # Create a certificate signing request for the CA.
   $ openssl req -new -batch -key internal-certs/ca-key.pem -out internal-certs/ca-csr.pem
   
   # Self sign the CA certificate.
   $ openssl x509 -req -in internal-certs/ca-csr.pem -signkey internal-certs/ca-key.pem -out internal-certs/ca.crt

2. Create a certificate for the router signed by the CA.

   These commands create a private key and a certificate, and sign the certificate using the CA created in the previous step:

   # Create a private key.
   $ openssl genrsa -out internal-certs/tls.key 2048
   
   # Create a certificate signing request for the router.
   $ openssl req -new -batch -subj "/CN=amq-interconnect.<project_name>.svc.cluster.local" -key internal-certs/tls.key -out internal-certs/server-csr.pem
   
   # Sign the certificate using the CA.
   $ openssl x509 -req -in internal-certs/server-csr.pem -CA internal-certs/ca.crt -CAkey internal-certs/ca-key.pem -out internal-certs/tls.crt -CAcreateserial

   where <project_name> is the name of the current OpenShift project.

3. Create a secret containing the private key, router certificate, and CA certificate.

   This command creates the secret using the key and certificates that were created in the previous steps:

   $ oc create secret generic inter-router-certs-secret --from-file=tls.crt=internal-certs/tls.crt  --from-file=tls.key=internal-certs/tls.key  --from-file=ca.crt=internal-certs/ca.crt

4. If you want to use SSL/TLS to authenticate client connections (as opposed to authenticating clients using SASL), create a CA certificate for client connections.

   These commands create a self-signed CA certificate for client connections:

   # Create a new directory for the client certificates.
   $ mkdir client-certs
   
   # Create a private key for the CA.
   $ openssl genrsa -out client-certs/ca-key.pem 2048
   
   # Create a certificate signing request for the CA.
   $ openssl req -new -batch -key client-certs/ca-key.pem -out client-certs/ca-csr.pem
   
   # Self sign the certificate.
   $ openssl x509 -req -in client-certs/ca-csr.pem -signkey client-certs/ca-key.pem -out client-certs/ca.crt

5. Create a certificate for client connections signed by the CA.

   These commands create a private key and a certificate, and then sign the certificate using the CA created in the previous step:

   # Create a private key.
   $ openssl genrsa -out client-certs/tls.key 2048
   
   # Create a certificate signing request for the client connections
   $ openssl req -new -batch -subj "/CN=<client_name>" -key client-certs/tls.key -out client-certs/client-csr.pem
   
   # Sign the certificate using the CA.
   $ openssl x509 -req -in client-certs/client-csr.pem -CA client-certs/ca.crt -CAkey client-certs/ca-key.pem -out client-certs/tls.crt -CAcreateserial

   where <client_name> is unique for each router client.

6. Create a secret containing the CA certificate used to sign client certificates using the certificate that was created in the previous steps:

   $ oc create secret generic client-ca-secret --from-file=ca.crt=client-certs/ca.crt --from-file=tls.crt=client-certs/ca.crt --from-file=tls.key=client-certs/ca-key.pem

Procedure

1. In the OpenShift Container Platform web console, navigate to Operators → OperatorHub.
2. Choose Red Hat Integration - AMQ Certificate Manager Operator from the list of available Operators, and then click Install.

3. On the Operator Installation page, select All namespaces on the cluster (default), and then click Install.

   The Installed Operators page appears displaying the status of the Operator installation.

4. Verify that the Red Hat Integration - AMQ Certificate Manager Operator is displayed and wait until the Status changes to Succeeded.

5. If the installation is not successful, troubleshoot the error:

   1. Click Red Hat Integration - AMQ Certificate Manager Operator on the Installed Operators page.
   2. Select the Subscription tab and view any failures or errors.

Procedure

1. In the OpenShift Container Platform web console, navigate to Operators → OperatorHub.
2. Choose Red Hat Integration - AMQ Interconnect Operator from the list of available Operators, and then click Install.

3. On the Operator Installation page, select the namespace into which you want to install the Operator, and then click Install.

   The Installed Operators page appears displaying the status of the Operator installation.

4. Verify that the AMQ Interconnect Operator is displayed and wait until the Status changes to Succeeded.

5. If the installation is not successful, troubleshoot the error:

   1. Click Red Hat Integration - AMQ Interconnect Operator on the Installed Operators page.
   2. Select the Subscription tab and view any failures or errors.

1. Log into the cluster as cluster-admin.

2. Disable the sources for the default catalogs using either the CLI or the OpenShift console:

   1. For the CLI, set disableAllDefaultSources: true for OperatorHub:

      $ oc patch OperatorHub cluster --type json \
      -p '[{"op": "add", "path": "/spec/disableAllDefaultSources", "value": true}]'

   2. For the OpenShift console, navigate to Administration → Cluster Settings → Configuration → OperatorHub.

      On the OperatorHub page, click the Sources tab, and disable the sources.

1. Create an Interconnect Custom Resource YAML file that describes the interior router deployment.

   Sample router-mesh.yaml file

   apiVersion: interconnectedcloud.github.io/v1alpha1
   kind: Interconnect
   metadata:
     name: router-mesh
   spec:
     deploymentPlan:
       role: interior  1
       size: 3  2
       placement:  Any  3

   1

   The operating mode of the routers in the deployment. The Operator will automatically connect interior routers in a mesh topology.

   2

   The number of routers to create.

   3

   Each router runs in a separate Pod. The placement defines where in the cluster the Operator should schedule and place the Pods. You can choose the following placement options:

   Any

   The Pods can run on any node in the OpenShift Container Platform cluster.

   Every

   The Operator places a router Pod on each node in the cluster. If you choose this option, the Size property is not needed - the number of routers corresponds to the number of nodes in the cluster.

   AntiAffinity

   The Operator ensures that multiple router Pods do not run on the same node in the cluster. If the size is greater than the number of nodes in the cluster, the extra Pods that cannot be scheduled will remain in a Pending state.

2. Create the router deployment described in the YAML file.

   $ oc apply -f router-mesh.yaml

   The Operator creates a deployment of interior routers in a mesh topology that uses default address semantics. It also creates a Service through which the routers can be accessed, and a Route through which you can access the web console.

3. Verify that the router mesh was created and the Pods are running.

   Each router runs in a separate Pod. They connect to each other automatically using the Service that the Operator created.

   $ oc get pods
   NAME                                     READY   STATUS    RESTARTS   AGE
   interconnect-operator-587f94784b-4bzdx   1/1     Running   0          52m
   router-mesh-6b48f89bd-588r5              1/1     Running   0          40m
   router-mesh-6b48f89bd-bdjc4              1/1     Running   0          40m
   router-mesh-6b48f89bd-h6d5r              1/1     Running   0          40m

4. Review the router deployment.

   $ oc get interconnect/router-mesh -o yaml
   apiVersion: interconnectedcloud.github.io/v1alpha1
   kind: Interconnect
   ...
   spec:
     addresses:  1
     - distribution: closest
       prefix: closest
     - distribution: multicast
       prefix: multicast
     - distribution: closest
       prefix: unicast
     - distribution: closest
       prefix: exclusive
     - distribution: multicast
       prefix: broadcast
     deploymentPlan:  2
       livenessPort: 8888
       placement: Any
       resources: {}
       role: interior
       size: 3
     edgeListeners:  3
     - port: 45672
     interRouterListeners:  4
     - authenticatePeer: true
       expose: true
       port: 55671
       saslMechanisms: EXTERNAL
       sslProfile: inter-router
     listeners:  5
     - port: 5672
     - authenticatePeer: true
       expose: true
       http: true
       port: 8080
     - port: 5671
       sslProfile: default
     sslProfiles:  6
     - credentials: router-mesh-default-tls
       name: default
     - caCert: router-mesh-inter-router-tls
       credentials: router-mesh-inter-router-tls
       mutualAuth: true
       name: inter-router
     users: router-mesh-users  7

   1

   The default address configuration. All messages sent to an address that does not match any of these prefixes are distributed in a balanced anycast pattern.

   2

   A router mesh of three interior routers was deployed.

   3

   Each interior router listens on port 45672 for connections from edge routers.

   4

   The interior routers connect to each other on port 55671. These inter-router connections are secured with SSL/TLS mutual authentication. The inter-router SSL Profile contains the details of the certificates that the Operator generated.

   5

   Each interior router listens for connections from external clients on the following ports:

   • 5672 - Unsecure connections from messaging applications.
   • 5671 - Secure connections from messaging applications.
   • 8080 - AMQ Interconnect web console access. Default user name/password security is applied.

   6

   Using the Red Hat Integration - AMQ Certificate Manager Operator, the Red Hat Integration - AMQ Interconnect Operator automatically creates two SSL profiles:

   • inter-router - The Operator secures the inter-router network with mutual TLS authentication by creating a Certificate Authority (CA) and generating certificates signed by the CA for each interior router.
   • default - The Operator creates TLS certificates for messaging applications to connect to the interior routers on port 5671.

   7

   The AMQ Interconnect web console is secured with user name/password authentication. The Operator automatically generates the credentials and stores them in the router-mesh-users Secret.

1. Create an Interconnect Custom Resource YAML file that describes the edge router deployment.

   Sample edge-routers.yaml file

   apiVersion: interconnectedcloud.github.io/v1alpha1
   kind: Interconnect
   metadata:
     name: edge-routers
   spec:
     deploymentPlan:
       role: edge
       placement: Every  1
     edgeConnectors:  2
       - host: router-mesh  3
         port: 45672  4

   1

   An edge router Pod will be deployed on each node in the OpenShift Container Platform cluster. This placement helps to balance messaging application traffic across the cluster. The Operator will create a DaemonSet to ensure that the number of Pods scheduled always corresponds to the number of nodes in the cluster.

   2

   Edge connectors define the connections from the edge routers to the interior routers.

   3

   The name of the Service that was created for the interior routers.

   4

   The port on which the interior routers listen for edge connections. The default is 45672.

2. Create the edge routers described in the YAML file:

   $ oc apply -f edge-routers.yaml

   The Operator deploys an edge router on each node of the OpenShift Container Platform cluster, and connects them to the interior routers.

3. Verify that the edge routers were created and the Pods are running.

   Each router runs in a separate Pod. Each edge router connects to any of the previously created interior routers.

   $ oc get pods
   NAME                                     READY   STATUS    RESTARTS   AGE
   edge-routers-2jz5j                       1/1     Running   0          33s
   edge-routers-fhlxv                       1/1     Running   0          33s
   edge-routers-gg2qb                       1/1     Running   0          33s
   edge-routers-hj72t                       1/1     Running   0          33s
   interconnect-operator-587f94784b-4bzdx   1/1     Running   0          54m
   router-mesh-6b48f89bd-588r5              1/1     Running   0          42m
   router-mesh-6b48f89bd-bdjc4              1/1     Running   0          42m
   router-mesh-6b48f89bd-h6d5r              1/1     Running   0          42m

Procedure

1. Start editing the Interconnect Custom Resource YAML file that describes the router deployment for which you want to expose ports.

   $ oc edit -f router-mesh.yaml

2. In the spec.listeners section, expose each port that you want clients outside of the cluster to be able to access.

   In this example, port 5671 is exposed. This enables clients outside of the cluster to authenticate with and connect to the router network.

   Sample router-mesh.yaml file

     apiVersion: interconnectedcloud.github.io/v1alpha1
     kind: Interconnect
     metadata:
       name: router-mesh
     spec:
       ...
       listeners:
         - port: 5672
         - authenticatePeer: true
           expose: true
           http: true
           port: 8080
         - port: 5671
           sslProfile: default
           expose: true
       ...

   The Red Hat Integration - AMQ Interconnect Operator creates a Route, which clients from outside the cluster can use to connect to the router network.

1. Start editing the Interconnect Custom Resource YAML file that describes the router deployment.

   $ oc edit -f router-mesh.yaml

2. In the spec section, define the semantics to be applied to addresses.

   Sample router-mesh.yaml file

   apiVersion: interconnectedcloud.github.io/v1alpha1
   kind: Interconnect
   metadata:
     name: router-mesh
   spec:
     ...
     addresses:
     - pattern: */orders  1
       distribution: multicast

   1

   Messages sent to any address that ends with “orders” will be distributed in a multicast pattern.

   The Operator applies the changes to the router network and restarts each Pod.

3. If you have additional router deployment Custom Resources that define routers in the router network, repeat this procedure for each CR.

   Each router in the router network must have the same address configuration.

Procedure

1. Start editing the Interconnect Custom Resource YAML file that describes the router deployment.

   $ oc edit -f router-mesh.yaml

2. In the spec section, add a connector to connect to the broker, a waypoint address to point to the broker queue, and autolinks to create the links to the queue.

   Sample router-mesh.yaml file

   apiVersion: interconnectedcloud.github.io/v1alpha1
   kind: Interconnect
   metadata:
     name: router-mesh
   spec:
     ...
     addresses:
     - prefix: my-queue  1
       waypoint: true
     autoLinks:  2
     - address: my-queue
       direction: in
       connection: my-broker
     - address: my-queue
       direction: out
       connection: my-broker
     connectors:  3
     - name: my-broker
       host: broker
       port: 5672
       routeContainer: true

   1

   The address (or set of addresses) for which messages should be stored on a broker queue.

   2

   The autolink configuration. It defines the incoming and outgoing links and connection to be used to send and receive the messages on the broker.

   3

   The connection to be used to connect the routers to the message broker.

   The Operator applies the changes to the router network and restarts each Pod.

3. Verify that the router has established the autolinks to the message broker.

   $ oc exec router-mesh-6d6dccb57f-x5cqf -it -- qdstat --autolinks
   AutoLinks
     addr      dir  phs  extAddr  link  status  lastErr
     ====================================================
     my-queue  in   1             26    active
     my-queue  out  0             27    active

4. If you have additional router deployment Custom Resources that define routers in the router network, repeat this procedure for each CR.

   Each router in the router network must have the same address configuration.

Procedure

1. Clone the qdr-monitoring GitHub repository.

   This repository contains example configuration files needed to set up Prometheus and Grafana to monitor AMQ Interconnect.

   $ git clone https://github.com/interconnectedcloud/qdr-monitoring

2. Set the NAMESPACE environment variable to the name of the project where you deployed AMQ Interconnect.

   For example, if you deployed AMQ Interconnect in the example project, set the NAMESPACE environment variable as follows:

   $ export NAMESPACE=example

3. Run the deploy-monitoring.sh script.

   This script creates and configures the OpenShift resources needed to deploy Prometheus, Alertmanager, and Grafana in your OpenShift project. It also configures two dashboards that provide metrics for the router network.

   $ ./deploy-monitoring.sh

   An alternative method of running this script is to to specify the target project as a parameter. For example:

   $ ./deploy-monitoring.sh example

Procedure

1. In the OpenShift web console, switch to Networking → Routes, and click the URL for the grafana Route.

   The Grafana Log In page appears.

2. Enter your user name and password, and then click Log In.

   The default Grafana user name and password are both admin. After logging in for the first time, you can change the password.

3. On the top header, click the dashboard drop-down menu, and then select the Qpid Dispatch Router or Qpid Dispatch Router - Delayed Deliveries dashboard.

   Figure 9.1. Delayed Deliveries dashboard