Chapter 3. Managing Fuse Online on OCP

Procedure

1. Log in to OpenShift with an account that gives you permission to edit the Fuse Online custom resource. For example:

   oc login -u admin -p admin-password

2. Switch to the project that is running the Fuse Online environment. For example:

   oc project my-fuse-online-project

3. Edit the syndesis custom resource:

   1. Invoke the following command, which typically opens the resource in an editor:

      oc edit syndesis

   2. Ensure that the following lines are in the resource. Edit as needed:

      components:
        server:
          features:
            auditing: true

4. Save the resource.

   When you enable the auditing feature in the syndesis custom resource, the running syndesis-server configuration reloads and Fuse Online starts logging relevant changes to Fuse Online components.

5. To view the Fuse Online audit log messages, type the following command:

Procedure

1. If there is an existing openshift-monitoring configuration, skip to Step 2.

   Otherwise, create an openshift-monitoring configuration, that sets the user workload monitoring option to true and then skip to Step 3:

   oc apply -f - <<EOF
   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |
      enableUserWorkload: true
   EOF

2. If there is an existing openshift-monitoring configuration:

   1. Check the existing openshift-monitoring configuration to determine whether the user workload monitoring option is set to true:

      oc get -n openshift-monitoring cm/cluster-monitoring-config -ojsonpath='{.data.config\.yaml}'

      If the result is enableUserWorkload: true, the user workload monitoring option is set to true. Skip to Step 3.

      If the result shows any other configurations, continue to the next step to enable the monitoring of user workloads by editing the ConfigMap.

   2. Open the ConfigMap file in an editor, for example:

      oc -n openshift-monitoring edit cm/cluster-monitoring-config

   3. Set enableUserWorkload to true. For example:

      apiVersion: v1
      kind: ConfigMap
      metadata:
        name: cluster-monitoring-config
        namespace: openshift-monitoring
      data:
        config.yaml: |
         enableUserWorkload: true

   4. Save the ConfigMap file.

3. Use the following command to watch the status of the pods in the openshift-user-workload-monitoring namespace:

   oc -n openshift-user-workload-monitoring get pods -w

   Wait until the status of the pods is Running, for example:

   prometheus-operator-5d989f48fd-2qbzd   2/2     Running
   prometheus-user-workload-0             5/5     Running   prometheus-user-workload-1             5/5     Running
   thanos-ruler-user-workload-0           3/3     Running
   thanos-ruler-user-workload-1           3/3     Running

4. Verify that the Fuse Online alert rules are enabled in Prometheus:

   1. Access the internal prometheus instance

      oc port-forward -n openshift-user-workload-monitoring pod/prometheus-user-workload-0 9090

   2. Open your browser to localhost:9090
   3. Select Status> Targets. You should see three syndesis endpoints.
   4. Press CTRL-C to terminate the port-forward process.
5. From the OperatorHub, install the Grafana Operator version 4 to a namespace of your choice, for example, to the grafana-middleware namespace. Use the update channel v4

6. Add a cluster role and a cluster role binding to allow the grafana-operator to list nodes and namespaces:

   1. Download the cluster role YAML file from the grafana-operator website:

      curl https://raw.githubusercontent.com/grafana-operator/grafana-operator/v4/deploy/cluster_roles/cluster_role_grafana_operator.yaml > tmp_role.yaml

   2. Add cluster permission for the grafana-operator to read other namespaces and nodes:

      cat <<EOF >> tmp_role.yaml
        - apiGroups:
            - ""
          resources:
            - namespaces
            - nodes
          verbs:
            - get
            - list
            - watch
      EOF

      oc apply -f tmp_role.yaml

      oc apply -f - <<EOF
      apiVersion: rbac.authorization.k8s.io/v1
      kind: ClusterRoleBinding
      metadata:
        name: grafana-operator
      roleRef:
        name: grafana-operator
        kind: ClusterRole
        apiGroup: ""
      subjects:
        - kind: ServiceAccount
          name: grafana-operator-controller-manager
          namespace: grafana-middleware
      EOF

7. Enable the grafana-operator to read Grafana dashboards from other namespaces by using the DASHBOARD_NAMESPACES_ALL environment variable to limit the namespaces:

   oc -n grafana-middleware patch subs/grafana-operator --type=merge -p '{"spec":{"config":{"env":[{"name":"DASHBOARD_NAMESPACES_ALL","value":"true"}]}}}'

8. Check that the grafana pods are recreated:

   oc -n grafana-middleware get pods -w

9. Optionally, view the grafana-operator logs:

   oc -n grafana-middleware logs -f `oc -n grafana-middleware get pods -oname|grep grafana-operator-controller-manager` -c manager

10. Add a Grafana custom resource to start a Grafana server pod, for example:

    oc apply -f - <<EOF
    apiVersion: integreatly.org/v1alpha1
    kind: Grafana
    metadata:
      name: grafana-middleware
      namespace: grafana-middleware
    spec:
      config:
        auth:
          disable_signout_menu: true
        auth.anonymous:
          enabled: true
        log:
          level: warn
          mode: console
        security:
          admin_password: secret
          admin_user: root
      dashboardLabelSelector:
      - matchExpressions:
        - key: app
          operator: In
          values:
          - grafana
          - syndesis
      ingress:
        enabled: true
    EOF

11. Allow the grafana-operator to read monitoring information:

    oc -n grafana-middleware adm policy add-cluster-role-to-user cluster-monitoring-view -z grafana-serviceaccount

12. Add a GrafanaDatasource to query thanos-querier:

    oc apply -f - <<EOF
    apiVersion: integreatly.org/v1alpha1
    kind: GrafanaDataSource
    metadata:
      name: prometheus-grafanadatasource
      namespace: grafana-middleware
    spec:
      datasources:
        - access: proxy
          editable: true
          isDefault: true
          jsonData:
            httpHeaderName1: 'Authorization'
            timeInterval: 5s
            tlsSkipVerify: true
          name: Prometheus
          secureJsonData:
            httpHeaderValue1: "Bearer $(oc get secret $(oc get secret | grep grafana-serviceaccount-token | awk '{print$1}') -o=jsonpath="{.data.token}" | base64 -d)"
          type: prometheus
          url: "https://$(oc get route thanos-querier -n openshift-monitoring -ojsonpath='{.spec.host}')"
      name: prometheus-grafanadatasource.yaml
    EOF

13. View the grafana server log:

    oc logs -f `oc get pods -l app=grafana -oname`

14. To access the grafana URL and view the Fuse Online dashboards:

    echo "https://"$(oc -n grafana-middleware get route/grafana-route -ojsonpath='{.spec.host}')

    In the left panel of the Grafana console, click the search button. A folder (OCP namespace name) containing the dashboards for each Syndesis instance is displayed.

    • For Fuse Online integrations, select Integration - Camel. This dashboard displays the standard metrics exposed by Apache Camel integration applications.

    • For Fuse Online infrastructure components, select one of the following infrastructure dashboards:

      Infrastructure - DB

      Displays metrics related to the Fuse Online Postgres instance.

      Infrastructure - JVM

      Displays metrics about the running JVM for the syndesis-meta or syndesis-server applications. Chose the application that you want to monitor from the Application drop down list at the top of the dashboard.

      Infrastructure - REST APIs

      Displays metrics relating to the Fuse Online infrastructure API endpoints, such as request throughput and latency. Choose the application that you want to monitor from the Application drop down list at the top of the dashboard.

Procedure

1. Log in to OpenShift with an account that has permission to install Fuse Online. For example:

   oc login -u developer -p developer

2. Switch to the OpenShift project that is or will be running the Fuse Online environment for which you want to configure backups. For example:

   oc project my-fuse-online-project

3. Create an OpenShift secret. In the command line:

   • Specify syndesis-backup-s3 as shown in the following command format.

   • Replace the AWS variables with your AWS access key, AWS secret key, AWS region in which the bucket resides, and the name of the bucket.

     Use the following command format to create the secret:

     oc create secret generic syndesis-backup-s3 \
     --from-literal=secret-key-id="my-aws-access-key" \
     --from-literal=secret-access-key="my-aws-secret-key" \
     --from-literal=region="aws-region" \
     --from-literal=bucket-name="aws-bucket-name"

     This secret must be present when the backup job is running.

4. If Fuse Online is not yet installed, you must edit the default-cr.yml file to enable backups. See Editing the syndesis custom resource before installing Fuse Online. After Fuse Online is installed, there will be backup jobs according to the schedule that you specified in the custom resource.

   If Fuse Online is running, you must edit the syndesis custom resource:

   1. Invoke the following command, which opens the syndesis custom resource in an editor:

      oc edit syndesis

   2. Add the following under spec::

      backup:
        schedule: my-backup-interval

      Replace my-backup-interval with the desired duration between backups. To determine how to specify the interval between backups, consult the following resources:

      • cron pre-defined schedules

      • cron intervals

        Do not specify the @ sign in front of the interval. For example, to configure daily backups, the custom resource would contain something like this:

        apiVersion: syndesis.io/v1beta1
        kind: Syndesis
        metadata:
          name: app
        spec:
          backup:
            schedule: daily

5. Save the file.

   This adds a backup job to syndesis-operator.

Procedure

1. Download the desired backup file from Amazon S3. Details for doing this are in the AWS documentation for How Do I Download an Object from an S3 Bucket?

2. Extract the content of the zip file. For example, the following command line unzips the 7.6-2020-03-15-23:30:00.zip file and copies the content into the /tmp/fuse-online-backup folder:

   unzip 7.6-2020-03-15-23:30:00.zip -d /tmp/fuse-online-backup

3. Decode the Fuse Online database, for example:

   base64 -d /tmp/fuse-online-backup/syndesis-db.dump > /tmp/fuse-online-backup/syndesis-db

4. Switch to the OpenShift project that is running the new Fuse Online environment. For example, if the new Fuse Online environment is in the my-fuse-online-project, then you would invoke the following command:

   oc project my-fuse-online-project

   The remainder of this procedure assumes that you have switched to the project that contains the new Fuse Online environment.

5. Obtain the name of the database pod.

   If the restored Fuse Online environment uses the provided, internal, PostgreSQL database, invoke the following command to obtain the name of the database pod:

   oc get pods -l deploymentconfig=syndesis-db -o jsonpath='{.items[*].metadata.name}'

   If the restored Fuse Online environment uses an external database, it is assumed that you know how to obtain the name of the pod for that database.

   In the remaining commands, where you see DATABASE_POD_NAME, insert the name of the database pod for the restored Fuse Online environment.

6. Scale down the components that are accessing the database in any way.

   1. Scale down syndesis-operator so that other components can be scaled down:

      oc scale deployment syndesis-operator --replicas 0

   2. Scale down the syndesis-server and syndesis-meta components:

      oc scale dc syndesis-server --replicas 0
oc scale dc syndesis-meta --replicas 0

7. Send the database backup file to the Fuse Online database pod:

   oc cp /tmp/fuse-online-backup/syndesis-db DATABASE_POD_NAME:/tmp/syndesis-db

8. Open a remote shell session in the Fuse Online database pod:

   oc rsh DATABASE_POD_NAME

9. Invoke the following commands to restore the Fuse Online database.

   If a psql command prompts for the database password, and the restored Fuse Online environment uses the provided, internal PostgreSQL database, you can find the password in the POSTGRESQL_PASSWORD environment variable in the syndesis-db deployment configuration. If the restored Fuse Online environment uses an external database, then it is assumed that you know the password.

   cd /tmp
   psql -c 'DROP database if exists syndesis_restore'
   psql -c 'CREATE database syndesis_restore'
   pg_restore -v -d syndesis_restore /tmp/syndesis-db
   psql -c "SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE datname = 'syndesis'"
   psql -c 'DROP database if exists syndesis'
   psql -c 'ALTER database syndesis_restore rename to syndesis'

   Fuse Online should now be restored. You can end the RSH session:

   exit

10. Scale up the Fuse Online components:

    oc scale deployment syndesis-operator --replicas 1

    Scaling syndesis-operator to 1 should bring up the other pods that were scaled down. However, if that does not happen, you can scale them up manually:

    oc scale dc syndesis-server --replicas 1
oc scale dc syndesis-meta --replicas 1

11. The server tries to start each restored integration but you need to update connections first. Consequently, ensure that the restored integrations are not running:

    1. Obtain the Fuse Online console route:

       echo "https://$(oc get route/syndesis -o jsonpath='{.spec.host}' )"

    2. Log in into the Fuse Online console with an OpenShift user account that has permission to install Fuse Online.
    3. Display the list of integrations and ensure that all integrations are stopped. If an integration is running, stop it.

12. For each connection that has a password, you need to update the connection to have the correct password for this Fuse Online environment. The following steps show how to do this for the provided PostgresDB connection.

    1. In the OpenShift console for the project in which this restored Fuse Online environment is running, retrieve the password for the PostgresDB connection. In the syndesis-db deployment, the password is available in the environment variables.
    2. In the Fuse Online console, display the connections.
    3. Edit the PostgresDB connection.
    4. In the connection details for the PostgresDB connection, paste the retrieved password in the Password field.

13. For each integration, confirm that there are no Configuration Required indicators. If there are, edit the integration to resolve the issues. When all steps in the integration are correct, publish the integration.

    If Fuse Online keeps rolling an integration back to a Stopped state right after the Build step, delete the deployment, ensure that no configuration is required, and try publishing the integration again.

    You can safely ignore the following message if you see it in the log:

    Error performing GET request to https://syndesis-my-fuse-online-project.my-cluster-url/api/v1/metrics/integrations

Procedure

1. Log in to OpenShift with an account that has permission to install Fuse Online. For example:

   oc login -u developer -p developer

2. Switch to the OpenShift project that is running the Fuse Online environment that you want to uninstall. For example:

   oc project my-fuse-online-project

3. Delete Fuse Online infrastructure:

   oc delete syndesis app

4. Delete syndesis-operator DeploymentConfig and ImageStream resources:

   oc delete deployment/syndesis-operator
oc delete is/syndesis-operator