Chapter 30. Enabling Cluster Metrics

The Metrics Deployer will be discussed in a later step, but you must first set up a service account for it:

The Heapster component requires access to the master server to list all available nodes and access the /stats endpoint for each node. Before it can do this, the Heapster service account requires the cluster-reader permission:

$ oadm policy add-cluster-role-to-user \
    cluster-reader system:serviceaccount:openshift-infra:heapster

Running OpenShift Container Platform cluster metrics with persistent storage means that your metrics will be stored to a persistent volume and be able to survive a pod being restarted or recreated. This is ideal if you require your metrics data to be guarded from data loss. For production environments it is highly recommended to configure persistent storage for your metrics pods.

The size of the persisted volume can be specified with the CASSANDRA_PV_SIZEtemplate parameter. By default it is set to 10 GB, which may or may not be sufficient for the size of the cluster you are using. If you require more space, for instance 100 GB, you could specify it with something like this:

$ oc process -f metrics-deployer.yaml \
    -v HAWKULAR_METRICS_HOSTNAME=hawkular-metrics.example.com \
    -v CASSANDRA_PV_SIZE=100Gi \
    | oc create -f -

The size requirement of the Cassandra storage is dependent on the number of pods. It is the administrator’s responsibility to ensure that the size requirements are sufficient for their setup and to monitor usage to ensure that the disk does not become full.

If you would like to use dynamically provisioned persistent volumes you must set the DYNAMICALLY_PROVISION_STORAGEtemplate option to true for the Metrics Deployer.

After the metrics deployer runs, the output of oc get pods should resemble the following:

 # oc get pods -n openshift-infra
 NAME                                READY             STATUS      RESTARTS          AGE
 hawkular-cassandra-1-l5y4g          1/1               Running     0                 17h
 hawkular-metrics-1t9so              1/1               Running     0                 17h
 heapster-febru                      1/1               Running     0                 17h

OpenShift Container Platform metrics are stored using the Cassandra database, which is deployed with settings of MAX_HEAP_SIZE=512M and NEW_HEAP_SIZE=100M. These values should cover most OpenShift Container Platform metrics installations, but you can modify them in the Cassandra Dockerfile prior to deploying cluster metrics.

By default, metrics data is stored for 7 days. You can configure this with the METRIC_DURATION parameter in the metrics.yaml configuration file. After seven days, Cassandra begins to purge the oldest metrics data. Metrics data for deleted pods and projects is not automatically purged; it is only removed once the data is over seven days old.

For cluster metrics to work with persistent storage, ensure that the persistent volume has the ReadWriteOnce access mode. If this mode is not active, then the persistent volume claim cannot locate the persistent volume, and Cassandra fails to start.

To use persistent storage with the metric components, ensure that a persistent volume of sufficient size is available. The creation of persistent volume claims is handled by the Metrics Deployer.

OpenShift Container Platform metrics also supports dynamically-provisioned persistent volumes. To use this feature with OpenShift Container Platform metrics, it is necessary to add an additional flag to the metrics-deployer: DYNAMICALLY_PROVISION_STORAGE=true. You can use EBS, GCE, and Cinder storage back-ends to dynamically provision persistent volumes.

In tests performed with with 210 and 990 OpenShift Container Platform nodes, where 10500 pods and 11000 pods were monitored respectively, the Cassandra database grew at the speed shown in the table below:

In the above calculation, approximately 20% of the expected size was added as overhead to ensure that the storage requirements do not exceed calculated value.

If the METRICS_DURATION and METRICS_RESOLUTION values are kept at the default (7 days and 15 seconds respectively), it is safe to plan Cassandra storage size requrements for week, as in the values above.

Known Issues and Limitations

Testing found that the heapster metrics component is capable of handling up to 12000 pods. If the amount of pods exceed that number, Heapster begins to fall behind in metrics processing, resulting in the possibility of metrics graphs no longer appearing. Work is ongoing to increase the number of pods that Heapster can gather metrics on, as well as upstream development of alternate metrics-gathering solutions.

Running OpenShift Container Platform cluster metrics with non-persistent storage means that any stored metrics will be deleted when the pod is deleted. While it is much easier to run cluster metrics with non-persistent data, running with non-persistent data does come with the risk of permanent data loss. However, metrics can still survive a container being restarted.

In order to use non-persistent storage, you must set the USE_PERSISTENT_STORAGEtemplate option to false for the Metrics Deployer.

The Metrics Deployer will auto-generate self-signed certificates for use between its components and will generate a re-encrypting route to expose the Hawkular Metrics service. This route is what allows the web console to access the Hawkular Metrics service.

In order for the browser running the web console to trust the connection through this route, it must trust the route’s certificate. This can be accomplished by providing your own certificates signed by a trusted Certificate Authority. The Metric Deployer’s secret allows you to pass your own certificates which it will then use when creating the route.

If you do not wish to provide your own certificates, the router’s default certificate will be used instead.

$ oc secrets new metrics-deployer \
    hawkular-metrics.pem=/home/openshift/metrics/hm.pem \
    hawkular-metrics-ca.cert=/home/openshift/metrics/hm-ca.cert

$ oc secrets new metrics-deployer nothing=/dev/null

The OpenShift Container Platform installer uses a template to deploy the metrics components. The default template can be found at the following path:

/usr/share/ansible/openshift-ansible/roles/openshift_hosted_templates/files/v1.4/enterprise/metrics-deployer.yaml

In case you need to make any changes to this file, copy it to another directory with the file name metrics-deployer.yaml and refer to the new location when using it in the following sections.

The metrics deployer does some validation both before and after deployment. If the pre-flight validation fails, the environment for deployment is considered unsuitable and the deployment is aborted. However, you can add IGNORE_PREFLIGHT=true to the deployer parameters if you believe the validation has erred.

If post-deployment validation fails, the deployer finishes in an Error state, which indicates that you should check the deployer logs for issues that may require addressing. For example, the validation may detect that the external hawkular-metrics route is not actually in use, because the route was already created somewhere else. The validation output at the end of a deployment should explain as clearly as possible any issues it finds and what you can do to address them.

Once you have addressed deployment validation issues, you can re-run just the validation by running the deployer again with the MODE=validate parameter added, for example:

$ oc new-app --as=system:serviceaccount:openshift-infra:metrics-deployer \
    -f metrics-deployer.yaml \
    -p HAWKULAR_METRICS_HOSTNAME=hawkular-metrics.example.com \
    -p MODE=validate

There is also a diagnostic for metrics:

$ oadm diagnostics MetricsApiProxy

Hawkular Metrics is a multi-tenanted application. It is configured so that a project in OpenShift Container Platform corresponds to a tenant in Hawkular Metrics.

As such, when accessing metrics for a project named MyProject you must set the Hawkular-Tenant header to MyProject.

There is also a special tenant named _system which contains system level metrics. This requires either a cluster-reader or cluster-admin level privileges to access.

The Hawkular Metrics service will authenticate the user against OpenShift Container Platform to determine if the user has access to the project it is trying to access.

Hawkular Metrics accepts a bearer token from the client and verifies that token with the OpenShift Container Platform server using a SubjectAccessReview. If the user has proper read privileges for the project, they are allowed to read the metrics for that project. For the _system tenant, the user requesting to read from this tenant must have cluster-reader permission.

When accessing the Hawkular Metrics API, you must pass a bearer token in the Authorization header.

If persistent storage was used to deploy OpenShift Container Platform metrics, then you must create a persistent volume (PV) for the new Cassandra pod to use before you can scale out the number of OpenShift Container Platform metrics Cassandra pods. However, if Cassandra was deployed with dynamically provisioned PVs, then this step is not necessary.

The Cassandra nodes use persistent storage, therefore scaling up or down is not possible with replication controllers.

Scaling a Cassandra cluster requires you to use the hawkular-cassandra-node template. By default, the Cassandra cluster is a single-node cluster.

To scale up the number of OpenShift Container Platform metrics hawkular pods to two replicas, run:

# oc scale -n openshift-infra --replicas=2 rc hawkular-metrics

Alternatively, update your inventory file and re-run the deployment.

To scale down: