Red Hat SummitChapter 3. Capacity Metering using the Telemetry Service
The Openstack Telemetry service provides usage metrics that can be leveraged for billing, charge-back, and show-back purposes. Such metrics data can also be used by third-party applications to plan for capacity on the cluster and can also be leveraged for auto-scaling virtual instances using Openstack Heat. For more information, see Auto Scaling for Compute.
The combination of ceilometer and gnocchi can be used for monitoring and alarms. This is supported on small-size clusters and with known limitations. For real-time monitoring, Red Hat Openstack Platform ships with agents that provide metrics data, and can be consumed by separate monitoring infrastructure and applications. For more information, see Monitoring Tools Configuration.
3.1. View Existing Alarms
To list the existing Telemetry alarms, use the aodh command. For example:
# aodh alarm list +--------------------------------------+--------------------------------------------+----------------------------+-------------------+----------+---------+ | alarm_id | type | name | state | severity | enabled | +--------------------------------------+--------------------------------------------+----------------------------+-------------------+----------+---------+ | 922f899c-27c8-4c7d-a2cf-107be51ca90a | gnocchi_aggregation_by_resources_threshold | iops-monitor-read-requests | insufficient data | low | True | +--------------------------------------+--------------------------------------------+----------------------------+-------------------+----------+---------+
To list the meters assigned to a resource, specify the UUID of the resource (an instance, image, or volume, among others). For example:
# gnocchi resource show 5e3fcbe2-7aab-475d-b42c-a440aa42e5ad
3.2. Create an Alarm
You can use aodh to create an alarm that activates when a threshold value is reached. In this example, the alarm activates and adds a log entry when the average CPU utilization for an individual instance exceeds 80%. A query is used to isolate the specific instance’s id (94619081-abf5-4f1f-81c7-9cedaa872403) for monitoring purposes:
# aodh alarm create --type gnocchi_aggregation_by_resources_threshold --name cpu_usage_high --metric cpu_util --threshold 80 --aggregation-method sum --resource-type instance --query '{"=": {"id": "94619081-abf5-4f1f-81c7-9cedaa872403"}}' --alarm-action 'log://'
+---------------------------+-------------------------------------------------------+
| Field | Value |
+---------------------------+-------------------------------------------------------+
| aggregation_method | sum |
| alarm_actions | [u'log://'] |
| alarm_id | b794adc7-ed4f-4edb-ace4-88cbe4674a94 |
| comparison_operator | eq |
| description | gnocchi_aggregation_by_resources_threshold alarm rule |
| enabled | True |
| evaluation_periods | 1 |
| granularity | 60 |
| insufficient_data_actions | [] |
| metric | cpu_util |
| name | cpu_usage_high |
| ok_actions | [] |
| project_id | 13c52c41e0e543d9841a3e761f981c20 |
| query | {"=": {"id": "94619081-abf5-4f1f-81c7-9cedaa872403"}} |
| repeat_actions | False |
| resource_type | instance |
| severity | low |
| state | insufficient data |
| state_timestamp | 2016-12-09T05:18:53.326000 |
| threshold | 80.0 |
| time_constraints | [] |
| timestamp | 2016-12-09T05:18:53.326000 |
| type | gnocchi_aggregation_by_resources_threshold |
| user_id | 32d3f2c9a234423cb52fb69d3741dbbc |
+---------------------------+-------------------------------------------------------+
To edit an existing threshold alarm, use the aodh alarm update command. For example, to increase the alarm threshold to 75%:
# aodh alarm update --name cpu_usage_high --threshold 75
3.3. Disable or Delete an Alarm
To disable an alarm:
# aodh alarm update --name cpu_usage_high --enabled=false
To delete an alarm:
# aodh alarm delete --name cpu_usage_high
3.4. View Measures
To list all the measures for a particular resource:
# gnocchi measures show --resource-id UUID METER_NAME
To list only measures for a particular resource, within a range of timestamps:
# gnocchi measures show --aggregation mean --start START_TIME --end STOP_TIME --resource-id UUID METER_NAME
Where START_TIME and END_TIME are in the form iso-dateThh:mm:ss.
3.5. Create New Measures
You can use measures to send data to the Telemetry service, and they do not need to correspond to a previously-defined meter. For example:
# gnocchi measures add -m 2015-01-12T17:56:23@42 --resource-id UUID METER_NAME
3.6. View Cloud Usage Measures
This example shows the average memory usage of all instances for each project.
gnocchi measures aggregation --resource-type instance --groupby project_id -m memory
3.7. View L3 Cache Usage
If your Intel hardware and libvirt version supports Cache Monitoring Technology (CMT), you can use the cpu_l3_cache meter to monitor the amount of L3 cache used by an instance.
3.8. Monitor the disk activity of instances
The following example demonstrates how to use an aodh alarm to monitor the cumulative disk activity for all the instances contained within a particular project.
1. Review the existing projects, and select the appropriate UUID of the project you need to monitor. This example uses the admin tenant:
$ openstack project list +----------------------------------+----------+ | ID | Name | +----------------------------------+----------+ | 745d33000ac74d30a77539f8920555e7 | admin | | 983739bb834a42ddb48124a38def8538 | services | | be9e767afd4c4b7ead1417c6dfedde2b | demo | +----------------------------------+----------+
2. Use the project’s UUID to create an alarm that analyses the sum() of all read requests generated by the instances in the admin tenant (the query can be further restrained with the --query parameter).
# aodh alarm create --type gnocchi_aggregation_by_resources_threshold --name iops-monitor-read-requests --metric disk.read.requests.rate --threshold 42000 --aggregation-method sum --resource-type instance --query '{"=": {"project_id": "745d33000ac74d30a77539f8920555e7"}}'
+---------------------------+-----------------------------------------------------------+
| Field | Value |
+---------------------------+-----------------------------------------------------------+
| aggregation_method | sum |
| alarm_actions | [] |
| alarm_id | 192aba27-d823-4ede-a404-7f6b3cc12469 |
| comparison_operator | eq |
| description | gnocchi_aggregation_by_resources_threshold alarm rule |
| enabled | True |
| evaluation_periods | 1 |
| granularity | 60 |
| insufficient_data_actions | [] |
| metric | disk.read.requests.rate |
| name | iops-monitor-read-requests |
| ok_actions | [] |
| project_id | 745d33000ac74d30a77539f8920555e7 |
| query | {"=": {"project_id": "745d33000ac74d30a77539f8920555e7"}} |
| repeat_actions | False |
| resource_type | instance |
| severity | low |
| state | insufficient data |
| state_timestamp | 2016-11-08T23:41:22.919000 |
| threshold | 42000.0 |
| time_constraints | [] |
| timestamp | 2016-11-08T23:41:22.919000 |
| type | gnocchi_aggregation_by_resources_threshold |
| user_id | 8c4aea738d774967b4ef388eb41fef5e |
+---------------------------+-----------------------------------------------------------+3.9. Manage Resource Types
Telemetry resource types that were previously hardcoded can now be managed by the gnocchi client. You can use the gnocchi client to create, view, and delete resource types, and you can use the gnocchi API to update or delete attributes.
1. Create a new resource-type:
$ gnocchi resource-type create testResource01 -a bla:string:True:min_length=123 +----------------+------------------------------------------------------------+ | Field | Value | +----------------+------------------------------------------------------------+ | attributes/bla | max_length=255, min_length=123, required=True, type=string | | name | testResource01 | | state | active | +----------------+------------------------------------------------------------+
2. Review the configuration of the resource-type:
$ gnocchi resource-type show testResource01 +----------------+------------------------------------------------------------+ | Field | Value | +----------------+------------------------------------------------------------+ | attributes/bla | max_length=255, min_length=123, required=True, type=string | | name | testResource01 | | state | active | +----------------+------------------------------------------------------------+
3. Delete the resource-type:
$ gnocchi resource-type delete testResource01
You cannot delete a resource type if a resource is using it.
3.10. Using the Time-Series-Database-as-a-Service
Time-Series-Database-as-a-Service (gnocchi) is a multi-tenant, metrics and resource database. It is designed to store metrics at a very large scale while providing access to metrics and resources information to operators and users.
Currently, the TSDaaS uses the Identity service for authentication, and Ceph, Object Storage to store data.
TDSaaS provides the statsd deamon that is compatible with the statsd protocol and can listen to the metrics sent over the network, named gnocchi-statsd. In order to enable statsd support in TDSaaS, you need to configure the [statsd] option in the configuration file. The resource ID parameter is used as the main generic resource where all the metrics are attached, a user and project ID that are associated with the resource and metrics, and an archive policy name thatis used to create the metrics.
All the metrics will be created dynamically as the metrics are sent to gnocchi-statsd, and attached with the provided name to the resource ID you configured. For more information on installing and configuring TSDaaS, see the Install Time-Series-Database-as-a-Service chapter in the Manual Installation Procedures available at: https://access.redhat.com/documentation/en/red-hat-enterprise-linux-openstack-platform/
3.10.1. Running Time-Series-Database-as-a-Service
Run Time-Series-Database-as-a-Service (TSDaaS) by running the HTTP server and metric daemon:
# gnocchi-api # gnocchi-metricd
3.10.2. Running As A WSGI Application
You can run the TSDaaS through a WSGI service such as mod_wsgi or any other WSGI application. The file gnocchi/rest/app.wsgi provided with TSDaaS allows you to enable Gnocchi as a WSGI application.
The TSDaaS API tier runs using WSGI. This means it can be run using Apache httpd and mod_wsgi, or another HTTP daemon such as uwsgi. You should configure the number of processes and threads according to the number of CPUs you have, usually around 1.5 × number of CPUs. If one server is not enough, you can spawn any number of new API servers to scale Gnocchi out, even on different machines.
3.10.3. metricd Workers
By default, the gnocchi-metricd daemon spans all your CPU power in order to maximize CPU utilisation when computing metric aggregation. You can use the gnocchi status command to query the HTTP API and get the cluster status for metric processing. This command displays the number of metrics to process, known as the processing backlog for the gnocchi-metricd. As long as this backlog is not continuously increasing, that means that gnocchi-metricd is able to cope with the amount of metric that are being sent. If the number of measure to process is continuously increasing, you will need to (maybe temporarily) increase the number of the gnocchi-metricd daemons. You can run any number of metricd daemons on any number of servers.
3.10.4. Monitoring the Time-Series-Database-as-a-Service
The /v1/status endpoint of the HTTP API returns various information, such as the number of measures to process (measures backlog), which you can easily monitor. Making sure that the HTTP server and the gnocchi-metricd daemon are running and are not writing anything alarming in their logs is a sign of good health of the overall system.
3.10.5. Backing up and Restoring Time-Series-Database-as-a-Service
In order to be able to recover from an unfortunate event, you need to backup both the index and the storage. That means creating a database dump (PostgreSQL or MySQL) and doing snapshots or copies of your data storage (Ceph, Swift or your file system). The procedure to restore is: restore your index and storage backups, reinstall TSDaaS if necessary, and restart it.
