Red Hat SummitQuesto contenuto non è disponibile nella lingua selezionata.
Chapter 4. Testing and troubleshooting autoscaling
Use the Orchestration service (heat) to automatically scale instances up and down based on threshold definitions. To troubleshoot your environment, you can look for errors in the log files and history records.
4.1. Testing automatic scaling up of instances
You can use the Orchestration service (heat) to scale instances automatically based on the cpu_alarm_high threshold definition. When the CPU use reaches a value defined in the threshold parameter, another instance starts up to balance the load. The threshold value in the template.yaml file is set to 80%.
Procedure
-
Log in to the host environment as the
stackuser. For standalone environments set the
OS_CLOUDenvironment variable:export OS_CLOUD=standalone
[stack@standalone ~]$ export OS_CLOUD=standaloneCopy to Clipboard Copied! Toggle word wrap Toggle overflow For director environments source the
overcloudrcfile:source ~/overcloudrc
[stack@undercloud ~]$ source ~/overcloudrcCopy to Clipboard Copied! Toggle word wrap Toggle overflow Log in to the instance:
ssh -i ~/mykey.pem cirros@192.168.122.8
$ ssh -i ~/mykey.pem cirros@192.168.122.8Copy to Clipboard Copied! Toggle word wrap Toggle overflow Run multiple
ddcommands to generate the load:sudo dd if=/dev/zero of=/dev/null & sudo dd if=/dev/zero of=/dev/null & sudo dd if=/dev/zero of=/dev/null &
[instance ~]$ sudo dd if=/dev/zero of=/dev/null & [instance ~]$ sudo dd if=/dev/zero of=/dev/null & [instance ~]$ sudo dd if=/dev/zero of=/dev/null &Copy to Clipboard Copied! Toggle word wrap Toggle overflow - Exit from the running instance and return to the host.
After you run the
ddcommands, you can expect to have 100% CPU use in the instance. Verify that the alarm has been triggered:Copy to Clipboard Copied! Toggle word wrap Toggle overflow After approximately 60 seconds, Orchestration starts another instance and adds it to the group. To verify that an instance has been created, enter the following command:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow After another short period of time, observe that the Orchestration service has autoscaled to three instances. The configuration is set to a maximum of three instances. Verify there are three instances:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow
4.2. Testing automatic scaling down of instances
You can use the Orchestration service (heat) to automatically scale down instances based on the cpu_alarm_low threshold. In this example, the instances are scaled down when CPU use is below 5%.
Procedure
From within the workload instance, terminate the running
ddprocesses and observe Orchestration begin to scale the instances back down.killall dd
$ killall ddCopy to Clipboard Copied! Toggle word wrap Toggle overflow -
Log in to the host environment as the
stackuser. For standalone environments set the
OS_CLOUDenvironment variable:export OS_CLOUD=standalone
[stack@standalone ~]$ export OS_CLOUD=standaloneCopy to Clipboard Copied! Toggle word wrap Toggle overflow For director environments source the
overcloudrcfile:source ~/overcloudrc
[stack@undercloud ~]$ source ~/overcloudrcCopy to Clipboard Copied! Toggle word wrap Toggle overflow When you stop the
ddprocesses, this triggers thecpu_alarm_low eventalarm. As a result, Orchestration begins to automatically scale down and remove the instances. Verify that the corresponding alarm has triggered:Copy to Clipboard Copied! Toggle word wrap Toggle overflow After a few minutes, Orchestration continually reduce the number of instances to the minimum value defined in the
min_sizeparameter of thescaleup_groupdefinition. In this scenario, themin_sizeparameter is set to1.
4.3. Troubleshooting for autoscaling
If your environment is not working properly, you can look for errors in the log files and history records.
Procedure
-
Log in to the host environment as the
stackuser. For standalone environments set the
OS_CLOUDenvironment variable:export OS_CLOUD=standalone
[stack@standalone ~]$ export OS_CLOUD=standaloneCopy to Clipboard Copied! Toggle word wrap Toggle overflow For director environments source the
overcloudrcfile:source ~/overcloudrc
[stack@undercloud ~]$ source ~/overcloudrcCopy to Clipboard Copied! Toggle word wrap Toggle overflow You must first launch the ephemeral Heat process to use the
openstack stackcommands:openstack tripleo launch heat --heat-dir /home/stack/overcloud-deploy/overcloud/heat-launcher --restore-db export OS_CLOUD=heat
(undercloud)$ openstack tripleo launch heat --heat-dir /home/stack/overcloud-deploy/overcloud/heat-launcher --restore-db (undercloud)$ export OS_CLOUD=heatCopy to Clipboard Copied! Toggle word wrap Toggle overflow To retrieve information on state transitions, list the stack event records:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Read the alarm history log:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow To view the records of scale-out or scale-down operations that heat collects for the existing stack, you can use the
awkcommand to parse theheat-engine.log:awk '/Stack UPDATE started/,/Stack CREATE completed successfully/ {print $0}' /var/log/containers/heat/heat-engine.log$ awk '/Stack UPDATE started/,/Stack CREATE completed successfully/ {print $0}' /var/log/containers/heat/heat-engine.logCopy to Clipboard Copied! Toggle word wrap Toggle overflow To view aodh-related information, examine the
evaluator.log:grep -i alarm /var/log/containers/aodh/evaluator.log | grep -i transition
$ grep -i alarm /var/log/containers/aodh/evaluator.log | grep -i transitionCopy to Clipboard Copied! Toggle word wrap Toggle overflow Remove the ephemeral Heat process from the undercloud:
openstack tripleo launch heat --kill
(undercloud)$ openstack tripleo launch heat --killCopy to Clipboard Copied! Toggle word wrap Toggle overflow
4.4. Using CPU telemetry values for autoscaling threshold when using rate:mean aggregration
When using the OS::Heat::Autoscaling heat orchestration template (HOT) and setting a threshold value for CPU, the value is expressed in nanoseconds of CPU time which is a dynamic value based on the number of virtual CPUs allocated to the instance workload. In this reference guide we’ll explore how to calculate and express the CPU nanosecond value as a percentage when using the Gnocchi rate:mean aggregration method.
4.4.1. Calculating CPU telemetry values as a percentage
CPU telemetry is stored in Gnocchi (OpenStack time-series data store) as CPU utilization in nanoseconds. When using CPU telemetry to define autoscaling thresholds it is useful to express the values as a percentage of CPU utilization since that is more natural when defining the threshold values. When defining the scaling policies used as part of an autoscaling group, we can take our desired threshold defined as a percentage and calculate the required threshold value in nanoseconds which is used in the policy definitions.
| Value (ns) | Granularity (s) | Percentage |
|---|---|---|
| 60000000000 | 60 | 100 |
| 54000000000 | 60 | 90 |
| 48000000000 | 60 | 80 |
| 42000000000 | 60 | 70 |
| 36000000000 | 60 | 60 |
| 30000000000 | 60 | 50 |
| 24000000000 | 60 | 40 |
| 18000000000 | 60 | 30 |
| 12000000000 | 60 | 20 |
| 6000000000 | 60 | 10 |
4.4.2. Displaying instance workload vCPU as a percentage
You can display the gnocchi-stored CPU telemetry data as a percentage rather than the nanosecond values for instances by using the openstack metric aggregates command.
Prerequisites
- Create a heat stack using the autoscaling group resource that results in an instance workload.
Procedure
- Login to your OpenStack environment as the cloud adminstrator.
Retrieve the ID of the autoscaling group heat stack:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Set the value of the stack ID to an environment variable:
export STACK_ID=$(openstack stack show vnf -c id -f value)
$ export STACK_ID=$(openstack stack show vnf -c id -f value)Copy to Clipboard Copied! Toggle word wrap Toggle overflow Return the metrics as an aggregate by resource type instance (server ID) with the value calculated as a percentage. The aggregate is returned as a value of nanoseconds of CPU time. We divide that number by 1000000000 to get the value in seconds. We then divide the value by our granularity, which in this example is 60 seconds. That value is then converted to a percentage by multiplying by 100. Finally, we divide the total value by the number of vCPU provided by the flavor assigned to the instance, in this example a value of 2 vCPU, providing us a value expressed as a percentage of CPU time:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow
4.4.3. Retrieving available telemetry for an instance workload
Retrieve the available telemetry for an instance workload and express the vCPU utilization as a percentage.
Prerequisites
- Create a heat stack using the autoscaling group resource that results in an instance workload.
Procedure
- Login to your OpenStack environment as the cloud adminstrator.
Retrieve the ID of the autoscaling group heat stack:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Set the value of the stack ID to an environment variable:
export STACK_ID=$(openstack stack show vnf -c id -f value)
$ export STACK_ID=$(openstack stack show vnf -c id -f value)Copy to Clipboard Copied! Toggle word wrap Toggle overflow Retrieve the ID of the workload instance you want to return data for. We are using the server list long form and filtering for instances that are part of our autoscaling group:
openstack server list --long --fit-width | grep "metering.server_group='$STACK_ID'"
$ openstack server list --long --fit-width | grep "metering.server_group='$STACK_ID'" | bc1811de-48ed-44c1-ae22-c01f36d6cb02 | vn-xlfb4jb-yhbq6fkk2kec-qsu2lr47zigs-vnf-y27wuo25ce4e | ACTIVE | None | Running | private=192.168.100.139, 192.168.25.179 | fedora36 | d21f1aaa-0077-4313-8a46-266c39b705c1 | m1.small | 692533fe-0912-417e-b706-5d085449db53 | nova | standalone.localdomain | metering.server_group='e0a15cee-34d1-418a-ac79-74ad07585730' |Copy to Clipboard Copied! Toggle word wrap Toggle overflow Set the instance ID for one of the returned instance workload names:
INSTANCE_NAME='vn-xlfb4jb-yhbq6fkk2kec-qsu2lr47zigs-vnf-y27wuo25ce4e' ; export INSTANCE_ID=$(openstack server list --name $INSTANCE_NAME -c ID -f value)
$ INSTANCE_NAME='vn-xlfb4jb-yhbq6fkk2kec-qsu2lr47zigs-vnf-y27wuo25ce4e' ; export INSTANCE_ID=$(openstack server list --name $INSTANCE_NAME -c ID -f value)Copy to Clipboard Copied! Toggle word wrap Toggle overflow Verify metrics have been stored for the instance resource ID. If no metrics are available it’s possible not enough time has elapsed since the instance was created. If enough time has elapsed, you can check the logs for the data collection service in
/var/log/containers/ceilometer/and logs for the time-series database service gnocchi in/var/log/containers/gnocchi/:Copy to Clipboard Copied! Toggle word wrap Toggle overflow Verify there are available measures for the resource metric and note the granularity value as we’ll use it when running the
openstack metric aggregatescommand:Copy to Clipboard Copied! Toggle word wrap Toggle overflow Retrieve the number of vCPU cores applied to the workload instance by reviewing the configured flavor for the instance workload:
openstack server show $INSTANCE_ID -cflavor -f value openstack flavor show 692533fe-0912-417e-b706-5d085449db53 -c vcpus -f value
$ openstack server show $INSTANCE_ID -cflavor -f value m1.small (692533fe-0912-417e-b706-5d085449db53) $ openstack flavor show 692533fe-0912-417e-b706-5d085449db53 -c vcpus -f value 2Copy to Clipboard Copied! Toggle word wrap Toggle overflow Return the metrics as an aggregate by resource type instance (server ID) with the value calculated as a percentage. The aggregate is returned as a value of nanoseconds of CPU time. We divide that number by 1000000000 to get the value in seconds. We then divide the value by our granularity, which in this example is 60 seconds (as previously retrieved with
openstack metric measures showcommand). That value is then converted to a percentage by multiplying by 100. Finally, we divide the total value by the number of vCPU provided by the flavor assigned to the instance, in this example a value of 2 vCPU, providing us a value expressed as a percentage of CPU time:Copy to Clipboard Copied! Toggle word wrap Toggle overflow
'%20d='M201.5%20305.5c-13.8%200-24.9-11.1-24.9-24.6%200-13.8%2011.1-24.9%2024.9-24.9%2013.6%200%2024.6%2011.1%2024.6%2024.9%200%2013.6-11.1%2024.6-24.6%2024.6zM504%20256c0%20137-111%20248-248%20248S8%20393%208%20256%20119%208%20256%208s248%20111%20248%20248zm-132.3-41.2c-9.4%200-17.7%203.9-23.8%2010-22.4-15.5-52.6-25.5-86.1-26.6l17.4-78.3%2055.4%2012.5c0%2013.6%2011.1%2024.6%2024.6%2024.6%2013.8%200%2024.9-11.3%2024.9-24.9s-11.1-24.9-24.9-24.9c-9.7%200-18%205.8-22.1%2013.8l-61.2-13.6c-3-.8-6.1%201.4-6.9%204.4l-19.1%2086.4c-33.2%201.4-63.1%2011.3-85.5%2026.8-6.1-6.4-14.7-10.2-24.1-10.2-34.9%200-46.3%2046.9-14.4%2062.8-1.1%205-1.7%2010.2-1.7%2015.5%200%2052.6%2059.2%2095.2%20132%2095.2%2073.1%200%20132.3-42.6%20132.3-95.2%200-5.3-.6-10.8-1.9-15.8%2031.3-16%2019.8-62.5-14.9-62.5zM302.8%20331c-18.2%2018.2-76.1%2017.9-93.6%200-2.2-2.2-6.1-2.2-8.3%200-2.5%202.5-2.5%206.4%200%208.6%2022.8%2022.8%2087.3%2022.8%20110.2%200%202.5-2.2%202.5-6.1%200-8.6-2.2-2.2-6.1-2.2-8.3%200zm7.7-75c-13.6%200-24.6%2011.1-24.6%2024.9%200%2013.6%2011.1%2024.6%2024.6%2024.6%2013.8%200%2024.9-11.1%2024.9-24.6%200-13.8-11-24.9-24.9-24.9z'/%3e%3c/svg%3e){kind=small}