OpenStack Integration Test Suite Guide
Introduction to the OpenStack Integration Test Suite
Abstract
Making open source more inclusive Copy linkLink copied to clipboard!
Red Hat is committed to replacing problematic language in our code, documentation, and web properties. We are beginning with these four terms: master, slave, blacklist, and whitelist. Because of the enormity of this endeavor, these changes will be implemented gradually over several upcoming releases. For more details, see our CTO Chris Wright’s message.
Providing feedback on Red Hat documentation Copy linkLink copied to clipboard!
We appreciate your input on our documentation. Tell us how we can make it better.
Using the Direct Documentation Feedback (DDF) function
Use the Add Feedback DDF function for direct comments on specific sentences, paragraphs, or code blocks.
- View the documentation in the Multi-page HTML format.
- Ensure that you see the Feedback button in the upper right corner of the document.
- Highlight the part of text that you want to comment on.
- Click Add Feedback.
- Complete the Add Feedback field with your comments.
- Optional: Add your email address so that the documentation team can contact you for clarification on your issue.
- Click Submit.
Chapter 1. OpenStack Integration Test Suite (tempest) validations Copy linkLink copied to clipboard!
Because Red Hat OpenStack Platform (RHOSP) consists of many different projects, it is important to test the interoperability of the projects within your RHOSP cluster. The OpenStack Integration Test Suite automates the integration testing of your RHOSP deployment. You can run tests to ensure that your cluster works as expected. Test output to provide early warning of potential problems, especially after an upgrade.
The Integration Test Suite contains tests for OpenStack API validation and scenario testing, as well as unit testing for self-validation. The Integration Test Suite performs black box testing by using the OpenStack public APIs, with tempest as the test runner.
The OpenStack Integration Test Suite (tempest) acts as a gate for commits to the Red Hat OpenStack Platform (RHOSP) core projects, it can stress test to generate load on a cloud deployment, and it can perform CLI tests to check the response formatting of the command line. You can run scenario tests and API tests against your RHOSP cloud deployment.
Scenario tests
Scenario tests simulate a typical end user action workflow to test the integration points between services. The testing framework conducts the configuration, tests the integration between services, and is then removed automatically. Tag the tests with the services that they relate to clarify, which client libraries the test uses.
The following scenarios are based on a use case:
- Uploading an image to the Image Service
- Deploying an instance from the image
- Attaching a volume to the instance
- Creating a snapshot of the instance
- Detaching the volume from the instance
API tests
API tests validate the OpenStack API. Tests use the OpenStack Integration Test Suite implementation of the OpenStack API. You can use both valid and invalid JSON to ensure that error responses are valid. You can run tests independently and you do not have to rely on the previous test state.
Chapter 2. Installing the Integration Test Suite (tempest) Copy linkLink copied to clipboard!
You can install the Integration Test Suite either with director or with a manual installation.
- To install the Integration Test Suite with director, see Installing the Integration Test Suite with director.
- To manually install the Integration Test Suite, Installing the Integration Test Suite manually.
2.1. Prerequisites Copy linkLink copied to clipboard!
- An undercloud installation. For more information, see Director installation and configuration.
- An overcloud deployment. For more information, see Planning your overcloud.
2.2. Installing the Integration Test Suite with director Copy linkLink copied to clipboard!
Use the Red Hat OpenStack Platform (RHOSP) director to install the test suite automatically.
Prerequisites
-
You have installed the
python3-tripleoclientpackages. For more information, see Installing director packages in the Director Installation and Usage guide.
Procedure
-
Log in to the undercloud host as the
stackuser. -
Edit the
undercloud.conffile located in the home directory of thestackuser. Set the
enable_tempestparameter totrue.enable_tempest = true
enable_tempest = trueCopy to Clipboard Copied! Toggle word wrap Toggle overflow Run the
openstack undercloud installcommand to include the extra configuration in the undercloud:openstack undercloud install
$ openstack undercloud installCopy to Clipboard Copied! Toggle word wrap Toggle overflow
2.3. Installing the Integration Test Suite manually Copy linkLink copied to clipboard!
If you do not want to install the Integration Test Suite (tempest) automatically with director, you can perform the installation manually later. You must ensure that you have a basic network configuration, install the Integration Test Suite packages, and create a configuration file that contains details about your OpenStack services and other testing behaviour switches.
Procedure
Ensure that the following networks are available within your Red Hat OpenStack Platform (RHOSP) environment:
- An external network that can provide a floating IP.
A private network.
Connect these networks through a router.
To create the private network, specify the following options according to your network deployment:
openstack network create <network_name> --share openstack subnet create <subnet_name> --subnet-range <address/prefix> \ --network <network_name> openstack router create <router_name> openstack router add subnet <router_name> <subnet_name>
$ openstack network create <network_name> --share $ openstack subnet create <subnet_name> --subnet-range <address/prefix> \ --network <network_name> $ openstack router create <router_name> $ openstack router add subnet <router_name> <subnet_name>Copy to Clipboard Copied! Toggle word wrap Toggle overflow To create the public network, specify the following options according to your network deployment:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow
Install the packages related to the Integration Test Suite:
sudo dnf -y install openstack-tempest
$ sudo dnf -y install openstack-tempestCopy to Clipboard Copied! Toggle word wrap Toggle overflow This command does not install any tempest plugins. You must install the plugins manually, depending on your RHOSP installation.
Install the appropriate tempest plugin for each component in your environment. For example, enter the following command to install the keystone, neutron, cinder, and telemetry plugins:
sudo dnf install python3-keystone-tests-tempest python3-neutron-tests-tempest python3-cinder-tests-tempest python3-telemetry-tests-tempest
$ sudo dnf install python3-keystone-tests-tempest python3-neutron-tests-tempest python3-cinder-tests-tempest python3-telemetry-tests-tempestCopy to Clipboard Copied! Toggle word wrap Toggle overflow For a full list of packages, see Integration Test Suite packages.
You can also install the openstack-tempest-all package. This package contains all of the tempest plugins.
2.3.1. Integration Test Suite packages Copy linkLink copied to clipboard!
Use dnf search to retrieve a list of tempest test packages:
sudo dnf search $(openstack service list -c Name -f value) 2>/dev/null | grep test | awk '{print $1}'
$ sudo dnf search $(openstack service list -c Name -f value) 2>/dev/null | grep test | awk '{print $1}'
| Component | Package Name |
|---|---|
| barbican | python3-barbican-tests-tempest |
| cinder | python3-cinder-tests-tempest |
| designate | python3-designate-tests-tempest |
| ec2-api | python3-ec2api-tests-tempest |
| heat | python3-heat-tests-tempest |
| ironic | python3-ironic-tests-tempest |
| keystone | python3-keystone-tests-tempest |
| kuryr | python3-kuryr-tests-tempest |
| manila | python3-manila-tests-tempest |
| mistral | python3-mistral-tests-tempest |
| networking-bgvpn | python3-networking-bgpvpn-tests-tempest |
| networking-l2gw | python3-networking-l2gw-tests-tempest |
| neutron | python3-neutron-tests-tempest |
| nova-join | python3-novajoin-tests-tempest |
| octavia | python3-octavia-tests-tempest |
| patrole | python3-patrole-tests-tempest |
| telemetry | python3-telemetry-tests-tempest |
| tripleo-common | python3-tripleo-common-tests-tempest |
| zaqar | python3-zaqar-tests-tempest |
The python3-telemetry-tests-tempest package contains plugins for aodh, panko, gnocchi, and ceilometer tests. The python3-ironic-tests-tempest package contains plugins for ironic and ironic-inspector.
Chapter 3. Configuring the Integration Test Suite (tempest) Copy linkLink copied to clipboard!
Before you begin validating your environment with the Integration Test Suite, you must create a workspace and generate the /etc/tempest.conf configuration file.
3.1. Prerequisites Copy linkLink copied to clipboard!
- An OpenStack environment that contains the Integration Test Suite packages. For more information, see Installing the Integration Test Suite with director.
3.2. Creating a workspace Copy linkLink copied to clipboard!
Create a workspace for your Integration Test Suite (tempest) configuration and output.
Procedure
Source the credentials for the target deployment:
If the target is in the undercloud, source the credentials for the undercloud:
source stackrc
# source stackrcCopy to Clipboard Copied! Toggle word wrap Toggle overflow If the target is in the overcloud, source the credentials for the overcloud:
source overcloudrc
# source overcloudrcCopy to Clipboard Copied! Toggle word wrap Toggle overflow
Initialize
tempest:tempest init mytempest cd mytempest
# tempest init mytempest # cd mytempestCopy to Clipboard Copied! Toggle word wrap Toggle overflow This command creates a tempest workspace named
mytempest.Optional: Enter the following command to view a list of existing workspaces:
tempest workspace list
# tempest workspace listCopy to Clipboard Copied! Toggle word wrap Toggle overflow Generate the
etc/tempest.conffile:discover-tempest-config --deployer-input ~/tempest-deployer-input.conf \ --debug --create --network-id <UUID>
# discover-tempest-config --deployer-input ~/tempest-deployer-input.conf \ --debug --create --network-id <UUID>Copy to Clipboard Copied! Toggle word wrap Toggle overflow Replace
UUIDwith the UUID of the external network.discover-tempest-configwas formerly calledconfig_tempest.pyand uses the same parameters.python-tempestconfis as a dependency ofopenstack-tempestand provides thediscover-tempest-config.NoteTo generate the
etc/tempest.conffile for the undercloud, ensure that the region name in thetempest-deployer-input.conffile is the same as the name in the undercloud deployment. If these names do not match, update the region name in thetempest-deployer-input.conffile to match the region name of your undercloud.To inspect the region name of your undercloud, enter the following commands:
source stackrc openstack region list
$ source stackrc $ openstack region listCopy to Clipboard Copied! Toggle word wrap Toggle overflow To inspect the region name of your overcloud, enter the following commands:
source overcloudrc openstack region list
$ source overcloudrc $ openstack region listCopy to Clipboard Copied! Toggle word wrap Toggle overflow
You might need to modify the default tempest.conf file to suit your environment. For more information, see Configuring extension lists and Configuring heat_plugin.
Verification
Verify your current tempest configuration:
tempest verify-config -o <output>
# tempest verify-config -o <output>Copy to Clipboard Copied! Toggle word wrap Toggle overflow
The value of output is the output file where Integration Test Suite writes your updated configuration. This is different from your original configuration file.
3.3. Configuring the Integration Test Suite manually Copy linkLink copied to clipboard!
The discover-tempest-config command generates the tempest.conf file automatically. However, you must ensure that the tempest.conf file corresponds to the configuration of your environment.
3.3.1. Configuring Integration Test Suite extension lists manually Copy linkLink copied to clipboard!
The default tempest.conf file contains lists of extensions for each component. Inspect the api_extensions attribute for each component in the tempest.conf file and verify that the lists of extensions correspond to your deployment.
If the extensions that are available in your deployment do not correspond to the list of extensions in the api_extensions attribute of the tempest.conf file, the component fails tempest tests. To prevent this failure, you must identify the extensions that are available in your deployment and include them in the api_extensions parameter. To get a list of Network, Compute, Volume, or Identity extensions in your deployment, run the following command:
Procedure
To retrieve a list of Network, Compute, Volume, or Identity extensions in your deployment, enter the following command:
openstack extension list [--network] [--compute] [--volume] [--identity]
$ openstack extension list [--network] [--compute] [--volume] [--identity]Copy to Clipboard Copied! Toggle word wrap Toggle overflow
3.3.2. Configuring heat_plugin manually Copy linkLink copied to clipboard!
You can configure heat_plugin manually in the tempest.conf file.
Procedure
Use the following example to configure
heat_pluginaccording to your deployment:Copy to Clipboard Copied! Toggle word wrap Toggle overflow
Use the openstack network list command to identify networks for the fixed_network_name, network_for_ssh, and floating_network_name parameters.
You must set heat to True in the [service_available] section of the tempest.conf file, and the user in the username attribute of the [heat_plugin] section must have the role member. For example, enter the following command to add the member role to the demo user:
openstack role add --user demo --project demo member
$ openstack role add --user demo --project demo member
3.4. Configuring Integration Test Suite logging Copy linkLink copied to clipboard!
You can change the default location for log files in the logs directory within your tempest workspace.
Procedure
In
tempest.conf, under the[DEFAULT]section, setlog_dirto the desired directory:[DEFAULT] log_dir = <directory>
[DEFAULT] log_dir = <directory>Copy to Clipboard Copied! Toggle word wrap Toggle overflow If you have your own logging configuration file, in
tempest.conf, under the[DEFAULT]section, setlog_config_appendto your file:[DEFAULT] log_config_append = <file>
[DEFAULT] log_config_append = <file>Copy to Clipboard Copied! Toggle word wrap Toggle overflow
If you set the log_config_append attribute, the Integration Test Suite ignores all other logging configuration in tempest.conf, including the log_dir attribute.
3.5. Configuring Integration Test Suite microversion tests Copy linkLink copied to clipboard!
The Integration Test Suite (tempest) provides stable interfaces to test the API microversions. To implement microversion tests by using these interfaces, complete the following steps.
Procedure
-
Configure options in the
tempest.confconfiguration file to specify the target microversions. Configure these options to ensure that the supported microversions correspond to the microversions in the OpenStack cloud. You can specify a range of target microversions to run multiple microversion tests in a single Integration Test Suite operation.
For example, to limit the range of microversions for the
computeservice, in the[compute]section of your configuration file, assign values to themin_microversionandmax_microversionparameters:[compute] min_microversion = 2.14 max_microversion = latest
[compute] min_microversion = 2.14 max_microversion = latestCopy to Clipboard Copied! Toggle word wrap Toggle overflow
Chapter 4. Cleaning Integration Test Suite (tempest) resources Copy linkLink copied to clipboard!
Before you validate your deployments by using OpenStack Integration Test Suite (tempest), run the cleanup command with the --init-saved-state flag. This command scans your environment to discover resources, for example networks, volumes, images, flavors, projects, and users. The discovered resources are saved in a file called saved_state.json. When the tempest cleanup command is executed all resources not recorded in the saved_state.json file are deleted.
Prerequisites
- An OpenStack environment that contains the Integration Test Suite packages. For more information, see Installing the Integration Test Suite with director.
- An Integration Test Suite configuration that corresponds to your OpenStack environment. For more information, see Creating a workspace.
- One or more completed Integration Test Suite validation tests.
4.1. Performing a dry run Copy linkLink copied to clipboard!
Perform a dry run before you execute the cleanup. A dry run lists the files that Integration Test Suite would delete by a cleanup, without actually deleting any files. The dry_run.json file contains the list of files that a cleanup deletes.
Procedure
Complete the dry run:
tempest cleanup --dry-run
# tempest cleanup --dry-runCopy to Clipboard Copied! Toggle word wrap Toggle overflow -
Review the
dry_run.jsonfile to ensure that the cleanup does not delete any files that you require for your environment.
4.2. Performing a tempest clean up Copy linkLink copied to clipboard!
Before you run any tempest tests, you must initialize the saved state. This creates the file saved_state.json, which prevents the cleanup from deleting objects that must be kept.
If you do not run the cleanup command with the --init-saved-state flag, RHOSP objects are deleted.
If you create objects after running the cleanup command with --init-saved-state, those objects can be deleted by subsequent tempest commands.
Procedure
Initialize the saved state to create the
saved_state.jsonfile:tempest cleanup --init-saved-state
# tempest cleanup --init-saved-stateCopy to Clipboard Copied! Toggle word wrap Toggle overflow Perform the cleanup:
tempest cleanup
# tempest cleanupCopy to Clipboard Copied! Toggle word wrap Toggle overflow
The tempest cleanup command deletes tempest resources but does not delete projects or the tempest administrator account.
You can modify the saved_state.json file to include or exclude objects that you want to retain or remove.
Chapter 5. Validating your OpenStack cloud with the Integration Test Suite (tempest) Copy linkLink copied to clipboard!
You can run Integration Test Suite validations in many ways with the tempest run command. You can also combine multiple options in a single tempest run command.
5.1. Prerequisites Copy linkLink copied to clipboard!
- An OpenStack environment that contains the Integration Test Suite packages. For more information, see Installing the Integration Test Suite with director.
- An Integration Test Suite configuration that corresponds to your OpenStack environment. For more information, see Creating a workspace.
5.2. Listing available tests Copy linkLink copied to clipboard!
Use the --list-tests option to list all available tests.
Procedure
Enter the
tempest runcommand with either the--list-testsor-loptions to get a list of available tempest tests:tempest run -l
# tempest run -lCopy to Clipboard Copied! Toggle word wrap Toggle overflow
5.3. Running smoke tests Copy linkLink copied to clipboard!
Smoke testing is a type of preliminary testing which covers only the most important functionality. Although these tests are not comprehensive, running smoke tests can save time if they do identify a problem.
Procedure
Enter the
tempest runcommand with the--whitelist-fileoption to use a whitelist file:tempest run --smoke
# tempest run --smokeCopy to Clipboard Copied! Toggle word wrap Toggle overflow
5.4. Passing tests by using allowlist files Copy linkLink copied to clipboard!
An allowlist file is a file that contains regular expressions to select tests that you want to include. If you use one or more regular expressions, specify each expression on a separate line.
Procedure
Enter the
tempest runcommand with either the--whitelist-fileor-woptions to use an allowlist file:tempest run -w <whitelist_file>
# tempest run -w <whitelist_file>Copy to Clipboard Copied! Toggle word wrap Toggle overflow
5.5. Skipping tests by using blocklist files Copy linkLink copied to clipboard!
A blocklist file is a file that contains regular expressions to select tests that you want to exclude. If you use one or more regular expressions, specify each expression on a separate line.
Procedure
Enter the
tempest runcommand with either the--blacklist-fileor-boptions to use a blocklist file:tempest run -b <blacklist_file>
# tempest run -b <blacklist_file>Copy to Clipboard Copied! Toggle word wrap Toggle overflow
5.6. Running tests in parallel or in series Copy linkLink copied to clipboard!
You can run tests in parallel, or in series. You can also define the number of workers that you want to use when you run parallel tests. By default, the Integration Test Suite uses one worker for each CPU available.
Choose to run the tests serially or in parallel:
Run the tests serially:
tempest run --serial
# tempest run --serialCopy to Clipboard Copied! Toggle word wrap Toggle overflow Run the tests in parallel (default):
tempest run --parallel
# tempest run --parallelCopy to Clipboard Copied! Toggle word wrap Toggle overflow Use the
--concurrencyor-coption to specify the number of workers to use when you run tests in parallel:tempest run --concurrency <workers>
# tempest run --concurrency <workers>Copy to Clipboard Copied! Toggle word wrap Toggle overflow
5.7. Running specific tests Copy linkLink copied to clipboard!
Run specific tests with the --regex option. The regular expression must be Python regular expression:
Procedure
Enter the following command:
tempest run --regex <regex>
# tempest run --regex <regex>Copy to Clipboard Copied! Toggle word wrap Toggle overflow For example, use the following example command to run all tests that have names that begin with
tempest.scenario:tempest run --regex ^tempest.scenario
# tempest run --regex ^tempest.scenarioCopy to Clipboard Copied! Toggle word wrap Toggle overflow
5.8. Deleting Integration Test Suite objects Copy linkLink copied to clipboard!
Enter the tempest cleanup command to delete all Integration Test Suite (tempest) resources. This command also deletes projects, but the command does not delete the administrator account:
Procedure
Delete the tempest resources:
tempest cleanup --delete-tempest-conf-objects
# tempest cleanup --delete-tempest-conf-objectsCopy to Clipboard Copied! Toggle word wrap Toggle overflow