Chapter 17. Managing containers with Ansible
Red Hat OpenStack Platform 17.0 uses the tripleo_container_manage
Ansible role to perform management operations on containers. You can also write custom playbooks to perform specific container management operations:
-
Collect the container configuration data that heat generates. The
tripleo_container_manage
role uses this data to orchestrate container deployment. - Start containers.
- Stop containers.
- Update containers.
- Delete containers.
- Run a container with a specific configuration.
Although director performs container management automatically, you might want to customize a container configuration, or apply a hotfix to a container without redeploying the overcloud.
This role supports only Podman container management.
17.1. tripleo-container-manage role defaults and variables
The following excerpt shows the defaults and variables for the tripleo_container_manage
Ansible role.
# All variables intended for modification should place placed in this file. tripleo_container_manage_hide_sensitive_logs: '{{ hide_sensitive_logs | default(true) }}' tripleo_container_manage_debug: '{{ ((ansible_verbosity | int) >= 2) | bool }}' tripleo_container_manage_clean_orphans: true # All variables within this role should have a prefix of "tripleo_container_manage" tripleo_container_manage_check_puppet_config: false tripleo_container_manage_cli: podman tripleo_container_manage_concurrency: 1 tripleo_container_manage_config: /var/lib/tripleo-config/ tripleo_container_manage_config_id: tripleo tripleo_container_manage_config_overrides: {} tripleo_container_manage_config_patterns: '*.json' # Some containers where Puppet is run, can take up to 10 minutes to finish # in slow environments. tripleo_container_manage_create_retries: 120 # Default delay is 5s so 120 retries makes a timeout of 10 minutes which is # what we have observed a necessary value for nova and neutron db-sync execs. tripleo_container_manage_exec_retries: 120 tripleo_container_manage_healthcheck_disabled: false tripleo_container_manage_log_path: /var/log/containers/stdouts tripleo_container_manage_systemd_teardown: true
17.2. tripleo-container-manage molecule scenarios
Molecule
is used to test the tripleo_container_manage
role. The following shows a molecule
default inventory:
hosts: all: hosts: instance: ansible_host: localhost ansible_connection: local ansible_distribution: centos8
Usage
Red Hat OpenStack 17.0 supports only Podman in this role. Docker support is on the roadmap.
The Molecule
Ansible role performs the following tasks:
-
Collect container configuration data, generated by the TripleO Heat Templates. This data is used as a source of truth. If a container is already managed by
Molecule
, no matter its present state, the configuration data will reconfigure the container as needed. -
Manage
systemd
shutdown files. It creates theTripleO Container systemd
service, required for service ordering when shutting down or starting a node. It also manages thenetns-placeholder
service. Delete containers that are nonger needed or that require reconfiguration. It uses a custom filter, named
needs_delete()
which has a set of rules to determine if the container needs to be deleted.-
A container will not be deleted if, the container is not managed by
tripleo_ansible
or the containerconfig_id
does not match the input ID. -
A container will be deleted, if the container has no
config_data
or the container hasconfig_data
which does not match data in input. Note that when a container is removed, the role also disables and removes thesystemd
services and healtchecks.
-
A container will not be deleted if, the container is not managed by
Create containers in a specific order defined by
start_order
container config, where the default is 0.-
If the container is an
exec
, a dedicated playbook forexecs
is run, usingasync
so multipleexecs
can be run at the same time. -
Otherwise, the
podman_container
is used, in async, to create the containers. If the container has arestart policy
,systemd
service is configured. If the container has a healthcheck script,systemd healthcheck
service is configured.
-
If the container is an
tripleo_container_manage_concurrency
parameter is set to 1 by default, and putting a value higher than 2 can expose issues with Podman locks.
Example of a playbook:
- name: Manage step_1 containers using tripleo-ansible block: - name: "Manage containers for step 1 with tripleo-ansible" include_role: name: tripleo_container_manage vars: tripleo_container_manage_config: "/var/lib/tripleo-config/container-startup-config/step_1" tripleo_container_manage_config_id: "tripleo_step1"
17.3. tripleo_container_manage role variables
The tripleo_container_manage
Ansible role contains the following variables:
Name | Default value | Description |
---|---|---|
tripleo_container_manage_check_puppet_config | false |
Use this variable if you want Ansible to check Puppet container configurations. Ansible can identify updated container configuration using the configuration hash. If a container has a new configuration from Puppet, set this variable to |
tripleo_container_manage_cli | podman |
Use this variable to set the command line interface that you want to use to manage containers. The |
tripleo_container_manage_concurrency | 1 | Use this variable to set the number of containers that you want to manage concurrently. |
tripleo_container_manage_config | /var/lib/tripleo-config/ | Use this variable to set the path to the container configuration directory. |
tripleo_container_manage_config_id | tripleo |
Use this variable to set the ID of a specific configuration step. For example, set this value to |
tripleo_container_manage_config_patterns | *.json | Use this variable to set the bash regular expression that identifies configuration files in the container configuration directory. |
tripleo_container_manage_debug | false |
Use this variable to enable or disable debug mode. Run the |
tripleo_container_manage_healthcheck_disable | false | Use this variable to enable or disable healthchecks. |
tripleo_container_manage_log_path | /var/log/containers/stdouts | Use this variable to set the stdout log path for containers. |
tripleo_container_manage_systemd_order | false | Use this variable to enable or disable systemd shutdown ordering with Ansible. |
tripleo_container_manage_systemd_teardown | true | Use this variable to trigger the cleanup of obsolete containers. |
tripleo_container_manage_config_overrides | {} | Use this variable to override any container configuration. This variable takes a dictionary of values where each key is the container name and the parameters that you want to override, for example, the container image or user. This variable does not write custom overrides to the JSON container configuration files and any new container deployments, updates, or upgrades revert to the content of the JSON configuration file. |
tripleo_container_manage_valid_exit_code | [] |
Use this variable to check if a container returns an exit code. This value must be a list, for example, |
17.4. tripleo-container-manage healthchecks
Until Red Hat OpenStack 17.0, container healthcheck was implemented by a systemd
timer which would run podman exec
to determine if a given container was healthy. Now, it uses the native healthcheck interface in Podman
which is easier to integrate and consume.
To check if a container (for example, keystone) is healthy, run the following command:
$ sudo podman healthcheck run keystone
The return code should be 0
and “healthy”
.
"Healthcheck": { "Status": "healthy", "FailingStreak": 0, "Log": [ { "Start": "2020-04-14T18:48:57.272180578Z", "End": "2020-04-14T18:48:57.806659104Z", "ExitCode": 0, "Output": "" }, (...) ] }
17.5. tripleo-container-manage debug
The tripleo_container_manage
Ansible role allows you to perform specific actions on a given container. This can be used to:
- Run a container with a specific one-off configuration.
- Output the container commands to manage containers lifecycle.
- Output the changes made on containers by Ansible.
To manage a single container, you need to know two things:
- At which step during the overcloud deployment was the container deployed.
- The name of the generated JSON file containing the container configuration.
The following is an example of a playbook to manage HAproxy
container at step 1
which overrides the image setting:
- hosts: localhost become: true tasks: - name: Manage step_1 containers using tripleo-ansible block: - name: "Manage HAproxy container at step 1 with tripleo-ansible" include_role: name: tripleo_container_manage vars: tripleo_container_manage_config_patterns: 'haproxy.json' tripleo_container_manage_config: "/var/lib/tripleo-config/container-startup-config/step_1" tripleo_container_manage_config_id: "tripleo_step1" tripleo_container_manage_clean_orphans: false tripleo_container_manage_config_overrides: haproxy: image: quay.io/tripleomastercentos9/centos-binary-haproxy:hotfix
If Ansible is run in check mode
, no container is removed or created, however at the end of the playbook run a list of commands is displayed to show the possible outcome of the playbook. This is useful for debugging purposes.
$ ansible-playbook haproxy.yaml --check
Adding the diff mode
will show the changes that would have been made on containers by Ansible.
$ ansible-playbook haproxy.yaml --check --diff
The tripleo_container_manage_clean_orphans
parameter is optional. It can be set to false meaning orphaned containers, with a specific config_id
, will not be removed. It can be used to manage a single container without impacting other running containers with same config_id
.
The tripleo_container_manage_config_overrides
parameter is optional and can be used to override a specific container attribute, for example the image or the container user. The parameter creates dictionary with container name and the parameters to override. These parameters have to exist and they define the container configuration in TripleO Heat Templates.
Note the dictionary does not update the overrides in the JSON file so if an update or upgrade is executed, the container will be re-configured with the configuration in the JSON file.