이 콘텐츠는 선택한 언어로 제공되지 않습니다.
Chapter 16. Converting your existing deployment to use TLS
You can configure your existing overcloud and undercloud endpoints to use TLS encryption. This approach uses novajoin
to integrate your deployment with Red Hat Identity Management (IdM), allowing access to DNS, Kerberos, and certmonger. Each overcloud node uses a certmonger client to retrieve certificates for each service.
For more information on TLS, see the Security and Hardening Guide.
16.1. Requirements
- You must have an existing IdM deployment, and it must also supply DNS services to the OpenStack deployment.
- The existing deployment must use FQDNs for public endpoints. Default configurations might use IP address-based endpoints, and will consequently generate IP address-based certificates; these must be changed to FQDNs before proceeding with these steps.
The overcloud and undercloud services will be unavailable for the duration of this procedure.
16.2. Reviewing your endpoints
By default, your existing Red Hat OpenStack Platform overcloud does not encrypt certain endpoints with TLS. For example, this output includes URLs that use http
instead of https
; these are not encrypted:
+----------------------------------+-----------+--------------+--------------+---------+-----------+--------------------------------------------------------+ | ID | Region | Service Name | Service Type | Enabled | Interface | URL | +----------------------------------+-----------+--------------+--------------+---------+-----------+--------------------------------------------------------+ | 0ad11e943e1f4ff988650cfba57b4031 | regionOne | nova | compute | True | internal | http://172.16.2.17:8774/v2.1 | | 1413eb9ef38a45b8bee1bee1b0dfe744 | regionOne | swift | object-store | True | public | https://overcloud.lab.local:13808/v1/AUTH_%(tenant_id)s | | 1a54f13f212044b0a20468861cd06f85 | regionOne | neutron | network | True | public | https://overcloud.lab.local:13696 | | 3477a3a052d2445697bb6642a8c26a91 | regionOne | placement | placement | True | internal | http://172.16.2.17:8778/placement | | 3f56445c0dd14721ac830d6afb2c2cd4 | regionOne | nova | compute | True | admin | http://172.16.2.17:8774/v2.1 | | 425b1773a55c4245bcbe3d051772ebba | regionOne | glance | image | True | internal | http://172.16.2.17:9292 | | 57cf09fa33ed446f8736d4228bdfa881 | regionOne | placement | placement | True | public | https://overcloud.lab.local:13778/placement | | 58600f3751e54f7e9d0a50ba618e4c54 | regionOne | glance | image | True | public | https://overcloud.lab.local:13292 | | 5c52f273c3284b068f2dc885c77174ca | regionOne | neutron | network | True | internal | http://172.16.2.17:9696 | | 8792a4dd8bbb456d9dea4643e57c43dc | regionOne | nova | compute | True | public | https://overcloud.lab.local:13774/v2.1 | | 94bbea97580a4c4b844478aad5a85e84 | regionOne | keystone | identity | True | public | https://overcloud.lab.local:13000 | | acbf11b5c76d44198af49e3b78ffedcd | regionOne | swift | object-store | True | internal | http://172.16.1.9:8080/v1/AUTH_%(tenant_id)s | | d4a1344f02a74f7ab0a50c5a7c13ca5c | regionOne | keystone | identity | True | internal | http://172.16.2.17:5000 | | d86c241dc97642419ddc12533447d73d | regionOne | placement | placement | True | admin | http://172.16.2.17:8778/placement | | de7d6c34533e4298a2752852427a7030 | regionOne | glance | image | True | admin | http://172.16.2.17:9292 | | e82086062ebd4d4b9e03c7f1544bdd3b | regionOne | swift | object-store | True | admin | http://172.16.1.9:8080 | | f8134cd9746247bca6a06389b563c743 | regionOne | keystone | identity | True | admin | http://192.168.24.6:35357 | | fe29177bd29545ca8fdc0c777a7cf03f | regionOne | neutron | network | True | admin | http://172.16.2.17:9696 | +----------------------------------+-----------+--------------+--------------+---------+-----------+--------------------------------------------------------+
The following sections explain how to encrypt these endpoints using TLS.
16.3. Apply workaround for known issue
There is currently a known issue for TLS Everywhere in-place upgrades, where overcloud nodes are consequently unable to enroll in IdM. As a workaround, remove /etc/ipa/ca.crt/
from all overcloud nodes before running the overcloud deploy. For more information, see https://bugzilla.redhat.com/show_bug.cgi?id=1732564.
For example, the following script is one way of applying the workaround. You might need to amend this to suit your deployment.
[stack@undercloud-0 ~]$ vi rm-ca.crt-dir.sh #!/bin/bash source /home/stack/stackrc NODES=$(openstack server list -f value -c Networks|sed s/ctlplane=//g) for NODE in $NODES do ssh heat-admin@$NODE sudo rm -rf /etc/ipa/ca.crt/ Done [stack@undercloud-0 ~]$ bash rm-ca.crt-dir.sh
16.4. Configuring endpoints to use TLS
This section explains how to enable TLS endpoint encryption for an existing deployment, and then how to check that the endpoints have been correctly configured.
When enabling TLS everywhere, there are different upgrade paths available, depending on how your domains are structured. These examples use sample domain names to describe the upgrade paths:
-
Reuse the existing public endpoint certificates, and enable TLS everywhere on the
internal
andadmin
endpoints where the overcloud domain (lab.local
) matches the IdM domain (lab.local
). -
Allow IdM to issue new public endpoints certificates, and enable TLS everywhere on the
internal
andadmin
endpoints where the overcloud domain (lab.local
) matches the IdM domain (lab.local
). -
Reuse existing public endpoint certificates, and enable TLS everywhere on the
internal
andadmin
endpoints where the overcloud domain (site1.lab.local
) is a subdomain of the IdM domain (lab.local
). -
Allow IdM to issue new public endpoints certificates, and enable TLS everywhere on the
internal
andadmin
endpoints where the overcloud domain (site1.lab.local
) is a subdomain of the IdM domain (lab.local
).
The following procedures in this section explain how to configure this integration using the various combinations described above.
16.4.1. Configuring undercloud integration for deployments using the same domain as IdM
This procedure describes how to configure undercloud integration for deployments that use the same domain as IdM.
Red Hat OpenStack Platform uses novajoin
to integrate with Red Hat Identity Management (IdM), which then issues and manages encryption certificates. In this procedure, you register the undercloud with IdM, generate a token, enable the token in the undercloud configuration, then re-run the undercloud and overcloud deployment scripts. For example:
Install
python-novajoin
for integration with IdM:[stack@undercloud-0 ~]$ sudo yum install python-novajoin
Run the
novajoin
configuration script and supply the configuration details for your IdM deployment. For example:[stack@undercloud-0 ~]$ sudo novajoin-ipa-setup --principal admin --password ComplexRedactedPassword \ --server ipa.lab.local --realm lab.local --domain lab.local \ --hostname undercloud-0.lab.local --precreate ... 0Uvua6NyIWVkfCSTOmwbdAobsqGH2GONRJrW24MoQ4wg
This output includes a one time password (OTP) for IdM, which will be a different value for your deployment.
Configure the undercloud to use
novajoin
, add the one-time password (OTP), use the IdM IP address for DNS, and describe the overcloud domain. You will need to adjust this example for your deployment:[stack@undercloud ~]$ vi undercloud.conf ... enable_novajoin = true ipa_otp = 0Uvua6NyIWVkfCSTOmwbdAobsqGH2GONRJrW24MoQ4wg undercloud_hostname = undercloud-0.lab.local undercloud_nameservers = X.X.X.X overcloud_domain_name = lab.local ...
Install the
novajoin
services in the undercloud:[stack@undercloud ~]$ openstack undercloud install
Add the overcloud IP address to DNS. You will need to amend this example to suit your deployment:
Note: Check the overcloud’s
network-environment.yaml
, and choose a VIP within each network’s range.[root@ipa ~]$ ipa dnsrecord-add lab.local overcloud --a-rec=10.0.0.101 [root@ipa ~]# ipa dnszone-add ctlplane.lab.local [root@ipa ~]# ipa dnsrecord-add ctlplane.lab.local overcloud --a-rec 192.168.24.101 [root@ipa ~]# ipa dnszone-add internalapi.lab.local [root@ipa ~]# ipa dnsrecord-add internalapi.lab.local overcloud --a-rec 172.17.1.101 [root@ipa ~]# ipa dnszone-add storage.lab.local [root@ipa ~]# ipa dnsrecord-add storage.lab.local overcloud --a-rec 172.17.3.101 [root@ipa ~]# ipa dnszone-add storagemgmt.lab.local [root@ipa ~]# ipa dnsrecord-add storagemgmt.lab.local overcloud --a-rec 172.17.4.101
Create a
public_vip.yaml
mapping for all the endpoints:Parameter_defaults: PublicVirtualFixedIPs: [{'ip_address':'10.0.0.101'}] ControlFixedIPs: [{'ip_address':'192.168.24.101'}] InternalApiVirtualFixedIPs: [{'ip_address':'172.17.1.101'}] StorageVirtualFixedIPs: [{'ip_address':'172.17.3.101'}] StorageMgmtVirtualFixedIPs: [{'ip_address':'172.17.4.101'}] RedisVirtualFixedIPs: [{'ip_address':'172.17.1.102'}]
16.4.2. Configuring overcloud integration for deployments that use the same domain as IdM, and retain the existing public endpoint certificates
Make sure the following parameters exist in your
openstack overcloud deploy
command (with valid settings) and then re-run the deployment command:- ` --ntp-server` - If not already set, specify the NTP server to suit your environment. The IdM server should be running ntp.
-
cloud-names.yaml
- Contains the FQDNs (not IPs) from the initial deployment command. -
enable-tls.yaml
- Contains the new overcloud certificate. For an example, see https://github.com/openstack/tripleo-heat-templates/blob/master/environments/ssl/enable-tls.yaml. -
public_vip.yaml
- The maps the endpoints to a specific ip so dns can match. -
enable-internal-tls.yaml
- Enables TLS for internal endpoints. -
tls-everywhere-endpoints-dns.yaml
- Configures TLS endpoints using DNS names. You can review the contents of this file to check the configuration scope. -
haproxy-internal-tls-certmonger.yaml
- certmonger will manage the internal certs in haproxy. inject-trust-anchor.yaml
- Adds the root certificate authority. This is only needed when the certificates rely on a CA chain that is not already part of the common set used by default; for example, when using self-signed.For example:
[ stack@undercloud ~]$ openstack overcloud deploy \ ... --ntp-server 10.13.57.78 \ -e /home/stack/cloud-names.yaml \ -e /home/stack/enable-tls.yaml \ -e /home/stack/public_vip.yaml \ -e <tripleo-heat-templates>/environments/ssl/enable-internal-tls.yaml \ -e <tripleo-heat-templates>/environments/ssl/tls-everywhere-endpoints-dns.yaml \ -e <tripleo-heat-templates>/environments/services/haproxy-internal-tls-certmonger.yaml \ -e /home/stack/inject-trust-anchor.yaml ...
NoteExamples of these environment files can be found here: https://github.com/openstack/tripleo-heat-templates/tree/master/environments/ssl.
16.4.3. Configuring overcloud integration for deployments that use the same domain as IdM, and replace the existing public endpoint certificates with an IdM generated certificate
Make sure the following parameters exist in your
openstack overcloud deploy
command (with valid settings) and then re-run the deployment command:- ` --ntp-server` - If not already set, specify the NTP server to suit your environment. The IdM server should be running ntp.
-
cloud-names.yaml
- Contains the FQDNs (not IPs) from the initial deployment command. -
enable-tls.yaml
- Contains the new overcloud certificate. For an example, see https://github.com/openstack/tripleo-heat-templates/blob/master/environments/ssl/enable-tls.yaml. -
public_vip.yaml
- The maps the endpoints to a specific ip so dns can match. -
enable-internal-tls.yaml
- Enables TLS for internal endpoints. -
tls-everywhere-endpoints-dns.yaml
- Configures TLS endpoints using DNS names. You can review the contents of this file to check the configuration scope. -
haproxy-public-tls-certmonger.yaml
- certmonger will manage the internal and public certs in haproxy. inject-trust-anchor.yaml
- Adds the root certificate authority. This is only needed when the certificates rely on a CA chain that is not already part of the common set used by default; for example, when using self-signed.For example:
[ stack@undercloud ~]$ openstack overcloud deploy \ ... --ntp-server 10.13.57.78 \ -e /home/stack/cloud-names.yaml \ -e /home/stack/enable-tls.yaml \ -e /home/stack/public_vip.yaml \ -e <tripleo-heat-templates>/environments/ssl/enable-internal-tls.yaml \ -e <tripleo-heat-templates>/environments/ssl/tls-everywhere-endpoints-dns.yaml \ -e <tripleo-heat-templates>/environments/services/haproxy-public-tls-certmonger.yaml \ -e /home/stack/inject-trust-anchor.yaml ...
NoteExamples of these environment files can be found at https://github.com/openstack/tripleo-heat-templates/tree/master/environments/ssl.
The template enable-internal-tls.j2.yaml
is referenced as enable-internal-tls.yaml
in the overcloud deploy command.
In addition, the old public endpoint certificates in enable-tls.yaml
will be replaced by certmonger with haproxy-public-tls-certmonger.yaml
, however, this file must still be referenced in the upgrade process.
16.4.4. Configuring undercloud integration for deployments that use an IdM subdomain
This procedure explains how to configure undercloud integration for deployments that use an IdM subdomain.
Red Hat OpenStack Platform uses novajoin
to integrate with Red Hat Identity Management (IdM), which then issues and manages encryption certificates. In this procedure, you register the undercloud with IdM, generate a token, enable the token in the undercloud configuration, then re-run the undercloud and overcloud deployment scripts. For example:
Install
python-novajoin
for integration with IdM:[stack@undercloud-0 ~]$
Run the
novajoin
configuration script and supply the configuration details for your IdM deployment. For example:[stack@undercloud-0 ~]$ sudo novajoin-ipa-setup --principal admin --password ComplexRedactedPassword \ --server ipa.lab.local --realm lab.local --domain lab.local \ --hostname undercloud-0.site1.lab.local --precreate ... 0Uvua6NyIWVkfCSTOmwbdAobsqGH2GONRJrW24MoQ4wg
This output includes a one time password (OTP) for IdM, which will be a different value for your deployment.
Configure the undercloud to use
novajoin
, and add the OTP, IdM IP for DNS and NTP, and overcloud domain:[stack@undercloud ~]$ vi undercloud.conf … [DEFAULT] undercloud_ntp_servers=X.X.X.X hieradata_override = /home/stack/hiera_override.yaml enable_novajoin = true ipa_otp = 0Uvua6NyIWVkfCSTOmwbdAobsqGH2GONRJrW24MoQ4wg undercloud_hostname = undercloud-0.site1.lab.local undercloud_nameservers = X.X.X.X overcloud_domain_name = site1.lab.local ...
Configure the undercloud to use
novajoin
, and add the OTP, IdM IP for DNS, and overcloud domain:[stack@undercloud-0 ~]$ vi hiera_override.yaml nova::metadata::novajoin::api::ipa_domain: site1.lab.local ...
Install the
novajoin
services in the undercloud:[stack@undercloud ~]$ openstack undercloud install
Add the overcloud IP address to DNS. You will need to amend this example to suit your deployment:
Note: Check the overcloud’s
network-environment.yaml
, and choose a VIP within each network’s range.[root@ipa ~]$ ipa dnsrecord-add site1.lab.local overcloud --a-rec=10.0.0.101 [root@ipa ~]# ipa dnszone-add site1.ctlplane.lab.local [root@ipa ~]# ipa dnsrecord-add site1.ctlplane.lab.local overcloud --a-rec 192.168.24.101 [root@ipa ~]# ipa dnszone-add site1.internalapi.lab.local [root@ipa ~]# ipa dnsrecord-add site1.internalapi.lab.local overcloud --a-rec 172.17.1.101 [root@ipa ~]# ipa dnszone-add site1.storage.lab.local [root@ipa ~]# ipa dnsrecord-add site1.storage.lab.local overcloud --a-rec 172.17.3.101 [root@ipa ~]# ipa dnszone-add site1.storagemgmt.lab.local [root@ipa ~]# ipa dnsrecord-add site1.storagemgmt.lab.local overcloud --a-rec 172.17.4.101
Create a
public_vip.yaml
mapping for each of the endpoints. For example:Parameter_defaults: PublicVirtualFixedIPs: [{'ip_address':'10.0.0.101'}] ControlFixedIPs: [{'ip_address':'192.168.24.101'}] InternalApiVirtualFixedIPs: [{'ip_address':'172.17.1.101'}] StorageVirtualFixedIPs: [{'ip_address':'172.17.3.101'}] StorageMgmtVirtualFixedIPs: [{'ip_address':'172.17.4.101'}] RedisVirtualFixedIPs: [{'ip_address':'172.17.1.102'}]
Create the
extras.yaml
mapping for each of the endpoints. For example:parameter_defaults: MakeHomeDir: True IdMNoNtpSetup: false IdMDomain: redhat.local DnsSearchDomains: "site1.redhat.local,redhat.local"
16.4.5. Configuring undercloud integration for deployments that use an IdM subdomain, and retain the existing public endpoint certificates
This procedure explains how to configure undercloud integration for deployments that use an IdM subdomain, and still retain the existing public endpoint certificates.
Make sure the following parameters exist in your
openstack overcloud deploy
command (with valid settings) and then re-run the deployment command:- ` --ntp-server` - If not already set, specify the NTP server to suit your environment. The IdM server should be running ntp.
-
cloud-names.yaml
- Contains the FQDNs (not IPs) from the initial deployment command. -
enable-tls.yaml
- Contains the new overcloud certificate. For an example, see https://github.com/openstack/tripleo-heat-templates/blob/master/environments/ssl/enable-tls.yaml. -
public_vip.yaml
- Contains endpoint maps to a specific ip so dns can match. - `extras.yaml ` - Contains settings for pam to make home directorys on login, no ntp setup, the base IdM domain, and the dns search for resolv.conf.
-
enable-internal-tls.yaml
- Enables TLS for internal endpoints. -
tls-everywhere-endpoints-dns.yaml
- Configures TLS endpoints using DNS names. You can review the contents of this file to check the configuration scope. -
haproxy-internal-tls-certmonger.yaml
- certmonger will manage the internal certs in haproxy. inject-trust-anchor.yaml
- Adds the root certificate authority. This is only needed when the certificates rely on a CA chain that is not already part of the common set used by default; for example, when using self-signed.For example:
[ stack@undercloud ~]$ openstack overcloud deploy \ ... --ntp-server 10.13.57.78 \ -e /home/stack/cloud-names.yaml \ -e /home/stack/enable-tls.yaml \ -e /home/stack/public_vip.yaml \ -e /home/stack/extras.yaml \ -e <tripleo-heat-templates>/environments/ssl/enable-internal-tls.yaml \ -e <tripleo-heat-templates>/environments/ssl/tls-everywhere-endpoints-dns.yaml \ -e <tripleo-heat-templates>/environments/services/haproxy-internal-tls-certmonger.yaml \ -e /home/stack/inject-trust-anchor.yaml ...
NoteExamples of these environment files can be found here: https://github.com/openstack/tripleo-heat-templates/tree/master/environments/ssl.
16.4.6. Configuring undercloud integration for deployments that use an IdM subdomain, and replace the existing public endpoint certificates with an IdM generated certificate
This procedure explains how to configure undercloud integration for deployments that use an IdM subdomain, and how to replace the existing public endpoint certificates with an IdM generated certificate.
Make sure the following parameters exist in your
openstack overcloud deploy
command (with valid settings) and then re-run the deployment command:- ` --ntp-server` - If not already set, specify the NTP server to suit your environment. The IdM server should be running ntp.
-
cloud-names.yaml
- Contains the FQDNs (not IPs) from the initial deployment command. -
enable-tls.yaml
- Contains the new overcloud certificate. For an example, see https://github.com/openstack/tripleo-heat-templates/blob/master/environments/ssl/enable-tls.yaml. -
public_vip.yaml
- The maps the endpoints to a specific ip so dns can match. - `extras.yaml ` - Contains settings for pam to make home directorys on login, no ntp setup, the base IdM domain, and the dns search for resolv.conf.
-
enable-internal-tls.yaml
- Enables TLS for internal endpoints. -
tls-everywhere-endpoints-dns.yaml
- Configures TLS endpoints using DNS names. You can review the contents of this file to check the configuration scope. -
haproxy-public-tls-certmonger.yaml
- certmonger will manage the internal and public certs in haproxy. inject-trust-anchor.yaml
- Adds the root certificate authority. This is only needed when the certificates rely on a CA chain that is not already part of the common set used by default; for example, when using self-signed.For example:
[ stack@undercloud ~]$ openstack overcloud deploy \ ... --ntp-server 10.13.57.78 \ -e /home/stack/cloud-names.yaml \ -e /home/stack/enable-tls.yaml \ -e /home/stack/public_vip.yaml \ -e /home/stack/extras.yaml \ -e <tripleo-heat-templates>/environments/ssl/enable-internal-tls.yaml \ -e <tripleo-heat-templates>/environments/ssl/tls-everywhere-endpoints-dns.yaml \ -e <tripleo-heat-templates>/environments/services/haproxy-public-tls-certmonger.yaml \ -e /home/stack/inject-trust-anchor.yaml ...
NoteExamples of these environment files can be found here: https://github.com/openstack/tripleo-heat-templates/tree/master/environments/ssl.
In this example, the template enable-internal-tls.j2.yaml
is referenced as enable-internal-tls.yaml
in the overcloud deploy
command. In addition, the old public endpoint certificates in enable-tls.yaml
will be replaced by certmonger using haproxy-public-tls-certmonger.yaml
, however, this file must still be referenced in the upgrade process.
16.5. Checking TLS encryption
Once the overcloud re-deployment has completed, check that all endpoints are now encrypted with TLS. In this example, all endpoints are configured to use https
, indicating that they are using TLS encryption:
+----------------------------------+-----------+--------------+----------------+---------+-----------+---------------------------------------------------------------+ | ID | Region | Service Name | Service Type | Enabled | Interface | URL | +----------------------------------+-----------+--------------+----------------+---------+-----------+---------------------------------------------------------------+ | 0fee4efdc4ae4310b6a139a25d9c0d9c | regionOne | neutron | network | True | public | https://overcloud.lab.local:13696 | | 220558ab1d2445139952425961a0c89a | regionOne | glance | image | True | public | https://overcloud.lab.local:13292 | | 24d966109ffa419da850da946f19c4ca | regionOne | placement | placement | True | admin | https://overcloud.internalapi.lab.local:8778/placement | | 27ac9e0d22804ee5bd3cd8c0323db49c | regionOne | nova | compute | True | internal | https://overcloud.internalapi.lab.local:8774/v2.1 | | 31d376853bd241c2ba1a27912fc896c6 | regionOne | swift | object-store | True | admin | https://overcloud.storage.lab.local:8080 | | 350806234c784332bfb8615e721057e3 | regionOne | nova | compute | True | admin | https://overcloud.internalapi.lab.local:8774/v2.1 | | 49c312f4db6748429d27c60164779302 | regionOne | keystone | identity | True | public | https://overcloud.lab.local:13000 | | 4e535265c35e486e97bb5a8bc77708b6 | regionOne | nova | compute | True | public | https://overcloud.lab.local:13774/v2.1 | | 5e93dd46b45f40fe8d91d3a5d6e847d3 | regionOne | keystone | identity | True | admin | https://overcloud.ctlplane.lab.local:35357 | | 6561984a90c742a988bf3d0acf80d1b6 | regionOne | swift | object-store | True | public | https://overcloud.lab.local:13808/v1/AUTH_%(tenant_id)s | | 76b8aad0bdda4313a02e4342e6a19fd6 | regionOne | placement | placement | True | public | https://overcloud.lab.local:13778/placement | | 96b004d5217c4d87a38cb780607bf9fb | regionOne | placement | placement | True | internal | https://overcloud.internalapi.lab.local:8778/placement | | 98489b4b107f4da596262b712c3fe883 | regionOne | glance | image | True | internal | https://overcloud.internalapi.lab.local:9292 | | bb7ab36f30b14b549178ef06ec74ff84 | regionOne | glance | image | True | admin | https://overcloud.internalapi.lab.local:9292 | | c1547f7bf9a14e9e85eaaaeea26413b7 | regionOne | neutron | network | True | admin | https://overcloud.internalapi.lab.local:9696 | | ca66f499ec544f42838eb78a515d9f1e | regionOne | keystone | identity | True | internal | https://overcloud.internalapi.lab.local:5000 | | df0181358c07431390bc66822176281d | regionOne | swift | object-store | True | internal | https://overcloud.storage.lab.local:8080/v1/AUTH_%(tenant_id)s | | e420350ef856460991c3edbfbae917c1 | regionOne | neutron | network | True | internal | https://overcloud.internalapi.lab.local:9696 | +----------------------------------+-----------+--------------+----------------+---------+-----------+---------------------------------------------------------------+