1.3. 检查收集的数据
您可以使用 OpenShift (RHOSO) must-gather 工具中的 Red Hat OpenStack Services 获取 collection-scripts 中定义的 kubernetes 资源,以及与 CoreOS 节点和 EDPM 相关联的 sos-reports。当 must-gather 执行结束时,它会创建一个包含所有收集的资源的目录,例如:
-
全局资源 :您可以使用它们获取有关 Red Hat OpenShift Container Platform (RHOCP)集群和 RHOSO 部署的资源的一些上下文。这些资源包括
crds、apiservices、csvs、packagemanifests、webhook和网络信息,如nncp、nnce、IPAddressPool等。 - Namespaced 资源:您需要它们获取 RHOSO 集群的状态并排除任何问题。
-
sos-reports :您可以从作为 RHOCP 集群一部分的
CoreOS节点收集它们,以及属于集群的 EDPM 节点。要连接到 EDPM 节点的信息由OpenStackDataplaneNodeSetsCR 检索,生成的 sos-report 从远程节点检索,然后在当前的must-gather目录中下载。 -
OpenStack Ctlplane Services:您可以通过
openstack-cli运行命令,以检查 RHOSO 集群中生成的相关资源,如端点列表、网络、子网、注册的服务等。
以下是 must-gather 输出的示例:
+-----------------------------------+ | . | +-----------------------------+ | ├── apiservices | | ctlplane/neutron/ | | ├── crd | | ├── agent_list | | ├── csv | (control plane resources) | ├── extension_list | | ├── ctlplane |------------------------------------| ├── floating_ip_list | | │ ├── neutron | | ├── network_list | | │ ├── nova |----------------- | ├── port_list | | │ └── placement | | | ├── router_list | | ├── dbs | +---------------------------+ | ├── security_group_list | | ├── namespaces | | namespaces/openstack/ | | └── subnet_list | | │ ├── cert-manager | | ├── all_resources.log | +-----------------------------+ | │ ├── openshift-machine-api | | ├── buildconfig |----------------------------------- | │ ├── openshift-nmstate | | ├── configmaps | | | │ ├── openstack | | ├── cronjobs | +--------------------------------------------------------------------+ | │ └── openstack-operators | | ├── crs | | namespaces/openstack/secrets/glance/ | | ├── network | | ├── daemonset | | ├── cert-glance-default-public-route.yaml | | │ ├── ipaddresspools | | ├── deployments | | ├── glance-config-data.yaml | | │ ├── nnce | | ├── events.log | | ├── glance-config-data.yaml-00-config.conf | | │ └── nncp | | ├── installplans | | ├── glance-default-single-config-data.yaml | | ├── nodes | | ├── jobs | | ├── glance-default-single-config-data.yaml-00-config.conf | | ├── sos-reports | | ├── nad.log | | ├── glance-default-single-config-data.yaml-10-glance-httpd.conf | | │ ├── _all_nodes | | ├── pods | | ├── glance-default-single-config-data.yaml-httpd.conf | | │ ├── barbican | | ├── pvc.log | | ├── glance-default-single-config-data.yaml-ssl.conf | | │ ├── ceilometer | | ├── replicaset | | └── glance-scripts.yaml | | │ ├── glance | | ├── routes | +--------------------------------------------------------------------+ | │ ├── keystone | | ├── secrets | | | │ ├── neutron | | ├── services | +--------------------------------------------------------------------+ | │ ├── nova | | ├── statefulsets | | Note: if DO_NOT_MASK is passed in CI, secrets are dumped without | | │ ├── ovn | | └── subscriptions | | hiding any sensitive information. | | │ ├── ovs | +---------------------------+ +--------------------------------------------------------------------+ | │ ├── placement | | │ └── swift | | └── webhooks | | ├── mutating | | └── validating | +-----------------------------------+
+-----------------------------------+
| . | +-----------------------------+
| ├── apiservices | | ctlplane/neutron/ |
| ├── crd | | ├── agent_list |
| ├── csv | (control plane resources) | ├── extension_list |
| ├── ctlplane |------------------------------------| ├── floating_ip_list |
| │ ├── neutron | | ├── network_list |
| │ ├── nova |----------------- | ├── port_list |
| │ └── placement | | | ├── router_list |
| ├── dbs | +---------------------------+ | ├── security_group_list |
| ├── namespaces | | namespaces/openstack/ | | └── subnet_list |
| │ ├── cert-manager | | ├── all_resources.log | +-----------------------------+
| │ ├── openshift-machine-api | | ├── buildconfig |-----------------------------------
| │ ├── openshift-nmstate | | ├── configmaps | |
| │ ├── openstack | | ├── cronjobs | +--------------------------------------------------------------------+
| │ └── openstack-operators | | ├── crs | | namespaces/openstack/secrets/glance/ |
| ├── network | | ├── daemonset | | ├── cert-glance-default-public-route.yaml |
| │ ├── ipaddresspools | | ├── deployments | | ├── glance-config-data.yaml |
| │ ├── nnce | | ├── events.log | | ├── glance-config-data.yaml-00-config.conf |
| │ └── nncp | | ├── installplans | | ├── glance-default-single-config-data.yaml |
| ├── nodes | | ├── jobs | | ├── glance-default-single-config-data.yaml-00-config.conf |
| ├── sos-reports | | ├── nad.log | | ├── glance-default-single-config-data.yaml-10-glance-httpd.conf |
| │ ├── _all_nodes | | ├── pods | | ├── glance-default-single-config-data.yaml-httpd.conf |
| │ ├── barbican | | ├── pvc.log | | ├── glance-default-single-config-data.yaml-ssl.conf |
| │ ├── ceilometer | | ├── replicaset | | └── glance-scripts.yaml |
| │ ├── glance | | ├── routes | +--------------------------------------------------------------------+
| │ ├── keystone | | ├── secrets | |
| │ ├── neutron | | ├── services | +--------------------------------------------------------------------+
| │ ├── nova | | ├── statefulsets | | Note: if DO_NOT_MASK is passed in CI, secrets are dumped without |
| │ ├── ovn | | └── subscriptions | | hiding any sensitive information. |
| │ ├── ovs | +---------------------------+ +--------------------------------------------------------------------+
| │ ├── placement |
| │ └── swift |
| └── webhooks |
| ├── mutating |
| └── validating |
+-----------------------------------+
当您进行故障排除时,不仅检查和分析 Secret 和服务配置文件,还要检查和分析与每个服务和 Pod 日志关联的 CR。这些是可在 CR 和 Pod 目录中找到的命名空间资源。另外,还会收集每个命名空间的一些通用信息。您可以使用 must-gather 工具来检索:
- 为当前命名空间记录的事件。
- 网络附加定义.
- 附加到部署的 Pod 的 PVC。
-
all_resources.log,它以部署的资源的形式概述了命名空间。
+---------------------------+ | namespaces/openstack/ | ------------------------------------ | ├── buildconfig | | | ├── cronjobs | +--------------------------------------------------------+ | ├── crs | | namespaces/openstack/crs/ | | ├── daemonset | | ├── barbicanapis.barbican.openstack.org | | ├── deployments | | ├── barbicankeystonelisteners.barbican.openstack.org | | ├── events.log | | ├── barbicans.barbican.openstack.org | | ├── installplans | | ├── barbicanworkers.barbican.openstack.org | | ├── jobs | | ... | | ├── nad.log | | ... | | ├── pods | | ├── glanceapis.glance.openstack.org | | ├── all_resources.log | | └── glance-default-single.yaml | | ├── configmaps | | ├── glances.glance.openstack.org | | ├── pvc.log | | └── glance.yaml | | ├── replicaset | | ├── keystoneapis.keystone.openstack.org | | ├── routes | | ├── keystoneendpoints.keystone.openstack.org | | ├── secrets | | ├── keystoneservices.keystone.openstack.org | | ├── services | | ... | | ├── statefulsets | | ├── telemetries.telemetry.openstack.org | | └── subscriptions | | └── transporturls.rabbitmq.openstack.org | +---------------------------+ +--------------------------------------------------------+
+---------------------------+
| namespaces/openstack/ | ------------------------------------
| ├── buildconfig | |
| ├── cronjobs | +--------------------------------------------------------+
| ├── crs | | namespaces/openstack/crs/ |
| ├── daemonset | | ├── barbicanapis.barbican.openstack.org |
| ├── deployments | | ├── barbicankeystonelisteners.barbican.openstack.org |
| ├── events.log | | ├── barbicans.barbican.openstack.org |
| ├── installplans | | ├── barbicanworkers.barbican.openstack.org |
| ├── jobs | | ... |
| ├── nad.log | | ... |
| ├── pods | | ├── glanceapis.glance.openstack.org |
| ├── all_resources.log | | └── glance-default-single.yaml |
| ├── configmaps | | ├── glances.glance.openstack.org |
| ├── pvc.log | | └── glance.yaml |
| ├── replicaset | | ├── keystoneapis.keystone.openstack.org |
| ├── routes | | ├── keystoneendpoints.keystone.openstack.org |
| ├── secrets | | ├── keystoneservices.keystone.openstack.org |
| ├── services | | ... |
| ├── statefulsets | | ├── telemetries.telemetry.openstack.org |
| └── subscriptions | | └── transporturls.rabbitmq.openstack.org |
+---------------------------+ +--------------------------------------------------------+
如前一个模式中所述,相同的模式适用于 Pod 资源。您可以使用 must-gather 工具来检索每个 Pod 的描述和相关日志,包括在 Pod 处于 CrashLookBackoff 状态时。
+---------------------------+
| namespaces/openstack/ | ------------------------------------
| ├── buildconfig | |
| ├── cronjobs | +-----------------------------------------------------------+
| ├── crs | | namespaces/openstack/pods/glance-dbpurge-28500481-f4jk9 |
| ├── daemonset | | ├── glance-dbpurge-28500481-f4jk9-describe |
| ├── deployments | | └── logs |
| ├── events.log | | └── glance-dbpurge.log |
| ├── installplans | | namespaces/openstack/pods/glance-default-single-0 |
| ├── jobs | | ├── glance-default-single-0-describe |
| ├── nad.log | | └── logs |
| ├── pods | | ├── glance-api.log |
| ├── all_resources.log | | ├── glance-httpd.log |
| ├── configmaps | | └── glance-log.log |
| ├── pvc.log | | namespaces/openstack/pods/glance-default-single-1 |
| ├── replicaset | | ├── glance-default-single-1-describe |
| ├── routes | | └── logs |
| ├── secrets | | ├── glance-api.log |
| ├── services | | ├── glance-httpd.log |
| ├── statefulsets | | └── glance-log.log |
| └── subscriptions | | namespaces/openstack/pods/glance-default-single-2 |
+---------------------------+ | ├── glance-default-single-2-describe |
| └── logs |
| ├── glance-api.log |
| ├── glance-httpd.log |
| └── glance-log.log |
+-----------------------------------------------------------+
+---------------------------+
| namespaces/openstack/ | ------------------------------------
| ├── buildconfig | |
| ├── cronjobs | +-----------------------------------------------------------+
| ├── crs | | namespaces/openstack/pods/glance-dbpurge-28500481-f4jk9 |
| ├── daemonset | | ├── glance-dbpurge-28500481-f4jk9-describe |
| ├── deployments | | └── logs |
| ├── events.log | | └── glance-dbpurge.log |
| ├── installplans | | namespaces/openstack/pods/glance-default-single-0 |
| ├── jobs | | ├── glance-default-single-0-describe |
| ├── nad.log | | └── logs |
| ├── pods | | ├── glance-api.log |
| ├── all_resources.log | | ├── glance-httpd.log |
| ├── configmaps | | └── glance-log.log |
| ├── pvc.log | | namespaces/openstack/pods/glance-default-single-1 |
| ├── replicaset | | ├── glance-default-single-1-describe |
| ├── routes | | └── logs |
| ├── secrets | | ├── glance-api.log |
| ├── services | | ├── glance-httpd.log |
| ├── statefulsets | | └── glance-log.log |
| └── subscriptions | | namespaces/openstack/pods/glance-default-single-2 |
+---------------------------+ | ├── glance-default-single-2-describe |
| └── logs |
| ├── glance-api.log |
| ├── glance-httpd.log |
| └── glance-log.log |
+-----------------------------------------------------------+