7.11. 查看 Master 和节点日志
OpenShift Container Platform 使用 systemd-journald.service
作为节点和名为 master-logs
的脚本,为 master 收集调试的日志消息。
web 控制台中显示的行的数量在 5000 中被硬编码,且无法更改。要查看整个日志,可使用 CLI。
日志记录根据 Kubernetes 日志记录惯例使用五个日志消息,如下所示:
选项 | 描述 |
---|---|
0 | 仅限错误和警告 |
2 | 普通信息 |
4 | 调试级别信息 |
6 | API 级调试信息(请求/响应) |
8 | 正文级别 API 调试信息 |
您可以根据需要 更改独立于 master 或节点的日志级别。
查看节点日志
要查看节点系统的日志,请运行以下命令:
# journalctl -r -u <journal_name>
使用 -r
选项显示最新的条目。
查看 master 日志
要查看 master 组件的日志,请运行以下命令:
# /usr/local/bin/master-logs <component> <container>
例如:
# /usr/local/bin/master-logs controllers controllers # /usr/local/bin/master-logs api api # /usr/local/bin/master-logs etcd etcd
将 master 日志重定向到文件
要将 master 日志的输出重定向到文件,请运行以下命令:
master-logs api api 2> file
7.11.1. 配置日志记录级别
您可以通过在 master 或 /etc/sysconfig/atomic-openshift-node 文件中的 /etc/origin/master/master.env 文件中设置 DEBUG_LOGLEVEL
选项来控制哪些 INFO 信息。将日志配置为收集所有信息可能会导致大型日志被解释,并可能会占用过量空间。仅在需要调试集群时收集所有消息。
不论日志配置是什么,日志中会出现带有 FATAL、ERROR、WARNING 和一些 INFO 严重性的消息。
更改日志记录级别:
- 编辑 master 或 /etc/sysconfig/atomic-openshift-node 文件的 /etc/origin/master/master.env 文件。
在
DEBUG_LOGLEVEL
字段中,输入 Log Level Options 表中的值。例如:
DEBUG_LOGLEVEL=4
- 根据需要重启 master 或节点主机。请参阅重启 OpenShift Container Platform 服务。
重启后,所有新日志消息均符合新设置。旧的消息不会更改。
可以使用标准集群安装过程来设置默认日志级别。如需更多信息,请参阅 集群变量。
以下示例是不同日志级别重定向的主日志文件的摘录。已从这些示例中删除系统信息。
loglevel=2 的 master-logs api api 2> 文件
输出摘录
W1022 15:08:09.787705 1 server.go:79] Unable to keep dnsmasq up to date, 0.0.0.0:8053 must point to port 53 I1022 15:08:09.787894 1 logs.go:49] skydns: ready for queries on cluster.local. for tcp4://0.0.0.0:8053 [rcache 0] I1022 15:08:09.787913 1 logs.go:49] skydns: ready for queries on cluster.local. for udp4://0.0.0.0:8053 [rcache 0] I1022 15:08:09.889022 1 dns_server.go:63] DNS listening at 0.0.0.0:8053 I1022 15:08:09.893156 1 feature_gate.go:190] feature gates: map[AdvancedAuditing:true] I1022 15:08:09.893500 1 master.go:431] Starting OAuth2 API at /oauth I1022 15:08:09.914759 1 master.go:431] Starting OAuth2 API at /oauth I1022 15:08:09.942349 1 master.go:431] Starting OAuth2 API at /oauth W1022 15:08:09.977088 1 swagger.go:38] No API exists for predefined swagger description /oapi/v1 W1022 15:08:09.977176 1 swagger.go:38] No API exists for predefined swagger description /api/v1 [restful] 2018/10/22 15:08:09 log.go:33: [restful/swagger] listing is available at https://openshift.com:443/swaggerapi [restful] 2018/10/22 15:08:09 log.go:33: [restful/swagger] https://openshift.com:443/swaggerui/ is mapped to folder /swagger-ui/ I1022 15:08:10.231405 1 master.go:431] Starting OAuth2 API at /oauth W1022 15:08:10.259523 1 swagger.go:38] No API exists for predefined swagger description /oapi/v1 W1022 15:08:10.259555 1 swagger.go:38] No API exists for predefined swagger description /api/v1 I1022 15:08:23.895493 1 logs.go:49] http: TLS handshake error from 10.10.94.10:46322: EOF I1022 15:08:24.449577 1 crdregistration_controller.go:110] Starting crd-autoregister controller I1022 15:08:24.449916 1 controller_utils.go:1019] Waiting for caches to sync for crd-autoregister controller I1022 15:08:24.496147 1 logs.go:49] http: TLS handshake error from 127.0.0.1:39140: EOF I1022 15:08:24.821198 1 cache.go:39] Caches are synced for APIServiceRegistrationController controller I1022 15:08:24.833022 1 cache.go:39] Caches are synced for AvailableConditionController controller I1022 15:08:24.865087 1 controller.go:537] quota admission added evaluator for: { events} I1022 15:08:24.865393 1 logs.go:49] http: TLS handshake error from 127.0.0.1:39162: read tcp4 127.0.0.1:443->127.0.0.1:39162: read: connection reset by peer I1022 15:08:24.966917 1 controller_utils.go:1026] Caches are synced for crd-autoregister controller I1022 15:08:24.967961 1 autoregister_controller.go:136] Starting autoregister controller I1022 15:08:24.967977 1 cache.go:32] Waiting for caches to sync for autoregister controller I1022 15:08:25.015924 1 controller.go:537] quota admission added evaluator for: { serviceaccounts} I1022 15:08:25.077984 1 cache.go:39] Caches are synced for autoregister controller W1022 15:08:25.304265 1 lease_endpoint_reconciler.go:176] Resetting endpoints for master service "kubernetes" to [10.10.94.10] E1022 15:08:25.472536 1 memcache.go:153] couldn't get resource list for servicecatalog.k8s.io/v1beta1: the server could not find the requested resource E1022 15:08:25.550888 1 memcache.go:153] couldn't get resource list for servicecatalog.k8s.io/v1beta1: the server could not find the requested resource I1022 15:08:29.480691 1 healthz.go:72] /healthz/log check I1022 15:08:30.981999 1 controller.go:105] OpenAPI AggregationController: Processing item v1beta1.servicecatalog.k8s.io E1022 15:08:30.990914 1 controller.go:111] loading OpenAPI spec for "v1beta1.servicecatalog.k8s.io" failed with: OpenAPI spec does not exists I1022 15:08:30.990965 1 controller.go:119] OpenAPI AggregationController: action for item v1beta1.servicecatalog.k8s.io: Rate Limited Requeue. I1022 15:08:31.530473 1 trace.go:76] Trace[1253590531]: "Get /api/v1/namespaces/openshift-infra/serviceaccounts/serviceaccount-controller" (started: 2018-10-22 15:08:30.868387562 +0000 UTC m=+24.277041043) (total time: 661.981642ms): Trace[1253590531]: [661.903178ms] [661.89217ms] About to write a response I1022 15:08:31.531366 1 trace.go:76] Trace[83808472]: "Get /api/v1/namespaces/aws-sb/secrets/aws-servicebroker" (started: 2018-10-22 15:08:30.831296749 +0000 UTC m=+24.239950203) (total time: 700.049245ms):
loglevel=4 的 master-logs api api 2> 文件
输出摘录
I1022 15:08:09.746980 1 plugins.go:149] Loaded 1 admission controller(s) successfully in the following order: AlwaysDeny. I1022 15:08:09.747597 1 plugins.go:149] Loaded 1 admission controller(s) successfully in the following order: ResourceQuota. I1022 15:08:09.748038 1 plugins.go:149] Loaded 1 admission controller(s) successfully in the following order: openshift.io/ClusterResourceQuota. I1022 15:08:09.786771 1 start_master.go:458] Starting master on 0.0.0.0:443 (v3.10.45) I1022 15:08:09.786798 1 start_master.go:459] Public master address is https://openshift.com:443 I1022 15:08:09.786844 1 start_master.go:463] Using images from "registry.access.redhat.com/openshift3/ose-<component>:v3.10.45" W1022 15:08:09.787046 1 dns_server.go:37] Binding DNS on port 8053 instead of 53, which may not be resolvable from all clients W1022 15:08:09.787705 1 server.go:79] Unable to keep dnsmasq up to date, 0.0.0.0:8053 must point to port 53 I1022 15:08:09.787894 1 logs.go:49] skydns: ready for queries on cluster.local. for tcp4://0.0.0.0:8053 [rcache 0] I1022 15:08:09.787913 1 logs.go:49] skydns: ready for queries on cluster.local. for udp4://0.0.0.0:8053 [rcache 0] I1022 15:08:09.889022 1 dns_server.go:63] DNS listening at 0.0.0.0:8053 I1022 15:08:09.893156 1 feature_gate.go:190] feature gates: map[AdvancedAuditing:true] I1022 15:08:09.893500 1 master.go:431] Starting OAuth2 API at /oauth I1022 15:08:09.914759 1 master.go:431] Starting OAuth2 API at /oauth I1022 15:08:09.942349 1 master.go:431] Starting OAuth2 API at /oauth W1022 15:08:09.977088 1 swagger.go:38] No API exists for predefined swagger description /oapi/v1 W1022 15:08:09.977176 1 swagger.go:38] No API exists for predefined swagger description /api/v1 [restful] 2018/10/22 15:08:09 log.go:33: [restful/swagger] listing is available at https://openshift.com:443/swaggerapi [restful] 2018/10/22 15:08:09 log.go:33: [restful/swagger] https://openshift.com:443/swaggerui/ is mapped to folder /swagger-ui/ I1022 15:08:10.231405 1 master.go:431] Starting OAuth2 API at /oauth W1022 15:08:10.259523 1 swagger.go:38] No API exists for predefined swagger description /oapi/v1 W1022 15:08:10.259555 1 swagger.go:38] No API exists for predefined swagger description /api/v1 [restful] 2018/10/22 15:08:10 log.go:33: [restful/swagger] listing is available at https://openshift.com:443/swaggerapi [restful] 2018/10/22 15:08:10 log.go:33: [restful/swagger] https://openshift.com:443/swaggerui/ is mapped to folder /swagger-ui/ I1022 15:08:10.444303 1 master.go:431] Starting OAuth2 API at /oauth W1022 15:08:10.492409 1 swagger.go:38] No API exists for predefined swagger description /oapi/v1 W1022 15:08:10.492507 1 swagger.go:38] No API exists for predefined swagger description /api/v1 [restful] 2018/10/22 15:08:10 log.go:33: [restful/swagger] listing is available at https://openshift.com:443/swaggerapi [restful] 2018/10/22 15:08:10 log.go:33: [restful/swagger] https://openshift.com:443/swaggerui/ is mapped to folder /swagger-ui/ I1022 15:08:10.774824 1 master.go:431] Starting OAuth2 API at /oauth I1022 15:08:23.808685 1 logs.go:49] http: TLS handshake error from 10.128.0.11:39206: EOF I1022 15:08:23.815311 1 logs.go:49] http: TLS handshake error from 10.128.0.14:53054: EOF I1022 15:08:23.822286 1 customresource_discovery_controller.go:174] Starting DiscoveryController I1022 15:08:23.822349 1 naming_controller.go:276] Starting NamingConditionController I1022 15:08:23.822705 1 logs.go:49] http: TLS handshake error from 10.128.0.14:53056: EOF +24.277041043) (total time: 661.981642ms): Trace[1253590531]: [661.903178ms] [661.89217ms] About to write a response I1022 15:08:31.531366 1 trace.go:76] Trace[83808472]: "Get /api/v1/namespaces/aws-sb/secrets/aws-servicebroker" (started: 2018-10-22 15:08:30.831296749 +0000 UTC m=+24.239950203) (total time: 700.049245ms): Trace[83808472]: [700.049245ms] [700.04027ms] END I1022 15:08:31.531695 1 trace.go:76] Trace[1916801734]: "Get /api/v1/namespaces/aws-sb/secrets/aws-servicebroker" (started: 2018-10-22 15:08:31.031163449 +0000 UTC m=+24.439816907) (total time: 500.514208ms): Trace[1916801734]: [500.514208ms] [500.505008ms] END I1022 15:08:44.675371 1 healthz.go:72] /healthz/log check I1022 15:08:46.589759 1 controller.go:537] quota admission added evaluator for: { endpoints} I1022 15:08:46.621270 1 controller.go:537] quota admission added evaluator for: { endpoints} I1022 15:08:57.159494 1 healthz.go:72] /healthz/log check I1022 15:09:07.161315 1 healthz.go:72] /healthz/log check I1022 15:09:16.297982 1 trace.go:76] Trace[2001108522]: "GuaranteedUpdate etcd3: *core.Node" (started: 2018-10-22 15:09:15.139820419 +0000 UTC m=+68.548473981) (total time: 1.158128974s): Trace[2001108522]: [1.158012755s] [1.156496534s] Transaction committed I1022 15:09:16.298165 1 trace.go:76] Trace[1124283912]: "Patch /api/v1/nodes/master-0.com/status" (started: 2018-10-22 15:09:15.139695483 +0000 UTC m=+68.548348970) (total time: 1.158434318s): Trace[1124283912]: [1.158328853s] [1.15713683s] Object stored in database I1022 15:09:16.298761 1 trace.go:76] Trace[24963576]: "GuaranteedUpdate etcd3: *core.Node" (started: 2018-10-22 15:09:15.13159057 +0000 UTC m=+68.540244112) (total time: 1.167151224s): Trace[24963576]: [1.167106144s] [1.165570379s] Transaction committed I1022 15:09:16.298882 1 trace.go:76] Trace[222129183]: "Patch /api/v1/nodes/node-0.com/status" (started: 2018-10-22 15:09:15.131269234 +0000 UTC m=+68.539922722) (total time: 1.167595526s): Trace[222129183]: [1.167517296s] [1.166135605s] Object stored in database
loglevel=8 中的 master-logs api api 2> 文件
输出摘录
1022 15:11:58.829357 1 plugins.go:84] Registered admission plugin "NamespaceLifecycle" I1022 15:11:58.839967 1 plugins.go:84] Registered admission plugin "Initializers" I1022 15:11:58.839994 1 plugins.go:84] Registered admission plugin "ValidatingAdmissionWebhook" I1022 15:11:58.840012 1 plugins.go:84] Registered admission plugin "MutatingAdmissionWebhook" I1022 15:11:58.840025 1 plugins.go:84] Registered admission plugin "AlwaysAdmit" I1022 15:11:58.840082 1 plugins.go:84] Registered admission plugin "AlwaysPullImages" I1022 15:11:58.840105 1 plugins.go:84] Registered admission plugin "LimitPodHardAntiAffinityTopology" I1022 15:11:58.840126 1 plugins.go:84] Registered admission plugin "DefaultTolerationSeconds" I1022 15:11:58.840146 1 plugins.go:84] Registered admission plugin "AlwaysDeny" I1022 15:11:58.840176 1 plugins.go:84] Registered admission plugin "EventRateLimit" I1022 15:11:59.850825 1 feature_gate.go:190] feature gates: map[AdvancedAuditing:true] I1022 15:11:59.859108 1 register.go:154] Admission plugin AlwaysAdmit is not enabled. It will not be started. I1022 15:11:59.859284 1 plugins.go:149] Loaded 1 admission controller(s) successfully in the following order: AlwaysAdmit. I1022 15:11:59.859809 1 register.go:154] Admission plugin NamespaceAutoProvision is not enabled. It will not be started. I1022 15:11:59.859939 1 plugins.go:149] Loaded 1 admission controller(s) successfully in the following order: NamespaceAutoProvision. I1022 15:11:59.860594 1 register.go:154] Admission plugin NamespaceExists is not enabled. It will not be started. I1022 15:11:59.860778 1 plugins.go:149] Loaded 1 admission controller(s) successfully in the following order: NamespaceExists. I1022 15:11:59.863999 1 plugins.go:149] Loaded 1 admission controller(s) successfully in the following order: NamespaceLifecycle. I1022 15:11:59.864626 1 register.go:154] Admission plugin EventRateLimit is not enabled. It will not be started. I1022 15:11:59.864768 1 plugins.go:149] Loaded 1 admission controller(s) successfully in the following order: EventRateLimit. I1022 15:11:59.865259 1 register.go:154] Admission plugin ProjectRequestLimit is not enabled. It will not be started. I1022 15:11:59.865376 1 plugins.go:149] Loaded 1 admission controller(s) successfully in the following order: ProjectRequestLimit. I1022 15:11:59.866126 1 plugins.go:149] Loaded 1 admission controller(s) successfully in the following order: OriginNamespaceLifecycle. I1022 15:11:59.866709 1 register.go:154] Admission plugin openshift.io/RestrictSubjectBindings is not enabled. It will not be started. I1022 15:11:59.866761 1 plugins.go:149] Loaded 1 admission controller(s) successfully in the following order: openshift.io/RestrictSubjectBindings. I1022 15:11:59.867304 1 plugins.go:149] Loaded 1 admission controller(s) successfully in the following order: openshift.io/JenkinsBootstrapper. I1022 15:11:59.867823 1 plugins.go:149] Loaded 1 admission controller(s) successfully in the following order: openshift.io/BuildConfigSecretInjector. I1022 15:12:00.015273 1 master_config.go:476] Initializing cache sizes based on 0MB limit I1022 15:12:00.015896 1 master_config.go:539] Using the lease endpoint reconciler with TTL=15s and interval=10s I1022 15:12:00.018396 1 storage_factory.go:285] storing { apiServerIPInfo} in v1, reading as __internal from storagebackend.Config{Type:"etcd3", Prefix:"kubernetes.io", ServerList:[]string{"https://master-0.com:2379"}, KeyFile:"/etc/origin/master/master.etcd-client.key", CertFile:"/etc/origin/master/master.etcd-client.crt", CAFile:"/etc/origin/master/master.etcd-ca.crt", Quorum:true, Paging:true, DeserializationCacheSize:0, Codec:runtime.Codec(nil), Transformer:value.Transformer(nil), CompactionInterval:300000000000, CountMetricPollPeriod:60000000000} I1022 15:12:00.037710 1 storage_factory.go:285] storing { endpoints} in v1, reading as __internal from storagebackend.Config{Type:"etcd3", Prefix:"kubernetes.io", ServerList:[]string{"https://master-0.com:2379"}, KeyFile:"/etc/origin/master/master.etcd-client.key", CertFile:"/etc/origin/master/master.etcd-client.crt", CAFile:"/etc/origin/master/master.etcd-ca.crt", Quorum:true, Paging:true, DeserializationCacheSize:0, Codec:runtime.Codec(nil), Transformer:value.Transformer(nil), CompactionInterval:300000000000, CountMetricPollPeriod:60000000000} I1022 15:12:00.054112 1 compact.go:54] compactor already exists for endpoints [https://master-0.com:2379] I1022 15:12:00.054678 1 start_master.go:458] Starting master on 0.0.0.0:443 (v3.10.45) I1022 15:12:00.054755 1 start_master.go:459] Public master address is https://openshift.com:443 I1022 15:12:00.054837 1 start_master.go:463] Using images from "registry.access.redhat.com/openshift3/ose-<component>:v3.10.45" W1022 15:12:00.056957 1 dns_server.go:37] Binding DNS on port 8053 instead of 53, which may not be resolvable from all clients W1022 15:12:00.065497 1 server.go:79] Unable to keep dnsmasq up to date, 0.0.0.0:8053 must point to port 53 I1022 15:12:00.066061 1 logs.go:49] skydns: ready for queries on cluster.local. for tcp4://0.0.0.0:8053 [rcache 0] I1022 15:12:00.066265 1 logs.go:49] skydns: ready for queries on cluster.local. for udp4://0.0.0.0:8053 [rcache 0] I1022 15:12:00.158725 1 dns_server.go:63] DNS listening at 0.0.0.0:8053 I1022 15:12:00.167910 1 htpasswd.go:118] Loading htpasswd file /etc/origin/master/htpasswd... I1022 15:12:00.168182 1 htpasswd.go:118] Loading htpasswd file /etc/origin/master/htpasswd... I1022 15:12:00.231233 1 storage_factory.go:285] storing {apps.openshift.io deploymentconfigs} in apps.openshift.io/v1, reading as apps.openshift.io/__internal from storagebackend.Config{Type:"etcd3", Prefix:"openshift.io", ServerList:[]string{"https://master-0.com:2379"}, KeyFile:"/etc/origin/master/master.etcd-client.key", CertFile:"/etc/origin/master/master.etcd-client.crt", CAFile:"/etc/origin/master/master.etcd-ca.crt", Quorum:true, Paging:true, DeserializationCacheSize:0, Codec:runtime.Codec(nil), Transformer:value.Transformer(nil), CompactionInterval:300000000000, CountMetricPollPeriod:60000000000} I1022 15:12:00.248136 1 compact.go:54] compactor already exists for endpoints [https://master-0.com:2379] I1022 15:12:00.248697 1 store.go:1391] Monitoring deploymentconfigs.apps.openshift.io count at <storage-prefix>//deploymentconfigs W1022 15:12:00.256861 1 swagger.go:38] No API exists for predefined swagger description /oapi/v1 W1022 15:12:00.258106 1 swagger.go:38] No API exists for predefined swagger description /api/v1