7.11. 마스터 및 노드 로그 보기
OpenShift Container Platform은 노드에 systemd-journald.service를 사용하고 마스터를 위해
라는 스크립트를 사용하여 디버깅을 위해 로그 메시지를 수집합니다.
master-
logs
웹 콘솔에 표시되는 행 수는 5000으로 하드 코딩되어 있으며 변경할 수 없습니다. 전체 로그를 보려면 CLI를 사용합니다.
로깅은 다음과 같이 Kubernetes 로깅 규칙에 따라 5개의 로그 메시지 심각도를 사용합니다.
옵션 | 설명 |
---|---|
0 | 오류 및 경고 만 |
2 | 일반 정보 |
4 | 디버깅 수준 정보 |
6 | API 수준 디버깅 정보(요청/응답) |
8 | 본문 수준 API 디버깅 정보 |
필요한 경우 마스터 또는 노드의 로그 수준을 독립적으로 변경할 수 있습니다.
노드 로그 보기
노드 시스템의 로그를 보려면 다음 명령을 실행합니다.
# journalctl -r -u <journal_name>
r 옵션을
사용하여 최신 항목을 먼저 표시합니다.
마스터 로그 보기
마스터 구성 요소의 로그를 보려면 다음 명령을 실행합니다.
# /usr/local/bin/master-logs <component> <container>
예를 들면 다음과 같습니다.
# /usr/local/bin/master-logs controllers controllers # /usr/local/bin/master-logs api api # /usr/local/bin/master-logs etcd etcd
마스터 로그를 파일로 리디렉션
마스터 로그인 출력을 파일로 리디렉션하려면 다음 명령을 실행합니다.
master-logs api api 2> file
7.11.1. 로깅 수준 구성
노드의 master 또는 /etc/ sysconfig/atomic-openshift-node 파일의 /etc/origin/master/master/env 파일에서 DEBUG_LOGLEVEL
옵션을 설정하여 로깅되는 INFO 메시지를 제어할 수 있습니다. 모든 메시지를 수집하도록 로그를 구성하면 해석하기 어려워지고 과도한 공간을 차지할 수 있는 대용량 로그가 발생할 수 있습니다. 클러스터를 디버깅해야 하는 경우에만 모든 메시지를 수집합니다.
FATAL, ERROR, WARNING 및 일부 INFO 심각도가 있는 메시지는 로그 구성에 관계없이 로그에 표시됩니다.
로깅 수준을 변경하려면 다음을 수행합니다.
- 노드의 마스터 또는 /etc/ sysconfig/atomic-openshift-node 파일의 /etc/origin/ master/master.env 파일을 편집합니다.
DEBUG_LOGLEVEL
(로그 수준 옵션) 필드에 Log Level Options (로그 수준 옵션) 테이블의 값을 입력합니다.예를 들면 다음과 같습니다.
DEBUG_LOGLEVEL=4
- 마스터 또는 노드 호스트를 적절하게 다시 시작합니다. OpenShift Container Platform 서비스 재시작을 참조하십시오.
재시작 후 모든 새 로그 메시지는 새 설정을 준수합니다. 이전 메시지는 변경되지 않습니다.
기본 로그 수준은 표준 클러스터 설치 프로세스를 사용하여 설정할 수 있습니다. 자세한 내용은 클러스터 변수를 참조하십시오.
다음 예제는 다양한 로그 수준에서 리디렉션된 마스터 로그 파일의 발췌 내용입니다. 이러한 예에서 시스템 정보가 제거되었습니다.
loglevel=2에서 master-logs api api 2> 파일
출력 발췌
W1022 15:08:09.787705 1 server.go:79] Unable to keep dnsmasq up to date, 0.0.0.0:8053 must point to port 53 I1022 15:08:09.787894 1 logs.go:49] skydns: ready for queries on cluster.local. for tcp4://0.0.0.0:8053 [rcache 0] I1022 15:08:09.787913 1 logs.go:49] skydns: ready for queries on cluster.local. for udp4://0.0.0.0:8053 [rcache 0] I1022 15:08:09.889022 1 dns_server.go:63] DNS listening at 0.0.0.0:8053 I1022 15:08:09.893156 1 feature_gate.go:190] feature gates: map[AdvancedAuditing:true] I1022 15:08:09.893500 1 master.go:431] Starting OAuth2 API at /oauth I1022 15:08:09.914759 1 master.go:431] Starting OAuth2 API at /oauth I1022 15:08:09.942349 1 master.go:431] Starting OAuth2 API at /oauth W1022 15:08:09.977088 1 swagger.go:38] No API exists for predefined swagger description /oapi/v1 W1022 15:08:09.977176 1 swagger.go:38] No API exists for predefined swagger description /api/v1 [restful] 2018/10/22 15:08:09 log.go:33: [restful/swagger] listing is available at https://openshift.com:443/swaggerapi [restful] 2018/10/22 15:08:09 log.go:33: [restful/swagger] https://openshift.com:443/swaggerui/ is mapped to folder /swagger-ui/ I1022 15:08:10.231405 1 master.go:431] Starting OAuth2 API at /oauth W1022 15:08:10.259523 1 swagger.go:38] No API exists for predefined swagger description /oapi/v1 W1022 15:08:10.259555 1 swagger.go:38] No API exists for predefined swagger description /api/v1 I1022 15:08:23.895493 1 logs.go:49] http: TLS handshake error from 10.10.94.10:46322: EOF I1022 15:08:24.449577 1 crdregistration_controller.go:110] Starting crd-autoregister controller I1022 15:08:24.449916 1 controller_utils.go:1019] Waiting for caches to sync for crd-autoregister controller I1022 15:08:24.496147 1 logs.go:49] http: TLS handshake error from 127.0.0.1:39140: EOF I1022 15:08:24.821198 1 cache.go:39] Caches are synced for APIServiceRegistrationController controller I1022 15:08:24.833022 1 cache.go:39] Caches are synced for AvailableConditionController controller I1022 15:08:24.865087 1 controller.go:537] quota admission added evaluator for: { events} I1022 15:08:24.865393 1 logs.go:49] http: TLS handshake error from 127.0.0.1:39162: read tcp4 127.0.0.1:443->127.0.0.1:39162: read: connection reset by peer I1022 15:08:24.966917 1 controller_utils.go:1026] Caches are synced for crd-autoregister controller I1022 15:08:24.967961 1 autoregister_controller.go:136] Starting autoregister controller I1022 15:08:24.967977 1 cache.go:32] Waiting for caches to sync for autoregister controller I1022 15:08:25.015924 1 controller.go:537] quota admission added evaluator for: { serviceaccounts} I1022 15:08:25.077984 1 cache.go:39] Caches are synced for autoregister controller W1022 15:08:25.304265 1 lease_endpoint_reconciler.go:176] Resetting endpoints for master service "kubernetes" to [10.10.94.10] E1022 15:08:25.472536 1 memcache.go:153] couldn't get resource list for servicecatalog.k8s.io/v1beta1: the server could not find the requested resource E1022 15:08:25.550888 1 memcache.go:153] couldn't get resource list for servicecatalog.k8s.io/v1beta1: the server could not find the requested resource I1022 15:08:29.480691 1 healthz.go:72] /healthz/log check I1022 15:08:30.981999 1 controller.go:105] OpenAPI AggregationController: Processing item v1beta1.servicecatalog.k8s.io E1022 15:08:30.990914 1 controller.go:111] loading OpenAPI spec for "v1beta1.servicecatalog.k8s.io" failed with: OpenAPI spec does not exists I1022 15:08:30.990965 1 controller.go:119] OpenAPI AggregationController: action for item v1beta1.servicecatalog.k8s.io: Rate Limited Requeue. I1022 15:08:31.530473 1 trace.go:76] Trace[1253590531]: "Get /api/v1/namespaces/openshift-infra/serviceaccounts/serviceaccount-controller" (started: 2018-10-22 15:08:30.868387562 +0000 UTC m=+24.277041043) (total time: 661.981642ms): Trace[1253590531]: [661.903178ms] [661.89217ms] About to write a response I1022 15:08:31.531366 1 trace.go:76] Trace[83808472]: "Get /api/v1/namespaces/aws-sb/secrets/aws-servicebroker" (started: 2018-10-22 15:08:30.831296749 +0000 UTC m=+24.239950203) (total time: 700.049245ms):
loglevel=4에서 master-logs api api 2> 파일
출력 발췌
I1022 15:08:09.746980 1 plugins.go:149] Loaded 1 admission controller(s) successfully in the following order: AlwaysDeny. I1022 15:08:09.747597 1 plugins.go:149] Loaded 1 admission controller(s) successfully in the following order: ResourceQuota. I1022 15:08:09.748038 1 plugins.go:149] Loaded 1 admission controller(s) successfully in the following order: openshift.io/ClusterResourceQuota. I1022 15:08:09.786771 1 start_master.go:458] Starting master on 0.0.0.0:443 (v3.10.45) I1022 15:08:09.786798 1 start_master.go:459] Public master address is https://openshift.com:443 I1022 15:08:09.786844 1 start_master.go:463] Using images from "registry.access.redhat.com/openshift3/ose-<component>:v3.10.45" W1022 15:08:09.787046 1 dns_server.go:37] Binding DNS on port 8053 instead of 53, which may not be resolvable from all clients W1022 15:08:09.787705 1 server.go:79] Unable to keep dnsmasq up to date, 0.0.0.0:8053 must point to port 53 I1022 15:08:09.787894 1 logs.go:49] skydns: ready for queries on cluster.local. for tcp4://0.0.0.0:8053 [rcache 0] I1022 15:08:09.787913 1 logs.go:49] skydns: ready for queries on cluster.local. for udp4://0.0.0.0:8053 [rcache 0] I1022 15:08:09.889022 1 dns_server.go:63] DNS listening at 0.0.0.0:8053 I1022 15:08:09.893156 1 feature_gate.go:190] feature gates: map[AdvancedAuditing:true] I1022 15:08:09.893500 1 master.go:431] Starting OAuth2 API at /oauth I1022 15:08:09.914759 1 master.go:431] Starting OAuth2 API at /oauth I1022 15:08:09.942349 1 master.go:431] Starting OAuth2 API at /oauth W1022 15:08:09.977088 1 swagger.go:38] No API exists for predefined swagger description /oapi/v1 W1022 15:08:09.977176 1 swagger.go:38] No API exists for predefined swagger description /api/v1 [restful] 2018/10/22 15:08:09 log.go:33: [restful/swagger] listing is available at https://openshift.com:443/swaggerapi [restful] 2018/10/22 15:08:09 log.go:33: [restful/swagger] https://openshift.com:443/swaggerui/ is mapped to folder /swagger-ui/ I1022 15:08:10.231405 1 master.go:431] Starting OAuth2 API at /oauth W1022 15:08:10.259523 1 swagger.go:38] No API exists for predefined swagger description /oapi/v1 W1022 15:08:10.259555 1 swagger.go:38] No API exists for predefined swagger description /api/v1 [restful] 2018/10/22 15:08:10 log.go:33: [restful/swagger] listing is available at https://openshift.com:443/swaggerapi [restful] 2018/10/22 15:08:10 log.go:33: [restful/swagger] https://openshift.com:443/swaggerui/ is mapped to folder /swagger-ui/ I1022 15:08:10.444303 1 master.go:431] Starting OAuth2 API at /oauth W1022 15:08:10.492409 1 swagger.go:38] No API exists for predefined swagger description /oapi/v1 W1022 15:08:10.492507 1 swagger.go:38] No API exists for predefined swagger description /api/v1 [restful] 2018/10/22 15:08:10 log.go:33: [restful/swagger] listing is available at https://openshift.com:443/swaggerapi [restful] 2018/10/22 15:08:10 log.go:33: [restful/swagger] https://openshift.com:443/swaggerui/ is mapped to folder /swagger-ui/ I1022 15:08:10.774824 1 master.go:431] Starting OAuth2 API at /oauth I1022 15:08:23.808685 1 logs.go:49] http: TLS handshake error from 10.128.0.11:39206: EOF I1022 15:08:23.815311 1 logs.go:49] http: TLS handshake error from 10.128.0.14:53054: EOF I1022 15:08:23.822286 1 customresource_discovery_controller.go:174] Starting DiscoveryController I1022 15:08:23.822349 1 naming_controller.go:276] Starting NamingConditionController I1022 15:08:23.822705 1 logs.go:49] http: TLS handshake error from 10.128.0.14:53056: EOF +24.277041043) (total time: 661.981642ms): Trace[1253590531]: [661.903178ms] [661.89217ms] About to write a response I1022 15:08:31.531366 1 trace.go:76] Trace[83808472]: "Get /api/v1/namespaces/aws-sb/secrets/aws-servicebroker" (started: 2018-10-22 15:08:30.831296749 +0000 UTC m=+24.239950203) (total time: 700.049245ms): Trace[83808472]: [700.049245ms] [700.04027ms] END I1022 15:08:31.531695 1 trace.go:76] Trace[1916801734]: "Get /api/v1/namespaces/aws-sb/secrets/aws-servicebroker" (started: 2018-10-22 15:08:31.031163449 +0000 UTC m=+24.439816907) (total time: 500.514208ms): Trace[1916801734]: [500.514208ms] [500.505008ms] END I1022 15:08:44.675371 1 healthz.go:72] /healthz/log check I1022 15:08:46.589759 1 controller.go:537] quota admission added evaluator for: { endpoints} I1022 15:08:46.621270 1 controller.go:537] quota admission added evaluator for: { endpoints} I1022 15:08:57.159494 1 healthz.go:72] /healthz/log check I1022 15:09:07.161315 1 healthz.go:72] /healthz/log check I1022 15:09:16.297982 1 trace.go:76] Trace[2001108522]: "GuaranteedUpdate etcd3: *core.Node" (started: 2018-10-22 15:09:15.139820419 +0000 UTC m=+68.548473981) (total time: 1.158128974s): Trace[2001108522]: [1.158012755s] [1.156496534s] Transaction committed I1022 15:09:16.298165 1 trace.go:76] Trace[1124283912]: "Patch /api/v1/nodes/master-0.com/status" (started: 2018-10-22 15:09:15.139695483 +0000 UTC m=+68.548348970) (total time: 1.158434318s): Trace[1124283912]: [1.158328853s] [1.15713683s] Object stored in database I1022 15:09:16.298761 1 trace.go:76] Trace[24963576]: "GuaranteedUpdate etcd3: *core.Node" (started: 2018-10-22 15:09:15.13159057 +0000 UTC m=+68.540244112) (total time: 1.167151224s): Trace[24963576]: [1.167106144s] [1.165570379s] Transaction committed I1022 15:09:16.298882 1 trace.go:76] Trace[222129183]: "Patch /api/v1/nodes/node-0.com/status" (started: 2018-10-22 15:09:15.131269234 +0000 UTC m=+68.539922722) (total time: 1.167595526s): Trace[222129183]: [1.167517296s] [1.166135605s] Object stored in database
loglevel=8에서 master-logs api api 2> 파일
출력 발췌
1022 15:11:58.829357 1 plugins.go:84] Registered admission plugin "NamespaceLifecycle" I1022 15:11:58.839967 1 plugins.go:84] Registered admission plugin "Initializers" I1022 15:11:58.839994 1 plugins.go:84] Registered admission plugin "ValidatingAdmissionWebhook" I1022 15:11:58.840012 1 plugins.go:84] Registered admission plugin "MutatingAdmissionWebhook" I1022 15:11:58.840025 1 plugins.go:84] Registered admission plugin "AlwaysAdmit" I1022 15:11:58.840082 1 plugins.go:84] Registered admission plugin "AlwaysPullImages" I1022 15:11:58.840105 1 plugins.go:84] Registered admission plugin "LimitPodHardAntiAffinityTopology" I1022 15:11:58.840126 1 plugins.go:84] Registered admission plugin "DefaultTolerationSeconds" I1022 15:11:58.840146 1 plugins.go:84] Registered admission plugin "AlwaysDeny" I1022 15:11:58.840176 1 plugins.go:84] Registered admission plugin "EventRateLimit" I1022 15:11:59.850825 1 feature_gate.go:190] feature gates: map[AdvancedAuditing:true] I1022 15:11:59.859108 1 register.go:154] Admission plugin AlwaysAdmit is not enabled. It will not be started. I1022 15:11:59.859284 1 plugins.go:149] Loaded 1 admission controller(s) successfully in the following order: AlwaysAdmit. I1022 15:11:59.859809 1 register.go:154] Admission plugin NamespaceAutoProvision is not enabled. It will not be started. I1022 15:11:59.859939 1 plugins.go:149] Loaded 1 admission controller(s) successfully in the following order: NamespaceAutoProvision. I1022 15:11:59.860594 1 register.go:154] Admission plugin NamespaceExists is not enabled. It will not be started. I1022 15:11:59.860778 1 plugins.go:149] Loaded 1 admission controller(s) successfully in the following order: NamespaceExists. I1022 15:11:59.863999 1 plugins.go:149] Loaded 1 admission controller(s) successfully in the following order: NamespaceLifecycle. I1022 15:11:59.864626 1 register.go:154] Admission plugin EventRateLimit is not enabled. It will not be started. I1022 15:11:59.864768 1 plugins.go:149] Loaded 1 admission controller(s) successfully in the following order: EventRateLimit. I1022 15:11:59.865259 1 register.go:154] Admission plugin ProjectRequestLimit is not enabled. It will not be started. I1022 15:11:59.865376 1 plugins.go:149] Loaded 1 admission controller(s) successfully in the following order: ProjectRequestLimit. I1022 15:11:59.866126 1 plugins.go:149] Loaded 1 admission controller(s) successfully in the following order: OriginNamespaceLifecycle. I1022 15:11:59.866709 1 register.go:154] Admission plugin openshift.io/RestrictSubjectBindings is not enabled. It will not be started. I1022 15:11:59.866761 1 plugins.go:149] Loaded 1 admission controller(s) successfully in the following order: openshift.io/RestrictSubjectBindings. I1022 15:11:59.867304 1 plugins.go:149] Loaded 1 admission controller(s) successfully in the following order: openshift.io/JenkinsBootstrapper. I1022 15:11:59.867823 1 plugins.go:149] Loaded 1 admission controller(s) successfully in the following order: openshift.io/BuildConfigSecretInjector. I1022 15:12:00.015273 1 master_config.go:476] Initializing cache sizes based on 0MB limit I1022 15:12:00.015896 1 master_config.go:539] Using the lease endpoint reconciler with TTL=15s and interval=10s I1022 15:12:00.018396 1 storage_factory.go:285] storing { apiServerIPInfo} in v1, reading as __internal from storagebackend.Config{Type:"etcd3", Prefix:"kubernetes.io", ServerList:[]string{"https://master-0.com:2379"}, KeyFile:"/etc/origin/master/master.etcd-client.key", CertFile:"/etc/origin/master/master.etcd-client.crt", CAFile:"/etc/origin/master/master.etcd-ca.crt", Quorum:true, Paging:true, DeserializationCacheSize:0, Codec:runtime.Codec(nil), Transformer:value.Transformer(nil), CompactionInterval:300000000000, CountMetricPollPeriod:60000000000} I1022 15:12:00.037710 1 storage_factory.go:285] storing { endpoints} in v1, reading as __internal from storagebackend.Config{Type:"etcd3", Prefix:"kubernetes.io", ServerList:[]string{"https://master-0.com:2379"}, KeyFile:"/etc/origin/master/master.etcd-client.key", CertFile:"/etc/origin/master/master.etcd-client.crt", CAFile:"/etc/origin/master/master.etcd-ca.crt", Quorum:true, Paging:true, DeserializationCacheSize:0, Codec:runtime.Codec(nil), Transformer:value.Transformer(nil), CompactionInterval:300000000000, CountMetricPollPeriod:60000000000} I1022 15:12:00.054112 1 compact.go:54] compactor already exists for endpoints [https://master-0.com:2379] I1022 15:12:00.054678 1 start_master.go:458] Starting master on 0.0.0.0:443 (v3.10.45) I1022 15:12:00.054755 1 start_master.go:459] Public master address is https://openshift.com:443 I1022 15:12:00.054837 1 start_master.go:463] Using images from "registry.access.redhat.com/openshift3/ose-<component>:v3.10.45" W1022 15:12:00.056957 1 dns_server.go:37] Binding DNS on port 8053 instead of 53, which may not be resolvable from all clients W1022 15:12:00.065497 1 server.go:79] Unable to keep dnsmasq up to date, 0.0.0.0:8053 must point to port 53 I1022 15:12:00.066061 1 logs.go:49] skydns: ready for queries on cluster.local. for tcp4://0.0.0.0:8053 [rcache 0] I1022 15:12:00.066265 1 logs.go:49] skydns: ready for queries on cluster.local. for udp4://0.0.0.0:8053 [rcache 0] I1022 15:12:00.158725 1 dns_server.go:63] DNS listening at 0.0.0.0:8053 I1022 15:12:00.167910 1 htpasswd.go:118] Loading htpasswd file /etc/origin/master/htpasswd... I1022 15:12:00.168182 1 htpasswd.go:118] Loading htpasswd file /etc/origin/master/htpasswd... I1022 15:12:00.231233 1 storage_factory.go:285] storing {apps.openshift.io deploymentconfigs} in apps.openshift.io/v1, reading as apps.openshift.io/__internal from storagebackend.Config{Type:"etcd3", Prefix:"openshift.io", ServerList:[]string{"https://master-0.com:2379"}, KeyFile:"/etc/origin/master/master.etcd-client.key", CertFile:"/etc/origin/master/master.etcd-client.crt", CAFile:"/etc/origin/master/master.etcd-ca.crt", Quorum:true, Paging:true, DeserializationCacheSize:0, Codec:runtime.Codec(nil), Transformer:value.Transformer(nil), CompactionInterval:300000000000, CountMetricPollPeriod:60000000000} I1022 15:12:00.248136 1 compact.go:54] compactor already exists for endpoints [https://master-0.com:2379] I1022 15:12:00.248697 1 store.go:1391] Monitoring deploymentconfigs.apps.openshift.io count at <storage-prefix>//deploymentconfigs W1022 15:12:00.256861 1 swagger.go:38] No API exists for predefined swagger description /oapi/v1 W1022 15:12:00.258106 1 swagger.go:38] No API exists for predefined swagger description /api/v1