5.2.
5.2.1. 링크 복사링크가 클립보드에 복사되었습니다!
5.2.2. 링크 복사링크가 클립보드에 복사되었습니다!
oc get etcd -o=jsonpath='{range .items[0].status.conditions[?(@.type=="EtcdMembersAvailable")]}{.message}{"\n"}'$ oc get etcd -o=jsonpath='{range .items[0].status.conditions[?(@.type=="EtcdMembersAvailable")]}{.message}{"\n"}'Copy to Clipboard Copied! Toggle word wrap Toggle overflow 2 of 3 members are available, ip-10-0-131-183.ec2.internal is unhealthy
2 of 3 members are available, ip-10-0-131-183.ec2.internal is unhealthyCopy to Clipboard Copied! Toggle word wrap Toggle overflow
5.2.3. 링크 복사링크가 클립보드에 복사되었습니다!
oc get machines -A -ojsonpath='{range .items[*]}{@.status.nodeRef.name}{"\t"}{@.status.providerStatus.instanceState}{"\n"}' | grep -v running$ oc get machines -A -ojsonpath='{range .items[*]}{@.status.nodeRef.name}{"\t"}{@.status.providerStatus.instanceState}{"\n"}' | grep -v runningCopy to Clipboard Copied! Toggle word wrap Toggle overflow ip-10-0-131-183.ec2.internal stopped
ip-10-0-131-183.ec2.internal stopped1 Copy to Clipboard Copied! Toggle word wrap Toggle overflow oc get nodes -o jsonpath='{range .items[*]}{"\n"}{.metadata.name}{"\t"}{range .spec.taints[*]}{.key}{" "}' | grep unreachable$ oc get nodes -o jsonpath='{range .items[*]}{"\n"}{.metadata.name}{"\t"}{range .spec.taints[*]}{.key}{" "}' | grep unreachableCopy to Clipboard Copied! Toggle word wrap Toggle overflow ip-10-0-131-183.ec2.internal node-role.kubernetes.io/master node.kubernetes.io/unreachable node.kubernetes.io/unreachable
ip-10-0-131-183.ec2.internal node-role.kubernetes.io/master node.kubernetes.io/unreachable node.kubernetes.io/unreachable1 Copy to Clipboard Copied! Toggle word wrap Toggle overflow oc get nodes -l node-role.kubernetes.io/master | grep "NotReady"
$ oc get nodes -l node-role.kubernetes.io/master | grep "NotReady"Copy to Clipboard Copied! Toggle word wrap Toggle overflow ip-10-0-131-183.ec2.internal NotReady master 122m v1.29.4
ip-10-0-131-183.ec2.internal NotReady master 122m v1.29.41 Copy to Clipboard Copied! Toggle word wrap Toggle overflow
oc get nodes -l node-role.kubernetes.io/master
$ oc get nodes -l node-role.kubernetes.io/masterCopy to Clipboard Copied! Toggle word wrap Toggle overflow NAME STATUS ROLES AGE VERSION ip-10-0-131-183.ec2.internal Ready master 6h13m v1.29.4 ip-10-0-164-97.ec2.internal Ready master 6h13m v1.29.4 ip-10-0-154-204.ec2.internal Ready master 6h13m v1.29.4
NAME STATUS ROLES AGE VERSION ip-10-0-131-183.ec2.internal Ready master 6h13m v1.29.4 ip-10-0-164-97.ec2.internal Ready master 6h13m v1.29.4 ip-10-0-154-204.ec2.internal Ready master 6h13m v1.29.4Copy to Clipboard Copied! Toggle word wrap Toggle overflow oc -n openshift-etcd get pods -l k8s-app=etcd
$ oc -n openshift-etcd get pods -l k8s-app=etcdCopy to Clipboard Copied! Toggle word wrap Toggle overflow etcd-ip-10-0-131-183.ec2.internal 2/3 Error 7 6h9m etcd-ip-10-0-164-97.ec2.internal 3/3 Running 0 6h6m etcd-ip-10-0-154-204.ec2.internal 3/3 Running 0 6h6m
etcd-ip-10-0-131-183.ec2.internal 2/3 Error 7 6h9m1 etcd-ip-10-0-164-97.ec2.internal 3/3 Running 0 6h6m etcd-ip-10-0-154-204.ec2.internal 3/3 Running 0 6h6mCopy to Clipboard Copied! Toggle word wrap Toggle overflow
5.2.4. 링크 복사링크가 클립보드에 복사되었습니다!
5.2.4.1. 링크 복사링크가 클립보드에 복사되었습니다!
- 중요
- 중요
oc -n openshift-etcd get pods -l k8s-app=etcd
$ oc -n openshift-etcd get pods -l k8s-app=etcdCopy to Clipboard Copied! Toggle word wrap Toggle overflow etcd-ip-10-0-131-183.ec2.internal 3/3 Running 0 123m etcd-ip-10-0-164-97.ec2.internal 3/3 Running 0 123m etcd-ip-10-0-154-204.ec2.internal 3/3 Running 0 124m
etcd-ip-10-0-131-183.ec2.internal 3/3 Running 0 123m etcd-ip-10-0-164-97.ec2.internal 3/3 Running 0 123m etcd-ip-10-0-154-204.ec2.internal 3/3 Running 0 124mCopy to Clipboard Copied! Toggle word wrap Toggle overflow oc rsh -n openshift-etcd etcd-ip-10-0-154-204.ec2.internal
$ oc rsh -n openshift-etcd etcd-ip-10-0-154-204.ec2.internalCopy to Clipboard Copied! Toggle word wrap Toggle overflow etcdctl member list -w table
sh-4.2# etcdctl member list -w tableCopy to Clipboard Copied! Toggle word wrap Toggle overflow Copy to Clipboard Copied! Toggle word wrap Toggle overflow etcdctl member remove 6fc1e7c9db35841d
sh-4.2# etcdctl member remove 6fc1e7c9db35841dCopy to Clipboard Copied! Toggle word wrap Toggle overflow Member 6fc1e7c9db35841d removed from cluster ead669ce1fbfb346
Member 6fc1e7c9db35841d removed from cluster ead669ce1fbfb346Copy to Clipboard Copied! Toggle word wrap Toggle overflow etcdctl member list -w table
sh-4.2# etcdctl member list -w tableCopy to Clipboard Copied! Toggle word wrap Toggle overflow Copy to Clipboard Copied! Toggle word wrap Toggle overflow
oc patch etcd/cluster --type=merge -p '{"spec": {"unsupportedConfigOverrides": {"useUnsupportedUnsafeNonHANonProductionUnstableEtcd": true}}}'$ oc patch etcd/cluster --type=merge -p '{"spec": {"unsupportedConfigOverrides": {"useUnsupportedUnsafeNonHANonProductionUnstableEtcd": true}}}'Copy to Clipboard Copied! Toggle word wrap Toggle overflow 중요참고oc delete node <node_name>
$ oc delete node <node_name>Copy to Clipboard Copied! Toggle word wrap Toggle overflow oc delete node ip-10-0-131-183.ec2.internal
$ oc delete node ip-10-0-131-183.ec2.internalCopy to Clipboard Copied! Toggle word wrap Toggle overflow oc get secrets -n openshift-etcd | grep ip-10-0-131-183.ec2.internal
$ oc get secrets -n openshift-etcd | grep ip-10-0-131-183.ec2.internal1 Copy to Clipboard Copied! Toggle word wrap Toggle overflow etcd-peer-ip-10-0-131-183.ec2.internal kubernetes.io/tls 2 47m etcd-serving-ip-10-0-131-183.ec2.internal kubernetes.io/tls 2 47m etcd-serving-metrics-ip-10-0-131-183.ec2.internal kubernetes.io/tls 2 47m
etcd-peer-ip-10-0-131-183.ec2.internal kubernetes.io/tls 2 47m etcd-serving-ip-10-0-131-183.ec2.internal kubernetes.io/tls 2 47m etcd-serving-metrics-ip-10-0-131-183.ec2.internal kubernetes.io/tls 2 47mCopy to Clipboard Copied! Toggle word wrap Toggle overflow oc delete secret -n openshift-etcd etcd-peer-ip-10-0-131-183.ec2.internal
$ oc delete secret -n openshift-etcd etcd-peer-ip-10-0-131-183.ec2.internalCopy to Clipboard Copied! Toggle word wrap Toggle overflow oc delete secret -n openshift-etcd etcd-serving-ip-10-0-131-183.ec2.internal
$ oc delete secret -n openshift-etcd etcd-serving-ip-10-0-131-183.ec2.internalCopy to Clipboard Copied! Toggle word wrap Toggle overflow oc delete secret -n openshift-etcd etcd-serving-metrics-ip-10-0-131-183.ec2.internal
$ oc delete secret -n openshift-etcd etcd-serving-metrics-ip-10-0-131-183.ec2.internalCopy to Clipboard Copied! Toggle word wrap Toggle overflow
oc get machines -n openshift-machine-api -o wide
$ oc get machines -n openshift-machine-api -o wideCopy to Clipboard Copied! Toggle word wrap Toggle overflow Copy to Clipboard Copied! Toggle word wrap Toggle overflow oc delete machine -n openshift-machine-api clustername-8qw5l-master-0
$ oc delete machine -n openshift-machine-api clustername-8qw5l-master-01 Copy to Clipboard Copied! Toggle word wrap Toggle overflow oc get machines -n openshift-machine-api -o wide
$ oc get machines -n openshift-machine-api -o wideCopy to Clipboard Copied! Toggle word wrap Toggle overflow Copy to Clipboard Copied! Toggle word wrap Toggle overflow
oc patch etcd/cluster --type=merge -p '{"spec": {"unsupportedConfigOverrides": null}}'$ oc patch etcd/cluster --type=merge -p '{"spec": {"unsupportedConfigOverrides": null}}'Copy to Clipboard Copied! Toggle word wrap Toggle overflow oc get etcd/cluster -oyaml
$ oc get etcd/cluster -oyamlCopy to Clipboard Copied! Toggle word wrap Toggle overflow EtcdCertSignerControllerDegraded: [Operation cannot be fulfilled on secrets "etcd-peer-sno-0": the object has been modified; please apply your changes to the latest version and try again, Operation cannot be fulfilled on secrets "etcd-serving-sno-0": the object has been modified; please apply your changes to the latest version and try again, Operation cannot be fulfilled on secrets "etcd-serving-metrics-sno-0": the object has been modified; please apply your changes to the latest version and try again]
EtcdCertSignerControllerDegraded: [Operation cannot be fulfilled on secrets "etcd-peer-sno-0": the object has been modified; please apply your changes to the latest version and try again, Operation cannot be fulfilled on secrets "etcd-serving-sno-0": the object has been modified; please apply your changes to the latest version and try again, Operation cannot be fulfilled on secrets "etcd-serving-metrics-sno-0": the object has been modified; please apply your changes to the latest version and try again]Copy to Clipboard Copied! Toggle word wrap Toggle overflow
oc -n openshift-etcd get pods -l k8s-app=etcd
$ oc -n openshift-etcd get pods -l k8s-app=etcdCopy to Clipboard Copied! Toggle word wrap Toggle overflow etcd-ip-10-0-133-53.ec2.internal 3/3 Running 0 7m49s etcd-ip-10-0-164-97.ec2.internal 3/3 Running 0 123m etcd-ip-10-0-154-204.ec2.internal 3/3 Running 0 124m
etcd-ip-10-0-133-53.ec2.internal 3/3 Running 0 7m49s etcd-ip-10-0-164-97.ec2.internal 3/3 Running 0 123m etcd-ip-10-0-154-204.ec2.internal 3/3 Running 0 124mCopy to Clipboard Copied! Toggle word wrap Toggle overflow oc patch etcd cluster -p='{"spec": {"forceRedeploymentReason": "recovery-'"$( date --rfc-3339=ns )"'"}}' --type=merge$ oc patch etcd cluster -p='{"spec": {"forceRedeploymentReason": "recovery-'"$( date --rfc-3339=ns )"'"}}' --type=merge1 Copy to Clipboard Copied! Toggle word wrap Toggle overflow oc rsh -n openshift-etcd etcd-ip-10-0-154-204.ec2.internal
$ oc rsh -n openshift-etcd etcd-ip-10-0-154-204.ec2.internalCopy to Clipboard Copied! Toggle word wrap Toggle overflow etcdctl member list -w table
sh-4.2# etcdctl member list -w tableCopy to Clipboard Copied! Toggle word wrap Toggle overflow Copy to Clipboard Copied! Toggle word wrap Toggle overflow 주의
5.2.4.2. 링크 복사링크가 클립보드에 복사되었습니다!
- 중요
oc debug node/ip-10-0-131-183.ec2.internal
$ oc debug node/ip-10-0-131-183.ec2.internal1 Copy to Clipboard Copied! Toggle word wrap Toggle overflow chroot /host
sh-4.2# chroot /hostCopy to Clipboard Copied! Toggle word wrap Toggle overflow mkdir /var/lib/etcd-backup
sh-4.2# mkdir /var/lib/etcd-backupCopy to Clipboard Copied! Toggle word wrap Toggle overflow mv /etc/kubernetes/manifests/etcd-pod.yaml /var/lib/etcd-backup/
sh-4.2# mv /etc/kubernetes/manifests/etcd-pod.yaml /var/lib/etcd-backup/Copy to Clipboard Copied! Toggle word wrap Toggle overflow mv /var/lib/etcd/ /tmp
sh-4.2# mv /var/lib/etcd/ /tmpCopy to Clipboard Copied! Toggle word wrap Toggle overflow
oc -n openshift-etcd get pods -l k8s-app=etcd
$ oc -n openshift-etcd get pods -l k8s-app=etcdCopy to Clipboard Copied! Toggle word wrap Toggle overflow etcd-ip-10-0-131-183.ec2.internal 2/3 Error 7 6h9m etcd-ip-10-0-164-97.ec2.internal 3/3 Running 0 6h6m etcd-ip-10-0-154-204.ec2.internal 3/3 Running 0 6h6m
etcd-ip-10-0-131-183.ec2.internal 2/3 Error 7 6h9m etcd-ip-10-0-164-97.ec2.internal 3/3 Running 0 6h6m etcd-ip-10-0-154-204.ec2.internal 3/3 Running 0 6h6mCopy to Clipboard Copied! Toggle word wrap Toggle overflow oc rsh -n openshift-etcd etcd-ip-10-0-154-204.ec2.internal
$ oc rsh -n openshift-etcd etcd-ip-10-0-154-204.ec2.internalCopy to Clipboard Copied! Toggle word wrap Toggle overflow etcdctl member list -w table
sh-4.2# etcdctl member list -w tableCopy to Clipboard Copied! Toggle word wrap Toggle overflow Copy to Clipboard Copied! Toggle word wrap Toggle overflow etcdctl member remove 62bcf33650a7170a
sh-4.2# etcdctl member remove 62bcf33650a7170aCopy to Clipboard Copied! Toggle word wrap Toggle overflow Member 62bcf33650a7170a removed from cluster ead669ce1fbfb346
Member 62bcf33650a7170a removed from cluster ead669ce1fbfb346Copy to Clipboard Copied! Toggle word wrap Toggle overflow etcdctl member list -w table
sh-4.2# etcdctl member list -w tableCopy to Clipboard Copied! Toggle word wrap Toggle overflow Copy to Clipboard Copied! Toggle word wrap Toggle overflow
oc patch etcd/cluster --type=merge -p '{"spec": {"unsupportedConfigOverrides": {"useUnsupportedUnsafeNonHANonProductionUnstableEtcd": true}}}'$ oc patch etcd/cluster --type=merge -p '{"spec": {"unsupportedConfigOverrides": {"useUnsupportedUnsafeNonHANonProductionUnstableEtcd": true}}}'Copy to Clipboard Copied! Toggle word wrap Toggle overflow oc get secrets -n openshift-etcd | grep ip-10-0-131-183.ec2.internal
$ oc get secrets -n openshift-etcd | grep ip-10-0-131-183.ec2.internal1 Copy to Clipboard Copied! Toggle word wrap Toggle overflow etcd-peer-ip-10-0-131-183.ec2.internal kubernetes.io/tls 2 47m etcd-serving-ip-10-0-131-183.ec2.internal kubernetes.io/tls 2 47m etcd-serving-metrics-ip-10-0-131-183.ec2.internal kubernetes.io/tls 2 47m
etcd-peer-ip-10-0-131-183.ec2.internal kubernetes.io/tls 2 47m etcd-serving-ip-10-0-131-183.ec2.internal kubernetes.io/tls 2 47m etcd-serving-metrics-ip-10-0-131-183.ec2.internal kubernetes.io/tls 2 47mCopy to Clipboard Copied! Toggle word wrap Toggle overflow oc delete secret -n openshift-etcd etcd-peer-ip-10-0-131-183.ec2.internal
$ oc delete secret -n openshift-etcd etcd-peer-ip-10-0-131-183.ec2.internalCopy to Clipboard Copied! Toggle word wrap Toggle overflow oc delete secret -n openshift-etcd etcd-serving-ip-10-0-131-183.ec2.internal
$ oc delete secret -n openshift-etcd etcd-serving-ip-10-0-131-183.ec2.internalCopy to Clipboard Copied! Toggle word wrap Toggle overflow oc delete secret -n openshift-etcd etcd-serving-metrics-ip-10-0-131-183.ec2.internal
$ oc delete secret -n openshift-etcd etcd-serving-metrics-ip-10-0-131-183.ec2.internalCopy to Clipboard Copied! Toggle word wrap Toggle overflow
oc patch etcd cluster -p='{"spec": {"forceRedeploymentReason": "single-master-recovery-'"$( date --rfc-3339=ns )"'"}}' --type=merge$ oc patch etcd cluster -p='{"spec": {"forceRedeploymentReason": "single-master-recovery-'"$( date --rfc-3339=ns )"'"}}' --type=merge1 Copy to Clipboard Copied! Toggle word wrap Toggle overflow oc patch etcd/cluster --type=merge -p '{"spec": {"unsupportedConfigOverrides": null}}'$ oc patch etcd/cluster --type=merge -p '{"spec": {"unsupportedConfigOverrides": null}}'Copy to Clipboard Copied! Toggle word wrap Toggle overflow oc get etcd/cluster -oyaml
$ oc get etcd/cluster -oyamlCopy to Clipboard Copied! Toggle word wrap Toggle overflow EtcdCertSignerControllerDegraded: [Operation cannot be fulfilled on secrets "etcd-peer-sno-0": the object has been modified; please apply your changes to the latest version and try again, Operation cannot be fulfilled on secrets "etcd-serving-sno-0": the object has been modified; please apply your changes to the latest version and try again, Operation cannot be fulfilled on secrets "etcd-serving-metrics-sno-0": the object has been modified; please apply your changes to the latest version and try again]
EtcdCertSignerControllerDegraded: [Operation cannot be fulfilled on secrets "etcd-peer-sno-0": the object has been modified; please apply your changes to the latest version and try again, Operation cannot be fulfilled on secrets "etcd-serving-sno-0": the object has been modified; please apply your changes to the latest version and try again, Operation cannot be fulfilled on secrets "etcd-serving-metrics-sno-0": the object has been modified; please apply your changes to the latest version and try again]Copy to Clipboard Copied! Toggle word wrap Toggle overflow
oc rsh -n openshift-etcd etcd-ip-10-0-154-204.ec2.internal
$ oc rsh -n openshift-etcd etcd-ip-10-0-154-204.ec2.internalCopy to Clipboard Copied! Toggle word wrap Toggle overflow etcdctl endpoint health
sh-4.2# etcdctl endpoint healthCopy to Clipboard Copied! Toggle word wrap Toggle overflow https://10.0.131.183:2379 is healthy: successfully committed proposal: took = 16.671434ms https://10.0.154.204:2379 is healthy: successfully committed proposal: took = 16.698331ms https://10.0.164.97:2379 is healthy: successfully committed proposal: took = 16.621645ms
https://10.0.131.183:2379 is healthy: successfully committed proposal: took = 16.671434ms https://10.0.154.204:2379 is healthy: successfully committed proposal: took = 16.698331ms https://10.0.164.97:2379 is healthy: successfully committed proposal: took = 16.621645msCopy to Clipboard Copied! Toggle word wrap Toggle overflow
5.2.4.3. 링크 복사링크가 클립보드에 복사되었습니다!
- 중요
oc -n openshift-etcd get pods -l k8s-app=etcd -o wide
$ oc -n openshift-etcd get pods -l k8s-app=etcd -o wideCopy to Clipboard Copied! Toggle word wrap Toggle overflow etcd-openshift-control-plane-0 5/5 Running 11 3h56m 192.168.10.9 openshift-control-plane-0 <none> <none> etcd-openshift-control-plane-1 5/5 Running 0 3h54m 192.168.10.10 openshift-control-plane-1 <none> <none> etcd-openshift-control-plane-2 5/5 Running 0 3h58m 192.168.10.11 openshift-control-plane-2 <none> <none>
etcd-openshift-control-plane-0 5/5 Running 11 3h56m 192.168.10.9 openshift-control-plane-0 <none> <none> etcd-openshift-control-plane-1 5/5 Running 0 3h54m 192.168.10.10 openshift-control-plane-1 <none> <none> etcd-openshift-control-plane-2 5/5 Running 0 3h58m 192.168.10.11 openshift-control-plane-2 <none> <none>Copy to Clipboard Copied! Toggle word wrap Toggle overflow oc rsh -n openshift-etcd etcd-openshift-control-plane-0
$ oc rsh -n openshift-etcd etcd-openshift-control-plane-0Copy to Clipboard Copied! Toggle word wrap Toggle overflow etcdctl member list -w table
sh-4.2# etcdctl member list -w tableCopy to Clipboard Copied! Toggle word wrap Toggle overflow Copy to Clipboard Copied! Toggle word wrap Toggle overflow - 주의
etcdctl member remove 7a8197040a5126c8
sh-4.2# etcdctl member remove 7a8197040a5126c8Copy to Clipboard Copied! Toggle word wrap Toggle overflow Member 7a8197040a5126c8 removed from cluster b23536c33f2cdd1b
Member 7a8197040a5126c8 removed from cluster b23536c33f2cdd1bCopy to Clipboard Copied! Toggle word wrap Toggle overflow etcdctl member list -w table
sh-4.2# etcdctl member list -w tableCopy to Clipboard Copied! Toggle word wrap Toggle overflow Copy to Clipboard Copied! Toggle word wrap Toggle overflow 중요
oc patch etcd/cluster --type=merge -p '{"spec": {"unsupportedConfigOverrides": {"useUnsupportedUnsafeNonHANonProductionUnstableEtcd": true}}}'$ oc patch etcd/cluster --type=merge -p '{"spec": {"unsupportedConfigOverrides": {"useUnsupportedUnsafeNonHANonProductionUnstableEtcd": true}}}'Copy to Clipboard Copied! Toggle word wrap Toggle overflow oc get secrets -n openshift-etcd | grep openshift-control-plane-2
$ oc get secrets -n openshift-etcd | grep openshift-control-plane-2Copy to Clipboard Copied! Toggle word wrap Toggle overflow etcd-peer-openshift-control-plane-2 kubernetes.io/tls 2 134m etcd-serving-metrics-openshift-control-plane-2 kubernetes.io/tls 2 134m etcd-serving-openshift-control-plane-2 kubernetes.io/tls 2 134m
etcd-peer-openshift-control-plane-2 kubernetes.io/tls 2 134m etcd-serving-metrics-openshift-control-plane-2 kubernetes.io/tls 2 134m etcd-serving-openshift-control-plane-2 kubernetes.io/tls 2 134mCopy to Clipboard Copied! Toggle word wrap Toggle overflow oc delete secret etcd-peer-openshift-control-plane-2 -n openshift-etcd
$ oc delete secret etcd-peer-openshift-control-plane-2 -n openshift-etcd secret "etcd-peer-openshift-control-plane-2" deletedCopy to Clipboard Copied! Toggle word wrap Toggle overflow oc delete secret etcd-serving-metrics-openshift-control-plane-2 -n openshift-etcd
$ oc delete secret etcd-serving-metrics-openshift-control-plane-2 -n openshift-etcd secret "etcd-serving-metrics-openshift-control-plane-2" deletedCopy to Clipboard Copied! Toggle word wrap Toggle overflow oc delete secret etcd-serving-openshift-control-plane-2 -n openshift-etcd
$ oc delete secret etcd-serving-openshift-control-plane-2 -n openshift-etcd secret "etcd-serving-openshift-control-plane-2" deletedCopy to Clipboard Copied! Toggle word wrap Toggle overflow
oc get machines -n openshift-machine-api -o wide
$ oc get machines -n openshift-machine-api -o wideCopy to Clipboard Copied! Toggle word wrap Toggle overflow Copy to Clipboard Copied! Toggle word wrap Toggle overflow oc get clusteroperator baremetal
$ oc get clusteroperator baremetalCopy to Clipboard Copied! Toggle word wrap Toggle overflow NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE MESSAGE baremetal 4.16.0 True False False 3d15h
NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE MESSAGE baremetal 4.16.0 True False False 3d15hCopy to Clipboard Copied! Toggle word wrap Toggle overflow oc delete bmh openshift-control-plane-2 -n openshift-machine-api
$ oc delete bmh openshift-control-plane-2 -n openshift-machine-apiCopy to Clipboard Copied! Toggle word wrap Toggle overflow baremetalhost.metal3.io "openshift-control-plane-2" deleted
baremetalhost.metal3.io "openshift-control-plane-2" deletedCopy to Clipboard Copied! Toggle word wrap Toggle overflow oc delete machine -n openshift-machine-api examplecluster-control-plane-2
$ oc delete machine -n openshift-machine-api examplecluster-control-plane-2Copy to Clipboard Copied! Toggle word wrap Toggle overflow 중요oc edit machine -n openshift-machine-api examplecluster-control-plane-2
$ oc edit machine -n openshift-machine-api examplecluster-control-plane-2Copy to Clipboard Copied! Toggle word wrap Toggle overflow finalizers: - machine.machine.openshift.io
finalizers: - machine.machine.openshift.ioCopy to Clipboard Copied! Toggle word wrap Toggle overflow machine.machine.openshift.io/examplecluster-control-plane-2 edited
machine.machine.openshift.io/examplecluster-control-plane-2 editedCopy to Clipboard Copied! Toggle word wrap Toggle overflow
oc get machines -n openshift-machine-api -o wide
$ oc get machines -n openshift-machine-api -o wideCopy to Clipboard Copied! Toggle word wrap Toggle overflow NAME PHASE TYPE REGION ZONE AGE NODE PROVIDERID STATE examplecluster-control-plane-0 Running 3h11m openshift-control-plane-0 baremetalhost:///openshift-machine-api/openshift-control-plane-0/da1ebe11-3ff2-41c5-b099-0aa41222964e externally provisioned examplecluster-control-plane-1 Running 3h11m openshift-control-plane-1 baremetalhost:///openshift-machine-api/openshift-control-plane-1/d9f9acbc-329c-475e-8d81-03b20280a3e1 externally provisioned examplecluster-compute-0 Running 165m openshift-compute-0 baremetalhost:///openshift-machine-api/openshift-compute-0/3d685b81-7410-4bb3-80ec-13a31858241f provisioned examplecluster-compute-1 Running 165m openshift-compute-1 baremetalhost:///openshift-machine-api/openshift-compute-1/0fdae6eb-2066-4241-91dc-e7ea72ab13b9 provisioned
NAME PHASE TYPE REGION ZONE AGE NODE PROVIDERID STATE examplecluster-control-plane-0 Running 3h11m openshift-control-plane-0 baremetalhost:///openshift-machine-api/openshift-control-plane-0/da1ebe11-3ff2-41c5-b099-0aa41222964e externally provisioned examplecluster-control-plane-1 Running 3h11m openshift-control-plane-1 baremetalhost:///openshift-machine-api/openshift-control-plane-1/d9f9acbc-329c-475e-8d81-03b20280a3e1 externally provisioned examplecluster-compute-0 Running 165m openshift-compute-0 baremetalhost:///openshift-machine-api/openshift-compute-0/3d685b81-7410-4bb3-80ec-13a31858241f provisioned examplecluster-compute-1 Running 165m openshift-compute-1 baremetalhost:///openshift-machine-api/openshift-compute-1/0fdae6eb-2066-4241-91dc-e7ea72ab13b9 provisionedCopy to Clipboard Copied! Toggle word wrap Toggle overflow Copy to Clipboard Copied! Toggle word wrap Toggle overflow Copy to Clipboard Copied! Toggle word wrap Toggle overflow 참고중요Copy to Clipboard Copied! Toggle word wrap Toggle overflow oc get machines -n openshift-machine-api -o wide
$ oc get machines -n openshift-machine-api -o wideCopy to Clipboard Copied! Toggle word wrap Toggle overflow Copy to Clipboard Copied! Toggle word wrap Toggle overflow oc get bmh -n openshift-machine-api
$ oc get bmh -n openshift-machine-apiCopy to Clipboard Copied! Toggle word wrap Toggle overflow Copy to Clipboard Copied! Toggle word wrap Toggle overflow oc get nodes
$ oc get nodesCopy to Clipboard Copied! Toggle word wrap Toggle overflow Copy to Clipboard Copied! Toggle word wrap Toggle overflow
oc patch etcd/cluster --type=merge -p '{"spec": {"unsupportedConfigOverrides": null}}'$ oc patch etcd/cluster --type=merge -p '{"spec": {"unsupportedConfigOverrides": null}}'Copy to Clipboard Copied! Toggle word wrap Toggle overflow oc get etcd/cluster -oyaml
$ oc get etcd/cluster -oyamlCopy to Clipboard Copied! Toggle word wrap Toggle overflow EtcdCertSignerControllerDegraded: [Operation cannot be fulfilled on secrets "etcd-peer-sno-0": the object has been modified; please apply your changes to the latest version and try again, Operation cannot be fulfilled on secrets "etcd-serving-sno-0": the object has been modified; please apply your changes to the latest version and try again, Operation cannot be fulfilled on secrets "etcd-serving-metrics-sno-0": the object has been modified; please apply your changes to the latest version and try again]
EtcdCertSignerControllerDegraded: [Operation cannot be fulfilled on secrets "etcd-peer-sno-0": the object has been modified; please apply your changes to the latest version and try again, Operation cannot be fulfilled on secrets "etcd-serving-sno-0": the object has been modified; please apply your changes to the latest version and try again, Operation cannot be fulfilled on secrets "etcd-serving-metrics-sno-0": the object has been modified; please apply your changes to the latest version and try again]Copy to Clipboard Copied! Toggle word wrap Toggle overflow
oc -n openshift-etcd get pods -l k8s-app=etcd
$ oc -n openshift-etcd get pods -l k8s-app=etcdCopy to Clipboard Copied! Toggle word wrap Toggle overflow etcd-openshift-control-plane-0 5/5 Running 0 105m etcd-openshift-control-plane-1 5/5 Running 0 107m etcd-openshift-control-plane-2 5/5 Running 0 103m
etcd-openshift-control-plane-0 5/5 Running 0 105m etcd-openshift-control-plane-1 5/5 Running 0 107m etcd-openshift-control-plane-2 5/5 Running 0 103mCopy to Clipboard Copied! Toggle word wrap Toggle overflow oc patch etcd cluster -p='{"spec": {"forceRedeploymentReason": "recovery-'"$( date --rfc-3339=ns )"'"}}' --type=merge$ oc patch etcd cluster -p='{"spec": {"forceRedeploymentReason": "recovery-'"$( date --rfc-3339=ns )"'"}}' --type=merge1 Copy to Clipboard Copied! Toggle word wrap Toggle overflow oc rsh -n openshift-etcd etcd-openshift-control-plane-0
$ oc rsh -n openshift-etcd etcd-openshift-control-plane-0Copy to Clipboard Copied! Toggle word wrap Toggle overflow etcdctl member list -w table
sh-4.2# etcdctl member list -w tableCopy to Clipboard Copied! Toggle word wrap Toggle overflow Copy to Clipboard Copied! Toggle word wrap Toggle overflow 참고etcdctl endpoint health --cluster
# etcdctl endpoint health --clusterCopy to Clipboard Copied! Toggle word wrap Toggle overflow https://192.168.10.10:2379 is healthy: successfully committed proposal: took = 8.973065ms https://192.168.10.9:2379 is healthy: successfully committed proposal: took = 11.559829ms https://192.168.10.11:2379 is healthy: successfully committed proposal: took = 11.665203ms
https://192.168.10.10:2379 is healthy: successfully committed proposal: took = 8.973065ms https://192.168.10.9:2379 is healthy: successfully committed proposal: took = 11.559829ms https://192.168.10.11:2379 is healthy: successfully committed proposal: took = 11.665203msCopy to Clipboard Copied! Toggle word wrap Toggle overflow oc get etcd -o=jsonpath='{range.items[0].status.conditions[?(@.type=="NodeInstallerProgressing")]}{.reason}{"\n"}{.message}{"\n"}'$ oc get etcd -o=jsonpath='{range.items[0].status.conditions[?(@.type=="NodeInstallerProgressing")]}{.reason}{"\n"}{.message}{"\n"}'Copy to Clipboard Copied! Toggle word wrap Toggle overflow AllNodesAtLatestRevision
AllNodesAtLatestRevisionCopy to Clipboard Copied! Toggle word wrap Toggle overflow