1.24.2. Resolving the problem: PVC does not change after editing statefulSetSize
If you have this problem, update the storage
parameter in each data-observability-observatorium-thanos-receive-default-x
PVC. Complete the following steps:
Confirm that the disk space is full by running a query.
- Log in to the Red Hat OpenShift Container Platform console.
- From the navigation menu, click Monitoring > Metrics.
Enter the following query in the Expression window and sort the query to ascend by the Volume column:
kubelet_volume_stats_available_bytes{namespace="open-cluster-management-observability"}/kubelet_volume_stats_capacity_bytes{namespace="open-cluster-management-observability"}
kubelet_volume_stats_available_bytes{namespace="open-cluster-management-observability"}/kubelet_volume_stats_capacity_bytes{namespace="open-cluster-management-observability"}
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Note: If the value is 0, then the disk is full. If the disk is full continue with the following tasks.
Expand the
data-observability-observatorium-thanos-receive-default-x
PVC to update thestorage
parameter to a greater value than thestatefulSetSize
value from the MultiClusterObservability (mco) CR. Run the following command for eachdata-observability-observatorium-thanos-receive-default-x
PVC:kubectl get pvc data-observability-observatorium-thanos-receive-default-0 -o yaml
kubectl get pvc data-observability-observatorium-thanos-receive-default-0 -o yaml
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Your
data-observability-observatorium-thanos-receive-default-x
PVC might resemble the following content:Copy to Clipboard Copied! Toggle word wrap Toggle overflow This might take some time to work. Your changes go into effect when the value of
storage
andstatus
match. Run the previous command to check.Verify the fix by completing the following steps:
- Log in to your Red Hat Advanced Cluster Management console.
- From the navigation menu, select Observe environments > Overview.
- Click the Grafana link that is near the console header to view the metrics from your managed clusters.
-
Check the
metrics-collector-deployment-x
pod logs. When the error is fixed in the logs, the following message appears:Metrics pushed successfully
. Confirm if the disk space is no longer full by running a query from your Red Hat OpenShift Container Platform console. Enter the following query in the Expression window and sort the query to ascend by the Volume column:
kubelet_volume_stats_available_bytes{namespace="open-cluster-management-observability"}/kubelet_volume_stats_capacity_bytes{namespace="open-cluster-management-observability"}
kubelet_volume_stats_available_bytes{namespace="open-cluster-management-observability"}/kubelet_volume_stats_capacity_bytes{namespace="open-cluster-management-observability"}
Copy to Clipboard Copied! Toggle word wrap Toggle overflow