Troubleshooting

Procedure

• Run the following command to check the version information:

  $ microshift version

  Example output

  Red Hat build of MicroShift Version: 4.17-0.microshift-e6980e25
  Base OCP Version: 4.17

Procedure

• To get the version number using the OpenShift CLI (oc), view the kube-public/microshift-version config map by running the following command:

  $ oc get configmap -n kube-public microshift-version -o yaml

  Example output

  apiVersion: v1
  data:
    major: "4"
    minor: "13"
    version: 4.13.8-0.microshift-fa441af87431
  kind: ConfigMap
  metadata:
    creationTimestamp: "2023-08-03T21:06:11Z"
    name: microshift-version
    namespace: kube-public

Procedure

• To display the base database version information, run the following command:

  $ microshift-etcd version

  Example output

  microshift-etcd Version: 4.17.1
  Base etcd Version: 3.5.13

• To display the full database version information, run the following command:

  $ microshift-etcd version -o json

  Example output

  {
    "major": "4",
    "minor": "16",
    "gitVersion": "4.17.1~rc.1",
    "gitCommit": "140777711962eb4e0b765c39dfd325fb0abb3622",
    "gitTreeState": "clean",
    "buildDate": "2024-05-10T16:37:53Z",
    "goVersion": "go1.21.9"
    "compiler": "gc",
    "platform": "linux/amd64",
    "patch": "",
    "etcdVersion": "3.5.13"
  }

Procedure

• Check the system status, which returns the cluster status, by running the following command:

  $ sudo systemctl status microshift

  If MicroShift fails to start, this command returns the logs from the previous run.

  Example healthy output

  ● microshift.service - MicroShift
       Loaded: loaded (/usr/lib/systemd/system/microshift.service; enabled; preset: disabled)
       Active: active (running) since <day> <date> 12:39:06 UTC; 47min ago
     Main PID: 20926 (microshift)
        Tasks: 14 (limit: 48063)
       Memory: 542.9M
          CPU: 2min 41.185s
       CGroup: /system.slice/microshift.service
               └─20926 microshift run
  
  <Month-Day> 13:23:06 i-06166fbb376f14a8b.<hostname> microshift[20926]: kube-apiserver I0528 13:23:06.876001   20926 controll>
  <Month-Day> 13:23:06 i-06166fbb376f14a8b.<hostname> microshift[20926]: kube-apiserver I0528 13:23:06.876574   20926 controll>
  # ...

• Optional: Get comprehensive logs by running the following command:

  $ sudo journalctl -u microshift

  Note

  The default configuration of the systemd journal service stores data in a volatile directory. To persist system logs across system starts and restarts, enable log persistence and set limits on the maximum journal data size.

• Optional: If MicroShift is running, check the status of active pods by entering the following command:

  $ oc get pods -A

  Example output

  NAMESPACE                   NAME                                                     READY   STATUS   RESTARTS  AGE
  default                     i-06166fbb376f14a8bus-west-2computeinternal-debug-qtwcr  1/1     Running  0		    46m
  kube-system                 csi-snapshot-controller-5c6586d546-lprv4                 1/1     Running  0		    51m
  kube-system                 csi-snapshot-webhook-6bf8ddc7f5-kz6k9                    1/1     Running  0		    51m
  openshift-dns               dns-default-45jl7                                        2/2     Running  0		    50m
  openshift-dns               node-resolver-7wmzf                                      1/1     Running  0		    51m
  openshift-ingress           router-default-78b86fbf9d-qvj9s                          1/1     Running  0		    51m
  openshift-ovn-kubernetes    ovnkube-master-5rfhh                                     4/4     Running  0		    51m
  openshift-ovn-kubernetes    ovnkube-node-gcnt6                                       1/1     Running  0		    51m
  openshift-service-ca        service-ca-bf5b7c9f8-pn6rk                               1/1     Running  0		    51m
  openshift-storage           topolvm-controller-549f7fbdd5-7vrmv                      5/5     Running  0		    51m
  openshift-storage           topolvm-node-rht2m                                       3/3     Running  0		    50m

  Note

  This example output shows basic MicroShift. If you have installed optional RPMs, the status of pods running those services is also expected to be shown in your output.

Prerequisites

• You must have the sos package installed.

• Not waiting several minutes after a system start to successfully stop MicroShift. The system must complete health checks and any other background processes before a back up can succeed.

• If MicroShift stopped running because of an error, you cannot perform a backup of the data.

  • Make sure the system is healthy.
  • Stop it in a healthy state before attempting a backup.
• If you do not have sufficient storage for the data, the backup fails. Ensure that you have enough storage for the MicroShift data.
• If you do not have sufficient permissions, a backup can fail. Ensure you have the correct user permissions to create a backup and perform the required configurations.

• Logs print to the terminal console during manual backups.

• Logs are automatically generated for rpm-ostree system automated backups as part of the MicroShift journal logs. You can check the logs by running the following command:

  $ sudo journalctl -u microshift

Procedure

• Get comprehensive MicroShift journal logs by running the following command:

  $ sudo journalctl -u microshift

• Check the Greenboot journal logs by running the following command:

  $ sudo journalctl -u greenboot-healthcheck

• Examining the comprehensive logs of a specific boot uses three steps. First list the boots, then select the one you want from the list you obtained:

  • List the boots present in the journal logs by running the following command:

    $ sudo journalctl --list-boots

    Example output

    IDX  BOOT ID                          	FIRST ENTRY                 LAST ENTRY
     0   681ece6f5c3047e183e9d43268c5527f 	<Day> <Date> 12:27:58 UTC 	<Day> <Date>> 13:39:41 UTC
    #....

  • Check the journal logs for the specific boot you want by running the following command:

    $ sudo journalctl --boot <idx_or_boot_id> 1

    1

    Replace <idx_or_boot_id> with the IDX or the BOOT ID number assigned to the specific boot that you want to check.

  • Check the journal logs for the boot of a specific service by running the following command:

    $ sudo journalctl --boot <idx_or_boot_id> -u <service_name> 1 2

    1

    Replace <idx_or_boot_id> with the IDX or the BOOT ID number assigned to the specific boot that you want to check.

    2

    Replace <service_name> with the name of the service that you want to check.

Procedure

• To see a report of health check status, use the following command:

  $ systemctl show --property=SubState --value greenboot-healthcheck.service

  • An output of start means that greenboot checks are still running.
  • An output of exited means that checks have passed and greenboot has exited. Greenboot runs the scripts in the green.d directory when the system is a healthy state.
  • An output of failed means that checks have not passed. Greenboot runs the scripts in red.d directory when the system is in this state and might restart the system.

• To see a report showing the numerical exit code of the service where 0 means success and non-zero values mean a failure occurred, use the following command:

  $ systemctl show --property=ExecMainStatus --value greenboot-healthcheck.service

• To see a report showing a message about boot status, such as Boot Status is GREEN - Health Check SUCCESS, use the following command:

  $ cat /run/motd.d/boot-status

Prerequisites

• You have installed jq.
• You have access to the cluster as a user with the cluster-admin role.

Procedure

• Edit the /etc/microshift/config.yaml file to set the memoryLimitMB value.

  etcd:
    memoryLimitMB: 128

  Note

  The minimum required value for memoryLimitMB on MicroShift is 128 MB. Values close to the minimum value are more likely to impact etcd performance. The lower the limit, the longer etcd takes to respond to queries. If the limit is too low or the etcd usage is high, queries time out.

• Stop and disable all MicroShift services
• Delete all MicroShift pods
• Delete all container image storage
• Reset network configuration
• Delete the /var/lib/microshift data directory
• Delete OVN-K networking configuration

Prerequisites

• You are logged into MicroShift as an administrator with root-user access.
• You have filed a support case.

• Stop and disable all MicroShift services
• Delete all MicroShift pods
• Reset network configuration
• Delete the /var/lib/microshift data directory
• Delete OVN-K networking configuration

Prerequisites

• You are logged into MicroShift as an administrator with root-user access.
• You have filed a support case.

• Stop all MicroShift services
• Delete all MicroShift pods
• Delete OVN-K networking configuration

Prerequisites

• You are logged into MicroShift as an administrator with root-user access.
• You have filed a support case.

• Stop all MicroShift services
• Delete all MicroShift pods
• Delete all MicroShift certificates

Prerequisites

• You are logged into MicroShift as an administrator with root-user access.
• You have filed a support case.