Chapter 45. Known Issues

RHEV OVA images of Atomic Host do not work

RHEV OVA images of Atomic Host currently cannot be imported into RHEV.

See this Bugzilla for details.

podman, buildah, and scopeo need full name of the image

Currently, to work with an image using podman, buildah, and scopeo, you cannot use the image’s short name, such as rhel7-aarch64. Instead, specify its full name, including the registry, for example registry.access.redhat.com/rhel7-aarch64:

$ podman pull registry.access.redhat.com/rhel7-aarch64

etcd 3.2.22-24 image contains incorrect version of the binary

On Thursday January 31, 2019 Red Hat released to the public a version of etcd that was incorrectly labeled. This incorrect image was released in the rhel-7-server-extras-rpms channel and the Red Hat Container Catalog. Specifically, an etcd container labeled 3.2.22-24 was released with etcd 3.3.11 inside of the container image. By Tuesday February 5th, 2019 Red Hat realized the issue and reverted the etcd container and RPM channels back to the correct image. The etcd container 3.2.22-18 is now the correct container.

OpenShift 3.0 to 3.9 uses RPMs by default for etcd installations. OpenShift 3.10 to 3.11 uses container images for etcd installations. Not all customers are affected by this issue. If you run an RPM-based etcd with a RHEL7 channel attached to the RHEL hosts and you issued a yum update for all RPMs installed on your etcd hosts between the 6 days of January 31 to Feb 5th, you have pulled the incorrect RPMs. Or, if you are running a container based etcd and you have upgraded your cluster, scaled up the etcd nodes or manually installed the latest etcd container image between the 6 days of January 31 to Feb 5th, you have pulled the incorrect container image.

See this Knowledgebase article for details on this issue and for how to check whether your systems have been affected.

The buildah package is not included by default

The buildah package is not included by default in RHEL Atomic Host 7.5.3. To add it, run:

# rpm-ostree install buildah

The current plan is to include the buildah package in RHEL Atomic Host 7.5.4, so it will not be necessary to install it separately.

The checkpoint and restore feature of podman is not working

Due to a bug introduced in CRIU, the checkpoint and restore feature of podman is not working". Docker is not impacted.

ostree remote configuration might be missing on new installations

The 'ostree' remote configuration might be missing on new installations of RHEL Atomic Host 7.5.0. Consequently, when the rpm-ostreed daemon starts, it does not find configuration of the remote, which causes the rpm-ostree command to hang.

So far, this issue has been found on new Kickstart installations, but not on ISO or cloud installations.

To fix the problem, follow these steps:

Containers running systemd do not work

Prior to Atomic Host 7.5.0, due to a bug, the container_manage_cgroup SELinux boolean permitted containers to modify cgroup settings whether the boolean is on or off. In 7.5.0, this has been fixed. Now, if you need to run containers with systemd, you need to set the boolean to on:

# setebool -P container_manage_cgroup on

See this Knowledgebase solution for more information.

Old LVM configuration file sometimes not available after upgrading

If an LVM operation happens during an Atomic Host upgrade, the old LVM configuration file might not be available after the upgrade. You would see this error message:

Failed to read modified config file 'lvm/...'

To work around this, ensure that no LVM operation happens during an upgrade.

A common LVM operation that might happen is thin-pool auto-extension. To prevent thin-pool auto-extension, upgrade as follows:

The root partition might have too little space for upgrades

The default Atomic Host root partition might be too small for upgrades. To upgrade, you might need to expand the root logical volume. See these sections:

Alternatively, you can free space on the root partition by pruning the previous deployment.

For background information on the root partition, see Managing Storage in Red Hat Enterprise Linux Atomic Host.

atomic uninstall uninstalls all sssd containers

Running this command on an sssd container:

$ atomic uninstall --name=container-name

incorrectly uninstalls not only the container-name sssd container, but all sssd containers.

To mitigate this, do not uninstall an sssd container if you use any other sssd containers.

Cannot use memory cgroups without swap on IBM POWER8 series

The "runc exec" command on the little-endian variant of IBM Power Systems uses significantly more memory than on AMD64 and Intel 64. Therefore, to prevent running out of memory, do not set cgroup memory limit to less than 100 megabytes.

By default, no user namespaces are allowed

By default, the new 7.4 kernel restricts the number of user namespaces to 0.

To work around this, increase the user namespace limit:

# echo 15000 > /proc/sys/user/max_user_namespaces

Cockpit can start dockerd when using docker, but not docker-latest

Beginning with RHEL Atomic Host 7.3.5, service-related functions in Cockpit might not work as expected if you run with docker-latest instead of docker. Notably, Cockpit fails to start the docker daemon when running with docker-latest.

Exposing the docker daemon through a TCP port is not secure

The docker daemon does no authentication, so binding it to a TCP port would give root access to any process with access to that TCP port. Red Hat advices against binding docker to a TCP port. See Access port options for details.

atomic scan will try to connect to the Internet if you do not use atomic install first

When you install the openscap container image with the atomic install command, the /etc/oscapd/oscapd.ini configuration file is placed on the host machine and gets exposed to the container. The oscapd.ini file contains the information about where to fetch Open Vulnerability and Assessment Language (OVAL) content from. The default setting is to use the CVE data from inside the container and won’t connect to the Internet unless you explicitly configure it so. When you do not use atomic install and directly start scanning with atomic scan, atomic will fetch the container and run it immediately ignoring the INSTALL label. This means that /etc/oscapd/oscapd.ini won’t be placed on the host system and be exposed to the container and the default behavior of the openscap-daemon itself inside the container will be used. The default behavior is to download CVE data from Red Hat’s URL, connecting to the Internet. Because of this. it is recommended that you use atomic install before scanning containers so that the settings from the opscapd.ini file are used. If not, scanning will still work, but be aware of the difference in the behavior of the openscap-daemon in both cases.

Red Hat Enterprise Linux Atomic Host does not support FIPS mode

FIPS mode cannot be enabled on RHEL Atomic Host.

Upgrade to 7.3 from release versions older than 7.2.7 fails with an error on Atomic Host

Attempting to upgrade from RHEL Atomic Host 7.2.6-1 or older to 7.3 fails with the following error:

"error: fsetxattr: Invalid argument"

There are three possible workarounds:

1) Disable SELinux and upgrade as usual:

# setenforce 0
# atomic host upgrade

2) Stop rpm-ostreed and change the SELinux context:

#	systemctl stop rpm-ostreed
#	cp /usr/libexec/rpm-ostreed /usr/local/bin/rpm-ostreed
#	chcon -t install_exec_t /usr/local/bin/rpm-ostreed
#	/usr/local/bin/rpm-ostreed
#	atomic host upgrade

3) Deploy Atomic Host 7.2.7 first and then upgrade:

#	 atomic host deploy 7.2.7
#	 systemctl reboot
#	 atomic host upgrade

Atomic Host does not support /usr as a mount point

Atomic Host does not support /usr as a mount point. As a consequence, Anaconda could crash if such a partition layout is configured. To work around this issue, do not make /usr a mount point.

etcdctl backup now reuses backup of the previous etcd member to avoid data loss

Previously, a member failed to be added to the etcd cluster when the database size was more than 700 MB, resulting in data loss. To work around this usse, the etcdctl backup command has been extended with options to reuse backup of the previous etcd member.

rhel-push-plugin service does not restart after package upgrade

The docker service requires rhel-push-plugin to be started before itself. However, after upgrading the docker and docker-rhel-push-plugin packages, the docker daemon restarts while using the already existing rhel-push-plugin service in memory without restarting it. To work around this issue, manually restart rhel-push-plugin first, and the docker service afterwards.

etcd will not start if its current version is older than the etcd cluster version

etcd checks if the etcd version is older than the etcd cluster version. If this is the case, etcd will not start and applications dependent on etcd can fail. This issue prevents RHEL Atomic Host from cleanly rolling back from version 7.2.6 to earlier versions.

In a kubernetes cluster, if the nodes are newer than the master, they may fail to start.

In a kubernetes cluster, if the master contains an older version of kubernetes than the nodes, the nodes may fail to start. To work around this issue, always upgrade the master nodes first. As a result, the cluster will continue to function as expected.

docker 1.10 introduced a secomp filter which will cause some syscalls to fail inside containers.

As a workaround, pass the --security-opt seccomp:unconfined option to docker when creating a container. Docker maintains a help page with a comprehensive list of blocked calls and the reasoning behind them, see https://docs.docker.com/engine/security/seccomp/. Note that the list is not entirely identical to what is blocked in Red Hat Enterprise Linux.

Upgrade of docker from 1.9 to 1.10 loses image metadata

Under certain circumstances, upgrading from docker 1.9 to docker 1.10 can result in a loss of docker image tag metadata. The underlying image layers remain intact and can be seen by running docker images -a. The metadata can be recovered, if it is present on a remote registry by simply re-running docker pull. This command will restore the metadata while avoiding a transfer of the already existing layer data.

Atomic Host installation offers BTRFS but it is not supported.

The RHEL Atomic Host installer offers BTRFS as a partition option, but the tree does not include btrfs-progs. Consequently, if you choose this option in the installer, you will not be able to proceed with the installation until you choose another option.

When the root partition runs out of free space

RHEL Atomic Host allocates 3GB of storage to the root partition, which includes the docker volumes (units of storage that a running container can request from the host system). This makes it easy for the root partition to run out of storage space. If insufficient space is available, upgrading with atomic host upgrade will fail. In order to support more volume space, more physical storage must be added to the system, or the root Logical Volume must be extended. By default, 40% from the other volume, will be reserved for storing the container images. The other 60% can be used to extend the root partition. For detailed instructions, see https://access.redhat.com/documentation/en/red-hat-enterprise-linux-atomic-host/version-7/getting-started-with-containers/#changing_the_size_of_the_root_partition_after_installation.

Rescue mode does not work in RHEL Atomic Host.

The Anaconda installer is unable to find a previously installed Atomic Host system when in rescue mode. Consequently, rescue mode does not work and should not be used.

The brandbot.path service may cause subscription-manager to change the /etc/os-release file in 7.1 installations.

The /etc/os-release file may still specify the 7.1 version even after Atomic Host has been upgraded to 7.2 using the atomic host upgrade command. This occurs because the underlying ostree tool preserves modified files in /etc. As a workaround, after upgrading to 7.2, run the following command:

cp /usr/etc/os-release /etc

This way, the /etc/os-release file will return to an unmodified state, and because brandbot.path is masked in 7.2.0, it will not be modified in the future by subscription-manager, and future upgrades will show the correct version.

When running kube-apiserver on port 443 in secure mode, some capabilities are missing.

As a workaround, the kube-apiserver binary has to be modified by running

# chown root:root /usr/bin/kube-apiserver
# chmod 700 /usr/bin/kube-apiserver
# setcap CAP_NET_BIND_SERVICE=ep /usr/bin/kube-apiserver