Chapter 45. Known Issues
RHEV OVA images of Atomic Host do not work
RHEV OVA images of Atomic Host currently cannot be imported into RHEV.
See this Bugzilla for details.
podman
,buildah
, andscopeo
need full name of the imageCurrently, to work with an image using
podman
,buildah
, andscopeo
, you cannot use the image’s short name, such asrhel7-aarch64
. Instead, specify its full name, including the registry, for exampleregistry.access.redhat.com/rhel7-aarch64
:$ podman pull registry.access.redhat.com/rhel7-aarch64
etcd 3.2.22-24 image contains incorrect version of the binary
On Thursday January 31, 2019 Red Hat released to the public a version of etcd that was incorrectly labeled. This incorrect image was released in the
rhel-7-server-extras-rpms
channel and the Red Hat Container Catalog. Specifically, an etcd container labeled3.2.22-24
was released with etcd3.3.11
inside of the container image. By Tuesday February 5th, 2019 Red Hat realized the issue and reverted the etcd container and RPM channels back to the correct image. The etcd container 3.2.22-18 is now the correct container.OpenShift 3.0 to 3.9 uses RPMs by default for etcd installations. OpenShift 3.10 to 3.11 uses container images for etcd installations. Not all customers are affected by this issue. If you run an RPM-based etcd with a RHEL7 channel attached to the RHEL hosts and you issued a
yum update
for all RPMs installed on your etcd hosts between the 6 days of January 31 to Feb 5th, you have pulled the incorrect RPMs. Or, if you are running a container based etcd and you have upgraded your cluster, scaled up the etcd nodes or manually installed the latest etcd container image between the 6 days of January 31 to Feb 5th, you have pulled the incorrect container image.See this Knowledgebase article for details on this issue and for how to check whether your systems have been affected.
The
buildah
package is not included by defaultThe
buildah
package is not included by default in RHEL Atomic Host 7.5.3. To add it, run:# rpm-ostree install buildah
The current plan is to include the
buildah
package in RHEL Atomic Host 7.5.4, so it will not be necessary to install it separately.The checkpoint and restore feature of
podman
is not workingDue to a bug introduced in CRIU, the checkpoint and restore feature of
podman
is not working". Docker is not impacted.ostree
remote configuration might be missing on new installationsThe 'ostree' remote configuration might be missing on new installations of RHEL Atomic Host 7.5.0. Consequently, when the
rpm-ostreed
daemon starts, it does not find configuration of the remote, which causes therpm-ostree
command to hang.So far, this issue has been found on new Kickstart installations, but not on ISO or cloud installations.
To fix the problem, follow these steps:
Populate the
/etc/ostree/remotes.d/
directory with anostree
remote configuration. This configuration should match the remote in the.origin
file that is in/sysroot/ostree/deploy/rhel-atomic-host/deploy/
. Example contents of/etc/ostree/remotes.d/redhat.conf
:[remote "rhel-atomic-host-ostree"] url=file:///install/ostree/repo
Restart the
rpm-ostreed
service:# systemctl restart rpm-ostreed.service
Alternatively, you can fix the problem by simply registering the system with
subscription-manager
.
Containers running systemd do not work
Prior to Atomic Host 7.5.0, due to a bug, the
container_manage_cgroup
SELinux boolean permitted containers to modify cgroup settings whether the boolean is on or off. In 7.5.0, this has been fixed. Now, if you need to run containers with systemd, you need to set the boolean toon
:# setebool -P container_manage_cgroup on
See this Knowledgebase solution for more information.
Old LVM configuration file sometimes not available after upgrading
If an LVM operation happens during an Atomic Host upgrade, the old LVM configuration file might not be available after the upgrade. You would see this error message:
Failed to read modified config file 'lvm/...'
To work around this, ensure that no LVM operation happens during an upgrade.
A common LVM operation that might happen is thin-pool auto-extension. To prevent thin-pool auto-extension, upgrade as follows:
Disable auto-extension:
# lvchange --monitor n VG/ThinPoolLV
Upgrade:
atomic host upgrade
After upgrade or reboot, enable auto-extension:
# lvchange --monitor y VG/ThinPoolLV
In an extremely rare case, this scenario will break LVM. To allow recovery from broken LVM, back up
/etc/lvm
before upgrading.
The root partition might have too little space for upgrades
The default Atomic Host root partition might be too small for upgrades. To upgrade, you might need to expand the root logical volume. See these sections:
Alternatively, you can free space on the root partition by pruning the previous deployment.
For background information on the root partition, see Managing Storage in Red Hat Enterprise Linux Atomic Host.
atomic uninstall
uninstalls all sssd containersRunning this command on an sssd container:
$ atomic uninstall --name=container-name
incorrectly uninstalls not only the
container-name
sssd container, but all sssd containers.To mitigate this, do not uninstall an sssd container if you use any other sssd containers.
Cannot use memory cgroups without swap on IBM POWER8 series
The "runc exec" command on the little-endian variant of IBM Power Systems uses significantly more memory than on AMD64 and Intel 64. Therefore, to prevent running out of memory, do not set cgroup memory limit to less than 100 megabytes.
By default, no user namespaces are allowed
By default, the new 7.4 kernel restricts the number of user namespaces to 0.
To work around this, increase the user namespace limit:
# echo 15000 > /proc/sys/user/max_user_namespaces
Cockpit can start dockerd when using docker, but not docker-latest
Beginning with RHEL Atomic Host 7.3.5, service-related functions in Cockpit might not work as expected if you run with
docker-latest
instead ofdocker
. Notably, Cockpit fails to start thedocker
daemon when running withdocker-latest
.Exposing the docker daemon through a TCP port is not secure
The docker daemon does no authentication, so binding it to a TCP port would give root access to any process with access to that TCP port. Red Hat advices against binding docker to a TCP port. See Access port options for details.
atomic scan will try to connect to the Internet if you do not use atomic install first
When you install the openscap container image with the
atomic install
command, the/etc/oscapd/oscapd.ini
configuration file is placed on the host machine and gets exposed to the container. Theoscapd.ini
file contains the information about where to fetch Open Vulnerability and Assessment Language (OVAL) content from. The default setting is to use the CVE data from inside the container and won’t connect to the Internet unless you explicitly configure it so. When you do not useatomic install
and directly start scanning withatomic scan
, atomic will fetch the container and run it immediately ignoring the INSTALL label. This means that/etc/oscapd/oscapd.ini
won’t be placed on the host system and be exposed to the container and the default behavior of theopenscap-daemon
itself inside the container will be used. The default behavior is to download CVE data from Red Hat’s URL, connecting to the Internet. Because of this. it is recommended that you useatomic install
before scanning containers so that the settings from theopscapd.ini
file are used. If not, scanning will still work, but be aware of the difference in the behavior of the openscap-daemon in both cases.Red Hat Enterprise Linux Atomic Host does not support FIPS mode
FIPS mode cannot be enabled on RHEL Atomic Host.
Upgrade to 7.3 from release versions older than 7.2.7 fails with an error on Atomic Host
Attempting to upgrade from RHEL Atomic Host 7.2.6-1 or older to 7.3 fails with the following error:
"error: fsetxattr: Invalid argument"
There are three possible workarounds:
1) Disable SELinux and upgrade as usual:
# setenforce 0 # atomic host upgrade
2) Stop
rpm-ostreed
and change the SELinux context:# systemctl stop rpm-ostreed # cp /usr/libexec/rpm-ostreed /usr/local/bin/rpm-ostreed # chcon -t install_exec_t /usr/local/bin/rpm-ostreed # /usr/local/bin/rpm-ostreed # atomic host upgrade
3) Deploy Atomic Host 7.2.7 first and then upgrade:
# atomic host deploy 7.2.7 # systemctl reboot # atomic host upgrade
Atomic Host does not support
/usr
as a mount pointAtomic Host does not support
/usr
as a mount point. As a consequence, Anaconda could crash if such a partition layout is configured. To work around this issue, do not make/usr
a mount point.etcdctl backup now reuses backup of the previous etcd member to avoid data loss
Previously, a member failed to be added to the etcd cluster when the database size was more than 700 MB, resulting in data loss. To work around this usse, the
etcdctl backup
command has been extended with options to reuse backup of the previous etcd member.rhel-push-plugin service does not restart after package upgrade
The docker service requires rhel-push-plugin to be started before itself. However, after upgrading the docker and docker-rhel-push-plugin packages, the docker daemon restarts while using the already existing rhel-push-plugin service in memory without restarting it. To work around this issue, manually restart rhel-push-plugin first, and the docker service afterwards.
etcd will not start if its current version is older than the etcd cluster version
etcd checks if the etcd version is older than the etcd cluster version. If this is the case, etcd will not start and applications dependent on etcd can fail. This issue prevents RHEL Atomic Host from cleanly rolling back from version 7.2.6 to earlier versions.
In a kubernetes cluster, if the nodes are newer than the master, they may fail to start.
In a kubernetes cluster, if the master contains an older version of kubernetes than the nodes, the nodes may fail to start. To work around this issue, always upgrade the master nodes first. As a result, the cluster will continue to function as expected.
docker 1.10 introduced a secomp filter which will cause some syscalls to fail inside containers.
As a workaround, pass the --security-opt seccomp:unconfined option to docker when creating a container. Docker maintains a help page with a comprehensive list of blocked calls and the reasoning behind them, see https://docs.docker.com/engine/security/seccomp/. Note that the list is not entirely identical to what is blocked in Red Hat Enterprise Linux.
Upgrade of docker from 1.9 to 1.10 loses image metadata
Under certain circumstances, upgrading from docker 1.9 to docker 1.10 can result in a loss of docker image tag metadata. The underlying image layers remain intact and can be seen by running docker images -a. The metadata can be recovered, if it is present on a remote registry by simply re-running docker pull. This command will restore the metadata while avoiding a transfer of the already existing layer data.
Atomic Host installation offers BTRFS but it is not supported.
The RHEL Atomic Host installer offers BTRFS as a partition option, but the tree does not include btrfs-progs. Consequently, if you choose this option in the installer, you will not be able to proceed with the installation until you choose another option.
When the root partition runs out of free space
RHEL Atomic Host allocates 3GB of storage to the root partition, which includes the docker volumes (units of storage that a running container can request from the host system). This makes it easy for the root partition to run out of storage space. If insufficient space is available, upgrading with
atomic host upgrade
will fail. In order to support more volume space, more physical storage must be added to the system, or the root Logical Volume must be extended. By default, 40% from the other volume, will be reserved for storing the container images. The other 60% can be used to extend the root partition. For detailed instructions, see https://access.redhat.com/documentation/en/red-hat-enterprise-linux-atomic-host/version-7/getting-started-with-containers/#changing_the_size_of_the_root_partition_after_installation.Rescue mode does not work in RHEL Atomic Host.
The Anaconda installer is unable to find a previously installed Atomic Host system when in rescue mode. Consequently, rescue mode does not work and should not be used.
The brandbot.path service may cause subscription-manager to change the /etc/os-release file in 7.1 installations.
The /etc/os-release file may still specify the 7.1 version even after Atomic Host has been upgraded to 7.2 using the atomic host upgrade command. This occurs because the underlying ostree tool preserves modified files in /etc. As a workaround, after upgrading to 7.2, run the following command:
cp /usr/etc/os-release /etc
This way, the /etc/os-release file will return to an unmodified state, and because
brandbot.path
is masked in 7.2.0, it will not be modified in the future by subscription-manager, and future upgrades will show the correct version.When running kube-apiserver on port 443 in secure mode, some capabilities are missing.
As a workaround, the kube-apiserver binary has to be modified by running
# chown root:root /usr/bin/kube-apiserver # chmod 700 /usr/bin/kube-apiserver # setcap CAP_NET_BIND_SERVICE=ep /usr/bin/kube-apiserver