Upgrading connected Red Hat Satellite to 6.16
Upgrade Satellite Server and Capsule
Abstract
Providing feedback on Red Hat documentation
We appreciate your feedback on our documentation. Let us know how we can improve it.
Use the Create Issue form in Red Hat Jira to provide your feedback. The Jira issue is created in the Red Hat Satellite Jira project, where you can track its progress.
Prerequisites
- Ensure you have registered a Red Hat account.
Procedure
- Click the following link: Create Issue. If Jira displays a login error, log in and proceed after you are redirected to the form.
- Complete the Summary and Description fields. In the Description field, include the documentation URL, chapter or section number, and a detailed description of the issue. Do not modify any other fields in the form.
- Click Create.
Chapter 1. Upgrading overview
Review prerequisites and available upgrade paths below before upgrading your current Red Hat Satellite installation to Red Hat Satellite 6.16.
For interactive upgrade instructions, you can also use the Red Hat Satellite Upgrade Helper on the Red Hat Customer Portal. This application provides you with an exact guide to match your current version number. You can find instructions that are specific to your upgrade path, as well as steps to prevent known issues. For more information, see Satellite Upgrade Helper on the Red Hat Customer Portal.
1.1. Upgrade paths
You can upgrade to Red Hat Satellite 6.16 from Red Hat Satellite 6.15. For complete instructions on how to upgrade, see Chapter 2, Upgrading Red Hat Satellite.
The high-level steps in upgrading Satellite to 6.16 are as follows:
- Ensure that your Satellite Servers and Capsule Servers have been upgraded to Satellite 6.15. For more information, see Upgrading connected Red Hat Satellite to 6.15 or Upgrading disconnected Red Hat Satellite to 6.15.
Upgrade your Satellite Server:
- Upgrade your Satellite Server to 6.16.
Optional: Upgrade the operating system on your Satellite Server to Red Hat Enterprise Linux 9.
NoteAlthough upgrading the operating system of your Satellite Server to Red Hat Enterprise Linux 9 is optional, you will need to do it before you can upgrade to the next Satellite version after 6.16.
- Synchronize the new 6.16 repositories.
Upgrade your Capsule Servers:
- Upgrade all Capsule Servers to 6.16.
Optional: Upgrade the operating system on your Capsule Servers to Red Hat Enterprise Linux 9.
NoteAlthough upgrading the operating system of your Capsule Servers to Red Hat Enterprise Linux 9 is optional, you will need to do it before you can upgrade to the next Satellite version after 6.16.
Capsules at version 6.15 will keep working with your upgraded Satellite Server 6.16. After you upgrade Satellite Server to 6.16, you can upgrade your Capsules separately over multiple maintenance windows. For more information, see Section 1.3, “Upgrading Capsules separately from Satellite”.
Satellite services are shut down during the upgrade. Ensure to plan for the required downtime. The upgrade process duration might vary depending on your hardware configuration, network speed, and the amount of data that is stored on the server.
- Upgrading Satellite Server takes approximately 1 – 2 hours.
- Upgrading Capsule Server takes approximately 10 – 30 minutes.
Hammer and API considerations
If you have any scripts that use the Hammer CLI tool, ensure that you modify these scripts according to the changes in Hammer. If you have any integrations that use the Satellite REST API, ensure that you modify these integrations according to the changes in the API. For more information about changes in Hammer and API, see Release notes.
1.2. Prerequisites
Upgrading to Satellite 6.16 affects your entire Satellite infrastructure. Before proceeding, complete the following:
- Read the Red Hat Satellite 6.16 Release Notes.
- Ensure that you have sufficient storage space on your server. For more information, see Preparing your Environment for Installation in Installing Satellite Server in a connected network environment and Preparing your Environment for Installation in Installing Capsule Server.
-
Ensure that you have at least the same amount of free space on
/var/lib/pgsql
as that consumed by/var/lib/pgsql/data
. Upgrading to Satellite 6.16 involves a PostgreSQL 12 to PostgreSQL 13 upgrade. The contents of/var/lib/pgsql/data
are backed up during the PostgreSQL upgrade. - Back up your Satellite Server and all Capsule Servers. For more information, see Backing Up Satellite Server and Capsule Server in Administering Red Hat Satellite.
- Plan for updating any scripts you use that contain Satellite API commands because some API commands differ between versions of Satellite.
- Migrate all organizations to Simple Content Access (SCA). For more information, see the Red Hat Knowledgebase solution Simple Content Access.
Ensure that all Satellite Servers are on the same version.
If you customize configuration files, manually or using a tool such as Hiera, these changes are overwritten when the maintenance script runs during upgrading or updating. You can use the --noop
option with the satellite-installer to test for changes. For more information, see the Red Hat Knowledgebase solution How to use the noop option to check for changes in Satellite config files during an upgrade.
1.3. Upgrading Capsules separately from Satellite
You can upgrade Satellite to version 6.16 and keep Capsules at version 6.15 until you have the capacity to upgrade them too.
All the functionality that worked previously works on 6.15 Capsules. However, the functionality added in the 6.16 release will not work until you upgrade Capsules to 6.16.
Upgrading Capsules after upgrading Satellite can be useful in the following example scenarios:
- If you want to have several smaller outage windows instead of one larger window.
- If Capsules in your organization are managed by several teams and are located in different locations.
- If you use a load-balanced configuration, you can upgrade one load-balanced Capsule and keep other load-balanced Capsules at one version lower. This allows you to upgrade all Capsules one after another without any outage.
1.4. Following the progress of the upgrade
Because of the lengthy upgrade time, use a utility such as tmux
to suspend and reattach a communication session. You can then check the upgrade progress without staying connected to the command shell continuously. For more information, see the tmux
manual page.
If you lose connection to the command shell where the upgrade command is running you can see the logs in /var/log/foreman-installer/satellite.log
to check if the process completed successfully.
Chapter 2. Upgrading Red Hat Satellite
Use the following procedures to upgrade your existing Red Hat Satellite to Red Hat Satellite 6.16.
2.1. Satellite Server upgrade considerations
This section describes how to upgrade Satellite Server from 6.15 to 6.16. You can upgrade from any minor version of Satellite Server 6.15.
Before you begin
- Review Section 1.2, “Prerequisites”.
- Note that you can upgrade Capsules separately from Satellite. For more information, see Section 1.3, “Upgrading Capsules separately from Satellite”.
- Review and update your firewall configuration. For more information, see Preparing your environment for installation in Installing Satellite Server in a connected network environment.
- Ensure that you do not delete the manifest from the Customer Portal or in the Satellite web UI because this removes all the entitlements of your content hosts.
- If you have edited any of the default templates, back up the files either by cloning or exporting them. Cloning is the recommended method because that prevents them being overwritten in future updates or upgrades. To confirm if a template has been edited, you can view its History before you upgrade or view the changes in the audit log after an upgrade. In the Satellite web UI, navigate to Monitor > Audits and search for the template to see a record of changes made. If you use the export method, restore your changes by comparing the exported template and the default template, manually applying your changes.
- Optional: Clone your Satellite Server to test the upgrade. After you successfully test the upgrade on the clone, you can repeat the upgrade on your primary Satellite Server and discard the clone, or you can promote the clone to your primary Satellite Server and discard the previous primary Satellite Server. For more information, see Cloning Satellite Server in Administering Red Hat Satellite.
Capsule considerations
- If you use content views to control updates to a Capsule Server’s base operating system, or for Capsule Server repository, you must publish updated versions of those content views.
- Note that Satellite Server upgraded from 6.15 to 6.16 can use Capsule Servers still at 6.15.
If you implemented custom certificates, you must retain the content of both the /root/ssl-build
directory and the directory in which you created any source files associated with your custom certificates.
Failure to retain these files during an upgrade causes the upgrade to fail. If these files have been deleted, they must be restored from a backup in order for the upgrade to proceed.
Upgrade scenarios
You cannot upgrade a self-registered Satellite. You must migrate a self-registered Satellite to the Red Hat Content Delivery Network (CDN) and then perform the upgrade.
FIPS mode
You cannot upgrade Satellite Server from a RHEL base system that is not operating in FIPS mode to a RHEL base system that is operating in FIPS mode.
To run Satellite Server on a Red Hat Enterprise Linux base system operating in FIPS mode, you must install Satellite on a freshly provisioned RHEL base system operating in FIPS mode. For more information, see Preparing your environment for installation in Installing Satellite Server in a connected network environment.
2.2. Upgrading a connected Satellite Server
Use this procedure for a Satellite Server with access to the public internet
If you customize configuration files, manually or using a tool such as Hiera, these changes are overwritten when the maintenance script runs during upgrading or updating. You can use the --noop
option with the satellite-installer to test for changes. For more information, see the Red Hat Knowledgebase solution How to use the noop option to check for changes in Satellite config files during an upgrade.
Upgrade Satellite Server
Stop all Satellite services:
# satellite-maintain service stop
Take a snapshot or create a backup:
- On a virtual machine, take a snapshot.
- On a physical machine, create a backup.
Start all Satellite services:
# satellite-maintain service start
-
Optional: If you made manual edits to DNS or DHCP configuration in the
/etc/zones.conf
or/etc/dhcp/dhcpd.conf
files, back up the configuration files because the installer only supports one domain or subnet, and therefore restoring changes from these backups might be required. Optional: If you made manual edits to DNS or DHCP configuration files and do not want to overwrite the changes, enter the following command:
# satellite-installer \ --foreman-proxy-dhcp-managed=false \ --foreman-proxy-dns-managed=false
- In the Satellite web UI, navigate to Hosts > Discovered hosts. On the Discovered Hosts page, power off and then delete the discovered hosts. From the Select an Organization menu, select each organization in turn and repeat the process to power off and delete the discovered hosts. Make a note to reboot these hosts when the upgrade is complete.
Upgrade satellite-maintain to its next version:
# satellite-maintain self-upgrade
- If you are using an external database, upgrade your database to PostgreSQL 13.
Use the health check option to determine if the system is ready for upgrade. When prompted, enter the hammer admin user credentials to configure
satellite-maintain
with hammer credentials. These changes are applied to the/etc/foreman-maintain/foreman-maintain-hammer.yml
file.# satellite-maintain upgrade check
Review the results and address any highlighted error conditions before performing the upgrade.
Optional: Because of the lengthy upgrade time, use a utility such as
tmux
to suspend and reattach a communication session. You can then check the upgrade progress without staying connected to the command shell continuously.If you lose connection to the command shell where the upgrade command is running, you can see the logged messages in the
/var/log/foreman-installer/satellite.log
file to check if the process completed successfully.Perform the upgrade:
# satellite-maintain upgrade run
If the previous command told you to reboot, then reboot the system:
# reboot
Next steps
- Optional: Upgrade the operating system to Red Hat Enterprise Linux 9 on the upgraded Satellite Server. For more information, see Chapter 3, Upgrading Red Hat Enterprise Linux on Satellite or Capsule.
2.3. Synchronizing the new repositories
You must enable and synchronize the new 6.16 repositories before you can upgrade Capsule Servers and Satellite clients.
Procedure
- In the Satellite web UI, navigate to Content > Red Hat Repositories.
- Toggle the Recommended Repositories switch to the On position.
From the list of results, expand the following repositories and click the Enable icon to enable the repositories:
- To upgrade Satellite clients, enable the Red Hat Satellite Client 6 repositories for all Red Hat Enterprise Linux versions that clients use.
If you have Capsule Servers, to upgrade them, enable the following repositories too:
Red Hat Satellite Capsule 6.16 (for RHEL 8 x86_64) (RPMs)
Red Hat Satellite Maintenance 6.16 (for RHEL 8 x86_64) (RPMs)
Red Hat Enterprise Linux 8 (for x86_64 – BaseOS) (RPMs)
Red Hat Enterprise Linux 8 (for x86_64 – AppStream) (RPMs)
NoteIf the 6.16 repositories are not available, refresh the Red Hat Subscription Manifest. In the Satellite web UI, navigate to Content > Subscriptions, click Manage Manifest, then click Refresh.
- In the Satellite web UI, navigate to Content > Sync Status.
- Click the arrow next to the product to view the available repositories.
- Select the repositories for 6.16. Note that Red Hat Satellite Client 6 does not have a 6.16 version. Choose Red Hat Satellite Client 6 instead.
Click Synchronize Now.
ImportantIf an error occurs when you try to synchronize a repository, refresh the manifest. If the problem persists, raise a support request. Do not delete the manifest from the Customer Portal or in the Satellite web UI; this removes all the entitlements of your content hosts.
- If you use content views to control updates to the base operating system of Capsule Server, update those content views with new repositories, publish, and promote their updated versions. For more information, see Managing content views in Managing content.
2.4. Performing post-upgrade tasks
- Optional: If the default provisioning templates have been changed during the upgrade, recreate any templates cloned from the default templates. If the custom code is executed before and/or after the provisioning process, use custom provisioning snippets to avoid recreating cloned templates. For more information about configuring custom provisioning snippets, see Creating Custom Provisioning Snippets in Provisioning hosts.
- Pulp is introducing more data about container manifests to the API. This information allows Katello to display manifest labels, annotations, and information about the manifest type, such as if it is bootable or represents flatpak content. As a result, migrations must be performed to pull this content from manifests into the database.
This migration takes time, so a pre-migration runs automatically after the upgrade to 6.16 to reduce future upgrade downtime. While the pre-migration is running, Satellite Server is fully functional but uses more hardware resources.
2.5. Upgrading Capsule Servers
This section describes how to upgrade Capsule Servers from 6.15 to 6.16.
Before you begin
- Review Section 1.2, “Prerequisites”.
- You must upgrade Satellite Server before you can upgrade any Capsule Servers. Note that you can upgrade Capsules separately from Satellite. For more information, see Section 1.3, “Upgrading Capsules separately from Satellite”.
- Ensure the Red Hat Satellite Capsule 6.16 repository is enabled in Satellite Server and synchronized.
- Ensure that you synchronize the required repositories on Satellite Server. For more information, see Section 2.3, “Synchronizing the new repositories”.
- If you use content views to control updates to the base operating system of Capsule Server, update those content views with new repositories, publish, and promote their updated versions. For more information, see Managing content views in Managing content.
- Ensure the Capsule’s base system is registered to the newly upgraded Satellite Server.
- Ensure the Capsule has the correct organization and location settings in the newly upgraded Satellite Server.
- Review and update your firewall configuration prior to upgrading your Capsule Server. For more information, see Preparing Your Environment for Capsule Installation in Installing Capsule Server.
If you implemented custom certificates, you must retain the content of both the /root/ssl-build
directory and the directory in which you created any source files associated with your custom certificates.
Failure to retain these files during an upgrade causes the upgrade to fail. If these files have been deleted, they must be restored from a backup in order for the upgrade to proceed.
Upgrading Capsule Servers
Create a backup.
- On a virtual machine, take a snapshot.
On a physical machine, create a backup.
For information on backups, see Backing Up Satellite Server and Capsule Server in Administering Red Hat Satellite.
Clean yum cache:
# yum clean metadata
-
Synchronize the
satellite-capsule-6.16-for-rhel-8-x86_64-rpms
repository in the Satellite Server. - Publish and promote a new version of the content view with which the Capsule is registered.
Optional: Because of the lengthy upgrade time, use a utility such as
tmux
to suspend and reattach a communication session. You can then check the upgrade progress without staying connected to the command shell continuously.If you lose connection to the command shell where the upgrade command is running, you can see the logged messages in the
/var/log/foreman-installer/capsule.log
file to check if the process completed successfully.The
rubygem-foreman_maintain
is installed from the Satellite Maintenance repository or upgraded from the Satellite Maintenance repository if currently installed.Ensure Capsule has access to
satellite-maintenance-6.16-for-rhel-8-x86_64-rpms
and execute:# satellite-maintain self-upgrade
On Capsule Server, verify that the
foreman_url
setting points to the Satellite FQDN:# grep foreman_url /etc/foreman-proxy/settings.yml
Use the health check option to determine if the system is ready for upgrade:
# satellite-maintain upgrade check
Review the results and address any highlighted error conditions before performing the upgrade.
Perform the upgrade:
# satellite-maintain upgrade run
If the previous command told you to reboot, then reboot the system:
# reboot
- Optional: If you made manual edits to DNS or DHCP configuration files, check and restore any changes required to the DNS and DHCP configuration files using the backups made earlier.
- Optional: If you use custom repositories, ensure that you enable these custom repositories after the upgrade completes.
Upgrading Capsule Servers using remote execution
Create a backup or take a snapshot.
For more information on backups, see Backing Up Satellite Server and Capsule Server in Administering Red Hat Satellite.
- In the Satellite web UI, navigate to Monitor > Jobs.
- Click Run Job.
- From the Job category list, select Maintenance Operations.
- From the Job template list, select Capsule Upgrade Playbook.
- In the Search Query field, enter the host name of the Capsule.
- Ensure that Apply to 1 host is displayed in the Resolves to field.
- In the target_version field, enter the target version of the Capsule.
- In the whitelist_options field, enter the options.
- Select the schedule for the job execution in Schedule.
- In the Type of query section, click Static Query.
Next steps
- Optional: Upgrade the operating system to Red Hat Enterprise Linux 9 on the upgraded Satellite Server. For more information, see Chapter 3, Upgrading Red Hat Enterprise Linux on Satellite or Capsule.
2.6. Upgrading the external database
You can upgrade an external database from Red Hat Enterprise Linux 8 to Red Hat Enterprise Linux 9 while upgrading Satellite from 6.15 to 6.16.
Prerequisites
- Create a new Red Hat Enterprise Linux 9 based host for PostgreSQL server that follows the external database on Red Hat Enterprise Linux 9 documentation. For more information, see Using External Databases with Satellite.
- Install PostgreSQL version 13 on the new Red Hat Enterprise Linux host.
Procedure
- Create a backup.
- Restore the backup on the new server.
If Satellite reaches the new database server via the old name, no further changes are required. Otherwise reconfigure Satellite to use the new name:
# satellite-installer \ --foreman-db-host newpostgres.example.com \ --katello-candlepin-db-host newpostgres.example.com \ --foreman-proxy-content-pulpcore-postgresql-host newpostgres.example.com
Chapter 3. Upgrading Red Hat Enterprise Linux on Satellite or Capsule
Satellite and Capsule are supported on both Red Hat Enterprise Linux 8 and Red Hat Enterprise Linux 9. You can use the following methods to upgrade your Satellite or Capsule operating system from Red Hat Enterprise Linux 8 to Red Hat Enterprise Linux 9:
- Leapp in-place upgrade
- With Leapp, you can upgrade your Satellite or Capsule in-place therefore it is faster but imposes a downtime on the services.
- Migration by using cloning
- The Red Hat Enterprise Linux 8 system remains operational during the migration using cloning, which reduces the downtime. You cannot use cloning for Capsule Server migrations.
- Migration by using backup and restore
- The Red Hat Enterprise Linux 8 system remains operational during the migration using cloning, which reduces the downtime. You can use backup and restore for migrating both Satellite and Capsule operating system from Red Hat Enterprise Linux 8 to Red Hat Enterprise Linux 9.
3.1. Upgrading Satellite or Capsule to RHEL 9 in-place by using Leapp
You can use the Leapp tool to upgrade as well as to help detect and resolve issues that could prevent you from upgrading successfully.
Prerequisites
- Review known issues before you begin an upgrade. For more information, see Known issues in Red Hat Satellite 6.16.
- If you use an HTTP proxy in your environment, configure the Subscription Manager to use the HTTP proxy for connection. For more information, see Troubleshooting in Upgrading from RHEL 8 to RHEL 9.
- Satellite 6.16 or Capsule 6.16 running on Red Hat Enterprise Linux 8.
For Capsule upgrade, enable and synchronize the following repositories to Satellite Server:
- Red Hat Enterprise Linux 9 for x86_64 - BaseOS (RPMs)
-
rhel-9-for-x86_64-baseos-rpms
- Red Hat Enterprise Linux 9 for x86_64 - AppStream (RPMs)
-
rhel-9-for-x86_64-appstream-rpms
- Red Hat Satellite Capsule 6.16 for RHEL 9 x86_64 RPMs
-
satellite-capsule-6.16-for-rhel-9-x86_64-rpms
- Red Hat Satellite Maintenance 6.16 for RHEL 9 x86_64 RPMs
-
satellite-maintenance-6.16-for-rhel-9-x86_64-rpms
Add the repositories to the lifecycle environment and content view that is attached to your Capsule Server.
Procedure
Install required packages:
# satellite-maintain packages install leapp leapp-upgrade-el8toel9
Let Leapp analyze your system:
# leapp preupgrade
The first run will most likely report issues and inhibit the upgrade. Examine the report in the
/var/log/leapp/leapp-report.txt
file, answer all questions by usingleapp answer
, and manually resolve other reported problems.-
Run
leapp preupgrade
again and make sure that it does not report any more issues. Let Leapp create the upgrade environment:
# leapp upgrade
Reboot the system to start the upgrade.
After the system reboots, a live system conducts the upgrade, reboots to fix SELinux labels and then reboots into the final Red Hat Enterprise Linux 9 system.
Wait for Leapp to finish the upgrade. You can monitor the process with
journalctl
:# journalctl -u leapp_resume -f
Unlock packages:
# satellite-maintain packages unlock
- Verify the post-upgrade state. For more information, see Verifying the post-upgrade state in Upgrading from RHEL 8 to RHEL 9.
- Perform post-upgrade tasks on the RHEL 9 system. For more information, see Performing post-upgrade tasks on the RHEL 9 system in Upgrading from RHEL 8 to RHEL 9.
Lock packages:
# satellite-maintain packages lock
- Change SELinux to enforcing mode. For more information, see Changing SELinux mode to enforcing in Upgrading from RHEL 8 to RHEL 9.
Unset the
subscription-manager
release:# subscription-manager release --unset
Additional resources
- For more information on customizing the Leapp upgrade for your environment, see Customizing your Red Hat Enterprise Linux in-place upgrade.
3.2. Migrating Satellite to RHEL 9 by using cloning
You can clone your existing Satellite Server from Red Hat Enterprise Linux 8 to a freshly installed Red Hat Enterprise Linux 9 system. Create a backup of the existing Satellite Server, which you then clone on the new Red Hat Enterprise Linux 9 system.
You cannot use cloning for Capsule Server backups.
Procedure
- Perform a full backup of your Satellite Server. This is the source Red Hat Enterprise Linux 8 server that you are migrating. For more information, see Performing a full backup of Satellite Server in Administering Red Hat Satellite.
- Deploy a system with Red Hat Enterprise Linux 9 and the same configuration as the source server. This is the target server.
- Clone the server. Clone configures hostname for the target server. For more information, see Cloning Satellite Server in Administering Red Hat Satellite
3.3. Migrating Satellite or Capsule to RHEL 9 using backup and restore
You can migrate your existing Satellite Server and Capsule Server from Red Hat Enterprise Linux 8 to a freshly installed Red Hat Enterprise Linux 9 system. The migration involves creating a backup of the existing Satellite Server and Capsule Server, which you then restore on the new Red Hat Enterprise Linux 9 system.
Procedure
- Perform a full backup of your Satellite Server or Capsule. This is the source Red Hat Enterprise Linux 8 server that you are migrating. For more information, see Performing a full backup of Satellite Server or Capsule Server in Administering Red Hat Satellite.
- Deploy a system with Red Hat Enterprise Linux 9 and the same hostname and configuration as the source server. This is the target server.
- Restore the backup. Restore does not significantly alter the target system and requires additional configuration. For more information, see Restoring Satellite Server or Capsule Server from a backup in Administering Red Hat Satellite.
- Restore the Capsule Server backup. For more information, see Restoring Satellite Server or Capsule Server from a backup in Administering Red Hat Satellite.
Appendix A. Troubleshooting permission issues
Satellite upgrades perform pre-upgrade checks. If the pre-upgrade check discovers permission issues, it fails with an error similar to the following one:
2024-01-29T20:50:09 [W|app|] Could not create role 'Ansible Roles Manager': ERF73-0602 [Foreman::PermissionMissingException]: some permissions were not found:
If you see an error like this on your Satellite Server, identify and remedy the permission issues.
Procedure
On your Satellite Server, identify permission issues:
# satellite-maintain health check --label duplicate_permissions
Fix permission issues:
# foreman-rake db:seed
Verification
Rerun the check to ensure no permission issues remain:
# satellite-maintain health check --label duplicate_permissions