Upgrading and Updating Red Hat Satellite

1. Clone your existing Satellite Servers. For more information, see Chapter 2, Cloning Satellite Server.
2. Upgrade Satellite Server and all Capsule Servers to Satellite 6.10. For more information, see Section 3.1, “Upgrading Satellite Server”.
3. Upgrade to Satellite Tools 6.10 on all Satellite clients. For more information, see Section 3.5, “Upgrading Satellite Clients”.

1. If you want to have several smaller outage windows instead of one larger window.
2. If Capsules in your organization are managed by several teams and are located in different locations.
3. If you use a load-balanced configuration, you can upgrade one load-balanced Capsule and keep other load-balanced Capsules at 1 version lower. This allows you to upgrade all Capsules one after another without any outage.

1. Back up the source server.
2. Clone the source server to the target server.
3. Power off the source server.
4. Update the network configuration on the target server to match the target server’s IP address with its new host name.
5. Restart goferd in Content hosts and Capsules to refresh the connection.
6. Test the new target server.

1. Perform a backup with PostgreSQL databases active excluding the Pulp data:

   # satellite-maintain backup offline --skip-pulp-content \
   --assumeyes /var/backup

2. Stop and disable the satellite-maintain services

   # satellite-maintain service stop
   # satellite-maintain service disable

3. Copy the Pulp data to the target server:

   # rsync --archive --partial --progress --compress \
   /var/lib/pulp target_server.example.com:/var/lib/pulp

Procedure

1. In the Satellite web UI, navigate to Content > Red Hat Repositories.
2. Toggle the Recommended Repositories switch to the On position.

3. From the list of results, expand the following repositories and click the Enable icon to enable the repositories:

   • To upgrade Satellite clients, enable the Red Hat Satellite Tools 6.10 repositories for all Red Hat Enterprise Linux versions that clients use.

   • If you have Capsule Servers, to upgrade them, enable the following repositories too:

     Red Hat Satellite Capsule 6.10 (for RHEL 7 Server) (RPMs)

     Red Hat Satellite Maintenance 6 (for RHEL 7 Server) (RPMs)

     Red Hat Ansible Engine 2.9 RPMs for Red Hat Enterprise Linux 7 Server

     Red Hat Software Collections RPMs for Red Hat Enterprise Linux 7 Server

   Note

   If the 6.10 repositories are not available, refresh the Subscription Manifest. Navigate to Content > Subscriptions, click Manage Manifest, then click Refresh.

4. Navigate to Content > Sync Status.
5. Click the arrow next to the product to view the available repositories.
6. Select the repositories for 6.10.

7. Click Synchronize Now.

   Important

   If an error occurs when you try to synchronize a repository, refresh the manifest. If the problem persists, raise a support request. Do not delete the manifest from the Customer Portal or in the Satellite web UI; this removes all the entitlements of your content hosts.

8. If you use Content Views to control updates to the base operating system of Capsule Server, update those Content Views with new repositories, publish, and promote their updated versions. For more information, see Managing Content Views in the Content Management Guide.

Upgrading Capsule Servers

1. Create a backup.

   • On a virtual machine, take a snapshot.

   • On a physical machine, create a backup.

     For information on backups, see Backing Up Satellite Server and Capsule Server in the Administering Red Hat Satellite 6.9 guide.

2. Clean yum cache:

   # yum clean metadata

3. Ensure that the rubygem-foreman_maintain package that provides satellite-maintain is installed and up to date:

   # yum install rubygem-foreman_maintain

4. On Capsule Server, verify that the foreman_url setting points to the Satellite FQDN:

   # grep foreman_url /etc/foreman-proxy/settings.yml

5. Ensure that the apache::purge_configs: false entry is either not present or commented out in the /etc/foreman-installer/custom-hiera.yaml file of the Satellite 6.9/Capsule 6.9 servers which will be upgraded to 6.10.

6. Check the available versions to confirm the version you want is listed:

   # satellite-maintain upgrade list-versions

7. Because of the lengthy upgrade time, use a utility such as screen to suspend and reattach a communication session. You can then check the upgrade progress without staying connected to the command shell continuously. For more information about using the screen command, see How do I use the screen command? article in the Red Hat Knowledge Base.

   If you lose connection to the command shell where the upgrade command is running you can see the logged messages in the /var/log/foreman-installer/satellite.log file to check if the process completed successfully.

8. Use the health check option to determine if the system is ready for upgrade:

   # satellite-maintain upgrade check --target-version 6.10

   Review the results and address any highlighted error conditions before performing the upgrade.

9. Perform the upgrade:

   # satellite-maintain upgrade run --target-version 6.10

   Warning

   If you run the command from a directory containing a config subdirectory, you will encounter the following error:

   ERROR: Scenario (config/capsule.yaml) was not found, can not continue.

   In such a case, change directory, for example to the root user’s home directory, and run the command again.

10. If you have migrated content from Pulp 2 to Pulp 3, remove all Pulp 2 content.

    # satellite-maintain content remove-pulp2

    This removes Pulp 2 RPMs, content in /var/lib/pulp/content/, the mongo database, and migration content in the Pulp 3 database.

    Accessing content through this Capsule will fail until a full synchronization of the upgraded Capsule is performed.

11. Check when the kernel packages were last updated:

    # rpm -qa --last | grep kernel

12. Optional: If a kernel update occurred since the last reboot, reboot the system:

    # reboot

13. Optional: If you made manual edits to DNS or DHCP configuration files, check and restore any changes required to the DNS and DHCP configuration files using the backups made earlier.
14. Optional: If you use custom repositories, ensure that you enable these custom repositories after the upgrade completes.

15. On Satellite Server, perform a complete synchronization of the upgraded Capsule as the MongoDB and RPM repositories are not automatically migrated with Satellite.

    # hammer capsule content synchronize --name ${Capsule} --skip-metadata-check true --async

    Note

    If you did not synchronize your repositories before upgrading your Satellite Server, running this command fails to synchronize your content with Capsule Servers. In this case, follow Section 3.4, “Synchronizing Capsule Servers Through Satellite Web UI” to synchronize your Capsule Servers.

Upgrading Capsule Servers Using Remote Execution in the Satellite web UI

1. Create a backup.

   • On a virtual machine, take a snapshot.

   • On a physical machine, create a backup.

     For information on backups, see Backing Up Satellite Server and Capsule Server in the Administering Red Hat Satellite 6.9 guide.

2. Optional: If you made manual edits to DNS or DHCP configuration in the /etc/zones.conf or /etc/dhcp/dhcpd.conf files, back up the configuration files because the installer only supports one domain or subnet, and therefore restoring changes from these backups might be required.

3. Optional: If you made manual edits to DNS or DHCP configuration files and do not want to overwrite the changes, enter the following command:

   # {foreman-installer} --foreman-proxy-dns-managed=false \
   --foreman-proxy-dhcp-managed=false

4. In the Satellite web UI, navigate to Monitor > Jobs.
5. Click Run Job.
6. From the Job category list, select Maintenance Operations.
7. From the Job template list, select Capsule Upgrade Playbook.
8. In the Search Query field, enter the host name of the Capsule.
9. Ensure that Resolves to shows 1 host.
10. In the target_version field, enter the target version of the Capsule.
11. In the whitelist_options field, enter the whitelist options.
12. For Type of query, click Static Query or Dynamic Query depending on the type of query.
13. Select the schedule for the job execution in Schedule.

Procedure

1. In the Satellite web UI, navigate to Infrastructure > Capsules.
2. Click on the name of Capsule Server you wish to synchronize.
3. Click Synchronize.
4. Select Complete Sync.

Upgrade Satellite Clients Using the Bulk Repository Set UI:

1. In the Satellite web UI, navigate to Hosts > Content Hosts and select the Content Hosts that you want to upgrade.
2. From the Select Action list, select Manage Repository Sets.
3. From the Repository Sets Management list, select the Red Hat Satellite Tools 6.9 check box.
4. From the Select Action list, select Override to Disabled, and click Done.
5. When the process completes, on the same set of hosts from the previous steps, from the Select Action list, select Manage Repository Sets.
6. From the Repository Sets Management list, select the Red Hat Satellite Tools 6.10 check box.
7. From the Select Action list, select Override to Enabled, and click Done.
8. When the process completes, on the same set of hosts from the previous steps, from the Select Action list, select Manage Packages.

9. In the Package search field, enter one of the following options depending on your configuration:

   • If your deployment uses katello-agent and goferd, enter katello-agent.
   • If your deployment does not use katello-agent and goferd, enter katello-host-tools.
10. From the Update list, you must select the via remote execution option. This is required because if you update the package using the Katello agent, the package update disrupts the communication between the client and Satellite or Capsule Server, which causes the update to fail. For more information, see Configuring and Setting Up Remote Jobs in the Managing Hosts guide.

Upgrade Satellite Clients Manually

1. Log into the client system.

2. Disable the repositories for the previous version of Satellite.

   # subscription-manager repos \
   --disable rhel-7-server-satellite-tools-6.9-rpms

3. Enable the Satellite Tools 6.10 repository for this version of Satellite.

   # subscription-manager repos \
   --enable=rhel-7-server-satellite-tools-6.10-rpms

4. Depending on your configuration, complete one of the following steps:

   • If your deployment uses katello-agent and goferd, enter the following command to install or upgrade katello-agent:

     # yum install katello-agent

   • If your deployment does not use katello-agent and goferd, enter the following command to install or upgrade katello-host-tools:

     # yum install katello-host-tools

Upgrade virt-who Manually

1. Upgrade virt-who.

   # yum upgrade virt-who

2. Restart the virt-who service so the new version is activated.

   # systemctl restart virt-who.service

1. In the Satellite web UI, navigate to Content > Red Hat Repositories.
2. In the Enabled Repositories area, locate Red Hat Satellite Tools 6.9 for RHEL 7 Server RPMs x86_64.
3. Click the Disable icon to the right.

Procedure

1. Stop all services, except for the postgresql service:

   # satellite-maintain service stop --exclude postgresql

2. Switch to the postgres user and reclaim space on the database:

   # su - postgres -c 'vacuumdb --full --dbname=foreman'

3. Start the other services when the vacuum completes:

   # satellite-maintain service start

4. Confirm that the files exist in the /var/lib/pgsql/ directory:

   # ls -l /var/lib/pgsql/
   
   # du -sh /var/lib/pgsql/

5. Delete the data from the /var/lib/pgsql/ directory:

   # rm -rf /var/lib/pgsql/*

1. The default tuning profile defined in the /usr/share/foreman-installer/config/foreman.hiera/tuning/common.yaml file
2. The tuning profile that you want to apply to your deployment and is defined in the /usr/share/foreman-installer/config/foreman.hiera/tuning/sizes/ directory
3. Optional: If you have configured a /etc/foreman-installer/custom-hiera.yaml file, Satellite applies these configuration settings.

Procedure

1. Optional: If you have configured the custom-hiera.yaml file on Satellite Server, back up the /etc/foreman-installer/custom-hiera.yaml file to custom-hiera.original. You can use the backup file to restore the /etc/foreman-installer/custom-hiera.yaml file to its original state if it becomes corrupted:

   # cp /etc/foreman-installer/custom-hiera.yaml \
   /etc/foreman-installer/custom-hiera.original

2. Optional: If you have configured the custom-hiera.yaml file on Satellite Server, review the definitions of the default tuning profile in /usr/share/foreman-installer/config/foreman.hiera/tuning/common.yaml and the tuning profile that you want to apply in /usr/share/foreman-installer/config/foreman.hiera/tuning/sizes/. Compare the configuration entries against the entries in your /etc/foreman-installer/custom-hiera.yaml file and remove any duplicated configuration settings in your /etc/foreman-installer/custom-hiera.yaml file.

3. Enter the satellite-installer command with the --tuning option for the profile that you want to apply. For example, to apply the medium tuning profile settings, enter the following command:

   # satellite-installer --tuning medium

Procedure

1. To view the Puppet environments, open the /etc/puppetlabs/code/environments file.

   # vim /etc/puppetlabs/code/environments

2. If any of the puppet environments are missing, copy them back from the Satellite Server or the backup taken prior to upgrade.
3. Ensure to have the right permissions and ownership of the missing contents inside the /etc/puppetlabs/code/environments/ file after they have been restored back.
4. Save and close the file.

To Update Satellite Server:

1. Ensure the Satellite Maintenance repository is enabled:

   # subscription-manager repos --enable \
   rhel-7-server-satellite-maintenance-6-rpms

2. Check the available versions to confirm the next minor version is listed:

   # satellite-maintain upgrade list-versions

3. Use the health check option to determine if the system is ready for upgrade. On first use of this command, satellite-maintain prompts you to enter the hammer admin user credentials and saves them in the /etc/foreman-maintain/foreman-maintain-hammer.yml file.

   # satellite-maintain upgrade check --target-version 6.10.z

   Review the results and address any highlighted error conditions before performing the upgrade.

4. Because of the lengthy update time, use a utility such as screen to suspend and reattach a communication session. You can then check the upgrade progress without staying connected to the command shell continuously. For more information about using the screen command, see How do I use the screen command? article in the Red Hat Knowledge Base.

   If you lose connection to the command shell where the upgrade command is running, you can see the logged messages in the /var/log/foreman-installer/satellite.log file to check if the process completed successfully.

5. Perform the upgrade:

   # satellite-maintain upgrade run --target-version 6.10.z

6. Check when the kernel packages were last updated:

   # rpm -qa --last | grep kernel

7. Optional: If a kernel update occurred since the last reboot, stop the satellite-maintain services and reboot the system:

   # satellite-maintain service stop
   # reboot

1. Ensure that you have synchronized the following repositories in your connected Satellite Server.

   rhel-7-server-ansible-2.9-rpms
   rhel-7-server-rpms
   rhel-7-server-satellite-6.10-rpms
   rhel-7-server-satellite-maintenance-6-rpms
   rhel-server-rhscl-7-rpms

2. Download the debug certificate of the organization and store it locally at, for example, /etc/pki/katello/certs/org-debug-cert.pem or a location of your choosing.

3. Create a Yum configuration file under /etc/yum.repos.d with the following repository information:

   [rhel-7-server-ansible-2.9-rpms]
   name=Ansible 2.9 RPMs for Red Hat Enterprise Linux 7 Server x86_64
   baseurl=https://satellite.example.com/pulp/content/My_Organization/Library/content/dist/rhel/server/7/$releasever/$basearch/ansible/2.9/os/
   enabled=1
   sslclientcert = /etc/pki/katello/certs/org-debug-cert.pem
   sslclientkey = /etc/pki/katello/certs/org-debug-cert.pem
   sslcacert = /etc/pki/katello/certs/katello-server-ca.crt
   sslverify = 1
   
   [rhel-7-server-rpms]
   name=Red Hat Enterprise Linux 7 Server RPMs x86_64
   baseurl=https://satellite.example.com/pulp/content/My_Organization/Library/content/dist/rhel/server/7/7Server/x86_64/os/
   enabled=1
   sslclientcert = /etc/pki/katello/certs/org-debug-cert.pem
   sslclientkey = /etc/pki/katello/certs/org-debug-cert.pem
   sslcacert = /etc/pki/katello/certs/katello-server-ca.crt
   sslverify = 1
   
   [rhel-7-server-satellite-6.10-rpms]
   name=Red Hat Satellite 6 for RHEL 7 Server RPMs x86_64
   baseurl=https://satellite.example.com/pulp/content/My_Organization/Library/content/dist/rhel/server/7/7Server/x86_64/satellite/6.10/os/
   enabled=1
   sslclientcert = /etc/pki/katello/certs/org-debug-cert.pem
   sslclientkey = /etc/pki/katello/certs/org-debug-cert.pem
   sslcacert = /etc/pki/katello/certs/katello-server-ca.crt
   
   [rhel-7-server-satellite-maintenance-6-rpms]
   name=Red Hat Satellite Maintenance 6 for RHEL 7 Server RPMs x86_64
   baseurl=https://satellite.example.com/pulp/content/My_Organization/Library/content/dist/rhel/server/7/7Server/x86_64/sat-maintenance/6/os/
   enabled=1
   sslclientcert = /etc/pki/katello/certs/org-debug-cert.pem
   sslclientkey = /etc/pki/katello/certs/org-debug-cert.pem
   sslcacert = /etc/pki/katello/certs/katello-server-ca.crt
   sslverify = 1
   
   [rhel-server-rhscl-7-rpms]
   name=Red Hat Software Collections RPMs for Red Hat Enterprise Linux 7 Server x86_64
   baseurl=https://satellite.example.com/pulp/content/My_Organization/Library/content/dist/rhel/server/7/7Server/x86_64/rhscl/1/os/
   enabled=1
   sslclientcert = /etc/pki/katello/certs/org-debug-cert.pem
   sslclientkey = /etc/pki/katello/certs/org-debug-cert.pem
   sslcacert = /etc/pki/katello/certs/katello-server-ca.crt
   sslverify = 1

4. In the configuration file, replace /etc/pki/katello/certs/org-debug-cert.pem in sslclientcert and sslclientkey with the location of the downloaded organization debug certificate.
5. Update satellite.example.com with correct FQDN for your deployment.

6. Replace My_Organization with the correct organization label in the baseurl. To obtain the organization label, enter the command:

   # hammer organization list

7. Enter the reposync command:

   # reposync --delete --download-metadata -p ~/Satellite-repos -n \
    -r rhel-7-server-ansible-2.9-rpms \
    -r rhel-7-server-rpms \
    -r rhel-7-server-satellite-6.10-rpms \
    -r rhel-7-server-satellite-maintenance-6-rpms \
    -r rhel-server-rhscl-7-rpms

   This downloads the contents of the repositories from the connected Satellite Server and stores them in the directory ~/Satellite-repos. The reposync command in Red Hat Enterprise Linux 7 downloads the RPMs but not the Yum metadata.

   Because of this, you must manually run createrepo in each sub-directory of Satellite-repos. Make sure you have the createrepo rpm installed. If not use the following command to install it.

   # satellite-maintain packages install createrepo

   Run the following command to create repodata in each sub-directory of ~/Satellite-repos. :

   # cd ~/Satellite-repos
   # for directory in */
   do
     echo "Processing $directory"
     cd $directory
     createrepo .
     cd ..
   done

8. Verify that the RPMs have been downloaded and the repository data directory is generated in each of the sub-directories of ~/Satellite-repos.

9. Archive the contents of the directory

   # cd ~
   # tar czf Satellite-repos.tgz Satellite-repos

10. Use the generated Satellite-repos.tgz file to upgrade in the disconnected Satellite Server.

1. Copy the generated Satellite-repos.tgz file to your disconnected Satellite Server

2. Extract the archive to anywhere accessible by the root user. In the following example /root is the extraction location.

   # cd /root
   # tar zxf Satellite-repos.tgz

3. Create a Yum configuration file under /etc/yum.repos.d with the following repository information:

   [rhel-7-server-ansible-2.9-rpms]
   name=Ansible 2.9 RPMs for Red Hat Enterprise Linux 7 Server x86_64
   baseurl=file:///root/Satellite-repos/rhel-7-server-ansible-2.9-rpms
   enabled=1
   
   [rhel-7-server-rpms]
   name=Red Hat Enterprise Linux 7 Server RPMs x86_64
   baseurl=file:///root/Satellite-repos/rhel-7-server-rpms
   enabled=1
   
   [rhel-7-server-satellite-6.10-rpms]
   name=Red Hat Satellite 6 for RHEL 7 Server RPMs x86_64
   baseurl=file:///root/Satellite-repos/rhel-7-server-satellite-6.10-rpms
   enabled=1
   
   [rhel-7-server-satellite-maintenance-6-rpms]
   name=Red Hat Satellite Maintenance 6 for RHEL 7 Server RPMs x86_64
   baseurl=file:///root/Satellite-repos/rhel-7-server-satellite-maintenance-6-rpms
   enabled=1
   
   [rhel-server-rhscl-7-rpms]
   name=Red Hat Software Collections RPMs for Red Hat Enterprise Linux 7 Server x86_64
   baseurl=file:///root/Satellite-repos/rhel-server-rhscl-7-rpms
   enabled=1

4. In the configuration file, replace the /root/Satellite-repos with the extracted location.

5. Check the available versions to confirm the next minor version is listed:

   # satellite-maintain upgrade list-versions

6. Use the health check option to determine if the system is ready for upgrade. On first use of this command, satellite-maintain prompts you to enter the hammer admin user credentials and saves them in the /etc/foreman-maintain/foreman-maintain-hammer.yml file.

   # satellite-maintain upgrade check --whitelist="check-upstream-repository,repositories-validate" --target-version 6.10.z

7. Review the results and address any highlighted error conditions before performing the upgrade.

8. Because of the lengthy update time, use a utility such as screen to suspend and reattach a communication session. You can then check the upgrade progress without staying connected to the command shell continuously. For more information about using the screen command, see How do I use the screen command? article in the Red Hat Knowledge Base.

   If you lose connection to the command shell where the upgrade command is running, you can see the logged messages in the /var/log/foreman-installer/satellite.log file to check if the process completed successfully.

9. Perform the upgrade:

   # satellite-maintain upgrade run --whitelist="check-upstream-repository,repositories-validate" --target-version 6.10.z

10. Check when the kernel packages were last updated:

    # rpm -qa --last | grep kernel

11. Optional: If a kernel update occurred since the last reboot, stop the satellite-maintain services and reboot the system:

    # satellite-maintain service stop
    # reboot

Procedure

1. Ensure that the Satellite Maintenance repository is enabled:

   # subscription-manager repos --enable \
   rhel-7-server-satellite-maintenance-6-rpms

2. Check the available versions to confirm the next minor version is listed:

   # satellite-maintain upgrade list-versions

3. Use the health check option to determine if the system is ready for upgrade:

   # satellite-maintain upgrade check --target-version 6.10.z

   Review the results and address any highlighted error conditions before performing the upgrade.

4. Because of the lengthy update time, use a utility such as screen to suspend and reattach a communication session. You can then check the upgrade progress without staying connected to the command shell continuously. For more information about using the screen command, see How do I use the screen command? article in the Red Hat Knowledge Base.

   If you lose connection to the command shell where the upgrade command is running, you can see the logged messages in the /var/log/foreman-installer/satellite.log file to check if the process completed successfully.

5. Perform the upgrade:

   # satellite-maintain upgrade run --target-version 6.10.z

6. Check when the kernel packages were last updated:

   # rpm -qa --last | grep kernel

7. Optional: If a kernel update occurred since the last reboot, stop the satellite-maintain services and reboot the system:

   # satellite-maintain service stop
   # reboot