6.3. Upgrading a RHEL or RHEV-H 6 Self-Hosted Engine Environment

Procedure 6.9. Upgrading the Manager

1. Enable global maintenance from one of the self-hosted engine hosts:

   # hosted-engine --set-maintenance --mode=global

   Copy to Clipboard Toggle word wrap
2. On the Manager virtual machine, subscribe to the required entitlements for receiving the Manager 3.6 packages:

   # subscription-manager repos --enable=rhel-6-server-rhevm-3.6-rpms

   Copy to Clipboard Toggle word wrap
3. Update the rhevm setup packages:

   # yum update rhevm\*setup\*

   Copy to Clipboard Toggle word wrap
4. Run the following command and follow the prompts to upgrade the Manager:

   # engine-setup --otopi-environment=OVESETUP_DB/forceUpgradeWithHeEl6Hosts=bool:True

   Copy to Clipboard Toggle word wrap
5. Remove or disable the Manager 3.5 repository to ensure the system does not use any Manager 3.5 packages:

   # subscription-manager repos --disable=rhel-6-server-rhevm-3.5-rpms

   Copy to Clipboard Toggle word wrap

Procedure 6.10. Upgrading the Self-Hosted Engine Host

1. In the Administration Portal, click the Hosts tab, select the host on which the self-hosted engine was originally deployed, click Maintenance, and click OK. The host's virtual machines, including the Manager, will migrate to other hosts.
2. Connect to the Manager virtual machine and run the host-direct-upgrade tool to import the configuration files:

   # /usr/share/ovirt-engine/bin/host-direct-upgrade.sh --import-configuration-files [-h 3.5-host-address]

   Copy to Clipboard Toggle word wrap

   Note

   If you do not specify the optional -h 3.5-host-address parameter, you will be prompted to input the 3.5 host's IP address or FQDN later.

   The host-direct-upgrade tool checks whether the self-hosted engine is configured to allow mixed RHEL versions in the same cluster. If not, the ovirt-engine service is restarted.
3. In the Administration Portal, click the Clusters tab and select the cluster.
4. Click Edit.
5. In the Scheduling Policy tab, select InClusterUpgrade from the Policy drop-down list and click OK.
6. Download Next Generation RHVH (for RHEV-M 3.6.11 Async) or RHEL 7.
7. Install the newer operating system on the host, keeping the same IP address and host name. See Installing Red Hat Virtualization Host in the Installation Guide to install Next Generation RHVH or the Red Hat Enterprise Linux Installation Guide to install RHEL 7.
8. Connect to the Manager virtual machine and run the host-direct-upgrade tool to export the configuration files:

   # /usr/share/ovirt-engine/bin/host-direct-upgrade.sh --export-and-configure [-h 3.6-host-address]

   Copy to Clipboard Toggle word wrap

   Note

   The 3.6-host-address is the IP address or FQDN of the host that you upgraded.

   This process may take 10 to 15 minutes.
9. Check the host's status:

   # hosted-engine --vm-status

   Copy to Clipboard Toggle word wrap
   The Score should be 2800:

   !! Cluster is in GLOBAL MAINTENANCE mode !!
   
   --== Host 1 status ==--
   
   conf_on_shared_storage             : False
   Status up-to-date                  : True
   Hostname                           : rhvhhe20170926h1.localdomain
   Host ID                            : 1
   Engine status                      : "reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"
   Score                              : 2800
   stopped                            : False
   Local maintenance                  : False
   crc32                              : ffe0a244
   local_conf_timestamp               : 0
   Host timestamp                     : 1375

   Copy to Clipboard Toggle word wrap
10. In the Administration Portal, click the Hosts tab and select the 3.5 host (still in maintenance mode).
11. Click Remove and OK.
12. Click New and add the upgraded host. Its status should be Up.
13. Migrate the Manager virtual machine manually to the upgraded host. See Manually Migrating Virtual Machines in the Virtual Machine Management Guide for details.

Procedure 6.11. Comparing the Manager's SPM ID to the Host IDs

1. Connect to the Manager virtual machine and run the following command to output the SPM ID:

   # sudo -u postgres psql engine -c "SELECT vds_spm_id, vds.vds_name FROM vds"

   Copy to Clipboard Toggle word wrap
2. Connect to each host and run the following command to ouput the Host ID:

   # grep "^host_id" /etc/ovirt-hosted-engine/hosted-engine.conf

   Copy to Clipboard Toggle word wrap
3. If a host's Host ID does not match the Manager's SPM ID:
   1. In the Administration Portal, click the Hosts tab, select the host with the mismatched Host ID, click Maintenance, and click OK.
   2. Connect to the host and correct the host_id value in the /etc/ovirt-hosted-engine/hosted-engine.conf file.
   3. Reboot the host.
   4. In the Administration Portal, click the Hosts tab and select the host.
   5. Click Activate and OK.

Procedure 6.12. Upgrading the Remaining Hosts

1. In the Administration Portal, click the Hosts tab, select another host, and click Maintenance.
   The host's virtual machines will migrate to other machines, preferring hosts with newer operating systems, because of the InClusterUpgrade policy. If the virtual machines do not migrate automatically, migrate them manually. See Manually Migrating Virtual Machines in the Virtual Machine Management Guide for details.
2. When the host has no virtual machines, click Remove.
3. Install Next Generation RHVH or RHEL 7 on the host, keeping its IP address and host name.
4. In the Administration Portal, click the Hosts tab and click Add to add the upgraded host.

   Note

   The upgraded host is added manually, before the hosted-engine setup, in order to preserve its Host ID.
5. Connect to the host, disable NetworkManager, and enable the network script:

   # systemctl stop NetworkManager; systemctl disable NetworkManager; systemctl start network; systemctl enable network

   Copy to Clipboard Toggle word wrap
6. Install the screen utility if necessary:

   # yum install screen

   Copy to Clipboard Toggle word wrap
7. Run the hosted-engine setup in a screen session:

   # hosted-engine --deploy
       
   [ INFO  ] Stage: Initializing
   [ INFO  ] Generating a temporary VNC password.   
   [ INFO  ] Stage: Environment setup
             Continuing will configure this host for serving as hypervisor and create a VM where you have to install the engine afterwards.
             Are you sure you want to continue? (Yes, No)[Yes]:
             Configuration files: []
             Log file: /var/log/ovirt-hosted-engine-setup/ovirt-hosted-engine-setup.log
             Version: otopi-1.4.2 (otopi-1.4.2-1.el7ev)
   [ INFO  ] Hardware supports virtualization
   [ INFO  ] Stage: Environment packages setup
   [ INFO  ] Stage: Programs detection
   [ INFO  ] Stage: Environment setup
   [ INFO  ] Stage: Environment customization
   
             --== STORAGE CONFIGURATION ==--
   
             During customization use CTRL-D to abort.
             Please specify the storage you would like to use (glusterfs, iscsi, fc, nfs3, nfs4)[nfs3]: nfs4
             Please specify the full shared storage connection path to use (example: host:/path): hostname:/path
             The specified storage location already contains a data domain. Is this an additional host setup (Yes, No)[Yes]?
   [ INFO  ] Installing on additional host
             Please specify the Host ID [Must be integer, default: 2]:
   
             --== SYSTEM CONFIGURATION ==--
   
   [WARNING] A configuration file must be supplied to deploy Hosted Engine on an additional host.
   [ INFO  ] Answer file successfully loaded
   
             --== NETWORK CONFIGURATION ==--
   
   [ INFO  ] Bridge rhevm already created
   [ INFO  ] Additional host deployment, firewall manager is 'iptables'
             The following CPU types are supported by this host:
                    - model_SandyBridge: Intel SandyBridge Family
                    - model_Westmere: Intel Westmere Family
                    - model_Nehalem: Intel Nehalem Family
                    - model_Penryn: Intel Penryn Family
                    - model_Conroe: Intel Conroe Family
   
             --== HOSTED ENGINE CONFIGURATION ==--  
   
             Enter the name which will be used to identify this host inside the Administrator Portal [hosted_engine_2]:
             Enter 'admin@internal' user password that will be used for accessing the Administrator Portal:
             Confirm 'admin@internal' user password:
   [ INFO  ] Stage: Setup validation
             Please provide the address of this host.
             Note: The engine VM and all the other hosts should be able to correctly resolve it.
             Host address:  [rhvh.example.com]:
   	  The Host ID is already known. Is this a re-deployment on an additional host that was previously set up (Yes, No)[Yes]?
   
             --== CONFIGURATION PREVIEW ==--
   
             Engine FQDN                        : enginevm.localdomain
             Bridge name                        : rhevm
             Host address                       : rhvh.example.com
             SSH daemon port                    : 22
             Firewall manager                   : iptables
             Gateway address                    : 192.168.1.1
             Host name for web application      : hosted_engine_2
             Storage Domain type                : nfs4
             Host ID                            : 2 
             Image size GB                      : 50
             GlusterFS Share Name               : hosted_engine_glusterfs
             GlusterFS Brick Provisioning       : False
             Storage connection                 : hostname:/path
             Console type                       : vnc
             Memory size MB                     : 4096
             MAC address                        : 00:16:3e:12:96:6b
             Boot type                          : disk
             Number of CPUs                     : 1 
             CPU Type                           : model_SandyBridge
   [ INFO  ] Stage: Transaction setup
   [ INFO  ] Stage: Misc configuration
   [ INFO  ] Stage: Package installation
   [ INFO  ] Stage: Misc configuration
   [ INFO  ] Configuring libvirt
   [ INFO  ] Configuring VDSM
   [ INFO  ] Starting vdsmd
   [ INFO  ] Waiting for VDSM hardware info
   [ INFO  ] Configuring VM
   [ INFO  ] Updating hosted-engine configuration   
   [ INFO  ] Stage: Transaction commit
   [ INFO  ] Stage: Closing up
   [ INFO  ] Acquiring internal CA cert from the engine
   [ INFO  ] The following CA certificate is going to be used, please immediately interrupt if not correct:
   [ INFO  ] Issuer: C=US, O=localdomain, CN=enginevm.localdomain.34970, Subject: C=US, O=localdomain, CN=enginevm.localdomain.34970, Fingerprint (SHA-1): 46A75DA2174D03B100D1811411C0C6627CB1D821
   [ INFO  ] Connecting to the Engine
   [ INFO  ] Waiting for the host to become operational in the engine. This may take several minutes...
   
   [ INFO  ] Still waiting for VDSM host to become operational...
   [ INFO  ] The VDSM Host is now operational
   [ ERROR ] Cannot automatically set CPU level of cluster Default: inclusterupgrade is not a member of SchedulingPolicyType. Possible values for SchedulingPolicyType are: evenly_distributed, power_saving, none, vm_evenly_distributed 
   [ INFO  ] Enabling and starting HA services
   [ INFO  ] Stage: Clean up
   [ INFO  ] Generating answer file '/var/lib/ovirt-hosted-engine-setup/answers/answers-20170927190113.conf'
   [ INFO  ] Generating answer file '/etc/ovirt-hosted-engine/answers.conf'
   [ INFO  ] Stage: Pre-termination
   [ INFO  ] Stage: Termination
   [ INFO  ] Hosted Engine successfully set up

   Copy to Clipboard Toggle word wrap

   Note

   You can ignore the [ ERROR ] Cannot automatically set CPU level of cluster Default message.

Procedure 6.13. Post-Upgrade Tasks

1. In the Administration Portal, click the Cluster tab and select the cluster.
2. Select 3.6 from the Compatibility Version drop-down list.
3. In the Scheduling Policy tab, select None from the Policy drop-down list and click OK.
4. Reboot each virtual machine (except the Manager virtual machine), one by one.
5. Connect to the Manager virtual machine and update its configuration:

   # engine-config -s CheckMixedRhelVersions=true --cver=3.5

   Copy to Clipboard Toggle word wrap
6. In the Administration Portal, click the Events tab and confirm that the Manager virtual machine and its storage domain were imported successfully:

   Sep 27, 2017 7:15:18 PM Hosted Engine VM was imported successfully
   Sep 27, 2017 7:15:18 PM Starting to import Vm HostedEngine to Data Center Default, Cluster Default
   Sep 27, 2017 7:15:16 PM VM HostedEngine was successfully removed.
   Sep 27, 2017 7:15:00 PM Hosted Engine storage domain imported successfully
   Sep 27, 2017 7:15:00 PM Storage Domain hosted_storage was attached to Data Center Default by SYSTEM
   Sep 27, 2017 7:15:00 PM Storage Domain hosted_storage (Data Center Default) was activated by SYSTEM
   Sep 27, 2017 7:14:57 PM The Hosted Engine Storage Domain isn't Active.

   Copy to Clipboard Toggle word wrap
7. In the Virtual Machines tab, select the Manager virtual machine, and click Shutdown.
8. Connect to a self-hosted engine host and disable global maintenance mode:

   # hosted-engine --set-maintenance --mode=none

   Copy to Clipboard Toggle word wrap
   Wait for the ovirt-ha-agent service to restart the Manager virtual machine.
9. Check the status of each host:

   # hosted-engine --vm-status

   Copy to Clipboard Toggle word wrap
   Each host should have a score of 3400:

   --== Host 1 status ==--
   
   conf_on_shared_storage             : False
   Status up-to-date                  : True
   Hostname                           : rhvhhe20170926h1.localdomain
   Host ID                            : 1
   Engine status                      : "health": "good", "vm": "up", "detail": "up"
   Score                              : 3400
   stopped                            : False
   Local maintenance                  : False
   crc32                              : 6a5a17a7
   local_conf_timestamp               : 7580
   Host timestamp                     : 7579
   
   
   --== Host 2 status ==--
   
   conf_on_shared_storage             : False
   Status up-to-date                  : True
   Hostname                           : rhvhhe20170926h2.localdomain
   Host ID                            : 2
   Engine status                      : "reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"
   Score                              : 3400
   stopped                            : False
   Local maintenance                  : False
   crc32                              : 1c87c83d
   local_conf_timestamp               : 4323
   Host timestamp                     : 4323

   Copy to Clipboard Toggle word wrap