Self-Hosted Engine Guide

Procedure 2.1. Installing the Self-Hosted Engine

1. Install the self-hosted engine packages:

   # yum install ovirt-hosted-engine-setup

2. Install the RHV-M Virtual Appliance package for the Manager virtual machine installation:

   # yum install rhevm-appliance

Procedure 2.2. Configuring a RHEL-based Self-Hosted Engine

1. Initiating Hosted Engine Deployment

   Run the hosted-engine script. To escape the script at any time, use the CTRL+D keyboard combination to abort deployment. It is recommended to use the screen window manager to run the script to avoid losing the session in case of network or terminal disruption. If not already installed, install the screen package, which is available in the standard Red Hat Enterprise Linux repository.

   # yum install screen

   # screen

   # hosted-engine --deploy

   Note

   In the event of session timeout or connection disruption, run screen -d -r to recover the hosted-engine deployment session.

2. Configuring Storage

   Select the type of storage to use.

   During customization use CTRL-D to abort.
   Please specify the storage you would like to use (glusterfs, iscsi, fc, nfs3, nfs4)[nfs3]:

   • For NFS storage types, specify the full address, using either the FQDN or IP address, and path name of the shared storage domain.

     Please specify the full shared storage connection path to use (example: host:/path): storage.example.com:/hosted_engine/nfs

   • For iSCSI, specify the iSCSI portal IP address, port, user name and password, and select a target name from the auto-detected list. You can only select one iSCSI target during the deployment.

     Please specify the iSCSI portal IP address:
     Please specify the iSCSI portal port [3260]:
     Please specify the iSCSI portal user:
     Please specify the iSCSI portal password:
     Please specify the target name (auto-detected values) [default]:

   • For Gluster storage, specify the full address, using either the FQDN or IP address, and path name of the shared storage domain.

     Important

     Only replica 3 Gluster storage is supported. Ensure the following configuration has been made:

     • In the /etc/glusterfs/glusterd.vol file on all three Gluster servers, set rpc-auth-allow-insecure to on.

       option rpc-auth-allow-insecure on

     • Configure the volume as follows:

       gluster volume set volume cluster.quorum-type auto
       gluster volume set volume network.ping-timeout 10
       gluster volume set volume auth.allow \*
       gluster volume set volume group virt
       gluster volume set volume storage.owner-uid 36
       gluster volume set volume storage.owner-gid 36
       gluster volume set volume server.allow-insecure on

     Please specify the full shared storage connection path to use (example: host:/path): storage.example.com:/hosted_engine/gluster_volume

   • For Fibre Channel, the host bus adapters must be configured and connected, and the hosted-engine script will auto-detect the LUNs available. The LUNs must not contain any existing data.

     The following luns have been found on the requested target:
     [1]     3514f0c5447600351       30GiB   XtremIO XtremApp
                             status: used, paths: 2 active
     
     [2]     3514f0c5447600352       30GiB   XtremIO XtremApp
                             status: used, paths: 2 active
     
     Please select the destination LUN (1, 2) [1]:

3. Configuring the Network

   The script detects possible network interface controllers (NICs) to use as a management bridge for the environment. It then checks your firewall configuration and offers to modify it for console (SPICE or VNC) access.

   Please indicate a nic to set rhvm bridge on: (eth1, eth0) [eth1]:
   iptables was detected on your computer, do you wish setup to configure it? (Yes, No)[Yes]: Yes
   Please indicate a pingable gateway IP address [X.X.X.X]:

4. Configuring the Virtual Machine

   Select disk for the boot device type, and the script will automatically detect the RHV-M Virtual Appliances available. Specify the memory size.

            Please specify the device to boot the VM from (choose disk for the oVirt engine appliance)
            (cdrom, disk, pxe) [disk]:
            Please specify the console type you would like to use to connect to the VM (vnc, spice) [vnc]: vnc
   [ INFO ] Detecting available oVirt engine appliances
            The following appliance have been found on your system:
                  [1] - The oVirt Engine Appliance image (OVA)
                  [2] - Directly select an OVA file
            Please select an appliance (1, 2) [1]:
   [ INFO ] Checking OVF archive content (could take a few minutes depending on archive size)

   Specify Yes if you want cloud-init to take care of the initial configuration of the Manager virtual machine. Specify Generate for cloud-init to take care of tasks like setting the root password, configuring networking, configuring the host name, injecting an answers file for engine-setup to use, and running engine-setup on boot. Optionally, select Existing if you have an existing cloud-init script to take care of more sophisticated functions of cloud-init. Specify the FQDN for the Manager virtual machine. Specify a MAC address for the HostedEngine-VM, or accept a randomly generated one. The MAC address can be used to update your DHCP and DNS server prior to installing the operating system on the virtual machine.

   Note

   For more information on cloud-init, see https://cloudinit.readthedocs.org/en/latest/.

   Would you like to use cloud-init to customize the appliance on the first boot (Yes, No)[Yes]? Yes
   Would you like to generate on-fly a cloud-init ISO image (of no-cloud type)
   or do you have an existing one (Generate, Existing)[Generate]? Generate
   Please provide the FQDN you would like to use for the engine appliance.
   Note: This will be the FQDN of the engine VM you are now going to launch.
   It should not point to the base host or to any other existing machine.
   Engine VM FQDN: (leave it empty to skip): manager.example.com
   Automatically execute engine-setup on the engine appliance on first boot (Yes, No)[Yes]? Yes
   Automatically restart the engine VM as a monitored service after engine-setup (Yes, No)[Yes]? Yes
   Enter root password that will be used for the engine appliance (leave it empty to skip): p@ssw0rd
   Confirm appliance root password: p@ssw0rd
   The following CPU types are supported by this host:
       - model_Penryn: Intel Penryn Family
       - model_Conroe: Intel Conroe Family
   Please specify the CPU type to be used by the VM [model_Penryn]:
   Please specify the number of virtual CPUs for the VM [Defaults to appliance OVF value: 4]:
   You may specify a MAC address for the VM or accept a randomly generated default [00:16:3e:77:b2:a4]:
   How should the engine VM network be configured (DHCP, Static)[DHCP]? Static
   Please enter the IP address to be used for the engine VM: 192.168.x.x
   Please provide a comma-separated list (max3) of IP addresses of domain name servers for the engine VM
   Engine VM DNS (leave it empty to skip):
   Add lines for the appliance itself and for this host to /etc/hosts on the engine VM?
   Note: ensuring that this host could resolve the engine VM hostname is still up to you (Yes, No)[No] Yes

5. Configuring the Hosted Engine

   Specify a name for the host to be identified in the Administration Portal, and the password for the admin@internal user to access the Administration Portal. Provide the name and TCP port number of the SMTP server, the email address used to send email notifications, and a comma-separated list of email addresses to receive these notifications.

   Enter engine admin password: p@ssw0rd
   Confirm engine admin password: p@ssw0rd
   Enter the name which will be used to identify this host inside the Administrator Portal [hosted_engine_1]:
   Please provide the FQDN for the engine you would like to use.
             This needs to match the FQDN that you will use for the engine installation within the VM.
             Note: This will be the FQDN of the VM you are now going to create,
             it should not point to the base host or to any other existing machine.
             Engine FQDN:  []: manager.example.com
   Please provide the name of the SMTP server through which we will send notifications [localhost]:
   Please provide the TCP port number of the SMTP server [25]:
   Please provide the email address from which notifications will be sent [root@localhost]:
   Please provide a comma-separated list of email addresses which will get notifications [root@localhost]:

6. Configuration Preview

   Before proceeding, the hosted-engine script displays the configuration values you have entered, and prompts for confirmation to proceed with these values.

7. Creating the Manager Virtual Machine

   The script creates the Manager virtual machine, starts the ovirt-engine and high availability services, and connects the host and shared storage domain to the Manager virtual machine.

   You can now connect to the VM with the following command:
   	/usr/bin/remote-viewer vnc://localhost:5900
   Use temporary password "3042QHpX" to connect to vnc console.
   Please note that in order to use remote-viewer you need to be able to run graphical applications.
   This means that if you are using ssh you have to supply the -Y flag (enables trusted X11 forwarding).
   Otherwise you can run the command from a terminal in your preferred desktop environment.
   If you cannot run graphical applications you can connect to the graphic console from another host or connect to the serial console using the following command:
   socat UNIX-CONNECT:/var/run/ovirt-vmconsole-console/fabbea5a-1989-411f-8ed7-7abe0917fc66.sock,user=ovirt-vmconsole STDIO,raw,echo=0,escape=1
   
   If you need to reboot the VM you will need to start it manually using the command:
   hosted-engine --vm-start
   You can then set a temporary password using the command:
   hosted-engine --add-console-password
   [ INFO ] Running engine-setup on the appliance
   ...
   [ INFO ] Engine-setup successfully completed
   [ INFO ] Engine is still unreachable
   [ INFO ] Engine is still unreachable, waiting...
   [ INFO ] Engine replied: DB Up!Welcome to Health Status!
   [ INFO ] Connecting to the Engine
            Enter the name of the cluster to which you want to add the host (Default) [Default]:
   [ INFO  ] Waiting for the host to become operational in the engine. This may take several minutes... [ INFO  ] Still waiting for VDSM host to become operational...
   [ INFO  ] The VDSM Host is now operational
   [ INFO  ] Shutting down the engine VM
   [ INFO  ] Enabling and starting HA services
   [ INFO  ] Saving hosted-engine configuration on the shared storage domain
             Hosted Engine successfully set up
   [ INFO  ] Stage: Clean up
   [ INFO  ] Generating answer file '/var/lib/ovirt-hosted-engine-setup/answers/answers-2015xx.conf'
   [ INFO  ] Generating answer file '/etc/ovirt-hosted-engine/answers.conf'
   [ INFO  ] Stage: Pre-termination
   [ INFO  ] Stage: Termination

   When the hosted-engine deployment script completes successfully, the Red Hat Virtualization Manager is configured and running on your host. The Manager has already configured the data center, cluster, host, the Manager virtual machine, and a shared storage domain dedicated to the Manager virtual machine.

   Important

   Log in as the admin@internal user to continue configuring the Manager and add further resources. You must create another data domain for the data center to be initialized to host regular virtual machine data, and for the Manager virtual machine to be visible. See Storage in the Administration Guide for different storage options and on how to add a data storage domain.
   Link your Red Hat Virtualization Manager to a directory server so you can add additional users to the environment. Red Hat Virtualization supports many directory server types; for example, Red Hat Directory Server (RHDS), Red Hat Identity Management (IdM), Active Directory, and many other types. Add a directory server to your environment using the ovirt-engine-extension-aaa-ldap-setup interactive setup script. For more information, see Configuring an External LDAP Provider in the Administration Guide.
   The ovirt-hosted-engine-setup script also saves the answers you gave during configuration to a file, to help with disaster recovery. If a destination is not specified using the --generate-answer=<file> argument, the answer file is generated at /etc/ovirt-hosted-engine/answers.conf.

   Note

   SSH password authentication is not enabled by default on the RHV-M Virtual Appliance. You can enable SSH password authentication by accessing the Red Hat Virtualization Manager virtual machine through the SPICE or VNC console. Verify that the sshd service is running. Edit /etc/ssh/sshd_config and change the following two options to yes:

   • PasswordAuthentication
   • PermitRootLogin

   Restart the sshd service for the changes to take effect.

8. Subscribing to the Required Entitlements

   Subscribe the Manager virtual machine to the required entitlements. See Subscribing to the Required Entitlements in the Installation Guide for more information.

Procedure 6.4. Creating a New Self-Hosted Environment to be Used as the Restored Environment

1. Updating DNS

   Update your DNS so that the fully qualified domain name of the Red Hat Virtualization environment correlates to the IP address of the new Manager. In this procedure, fully qualified domain name was set as Manager.example.com. The fully qualified domain name provided for the engine must be identical to that given in the engine setup of the original engine that was backed up.

2. Initiating Hosted Engine Deployment

   On the newly installed Red Hat Enterprise Linux host, run the hosted-engine deployment script. To escape the script at any time, use the CTRL+D keyboard combination to abort deployment. If running the hosted-engine deployment script over a network, it is recommended to use the screen window manager to avoid losing the session in case of network or terminal disruption. Install the screen package first if not installed.

   # screen

   # hosted-engine --deploy

3. Preparing for Initialization

   The script begins by requesting confirmation to use the host as a hypervisor for use in a self-hosted engine environment.

   Continuing will configure this host for serving as hypervisor and create a VM where you have to install oVirt Engine afterwards. 
   Are you sure you want to continue? (Yes, No)[Yes]:

4. Configuring Storage

   Select the type of storage to use.

   During customization use CTRL-D to abort.
   Please specify the storage you would like to use (glusterfs, iscsi, fc, nfs3, nfs4)[nfs3]:

   • For NFS storage types, specify the full address, using either the FQDN or IP address, and path name of the shared storage domain.

     Please specify the full shared storage connection path to use (example: host:/path): storage.example.com:/hosted_engine/nfs

   • For iSCSI, specify the iSCSI portal IP address, port, user name and password, and select a target name from the auto-detected list. You can only select one iSCSI target during the deployment.

     Please specify the iSCSI portal IP address:           
     Please specify the iSCSI portal port [3260]:           
     Please specify the iSCSI portal user:           
     Please specify the iSCSI portal password:
     Please specify the target name (auto-detected values) [default]:

   • For Gluster storage, specify the full address, using either the FQDN or IP address, and path name of the shared storage domain.

     Important

     Only replica 3 Gluster storage is supported. Ensure the following configuration has been made:

     • In the /etc/glusterfs/glusterd.vol file on all three Gluster servers, set rpc-auth-allow-insecure to on.

       option rpc-auth-allow-insecure on

     • Configure the volume as follows:

       gluster volume set volume cluster.quorum-type auto
       gluster volume set volume network.ping-timeout 10
       gluster volume set volume auth.allow \*
       gluster volume set volume group virt
       gluster volume set volume storage.owner-uid 36
       gluster volume set volume storage.owner-gid 36
       gluster volume set volume server.allow-insecure on

     Please specify the full shared storage connection path to use (example: host:/path): storage.example.com:/hosted_engine/gluster_volume

   • For Fibre Channel, the host bus adapters must be configured and connected, and the hosted-engine script will auto-detect the LUNs available. The LUNs must not contain any existing data.

     The following luns have been found on the requested target:
     [1]     3514f0c5447600351       30GiB   XtremIO XtremApp
                             status: used, paths: 2 active
               
     [2]     3514f0c5447600352       30GiB   XtremIO XtremApp
                             status: used, paths: 2 active
     
     Please select the destination LUN (1, 2) [1]:

5. Configuring the Network

   The script detects possible network interface controllers (NICs) to use as a management bridge for the environment. It then checks your firewall configuration and offers to modify it for console (SPICE or VNC) access the Manager virtual machine. Provide a pingable gateway IP address, to be used by the ovirt-ha-agent, to help determine a host's suitability for running a Manager virtual machine.

   Please indicate a nic to set ovirtmgmt bridge on: (eth1, eth0) [eth1]:
   iptables was detected on your computer, do you wish setup to configure it? (Yes, No)[Yes]: 
   Please indicate a pingable gateway IP address [X.X.X.X]:
   

6. Configuring the New Manager Virtual Machine

   The script creates a virtual machine to be configured as the new Manager virtual machine. Specify the boot device and, if applicable, the path name of the installation media, the image alias, the CPU type, the number of virtual CPUs, and the disk size. Specify a MAC address for the Manager virtual machine, or accept a randomly generated one. The MAC address can be used to update your DHCP server prior to installing the operating system on the Manager virtual machine. Specify memory size and console connection type for the creation of Manager virtual machine.

   Please specify the device to boot the VM from (cdrom, disk, pxe) [cdrom]: 
   Please specify an alias for the Hosted Engine image [hosted_engine]:  
   The following CPU types are supported by this host:
             - model_Penryn: Intel Penryn Family
             - model_Conroe: Intel Conroe Family
   Please specify the CPU type to be used by the VM [model_Penryn]: 
   Please specify the number of virtual CPUs for the VM [Defaults to minimum requirement: 2]: 
   Please specify the disk size of the VM in GB [Defaults to minimum requirement: 25]: 
   You may specify a MAC address for the VM or accept a randomly generated default [00:16:3e:77:b2:a4]: 
   Please specify the memory size of the VM in MB [Defaults to minimum requirement: 4096]: 
   Please specify the console type you want to use to connect to the VM (vnc, spice) [vnc]:
   

7. Identifying the Name of the Host

   Specify the password for the admin@internal user to access the Administration Portal.
   A unique name must be provided for the name of the host, to ensure that it does not conflict with other resources that will be present when the engine has been restored from the backup. The name hosted_engine_1 can be used in this procedure because this host was placed into maintenance mode before the environment was backed up, enabling removal of this host between the restoring of the engine and the final synchronization of the host and the engine.

   Enter engine admin password: 
   Confirm engine admin password:
   Enter the name which will be used to identify this host inside the Administration Portal [hosted_engine_1]:

8. Configuring the Hosted Engine

   Provide the fully qualified domain name for the new Manager virtual machine. This procedure uses the fully qualified domain name Manager.example.com. Provide the name and TCP port number of the SMTP server, the email address used to send email notifications, and a comma-separated list of email addresses to receive these notifications.

   Important

   The fully qualified domain name provided for the engine (Manager.example.com) must be the same fully qualified domain name provided when original Manager was initially set up.

   Please provide the FQDN for the engine you would like to use.
   This needs to match the FQDN that you will use for the engine installation within the VM.
    Note: This will be the FQDN of the VM you are now going to create,
    it should not point to the base host or to any other existing machine.
    Engine FQDN: Manager.example.com
   Please provide the name of the SMTP server through which we will send notifications [localhost]: 
   Please provide the TCP port number of the SMTP server [25]: 
   Please provide the email address from which notifications will be sent [root@localhost]: 
   Please provide a comma-separated list of email addresses which will get notifications [root@localhost]:

9. Configuration Preview

   Before proceeding, the hosted-engine deployment script displays the configuration values you have entered, and prompts for confirmation to proceed with these values.

   Bridge interface                   : eth1
   Engine FQDN                        : Manager.example.com
   Bridge name                        : ovirtmgmt
   SSH daemon port                    : 22
   Firewall manager                   : iptables
   Gateway address                    : X.X.X.X
   Host name for web application      : hosted_engine_1
   Host ID                            : 1
   Image alias                        : hosted_engine
   Image size GB                      : 25
   Storage connection                 : storage.example.com:/hosted_engine/nfs
   Console type                       : vnc
   Memory size MB                     : 4096
   MAC address                        : 00:16:3e:77:b2:a4
   Boot type                          : pxe
   Number of CPUs                     : 2
   CPU Type                           : model_Penryn
   
   Please confirm installation settings (Yes, No)[Yes]:
   

10. Creating the New Manager Virtual Machine

    The script creates the virtual machine to be configured as the Manager virtual machine and provides connection details. You must install an operating system on it before the hosted-engine deployment script can proceed on Hosted Engine configuration.

    [ INFO  ] Stage: Transaction setup
    [ INFO  ] Stage: Misc configuration
    [ INFO  ] Stage: Package installation
    [ INFO  ] Stage: Misc configuration
    [ INFO  ] Configuring libvirt
    [ INFO  ] Configuring VDSM
    [ INFO  ] Starting vdsmd
    [ INFO  ] Waiting for VDSM hardware info
    [ INFO  ] Waiting for VDSM hardware info
    [ INFO  ] Configuring the management bridge
    [ INFO  ] Creating Storage Domain
    [ INFO  ] Creating Storage Pool
    [ INFO  ] Connecting Storage Pool
    [ INFO  ] Verifying sanlock lockspace initialization
    [ INFO  ] Creating VM Image
    [ INFO  ] Disconnecting Storage Pool
    [ INFO  ] Start monitoring domain
    [ INFO  ] Configuring VM
    [ INFO  ] Updating hosted-engine configuration
    [ INFO  ] Stage: Transaction commit
    [ INFO  ] Stage: Closing up
    [ INFO  ] Creating VM
    You can now connect to the VM with the following command:
          /usr/bin/remote-viewer vnc://localhost:5900
    Use temporary password "3477XXAM" to connect to vnc console.
    Please note that in order to use remote-viewer you need to be able to run graphical applications.
    This means that if you are using ssh you have to supply the -Y flag (enables trusted X11 forwarding).
    Otherwise you can run the command from a terminal in your preferred desktop environment.
    If you cannot run graphical applications you can connect to the graphic console from another host or connect to the console using the following command:
    virsh -c qemu+tls://Test/system console HostedEngine
    If you need to reboot the VM you will need to start it manually using the command:
    hosted-engine --vm-start
    You can then set a temporary password using the command:
    hosted-engine --add-console-password
    The VM has been started.  Install the OS and shut down or reboot it.  To continue please make a selection:
             
      (1) Continue setup - VM installation is complete
      (2) Reboot the VM and restart installation
      (3) Abort setup
      (4) Destroy VM and abort setup
             
      (1, 2, 3, 4)[1]:

    Using the naming convention of this procedure, connect to the virtual machine using VNC with the following command:

    /usr/bin/remote-viewer vnc://hosted_engine_1.example.com:5900

11. Installing the Virtual Machine Operating System

    Connect to Manager virtual machine and install a Red Hat Enterprise Linux 7 operating system.

12. Synchronizing the Host and the Manager

    Return to the host and continue the hosted-engine deployment script by selecting option 1:

    (1) Continue setup - VM installation is complete

    Waiting for VM to shut down...
    [ INFO  ] Creating VM
    You can now connect to the VM with the following command:
          /usr/bin/remote-viewer vnc://localhost:5900
    Use temporary password "3477XXAM" to connect to vnc console.
    Please note that in order to use remote-viewer you need to be able to run graphical applications.
    This means that if you are using ssh you have to supply the -Y flag (enables trusted X11 forwarding).
    Otherwise you can run the command from a terminal in your preferred desktop environment.
    If you cannot run graphical applications you can connect to the graphic console from another host or connect to the console using the following command:
    virsh -c qemu+tls://Test/system console HostedEngine
    If you need to reboot the VM you will need to start it manually using the command:
    hosted-engine --vm-start
    You can then set a temporary password using the command:
    hosted-engine --add-console-password
    Please install and setup the engine in the VM.
    You may also be interested in subscribing to "agent" RHN/Satellite channel and installing rhevm-guest-agent-common package in the VM.
    To continue make a selection from the options below:
      (1) Continue setup - engine installation is complete
      (2) Power off and restart the VM
      (3) Abort setup
      (4) Destroy VM and abort setup
             
      (1, 2, 3, 4)[1]:

13. Installing the Manager

    Connect to the new Manager virtual machine, register it with Red Hat Subscription Management, and enable the required repositories. See Subscribing to the Required Entitlements in the Installation Guide.
    Ensure the latest versions of all installed packages are in use, and install the rhevm packages.

    # yum update

    Note

    Reboot the machine if any kernel related packages have been updated.

    # yum install rhevm

Procedure 6.5. Restoring the Self-Hosted Engine Manager

1. Secure copy the backup files to the new Manager virtual machine. This example copies the files from a network storage server to which the files were copied in Section 6.1, “Backing up the Self-Hosted Engine Manager Virtual Machine”. In this example, Storage.example.com is the fully qualified domain name of the storage server, /backup/EngineBackupFiles is the designated file path for the backup files on the storage server, and /backup/ is the path to which the files will be copied on the new Manager.

   # scp -p Storage.example.com:/backup/EngineBackupFiles /backup/

2. Use the engine-backup tool to restore a complete backup.
   • If you are only restoring the Manager, run:

     # engine-backup --mode=restore --file=file_name --log=log_file_name --provision-db --restore-permissions

   • If you are restoring the Manager and Data Warehouse, run:

     # engine-backup --mode=restore --file=file_name --log=log_file_name --provision-db --provision-dwh-db --restore-permissions

   If successful, the following output displays:

   You should now run engine-setup.
   Done.

3. Configure the restored Manager virtual machine. This process identifies the existing configuration settings and database content. Confirm the settings. Upon completion, the setup provides an SSH fingerprint and an internal Certificate Authority hash.

   # engine-setup

   [ INFO  ] Stage: Initializing
   [ INFO  ] Stage: Environment setup
   Configuration files: ['/etc/ovirt-engine-setup.conf.d/10-packaging.conf', '/etc/ovirt-engine-setup.conf.d/20-setup-ovirt-post.conf']
   Log file: /var/log/ovirt-engine/setup/ovirt-engine-setup-20140304075238.log
   Version: otopi-1.1.2 (otopi-1.1.2-1.el6ev)
   [ INFO  ] Stage: Environment packages setup
   [ INFO  ] Yum Downloading: rhel-65-zstream/primary_db 2.8 M(70%)
   [ INFO  ] Stage: Programs detection
   [ INFO  ] Stage: Environment setup
   [ INFO  ] Stage: Environment customization
            
             --== PACKAGES ==--
            
   [ INFO  ] Checking for product updates...
   [ INFO  ] No product updates found
            
             --== NETWORK CONFIGURATION ==--
            
   Setup can automatically configure the firewall on this system.
   Note: automatic configuration of the firewall may overwrite current settings.
   Do you want Setup to configure the firewall? (Yes, No) [Yes]: 
   [ INFO  ] iptables will be configured as firewall manager.
            
             --== DATABASE CONFIGURATION ==--
            
            
             --== OVIRT ENGINE CONFIGURATION ==--
            
             Skipping storing options as database already prepared
            
             --== PKI CONFIGURATION ==--
            
             PKI is already configured
            
             --== APACHE CONFIGURATION ==--
            
            
             --== SYSTEM CONFIGURATION ==--
            
            
             --== END OF CONFIGURATION ==--
            
   [ INFO  ] Stage: Setup validation
   [ INFO  ] Cleaning stale zombie tasks
            
             --== CONFIGURATION PREVIEW ==--
            
             Database name                      : engine
             Database secured connection        : False
             Database host                      : X.X.X.X
             Database user name                 : engine
             Database host name validation      : False
             Database port                      : 5432
             NFS setup                          : True
             Firewall manager                   : iptables
             Update Firewall                    : True
             Configure WebSocket Proxy          : True
             Host FQDN                          : Manager.example.com
             NFS mount point                    : /var/lib/exports/iso
             Set application as default page    : True
             Configure Apache SSL               : True
            
             Please confirm installation settings (OK, Cancel) [OK]:

4. Removing the Host from the Restored Environment

   If the deployment of the restored self-hosted engine is on new hardware that has a unique name not present in the backed-up engine, skip this step. This step is only applicable to deployments occurring on the failover host, hosted_engine_1. Because this host was present in the environment at time the backup was created, it maintains a presence in the restored engine and must first be removed from the environment before final synchronization can take place.
   1. Log in to the Administration Portal.
   2. Click the Hosts tab. The failover host, hosted_engine_1, will be in maintenance mode and without a virtual load, as this was how it was prepared for the backup.
   3. Click Remove.
   4. Click Ok.

   Note

   If the host you are trying to remove becomes non-operational, see Section 6.2.4, “Removing Non-Operational Hosts from a Restored Self-Hosted Engine Environment” for instructions on how to force the removal of a host.

5. Synchronizing the Host and the Manager

   Return to the host and continue the hosted-engine deployment script by selecting option 1:

   (1) Continue setup - engine installation is complete

   [ INFO  ] Engine replied: DB Up!Welcome to Health Status!
   [ INFO  ] Waiting for the host to become operational in the engine. This may take several minutes...
   [ INFO  ] Still waiting for VDSM host to become operational...

   At this point, hosted_engine_1 will become visible in the Administration Portal with Installing and Initializing states before entering a Non Operational state. The host will continue to wait for the VDSM host to become operational until it eventually times out. This happens because another host in the environment maintains the Storage Pool Manager (SPM) role and hosted_engine_1 cannot interact with the storage domain because the SPM host is in a Non Responsive state. When this process times out, you are prompted to shut down the virtual machine to complete the deployment. When deployment is complete, the host can be manually placed into maintenance mode and activated through the Administration Portal.

   [ INFO  ] Still waiting for VDSM host to become operational...
   [ ERROR ] Timed out while waiting for host to start. Please check the logs.
   [ ERROR ] Unable to add hosted_engine_2 to the manager
             Please shutdown the VM allowing the system to launch it as a monitored service.
             The system will wait until the VM is down.

6. Shut down the new Manager virtual machine.

   # shutdown -h now

7. Return to the host to confirm it has detected that the Manager virtual machine is down.

   [ INFO  ] Enabling and starting HA services
             Hosted Engine successfully set up
   [ INFO  ] Stage: Clean up
   [ INFO  ] Stage: Pre-termination
   [ INFO  ] Stage: Termination
   

8. Activate the host.
   1. Log in to the Administration Portal.
   2. Click the Hosts tab.
   3. Select hosted_engine_1 and click the Maintenance button. The host may take several minutes before it enters maintenance mode.
   4. Click the Activate button.
   Once active, hosted_engine_1 immediately contends for SPM, and the storage domain and data center become active.
9. Migrate virtual machines to the active host by manually fencing the Non Responsive hosts. In the Administration Portal, right-click the hosts and select Confirm 'Host has been Rebooted'.
   Any virtual machines that were running on that host at the time of the backup will now be removed from that host, and move from an Unknown state to a Down state. These virtual machines can now be run on hosted_engine_1. The host that was fenced can now be forcefully removed using the REST API.

Procedure 6.6. Restoring the Self-Hosted Engine Manager

1. Manually create an empty database to which the database content in the backup can be restored. The following steps must be performed on the machine where the database is to be hosted.
   1. If the database is to be hosted on a machine other than the Manager virtual machine, install the postgresql-server package. This step is not required if the database is to be hosted on the Manager virtual machine because this package is included with the rhevm package.

      # yum install postgresql-server

   2. Initialize the postgresql database, start the postgresql service, and ensure this service starts on boot:

      # postgresql-setup initdb
      # systemctl start postgresql.service
      # systemctl enable postgresql.service

   3. Enter the postgresql command line:

      # su postgres
      $ psql

   4. Create the engine user:

      postgres=# create role engine with login encrypted password 'password';

      If you are also restoring Data Warehouse, create the ovirt_engine_history user on the relevant host:

      postgres=# create role ovirt_engine_history with login encrypted password 'password';

   5. Create the new database:

      postgres=# create database database_name owner engine template template0 encoding 'UTF8' lc_collate 'en_US.UTF-8' lc_ctype 'en_US.UTF-8';

      If you are also restoring the Data Warehouse, create the database on the relevant host:

      postgres=# create database database_name owner ovirt_engine_history template template0 encoding 'UTF8' lc_collate 'en_US.UTF-8' lc_ctype 'en_US.UTF-8';

   6. Exit the postgresql command line and log out of the postgres user:

      postgres=# \q
      $ exit

   7. Edit the /var/lib/pgsql/data/pg_hba.conf file as follows:
      • For each local database, replace the existing directives in the section starting with local at the bottom of the file with the following directives:

        host    database_name    user_name    0.0.0.0/0  md5
        host    database_name    user_name    ::0/0      md5

      • For each remote database:
        • Add the following line immediately underneath the line starting with Local at the bottom of the file, replacing X.X.X.X with the IP address of the Manager:

          host    database_name    user_name    X.X.X.X/32   md5

        • Allow TCP/IP connections to the database. Edit the /var/lib/pgsql/data/postgresql.conf file and add the following line:

          listen_addresses='*'

          This example configures the postgresql service to listen for connections on all interfaces. You can specify an interface by giving its IP address.
        • Open the default port used for PostgreSQL database connections, and save the updated firewall rules:

          # iptables -I INPUT 5 -p tcp -s Manager_IP_Address --dport 5432 -j ACCEPT
          # service iptables save

   8. Restart the postgresql service:

      # systemctl restart postgresql.service

2. Secure copy the backup files to the new Manager virtual machine. This example copies the files from a network storage server to which the files were copied in Section 6.1, “Backing up the Self-Hosted Engine Manager Virtual Machine”. In this example, Storage.example.com is the fully qualified domain name of the storage server, /backup/EngineBackupFiles is the designated file path for the backup files on the storage server, and /backup/ is the path to which the files will be copied on the new Manager.

   # scp -p Storage.example.com:/backup/EngineBackupFiles /backup/

3. Restore a complete backup or a database-only backup with the --change-db-credentials parameter to pass the credentials of the new database. The database_location for a database local to the Manager is localhost.

   Note

   The following examples use a --*password option for each database without specifying a password, which will prompt for a password for each database. Passwords can be supplied for these options in the command itself, however this is not recommended as the password will then be stored in the shell history. Alternatively, --*passfile=password_file options can be used for each database to securely pass the passwords to the engine-backup tool without the need for interactive prompts.
   • Restore a complete backup:

     # engine-backup --mode=restore --file=file_name --log=log_file_name --change-db-credentials --db-host=database_location --db-name=database_name --db-user=engine --db-password

     If Data Warehouse is also being restored as part of the complete backup, include the revised credentials for the additional database:

     engine-backup --mode=restore --file=file_name --log=log_file_name --change-db-credentials --db-host=database_location --db-name=database_name --db-user=engine --db-password --change-dwh-db-credentials --dwh-db-host=database_location --dwh-db-name=database_name --dwh-db-user=ovirt_engine_history --dwh-db-password

   • Restore a database-only backup restoring the configuration files and the database backup:

     # engine-backup --mode=restore --scope=files --scope=db --file=file_name --log=file_name --change-db-credentials --db-host=database_location --db-name=database_name --db-user=engine --db-password

     The example above restores a backup of the Manager database.

     # engine-backup --mode=restore --scope=files --scope=dwhdb --file=file_name --log=file_name --change-dwh-db-credentials --dwh-db-host=database_location --dwh-db-name=database_name --dwh-db-user=ovirt_engine_history --dwh-db-password

     The example above restores a backup of the Data Warehouse database.
   If successful, the following output displays:

   You should now run engine-setup.
   Done.

4. Configure the restored Manager virtual machine. This process identifies the existing configuration settings and database content. Confirm the settings. Upon completion, the setup provides an SSH fingerprint and an internal Certificate Authority hash.

   # engine-setup

   [ INFO  ] Stage: Initializing
   [ INFO  ] Stage: Environment setup
   Configuration files: ['/etc/ovirt-engine-setup.conf.d/10-packaging.conf', '/etc/ovirt-engine-setup.conf.d/20-setup-ovirt-post.conf']
   Log file: /var/log/ovirt-engine/setup/ovirt-engine-setup-20140304075238.log
   Version: otopi-1.1.2 (otopi-1.1.2-1.el6ev)
   [ INFO  ] Stage: Environment packages setup
   [ INFO  ] Yum Downloading: rhel-65-zstream/primary_db 2.8 M(70%)
   [ INFO  ] Stage: Programs detection
   [ INFO  ] Stage: Environment setup
   [ INFO  ] Stage: Environment customization
            
             --== PACKAGES ==--
            
   [ INFO  ] Checking for product updates...
   [ INFO  ] No product updates found
            
             --== NETWORK CONFIGURATION ==--
            
   Setup can automatically configure the firewall on this system.
   Note: automatic configuration of the firewall may overwrite current settings.
   Do you want Setup to configure the firewall? (Yes, No) [Yes]: 
   [ INFO  ] iptables will be configured as firewall manager.
            
             --== DATABASE CONFIGURATION ==--
            
            
             --== OVIRT ENGINE CONFIGURATION ==--
            
             Skipping storing options as database already prepared
            
             --== PKI CONFIGURATION ==--
            
             PKI is already configured
            
             --== APACHE CONFIGURATION ==--
            
            
             --== SYSTEM CONFIGURATION ==--
            
            
             --== END OF CONFIGURATION ==--
            
   [ INFO  ] Stage: Setup validation
   [ INFO  ] Cleaning stale zombie tasks
            
             --== CONFIGURATION PREVIEW ==--
            
             Database name                      : engine
             Database secured connection        : False
             Database host                      : X.X.X.X
             Database user name                 : engine
             Database host name validation      : False
             Database port                      : 5432
             NFS setup                          : True
             Firewall manager                   : iptables
             Update Firewall                    : True
             Configure WebSocket Proxy          : True
             Host FQDN                          : Manager.example.com
             NFS mount point                    : /var/lib/exports/iso
             Set application as default page    : True
             Configure Apache SSL               : True
            
             Please confirm installation settings (OK, Cancel) [OK]:

5. Removing the Host from the Restored Environment

   If the deployment of the restored self-hosted engine is on new hardware that has a unique name not present in the backed-up engine, skip this step. This step is only applicable to deployments occurring on the failover host, hosted_engine_1. Because this host was present in the environment at time the backup was created, it maintains a presence in the restored engine and must first be removed from the environment before final synchronization can take place.
   1. Log in to the Administration Portal.
   2. Click the Hosts tab. The failover host, hosted_engine_1, will be in maintenance mode and without a virtual load, as this was how it was prepared for the backup.
   3. Click Remove.
   4. Click Ok.

6. Synchronizing the Host and the Manager

   Return to the host and continue the hosted-engine deployment script by selecting option 1:

   (1) Continue setup - engine installation is complete

   [ INFO  ] Engine replied: DB Up!Welcome to Health Status!
   [ INFO  ] Waiting for the host to become operational in the engine. This may take several minutes...
   [ INFO  ] Still waiting for VDSM host to become operational...

   At this point, hosted_engine_1 will become visible in the Administration Portal with Installing and Initializing states before entering a Non Operational state. The host will continue to wait for the VDSM host to become operational until it eventually times out. This happens because another host in the environment maintains the Storage Pool Manager (SPM) role and hosted_engine_1 cannot interact with the storage domain because the SPM host is in a Non Responsive state. When this process times out, you are prompted to shut down the virtual machine to complete the deployment. When deployment is complete, the host can be manually placed into maintenance mode and activated through the Administration Portal.

   [ INFO  ] Still waiting for VDSM host to become operational...
   [ ERROR ] Timed out while waiting for host to start. Please check the logs.
   [ ERROR ] Unable to add hosted_engine_2 to the manager
             Please shutdown the VM allowing the system to launch it as a monitored service.
             The system will wait until the VM is down.

7. Shut down the new Manager virtual machine.

   # shutdown -h now

8. Return to the host to confirm it has detected that the Manager virtual machine is down.

   [ INFO  ] Enabling and starting HA services
             Hosted Engine successfully set up
   [ INFO  ] Stage: Clean up
   [ INFO  ] Stage: Pre-termination
   [ INFO  ] Stage: Termination
   

9. Activate the host.
   1. Log in to the Administration Portal.
   2. Click the Hosts tab.
   3. Select hosted_engine_1 and click the Maintenance button. The host may take several minutes before it enters maintenance mode.
   4. Click the Activate button.
   Once active, hosted_engine_1 immediately contends for SPM, and the storage domain and data center become active.
10. Migrate virtual machines to the active host by manually fencing the Non Responsive hosts. In the Administration Portal, right-click the hosts and select Confirm 'Host has been Rebooted'.
    Any virtual machines that were running on that host at the time of the backup will now be removed from that host, and move from an Unknown state to a Down state. These virtual machines can now be run on hosted_engine_1. The host that was fenced can now be forcefully removed using the REST API.

1. Fencing the Non-Operational Host

   In the Administration Portal, right-click the hosts and select Confirm 'Host has been Rebooted'.
   Any virtual machines that were running on that host at the time of the backup will now be removed from that host, and move from an Unknown state to a Down state. The host that was fenced can now be forcefully removed using the REST API.

2. Retrieving the Manager Certificate Authority

   Connect to the Manager virtual machine and use the command line to perform the following requests with cURL.
   Use a GET request to retrieve the Manager Certificate Authority (CA) certificate for use in all future API requests. In the following example, the --output option is used to designate the file hosted-engine.ca as the output for the Manager CA certificate. The --insecure option means that this initial request will be without a certificate.

   # curl --output hosted-engine.ca --insecure https://[Manager.example.com]/ca.crt

3. Retrieving the GUID of the Host to be Removed

   Use a GET request on the hosts collection to retrieve the Global Unique Identifier (GUID) for the host to be removed. The following example includes the Manager CA certificate file, and uses the admin@internal user for authentication, the password for which will be prompted once the command is executed.

   # curl --request GET --cacert hosted-engine.ca --user admin@internal https://[Manager.example.com]/api/hosts

   This request returns the details of all of the hosts in the environment. The host GUID is a hexadecimal string associated with the host name. For more information on the Red Hat Virtualization REST API, see the Red Hat Virtualization REST API Guide.

4. Removing the Fenced Host

   Use a DELETE request using the GUID of the fenced host to remove the host from the environment. In addition to the previously used options this example specifies headers to specify that the request is to be sent and returned using eXtensible Markup Language (XML), and the body in XML that sets the force action to be true.

   curl --request DELETE --cacert hosted-engine.ca --user admin@internal --header "Content-Type: application/xml" --header "Accept: application/xml" --data "<action><force>true</force></action>" https://[Manager.example.com]/api/hosts/ecde42b0-de2f-48fe-aa23-1ebd5196b4a5

   This DELETE request can be used to remove every fenced host in the self-hosted engine environment, as long as the appropriate GUID is specified.

5. Removing the Self-Hosted Engine Configuration from the Host

   Remove the host's self-hosted engine configuration so it can be reconfigured when the host is re-installed to a self-hosted engine environment.
   Log in to the host and remove the configuration file:

   # rm /etc/ovirt-hosted-engine/hosted-engine.conf