Chapter 7. Backup and disaster recovery
CodeReady Workspaces Operator can create backups of CodeReady Workspaces instances and restore them from a backup snapshot if needed. The following chapter describes ways of preparing such backups and their use in the follow-up recovery phase:
- The standard backup mechanism of CodeReady Workspaces does not back up the content of users' workspaces. To preserve local changes, see Section 7.6, “Persistent Volumes backups”.
- Backup snapshots are bound to their own specific cluster and must be used only there.
- CodeReady Workspaces Operator creates a new backup on every CodeReady Workspaces update.
- Configured backup server is automatically used to store the backup.
-
When a CodeReady Workspaces administrator configures more than one backup server, the CodeReady Workspaces Operator uses the server with the
che.eclipse.org/backup-before-update: true
annotation by default. CodeReady Workspaces Operator uses the internal backup server:
- Every time the CodeReady Workspaces administrator does not configure the backup server.
- When several backup servers do not have any annotation.
Additional resources
7.1. Setting up a backup server
The following section describes the supported CodeReady Workspaces backup servers and provides information for their setup.
- Red Hat CodeReady Workspaces Operator can automatically configure a backup server inside the same cluster; however, it is not recommended for production use.
- Users who agreed to the limitations coming from the decision to back up their data inside the same OpenShift project as CodeReady Workspaces installation may skip this section.
CodeReady Workspaces uses the restic
tool to:
- manage backup snapshots
push to or to pull backup data from a backup server
NoteThe
restic
backup tool is licensed under the BSD 2-Clause license.
The backup servers currently supported for CodeReady Workspaces:
- REST
-
The REST server is a solution designed to cooperate with the
restic
tool. See How to set up a REST server documentation. - Amazon S3 and API compatible alternatives
- See AWS S3 Simple Storage Service Documentation or the docs of alternative services that have compatible API with AWS.
- SFTP
- See How to configure an SFTP server .
7.2. Managing backups using crwctl
The following section describes how to create and use backups of a CodeReady Workspaces installation to perform a recovery or a rollback to a previous version using crwctl
.
- Red Hat CodeReady Workspaces Operator can automatically configure a backup server inside the same cluster; however, it is not recommended for production use.
- Users who agreed to the limitations coming from the decision to back up their data inside the same OpenShift project as CodeReady Workspaces installation may skip this section.
7.2.1. Creating a new backup
To create a backup snapshot and send it to a pre-configured backup server:
$ crwctl server:backup --repository-url=<repository-url> --repository-password=<repository-password>
-
You can create other backups to the same backup server using the
server:backup
command with no arguments. -
Using the
server:backup
command with no arguments for the first time will configure and use an internal backup server.
-
You can create other backups to the same backup server using the
7.2.2. Restoring from a backup
A CodeReady Workspaces administrator can use an existing snapshot of a particular CodeReady Workspaces version to restore a desired state or version. The following instructions describe several variations of the restoration command. Adjust the command arguments according to your use case.
To restore the previous functional state of the same version of CodeReady Workspaces:
$ crwctl server:restore --repository-url=<repository-url> --repository-password=<repository-password> --snapshot-id=<snapshot-id>
To roll back to a version different from the current version of CodeReady Workspaces:
$ crwctl server:restore --version=<version> --snapshot-id=<snapshot-id> --repository-url=<repository-url> --repository-password=<repository-password>
This performs a version rollback and restores a snapshot made from a previous version of CodeReady Workspaces. The provided snapshot must be created from the version of CodeReady Workspaces to which you want to roll back.
NoteIf you have a dedicated backup repository for each CodeReady Workspaces version and want to use the most recent backup for the version, you can provide the
latest
argument as a snapshot ID. By doing so, thelatest
argument will be converted to the latest known ID in the given repository, which will be then used by the CodeReady Workspaces Operator to recover.To restore a state described by an existing backup Custom Resource:
$ crwctl server:restore --backup-cr-name=<CheClusterBackupCRName>
To roll back a version upgrade of CodeReady Workspaces:
$ crwctl server:restore --rollback
This recovers the version that CodeReady Workspaces was using before upgrading to a later version.
NoteCodeReady Workspaces Operator automatically creates a backup before every upgrade.
7.3. Configuring crwctl to use a backup server
The following section describes how to define environment variables for a specific backup server using the crwctl tool.
Procedure
Determine backup server type and the server URL. Use the restic repository documentation as the reference.
The URL can be specified with the
-r
parameter or defined using theBACKUP_REPOSITORY_URL
environment variable.Retrieve or create a password for the backup repository.
The password can be specified with the
-p
parameter or defined using theBACKUP_REPOSITORY_PASSWORD
environment variable.WarningBackup data are encrypted with this password. The loss of the backup repository password will cause losing the data.
Set the following environment variables for the chosen backup server type:
- REST
-
When optional authentication is turned on, export
REST_SERVER_USERNAME
andREST_SERVER_PASSWORD
environment variables. - AWS S3
-
Export the
AWS_ACCESS_KEY_ID
andAWS_SECRET_ACCESS_KEY
environment variables with AWS user credentials. - SFTP
For login without a password, export the
SSH_KEY_FILE
environment variable that holds the path to a file with a corresponding SSH key, or provide the--ssh-key-file
parameter.Alternatively, the
SSH_KEY
environment variable that holds an SSH key itself can be used.
It is possible to point directly to the backup server configuration object using --backup-server-config-name
parameter or BACKUP_SERVER_CONFIG_NAME
environment variable. In such a case, all the configuration above is not needed. For more details, see Section 7.4, “Managing backups using custom resources”
7.4. Managing backups using custom resources
The following section describes how to create backups of CodeReady Workspaces installation and recover directly using Custom Resource objects.
- Red Hat CodeReady Workspaces Operator can automatically configure a backup server inside the same cluster; however, it is not recommended for production use.
- Users who agreed to the limitations coming from the decision to back up their data inside the same OpenShift project as CodeReady Workspaces installation may skip this section.
Prerequisites
7.4.1. Creating a new backup
Create a
CheClusterBackup
object to create a new backup:apiVersion: org.eclipse.che/v1 kind: CheClusterBackup metadata: name: CodeReady Workspaces-backup spec: backupServerConfigRef: backup-server-configuration 1
- 1
- Name of the
CheBackupServerConfiguration
object defining what backup server to use.-
The creation of a
CheClusterBackup
object starts a new backup. Before reusing the same name for a new backup object, delete the old object:
oc delete CheClusterBackup <name> -n openshift-workspaces
-
The creation of a
Editing the CheClusterBackup
objects has no effect.
Alternative
To use the internal backup server, request automatic configuration from CodeReady Workspaces Operator. The preparation described above is not required.
Configure the automatic setup and sending of the backup to the internal backup server:
apiVersion: org.eclipse.che/v1 kind: CheClusterBackup metadata: name: CodeReady Workspaces-backup spec: useInternalBackupServer: true
7.4.2. Restoring from a backup
The approach described in this chapter can not be used to recover to a different version of CodeReady Workspaces. To recover CodeReady Workspaces to another version, use the crwctl
tool. See the Section 7.2, “Managing backups using crwctl” chapter for more information.
Create a new object of
CheClusterRestore
to recover a CodeReady Workspaces installation from a backup:apiVersion: org.eclipse.che/v1 kind: CheClusterRestore metadata: name: CodeReady Workspaces-restore spec: backupServerConfigRef: backup-server-configuration 1 snapshotId: ba92c7e0 2
- 1
- Name of the
CheBackupServerConfiguration
object that defines what backup server to use. - 2
- Optional parameter defining the Snapshot ID to restore from. The default value is the last snapshot on the backup server.
Create a new
CheClusterRestore
object to request a new recovery.Before reusing the same name for a new backup object, delete the old object first:
oc delete CheClusterBackup <name> -n openshift-workspaces
Wait until the recovery process finishes.
In a case of errors occurrences in your browser after the recovery, clean up the browser data for the CodeReady Workspaces domain.
Editing of CheClusterRestore
objects has no effect.
Verification
Verify backup process state:
Read the
status
section of theCheClusterBackup
object to check the backup process:status: message: 'Backup is in progress. Start time: <timestamp>' 1 stage: Collecting CodeReady Workspaces installation data 2 state: InProgress 3 snapshotId: ba92c7e0 4
Verify recovery process state
Read the
status
section of theCheClusterRestore
object to check the recovery process:status: message: 'Restore is in progress. Start time: <timestamp>' 1 stage: Restoring CodeReady Workspaces related cluster objects 2 state: InProgress 3
7.5. Configuring CodeReady Workspaces to use a backup server
To configure a backup server for CodeReady Workspaces, a user needs to create the CheBackupServerConfiguration
Custom Resource object in the openshift-workspaces namespace. The object’s spec
property is divided in several sections where each corresponds to a specific backup server type:
- REST
- AWS S3 or API compatible
- Note
-
The Custom Resource object, stored in the
openshift-workspaces
namespace, must have only one section configured in thespec
property. -
It is possible to configure as many backup servers as needed, but each in a separate
Custom Resource
. - Referenced secrets for each server type must exist and have required fields specified. See the description of each secret in the corresponding server-type chapters.
-
The Custom Resource object, stored in the
7.5.1. Configuring REST server
apiVersion: org.eclipse.che/v1 kind: CheBackupServerConfiguration metadata: name: backup-server-configuration spec: rest: protocol: http 1 hostname: my-domain.net 2 port: 1234 3 repositoryPath: CodeReady Workspaces-backups 4 repositoryPasswordSecretRef: backup-encryption-password-secret 5 credentialsSecretRef: rest-server-auth-secret 6
- 1
- Optional property that specifies the protocol to be used. The default value is
https
withhttp
as the second allowed option. - 2
- Backup server host name.
- 3
- Optional property that specifies the port on which the backup server is running. The default value is
8000
. - 4
- Path on the backup server where the backup snapshots are stored.
- 5
- Secret name containing a repository password, stored in the
repo-password
field. If the secret contains only one field, its name is arbitrary. The password is used to encrypt and decrypt backup snapshots data. - 6
- Optional property that specifies the name of the secret with the REST server user credentials, stored in the
username
andpassword
fields.
7.5.2. Configuring AWS S3 or API compatible server
apiVersion: org.eclipse.che/v1 kind: CheBackupServerConfiguration metadata: name: backup-server-configuration spec: awss3: protocol: https 1 hostname: my-domain.net 2 port: 1234 3 repositoryPath: CodeReady Workspaces-backups 4 repositoryPasswordSecretRef: backup-encryption-password-secret 5 awsAccessKeySecretRef: aws-user-credentials-secret 6
- 1
- Optional property that specifies the protocol to be used. The default value is
https
withhttp
as the second allowed option. - 2
- Optional property that specifies the S3 host name. The default value is
s3.amazonaws.com
. - 3
- Optional property that specifies the port on which the backup server is running.
- 4
- The name of the bucket resource where the backup snapshots are stored. The bucket resource must be manually pre-created.
- 5
- The name of the secret containing a repository password, stored in the
repo-password
field. If the secret contains only one field, this name is arbitrary. The password is used to encrypt and decrypt backup snapshots data. - 6
- The name of the secret containing user credentials stored in the
awsAccessKeyId
andawsSecretAccessKey
fields.
7.5.3. Configuring SFTP server
apiVersion: org.eclipse.che/v1 kind: CheBackupServerConfiguration metadata: name: backup-server-configuration spec: awss3: username: user 1 hostname: my-domain.net 2 port: 1234 3 repositoryPath: CodeReady Workspaces-backups 4 repositoryPasswordSecretRef: backup-encryption-password-secret 5 sshKeySecretRef: ssh-key-secret 6
- 1
- User name on the remote server to login with using the SSH protocol.
- 2
- Remote server host name.
- 3
- Optional property that specifies the port on which an SFTP server is running. The default value is
22
. - 4
- Absolute or relative path on the server where backup snapshots are stored.
- 5
- The name of the secret containing a repository password, stored in the
repo-password
field. If the secret contains only one field, this name is arbitrary. The password is used to encrypt and decrypt backup snapshots data. - 6
- The name of the secret containing a private SSH key, stored in the
ssh-privatekey
field. This SSH key can be used to perform a login without a password on an SFTP server.
7.6. Persistent Volumes backups
Persistent Volumes (PVs) store the CodeReady Workspaces workspace data similarly to how workspace data is stored for desktop IDEs on the local hard disk drive.
To prevent data loss, back up PVs periodically. The recommended approach is to use storage-agnostic tools for backing up and restoring OpenShift resources, including PVs.
7.6.1. Recommended backup tool: Velero
Velero is an open-source tool for backing up OpenShift applications and their PVs. Velero allows you to:
- Deploy in the cloud or on premises.
- Back up the cluster and restore in case of data loss.
- Migrate cluster resources to other clusters.
- Replicate a production cluster to development and testing clusters.
Alternatively, you can use backup solutions dependent on the underlying storage system. For example, solutions that are Gluster or Ceph-specific.
Additional resources
7.7. External database setup
The PostgreSQL database is used by the CodeReady Workspaces server for persisting data about the state of CodeReady Workspaces. It contains information about user accounts, workspaces, preferences, and other details.
By default, the CodeReady Workspaces Operator creates and manages the database deployment.
However, the CodeReady Workspaces Operator does not support full life-cycle capabilities, such as backups and recovery.
For a business-critical setup, configure an external database with the following recommended disaster-recovery options:
- High Availability (HA)
- Point In Time Recovery (PITR)
Configure an external PostgreSQL instance on-premises or use a cloud service, such as Amazon Relational Database Service (Amazon RDS). With Amazon RDS, it is possible to deploy production databases in a Multi-Availability Zone configuration for a resilient disaster recovery strategy with daily and on-demand snapshots.
The recommended configuration of the example database is:
Parameter | Value |
---|---|
Instance class | db.t2.small |
vCPU | 1 |
RAM | 2 GB |
Multi-az | true, 2 replicas |
Engine version | 9.6.11 |
TLS | enabled |
Automated backups | enabled (30 days) |
7.7.1. Configuring external PostgreSQL
By configuring the external PostgreSQL, you can make the workspace metadata and the user information persistent.
Procedure
Define the values of the following placeholders:
-
<database-user>
is the CodeReady Workspaces server database user name -
<database-password>
is the CodeReady Workspaces server database password -
<database>
is the CodeReady Workspaces server database name
-
Use the following SQL script to create a user and a database for the CodeReady Workspaces server to make workspace metadata persistent:
CREATE USER <database-user> WITH PASSWORD '<database-password>' CREATE DATABASE <database> GRANT ALL PRIVILEGES ON DATABASE <database> TO <database-user> ALTER USER <database-user> WITH SUPERUSER
Define the value of the following placeholder:
-
<identity-database-password>
is the RH-SSO database password
-
Use the following SQL script to create a database for the RH-SSO back end to make the user information persistent:
CREATE USER keycloak WITH PASSWORD '<identity-database-password>' CREATE DATABASE keycloak GRANT ALL PRIVILEGES ON DATABASE keycloak TO keycloak
7.7.2. Configuring CodeReady Workspaces to work with an external PostgreSQL
Prerequisites
-
The
oc
tool is available.
Procedure
Pre-create a project for CodeReady Workspaces:
$ oc create namespace openshift-workspaces
Create a secret to store CodeReady Workspaces server database credentials:
$ oc create secret generic <server-database-credentials> \ 1 --from-literal=user=<database-user> \ 2 --from-literal=password=<database-password> \ 3 -n openshift-workspaces
Create a secret to store RH-SSO database credentials:
$ oc create secret generic <identity-database-credentials> \ 1 --from-literal=password=<identity-database-password> \ 2 -n openshift-workspaces
Deploy Red Hat CodeReady Workspaces by executing the
crwctl
command with applying a patch. For example:$ crwctl server:deploy --che-operator-cr-patch-yaml=patch.yaml ...
patch.yaml should contain the following to make the Operator skip deploying a database and pass connection details of an existing database to a CodeReady Workspaces server:
spec: database: externalDb: true chePostgresHostName: <hostname> 1 chePostgresPort: <port> 2 chePostgresSecret: <server-database-credentials> 3 chePostgresDb: <database> 4 spec: auth: identityProviderPostgresSecret: <identity-database-credentials> 5
Additional resources