Red Hat Summit Red Hat SummitSupportConsoleDéveloppeursCommencer un essaiContact

Ce contenu n'est pas disponible dans la langue sélectionnée.

Chapter 8. Backup and disaster recovery

CodeReady Workspaces Operator can create backups of CodeReady Workspaces instances and restore them from a backup snapshot if needed. The following chapter describes ways of preparing such backups and their use in the follow-up recovery phase:

Important
  • The standard backup mechanism of CodeReady Workspaces does not back up the content of users' workspaces. To preserve local changes, see Section 8.6, “Persistent Volumes backups”.
  • Backup snapshots are bound to their own specific cluster and must be used only there.
  • CodeReady Workspaces Operator creates a new backup on every CodeReady Workspaces update.
  • Configured backup server is automatically used to store the backup.
  • When a CodeReady Workspaces administrator configures more than one backup server, the CodeReady Workspaces Operator uses the server with the che.eclipse.org/backup-before-update: true annotation by default.

  • CodeReady Workspaces Operator uses the internal backup server:

    • Every time the CodeReady Workspaces administrator does not configure the backup server.
    • When several backup servers do not have any annotation.

Additional resources

8.1. Setting up a backup server
Copier lien

The following section describes the supported CodeReady Workspaces backup servers and provides information for their setup.

Note
  • Red Hat CodeReady Workspaces Operator can automatically configure a backup server inside the same cluster; however, it is not recommended for production use.
  • Users who agreed to the limitations coming from the decision to back up their data inside the same OpenShift project as CodeReady Workspaces installation may skip this section.

CodeReady Workspaces uses the restic tool to:

  • manage backup snapshots

  • push to or to pull backup data from a backup server

    Note

    The restic backup tool is licensed under the BSD 2-Clause license.

The backup servers currently supported for CodeReady Workspaces:

REST
The REST server is a solution designed to cooperate with the restic tool. See How to set up a REST server documentation.
Amazon S3 and API compatible alternatives
See AWS S3 Simple Storage Service Documentation or the docs of alternative services that have compatible API with AWS.
SFTP
See How to configure an SFTP server .

8.2. Managing backups using crwctl
Copier lien

The following section describes how to create and use backups of a CodeReady Workspaces installation to perform a recovery or a rollback to a previous version using crwctl.

Prerequisites
  • Red Hat CodeReady Workspaces Operator can automatically configure a backup server inside the same cluster; however, it is not recommended for production use.
  • Users who agreed to the limitations coming from the decision to back up their data inside the same OpenShift project as CodeReady Workspaces installation may skip this section.

Procedure

8.2.1. Creating a new backup
Copier lien

  1. To create a backup snapshot and send it to a pre-configured backup server: 

    $ crwctl server:backup --repository-url=<repository-url> --repository-password=<repository-password>
    Copy to Clipboard Toggle word wrap
    • You can create other backups to the same backup server using the server:backup command with no arguments.
    • Using the server:backup command with no arguments for the first time will configure and use an internal backup server.

8.2.2. Restoring from a backup
Copier lien

A CodeReady Workspaces administrator can use an existing snapshot of a particular CodeReady Workspaces version to restore a desired state or version. The following instructions describe several variations of the restoration command. Adjust the command arguments according to your use case.

  • To restore the previous functional state of the same version of CodeReady Workspaces: 

    $ crwctl server:restore --repository-url=<repository-url> --repository-password=<repository-password> --snapshot-id=<snapshot-id>
    Copy to Clipboard Toggle word wrap

  • To roll back to a version different from the current version of CodeReady Workspaces: 

    $ crwctl server:restore --version=<version> --snapshot-id=<snapshot-id> --repository-url=<repository-url> --repository-password=<repository-password>
    Copy to Clipboard Toggle word wrap

    This performs a version rollback and restores a snapshot made from a previous version of CodeReady Workspaces. The provided snapshot must be created from the version of CodeReady Workspaces to which you want to roll back.

    Note

    If you have a dedicated backup repository for each CodeReady Workspaces version and want to use the most recent backup for the version, you can provide the latest argument as a snapshot ID. By doing so, the latest argument will be converted to the latest known ID in the given repository, which will be then used by the CodeReady Workspaces Operator to recover.

  • To restore a state described by an existing backup Custom Resource: 

    $ crwctl server:restore --backup-cr-name=<CheClusterBackupCRName>
    Copy to Clipboard Toggle word wrap

  • To roll back a version upgrade of CodeReady Workspaces: 

    $ crwctl server:restore --rollback
    Copy to Clipboard Toggle word wrap

    This recovers the version that CodeReady Workspaces was using before upgrading to a later version.

    Note

    CodeReady Workspaces Operator automatically creates a backup before every upgrade.

8.3. Configuring crwctl to use a backup server
Copier lien

The following section describes how to define environment variables for a specific backup server using the crwctl tool.

Procedure

  1. Determine backup server type and the server URL. Use the restic repository documentation as the reference.

    The URL can be specified with the -r parameter or defined using the BACKUP_REPOSITORY_URL environment variable.

  2. Retrieve or create a password for the backup repository.

    The password can be specified with the -p parameter or defined using the BACKUP_REPOSITORY_PASSWORD environment variable.

    Warning

    Backup data are encrypted with this password. The loss of the backup repository password will cause losing the data.

  3. Set the following environment variables for the chosen backup server type:

    REST
    When optional authentication is turned on, export REST_SERVER_USERNAME and REST_SERVER_PASSWORD environment variables.
    AWS S3
    Export the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables with AWS user credentials.
    SFTP

    For login without a password, export the SSH_KEY_FILE environment variable that holds the path to a file with a corresponding SSH key, or provide the --ssh-key-file parameter.

    Alternatively, the SSH_KEY environment variable that holds an SSH key itself can be used.

Note

It is possible to point directly to the backup server configuration object using --backup-server-config-name parameter or BACKUP_SERVER_CONFIG_NAME environment variable. In such a case, all the configuration above is not needed. For more details, see Section 8.4, “Managing backups using custom resources”

8.4. Managing backups using custom resources
Copier lien

The following section describes how to create backups of CodeReady Workspaces installation and recover directly using Custom Resource objects.

Note
  • Red Hat CodeReady Workspaces Operator can automatically configure a backup server inside the same cluster; however, it is not recommended for production use.
  • Users who agreed to the limitations coming from the decision to back up their data inside the same OpenShift project as CodeReady Workspaces installation may skip this section.

Prerequisites

Procedure

8.4.1. Creating a new backup
Copier lien

  1. Create a CheClusterBackup object to create a new backup: 

    apiVersion: org.eclipse.che/v1
kind: CheClusterBackup
metadata:
  name: CodeReady Workspaces-backup
spec:
  backupServerConfigRef: backup-server-configuration
    1
    Copy to Clipboard Toggle word wrap
1
Name of the CheBackupServerConfiguration object defining what backup server to use.
  • The creation of a CheClusterBackup object starts a new backup.

  • Before reusing the same name for a new backup object, delete the old object: 

    oc delete CheClusterBackup <name> -n openshift-workspaces
    Copy to Clipboard Toggle word wrap
Note

Editing the CheClusterBackup objects has no effect.

Alternative

To use the internal backup server, request automatic configuration from CodeReady Workspaces Operator. The preparation described above is not required.

  • Configure the automatic setup and sending of the backup to the internal backup server: 

    apiVersion: org.eclipse.che/v1
kind: CheClusterBackup
metadata:
  name: CodeReady Workspaces-backup
spec:
  useInternalBackupServer: true
    Copy to Clipboard Toggle word wrap

8.4.2. Restoring from a backup
Copier lien

Note

The approach described in this chapter can not be used to recover to a different version of CodeReady Workspaces. To recover CodeReady Workspaces to another version, use the crwctl tool. See the Section 8.2, “Managing backups using crwctl” chapter for more information.

  1. Create a new object of CheClusterRestore to recover a CodeReady Workspaces installation from a backup: 

    apiVersion: org.eclipse.che/v1
kind: CheClusterRestore
metadata:
  name: CodeReady Workspaces-restore
spec:
  backupServerConfigRef: backup-server-configuration
    1 
    
  snapshotId: ba92c7e0
    2
    Copy to Clipboard Toggle word wrap
1
Name of the CheBackupServerConfiguration object that defines what backup server to use.
2
Optional parameter defining the Snapshot ID to restore from. The default value is the last snapshot on the backup server.

  1. Create a new CheClusterRestore object to request a new recovery.

    • Before reusing the same name for a new backup object, delete the old object first: 

      oc delete CheClusterBackup <name> -n openshift-workspaces
      Copy to Clipboard Toggle word wrap

  2. Wait until the recovery process finishes.

    In a case of errors occurrences in your browser after the recovery, clean up the browser data for the CodeReady Workspaces domain.

Note

Editing of CheClusterRestore objects has no effect.

Verification

  1. Verify backup process state:

    1. Read the status section of the CheClusterBackup object to check the backup process: 

      status:
  message: 'Backup is in progress. Start time: <timestamp>'
      1 
      
  stage: Collecting CodeReady Workspaces installation data
      2 
      
  state: InProgress
      3 
      
  snapshotId: ba92c7e0
      4
      Copy to Clipboard Toggle word wrap
      1
      Displays the overall state or error message.
      2
      Current phase of the backup process in a human-readable format.
      3
      Backup process state. One of InProgress, Succeeded, or Failed.
      4
      ID of the created backup snapshot. The field appears only when state is Succeeded.

  2. Verify recovery process state

    1. Read the status section of the CheClusterRestore object to check the recovery process: 

      status:
  message: 'Restore is in progress. Start time: <timestamp>'
      1 
      
  stage: Restoring CodeReady Workspaces related cluster objects
      2 
      
  state: InProgress
      3
      Copy to Clipboard Toggle word wrap
      1
      Overall state or error message.
      2
      Current phase of the recovery process in a human-readable format.
      3
      Recovery process state. One of InProgress, Succeeded, or Failed.

8.5. Configuring CodeReady Workspaces to use a backup server
Copier lien

To configure a backup server for CodeReady Workspaces, a user needs to create the CheBackupServerConfiguration Custom Resource object in the openshift-workspaces namespace. The object’s spec property is divided in several sections where each corresponds to a specific backup server type:

  • REST
  • AWS S3 or API compatible

  • SFTP

    Note
    • The Custom Resource object, stored in the openshift-workspaces namespace, must have only one section configured in the spec property.
    • It is possible to configure as many backup servers as needed, but each in a separate Custom Resource.
    • Referenced secrets for each server type must exist and have required fields specified. See the description of each secret in the corresponding server-type chapters.

8.5.1. Configuring REST server
Copier lien

apiVersion: org.eclipse.che/v1
kind: CheBackupServerConfiguration
metadata:
  name: backup-server-configuration
spec:
  rest:
    protocol: http
1 

    hostname: my-domain.net
2 

    port: 1234
3 

    repositoryPath: CodeReady Workspaces-backups
4 

    repositoryPasswordSecretRef: backup-encryption-password-secret
5 

    credentialsSecretRef: rest-server-auth-secret
6
Copy to Clipboard Toggle word wrap
1
Optional property that specifies the protocol to be used. The default value is https with http as the second allowed option.
2
Backup server host name.
3
Optional property that specifies the port on which the backup server is running. The default value is 8000.
4
Path on the backup server where the backup snapshots are stored.
5
Secret name containing a repository password, stored in the repo-password field. If the secret contains only one field, its name is arbitrary. The password is used to encrypt and decrypt backup snapshots data.
6
Optional property that specifies the name of the secret with the REST server user credentials, stored in the username and password fields.

8.5.2. Configuring AWS S3 or API compatible server
Copier lien

apiVersion: org.eclipse.che/v1
kind: CheBackupServerConfiguration
metadata:
  name: backup-server-configuration
spec:
  awss3:
    protocol: https
1 

    hostname: my-domain.net
2 

    port: 1234
3 

    repositoryPath: CodeReady Workspaces-backups
4 

    repositoryPasswordSecretRef: backup-encryption-password-secret
5 

    awsAccessKeySecretRef: aws-user-credentials-secret
6
Copy to Clipboard Toggle word wrap
1
Optional property that specifies the protocol to be used. The default value is https with http as the second allowed option.
2
Optional property that specifies the S3 host name. The default value is s3.amazonaws.com.
3
Optional property that specifies the port on which the backup server is running.
4
The name of the bucket resource where the backup snapshots are stored. The bucket resource must be manually pre-created.
5
The name of the secret containing a repository password, stored in the repo-password field. If the secret contains only one field, this name is arbitrary. The password is used to encrypt and decrypt backup snapshots data.
6
The name of the secret containing user credentials stored in the awsAccessKeyId and awsSecretAccessKey fields.

8.5.3. Configuring SFTP server
Copier lien

apiVersion: org.eclipse.che/v1
kind: CheBackupServerConfiguration
metadata:
  name: backup-server-configuration
spec:
  awss3:
    username: user
1 

    hostname: my-domain.net
2 

    port: 1234
3 

    repositoryPath: CodeReady Workspaces-backups
4 

    repositoryPasswordSecretRef: backup-encryption-password-secret
5 

    sshKeySecretRef: ssh-key-secret
6
Copy to Clipboard Toggle word wrap
1
User name on the remote server to login with using the SSH protocol.
2
Remote server host name.
3
Optional property that specifies the port on which an SFTP server is running. The default value is 22.
4
Absolute or relative path on the server where backup snapshots are stored.
5
The name of the secret containing a repository password, stored in the repo-password field. If the secret contains only one field, this name is arbitrary. The password is used to encrypt and decrypt backup snapshots data.
6
The name of the secret containing a private SSH key, stored in the ssh-privatekey field. This SSH key can be used to perform a login without a password on an SFTP server.

8.6. Persistent Volumes backups
Copier lien

Persistent Volumes (PVs) store the CodeReady Workspaces workspace data similarly to how workspace data is stored for desktop IDEs on the local hard disk drive.

To prevent data loss, back up PVs periodically. The recommended approach is to use storage-agnostic tools for backing up and restoring OpenShift resources, including PVs.

8.7. External database setup
Copier lien

The PostgreSQL database is used by the CodeReady Workspaces server for persisting data about the state of CodeReady Workspaces. It contains information about user accounts, workspaces, preferences, and other details.

By default, the CodeReady Workspaces Operator creates and manages the database deployment.

However, the CodeReady Workspaces Operator does not support full life-cycle capabilities, such as backups and recovery.

For a business-critical setup, configure an external database with the following recommended disaster-recovery options:

  • High Availability (HA)
  • Point In Time Recovery (PITR)

Configure an external PostgreSQL instance on-premises or use a cloud service, such as Amazon Relational Database Service (Amazon RDS). With Amazon RDS, it is possible to deploy production databases in a Multi-Availability Zone configuration for a resilient disaster recovery strategy with daily and on-demand snapshots.

The recommended configuration of the example database is:

Expand
ParameterValue

Instance class

db.t2.small

vCPU

1

RAM

2 GB

Multi-az

true, 2 replicas

Engine version

9.6.11

TLS

enabled

Automated backups

enabled (30 days)

8.7.1. Configuring external PostgreSQL
Copier lien

By configuring the external PostgreSQL, you can make the workspace metadata and the user information persistent.

Procedure

  1. Define the values of the following placeholders:

    • <database-user> is the CodeReady Workspaces server database user name
    • <database-password> is the CodeReady Workspaces server database password
    • <database> is the CodeReady Workspaces server database name

  2. Use the following SQL script to create a user and a database for the CodeReady Workspaces server to make workspace metadata persistent: 

    CREATE USER <database-user> WITH PASSWORD '<database-password>'
CREATE DATABASE <database>
GRANT ALL PRIVILEGES ON DATABASE <database> TO <database-user>
ALTER USER <database-user> WITH SUPERUSER
    Copy to Clipboard Toggle word wrap

  3. Define the value of the following placeholder:

    • <identity-database-password> is the RH-SSO database password

  4. Use the following SQL script to create a database for the RH-SSO back end to make the user information persistent: 

    CREATE USER keycloak WITH PASSWORD '<identity-database-password>'
CREATE DATABASE keycloak
GRANT ALL PRIVILEGES ON DATABASE keycloak TO keycloak
    Copy to Clipboard Toggle word wrap

Prerequisites

  • The oc tool is available.

Procedure

  1. Pre-create a project for CodeReady Workspaces: 

    $ oc create namespace openshift-workspaces
    Copy to Clipboard Toggle word wrap

  2. Create a secret to store CodeReady Workspaces server database credentials: 

    $ oc create secret generic <server-database-credentials> \
    1 
    
--from-literal=user=<database-user> \
    2 
    
--from-literal=password=<database-password> \
    3 
    
-n openshift-workspaces
    Copy to Clipboard Toggle word wrap
    1
    Secret name to store CodeReady Workspaces server database credentials
    2
    CodeReady Workspaces server database username
    3
    CodeReady Workspaces server database password

  3. Add the required labels to the CodeReady Workspaces server database credentials secret: 

    $ oc label secret <server-database-credentials> \
    1 
    
 app.kubernetes.io/part-of=che.eclipse.org -n openshift-workspaces
    Copy to Clipboard Toggle word wrap
    1
    Secret name to store CodeReady Workspaces server database credentials

  4. Create a secret to store RH-SSO database credentials: 

    $ oc create secret generic <identity-database-credentials> \
    1 
    
--from-literal=password=<identity-database-password> \
    2 
    
-n openshift-workspaces
    Copy to Clipboard Toggle word wrap
    1
    Secret name to store RH-SSO database credentials
    2
    RH-SSO database password

  5. Add the required labels to the RH-SSO database credentials secret: 

    $ oc label secret <identity-database-credentials> \
    1 
    
app.kubernetes.io/part-of=che.eclipse.org -n openshift-workspaces
    Copy to Clipboard Toggle word wrap
    1
    Secret name to store RH-SSO database credentials

  6. Deploy Red Hat CodeReady Workspaces by executing the crwctl command with applying a patch. For example: 

    $ crwctl server:deploy --che-operator-cr-patch-yaml=patch.yaml ...
    Copy to Clipboard Toggle word wrap

patch.yaml should contain the following to make the Operator skip deploying a database and pass connection details of an existing database to a CodeReady Workspaces server: 

spec:
  database:
    externalDb: true
    chePostgresHostName: <hostname>
1 

    chePostgresPort: <port>
2 

    chePostgresSecret: <server-database-credentials>
3 

    chePostgresDb: <database>
4 

spec:
  auth:
    identityProviderPostgresSecret: <identity-database-credentials>
5
Copy to Clipboard Toggle word wrap
1
External database host name
2
External database port
3
Secret name with CodeReady Workspaces server database credentials
4
CodeReady Workspaces server database name
5
Secret name with RH-SSO database credentials

Additional resources

Retour au début
Red Hat logoGithubredditYoutubeTwitter

Apprendre

Essayez, achetez et vendez

Communautés

À propos de la documentation Red Hat

Nous aidons les utilisateurs de Red Hat à innover et à atteindre leurs objectifs grâce à nos produits et services avec un contenu auquel ils peuvent faire confiance. Découvrez nos récentes mises à jour.

Rendre l’open source plus inclusif

Red Hat s'engage à remplacer le langage problématique dans notre code, notre documentation et nos propriétés Web. Pour plus de détails, consultez le Blog Red Hat.

À propos de Red Hat

Nous proposons des solutions renforcées qui facilitent le travail des entreprises sur plusieurs plates-formes et environnements, du centre de données central à la périphérie du réseau.

Theme

© 2025 Red Hat