红帽全球峰会此内容没有您所选择的语言版本。
Chapter 6. 3scale backup and restore
This section provides you, as the administrator of a Red Hat 3scale API Management installation, the information needed to:
- Set up the backup procedures for persistent data
- Perform a restore from backup of the persistent data
In case of a failure with one or more of the MySQL databases, you will be able to restore 3scale correctly to its previous operational state.
6.1. Prerequisites
- A 3scale 2.6 instance. For more information about how to install 3scale, see Chapter 2. Installation guide for 3scale on OpenShift.
An OpenShift Container Platform 4.x user account with one of the following roles in the OpenShift cluster:
- cluster-admin
- admin
- edit
A user with an edit cluster role locally binded in the namespace of a 3scale installation can perform backup and restore procedures.
6.2. Persistent volumes
In a 3scale deployment on OpenShift, all persistent data is stored either in a storage service in the cluster (not currently used), a persistent volume (PV) provided to the cluster by the underlying infrastructure, or a storage service external to the cluster, either in the same data center or elsewhere.
6.3. Considerations
The backup and restore procedures for persistent data vary depending on the storage used, to ensure the backups and restores preserve data consistency, for example, that a partial write, or a partial transaction is not captured. That is, it is not sufficient to backup the underlying PV for a database, but instead the databases backup mechanisms should be used.
Also, some parts of the data are synchronized between different components. One copy is considered the source of truth for the data set, and the other is a copy that is not modified locally, but synchronized from the source of truth. In these cases, upon restore, the source of truth should be restored and then the copies in other components synchronized from it.
6.4. Using data sets
This section explains in more detail about different data sets in the different persistent stores, their purpose, the storage type used, and whether or not it is the source of truth.
The full state of a 3scale deployment is stored across the following DeploymentConfig objects and their PVs:
| Name | Description |
|---|---|
|
MySQL database ( | |
| Volume for Files | |
| |
|
Redis database ( | |
|
Redis database ( |
6.4.1. Defining system-mysql
system-mysql is a relational database which stores information about users, accounts, APIs, plans, and more, in the 3scale Admin Console.
A subset of this information related to services is synchronized to the Backend component and stored in backend-redis. system-mysql is the source of truth for this information.
6.4.2. Defining system-storage
system-storage stores files to be read and written by the System component.
They fall into two categories:
-
Configuration files read by the
Systemcomponent at run-time - Static files, for example, HTML, CSS, JS, uploaded to system by its CMS feature, for the purpose of creating a Developer Portal
System can be scaled horizontally with multiple pods uploading and reading said static files, hence the need for a ReadWriteMany (RWX) PersistentVolume.
6.4.3. Defining zync-database
A zync-database is a relational database which stores information related to the synchronization of identities between 3scale and an Identity provider (IdP). This information is not duplicated in other components and is the sole source of truth.
6.4.4. Defining backend-redis
backend-redis contains multiple data sets used by the Backend component:
-
Usages: This is API usage information aggregated by
Backend. It is used byBackendfor rate-limiting decisions and bySystemto display analytics information in the UI or via API. -
Config: This is configuration information about services, rate-limits, and more, that is synchronized from
Systemvia an internal API. This is not the source of truth of this information, howeverSystemandsystem-mysqlis. -
AuthKeys: Storage of OAuth keys created directly in
Backend. This is the source of truth for this information. - Queues: This is queues of background jobs to be executed by worker processes. These are ephemeral and are deleted once processed.
6.4.5. Defining system-redis
system-redis contains queues for jobs to be processed in background. These are ephemeral and are deleted once processed.
6.5. Backup procedures
The following commands are used to back up and archive system databases.
6.5.1. Backing up system-mysql
Execute MySQL Backup Command:
oc rsh $(oc get pods -l 'deploymentConfig=system-mysql' -o json | jq -r '.items[0].metadata.name') bash -c 'export MYSQL_PWD=${MYSQL_ROOT_PASSWORD}; mysqldump --single-transaction -hsystem-mysql -uroot system' | gzip > system-mysql-backup.gz
oc rsh $(oc get pods -l 'deploymentConfig=system-mysql' -o json | jq -r '.items[0].metadata.name') bash -c 'export MYSQL_PWD=${MYSQL_ROOT_PASSWORD}; mysqldump --single-transaction -hsystem-mysql -uroot system' | gzip > system-mysql-backup.gz6.5.2. Backing up system-storage
Archive the system-storage files to another storage:
oc rsync $(oc get pods -l 'deploymentConfig=system-app' -o json | jq '.items[0].metadata.name' -r):/opt/system/public/system ./local/dir
oc rsync $(oc get pods -l 'deploymentConfig=system-app' -o json | jq '.items[0].metadata.name' -r):/opt/system/public/system ./local/dir6.5.3. Backing up zync-database
Execute Postgres Backup Command:
oc rsh $(oc get pods -l 'deploymentConfig=zync-database' -o json | jq '.items[0].metadata.name' -r) bash -c 'pg_dumpall -c --if-exists' | gzip > zync-database-backup.gz
oc rsh $(oc get pods -l 'deploymentConfig=zync-database' -o json | jq '.items[0].metadata.name' -r) bash -c 'pg_dumpall -c --if-exists' | gzip > zync-database-backup.gz6.5.4. Backing up backend-redis
Backup the dump.rb file from redis:
oc cp $(oc get pods -l 'deploymentConfig=backend-redis' -o json | jq '.items[0].metadata.name' -r):/var/lib/redis/data/dump.rdb ./backend-redis-dump.rdb
oc cp $(oc get pods -l 'deploymentConfig=backend-redis' -o json | jq '.items[0].metadata.name' -r):/var/lib/redis/data/dump.rdb ./backend-redis-dump.rdb6.5.5. Backing up system-redis
Backup the dump.rb file from redis:
oc cp $(oc get pods -l 'deploymentConfig=system-redis' -o json | jq '.items[0].metadata.name' -r):/var/lib/redis/data/dump.rdb ./system-redis-dump.rdb
oc cp $(oc get pods -l 'deploymentConfig=system-redis' -o json | jq '.items[0].metadata.name' -r):/var/lib/redis/data/dump.rdb ./system-redis-dump.rdb6.6. Procedures to restore databases
You can use the following commands to restore system databases after a system failure has occurred.
6.6.1. Restoring system-mysql
Copy the MySQL dump to the system-mysql pod:
oc cp ./system-mysql-backup.gz $(oc get pods -l 'deploymentConfig=system-mysql' -o json | jq '.items[0].metadata.name' -r):/var/lib/mysql
oc cp ./system-mysql-backup.gz $(oc get pods -l 'deploymentConfig=system-mysql' -o json | jq '.items[0].metadata.name' -r):/var/lib/mysqlCopy to Clipboard Copied! Toggle word wrap Toggle overflow Decompress the Backup File:
oc rsh $(oc get pods -l 'deploymentConfig=system-mysql' -o json | jq -r '.items[0].metadata.name') bash -c 'gzip -d ${HOME}/system-mysql-backup.gz'oc rsh $(oc get pods -l 'deploymentConfig=system-mysql' -o json | jq -r '.items[0].metadata.name') bash -c 'gzip -d ${HOME}/system-mysql-backup.gz'Copy to Clipboard Copied! Toggle word wrap Toggle overflow Restore the MySQL DB Backup file:
oc rsh $(oc get pods -l 'deploymentConfig=system-mysql' -o json | jq -r '.items[0].metadata.name') bash -c 'export MYSQL_PWD=${MYSQL_ROOT_PASSWORD}; mysql -hsystem-mysql -uroot system < ${HOME}/system-mysql-backup'oc rsh $(oc get pods -l 'deploymentConfig=system-mysql' -o json | jq -r '.items[0].metadata.name') bash -c 'export MYSQL_PWD=${MYSQL_ROOT_PASSWORD}; mysql -hsystem-mysql -uroot system < ${HOME}/system-mysql-backup'Copy to Clipboard Copied! Toggle word wrap Toggle overflow
6.6.2. Restoring system-storage
Restore the Backup file to system-storage:
oc rsync ./local/dir $(oc get pods -l 'deploymentConfig=system-app' -o json | jq '.items[0].metadata.name' -r):/opt/system/public/system
oc rsync ./local/dir $(oc get pods -l 'deploymentConfig=system-app' -o json | jq '.items[0].metadata.name' -r):/opt/system/public/system6.6.3. Restoring zync-database
Copy the Zync Database dump to the
zync-databasepod:oc cp ./zync-database-backup.gz $(oc get pods -l 'deploymentConfig=zync-database' -o json | jq '.items[0].metadata.name' -r):/var/lib/pgsql/
oc cp ./zync-database-backup.gz $(oc get pods -l 'deploymentConfig=zync-database' -o json | jq '.items[0].metadata.name' -r):/var/lib/pgsql/Copy to Clipboard Copied! Toggle word wrap Toggle overflow Decompress the Backup File:
oc rsh $(oc get pods -l 'deploymentConfig=zync-database' -o json | jq -r '.items[0].metadata.name') bash -c 'gzip -d ${HOME}/zync-database-backup.gz'oc rsh $(oc get pods -l 'deploymentConfig=zync-database' -o json | jq -r '.items[0].metadata.name') bash -c 'gzip -d ${HOME}/zync-database-backup.gz'Copy to Clipboard Copied! Toggle word wrap Toggle overflow Restore the PostgreSQL DB Backup file:
oc rsh $(oc get pods -l 'deploymentConfig=zync-database' -o json | jq -r '.items[0].metadata.name') bash -c 'psql -f ${HOME}/zync-database-backup'oc rsh $(oc get pods -l 'deploymentConfig=zync-database' -o json | jq -r '.items[0].metadata.name') bash -c 'psql -f ${HOME}/zync-database-backup'Copy to Clipboard Copied! Toggle word wrap Toggle overflow
After restoring backend-redis a sync of the Config information from System should be forced to ensure the information in Backend is consistent with that in System, which is the source of truth.
These steps are intended for running instances of backend-redis.
Edit the
redis-configconfigmap:oc edit configmap redis-config
oc edit configmap redis-configCopy to Clipboard Copied! Toggle word wrap Toggle overflow Comment
SAVEcommands in theredis-configconfigmap:#save 900 1 #save 300 10 #save 60 10000
#save 900 1 #save 300 10 #save 60 10000Copy to Clipboard Copied! Toggle word wrap Toggle overflow Set
appendonlyto no in theredis-configconfigmap:appendonly no
appendonly noCopy to Clipboard Copied! Toggle word wrap Toggle overflow Redeploy
backend-redisto load the new configurations:oc rollout latest dc/backend-redis
oc rollout latest dc/backend-redisCopy to Clipboard Copied! Toggle word wrap Toggle overflow Rename the
dump.rbfile:oc rsh $(oc get pods -l 'deploymentConfig=backend-redis' -o json | jq '.items[0].metadata.name' -r) bash -c 'mv ${HOME}/data/dump.rdb ${HOME}/data/dump.rdb-old'oc rsh $(oc get pods -l 'deploymentConfig=backend-redis' -o json | jq '.items[0].metadata.name' -r) bash -c 'mv ${HOME}/data/dump.rdb ${HOME}/data/dump.rdb-old'Copy to Clipboard Copied! Toggle word wrap Toggle overflow Rename the
appendonly.aoffile:oc rsh $(oc get pods -l 'deploymentConfig=backend-redis' -o json | jq '.items[0].metadata.name' -r) bash -c 'mv ${HOME}/data/appendonly.aof ${HOME}/data/appendonly.aof-old'oc rsh $(oc get pods -l 'deploymentConfig=backend-redis' -o json | jq '.items[0].metadata.name' -r) bash -c 'mv ${HOME}/data/appendonly.aof ${HOME}/data/appendonly.aof-old'Copy to Clipboard Copied! Toggle word wrap Toggle overflow Move the Backup file to the POD:
oc cp ./backend-redis-dump.rdb $(oc get pods -l 'deploymentConfig=backend-redis' -o json | jq '.items[0].metadata.name' -r):/var/lib/redis/data/dump.rdb
oc cp ./backend-redis-dump.rdb $(oc get pods -l 'deploymentConfig=backend-redis' -o json | jq '.items[0].metadata.name' -r):/var/lib/redis/data/dump.rdbCopy to Clipboard Copied! Toggle word wrap Toggle overflow Redeploy
backend-redisto load the backup:oc rollout latest dc/backend-redis
oc rollout latest dc/backend-redisCopy to Clipboard Copied! Toggle word wrap Toggle overflow Create the
appendonlyfile:oc rsh $(oc get pods -l 'deploymentConfig=backend-redis' -o json | jq '.items[0].metadata.name' -r) bash -c 'redis-cli BGREWRITEAOF'
oc rsh $(oc get pods -l 'deploymentConfig=backend-redis' -o json | jq '.items[0].metadata.name' -r) bash -c 'redis-cli BGREWRITEAOF'Copy to Clipboard Copied! Toggle word wrap Toggle overflow After a while, ensure that the AOF rewrite is complete:
oc rsh $(oc get pods -l 'deploymentConfig=backend-redis' -o json | jq '.items[0].metadata.name' -r) bash -c 'redis-cli info' | grep aof_rewrite_in_progress
oc rsh $(oc get pods -l 'deploymentConfig=backend-redis' -o json | jq '.items[0].metadata.name' -r) bash -c 'redis-cli info' | grep aof_rewrite_in_progressCopy to Clipboard Copied! Toggle word wrap Toggle overflow -
While
aof_rewrite_in_progress = 1, the execution is in progress. -
Check periodically until
aof_rewrite_in_progress = 0. Zero indicates that the execution is complete.
-
While
Edit the
redis-configconfigmap:oc edit configmap redis-config
oc edit configmap redis-configCopy to Clipboard Copied! Toggle word wrap Toggle overflow Uncomment
SAVEcommands in theredis-configconfigmap:save 900 1 save 300 10 save 60 10000
save 900 1 save 300 10 save 60 10000Copy to Clipboard Copied! Toggle word wrap Toggle overflow Set
appendonlyto yes in theredis-configconfigmap:appendonly yes
appendonly yesCopy to Clipboard Copied! Toggle word wrap Toggle overflow Redeploy
backend-redisto reload the default configurations:oc rollout latest dc/backend-redis
oc rollout latest dc/backend-redisCopy to Clipboard Copied! Toggle word wrap Toggle overflow
These steps are intended for running instances of system-redis.
Edit the
redis-configconfigmap:oc edit configmap redis-config
oc edit configmap redis-configCopy to Clipboard Copied! Toggle word wrap Toggle overflow Comment
SAVEcommands in theredis-configconfigmap:#save 900 1 #save 300 10 #save 60 10000
#save 900 1 #save 300 10 #save 60 10000Copy to Clipboard Copied! Toggle word wrap Toggle overflow Set
appendonlyto no in theredis-configconfigmap:appendonly no
appendonly noCopy to Clipboard Copied! Toggle word wrap Toggle overflow Redeploy
system-redisto load the new configurations:oc rollout latest dc/system-redis
oc rollout latest dc/system-redisCopy to Clipboard Copied! Toggle word wrap Toggle overflow Rename the
dump.rbfile:oc rsh $(oc get pods -l 'deploymentConfig=system-redis' -o json | jq '.items[0].metadata.name' -r) bash -c 'mv ${HOME}/data/dump.rdb ${HOME}/data/dump.rdb-old'oc rsh $(oc get pods -l 'deploymentConfig=system-redis' -o json | jq '.items[0].metadata.name' -r) bash -c 'mv ${HOME}/data/dump.rdb ${HOME}/data/dump.rdb-old'Copy to Clipboard Copied! Toggle word wrap Toggle overflow Rename the
appendonly.aoffile:oc rsh $(oc get pods -l 'deploymentConfig=system-redis' -o json | jq '.items[0].metadata.name' -r) bash -c 'mv ${HOME}/data/appendonly.aof ${HOME}/data/appendonly.aof-old'oc rsh $(oc get pods -l 'deploymentConfig=system-redis' -o json | jq '.items[0].metadata.name' -r) bash -c 'mv ${HOME}/data/appendonly.aof ${HOME}/data/appendonly.aof-old'Copy to Clipboard Copied! Toggle word wrap Toggle overflow Move the
Backupfile to the POD:oc cp ./system-redis-dump.rdb $(oc get pods -l 'deploymentConfig=system-redis' -o json | jq '.items[0].metadata.name' -r):/var/lib/redis/data/dump.rdb
oc cp ./system-redis-dump.rdb $(oc get pods -l 'deploymentConfig=system-redis' -o json | jq '.items[0].metadata.name' -r):/var/lib/redis/data/dump.rdbCopy to Clipboard Copied! Toggle word wrap Toggle overflow Redeploy
system-redisto load the backup:oc rollout latest dc/system-redis
oc rollout latest dc/system-redisCopy to Clipboard Copied! Toggle word wrap Toggle overflow Create the
appendonlyfile:oc rsh $(oc get pods -l 'deploymentConfig=system-redis' -o json | jq '.items[0].metadata.name' -r) bash -c 'redis-cli BGREWRITEAOF'
oc rsh $(oc get pods -l 'deploymentConfig=system-redis' -o json | jq '.items[0].metadata.name' -r) bash -c 'redis-cli BGREWRITEAOF'Copy to Clipboard Copied! Toggle word wrap Toggle overflow After a while, ensure that the AOF rewrite is complete:
oc rsh $(oc get pods -l 'deploymentConfig=system-redis' -o json | jq '.items[0].metadata.name' -r) bash -c 'redis-cli info' | grep aof_rewrite_in_progress
oc rsh $(oc get pods -l 'deploymentConfig=system-redis' -o json | jq '.items[0].metadata.name' -r) bash -c 'redis-cli info' | grep aof_rewrite_in_progressCopy to Clipboard Copied! Toggle word wrap Toggle overflow -
While
aof_rewrite_in_progress = 1, the execution is in progress. -
Check periodically until
aof_rewrite_in_progress = 0. Zero indicates that the execution is complete.
-
While
Edit the
redis-configconfigmap:oc edit configmap redis-config
oc edit configmap redis-configCopy to Clipboard Copied! Toggle word wrap Toggle overflow Uncomment
SAVEcommands in theredis-configconfigmap:save 900 1 save 300 10 save 60 10000
save 900 1 save 300 10 save 60 10000Copy to Clipboard Copied! Toggle word wrap Toggle overflow Set
appendonlyto yes in theredis-configconfigmap:appendonly yes
appendonly yesCopy to Clipboard Copied! Toggle word wrap Toggle overflow Redeploy
system-redisto reload the default configurations:oc rollout latest dc/system-redis
oc rollout latest dc/system-redisCopy to Clipboard Copied! Toggle word wrap Toggle overflow
'%20d='M201.5%20305.5c-13.8%200-24.9-11.1-24.9-24.6%200-13.8%2011.1-24.9%2024.9-24.9%2013.6%200%2024.6%2011.1%2024.6%2024.9%200%2013.6-11.1%2024.6-24.6%2024.6zM504%20256c0%20137-111%20248-248%20248S8%20393%208%20256%20119%208%20256%208s248%20111%20248%20248zm-132.3-41.2c-9.4%200-17.7%203.9-23.8%2010-22.4-15.5-52.6-25.5-86.1-26.6l17.4-78.3%2055.4%2012.5c0%2013.6%2011.1%2024.6%2024.6%2024.6%2013.8%200%2024.9-11.3%2024.9-24.9s-11.1-24.9-24.9-24.9c-9.7%200-18%205.8-22.1%2013.8l-61.2-13.6c-3-.8-6.1%201.4-6.9%204.4l-19.1%2086.4c-33.2%201.4-63.1%2011.3-85.5%2026.8-6.1-6.4-14.7-10.2-24.1-10.2-34.9%200-46.3%2046.9-14.4%2062.8-1.1%205-1.7%2010.2-1.7%2015.5%200%2052.6%2059.2%2095.2%20132%2095.2%2073.1%200%20132.3-42.6%20132.3-95.2%200-5.3-.6-10.8-1.9-15.8%2031.3-16%2019.8-62.5-14.9-62.5zM302.8%20331c-18.2%2018.2-76.1%2017.9-93.6%200-2.2-2.2-6.1-2.2-8.3%200-2.5%202.5-2.5%206.4%200%208.6%2022.8%2022.8%2087.3%2022.8%20110.2%200%202.5-2.2%202.5-6.1%200-8.6-2.2-2.2-6.1-2.2-8.3%200zm7.7-75c-13.6%200-24.6%2011.1-24.6%2024.9%200%2013.6%2011.1%2024.6%2024.6%2024.6%2013.8%200%2024.9-11.1%2024.9-24.6%200-13.8-11-24.9-24.9-24.9z'/%3e%3c/svg%3e){kind=small}