Chapter 3. Database Images
3.1. Overview
This topic group includes information on the different database images available for OpenShift Online users.
3.2. MySQL
3.2.1. Overview
OpenShift Online provides a container image for running MySQL. This image can provide database services based on username, password, and database name settings provided via configuration.
3.2.2. Versions
Currently, OpenShift Online provides versions 5.6 and 5.7 of MySQL.
3.2.3. Images
RHEL 7 images are available through the Red Hat Registry:
$ docker pull registry.access.redhat.com/rhscl/mysql-56-rhel7 $ docker pull registry.access.redhat.com/rhscl/mysql-57-rhel7
You can use these images through the mysql
image stream.
3.2.4. Configuration and Usage
3.2.4.1. Initializing the Database
The first time you use the shared volume, the database is created along with the database administrator user and the MySQL root user (if you specify the MYSQL_ROOT_PASSWORD
environment variable). Afterwards, the MySQL daemon starts up. If you are re-attaching the volume to another container, then the database, database user, and the administrator user are not created, and the MySQL daemon starts.
The following command creates a new database pod with MySQL running in a container:
$ oc new-app \ -e MYSQL_USER=<username> \ -e MYSQL_PASSWORD=<password> \ -e MYSQL_DATABASE=<database_name> \ mysql:5.6
3.2.4.2. Running MySQL Commands in Containers
OpenShift Online uses Software Collections (SCLs) to install and launch MySQL. If you want to execute a MySQL command inside of a running container (for debugging), you must invoke it using bash.
To do so, first identify the name of the pod. For example, you can view the list of pods in your current project:
$ oc get pods
Then, open a remote shell session to the pod:
$ oc rsh <pod>
When you enter the container, the required SCL is automatically enabled.
You can now run the mysql command from the bash shell to start a MySQL interactive session and perform normal MySQL operations. For example, to authenticate as the database user:
bash-4.2$ mysql -u $MYSQL_USER -p$MYSQL_PASSWORD -h $HOSTNAME $MYSQL_DATABASE Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 4 Server version: 5.6.37 MySQL Community Server (GPL) ... mysql>
When you are finished, enter quit or exit to leave the MySQL session.
3.2.4.3. Environment Variables
The MySQL user name, password, and database name must be configured with the following environment variables:
Variable Name | Description |
---|---|
| Specifies the user name for the database user that is created for use by your application. |
|
Password for the |
|
Name of the database to which |
| Optional password for the root user. If this is not set, then remote login to the root account is not possible. Local connections from within the container are always permitted without a password. |
| Service host variable automatically created by Kubernetes. |
| Service port variable automatically created by Kubernetes. |
You must specify the user name, password, and database name. If you do not specify all three, the pod will fail to start and OpenShift Online will continuously try to restart it.
MySQL settings can be configured with the following environment variables:
Variable Name | Description | Default |
---|---|---|
| Sets how the table names are stored and compared. | 0 |
| The maximum permitted number of simultaneous client connections. | 151 |
| The maximum size of one packet or any generated/intermediate string. | 200M |
| The minimum length of the word to be included in a FULLTEXT index. | 4 |
| The maximum length of the word to be included in a FULLTEXT index. | 20 |
| Controls the innodb_use_native_aio setting value if the native AIO is broken. | 1 |
| The number of open tables for all threads. | 400 |
| The size of the buffer used for index blocks. | 10% of available memory |
| The size of the buffer used for sorting. | 256K |
| The size of the buffer used for a sequential scan. | 5% of available memory |
| The size of the buffer pool where InnoDB caches table and index data. | 50% of available memory |
| The size of each log file in a log group. | 15% of available memory |
| The size of the buffer that InnoDB uses to write to the log files on disk. | 15% of available memory |
Some of the memory-related parameters have percentages as default values. These values are calculated dynamically during a container’s startup based on memory limits.
3.2.4.4. Volume Mount Points
The MySQL image can be run with mounted volumes to enable persistent storage for the database:
- /var/lib/mysql/data - This is the data directory where MySQL stores database files.
3.2.4.5. Changing Passwords
Passwords are part of the image configuration, therefore the only supported method to change passwords for the database user (MYSQL_USER
) and root user is by changing the environment variables MYSQL_PASSWORD
and MYSQL_ROOT_PASSWORD
, respectively.
You can view the current passwords by viewing the pod or deployment configuration in the web console or by listing the environment variables with the CLI:
$ oc set env pod <pod_name> --list
Whenever MYSQL_ROOT_PASSWORD
is set, it enables remote access for the root user with the given password, and whenever it is unset, remote access for the root user is disabled. This does not affect the regular user MYSQL_USER
, who always has remote access. This also does not affect local access by the root user, who can always log in without a password in localhost.
Changing database passwords through SQL statements or any way other than through the environment variables aforementioned causes a mismatch between the values stored in the variables and the actual passwords. Whenever a database container starts, it resets the passwords to the values stored in the environment variables.
To change these passwords, update one or both of the desired environment variables for the related deployment configuration(s) using the oc set env
command. If multiple deployment configurations utilize these environment variables, for example in the case of an application created from a template, you must update the variables on each deployment configuration so that the passwords are in sync everywhere. This can be done all in the same command:
$ oc set env dc <dc_name> [<dc_name_2> ...] \ MYSQL_PASSWORD=<new_password> \ MYSQL_ROOT_PASSWORD=<new_root_password>
Depending on your application, there may be other environment variables for passwords in other parts of the application that should also be updated to match. For example, there could be a more generic DATABASE_USER
variable in a front-end pod that should match the database user’s password. Ensure that passwords are in sync for all required environment variables per your application, otherwise your pods may fail to redeploy when triggered.
Updating the environment variables triggers the redeployment of the database server if you have a configuration change trigger. Otherwise, you must manually start a new deployment in order to apply the password changes.
To verify that new passwords are in effect, first open a remote shell session to the running MySQL pod:
$ oc rsh <pod>
From the bash shell, verify the database user’s new password:
bash-4.2$ mysql -u $MYSQL_USER -p<new_password> -h $HOSTNAME $MYSQL_DATABASE -te "SELECT * FROM (SELECT database()) db CROSS JOIN (SELECT user()) u"
If the password was changed correctly, you should see a table like this:
+------------+---------------------+ | database() | user() | +------------+---------------------+ | sampledb | user0PG@172.17.42.1 | +------------+---------------------+
To verify the root user’s new password:
bash-4.2$ mysql -u root -p<new_root_password> -h $HOSTNAME $MYSQL_DATABASE -te "SELECT * FROM (SELECT database()) db CROSS JOIN (SELECT user()) u"
If the password was changed correctly, you should see a table like this:
+------------+------------------+ | database() | user() | +------------+------------------+ | sampledb | root@172.17.42.1 | +------------+------------------+
3.2.5. Creating a Database Service from a Template
OpenShift Online provides a template to make creating a new database service easy. The template provides parameter fields to define all the mandatory environment variables (user, password, database name, etc) with predefined defaults including auto-generation of password values. It will also define both a deployment configuration and a service.
The MySQL template should have been registered in the default openshift project by your cluster administrator during the initial cluster setup.
The following template is available:
-
mysql-persistent
uses a persistent volume store for the database data which means the data will survive a pod restart.
You can instantiate templates by following these instructions.
Once you have instantiated the service, you can copy the user name, password, and database name environment variables into a deployment configuration for another component that intends to access the database. That component can then access the database via the service that was defined.
3.2.5.1. Creating the Deployment Configuration for the MySQL Master
To set up MySQL replication, a deployment configuration is defined in the example template that defines a replication controller. For MySQL master-slave replication, two deployment configurations are needed. One deployment configuration defines the MySQL master server and second the MySQL slave servers.
To tell a MySQL server to act as the master, the command
field in the container’s definition in the deployment configuration must be set to run-mysqld-master. This script acts as an alternative entrypoint for the MySQL image and configures the MySQL server to run as the master in replication.
MySQL replication requires a special user that relays data between the master and slaves. The following environment variables are defined in the template for this purpose:
Variable Name | Description | Default |
---|---|---|
| The user name of the replication user | master |
| The password for the replication user | generated |
Example 3.1. MySQL Master Deployment Configuration Object Definition in the Example Template
kind: "DeploymentConfig" apiVersion: "v1" metadata: name: "mysql-master" spec: strategy: type: "Recreate" triggers: - type: "ConfigChange" replicas: 1 selector: name: "mysql-master" template: metadata: labels: name: "mysql-master" spec: volumes: - name: "mysql-master-data" persistentVolumeClaim: claimName: "mysql-master" containers: - name: "server" image: "openshift/mysql-56-centos7" command: - "run-mysqld-master" ports: - containerPort: 3306 protocol: "TCP" env: - name: "MYSQL_MASTER_USER" value: "${MYSQL_MASTER_USER}" - name: "MYSQL_MASTER_PASSWORD" value: "${MYSQL_MASTER_PASSWORD}" - name: "MYSQL_USER" value: "${MYSQL_USER}" - name: "MYSQL_PASSWORD" value: "${MYSQL_PASSWORD}" - name: "MYSQL_DATABASE" value: "${MYSQL_DATABASE}" - name: "MYSQL_ROOT_PASSWORD" value: "${MYSQL_ROOT_PASSWORD}" volumeMounts: - name: "mysql-master-data" mountPath: "/var/lib/mysql/data" resources: {} terminationMessagePath: "/dev/termination-log" imagePullPolicy: "IfNotPresent" securityContext: capabilities: {} privileged: false restartPolicy: "Always" dnsPolicy: "ClusterFirst"
After the deployment configuration is created and the pod with MySQL master server is started, it will create the database defined by MYSQL_DATABASE
and configure the server to replicate this database to slaves.
The example provided defines only one replica of the MySQL master server. This causes OpenShift Online to start only one instance of the server. Multiple instances (multi-master) is not supported and therefore you can not scale this replication controller.
To replicate the database created by the MySQL master, a deployment configuration is defined in the template. This deployment configuration creates a replication controller that launches the MySQL image with the command
field set to run-mysqld-slave. This alternative entrypoints skips the initialization of the database and configures the MySQL server to connect to the mysql-master service, which is also defined in example template.
Example 3.2. MySQL Slave Deployment Configuration Object Definition in the Example Template
kind: "DeploymentConfig" apiVersion: "v1" metadata: name: "mysql-slave" spec: strategy: type: "Recreate" triggers: - type: "ConfigChange" replicas: 1 selector: name: "mysql-slave" template: metadata: labels: name: "mysql-slave" spec: containers: - name: "server" image: "openshift/mysql-56-centos7" command: - "run-mysqld-slave" ports: - containerPort: 3306 protocol: "TCP" env: - name: "MYSQL_MASTER_USER" value: "${MYSQL_MASTER_USER}" - name: "MYSQL_MASTER_PASSWORD" value: "${MYSQL_MASTER_PASSWORD}" - name: "MYSQL_DATABASE" value: "${MYSQL_DATABASE}" resources: {} terminationMessagePath: "/dev/termination-log" imagePullPolicy: "IfNotPresent" securityContext: capabilities: {} privileged: false restartPolicy: "Always" dnsPolicy: "ClusterFirst"
This example deployment configuration starts the replication controller with the initial number of replicas set to 1. You can scale this replication controller in both directions, up to the resources capacity of your account.
3.2.5.2. Creating a Headless Service
The pods created by the MySQL slave replication controller must reach the MySQL master server in order to register for replication. The example template defines a headless service named mysql-master for this purpose. This service is not used only for replication, but the clients can also send the queries to mysql-master:3306 as the MySQL host.
To have a headless service, the clusterIP
parameter in the service definition is set to None. Then you can use a DNS query to get a list of the pod IP addresses that represents the current endpoints for this service.
Example 3.3. Headless Service Object Definition in the Example Template
kind: "Service" apiVersion: "v1" metadata: name: "mysql-master" labels: name: "mysql-master" spec: ports: - protocol: "TCP" port: 3306 targetPort: 3306 nodePort: 0 selector: name: "mysql-master" clusterIP: "None" type: "ClusterIP" sessionAffinity: "None" status: loadBalancer: {}
3.2.5.3. Scaling the MySQL Slaves
To increase the number of members in the cluster:
$ oc scale rc mysql-slave-1 --replicas=<number>
This tells the replication controller to create a new MySQL slave pod. When a new slave is created, the slave entrypoint first attempts to contact the mysql-master service and register itself to the replication set. Once that is done, the MySQL master server sends the slave the replicated database.
When scaling down, the MySQL slave is shut down and, because the slave does not have any persistent storage defined, all data on the slave is lost. The MySQL master server then discovers that the slave is not reachable anymore, and it automatically removes it from the replication.
3.2.6. Troubleshooting
This section describes some troubles you might encounter and presents possible resolutions.
3.2.6.1. Linux Native AIO Failure
Symptom
The MySQL container fails to start and the logs show something like:
151113 5:06:56 InnoDB: Using Linux native AIO 151113 5:06:56 InnoDB: Warning: io_setup() failed with EAGAIN. Will make 5 attempts before giving up. InnoDB: Warning: io_setup() attempt 1 failed. InnoDB: Warning: io_setup() attempt 2 failed. Waiting for MySQL to start ... InnoDB: Warning: io_setup() attempt 3 failed. InnoDB: Warning: io_setup() attempt 4 failed. Waiting for MySQL to start ... InnoDB: Warning: io_setup() attempt 5 failed. 151113 5:06:59 InnoDB: Error: io_setup() failed with EAGAIN after 5 attempts. InnoDB: You can disable Linux Native AIO by setting innodb_use_native_aio = 0 in my.cnf 151113 5:06:59 InnoDB: Fatal error: cannot initialize AIO sub-system 151113 5:06:59 [ERROR] Plugin 'InnoDB' init function returned error. 151113 5:06:59 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed. 151113 5:06:59 [ERROR] Unknown/unsupported storage engine: InnoDB 151113 5:06:59 [ERROR] Aborting
Explanation
MySQL’s storage engine was unable to use the kernel’s AIO (Asynchronous I/O) facilities due to resource limits.
Resolution
Turn off AIO usage entirely by setting environment variable MYSQL_AIO
to have value 0
. On subsequent deployments, this arranges for the MySQL configuration variable innodb_use_native_aio
to have value 0
.
3.3. PostgreSQL
3.3.1. Overview
OpenShift Online provides a container image for running PostgreSQL. This image can provide database services based on username, password, and database name settings provided via configuration.
3.3.2. Versions
Currently, OpenShift Online supports versions 9.4 and 9.5 of PostgreSQL.
3.3.3. Images
RHEL 7 images are available through the Red Hat Registry:
$ docker pull registry.access.redhat.com/rhscl/postgresql-94-rhel7 $ docker pull registry.access.redhat.com/rhscl/postgresql-95-rhel7
You can use these images through the postgresql
image stream.
3.3.4. Configuration and Usage
3.3.4.1. Initializing the Database
The first time you use the shared volume, the database is created along with the database administrator user and the PostgreSQL postgres user (if you specify the POSTGRESQL_ADMIN_PASSWORD
environment variable). Afterwards, the PostgreSQL daemon starts up. If you are re-attaching the volume to another container, then the database, the database user, and the administrator user are not created, and the PostgreSQL daemon starts.
The following command creates a new database pod with PostgreSQL running in a container:
$ oc new-app \ -e POSTGRESQL_USER=<username> \ -e POSTGRESQL_PASSWORD=<password> \ -e POSTGRESQL_DATABASE=<database_name> \ postgresql:9.5
3.3.4.2. Running PostgreSQL Commands in Containers
OpenShift Online uses Software Collections (SCLs) to install and launch PostgreSQL. If you want to execute a PostgreSQL command inside of a running container (for debugging), you must invoke it using bash.
To do so, first identify the name of the running PostgreSQL pod. For example, you can view the list of pods in your current project:
$ oc get pods
Then, open a remote shell session to the desired pod:
$ oc rsh <pod>
When you enter the container, the required SCL is automatically enabled.
You can now run the psql command from the bash shell to start a PostgreSQL interactive session and perform normal PostgreSQL operations. For example, to authenticate as the database user:
bash-4.2$ PGPASSWORD=$POSTGRESQL_PASSWORD psql -h postgresql $POSTGRESQL_DATABASE $POSTGRESQL_USER psql (9.5.16) Type "help" for help. default=>
When you are finished, enter \q to leave the PostgreSQL session.
3.3.4.3. Environment Variables
The PostgreSQL user name, password, and database name must be configured with the following environment variables:
Variable Name | Description |
---|---|
| User name for the PostgreSQL account to be created. This user has full rights to the database. |
| Password for the user account. |
| Database name. |
| Optional password for the postgres administrator user. If this is not set, then remote login to the postgres account is not possible. Local connections from within the container are always permitted without a password. |
You must specify the user name, password, and database name. If you do not specify all three, the pod will fail to start and OpenShift Online will continuously try to restart it.
PostgreSQL settings can be configured with the following environment variables:
Variable Name | Description | Default |
---|---|---|
| Maximum number of client connections allowed. | 100 |
|
Maximum number of transactions that can be in the "prepared" state. If using prepared transactions, the value should be at least as large as | 0 |
| Amount of memory dedicated to PostgreSQL for caching data. | 32M |
| Estimated amount of memory available for disk caching by the operating system and within PostgreSQL itself. | 128M |
3.3.4.4. Volume Mount Points
The PostgreSQL image can be run with mounted volumes to enable persistent storage for the database:
- /var/lib/pgsql/data - This is the database cluster directory where PostgreSQL stores database files.
3.3.4.5. Changing Passwords
Passwords are part of the image configuration, therefore the only supported method to change passwords for the database user (POSTGRESQL_USER
) and postgres administrator user is by changing the environment variables POSTGRESQL_PASSWORD
and POSTGRESQL_ADMIN_PASSWORD
, respectively.
You can view the current passwords by viewing the pod or deployment configuration in the web console or by listing the environment variables with the CLI:
$ oc set env pod <pod_name> --list
Changing database passwords through SQL statements or any way other than through the environment variables aforementioned will cause a mismatch between the values stored in the variables and the actual passwords. Whenever a database container starts, it resets the passwords to the values stored in the environment variables.
To change these passwords, update one or both of the desired environment variables for the related deployment configuration(s) using the oc set env
command. If multiple deployment configurations utilize these environment variables, for example in the case of an application created from a template, you must update the variables on each deployment configuration so that the passwords are in sync everywhere. This can be done all in the same command:
$ oc set env dc <dc_name> [<dc_name_2> ...] \ POSTGRESQL_PASSWORD=<new_password> \ POSTGRESQL_ADMIN_PASSWORD=<new_admin_password>
Depending on your application, there may be other environment variables for passwords in other parts of the application that should also be updated to match. For example, there could be a more generic DATABASE_USER
variable in a front-end pod that should match the database user’s password. Ensure that passwords are in sync for all required environment variables per your application, otherwise your pods may fail to redeploy when triggered.
Updating the environment variables triggers the redeployment of the database server if you have a configuration change trigger. Otherwise, you must manually start a new deployment in order to apply the password changes.
To verify that new passwords are in effect, first open a remote shell session to the running PostgreSQL pod:
$ oc rsh <pod>
From the bash shell, verify the database user’s new password:
bash-4.2$ PGPASSWORD=<new_password> psql -h postgresql $POSTGRESQL_DATABASE $POSTGRESQL_USER -c "SELECT * FROM (SELECT current_database()) cdb CROSS JOIN (SELECT current_user) cu"
If the password was changed correctly, you should see a table like this:
current_database | current_user ------------------+-------------- default | django (1 row)
From the bash shell, verify the postgres administrator user’s new password:
bash-4.2$ PGPASSWORD=<new_admin_password> psql -h postgresql $POSTGRESQL_DATABASE postgres -c "SELECT * FROM (SELECT current_database()) cdb CROSS JOIN (SELECT current_user) cu"
If the password was changed correctly, you should see a table like this:
current_database | current_user ------------------+-------------- default | postgres (1 row)
3.3.5. Creating a Database Service from a Template
OpenShift Online provides a template to make creating a new database service easy. The template provides parameter fields to define all the mandatory environment variables (user, password, database name, etc) with predefined defaults including auto-generation of password values. It will also define both a deployment configuration and a service.
The PostgreSQL template should have been registered in the default openshift project by your cluster administrator during the initial cluster setup.
The following template is available:
-
PostgreSQL-persistent
uses a persistent volume store for the database data which means the data will survive a pod restart.
You can instantiate templates by following these instructions.
Once you have instantiated the service, you can copy the user name, password, and database name environment variables into a deployment configuration for another component that intends to access the database. That component can then access the database via the service that was defined.
3.4. MongoDB
3.4.1. Overview
OpenShift Online provides a container image for running MongoDB. This image can provide database services based on username, password, and database name settings provided via configuration.
3.4.2. Versions
Currently, OpenShift Online provides versions 2.6, 3.2, and 3.4 of MongoDB.
3.4.3. Images
RHEL 7 images are available through the Red Hat Registry:
$ docker pull registry.access.redhat.com/rhscl/mongodb-26-rhel7 $ docker pull registry.access.redhat.com/rhscl/mongodb-32-rhel7 $ docker pull registry.access.redhat.com/rhscl/mongodb-34-rhel7
You can use these images through the mongodb
image stream.
3.4.4. Configuration and usage
3.4.4.1. Initializing the database
You can configure MongoDB with an ephemeral volume or a persistent volume. The first time you use the volume, the database is created along with the database administrator user. Afterwards, the MongoDB daemon starts up. If you are re-attaching the volume to another container, then the database, database user, and the administrator user are not created, and the MongoDB daemon starts.
The following command creates a new database pod with MongoDB running in a container with an ephemeral volume:
$ oc new-app \ -e MONGODB_USER=<username> \ -e MONGODB_PASSWORD=<password> \ -e MONGODB_DATABASE=<database_name> \ -e MONGODB_ADMIN_PASSWORD=<admin_password> \ mongodb:2.6
3.4.4.2. Running MongoDB commands in containers
OpenShift Online uses Software Collections (SCLs) to install and launch MongoDB. If you want to execute a MongoDB command inside of a running container (for debugging), you must invoke it using bash.
To do so, first identify the name of the running MongoDB pod. For example, you can view the list of pods in your current project:
$ oc get pods
Then, open a remote shell session to the desired pod:
$ oc rsh <pod>
When you enter the container, the required SCL is automatically enabled.
You can now run mongo commands from the bash shell to start a MongoDB interactive session and perform normal MongoDB operations. For example, to switch to the sampledb database and authenticate as the database user:
bash-4.2$ mongo -u $MONGODB_USER -p $MONGODB_PASSWORD $MONGODB_DATABASE MongoDB shell version: 2.6.9 connecting to: sampledb >
When you are finished, press CTRL+D to leave the MongoDB session.
3.4.4.3. Environment Variables
The MongoDB user name, password, database name, and admin password must be configured with the following environment variables:
Variable Name | Description |
---|---|
| User name for MongoDB account to be created. |
| Password for the user account. |
| Database name. |
| Password for the admin user. |
You must specify the user name, password, database name, and admin password. If you do not specify all four, the pod will fail to start and OpenShift Online will continuously try to restart it.
The administrator user name is set to admin and you must specify its password by setting the MONGODB_ADMIN_PASSWORD
environment variable. This process is done upon database initialization.
MongoDB settings can be configured with the following environment variables:
Variable Name | Description | Default |
---|---|---|
| Disable data file preallocation. |
|
| Set MongoDB to use a smaller default data file size. |
|
| Runs MongoDB in a quiet mode that attempts to limit the amount of output. |
|
Text search is enabled by default in MongoDB versions 2.6 and higher, and therefore has no configurable parameter.
3.4.4.4. Volume mount points
The MongoDB image can be run with mounted volumes to enable persistent storage for the database:
- /var/lib/mongodb/data - This is the database directory where MongoDB stores database files.
3.4.4.5. Changing passwords
Passwords are part of the image configuration, therefore the only supported method to change passwords for the database user (MONGODB_USER
) and admin user is by changing the environment variables MONGODB_PASSWORD
and MONGODB_ADMIN_PASSWORD
, respectively.
You can view the current passwords by viewing the pod or deployment configuration in the web console or by listing the environment variables with the CLI:
$ oc set env pod <pod_name> --list
Changing database passwords directly in MongoDB causes a mismatch between the values stored in the variables and the actual passwords. Whenever a database container starts, it resets the passwords to the values stored in the environment variables.
To change these passwords, update one or both of the desired environment variables for the related deployment configuration(s) using the oc set env
command. If multiple deployment configurations utilize these environment variables, for example in the case of an application created from a template, you must update the variables on each deployment configuration so that the passwords are in sync everywhere. This can be done all in the same command:
$ oc set env dc <dc_name> [<dc_name_2> ...] \ MONGODB_PASSWORD=<new_password> \ MONGODB_ADMIN_PASSWORD=<new_admin_password>
Depending on your application, there may be other environment variables for passwords in other parts of the application that should also be updated to match. For example, there could be a more generic DATABASE_USER
variable in a front-end pod that should match the database user’s password. Ensure that passwords are in sync for all required environment variables per your application, otherwise your pods may fail to redeploy when triggered.
Updating the environment variables triggers the redeployment of the database server if you have a configuration change trigger. Otherwise, you must manually start a new deployment in order to apply the password changes.
To verify that new passwords are in effect, first open a remote shell session to the running MongoDB pod:
$ oc rsh <pod>
From the bash shell, verify the database user’s new password:
bash-4.2$ mongo -u $MONGODB_USER -p <new_password> $MONGODB_DATABASE --eval "db.version()"
If the password was changed correctly, you should see output like this:
MongoDB shell version: 2.6.9 connecting to: sampledb 2.6.9
To verify the admin user’s new password:
bash-4.2$ mongo -u admin -p <new_admin_password> admin --eval "db.version()"
If the password was changed correctly, you should see output like this:
MongoDB shell version: 2.6.9 connecting to: admin 2.6.9
3.4.5. Creating a database service from a template
OpenShift Online provides a template to make creating a new database service easy. The template provides parameter fields to define all the mandatory environment variables (user, password, database name, etc) with predefined defaults including auto-generation of password values. It will also define both a deployment configuration and a service.
The MongoDB templates should have been registered in the default openshift project by your cluster administrator during the initial cluster setup.
The following template is available:
-
mongodb-persistent
uses a persistent volume store for the database data which means the data will survive a pod restart.
You can instantiate templates by following these instructions.
Once you have instantiated the service, you can copy the user name, password, and database name environment variables into a deployment configuration for another component that intends to access the database. That component can then access the database via the service that was defined.
3.4.6. MongoDB replication
Red Hat provides a proof-of-concept template for MongoDB replication (clustering) using StatefulSet. You can obtain the example template from GitHub.
For example, to upload the example template into the current project’s template library:
$ oc create -f \ https://raw.githubusercontent.com/sclorg/mongodb-container/master/examples/petset/mongodb-petset-persistent.yaml
The example template uses persistent storage. You must have persistent volumes available in your cluster to use this template.
As OpenShift Online automatically restarts unhealthy pods (containers), it will restart replica set members if one or more of these members crashes or fails.
While a replica set member is down or being restarted, it may be one of these scenarios:
PRIMARY member is down:
In this case, the other two members elect a new PRIMARY. Until then, reads are not affected, but the writes fail. After a successful election, writes and reads process normally.
One of the SECONDARY member is down:
Reads and writes are unaffected. Depending on the
oplogSize
configuration and the write rate, the third member might fail to join back the replica set, requiring manual intervention to re-sync its copy of the database.Any two members are down:
When a three-member replica set member cannot reach any other member, it will step down from the PRIMARY role if it had it. In this case, reads might be served by a SECONDARY member, and writes fail. As soon as one more member is back up, an election picks a new PRIMARY member and reads and writes process normally.
All members are down:
In this extreme case, both reads and writes fail. After two or more members are back up, an election reestablishes the replica set to have a PRIMARY and a SECONDARY member, after which reads and writes process normally.
This is the recommended replication strategy for MongoDB.
For production environments, you must maintain as much separation between members as possible. It is recommended to use one or more of the node selection features to schedule StatefulSet pods onto different nodes, and to provide them storage backed by independent volumes.
3.4.6.1. Limitations
- Only MongoDB 3.2 is supported.
- You have to manually update replica set configuration in case of scaling down.
Changing a user and administrator password is a manual process. It requires:
- updating values of environment variables in the StatefulSet configuration,
- changing password in the database, and
- restarting all pods one after another.
3.4.6.2. Using the example template
Assuming you already have three pre-created persistent volumes or configured persistent volume provisioning.
Create a new poject where you want to create a MongoDB cluster:
$ oc new-project mongodb-cluster-example
Create a new application using the example template:
$ oc new-app https://raw.githubusercontent.com/sclorg/mongodb-container/master/examples/petset/mongodb-petset-persistent.yaml
This command created a a MongoDB cluster with three replica set members.
Check the status of the new MongoDB pods:
$ oc get pods NAME READY STATUS RESTARTS AGE mongodb-0 1/1 Running 0 50s mongodb-1 1/1 Running 0 50s mongodb-2 1/1 Running 0 49s
After creating a cluster from the example template, you have a replica set with three members. Once the pods are running you can perform various actions on these pods such as:
Checking logs for one of the pods:
$ oc logs mongodb-0
Log in to the pod:
$ oc rsh mongodb-0 sh-4.2$
Log into a MongoDB instance:
sh-4.2$ mongo $MONGODB_DATABASE -u $MONGODB_USER -p$MONGODB_PASSWORD MongoDB shell version: 3.2.6 connecting to: sampledb rs0:PRIMARY>
3.4.6.3. Scale up
MongoDB recommends an odd number of members in a replica set. If there are sufficient available persistent volumes, or a dynamic storage provisioner is present, scaling up is done by using the oc scale
command:
$ oc scale --replicas=5 statefulsets/mongodb $ oc get pods NAME READY STATUS RESTARTS AGE mongodb-0 1/1 Running 0 9m mongodb-1 1/1 Running 0 8m mongodb-2 1/1 Running 0 8m mongodb-3 1/1 Running 0 1m mongodb-4 1/1 Running 0 57s
This creates new pods which connect to the replica set and updates its configuration.
Scaling up an existing database requires manual intervention if the database size is greater than the oplogSize
configuration. For such cases, a manual initial sync of the new members is required. For more information, see Check the Size of the Oplog and the MongoDB Replication documentation.
3.4.6.4. Scale down
To scale down a replica set it is possible to go from five to three members, or from three to only one member.
Although scaling up may be done without manual intervention when the preconditions are met (storage availability, size of existing database and oplogSize
), scaling down always require manual intervention.
To scale down:
Set the new number of replicas by using the
oc scale
command:$ oc scale --replicas=3 statefulsets/mongodb
If the new number of replicas still constitutes a majority of the previous number, the replica set may elect a new PRIMARY in case one of the pods that was deleted had the PRIMARY member role. For example, when scaling down from five members to three members.
Alternatively, scaling down to a lower number temporarily renders the replica set to have only SECONDARY members and be in read-only mode. For example, when scaling down from five members to only one member.
Update the replica set configuration to remove members that no longer exist.
This may be improved in the future, a possible implementation being setting a
PreStop
pod hook that inspects the number of replicas (exposed via the downward API) and determines that the pod is being removed from the StatefulSet, and not being restarted for some other reason.- Purge the volume used by the decommissioned pods.
3.5. MariaDB
3.5.1. Overview
OpenShift Online provides a container image for running MariaDB. This image can provide database services based on username, password, and database name settings provided in a configuration file.
3.5.2. Versions
Currently, OpenShift Online provides versions 10.0 and 10.1 of MariaDB.
3.5.3. Images
RHEL 7 images are available through the Red Hat Registry:
$ docker pull registry.access.redhat.com/rhscl/mariadb-100-rhel7 $ docker pull registry.access.redhat.com/rhscl/mariadb-101-rhel7
You can use the MariaDB 10.1 image through the mariadb
image stream.
3.5.4. Configuration and Usage
3.5.4.1. Initializing the Database
The first time you use the shared volume, the database is created along with the database administrator user and the MariaDB root user (if you specify the MYSQL_ROOT_PASSWORD
environment variable). Afterwards, the MariaDB daemon starts up. If you are re-attaching the volume to another container, then the database, database user, and the administrator user are not created, and the MariaDB daemon starts.
The following command creates a new database pod with MariaDB running in a container:
$ oc new-app \ -e MYSQL_USER=<username> \ -e MYSQL_PASSWORD=<password> \ -e MYSQL_DATABASE=<database_name> \ mariadb:10.1
3.5.4.2. Running MariaDB Commands in Containers
OpenShift Online uses Software Collections (SCLs) to install and launch MariaDB. If you want to execute a MariaDB command inside of a running container (for debugging), you must invoke it using bash.
To do so, first identify the name of the running MariaDB pod. For example, you can view the list of pods in your current project:
$ oc get pods
Then, open a remote shell session to the pod:
$ oc rsh <pod>
When you enter the container, the required SCL is automatically enabled.
You can now run mysql commands from the bash shell to start a MariaDB interactive session and perform normal MariaDB operations. For example, to authenticate as the database user:
bash-4.2$ mysql -u $MYSQL_USER -p$MYSQL_PASSWORD -h $HOSTNAME $MYSQL_DATABASE Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 4 Server version: 5.5.37 MySQL Community Server (GPL) ... mysql>
When you are finished, enter quit or exit to leave the MySQL session.
3.5.4.3. Environment Variables
The MariaDB user name, password, and database name must be configured with the following environment variables:
Variable Name | Description |
---|---|
| User name for MySQL account to be created. |
| Password for the user account. |
| Database name. |
| Password for the root user (optional). |
You must specify the user name, password, and database name. If you do not specify all three, the pod will fail to start and OpenShift Online will continuously try to restart it.
MariaDB settings can be configured with the following environment variables:
Variable Name | Description | Default |
---|---|---|
| Sets how the table names are stored and compared. | 0 |
| The maximum permitted number of simultaneous client connections. | 151 |
| The maximum size of one packet or any generated/intermediate string. | 200M |
| The minimum length of the word to be included in a FULLTEXT index. | 4 |
| The maximum length of the word to be included in a FULLTEXT index. | 20 |
| Controls the innodb_use_native_aio setting value if the native AIO is broken. | 1 |
| The number of open tables for all threads. | 400 |
| The size of the buffer used for index blocks. | 32M (or 10% of available memory) |
| The size of the buffer used for sorting. | 256K |
| The size of the buffer used for a sequential scan. | 8M (or 5% of available memory) |
| The size of the buffer pool where InnoDB caches table and index data. | 32M (or 50% of available memory) |
| The size of each log file in a log group. | 8M (or 15% of available memory) |
| The size of the buffer that InnoDB uses to write to the log files on disk. | 8M (or 15% of available memory) |
| Point to an alternative configuration file. | /etc/my.cnf |
|
Set sets the binlog format, supported values are | statement |
3.5.4.4. Volume Mount Points
The MariaDB image can be run with mounted volumes to enable persistent storage for the database:
- /var/lib/mysql/data - The MySQL data directory is where MariaDB stores database files.
3.5.4.5. Changing Passwords
Passwords are part of the image configuration, therefore the only supported method to change passwords for the database user (MYSQL_USER
) and admin user is by changing the environment variables MYSQL_PASSWORD
and MYSQL_ROOT_PASSWORD
, respectively.
You can view the current passwords by viewing the pod or deployment configuration in the web console or by listing the environment variables with the CLI:
$ oc set env pod <pod_name> --list
Changing database passwords through SQL statements or any way other than through the environment variables aforementioned causes a mismatch between the values stored in the variables and the actual passwords. Whenever a database container starts, it resets the passwords to the values stored in the environment variables.
To change these passwords, update one or both of the desired environment variables for the related deployment configuration(s) using the oc set env
command. If multiple deployment configurations utilize these environment variables, for example in the case of an application created from a template, you must update the variables on each deployment configuration so that the passwords are in sync everywhere. This can be done all in the same command:
$ oc set env dc <dc_name> [<dc_name_2> ...] \ MYSQL_PASSWORD=<new_password> \ MYSQL_ROOT_PASSWORD=<new_root_password>
Depending on your application, there may be other environment variables for passwords in other parts of the application that should also be updated to match. For example, there could be a more generic DATABASE_USER
variable in a front-end pod that should match the database user’s password. Ensure that passwords are in sync for all required environment variables per your application, otherwise your pods may fail to redeploy when triggered.
Updating the environment variables triggers the redeployment of the database server if you have a configuration change trigger. Otherwise, you must manually start a new deployment in order to apply the password changes.
To verify that new passwords are in effect, first open a remote shell session to the running MariaDB pod:
$ oc rsh <pod>
From the bash shell, verify the database user’s new password:
bash-4.2$ mysql -u $MYSQL_USER -p<new_password> -h $HOSTNAME $MYSQL_DATABASE -te "SELECT * FROM (SELECT database()) db CROSS JOIN (SELECT user()) u"
If the password was changed correctly, you should see a table like this:
+------------+---------------------+ | database() | user() | +------------+---------------------+ | sampledb | user0PG@172.17.42.1 | +------------+---------------------+
To verify the root user’s new password:
bash-4.2$ mysql -u root -p<new_root_password> -h $HOSTNAME $MYSQL_DATABASE -te "SELECT * FROM (SELECT database()) db CROSS JOIN (SELECT user()) u"
If the password was changed correctly, you should see a table like this:
+------------+------------------+ | database() | user() | +------------+------------------+ | sampledb | root@172.17.42.1 | +------------+------------------+
3.5.5. Creating a Database Service from a Template
OpenShift Online provides a template to make creating a new database service easy. The template provides parameter fields to define all the mandatory environment variables (user, password, database name, etc) with predefined defaults including auto-generation of password values. It will also define both a deployment configuration and a service.
The MariaDB template should have been registered in the default openshift project by your cluster administrator during the initial cluster setup.
The following template is available:
-
mariadb-persistent
uses a persistent volume store for the database data which means the data will survive a pod restart.
You can instantiate templates by following these instructions.
Once you have instantiated the service, you can copy the user name, password, and database name environment variables into a deployment configuration for another component that intends to access the database. That component can then access the database through the service that was defined.
3.5.6. Troubleshooting
This section describes some troubles you might encounter and presents possible resolutions.
3.5.6.1. Linux Native AIO Failure
Symptom
The MySQL container fails to start and the logs show something like:
151113 5:06:56 InnoDB: Using Linux native AIO 151113 5:06:56 InnoDB: Warning: io_setup() failed with EAGAIN. Will make 5 attempts before giving up. InnoDB: Warning: io_setup() attempt 1 failed. InnoDB: Warning: io_setup() attempt 2 failed. Waiting for MySQL to start ... InnoDB: Warning: io_setup() attempt 3 failed. InnoDB: Warning: io_setup() attempt 4 failed. Waiting for MySQL to start ... InnoDB: Warning: io_setup() attempt 5 failed. 151113 5:06:59 InnoDB: Error: io_setup() failed with EAGAIN after 5 attempts. InnoDB: You can disable Linux Native AIO by setting innodb_use_native_aio = 0 in my.cnf 151113 5:06:59 InnoDB: Fatal error: cannot initialize AIO sub-system 151113 5:06:59 [ERROR] Plugin 'InnoDB' init function returned error. 151113 5:06:59 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed. 151113 5:06:59 [ERROR] Unknown/unsupported storage engine: InnoDB 151113 5:06:59 [ERROR] Aborting
Explanation
MariaDB’s storage engine was unable to use the kernel’s AIO (Asynchronous I/O) facilities due to resource limits.
Resolution
Turn off AIO usage entirely, by setting environment variable MYSQL_AIO
to have value 0
. On subsequent deployments, this arranges for the MySQL configuration variable innodb_use_native_aio
to have value 0
.