Chapter 22. Persistent Storage Examples

Run the ceph auth get-key command on a Ceph MON node to display the key value for the client.admin user:

apiVersion: v1
kind: Secret
metadata:
  name: ceph-secret
  namespace: kube-system
data:
  key: QVFBOFF2SlZheUJQRVJBQWgvS2cwT1laQUhPQno3akZwekxxdGc9PQ== 1
type: kubernetes.io/rbd 2

Save the secret definition to a file, for example ceph-secret.yaml, then create the secret:

$ oc create -f ceph-secret.yaml
secret "ceph-secret" created

Verify that the secret was created:

# oc get secret ceph-secret
NAME          TYPE                DATA      AGE
ceph-secret   kubernetes.io/rbd   1         5d

Run the ceph auth get-key command on a Ceph MON node to display the key value for the client.kube user:

apiVersion: v1
kind: Secret
metadata:
  name: ceph-user-secret
  namespace: default
data:
  key: QVFCbEV4OVpmaGJtQ0JBQW55d2Z0NHZtcS96cE42SW1JVUQvekE9PQ== 1
type: kubernetes.io/rbd 2

Save the secret definition to a file, for example ceph-user-secret.yaml, then create the secret:

$ oc create -f ceph-user-secret.yaml
secret "ceph-user-secret" created

Verify that the secret was created:

# oc get secret ceph-user-secret
NAME               TYPE                DATA      AGE
ceph-user-secret   kubernetes.io/rbd   1         5d

Verify and view the cluster environment, including nodes and pods:

$ oc get nodes,pods
NAME      STATUS    AGE
master    Ready     22h
node0     Ready     22h
node1     Ready     22h
node2     Ready     22h
NAME                               READY     STATUS              RESTARTS   AGE           1/1       Running             0          1d
glusterfs-node0-2509304327-vpce1   1/1       Running   0          1d        192.168.10.100   node0
glusterfs-node1-3290690057-hhq92   1/1       Running   0          1d        192.168.10.101   node1 1
glusterfs-node2-4072075787-okzjv   1/1       Running   0          1d        192.168.10.102   node2
heketi-3017632314-yyngh            1/1       Running   0          1d        10.42.0.0        node0 2

If not already set in the environment, export the HEKETI_CLI_SERVER:

$ export HEKETI_CLI_SERVER=$(oc describe svc/heketi | grep "Endpoints:" | awk '{print "http://"$2}')

Identify the Heketi REST URL and server IP address:

$ echo $HEKETI_CLI_SERVER
http://10.42.0.0:8080

Identify the Gluster endpoints that are needed to pass in as a parameter into the storage class, which is used in a later step (heketi-storage-endpoints).

$ oc get endpoints
NAME                       ENDPOINTS                                            AGE
heketi                     10.42.0.0:8080                                       22h
heketi-storage-endpoints   192.168.10.100:1,192.168.10.101:1,192.168.10.102:1   22h 1
kubernetes                 192.168.10.90:6443                                   23h

Create the Storage Class YAML file, save it, then submit it to OpenShift Container Platform:

$ oc create -f gluster-storage-class.yaml
storageclass "gluster-heketi" created

View the storage class:

$ oc get storageclass
NAME              TYPE
gluster-heketi    kubernetes.io/glusterfs

Create a persistent volume claim (PVC) requesting 5GB of storage.

During that time, the Dynamic Provisioning Framework and Heketi will automatically provision a new GlusterFS volume and generate the persistent volume (PV) object:

Create the PVC YAML file, save it, then submit it to OpenShift Container Platform:

$ oc create -f gluster-pvc.yaml
persistentvolumeclaim "gluster1" created

View the PVC:

$ oc get pvc
NAME       STATUS    VOLUME                                     CAPACITY   ACCESSMODES   AGE
gluster1   Bound     pvc-7d37c7bd-bb5b-11e6-b81e-525400d87180   5Gi        RWO           14h

Notice that the PVC is bound to a dynamically created volume.

View the persistent volume (PV):

$ oc get pv
NAME                                       CAPACITY   ACCESSMODES   RECLAIMPOLICY   STATUS    CLAIM              REASON    AGE
pvc-7d37c7bd-bb5b-11e6-b81e-525400d87180   5Gi        RWO           Delete          Bound     default/gluster1             14h

Create the Pod YAML file, save it, then submit it to OpenShift Container Platform:

$ oc create -f nginx-pod.yaml
pod "gluster-pod1" created

View the pod:

$ oc get pods -o wide
NAME                               READY     STATUS    RESTARTS   AGE       IP               NODE
nginx-pod                          1/1       Running   0          9m        10.38.0.0        node1
glusterfs-node0-2509304327-vpce1   1/1       Running   0          1d        192.168.10.100   node0
glusterfs-node1-3290690057-hhq92   1/1       Running   0          1d        192.168.10.101   node1
glusterfs-node2-4072075787-okzjv   1/1       Running   0          1d        192.168.10.102   node2
heketi-3017632314-yyngh            1/1       Running   0          1d        10.42.0.0        node0

oc exec into the container and create an index.html file in the mountPath definition of the pod:

$ oc exec -ti nginx-pod /bin/sh
$ cd /usr/share/nginx/html
$ echo 'Hello World from GlusterFS!!!' > index.html
$ ls
index.html
$ exit

Using the curl command from the master node, curl the URL of the pod:

$ curl http://10.38.0.0
Hello World from GlusterFS!!!

Check your Gluster pod to ensure that the index.html file was written. Choose any of the Gluster pods:

$ oc exec -ti glusterfs-node1-3290690057-hhq92 /bin/sh
$ mount | grep heketi
/dev/mapper/VolGroup00-LogVol00 on /var/lib/heketi type xfs (rw,relatime,seclabel,attr2,inode64,noquota)
/dev/mapper/vg_f92e09091f6b20ab12b02a2513e4ed90-brick_1e730a5462c352835055018e1874e578 on /var/lib/heketi/mounts/vg_f92e09091f6b20ab12b02a2513e4ed90/brick_1e730a5462c352835055018e1874e578 type xfs (rw,noatime,seclabel,nouuid,attr2,inode64,logbsize=256k,sunit=512,swidth=512,noquota)
/dev/mapper/vg_f92e09091f6b20ab12b02a2513e4ed90-brick_d8c06e606ff4cc29ccb9d018c73ee292 on /var/lib/heketi/mounts/vg_f92e09091f6b20ab12b02a2513e4ed90/brick_d8c06e606ff4cc29ccb9d018c73ee292 type xfs (rw,noatime,seclabel,nouuid,attr2,inode64,logbsize=256k,sunit=512,swidth=512,noquota)

$ cd /var/lib/heketi/mounts/vg_f92e09091f6b20ab12b02a2513e4ed90/brick_d8c06e606ff4cc29ccb9d018c73ee292/brick
$ ls
index.html
$ cat index.html
Hello World from GlusterFS!!!

Install Heketi and the Heketi client. From the host designated to run Heketi and the Heketi client, run:

# yum install heketi heketi-client -y

Create and install Heketi private keys on each GlusterFS cluster node. From the host that is running Heketi:

# ssh-keygen -f /etc/heketi/heketi_key -t rsa -N ''
# ssh-copy-id -i /etc/heketi/heketi_key.pub root@gluster23.rhs
# ssh-copy-id -i /etc/heketi/heketi_key.pub root@gluster24.rhs
# ssh-copy-id -i /etc/heketi/heketi_key.pub root@gluster25.rhs
# chown heketi:heketi /etc/heketi/heketi_key*

Edit the /etc/heketi/heketi.json file to setup the SSH executor. Below is an excerpt from the /etc/heketi/heketi.json file; the parts to configure are the executor and SSH sections:

	"executor": "ssh", 1

	"_sshexec_comment": "SSH username and private key file information",
	"sshexec": {
  	  "keyfile": "/etc/heketi/heketi_key", 2
  	  "user": "root", 3
  	  "port": "22", 4
  	  "fstab": "/etc/fstab" 5
	},

Restart and enable service:

# systemctl restart heketi
# systemctl enable heketi

Test the connection to Heketi:

# curl http://glusterclient2.rhs:8080/hello
Hello from Heketi

Set an environment variable for the Heketi server:

# export HEKETI_CLI_SERVER=http://glusterclient2.rhs:8080

Create and load the topology file. There is a sample file located in /usr/share/heketi/topology-sample.json by default, or /etc/heketi depending on how it was installed.

{
  "clusters": [
    {
      "nodes": [
        {
          "node": {
            "hostnames": {
              "manage": [
                "gluster23.rhs"
              ],
              "storage": [
                "192.168.1.200"
              ]
            },
            "zone": 1
          },
          "devices": [
            "/dev/sde",
            "/dev/sdf"
          ]
        },
        {
          "node": {
            "hostnames": {
              "manage": [
                "gluster24.rhs"
              ],
              "storage": [
                "192.168.1.201"
              ]
            },
            "zone": 1
          },
          "devices": [
            "/dev/sde",
            "/dev/sdf"
          ]
        },
        {
          "node": {
            "hostnames": {
              "manage": [
                "gluster25.rhs"
              ],
              "storage": [
                "192.168.1.202"
              ]
            },
            "zone": 1
          },
          "devices": [
            "/dev/sde",
            "/dev/sdf"
          ]
        }
      ]
    }
  ]
}

Using heketi-cli, run the following command to load the topology of your environment.

# heketi-cli topology load --json=topology.json

    	Found node gluster23.rhs on cluster bdf9d8ca3fa269ff89854faf58f34b9a
   		Adding device /dev/sde ... OK
   	 	Adding device /dev/sdf ... OK
    	Creating node gluster24.rhs ... ID: 8e677d8bebe13a3f6846e78a67f07f30
   	 	Adding device /dev/sde ... OK
   	 	Adding device /dev/sdf ... OK
...
...

Create a Gluster volume to verify Heketi:

# heketi-cli volume create --size=50

View the volume information from one of the the Gluster nodes:

# gluster volume info

	Volume Name: vol_335d247ac57ecdf40ac616514cc6257f 1
	Type: Distributed-Replicate
	Volume ID: 75be7940-9b09-4e7f-bfb0-a7eb24b411e3
	Status: Started
...
...

Create a StorageClass object definition. The definition below is based on the minimum requirements needed for this example to work with OpenShift Container Platform. See Dynamic Provisioning and Creating Storage Classes for additional parameters and specification definitions.

kind: StorageClass
apiVersion: storage.k8s.io/v1beta1
metadata:
  name: gluster-dyn
provisioner: kubernetes.io/glusterfs
parameters:
  resturl: "http://glusterclient2.rhs:8080" 1
  restauthenabled: "false" 2

From the OpenShift Container Platform master host, create the storage class:

# oc create -f glusterfs-storageclass1.yaml
storageclass "gluster-dyn" created

Create a persistent volume claim (PVC), requesting the newly-created storage class. For example:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
 name: gluster-dyn-pvc
 annotations:
   volume.beta.kubernetes.io/storage-class: gluster-dyn
spec:
 accessModes:
  - ReadWriteMany
 resources:
   requests:
        storage: 30Gi

From the OpenShift Container Platform master host, create the PVC:

# oc create -f glusterfs-pvc-storageclass.yaml
persistentvolumeclaim "gluster-dyn-pvc" created

View the PVC to see that the volume was dynamically created and bound to the PVC:

# oc get pvc
NAME          	STATUS	VOLUME                                 		CAPACITY   	ACCESSMODES   	STORAGECLASS   	AGE
gluster-dyn-pvc Bound	pvc-78852230-d8e2-11e6-a3fa-0800279cf26f   	30Gi   		RWX       	gluster-dyn	42s

Verify and view the new volume on one of the Gluster nodes:

# gluster volume info

	Volume Name: vol_335d247ac57ecdf40ac616514cc6257f 1
	Type: Distributed-Replicate
	Volume ID: 75be7940-9b09-4e7f-bfb0-a7eb24b411e3
	Status: Started
        ...
	Volume Name: vol_f1404b619e6be6ef673e2b29d58633be 2
	Type: Distributed-Replicate
	Volume ID: 7dc234d0-462f-4c6c-add3-fb9bc7e8da5e
	Status: Started
	Number of Bricks: 2 x 2 = 4
	...

Create the pod object definition:

apiVersion: v1
kind: Pod
metadata:
  name: gluster-pod1
  labels:
    name: gluster-pod1
spec:
  containers:
  - name: gluster-pod1
    image: gcr.io/google_containers/nginx-slim:0.8
    ports:
    - name: web
      containerPort: 80
    securityContext:
      privileged: true
    volumeMounts:
    - name: gluster-vol1
      mountPath: /usr/share/nginx/html
  volumes:
  - name: gluster-vol1
    persistentVolumeClaim:
      claimName: gluster-dyn-pvc 1

From the OpenShift Container Platform master host, create the pod:

# oc create -f nginx-pod.yaml
pod "gluster-pod1" created

View the pod. Give it a few minutes, as it might need to download the image if it does not already exist:

# oc get pods -o wide
NAME                               READY     STATUS    RESTARTS   AGE       IP               NODE
gluster-pod1                       1/1       Running   0          9m        10.38.0.0        node1

Now remote into the container with oc exec and create an index.html file:

# oc exec -ti gluster-pod1 /bin/sh
$ cd /usr/share/nginx/html
$ echo 'Hello World from GlusterFS!!!' > index.html
$ ls
index.html
$ exit

Now curl the URL of the pod:

# curl http://10.38.0.0
Hello World from GlusterFS!!!

Create a new project:

$ oc new-project <project-name>

Enable privileged containers in the new project:

$ oc adm policy add-scc-to-user privileged -z default

Register the deploy-heketi template:

$ oc create -f <template-path>/deploy-heketi-template

Deploy the bootstrap Heketi container:

$ oc process deploy-heketi -v \
         HEKETI_KUBE_NAMESPACE=<project-name> \
         HEKETI_KUBE_APIHOST=<master-url-and-port> \
         HEKETI_KUBE_INSECURE=y \
         HEKETI_KUBE_USER=<cluster-admin-username> \
         HEKETI_KUBE_PASSWORD=<cluster-admin-password> | oc create -f -

Wait until the deploy-heketi pod starts and all services are running. Then get Heketi service details:

$ oc get svc
NAME            CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
deploy-heketi   172.30.96.173   <none>        8080/TCP   2m

Check if Heketi services are running properly, it must return Hello from Heketi.

$ curl http://<cluster-ip>:8080/hello
Hello from Heketi

Set an environment variable for the Heketi server:

$ export HEKETI_CLI_SERVER=http://<cluster-ip>:8080

Create and load the topology file. There is a sample file located in /usr/share/heketi/topology-sample.json by default, or /etc/heketi depending on how it was installed.

{
  "clusters": [
    {
      "nodes": [
        {
          "node": {
            "hostnames": {
              "manage": [
                "gluster23.rhs"
              ],
              "storage": [
                "192.168.1.200"
              ]
            },
            "zone": 1
          },
          "devices": [
            "/dev/sde",
            "/dev/sdf"
          ]
        },
        {
          "node": {
            "hostnames": {
              "manage": [
                "gluster24.rhs"
              ],
              "storage": [
                "192.168.1.201"
              ]
            },
            "zone": 1
          },
          "devices": [
            "/dev/sde",
            "/dev/sdf"
          ]
        },
        {
          "node": {
            "hostnames": {
              "manage": [
                "gluster25.rhs"
              ],
              "storage": [
                "192.168.1.202"
              ]
            },
            "zone": 1
          },
          "devices": [
            "/dev/sde",
            "/dev/sdf"
          ]
        }
      ]
    }
  ]
}

Run the following command to load the topology of your environment.

$ heketi-cli topology load --json=topology-sample.json

    	Found node gluster23.rhs on cluster bdf9d8ca3fa269ff89854faf58f34b9a
   		Adding device /dev/sde ... OK
   	 	Adding device /dev/sdf ... OK
    	Creating node gluster24.rhs ... ID: 8e677d8bebe13a3f6846e78a67f07f30
   	 	Adding device /dev/sde ... OK
   	 	Adding device /dev/sdf ... OK
...

Create a Gluster volume to verify Heketi:

$ heketi-cli volume create --size=50

View the volume information from one of the the Gluster nodes:

$ gluster volume info

	Volume Name: vol_335d247ac57ecdf40ac616514cc6257f 1
	Type: Distributed-Replicate
	Volume ID: 75be7940-9b09-4e7f-bfb0-a7eb24b411e3
	Status: Started
...

Create a StorageClass object definition. The following definition is based on the minimum requirements needed for this example to work with OpenShift Container Platform. See Dynamic Provisioning and Creating Storage Classes for additional parameters and specification definitions.

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: gluster-dyn
provisioner: kubernetes.io/glusterfs
parameters:
  resturl: "http://glusterclient2.rhs:8080" 1
  restauthenabled: "false" 2

From the OpenShift Container Platform master host, create the storage class:

$ oc create -f glusterfs-storageclass1.yaml
storageclass "gluster-dyn" created

Create a persistent volume claim (PVC), requesting the newly-created storage class. For example:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
 name: gluster-dyn-pvc
spec:
 accessModes:
  - ReadWriteMany
 resources:
   requests:
        storage: 30Gi
 storageClassName: gluster-dyn

From the OpenShift Container Platform master host, create the PVC:

$ oc create -f glusterfs-pvc-storageclass.yaml
persistentvolumeclaim "gluster-dyn-pvc" created

View the PVC to see that the volume was dynamically created and bound to the PVC:

$ oc get pvc
NAME          	STATUS	VOLUME                                 		CAPACITY   	ACCESSMODES   	STORAGECLASS   	AGE
gluster-dyn-pvc Bound	pvc-78852230-d8e2-11e6-a3fa-0800279cf26f   	30Gi   		RWX       	gluster-dyn	42s

Verify and view the new volume on one of the Gluster nodes:

$ gluster volume info

	Volume Name: vol_335d247ac57ecdf40ac616514cc6257f 1
	Type: Distributed-Replicate
	Volume ID: 75be7940-9b09-4e7f-bfb0-a7eb24b411e3
	Status: Started
        ...
	Volume Name: vol_f1404b619e6be6ef673e2b29d58633be 2
	Type: Distributed-Replicate
	Volume ID: 7dc234d0-462f-4c6c-add3-fb9bc7e8da5e
	Status: Started
	Number of Bricks: 2 x 2 = 4
	...

Create the pod object definition:

apiVersion: v1
kind: Pod
metadata:
  name: gluster-pod1
  labels:
    name: gluster-pod1
spec:
  containers:
  - name: gluster-pod1
    image: gcr.io/google_containers/nginx-slim:0.8
    ports:
    - name: web
      containerPort: 80
    securityContext:
      privileged: true
    volumeMounts:
    - name: gluster-vol1
      mountPath: /usr/share/nginx/html
  volumes:
  - name: gluster-vol1
    persistentVolumeClaim:
      claimName: gluster-dyn-pvc 1

From the OpenShift Container Platform master host, create the pod:

$ oc create -f nginx-pod.yaml
pod "gluster-pod1" created

View the pod. Give it a few minutes, as it might need to download the image if it does not already exist:

$ oc get pods -o wide
NAME                               READY     STATUS    RESTARTS   AGE       IP               NODE
gluster-pod1                       1/1       Running   0          9m        10.38.0.0        node1

Now remote into the container with oc exec and create an index.html file:

$ oc exec -ti gluster-pod1 /bin/sh
$ cd /usr/share/nginx/html
$ echo 'Hello World from GlusterFS!!!' > index.html
$ ls
index.html
$ exit

Now curl the URL of the pod:

$ curl http://10.38.0.0
Hello World from GlusterFS!!!

As the admin, create the service, endpoint object, and persistent volume:

$ oc create -f gluster-endpoints-service.yaml
$ oc create -f gluster-endpoints.yaml
$ oc create -f gluster-pv.yaml

Verify that the objects were created:

$ oc get svc
NAME              CLUSTER_IP      EXTERNAL_IP   PORT(S)   SELECTOR   AGE
gluster-cluster   172.30.151.58   <none>        1/TCP     <none>     24s

$ oc get ep
NAME              ENDPOINTS                           AGE
gluster-cluster   192.168.59.102:1,192.168.59.103:1   2m

$ oc get pv
NAME                     LABELS    CAPACITY   ACCESSMODES   STATUS      CLAIM     REASON    AGE
gluster-default-volume   <none>    2Gi        RWX           Available                       2d

As the admin, add a user to the SCC:

$ oc adm policy add-scc-to-user privileged <username>

Log in as the regular user:

$ oc login -u <username> -p <password>

Then, create a new project:

$ oc new-project <project_name>

As a regular user, create the PersistentVolumeClaim to access the volume:

$ oc create -f gluster-pvc.yaml -n <project_name>

Define your pod to access the claim:

apiVersion: v1
id: gluster-S3-pvc
kind: Pod
metadata:
  name: gluster-nginx-priv
spec:
  containers:
    - name: gluster-nginx-priv
      image: fedora/nginx
      volumeMounts:
        - mountPath: /mnt/gluster 1
          name: gluster-volume-claim
      securityContext:
        privileged: true
  volumes:
    - name: gluster-volume-claim
      persistentVolumeClaim:
        claimName: gluster-claim 2

Upon pod creation, the mount directory is created and the volume is attached to that mount point.

As regular user, create a pod from the definition:

$ oc create -f gluster-S3-pod.yaml

Verify that the pod created successfully:

$ oc get pods
NAME                 READY     STATUS    RESTARTS   AGE
gluster-S3-pod   1/1       Running   0          36m

It can take several minutes for the pod to create.

Create a new registry configuration file called registryconfig.yaml and add the following information:

version: 0.1
log:
  level: debug
http:
  addr: :5000
storage:
  cache:
    blobdescriptor: inmemory
  delete:
    enabled: true
  azure: 1
    accountname: azureblobacc
    accountkey:  azureblobacckey
    container: azureblobname
    realm: core.windows.net 2
auth:
  openshift:
    realm: openshift
middleware:
  registry:
    - name: openshift
  repository:
    - name: openshift
      options:
        acceptschema2: false
        pullthrough: true
        enforcequota: false
        projectcachettl: 1m
        blobrepositorycachettl: 10m
  storage:
    - name: openshift