For various reasons I wanted to play with docker containers backed by persistent Ceph storage. rexray seemed like the way to do that, so here are my notes on getting that working…
First off, I needed to install rexray:
root@labosa:~/rexray# curl -sSL https://dl.bintray.com/emccode/rexray/install | sh Selecting previously unselected package rexray. (Reading database ... 177547 files and directories currently installed.) Preparing to unpack rexray_0.9.0-1_amd64.deb ... Unpacking rexray (0.9.0-1) ... Setting up rexray (0.9.0-1) ... rexray has been installed to /usr/bin/rexray REX-Ray ------- Binary: /usr/bin/rexray Flavor: client+agent+controller SemVer: 0.9.0 OsArch: Linux-x86_64 Branch: v0.9.0 Commit: 2a7458dd90a79c673463e14094377baf9fc8695e Formed: Thu, 04 May 2017 07:38:11 AEST libStorage ---------- SemVer: 0.6.0 OsArch: Linux-x86_64 Branch: v0.9.0 Commit: fa055d6da595602715bdfd5541b4aa6d4dcbcbd9 Formed: Thu, 04 May 2017 07:36:11 AEST
Which is of course horrid. What that script seems to have done is install a deb’d version of rexray based on an alien’d package:
root@labosa:~/rexray# dpkg -s rexray Package: rexray Status: install ok installed Priority: extra Section: alien Installed-Size: 36140 Maintainer: Travis CI User <travis@testing-gce-7fbf00fc-f7cd-4e37-a584-810c64fdeeb1> Architecture: amd64 Version: 0.9.0-1 Depends: libc6 (>= 2.3.2) Description: Tool for managing remote & local storage. A guest based storage introspection tool that allows local visibility and management from cloud and storage platforms. . (Converted from a rpm package by alien version 8.86.)
If I was building anything more than a test environment I think I’d want to do a better job of installing rexray than this, so you’ve been warned.
Next to configure rexray to use Ceph. The configuration details are cunningly hidden in the libstorage docs, and aren’t mentioned at all in the rexray docs, so you probably want to take a look at the libstorage docs on ceph. First off, we need to install the ceph tools, and copy the ceph authentication information from the the ceph we installed using openstack-ansible earlier.
root@labosa:/etc# apt-get install ceph-common root@labosa:/etc# scp -rp 172.29.239.114:/etc/ceph . The authenticity of host '172.29.239.114 (172.29.239.114)' can't be established. ECDSA key fingerprint is SHA256:SA6U2fuXyVbsVJIoCEHL+qlQ3xEIda/MDOnHOZbgtnE. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added '172.29.239.114' (ECDSA) to the list of known hosts. rbdmap 100% 92 0.1KB/s 00:00 ceph.conf 100% 681 0.7KB/s 00:00 ceph.client.admin.keyring 100% 63 0.1KB/s 00:00 ceph.client.glance.keyring 100% 64 0.1KB/s 00:00 ceph.client.cinder.keyring 100% 64 0.1KB/s 00:00 ceph.client.cinder-backup.keyring 71 0.1KB/s 00:00 root@labosa:/etc# modprobe rbd
You also need to configure rexray. My first attempt looked like this:
root@labosa:/var/log# cat /etc/rexray/config.yml libstorage: service: ceph
And the rexray output sure made it look like it worked…
root@labosa:/etc# rexray service start
● rexray.service - rexray
Loaded: loaded (/etc/systemd/system/rexray.service; enabled; vendor preset: enabled)
Active: active (running) since Mon 2017-05-29 10:14:07 AEST; 33ms ago
Main PID: 477423 (rexray)
Tasks: 5
Memory: 1.5M
CPU: 9ms
CGroup: /system.slice/rexray.service
└─477423 /usr/bin/rexray start -f
May 29 10:14:07 labosa systemd[1]: Started rexray.
Which looked good, but /var/log/syslog said:
May 29 10:14:08 labosa rexray[477423]: REX-Ray May 29 10:14:08 labosa rexray[477423]: ------- May 29 10:14:08 labosa rexray[477423]: Binary: /usr/bin/rexray May 29 10:14:08 labosa rexray[477423]: Flavor: client+agent+controller May 29 10:14:08 labosa rexray[477423]: SemVer: 0.9.0 May 29 10:14:08 labosa rexray[477423]: OsArch: Linux-x86_64 May 29 10:14:08 labosa rexray[477423]: Branch: v0.9.0 May 29 10:14:08 labosa rexray[477423]: Commit: 2a7458dd90a79c673463e14094377baf9fc8695e May 29 10:14:08 labosa rexray[477423]: Formed: Thu, 04 May 2017 07:38:11 AEST May 29 10:14:08 labosa rexray[477423]: libStorage May 29 10:14:08 labosa rexray[477423]: ---------- May 29 10:14:08 labosa rexray[477423]: SemVer: 0.6.0 May 29 10:14:08 labosa rexray[477423]: OsArch: Linux-x86_64 May 29 10:14:08 labosa rexray[477423]: Branch: v0.9.0 May 29 10:14:08 labosa rexray[477423]: Commit: fa055d6da595602715bdfd5541b4aa6d4dcbcbd9 May 29 10:14:08 labosa rexray[477423]: Formed: Thu, 04 May 2017 07:36:11 AEST May 29 10:14:08 labosa rexray[477423]: time="2017-05-29T10:14:08+10:00" level=error msg="error starting libStorage server" error.driver=ceph time=1496016848215 May 29 10:14:08 labosa rexray[477423]: time="2017-05-29T10:14:08+10:00" level=error msg="default module(s) failed to initialize" error.driver=ceph time=1496016848216 May 29 10:14:08 labosa rexray[477423]: time="2017-05-29T10:14:08+10:00" level=error msg="daemon failed to initialize" error.driver=ceph time=1496016848216 May 29 10:14:08 labosa rexray[477423]: time="2017-05-29T10:14:08+10:00" level=error msg="error starting rex-ray" error.driver=ceph time=1496016848216
That’s because the service is called rbd it seems. So, the config file ended up looking like this:
root@labosa:/var/log# cat /etc/rexray/config.yml libstorage: service: rbd rbd: defaultPool: rbd
Now to install docker:
root@labosa:/var/log# sudo apt-get update
root@labosa:/var/log# sudo apt-get install linux-image-extra-$(uname -r) \
linux-image-extra-virtual
root@labosa:/var/log# sudo apt-get install apt-transport-https \
ca-certificates curl software-properties-common
root@labosa:/var/log# curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
root@labosa:/var/log# sudo add-apt-repository \
"deb [arch=amd64] https://download.docker.com/linux/ubuntu \
$(lsb_release -cs) \
stable"
root@labosa:/var/log# sudo apt-get update
root@labosa:/var/log# sudo apt-get install docker-ce
Now let’s make a rexray volume.
root@labosa:/var/log# rexray volume ls
ID Name Status Size
root@labosa:/var/log# docker volume create --driver=rexray --name=mysql \
--opt=size=1
A size of 1 here means 1gb
mysql
root@labosa:/var/log# rexray volume ls
ID Name Status Size
rbd.mysql mysql available 1
Let’s start the container.
root@labosa:/var/log# docker run --name some-mysql --volume-driver=rexray \
-v mysql:/var/lib/mysql -e MYSQL_ROOT_PASSWORD=my-secret-pw -d mysql
Unable to find image 'mysql:latest' locally
latest: Pulling from library/mysql
10a267c67f42: Pull complete
c2dcc7bb2a88: Pull complete
17e7a0445698: Pull complete
9a61839a176f: Pull complete
a1033d2f1825: Pull complete
0d6792140dcc: Pull complete
cd3adf03d6e6: Pull complete
d79d216fd92b: Pull complete
b3c25bdeb4f4: Pull complete
02556e8f331f: Pull complete
4bed508a9e77: Pull complete
Digest: sha256:2f4b1900c0ee53f344564db8d85733bd8d70b0a78cd00e6d92dc107224fc84a5
Status: Downloaded newer image for mysql:latest
ccc251e6322dac504e978f4b95b3787517500de61eb251017cc0b7fd878c190b
And now to prove that persistence works and that there’s nothing up my sleeve…
root@labosa:/var/log# docker run -it --link some-mysql:mysql --rm mysql \
sh -c 'exec mysql -h"$MYSQL_PORT_3306_TCP_ADDR" \
-P"$MYSQL_PORT_3306_TCP_PORT" -uroot -p"$MYSQL_ENV_MYSQL_ROOT_PASSWORD"'
mysql: [Warning] Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 3
Server version: 5.7.18 MySQL Community Server (GPL)
Copyright (c) 2000, 2017, Oracle and/or its affiliates. All rights reserved.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql> show databases;
+--------------------+
| Database |
+--------------------+
| information_schema |
| mysql |
| performance_schema |
| sys |
+--------------------+
4 rows in set (0.00 sec)
mysql> create database demo;
Query OK, 1 row affected (0.03 sec)
mysql> use demo;
Database changed
mysql> create table foo(val char(5));
Query OK, 0 rows affected (0.14 sec)
mysql> insert into foo(val) values ('a'), ('b'), ('c');
Query OK, 3 rows affected (0.08 sec)
Records: 3 Duplicates: 0 Warnings: 0
mysql> select * from foo;
+------+
| val |
+------+
| a |
| b |
| c |
+------+
3 rows in set (0.00 sec)
Now let’s re-create the container and prove the data remains.
root@labosa:/var/log# docker stop some-mysql
some-mysql
root@labosa:/var/log# docker rm some-mysql
some-mysql
root@labosa:/var/log# docker run --name some-mysql --volume-driver=rexray \
-v mysql:/var/lib/mysql -e MYSQL_ROOT_PASSWORD=my-secret-pw -d mysql
99a7ccae1ad1865eb1bcc8c757251903dd2f1ac7d3ce4e365b5cdf94f539fe05
root@labosa:/var/log# docker run -it --link some-mysql:mysql --rm mysql \
sh -c 'exec mysql -h"$MYSQL_PORT_3306_TCP_ADDR" -\
P"$MYSQL_PORT_3306_TCP_PORT" -uroot -p"$MYSQL_ENV_MYSQL_ROOT_PASSWORD"'
mysql: [Warning] Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 3
Server version: 5.7.18 MySQL Community Server (GPL)
Copyright (c) 2000, 2017, Oracle and/or its affiliates. All rights reserved.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql> use demo;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A
Database changed
mysql> select * from foo;
+------+
| val |
+------+
| a |
| b |
| c |
+------+
3 rows in set (0.00 sec)
So there you go.