Installing Ceph docker plugin to DCOS using Rex-Ray
- Get link
- X
- Other Apps
Installing Ceph docker plugin to DCOS using Rex-Ray
Introduction
From the previous blog, DC/OS 1.9 and Ceph have been installed. To install Jenkins using external volume, I decided to use Rex-Ray because most of other plugins have not been maintained properly to support latest version.
Reference
- dcos-rexray
- Marathon rexray
- Configuring docker to use rexray and Ceph for persistent storage
- Rex-ray configuration generator
- rexray Docker volume plug-ins
Install Rex-Ray
[dcosadmin@dcotest02 ~]$ sudo su -
Last login: Mon Jun 26 12:50:35 MDT 2017 from 172.16.110.20 on pts/0
[root@dcotest02 ~]# curl -sSL https://dl.bintray.com/emccode/rexray/install | sh
INFO[0000] updated log level logLevel=info
INFO[0000] exiting process time=1498592066094
rexray has been installed to /usr/bin/rexray
INFO[0000] updated log level logLevel=info
REX-Ray
-------
Binary: /usr/bin/rexray
Flavor: client+agent+controller
SemVer: 0.9.1
OsArch: Linux-x86_64
Branch: v0.9.1
Commit: 2373541479478b817a8b143629e552f404f75226
Formed: Fri, 09 Jun 2017 13:23:38 MDT
libStorage
----------
SemVer: 0.6.1
OsArch: Linux-x86_64
Branch: v0.9.1
Commit: fd26f0ec72b077ffa7c82160fbd12a276e12c2ad
Formed: Fri, 09 Jun 2017 13:23:05 MDT
INFO[0000] exiting process time=1498592066292
[root@dcotest02 ~]#
Checking package
[root@dcotest02 ~]# yum search rexray
Loaded plugins: fastestmirror
Determining fastest mirrors
* base: mirror.csclub.uwaterloo.ca
* extras: centos.mirror.ca.planethoster.net
* updates: centos.mirror.ca.planethoster.net
======================================= N/S matched: rexray ========================================
rexray.x86_64 : Tool for managing remote & local storage.
Name and summary matches only, use "search all" for everything.
[root@dcotest02 ~]#
Configure Rex-Ray
If ceph-common is not installed on master node and each agent, ceph-common page should be installed.
Go to configurator site. refer reference
[root@dcotest02 ~]# yum install ceph-common
Loaded plugins: fastestmirror
base | 3.6 kB 00:00:00
centos-ceph-jewel | 2.9 kB 00:00:00
dockerrepo | 2.9 kB 00:00:00
extras | 3.4 kB 00:00:00
updates | 3.4 kB 00:00:00
Loading mirror speeds from cached hostfile
* base: mirror.csclub.uwaterloo.ca
* extras: centos.mirror.ca.planethoster.net
* updates: centos.mirror.ca.planethoster.net
Package 1:ceph-common-10.2.7-0.el7.x86_64 already installed and latest version
Nothing to do
[root@dcotest02 ~]# modprobe rbd
[root@dcotest02 ~]# cat /etc/rexray/config.yml
rexray:
loglevel: info
modules:
default-admin:
host: tcp://127.0.0.1:61003
default-docker:
disabled: true
[root@dcotest02 ~]# vi /etc/rexray/config.yml
[root@dcotest02 ~]# rexray service start
INFO[0000] updated log level logLevel=info
● rexray.service - rexray
Loaded: loaded (/etc/systemd/system/rexray.service; enabled; vendor preset: disabled)
Active: active (running) since Tue 2017-06-27 13:42:01 MDT; 5ms ago
Main PID: 11838 (rexray)
Memory: 1.1M
CGroup: /system.slice/rexray.service
└─11838 /usr/bin/rexray start -f
Jun 27 13:42:01 dcotest02.test.flairpackaging.com systemd[1]: Started rexray.
Jun 27 13:42:01 dcotest02.test.flairpackaging.com systemd[1]: Starting rexray...
INFO[0000] exiting process time=1498592521889
[root@dcotest02 log]
Correct configuration file /etc/rexray/config.yaml
[root@dcotest02 log]# cat /etc/rexray/config.yml
libstorage:
service: rbd
rbd:
defaultPool: rbd
[root@dcotest02 log]#
Install to all agents
- copy configuration file
- install ceph-commons
- install rexray
- update rexray configuration file
[dcosadmin@dcotest08 ~]$ sudo mkdir -p /etc/ceph
[dcosadmin@dcotest08 ~]$ sudo scp 172.16.110.30:/etc/ceph/* /etc/ceph
The authenticity of host '172.16.110.30 (172.16.110.30)' can't be established.
ECDSA key fingerprint is dc:75:bb:70:96:f7:76:45:0f:13:6b:92:37:3b:f5:3e.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '172.16.110.30' (ECDSA) to the list of known hosts.
root@172.16.110.30's password:
ceph.client.admin.keyring 100% 142 0.1KB/s 00:00
ceph.conf 100% 540 0.5KB/s 00:00
ceph.mon.keyring 100% 77 0.1KB/s 00:00
rbdmap 100% 92 0.1KB/s 00:00
[dcosadmin@dcotest08 ~]$ sudo yum install ceph-common
....
Complete!
[dcosadmin@dcotest08 ~]$
[dcosadmin@dcotest08 ~]$ sudo curl -sSL https://dl.bintray.com/emccode/rexray/install | sudo sh
INFO[0000] updated log level logLevel=info
INFO[0000] exiting process time=1498599121167
rexray has been installed to /usr/bin/rexray
INFO[0000] updated log level logLevel=info
REX-Ray
-------
Binary: /usr/bin/rexray
Flavor: client+agent+controller
SemVer: 0.9.1
OsArch: Linux-x86_64
Branch: v0.9.1
Commit: 2373541479478b817a8b143629e552f404f75226
Formed: Fri, 09 Jun 2017 13:23:38 MDT
libStorage
----------
SemVer: 0.6.1
OsArch: Linux-x86_64
Branch: v0.9.1
Commit: fd26f0ec72b077ffa7c82160fbd12a276e12c2ad
Formed: Fri, 09 Jun 2017 13:23:05 MDT
INFO[0000] exiting process time=1498599121364
[dcosadmin@dcotest08 ~]$
[dcosadmin@dcotest08 ~]$ sudo vi /etc/rexray/config.yml
[dcosadmin@dcotest08 ~]$ cat /etc/rexray/config.yml
libstorage:
service: rbd
rbd:
defaultPool: rbd
[dcosadmin@dcotest08 ~]$ sudo rexray service start
[root@dcotest03 ~]# rexray volume ls
ID Name Status Size
[root@dcotest03 ~]#
Installing Rexray RBD plugin for Docker
[root@dcotest03 ~]# docker plugin install rexray/rbd:0.9.2-rc2
Plugin "rexray/rbd:0.9.2-rc2" is requesting the following privileges:
- network: [host]
- mount: [/dev]
- mount: [/etc/ceph]
- allow-all-devices: [true]
- capabilities: [CAP_SYS_ADMIN]
Do you grant the above permissions? [y/N] y
0.9.2-rc2: Pulling from rexray/rbd
f683ee9c713a: Download complete
Digest: sha256:27c63f276f21735c2bb2091fcca9cb9ac3992d76dec178ec422b63e0864d1c86
Status: Downloaded newer image for rexray/rbd:0.9.2-rc2
Installed plugin rexray/rbd:0.9.2-rc2
[root@dcotest03 ~]#
[root@dcotest03 ~]# docker plugin ls
ID NAME DESCRIPTION ENABLED
eb1765646ce0 rexray/rbd:0.9.2-rc2 REX-Ray for Ceph RBD true
[root@dcotest03 ~]#
[root@dcotest03 ~]# docker volume create --driver=rexray/rbd:0.9.2-rc2 --name=jenkins --opt=size=4
jenkins
[root@dcotest03 ~]# docker volume inspect jenkins
[
{
"Driver": "rexray/rbd:0.9.2-rc2",
"Labels": {},
"Mountpoint": "/var/lib/docker/plugins/eb1765646ce06176b78d3d357af3c86dc284cb49c47911e6aef46bf8fa03fe68/rootfs",
"Name": "jenkins",
"Options": {
"size": "4"
},
"Scope": "global",
"Status": {
"availabilityZone": "",
"fields": null,
"iops": 0,
"name": "jenkins",
"server": "rbd",
"service": "rbd",
"size": 4,
"type": "rbd"
}
}
]
[root@dcotest03 ~]#
Repeate installing ceph-common and docker plugin through all agensts. In my case, I installed from 172.16.110.31 to 172.16.110.36. After completing the installation, you can see volume(rbd image) from all node by running sudo docker volume ls.
[dcosadmin@dcotest06 ~]$ sudo docker plugin install rexray/rbd:0.9.2-rc2
Plugin "rexray/rbd:0.9.2-rc2" is requesting the following privileges:
- network: [host]
- mount: [/dev]
- mount: [/etc/ceph]
- allow-all-devices: [true]
- capabilities: [CAP_SYS_ADMIN]
Do you grant the above permissions? [y/N] y
0.9.2-rc2: Pulling from rexray/rbd
f683ee9c713a: Download complete
Digest: sha256:27c63f276f21735c2bb2091fcca9cb9ac3992d76dec178ec422b63e0864d1c86
Status: Downloaded newer image for rexray/rbd:0.9.2-rc2
Installed plugin rexray/rbd:0.9.2-rc2
[dcosadmin@dcotest06 ~]$ sudo docker volume ls
DRIVER VOLUME NAME
local 8969e2cd1246282ab702ff4390ee94474306806fc5715109baf69f30335c4459
rexray/rbd:0.9.2-rc2 jenkins
[dcosadmin@dcotest06 ~]$
Summary
It took a while for me to prepare the core infrastructure to test external volume from marathon, shared storage, and installing DC/OS. By completing until this stage, it is ready to enjoy new world of microservice. Next tutorial will be installing Jenkins service using external volume which we create from this tutorial.
- Get link
- X
- Other Apps
Comments
Post a Comment