准备工作
1.假设集群: 选一台作管理机
注意: ceph集群引用hostname,而非ip.
172.17.4.16 test16 #hostname必须是test16
172.17.4.17 test17 #hostname必须是test17
172.17.4.18 test18 #hostname必须是test18
172.17.4.19 test19 #hostname必须是test19, 选作管理机
172.17.4.20 test20 #hostname必须是test20
172.17.4.21 test21 #hostname必须是test21
172.17.4.22 test22 #hostname必须是test22
2.安装用户
注意: 建议不要使用"ceph",因为"apt install ceph"需要使用ceph. 用root最省事。
root@test19:~# ssh-keygen #创建安装用rsa,安装完记得删除
root@test19:~# less .ssh/id_rsa.pub
ssh-rsa 
# 在各台机执行,authorized_keys必须是600
mkdir /root/.ssh
echo -n 'ssh-rsa 
测试用户
#在test19无密ssh
for node in 16 17 18 19 20 21 22; do
ssh test$node echo 123;
done;
#确认yes
The authenticity of host 'test22 (172.17.4.22)' can't be established.
ECDSA key fingerprint is SHA256:6S9mp0m8bV15bKX9RugP2CflINUvxORw82yCwJqVyrE.
Are you sure you want to continue connecting (yes/no)? yes
完成后记得将/root/.ssh/id_rsa从test19移除并妥善保管,以后可以用来维护集群.
3.更新源
root@test19: for node in 16 17 18 19 20 21 22; do
ssh test$node \
wget -q -O- https://download.ceph.com/keys/release.asc | sudo apt-key add -;\
echo deb https://download.ceph.com/debian-jewel/ $(lsb_release -sc) main | sudo tee /etc/apt/sources.list.d/ceph.list;\
apt update && apt -y dist-upgrade && apt -y autoremove;
done
#apt使用-y默认Yes
4.安装ceph-deploy
root@test19: apt install ceph-deploy
注意: 16.04下这二个版本都有bug.
1.5.32在执行ceph-deploy mon create-initial命令后生成的ceph.client.admin.keyring文件是空的!
1.5.34则在重装时不会替换/var/lib/ceph/bootstrap-osd/ceph.keyring, 造成ceph-deploy osd activate失败.
root@test19:~# apt-cache policy ceph-deploy
ceph-deploy:
Installed: 1.5.34
Candidate: 1.5.34
Version table:
*** 1.5.34 999
999 https://download.ceph.com/debian-jewel xenial/main amd64 Packages
999 https://download.ceph.com/debian-jewel xenial/main i386 Packages
100 /var/lib/dpkg/status
1.5.32-0ubuntu1 500
500 http://hk.archive.ubuntu.com/ubuntu xenial/universe amd64 Packages
500 http://hk.archive.ubuntu.com/ubuntu xenial/universe i386 Packages
安装过程
参考: INSTALLATION (QUICK)（http://docs.ceph.com/docs/master/start/）
1.创建管理目录
root@test19:~# mkdir ceph-admin
root@test19:~# cd ceph-admin/
2.初始配置
root@test19:~/ceph-admin# ceph-deploy new test16 test17
drwxr-xr-x 2 root root 4096 Sep 14 10:28 ./
drwx------ 6 root root 4096 Sep 14 10:28 ../
-rw-r--r-- 1 root root 215 Sep 14 10:28 ceph.conf
-rw-r--r-- 1 root root 4080 Sep 14 10:28 ceph-deploy-ceph.log
-rw------- 1 root root 73 Sep 14 10:28 ceph.mon.keyring
root@test19:~/ceph-admin# vi ceph.conf
[global]
fsid = c96ee478-4df7-4100-9792-c078cf44b3c4
mon_initial_members = test16, test17
mon_host = 172.17.4.16,172.17.4.17
auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx
public_network = 172.17.4.0/24
cluster_network = 172.17.4.0/24
# 以下是针对ext4的bug
osd_max_object_name_len = 256
osd_max_object_namespace_len = 64
osd_check_max_object_name_len_on_startup = false
需要注意: jewel 10.2.1对EXT4格式有个文件名长度BUG!
3.安装ceph
root@test19:~/ceph-admin# for node in 16 17 18 19 20 21 22; do ceph-deploy install test$node; done;
4.创建初始mon结点
root@test19:~/ceph-admin# ceph-deploy mon create-initial
5.创建激活osd结点, 去掉16,17
root@test19:~/ceph-admin# for node in 18 19 20 21 22; do ssh test$node 'mkdir -R /data/ceph/osd; chown ceph:ceph -R /data/ceph; rm -rf /data/ceph/osd/*'; ceph-deploy osd prepare test$node:/data/ceph/osd; ceph-deploy osd activate test$node:/data/ceph/osd; done;
6.分发管理配置
root@test19:~/ceph-admin# for node in 16 17 18 19 20 21 22; do ceph-deploy admin test$node; ssh test$node 'chmod a+r /etc/ceph/ceph.client.admin.keyring'; done;
如果上述步骤需要重试,可以执行下述清理:
root@test19:~/ceph-admin# ceph-deploy purgedata test16 test17 test18 test19 test20 test21 test22 root@test19:~/ceph-admin# ceph-deploy forgetkeys root@test19:~/ceph-admin# for node in 16 17 18 19 20 21 22; do ssh test$node rm -rf /data/ceph/osd/*; done;
另外, 重试命令需要带上"--overwrite-conf"选项!
测试集群是否安装成功
root@test19:~/ceph-admin# ceph -s
root@test19:~/ceph-admin# ceph osd tree
root@test19:~/ceph-admin# ceph osd perf