使用 Gluster FS 做 Kubernetes 持久化存储

安装 Gluster FS

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
# 先安装 gluster 源
$ yum install centos-release-gluster -y

# 安装 glusterfs 组件
$ yum install -y glusterfs glusterfs-server glusterfs-fuse glusterfs-rdma glusterfs-geo-replication glusterfs-devel

# 创建 glusterfs 目录
$ mkdir /opt/glusterd

# 修改 glusterd 目录
$ sed -i 's/var\/lib/opt/g' /etc/glusterfs/glusterd.vol

# 启动 glusterfs
$ systemctl start glusterd.service

# 设置开机启动
$ systemctl enable glusterd.service

#查看状态
$ systemctl status glusterd.service

配置 Gluster FS

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
# 配置 hosts
$ vi /etc/hosts
172.31.21.208 gfs01.isoftone.com
172.31.21.209 gfs02.isoftone.com
172.31.21.210 gfs03.isoftone.com
172.31.21.211 gfs04.isoftone.com

# 开放端口
$ iptables -I INPUT -p tcp --dport 24007 -j ACCEPT

# 创建存储目录
$ mkdir /opt/gfs_data

# 添加节点到 集群
# 执行操作的本机不需要probe 本机
[root@sz-pg-oam-docker-test-001 ~]#
gluster peer probe gfs02.isoftone.com
gluster peer probe gfs03.isoftone.com
gluster peer probe gfs04.isoftone.com

# 查看集群状态
$ gluster peer status
Number of Peers: 3

Hostname: gfs02.isoftone.com
Uuid: 0697cfd7-4c88-4d29-b907-59894b158df7
State: Peer in Cluster (Connected)

Hostname: gfs03.isoftone.com
Uuid: 7754d301-00ac-4c28-ac22-4d5f89d2d1d3
State: Peer in Cluster (Connected)

Hostname: gfs04.isoftone.com
Uuid: 7b33777c-0d3b-45f9-acbd-4f8d09da6431
State: Peer in Cluster (Connected)

配置 Gluster FS Volume

GlusterFS中的volume的模式有很多中,包括以下几种:

  • 分布卷(默认模式):即DHT, 也叫 分布卷: 将文件已hash算法随机分布到 一台服务器节点中存储。
  • 复制模式:即AFR, 创建volume 时带 replica x 数量: 将文件复制到 replica x 个节点中。
  • 条带模式:即Striped, 创建volume 时带 stripe x 数量: 将文件切割成数据块,分别存储到 stripe x 个节点中 ( 类似raid 0 )。
  • 分布式条带模式:最少需要4台服务器才能创建。 创建volume 时 stripe 2 server = 4 个节点: 是DHT 与 Striped 的组合型。
  • 分布式复制模式:最少需要4台服务器才能创建。 创建volume 时 replica 2 server = 4 个节点:是DHT 与 AFR 的组合型。
  • 条带复制卷模式:最少需要4台服务器才能创建。 创建volume 时 stripe 2 replica 2 server = 4 个节点: 是 Striped 与 AFR 的组合型。
  • 三种模式混合: 至少需要8台 服务器才能创建。 stripe 2 replica 2 , 每4个节点 组成一个 组。

这几种模式的示例图参考:CentOS7安装GlusterFS

因为我们只有四台主机,在此我们使用默认的分布式复制模式

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
# 创建分布卷
$ gluster volume create k8s-volume replica 2 transport tcp gfs01.isoftone.com:/opt/gfs_data gfs02.isoftone.com:/opt/gfs_data gfs03.isoftone.com:/opt/gfs_data gfs04.isoftone.com:/opt/gfs_data force

# 查看volume状态
$ gluster volume info
Volume Name: k8s-volume
Type: Distributed-Replicate
Volume ID: f5438a2b-22b3-4608-9a45-ed9fa0028d38
Status: Created
Snapshot Count: 0
Number of Bricks: 2 x 2 = 4
Transport-type: tcp
Bricks:
Brick1: gfs01.isoftone.com:/opt/gfs_data
Brick2: gfs02.isoftone.com:/opt/gfs_data
Brick3: gfs03.isoftone.com:/opt/gfs_data
Brick4: gfs04.isoftone.com:/opt/gfs_data
Options Reconfigured:
transport.address-family: inet
nfs.disable: on
performance.client-io-threads: off

# 启动 k8s-volume 卷
$ gluster volume start k8s-volume

调优 Gluster FS

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
# 开启 指定 volume 的配额
$ gluster volume quota k8s-volume enable

# 限制 指定 volume 的配额
$ gluster volume quota k8s-volume limit-usage / 1TB

# 设置 cache 大小, 默认32MB
$ gluster volume set k8s-volume performance.cache-size 4GB

# 设置 io 线程, 太大会导致进程崩溃
$ gluster volume set k8s-volume performance.io-thread-count 16

# 设置 网络检测时间, 默认42s
$ gluster volume set k8s-volume network.ping-timeout 10

# 设置 写缓冲区的大小, 默认1M
$ gluster volume set k8s-volume performance.write-behind-window-size 1024MB

在 Kubernetes 使用 Gluster FS

安装 Gluster FS 客户端

1
2
3
4
5
6
7
8
9
# 在所有 k8s node 中安装 glusterfs 客户端
$ yum install -y glusterfs glusterfs-fuse

# 配置 hosts
$ vi /etc/hosts
172.31.21.208 gfs01.isoftone.com
172.31.21.209 gfs02.isoftone.com
172.31.21.210 gfs03.isoftone.com
172.31.21.211 gfs04.isoftone.com

创建 endpoints

1
2
3
4
5
6
7
$ curl -O https://raw.githubusercontent.com/batizhao/dockerfile/master/k8s/glusterfs/glusterfs-endpoints.yaml

# 导入 glusterfs-pod.json
$ kubectl apply -f glusterfs-endpoints.yaml
$ kubectl get ep
NAME ENDPOINTS AGE
glusterfs-cluster 172.31.21.208:1,172.31.21.209:1,172.31.21.210:1 + 1 more... 59m

创建 PersistentVolume

1
2
3
4
5
$ curl -O https://raw.githubusercontent.com/batizhao/dockerfile/master/k8s/glusterfs/glusterfs-pv.yaml

$ kubectl apply -f glusterfs-pv.yaml
$ kubectl get pv | grep glus
glusterfs-pv 8Gi RWX Retain Bound default/glusterfs-pvc 55m

创建 PersistentVolumeClaim

1
2
3
4
5
$ curl -O https://raw.githubusercontent.com/batizhao/dockerfile/master/k8s/glusterfs/glusterfs-pvc.yaml

$ kubectl apply -f glusterfs-pvc.yaml
$ kubectl get pvc | grep glus
glusterfs-pvc Bound glusterfs-pv 8Gi RWX 56m

创建测试 nginx 挂载 volume

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
$ curl -O https://raw.githubusercontent.com/batizhao/dockerfile/master/k8s/glusterfs/nginx-deployment.yaml

$ kubectl apply -f nginx-deployment.yaml
$ kubectl get deploy,pod
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
deploy/nginx-dm 2 2 2 2 54m

NAME READY STATUS RESTARTS AGE
po/nginx-dm-5d49ccb7f5-h62dc 1/1 Running 0 54m
po/nginx-dm-5d49ccb7f5-szsfg 1/1 Running 0 54m

# 登陆 k8s node 物理机,使用 df 可查看挂载目录
$ df -h | grep glusterfs
172.31.21.208:k8s-volume 560G 8.7G 552G 2% /var/lib/kubelet/pods/c403020c-0ca8-11e8-ace7-000c29c217e5/volumes/kubernetes.io~glusterfs/glusterfs-pv

# 查看挂载
$ kubectl exec -it nginx-dm-5d49ccb7f5-h62dc -- df -h|grep k8s-volume
172.31.21.208:k8s-volume 560G 8.7G 552G 2% /usr/share/nginx/html

# 创建文件 测试
$ kubectl exec -it nginx-dm-5d49ccb7f5-h62dc -- touch /usr/share/nginx/html/index.html

$ kubectl exec -it nginx-dm-5d49ccb7f5-h62dc -- ls -lt /usr/share/nginx/html/index.html
-rw-r--r-- 1 root root 0 Feb 8 09:24 /usr/share/nginx/html/index.html

# 验证 glusterfs
# 因为我们使用分布复制卷 replica 2,所以可以看到有 2 个节点中有文件
[root@glusterfs-01 gfs_data]# ls
index.html
[root@glusterfs-02 gfs_data]# ls
index.html
[root@glusterfs-03 gfs_data]# ls
[root@glusterfs-04 gfs_data]# ls

安装 Heketi

GlusterFS 是个开源的分布式文件系统,而 Heketi 在其上提供了 REST 形式的 API,二者协同为 Kubernetes 提供了存储卷的自动供给能力。

  • Heketi 服务器:172.31.21.208
  • Gluster 服务器:
    • 172.31.21.208
    • 172.31.21.209
    • 172.31.21.210
    • 172.31.21.211
1
2
3
4
5
6
7
# 安装
$ yum -y install epel-release
$ yum -y install heketi heketi-client

# 管理证书
$ ssh-keygen -f /etc/heketi/heketi_key -t rsa -N ''
$ chown heketi:heketi /etc/heketi/heketi_key*

制作完成后会在当前目录下生成 heketi_key、heketi_key.pub。接下来,修改 /etc/heketi/heketi.json 中的 keyfile 指向生成的 key。

1
2
3
4
# 将公钥 heketi_key.pub 拷贝到所有 glusterfs 节点上 /etc/heketi/keketi_key.pub
$ ssh-copy-id -i /etc/heketi/heketi_key.pub root@172.31.21.209
$ ssh-copy-id -i /etc/heketi/heketi_key.pub root@172.31.21.210
$ ssh-copy-id -i /etc/heketi/heketi_key.pub root@172.31.21.211
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
$ cat /etc/heketi/heketi.json
{
"_port_comment": "Heketi Server Port Number",
"port": "8080",

"_use_auth": "Enable JWT authorization. Please enable for deployment",
"use_auth": false,

"_jwt": "Private keys for access",
"jwt": {
"_admin": "Admin has access to all APIs",
"admin": {
"key": "123456"
},
"_user": "User only has access to /volumes endpoint",
"user": {
"key": "123456"
}
},

"_glusterfs_comment": "GlusterFS Configuration",
"glusterfs": {
"_executor_comment": [
"Execute plugin. Possible choices: mock, ssh",
"mock: This setting is used for testing and development.",
" It will not send commands to any node.",
"ssh: This setting will notify Heketi to ssh to the nodes.",
" It will need the values in sshexec to be configured.",
"kubernetes: Communicate with GlusterFS containers over",
" Kubernetes exec api."
],
"executor": "ssh",

"_sshexec_comment": "SSH username and private key file information",
"sshexec": {
"keyfile": "/etc/heketi/heketi_key",
"user": "root",
"port": "Optional: ssh port. Default is 22",
"fstab": "Optional: Specify fstab file on node. Default is /etc/fstab"
},

"_db_comment": "Database file name",
"db": "/var/lib/heketi/heketi.db",

"_loglevel_comment": [
"Set log level. Choices are:",
" none, critical, error, warning, info, debug",
"Default is warning"
],
"loglevel" : "debug"
}
}

# 启动
systemctl enable heketi && systemctl restart heketi && systemctl status heketi

# 测试
$ curl http://localhost:8080/hello
Hello from Heketi

参考