存储与镜像排查笔记合集
本文把若干零散的排查与操作记录合并在一起，每条都是当时环境下的实测结果；环境变化后命令可能失效，以官方文档为准。
迁移 Docker 存储到新的硬盘
1. 硬盘格式化
通常，第二块硬盘的名字会是 /dev/sdb 。
会有提示输入参数：
command (m for help):n
Partition number(1-4):1
First cylinder (1-22800,default 1):Enter
command (m for help):w
1
2
| mkdir /data
mount -t ext4 /dev/sdb /data
|
先找到设备的 UUID。
1
2
3
| blkid |grep /dev/sdb
/dev/sdb: UUID="328a9d32-abb6-492a-aabe-b6a63583674d" TYPE="ext4"
|
编辑 /etc/fstab 新增挂载项。
1
2
3
| vim /etc/fstab
UUID=328a9d32-abb6-492a-aabe-b6a63583674d /dev/sdb ext4 defaults 0 0
|
1.1 云服务器新增数据盘
云主机扩容后，新磁盘盘符可能是 /dev/vdc 等，操作步骤如下。
- 查找新磁盘盘符
- 分区
依次选择：
- n add a new partition
- p primary partition (1-4)，根据需要选择
- 格式化
- 挂载
1
2
| mkdir /local_dir
mount /dev/vdc /local_dir
|
2. 迁移 Docker 存储
1
| mv /var/lib/docker /data/
|
1
| ln -s /data/docker /var/lib/docker
|
使用 Alpine 镜像常见问题
1. DNS 请求超时
原因: alpine 使用的是 musl 库，在 DNS 解析上会有一些限制[1]
解决方式：
不使用 apline 镜像，并在容器 resolv.conf 文件中增加 options single-request-reopen 配置。因为 single-request-reopen 配置项只对 glibc 库生效，但是 apline 镜像使用的是 musl 库
2. Docker 下无法解析 hosts
原因:
alpine 没有 /etc/nsswitch.conf，导致依赖 hosts 的进程无法解析 hosts。
解决方式：
在 Dockerfile 中添加如下命令：
1
| RUN if [ ! -e /etc/nsswitch.conf ];then echo 'hosts: files dns' > /etc/nsswitch.conf; fi
|
3. 参考
- https://wiki.musl-libc.org/functional-differences-from-glibc.html#Name-Resolver/DNS
Fluid 挂载 S3 为 PVC 以及性能测试
1. 创建 Dataset
1
2
3
4
5
6
7
8
9
10
| kubectl apply -f - <<EOF
apiVersion: v1
kind: Secret
metadata:
name: my-s3
type: Opaque
stringData:
aws.accessKeyId: xxx
aws.secretKey: xxx
EOF
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
| kubectl apply -f - <<EOF
apiVersion: data.fluid.io/v1alpha1
kind: Dataset
metadata:
name: my-s3
spec:
mounts:
- mountPoint: s3://BUCKET/
name: s3
options:
alluxio.underfs.s3.endpoint: ks3-cn-beijing-internal.ksyun.com
alluxio.underfs.s3.disable.dns.buckets: "false"
encryptOptions:
- name: aws.accessKeyId
valueFrom:
secretKeyRef:
name: my-s3
key: aws.accessKeyId
- name: aws.secretKey
valueFrom:
secretKeyRef:
name: my-s3
key: aws.secretKey
accessModes:
- ReadWriteMany
EOF
|
2. 创建 Runtime
1
2
3
4
5
6
7
8
9
10
11
12
| kubectl apply -f - <<EOF
apiVersion: data.fluid.io/v1alpha1
kind: AlluxioRuntime
metadata:
name: my-s3
spec:
tieredstore:
levels:
- mediumtype: MEM
path: /dev/shm
quota: 50Gi
EOF
|
3. 创建测试 Pod
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
| kubectl apply -f - <<EOF
apiVersion: v1
kind: Pod
metadata:
name: s3-demo
spec:
containers:
- name: demo
image: shaowenchen/demo:ubuntu
volumeMounts:
- mountPath: /data
name: data
volumes:
- name: data
persistentVolumeClaim:
claimName: my-s3
EOF
|
4. 性能测试
1
2
3
4
5
6
7
8
9
| time dd if=/dev/zero of=./dd.txt bs=4M count=250
250+0 records in
250+0 records out
1048576000 bytes (1.0 GB, 1000 MiB) copied, 57.8283 s, 18.1 MB/s
real 1m14.210s
user 0m0.000s
sys 0m0.363s
|
1
2
3
4
5
| time cp ./dd.txt /dev/null
real 0m13.823s
user 0m0.000s
sys 0m0.386s
|
1
2
3
4
5
| time cp ./dd.txt /dev/null
real 0m0.226s
user 0m0.004s
sys 0m0.221s
|
5. 清理
1
2
3
4
| kubectl delete pod s3-demo
kubectl delete alluxioruntime my-s3
kubectl delete dataset my-s3
kubectl delete secret my-s3
|
6. 总结
使用 Fluid 直接将 S3 挂载为 PVC 性能不算好，以上的测试桶和集群在一个区域。
- 已经挂载的 PVC 并不能看到 S3 桶的更新。也就是说，PVC 只是 S3 挂载瞬间的 Snapshort
- 不要在 PVC 中创建目录，目录不会同步到 S3 桶
- 不要在 PVC 中创建文件，文件不会同步到 S3 桶
- 读写速度都很慢
使用 s3fs + ThinRuntimeProfile 的方式应该更好，但目前暂时没有业务需求，另外如果是 OSS 可以使用 JindoRuntime 支持对象存储的加速。
使用 Fluid 和 S3FS 对接 S3 存储及性能测试
本文使用的是 Fluid 1.0 版本，高版本的配置文件路径发生了变化，需要根据实际情况调整。
1. 制作镜像
1.1 fluid_config_init.py
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
| #!/usr/bin/env python
import json
import os
rawStr = ""
with open("/etc/fluid/config.json", "r") as f:
rawStr = f.readlines()
rawStr = rawStr[0]
script = """
#!/bin/sh
set -ex
MNT_TO=$targetPath
trap "umount ${MNT_TO}" SIGTERM
mkdir -p ${MNT_TO} || true
s3fs $bucket:$bucketPath $MNT_TO -o url=$url -o allow_other -f -d -o f2
echo "mounted and exit code: $?"
sleep inf
"""
obj = json.loads(rawStr)
with open("/root/.passwd-s3fs", "w") as f:
f.write("%s:%s\n" % (obj["mounts"][0]["options"]["s3-access-key"], obj["mounts"][0]["options"]["s3-access-secret"]))
# set /root/.passwd-s3fs to be read only by owner
os.chmod("/root/.passwd-s3fs", 0o600)
bucketPath = obj["mounts"][0]["mountPoint"].lstrip("s3://").rstrip("/")
bucket = bucketPath.split("/")[0]
bucketPath = bucketPath[len(bucket):]
with open("/mount-s3.sh", "w") as f:
f.write('targetPath="%s"\n' % obj["targetPath"])
f.write('bucket="%s"\n' % bucket)
f.write('bucketPath="%s"\n' % bucketPath)
f.write('url="%s"\n' % obj["mounts"][0]["options"]["url"])
f.write(script)
|
1.2 entrypoint.sh
1
2
3
4
5
6
| #!/usr/bin/env bash
set +x
python /fluid_config_init.py
chmod u+x /mount-s3.sh
bash /mount-s3.sh
|
1.3 Dockerfile
1
2
3
4
5
6
| FROM shaowenchen/runtime-ubuntu:20.04
RUN apt-get update && apt-get install -y python s3fs
COPY ./fluid_config_init.py /
COPY ./entrypoint.sh /usr/local/bin/
RUN chmod +x /usr/local/bin/entrypoint.sh
ENTRYPOINT []
|
1.4 构建镜像
1
2
| docker build . --network=host -f Dockerfile -t shaowenchen/demo:fluid-s3fs
docker push shaowenchen/demo:fluid-s3fs
|
1. 挂载 S3 存储
1
2
3
4
| export ENDPOINT=obs.ap-southeast-3.myhuaweicloud.com
export BUCKET=
export AK=
export SK=
|
1
2
3
4
5
6
7
8
9
10
11
12
13
| kubectl apply -f - <<EOF
apiVersion: data.fluid.io/v1alpha1
kind: Dataset
metadata:
name: mys3fs
spec:
mounts:
- mountPoint: s3://${BUCKET}/test1
options:
s3-access-key: ${AK}
s3-access-secret: ${SK}
url: https://${ENDPOINT}
EOF
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
| kubectl apply -f - <<EOF
apiVersion: data.fluid.io/v1alpha1
kind: ThinRuntimeProfile
metadata:
name: s3fs-profile
spec:
fileSystemType: s3fs
fuse:
image: shaowenchen/demo:fluid-s3fs
imageTag: latest
imagePullPolicy: Always
command:
- "/usr/local/bin/entrypoint.sh"
EOF
|
1
2
3
4
5
6
7
8
| kubectl apply -f - <<EOF
apiVersion: data.fluid.io/v1alpha1
kind: ThinRuntime
metadata:
name: mys3fs-37
spec:
profileName: s3fs-profile
EOF
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
| kubectl apply -f - <<EOF
apiVersion: v1
kind: Pod
metadata:
name: mys3fs
spec:
containers:
- name: mys3fs
image: shaowenchen/demo:ubuntu
volumeMounts:
- mountPath: /data
name: mys3fs
volumes:
- name: mys3fs
persistentVolumeClaim:
claimName: mys3fs
EOF
|
2. 性能测试
进入 Pod
1
| kubectl exec -it mys3fs -- bash
|
执行
1
| curl -sSL https://d.juicefs.com/install | sh -
|
安装 JuiceFS 客户端
1
2
3
4
5
6
7
8
9
10
11
12
13
| juicefs bench --block-size 4 --big-file-size 1024 /data
Benchmark finished!
BlockSize: 4.0 MiB, BigFileSize: 1.0 GiB, SmallFileSize: 128 KiB, SmallFileCount: 100, NumThreads: 1
+------------------+-----------------+----------------+
| ITEM | VALUE | COST |
+------------------+-----------------+----------------+
| Write big file | 194.80 MiB/s | 5.26 s/file |
| Read big file | 100.50 MiB/s | 10.19 s/file |
| Write small file | 3.6 files/s | 277.70 ms/file |
| Read small file | 51.1 files/s | 19.59 ms/file |
| Stat file | 26301.5 files/s | 0.04 ms/file |
+------------------+-----------------+----------------+
|
3. 清理资源
1
2
3
| kubectl delete pod mys3fs
kubectl delete thinruntime mys3fs
kubectl delete dataset mys3fs
|
使用 Fluid 对接 OBS 存储及性能测试
1. Jindo 挂载 OBS
1
2
3
4
| export ENDPOINT=obs.cn-north-4.myhuaweicloud.com
export BUCKET=
export AK=
export SK=
|
1
2
3
4
5
6
7
8
9
10
| kubectl apply -f - <<EOF
apiVersion: v1
kind: Secret
metadata:
name: myobssecret
type: Opaque
stringData:
fs.obs.accessKeyId: ${AK}
fs.obs.accessKeySecret: ${SK}
EOF
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
| kubectl apply -f - <<EOF
apiVersion: data.fluid.io/v1alpha1
kind: Dataset
metadata:
name: myobs-jindo
spec:
mounts:
- mountPoint: obs://${BUCKET}/test2/
options:
fs.obs.endpoint: ${ENDPOINT}
name: default
path: "/"
encryptOptions:
- name: fs.obs.accessKeyId
valueFrom:
secretKeyRef:
name: myobssecret
key: fs.obs.accessKeyId
- name: fs.obs.accessKeySecret
valueFrom:
secretKeyRef:
name: myobssecret
key: fs.obs.accessKeySecret
accessModes:
- ReadWriteMany
EOF
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
| kubectl apply -f - <<EOF
apiVersion: data.fluid.io/v1alpha1
kind: JindoRuntime
metadata:
name: myobs-jindo
spec:
replicas: 2
tieredstore:
levels:
- mediumtype: SSD
path: /cache
quota: 40960
fuse:
args:
- -oro
- -ometrics_port=0
- -okernel_cache
- -oattr_timeout=7200
- -oentry_timeout=7200
- -onegative_timeout=7200
- -opread
EOF
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
| kubectl apply -f - <<EOF
apiVersion: v1
kind: Pod
metadata:
name: myobs-jindo
spec:
containers:
- name: demo
image: shaowenchen/demo:ubuntu
volumeMounts:
- mountPath: /data
name: data
volumes:
- name: data
persistentVolumeClaim:
claimName: myobs-jindo
EOF
|
2. 性能测试
进入 Pod
1
| kubectl exec -it myobs-jindo -- bash
|
执行
1
| curl -sSL https://d.juicefs.com/install | sh -
|
安装 JuiceFS 客户端
1
2
3
4
5
6
7
8
9
10
11
12
13
| juicefs bench --block-size 4 --big-file-size 1024 /data
Benchmark finished!
BlockSize: 4.0 MiB, BigFileSize: 1.0 GiB, SmallFileSize: 128 KiB, SmallFileCount: 100, NumThreads: 1
+------------------+--------------+----------------+
| ITEM | VALUE | COST |
+------------------+--------------+----------------+
| Write big file | 389.70 MiB/s | 2.63 s/file |
| Read big file | 242.28 MiB/s | 4.23 s/file |
| Write small file | 6.4 files/s | 157.08 ms/file |
| Read small file | 15.5 files/s | 64.39 ms/file |
| Stat file | 34.9 files/s | 28.62 ms/file |
+------------------+--------------+----------------+
|
用来分发模型，性能基本足够。
3. 清理资源
1
2
3
4
| kubectl delete pod myobs-jindo
kubectl delete jindoruntime myobs-jindo
kubectl delete dataset myobs-jindo
kubectl delete secret myobssecret
|
4. 参考
在容器中使用 obsutil 命令问题
1. 容器的 ulimit 设置太小
报错信息:
1
2
3
4
5
6
7
| CheckpointDir: /root/.obsutil_checkpoint
OutputDir: /root/.obsutil_output
runtime: mlock of signal stack failed: 12
runtime: increase the mlock limit (ulimit -l) or
runtime: update your kernel to 5.3.15+, 5.4.2+, or 5.5+
fatal error: mlock failed
|
- 如果是 Docker 下，可以在启动时添加
ulimit 参数
1
| docker run -it --ulimit memlock=-1 ...
|
- 如果是 Kubernetes 下，需要以特权模式启动 Pod
1
2
3
| securityContext:
privileged: true
runAsUser: 0
|
在 yaml 配置中添加上面的片段，然后在启动脚本中添加 ulimit -c unlimited 修改 ulimit 。
2. 只读权限的凭证无法使用
报错信息:
1
| Error: open /root/.obsutilconfig: read-only file system
|
由于以只读权限挂载 .obsutilconfig 凭证到容器，而 obsutil 命令需要对凭证的写权限。
因此需要在容器启动时，读取并生成凭证。
1
2
| cat /root/.obsutilconfig1 > /root/.obsutilconfig
ulimit -c unlimited
|
Docker 20.10.6 拉取某些镜像报错 unlinkat
使用的是 CentOS 7，内核版本 3.10.0-327 。
1. 查看当前 Docker 的版本
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
| docker version
Client: Docker Engine - Community
Version: 20.10.6
API version: 1.41
Go version: go1.13.15
Git commit: 370c289
Built: Fri Apr 9 22:46:01 2021
OS/Arch: linux/amd64
Context: default
Experimental: true
Server: Docker Engine - Community
Engine:
Version: 20.10.6
API version: 1.41 (minimum version 1.12)
Go version: go1.13.15
Git commit: 8728dd2
Built: Fri Apr 9 22:44:13 2021
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.4.4
GitCommit: 05f951a3781f4f2c1911b05e61c160e9c30eaa8e
runc:
Version: 1.0.0-rc93
GitCommit: 12644e614e25b05da6fd08a38ffa0cfe1903fdec
docker-init:
Version: 0.19.0
GitCommit: de40ad0
|
能够拉取大部分镜像，但是部分镜像会报错，如下:
1
2
3
4
5
6
7
8
9
10
11
| docker pull k8s.gcr.io/kube-proxy:v1.16.12
v1.16.12: Pulling from kube-proxy
83b4483280e5: Pull complete
cedd2715c2e4: Pull complete
297e97c9c472: Extracting [==================================================>] 2.052MB/2.052MB
67b649411e75: Download complete
d97928a1765f: Download complete
ffa39a529ef3: Download complete
ed820abd805b: Download complete
failed to register layer: ApplyLayer exit status 1 stdout: stderr: unlinkat /var/log/apt: invalid argument
|
2. 降低 Docker 的版本
参考文档，安装指定版本的 Docker
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
| docker version
Client: Docker Engine - Community
Version: 20.10.6
API version: 1.40
Go version: go1.13.15
Git commit: 370c289
Built: Fri Apr 9 22:45:33 2021
OS/Arch: linux/amd64
Context: default
Experimental: true
Server: Docker Engine - Community
Engine:
Version: 19.03.8
API version: 1.40 (minimum version 1.12)
Go version: go1.12.17
Git commit: afacb8b
Built: Wed Mar 11 01:25:42 2020
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.4.4
GitCommit: 05f951a3781f4f2c1911b05e61c160e9c30eaa8e
runc:
Version: 1.0.0-rc93
GitCommit: 12644e614e25b05da6fd08a38ffa0cfe1903fdec
docker-init:
Version: 0.18.0
GitCommit: fec3683
|
使用 Fluid 对接 S3 存储及性能测试
1. Jindo 挂载 S3
1
2
3
4
| export ENDPOINT=obs.ap-southeast-3.myhuaweicloud.com
export BUCKET=
export AK=
export SK=
|
1
2
3
4
5
6
7
8
9
10
| kubectl apply -f - <<EOF
apiVersion: v1
kind: Secret
metadata:
name: mys3secret
type: Opaque
stringData:
fs.s3.accessKeyId: ${AK}
fs.s3.accessKeySecret: ${SK}
EOF
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
| kubectl apply -f - <<EOF
apiVersion: data.fluid.io/v1alpha1
kind: Dataset
metadata:
name: mys3-jindo
spec:
mounts:
- mountPoint: s3://${BUCKET}/test2/
options:
fs.s3.endpoint: ${ENDPOINT}
encryptOptions:
- name: fs.s3.accessKeyId
valueFrom:
secretKeyRef:
name: mys3secret
key: fs.s3.accessKeyId
- name: fs.s3.accessKeySecret
valueFrom:
secretKeyRef:
name: mys3secret
key: fs.s3.accessKeySecret
accessModes:
- ReadWriteMany
EOF
|
1
2
3
4
5
6
7
8
9
10
11
12
13
| kubectl apply -f - <<EOF
apiVersion: data.fluid.io/v1alpha1
kind: JindoRuntime
metadata:
name: mys3-jindo
spec:
replicas: 2
tieredstore:
levels:
- mediumtype: SSD
path: /cache
quota: 40960
EOF
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
| kubectl apply -f - <<EOF
apiVersion: v1
kind: Pod
metadata:
name: mys3-jindo
spec:
containers:
- name: demo
image: shaowenchen/demo:ubuntu
volumeMounts:
- mountPath: /data
name: data
volumes:
- name: data
persistentVolumeClaim:
claimName: mys3-jindo
EOF
|
2. 性能测试
进入 Pod
1
| kubectl exec -it mys3-jindo -- bash
|
执行
1
| curl -sSL https://d.juicefs.com/install | sh -
|
安装 JuiceFS 客户端
1
2
3
4
5
6
7
8
9
10
11
12
13
| juicefs bench --block-size 4 --big-file-size 1024 /data
Benchmark finished!
BlockSize: 4.0 MiB, BigFileSize: 1.0 GiB, SmallFileSize: 128 KiB, SmallFileCount: 100, NumThreads: 1
+------------------+--------------+----------------+
| ITEM | VALUE | COST |
+------------------+--------------+----------------+
| Write big file | 394.04 MiB/s | 2.60 s/file |
| Read big file | 304.28 MiB/s | 3.37 s/file |
| Write small file | 7.0 files/s | 143.11 ms/file |
| Read small file | 18.2 files/s | 55.07 ms/file |
| Stat file | 41.5 files/s | 24.09 ms/file |
+------------------+--------------+----------------+
|
3. 清理资源
1
2
3
| kubectl delete secret mys3secret
kubectl delete dataset mys3-jindo
kubectl delete jindoruntime mys3-jindo
|
部署 JuiceFS CSI Driver
1. 安装部署
1
| wget https://raw.githubusercontent.com/juicedata/juicefs-csi-driver/master/deploy/k8s.yaml
|
- 任意非 master 节点查看 kubelet 的安装目录
1
| ps -ef|grep kubelet |grep root-dir
|
如果有值，则需要执行替换；否则直接安装。
1
| sed 's@/var/lib/kubelet@{{KUBELET_DIR}}@g' k8s.yaml
|
1
| kubectl apply -f k8s.yaml
|
1
| kubectl -n kube-system get pods -l app.kubernetes.io/name=juicefs-csi-driver
|
2. 创建存储
1
2
3
4
5
6
7
8
9
10
11
12
13
14
| apiVersion: v1
metadata:
name: juicefs-secret
namespace: default
labels:
juicefs.com/validate-secret: "true"
kind: Secret
type: Opaque
stringData:
name: ${JUICEFS_NAME}
token: ${JUICEFS_TOKEN}
access-key: ${ACCESS_KEY}
secret-key: ${SECRET_KEY}
envs: '{"BASE_URL": "http://console.example.com/static"}'
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
| apiVersion: v1
kind: PersistentVolume
metadata:
name: juicefs-pv
namespace: default
labels:
juicefs-name: ten-pb-fs
spec:
capacity:
storage: 10Pi
volumeMode: Filesystem
accessModes:
- ReadWriteMany
persistentVolumeReclaimPolicy: Retain
csi:
driver: csi.juicefs.com
# volumeHandle 需要保证集群内唯一，因此一般直接用 PV 名即可
volumeHandle: juicefs-pv
fsType: juicefs
# 在先前的步骤中已经创建好文件系统认证信息（Secret），在这里引用
# 如果要在静态配置下使用不同的认证信息，甚至使用不同的 JuiceFS 文件系统，则需要创建不同的 Secret
nodePublishSecretRef:
name: juicefs-secret
namespace: default
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: juicefs-pvc
namespace: default
spec:
accessModes:
- ReadWriteMany
volumeMode: Filesystem
# 静态配置下必须指定 storageClassName 为空字符串
# 代表该 PV 不采用任何 StorageClass，而是直接使用 selector 所指定的 PV
storageClassName: ""
# 由于目前 JuiceFS CSI 驱动不支持给静态 PV 设置存储容量，此处 requests.storage 填写任意小于 PV capacity 的有效值即可
resources:
requests:
storage: 10Pi
selector:
matchLabels:
juicefs-name: ten-pb-fs
|
3. 参考
