Raspberry Pi K3s NFS FS

June 28, 2020

3 分钟

NFS on Raspberry Pi

目前正学习如何在树莓派4集群上结合gromacs+mpich的分子动力学模拟，所以需要搭建一款分布式存储系统，又由于树莓派性能的限制，搭建OpenEBS会比较浪费计算资源，最后就采用轻量级的NFS来完成文件系统的共享和存储。

NFS on K3s aarch64

基础依赖

NFS Server:

sudo apt install nfs-kernel-server

添加共享文件目录到/etc/exports:

/mnt/data/nfs	192.168.1.0/24(rw,sync,no_root_squash,no_subtree_check) # 外部使用
/mnt/data/kubedata 192.168.1.0/24(rw,sync,no_root_squash,no_subtree_check) # k3s 使用

启动 NFS Server:

sudo systemctl restart nfs-kernel-server

NFS Client:

sudo apt install nfs-common

sudo mount 192.168.1.145:/mnt/data/nfs ./nfs

添加到/etc/fstab:

192.168.1.145:/mnt/data/nfs /home/chenfeng/nfs nfs auto,nofail,noatime,nolock 0 0

K3s NFS Volume

需要使用Kubernets提供的NFS Client Provisioner

k3s是跑在Ubuntu 20.04 aarch64 系统上的，而官方提供的NFS Client Provisioner是基于ARM v7的，所以需要重新编译Provisioner。

以下的操作全部在Raspberry pi 4上完成的。

以下为PATH:

diff -ur ./Makefile /tmp/nfs-client/Makefile
--- ./Makefile	2020-06-28 07:47:43.883181030 +0000
+++ /tmp/nfs-client/Makefile	2020-06-28 10:06:00.588586966 +0000
@@ -28,17 +28,16 @@
 container: build image build_arm image_arm

 build:
-	CGO_ENABLED=0 GOOS=linux go build -a -ldflags '-extldflags "-static"' -o docker/x86_64/nfs-client-provisioner ./cmd/nfs-client-provisioner
+	go build -a -ldflags '-extldflags "-static"' -o docker/x86_64/nfs-client-provisioner ./cmd/nfs-client-provisioner

 build_arm:
-	CGO_ENABLED=0 GOOS=linux GOARCH=arm GOARM=7 go build -a -ldflags '-extldflags "-static"' -o docker/arm/nfs-client-provisioner ./cmd/nfs-client-provisioner
+	go build -a -ldflags '-extldflags "-static"' -o docker/arm/nfs-client-provisioner ./cmd/nfs-client-provisioner

 image:
 	docker build -t $(MUTABLE_IMAGE) docker/x86_64
 	docker tag $(MUTABLE_IMAGE) $(IMAGE)

 image_arm:
-	docker run --rm --privileged multiarch/qemu-user-static:register --reset
 	docker build -t $(MUTABLE_IMAGE_ARM) docker/arm
 	docker tag $(MUTABLE_IMAGE_ARM) $(IMAGE_ARM)

Only in /tmp/nfs-client/deploy: .deployment-arm.yaml.swp
diff -ur ./deploy/deployment-arm.yaml /tmp/nfs-client/deploy/deployment-arm.yaml
--- ./deploy/deployment-arm.yaml	2020-06-28 09:24:48.499572298 +0000
+++ /tmp/nfs-client/deploy/deployment-arm.yaml	2020-06-28 10:06:11.876117004 +0000
@@ -4,7 +4,8 @@
   name: nfs-client-provisioner
   labels:
     app: nfs-client-provisioner
-  namespace: nfs
+  # replace with namespace where provisioner is deployed
+  namespace: default
 spec:
   replicas: 1
   strategy:
@@ -20,7 +21,7 @@
       serviceAccountName: nfs-client-provisioner
       containers:
         - name: nfs-client-provisioner
-          image: 192.168.1.114:5000/nfs-client-provisioner-arm
+          image: quay.io/external_storage/nfs-client-provisioner-arm:latest
           volumeMounts:
             - name: nfs-client-root
               mountPath: /persistentvolumes
@@ -28,11 +29,11 @@
             - name: PROVISIONER_NAME
               value: fuseim.pri/ifs
             - name: NFS_SERVER
-              value: 192.168.1.145
+              value: 10.10.10.60
             - name: NFS_PATH
-              value: /mnt/data/kubedata
+              value: /ifs/kubernetes
       volumes:
         - name: nfs-client-root
           nfs:
-            server: 192.168.1.145
-            path: /mnt/data/kubedata
+            server: 10.10.10.60
+            path: /ifs/kubernetes
diff -ur ./deploy/test-pod.yaml /tmp/nfs-client/deploy/test-pod.yaml
--- ./deploy/test-pod.yaml	2020-06-28 09:36:06.872994438 +0000
+++ /tmp/nfs-client/deploy/test-pod.yaml	2020-06-28 10:06:11.920115180 +0000
@@ -5,7 +5,7 @@
 spec:
   containers:
   - name: test-pod
-    image: 192.168.1.114:5000/ubuntu:20.04
+    image: gcr.io/google_containers/busybox:1.24
     command:
       - "/bin/sh"
     args:
diff -ur ./docker/arm/Dockerfile /tmp/nfs-client/docker/arm/Dockerfile
--- ./docker/arm/Dockerfile	2020-06-28 07:47:43.759177680 +0000
+++ /tmp/nfs-client/docker/arm/Dockerfile	2020-06-28 10:06:00.628585293 +0000
@@ -12,7 +12,6 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.

-FROM hypriot/rpi-alpine:3.6
-RUN apk update --no-cache && apk add ca-certificates
+FROM ubuntu:20.04
 COPY nfs-client-provisioner /nfs-client-provisioner
 ENTRYPOINT ["/nfs-client-provisioner"]

nfs-client/cmd/nfs-client-provisioner为Provisioner源码目录。

在调用make build_arm之前，我们需要安装golang，并且将编译需要的依赖从仓库external-storage/vendor/拷贝到$GOPATH/src中，编译完成的二进制文件位于nfs-client/docker/arm/nfs-client-provisioner。

构建打包的image直接修改为ubuntu:20.04，运行make image_arm。

根据上述PATCH修改完对应的文件之后，执行:

kubectl apply -f rbac.yaml
kubectl apply -f deployment-arm.yaml
kubectl apply -f class.yaml

最后测试test pod

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: test-claim
  annotations:
    volume.beta.kubernetes.io/storage-class: "managed-nfs-storage"
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 1Mi

kubectl  get pvc

NAME         STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS          AGE
test-claim   Bound    pvc-dff7c47d-91ae-483e-9d95-d33074ec9e0b   1Mi        RWX            managed-nfs-storage   48m

kind: Pod
apiVersion: v1
metadata:
  name: test-pod
spec:
  containers:
  - name: test-pod
    image: 192.168.1.114:5000/ubuntu:20.04
    command:
      - "/bin/sh"
    args:
      - "-c"
      - "touch /mnt/SUCCESS && exit 0 || exit 1"
    volumeMounts:
      - name: nfs-pvc
        mountPath: "/mnt"
  restartPolicy: "Never"
  volumes:
    - name: nfs-pvc
      persistentVolumeClaim:
        claimName: test-claim

如果PVC test-claim是Bound状态，并且test-pod运行结束之后，会在对应的NFS目录写入SUCCESS文件，则k3s NFS 搭建完成。

Sculpting in time