Replace Disk

The IOMesh Dashboard displays the health status of physical disks for easy monitoring. If any disk is indicated as Unhealthy, Failing, or S.M.A.R.T not passed, you should replace it with a new disk as soon as possible.

Procedure

1. Get the meta leader pod name.

   kubectl get pod -n iomesh-system -l=iomesh.com/meta-leader -o=jsonpath='{.items[0].metadata.name}'
   

   iomesh-meta-0
   

2. Access the meta leader pod.

   kubectl exec -it iomesh-meta-0 -n iomesh-system -c iomesh-meta bash
   

3. Run the following command multiple times to verify that there are no ongoing migration or recovery tasks in the cluster.

   Ensure that the output value is 0. If any field has a non-zero value, you should wait for it to reach 0.

   /opt/iomesh/iomeshctl summary cluster | egrep "recovers|migrates" 
   

   num_ongoing_recovers: 0
   num_pending_recovers: 0
   num_ongoing_migrates: 0
   num_pending_migrates: 0
     pending_migrates_bytes: 0
     pending_recovers_bytes: 0
         pending_migrates_bytes: 0
         pending_recovers_bytes: 0
         pending_migrates_bytes: 0
         pending_recovers_bytes: 0
         pending_migrates_bytes: 0
         pending_recovers_bytes: 0
     num_ongoing_recovers: 0
     num_pending_recovers: 0
     num_ongoing_migrates: 0
     num_pending_migrates: 0
       pending_migrates_bytes: 0
       pending_recovers_bytes: 0
   

4. View the disk that requires replacement. In the given example, let's assume that the disk blockdevice-66312cce9037ae891a099ad83f44d7c9 needs to be replaced.

   kubectl --namespace iomesh-system get bd -o wide
   

   NAME                                           NODENAME      PATH       FSTYPE   SIZE          CLAIMSTATE   STATUS   AGE
   blockdevice-41f0c2b60f5d63c677c3aca05c2981ef   qtest-k8s-0   /dev/sdc            53687091200   Unclaimed    Active   29h
   blockdevice-66312cce9037ae891a099ad83f44d7c9   qtest-k8s-1   /dev/sdc            69793218560   Claimed      Active   44h
   blockdevice-7aff82fe93fac5153b14af3c82d68856   qtest-k8s-2   /dev/sdb            69793218560   Claimed      Active   44h
   

5. Run the following command to edit the deviceMap of the disk. Add the disk name to the field exclude under devicemap.

   kubectl edit iomesh iomesh -n iomesh-system
   

       # ...
       deviceMap:
         # ...
         dataStore:
           selector:
             matchExpressions:
             - key: iomesh.com/bd-driverType
               operator: In
               values:
               - HDD
             matchLabels:
               iomesh.com/bd-deviceType: disk
           exclude:
           - blockdevice-66312cce9037ae891a099ad83f44d7c9
     # ...
   

6. Repeat Step 2 and 3 to verify that there are no ongoing migration or recovery tasks in the cluster.

7. Verify that the block device is in the Unclaimed state.

   kubectl get bd blockdevice-66312cce9037ae891a099ad83f44d7c9 -n iomesh-system
   

   NAME                                           NODENAME      PATH       FSTYPE   SIZE          CLAIMSTATE   STATUS   AGE
   blockdevice-66312cce9037ae891a099ad83f44d7c9   qtest-k8s-1   /dev/sdc            69793218560   Unclaimed     Active  44h
   

8. Unplug the disk. Then the disk will enter the Inactive state.

   Run the following commands simultaneously to remove the block device and its corresponding blockdeviceclaim.

   NOTE: It is normal to see a prompt indicating that bdc cannot be found when running the following commands to clear it.

   kubectl patch bdc/blockdevice-66312cce9037ae891a099ad83f44d7c9 -p '{"metadata":{"finalizers":[]}}' --type=merge -n iomesh-system
   kubectl patch bd/blockdevice-66312cce9037ae891a099ad83f44d7c9 -p '{"metadata":{"finalizers":[]}}' --type=merge -n iomesh-system
   kubectl delete bdc blockdevice-66312cce9037ae891a099ad83f44d7c9 -n iomesh-system
   kubectl delete bd blockdevice-66312cce9037ae891a099ad83f44d7c9 -n iomesh-system
   

9. Plug the new disk. Refer to Set Up IOMesh for mounting steps.