IOMesh as External Storage

In addition to providing storage within its Kubernetes environment where IOMesh is located, IOMesh can also offer storage externally to a Kubernetes cluster through CSI, or function as an iSCSI target for an iSCSI client.

Configure iSCSI Access Point

To use IOMesh storage via the CSI driver or iSCSI target, it is necessary to configure an iSCSI access point first.

To ensure that the access point has a consistent and functional IP address, IOMesh employs iomesh-access service with a service type of LoadBalancer. This service must have an IP externally exposed, and the specific method for configuring this external IP will vary depending on the cloud environment in which IOMesh is deployed.

For IOMesh deployed in LoadBalancer-supported cloud environments, see In-Tree LoadBalancer.
For IOMesh deployed on bare metal or other LoadBalancer-not-supported cloud environments, see Out-of-Tree LoadBalancer.

In-Tree LoadBalancer

Out-of-Tree LoadBalancer

If IOMesh is deployed in a cloud environment that supports LoadBalancer, Kubernetes will automatically call the API of the cloud provider and assign an external IP to iomesh-access service.

Check the status of iomesh-access service, ensuring it has been assigned an external IP.

kubectl get service iomesh-access -n iomesh-system

NAME            TYPE           CLUSTER-IP     EXTERNAL-IP     PORT(S)                                                          AGE
iomesh-access   LoadBalancer   10.96.22.212   192.168.2.1     3260:32012/TCP,10206:31920/TCP,10201:31402/TCP,10207:31802/TCP   45h

Set spec.redirector.iscsiVirtualIP to the external IP of iomesh-access service by running the following command. Once edited, the iomesh-iscsi-redirector pod will automatically restart to make the modification take effect.
```
kubectl edit iomesh iomesh -n iomesh-system
```
NOTE: spec.redirector.iscsiVirtualIP must be the same as the external IP. If the external IP is changed, update spec.redirector.iscsiVirtualIP.
An optional step. To use storage from one more IOMesh cluster, run the kubectl edit command to set the spec.redirector.iscsiVirtualIP field in its YAML file to the external IP of the iomesh-access service.

In case IOMesh is deployed on bare metal or any cloud environment that does not support Kubernetes LoadBalancer, you will need to install MetallLB as the default LocalBalancer in the Kubernetes cluster.

Verify that the Kubernetes cluster meets the MetalLB installation requirements.

Install MetalLB.

helm repo add metallb https://metallb.github.io/metallb
helm install metallb metallb/metallb --version 0.12.1 -n iomesh-system

Create a MetalLB YAML configMap metallb-config.yaml.

Set protocol to layer2. Fill in an IP pool in addresses. The IP pool must be an IP range like "192.168.1.100-192.168.1.110" or the CIDR like "192.168.2.0/24".
```
apiVersion: v1
kind: ConfigMap
metadata:
  namespace: iomesh-system
  name: metallb
data:
  config: |
    address-pools:
    - name: iomesh-access-address-pools
      protocol: layer2
      addresses:
      - <fill-in-your-ip-address-pool-here> # Fill in an IP pool.
```
NOTE: metallb 0.12.1 does not support binding address-pools to a specific NIC. To use the NIC binding feature, metallb must be version 0.13.0 or higher. Refer to https://metallb.universe.tf/concepts/layer2/ for configuration.

NOTE: metallb 0.12.1 does not support Kubernetes 1.25 or above. For Kubernetes 1.25 or above, refer to https://metallb.universe.tf to deploy metallb 0.13.0 or higher.

NOTE: If you have updated this ConfigMap, you will need to restart all metallb pods in order for the configuration to take effect.
Apply the YAML config.
```
kubectl apply -f metallb-config.yaml
```

Check the status of iomesh-access service, ensuring an external IP from the IP pool has been assigned to MetalLB.

watch kubectl get service iomesh-access -n iomesh-system

NAME            TYPE           CLUSTER-IP     EXTERNAL-IP     PORT(S)                                                          AGE
iomesh-access   LoadBalancer   10.96.22.212   192.168.2.1     3260:32012/TCP,10206:31920/TCP,10201:31402/TCP,10207:31802/TCP   45h

Set spec.redirector.iscsiVirtualIP to the external IP by running the following command. Once the changes are saved, the iomesh-iscsi-redirector pod will automatically restart to make the modification take effect.
```
kubectl edit iomesh iomesh -n iomesh-system
```
NOTE: spec.redirector.iscsiVirtualIP should be the same as the external IP. If the external IP is changed, update spec.redirector.iscsiVirtualIP.
An optional step. To use storage from one more IOMesh cluster, run kubectl edit command to set the spec.redirector.iscsiVirtualIP field in its YAML file to the external IP of the iomesh-access service.

External Storage vis CSI

After the access point is configured, IOMesh can provide storage externally to a Kubernetes cluster. However, before doing this, it is necessary to install IOMesh CSI Driver in the cluster using the instructions provided below. Note that the installation should be online.

NOTE: To use this function, the external Kubernetes cluster should be able to access IOMesh DATA_CIDR that was configured in Prerequisites.

Set up open-iscsi on the worker nodes of the external Kubernetes cluster.

Add the Helm chart repository.

helm repo add iomesh http://iomesh.com/charts

Export the IOMesh CSI Driver configuration file csi-driver.yaml.
```
helm show values iomesh/csi-driver > csi-driver.yaml
```

Edit the following fields in csi-driver.yaml.

 fullnameOverride: "iomesh-csi-driver"
 # ...
 # Specify the container platform, either "kubernetes" or "openshift". 
 co: "kubernetes" 
 # Specify the container platform version, which should be the same as the version of the cluster hosting IOMesh.
 coVersion: "1.18"
 # ...
 storageClass:
   nameOverride: "iomesh-csi-driver"
   parameters:
     csi.storage.k8s.io/fstype: "ext4"
     replicaFactor: "2"
     thinProvision: "true"
 # ...
 driver:
   # ...
   # The Kubernetes cluster ID.
   clusterID: "my-iomesh"
   # IOMesh meta server iSCSI portal address.
   metaAddr: "iomesh-cluster-vip:10206"
   # IOMesh chunk server iSCSI portal address.
   iscsiPortal: "iomesh-cluster-vip:3260"
   # Access IOMesh as external storage.
   deploymentMode: "EXTERNAL"
   # The unique CSI driver name in the Kubernetes cluster.
   nameOverride: "com.iomesh.csi-driver"
   # ...
   controller:
     driver:
       podDeletePolicy: "no-delete-pod"
   node:
     driver:
       # ...
       # The root directory of `kubelet` service.
       kubeletRootDir: "/var/lib/kubelet"
   # ...

Field	Value	Description
`fullnameOverride`	`"iomesh-csi-driver"`	The CSI driver name.
`co`	`"kubernetes"`	The container platform. If your container platform is OpenShift, type `"openshift"`.
`coVersion`	`"1.18"`	The container platform version, which should be the same as the version of the cluster hosting IOMesh.
`storageClass.nameOverride`	`"iomesh-csi-driver"`	The default StorageClass name, which is customizable during installation.
`storageClass.parameters`	`"parameters"`	The parameters for the default StorageClass, which is customizable during installation.
`driver.clusterID`	`"my-iomesh"`	The ID of the Kubernetes cluster, which is used to identify the cluster when IOMesh provides storage for multiple Kubernetes clusters.
`driver.metaAddr`	`"iomesh-cluster-vip:10206"`	The iSCSI portal address of the IOMesh meta server.
`driver.iscsiPortal`	`"iomesh-cluster-vip:3260"`	The iSCSI portal address of the IOMesh chunk server.
`driver.deploymentMode`	`"EXTERNAL"`	`EXTERNAL` means accessing IOMesh as external storage.
`driver.controller.driver.podDeletePolicy`	`"no-delete-pod"`(default) `"delete-deployment-pod"` `"delete-statefulset-pod"` `"delete-both-statefulset-and-deployment-pod"`	When creating a PVC using IOMesh CSI Driver, it will be bound to a pod. This field allows you to decide whether the pod should be automatically deleted and rebuilt on another healthy worker node if its original worker node has an issue.
`driver.node.driver.kubeletRootDir`	`"/var/lib/kubelet"`	The root directory for `kubelet` service to manage pod-mounted volumes. Default value is `/var/lib/kubelet`.

Deploy IOMesh CSI Driver.

helm install csi-driver iomesh/csi-driver \
    --namespace iomesh-system \
    --create-namespace \
    --values csi-driver.yaml \
    --wait

Verify that IOMesh CSI Driver has been successfully installed.

If all pods are shown in Running state, then IOMesh CSI Driver has been successfully installed.

watch kubectl get --namespace iomesh-system pods

NAME                                            READY   STATUS    RESTARTS   AGE
iomesh-csi-driver-controller-plugin-5dbfb48d5c-2sk97   6/6     Running   0          42s
iomesh-csi-driver-controller-plugin-5dbfb48d5c-cfhwt   6/6     Running   0          42s
iomesh-csi-driver-controller-plugin-5dbfb48d5c-drl7s   6/6     Running   0          42s
iomesh-csi-driver-node-plugin-25585                    3/3     Running   0          39s
iomesh-csi-driver-node-plugin-fscsp                    3/3     Running   0          30s
iomesh-csi-driver-node-plugin-g4c4v                    3/3     Running   0          39s

Once IOMesh CSI Driver is installed, you may refer to Volume Operations and VolumeSnapshot Operations for storage operations.

IOMesh as iSCSI Target

IOMesh provides support for the external iSCSI access service. You can create an iSCSI LUN by creating a PVC that can be accessed via any iSCSI client such as open-iscsi located beyond the Kubernetes cluster.

NOTE: To use this function, the iSCSI client should be able to access IOMesh DATA_CIDR that was configured in Prerequisites.

Create iSCSI LUN using PVC

Create a PVC as an external iSCSI LUN. You may obtain the iSCSI client IQNs by using the command cat /etc/iscsi/initiatorname.iscsi.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: external-iscsi
  annotations:
    # Mark this PVC as an iSCSI LUN for external use.
    iomesh.com/external-use: "true"
    # Set `initiator iqn acl` for iSCSI LUN. If left unspecified, all initiators will be prohibited from accessing this PVC.
    # If `accessModes` is `RWO`, you can only set 1 value in this field.
    # If `accessModes` is `RWX`, you can set multiple values in this field and separate them with the comma (,).
    # To allow all IQNs to access this PVC, set the value to "*/*".
    iomesh.com/iscsi-lun-iqn-allow-list: "iqn.1994-05.com.example:a6c97f775dcb"
spec:
  storageClassName: iomesh-csi-driver
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi

NOTE: You can also set the field iomesh.com/iscsi-lun-iqn-allow-list after the PVC is created.

Once the PVC transitions to the Bound state, run the following command to view the field spec.volumeAttributes.iscsiEntrypoint.

kubectl get pv pvc-d84b4657-7ab5-4212-9270-ce40e6a1356a -o jsonpath='{.spec.csi.volumeAttributes.iscsiEndpoint}'

iscsi://cluster-loadbalancer-ip:3260/iqn.2016-02.com.smartx:system:54e7022b-2dcc-4b43-800c-e52b6fad07d3/1
# The IP of iSCSI Portal is the cluster LoadBalancer IP, and port number is 3260.
# The target is `iqn.2016-02.com.smartx:system:54e7022b-2dcc-4b43-800c-e52b6fad07d3`.
# LUN ID is 1.

With the above information, you can use any iSCSI client to access the LUN. For example, if cluster-loadbalancer-ip is 192.168.25.101 and the iSCSI client is open-iscsi, run the following command:

iscsiadm -m discovery -t sendtargets  -p 192.168.25.101:3260 --discover
iscsiadm -m node -T iqn.2016-02.com.smartx:system:54e7022b-2dcc-4b43-800c-e52b6fad07d3 -p 192.168.25.101:3260  --login

Delete PVC

To delete a PVC for an iSCSI LUN, ensure that the iomesh.com/iscsi-lun-iqn-allow-list field in the PVC is blank or is already deleted. Whether the external iSCSI LUN is retained after deleting the PVC depends on the reclaimPolicy field in the StorageClass of the PVC.

Documentation

next