Use Persistent Volumes and Persistent Volume Claims with Oracle Cloud Native Environment

Introduction

Because container-based applications are stateless by default, any file updates during the container lifetime are lost. However, if your application is stateful, the ability to persist any changes to the filesystem becomes relevant. It is this requirement that has led to Kubernetes supporting many types of volumes. Many volume types that you may have encountered already are ephemeral (they only exist during the pod lifetime). Kubernetes also supports persistent volumes, which persist any data stored beyond the pod lifetime. There are several Persistent Volume types supported by Kubernetes, and this tutorial will demonstrate one of the simplest - the local type, which stores data on devices stored locally on one of the cluster nodes.

This tutorial shows how to create and use Persistent Volumes and Persistent Volume Claims with Oracle Cloud Native Environment (Oracle CNE). Persistent Volumes and Persistent Volume Claims work together to provide persistence to any container-based applications deployed onto Oracle CNE. You will start by covering how to use Persistent Volumes initially, then cover the use of Persistent Volume Claims.

Objectives

In this tutorial, you will learn to:

Install the prerequisites for the oci provider
Use the oci provider to create and manage a Kubernetes cluster
The difference between a Persistent Volume (PV) and a Persistent Volume Claim (PVC)
How to use Persistent Volumes and Persistent Volume Claims

Prerequisites

Minimum of one Oracle Linux instance
Each system should have Oracle Linux installed and configured with:
- An Oracle user account (used during the installation) with sudo access
- Key-based SSH, also known as password-less SSH, between the hosts
OCI cluster creation requires access to the following resources in an Oracle Cloud Infrastructure tenancy:
- Virtual cloud network with four subnets
- Network load balancer
- Object Storage bucket with minimum 5 GiB available
- Compute Custom Image
- Compute Arm Shape for the control plane node
  - VM.Standard.A1.Flex with two OCPU and 12 memory
- Compute for each additional control plane and worker node
  - VM.Standard.E4.Flex with four OCPU and 64 memory

Configure Oracle CNE

Note: If running in your own tenancy, read the linux-virt-labs GitHub project README.md and complete the prerequisites before deploying the lab environment.

Open a terminal on the Luna Desktop.

Clone the linux-virt-labs GitHub project.

git clone https://github.com/oracle-devrel/linux-virt-labs.git

Change into the working directory.
```
cd linux-virt-labs/ocne2
```

Install the required collections.

ansible-galaxy collection install -r requirements.yml

Increase the Boot volume size.

cat << EOF | tee instances.yml > /dev/null
compute_instances:
  1:
    instance_name: "ocne"
    type: "server"
    boot_volume_size_in_gbs: 128
ocne_type: "oci"
install_ocne_rpm: true
create_ocne_oci_cluster: true
ocne_cluster_name: "mycluster"
num_cp_nodes: 1
num_wk_nodes: 3
update_all: true
EOF

Deploy the lab environment.
```
ansible-playbook create_instance.yml -e localhost_python_interpreter="/usr/bin/python3.6" -e "@instances.yml"
```
The free lab environment requires the extra variable local_python_interpreter, which sets ansible_python_interpreter for plays running on localhost. This variable is needed because the environment installs the RPM package for the Oracle Cloud Infrastructure SDK for Python, located under the python3.6 modules.
The default deployment shape uses the AMD CPU and Oracle Linux 8. To use an Intel CPU or Oracle Linux 9, add -e instance_shape="VM.Standard3.Flex" or -e os_version="9" to the deployment command.
Important: Wait for the playbook to run successfully and reach the pause task. At this stage of the playbook, the installation of the Oracle CNE is complete, and the instances are ready. Take note of the previous play, which prints the public and private IP addresses of the nodes it deploys and any other deployment information needed while running the lab.

Get the Cluster Nodes

Open a terminal and connect via SSH to the ocne instance
```
ssh oracle@<ip_address_of_instance>
```
Get a list of known clusters using the CLI.
```
ocne cluster list
```
Get the location of the kube configuration.
```
ocne cluster show -C mycluster
```
We use the -C to specify a specific cluster from the cluster list.

Set the KUBECONFIG environment variable.

export KUBECONFIG=$(ocne cluster show -C mycluster)

Wait for the cluster to stabilize and all pods to report in a running state.
```
watch kubectl get pods -A
```
Once all the pods show a STATUS of Running, type ctrl-c to exit the watch command.

Confirm the Storage Class Used on your Cluster

There are different storage classes available in different environments. The free lab environment uses Oracle Cloud Storage and has been pre-configured to work with oci-ccm (The Oracle Cloud Infrastructure Cloud Controller Manager ).

Confirm which StorageClass is available in the free lab environment.

kubectl get storageclass

Example Output:

[oracle@ocne ~]$ kubectl get storageclass
NAME               PROVISIONER                       RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
oci                oracle.com/oci                    Delete          Immediate              false                  8m
oci-bv (default)   blockvolume.csi.oraclecloud.com   Delete          WaitForFirstConsumer   true                   8m
oci-bv-encrypted   blockvolume.csi.oraclecloud.com   Delete          WaitForFirstConsumer   true                   8m

Note: If you are using your own Kubernetes cluster, it is essential to confirm what Storage Classes are available and then modify the value in your Persistent Volume/Persistent Volume Claim so they work. The free lab uses the default value of oci-bv.

Create a Persistent Volume

Enabling a container to persist its data between sessions is a two-stage process. Firstly, the disk resource has to be defined (the Persistent Volume), and then it needs to be assigned to the container (Persistent Volume Claim). This section will complete the first of these.

Create the Persistent Volume Claim YAML file.
```
cat << 'EOF' > ~/pv.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
  name: test-pv
spec:
  storageClassName: oci-bv
  capacity:
    storage: 1Gi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: "/tmp/test-pv"
EOF
```
The principal fields to note are:
- accessModes: the value used here (ReadWriteOnce) means the volume can be mounted as ReadWrite by only one Pod, whilst being mounted by any other Pods on the same node in Read mode only (for more information look here ).
- storage: 1Gi defines the size of the Persistent Volume. This example is 1Gb.
- The storageClassName variable is set to oci-bv
- hostPath defines the 'type' of Persistent Volume. The hostPath value used here will ONLY work on a single node, making it only suitable for testing (more information is available here ). The hostPath value allows the Persistent Volume to emulate networked storage. In this example, the hostPath variable configures the storage class to store the Persistent Volume data at /tmp/test-pv. (Note: If you use a different storage driver, you may need to use a different provisioner from hostPath.)
Deploy the Persistent Volume.
```
kubectl apply -f pv.yaml
```

Confirm the Persistent Volume deployed.

kubectl get pv

Example Output:

[oracle@ocne ~]$ kubectl get pv
NAME      CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM   STORAGECLASS   VOLUMEATTRIBUTESCLASS   REASON   AGE
test-pv   1Gi        RWO            Retain           Available           oci-bv         <unset>                          8s

(Optional) You can also view more details about the Persistent Volume.

kubectl describe pv/test-pv

Example Output:

Name:            test-pv
Labels:          <none>
Annotations:     <none>
Finalizers:      [kubernetes.io/pv-protection]
StorageClass:    oci-bv
Status:          Available
Claim:           
Reclaim Policy:  Retain
Access Modes:    RWO
VolumeMode:      Filesystem
Capacity:        1Gi
Node Affinity:   <none>
Message:         
Source:
    Type:          HostPath (bare host directory volume)
   Path:          /tmp/test-pv
    HostPathType:  
Events:            <none>

Interesting Information: This illustrates how the fields defined in the YAML file map to the Persistent Volume. Did you spot some unused fields, such as Node Affinity ? They are used to test functionality if your install has multiple Worker nodes, which is out of scope for this lab.

Create a Persistent Volume Claim

You cannot use a Persistent Volume in isolation. Instead, a Persistent Volume Claim must be defined and created. Next, you will complete the second of the two steps.

Create the Persistent Volume Claim YAML file.

The Persistent Volume Claim enables the deployed application to request physical storage.

cat << 'EOF' > ~/pvc.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: test-pvc
spec:
  storageClassName: oci-bv
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
EOF

Create the Persistent Volume Claim.
```
kubectl apply -f pvc.yaml
```
This command instructs Kubernetes to look for a Persistent Volume that matches the PVC claims requirements. Kubernetes looks to match the storageClass type requested and, if located, binds this claim to the identified volume.

Confirm the Persistent Volume Claim created.

kubectl get pvc/test-pvc

Example Output:

[oracle@ocne ~]$ kubectl get pvc/test-pvc
NAME       STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS   VOLUMEATTRIBUTESCLASS   AGE
test-pvc   Pending                                      oci-bv         <unset>                 16s

(Optional) Retrieve more detail related to the Persistent Volume Claim.

kubectl describe pvc/test-pvc

Example Output:

Name:          test-pvc
Namespace:     default
StorageClass:  oci-bv
Status:        Pending
Volume:        
Labels:        <none>
Annotations:   <none>
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      
Access Modes:  
VolumeMode:    Filesystem
Used By:       <none>
Events:
  Type    Reason                Age                   From                         Message
  ----    ------                ----                  ----                         -------
 Normal  WaitForFirstConsumer  11s (x22 over 5m22s)  persistentvolume-controller  waiting for first consumer to be created before binding

Next, you will test that this delivers persistence by creating a new deployment that uses this Persistent Volume Claim.

Create a Stateful Application Deployment

Next, you will create a deployment descriptor YAML file to deploy a stateful application using the Persistent Volume Claim.

Create a new deployment YAML file to deploy Nginx on Oracle CNE.

cat << 'EOF' > ~/pod.yaml
apiVersion: v1
kind: Pod
metadata:
  name: pvc-pod
spec:
  containers:
    - name: pvc-pod-container
      image: ghcr.io/oracle/oraclelinux9-nginx:1.20
      ports:
        - containerPort: 80
      volumeMounts:
        - mountPath: "/tmp/test-pv"
          name: data
  volumes:
    - name: data
      persistentVolumeClaim:
        claimName: test-pvc
EOF

Where:

The volumeMounts variable defines the path mounted inside the deployed Pod. This example mounts as tmp/test-pv
The volumes variable defines which Persistent Volume Claim the Pod deployment should use. This example uses test-pvc

Deploy the Pod.
```
kubectl apply -f pod.yaml
```
Confirm the Pod has deployed and is running.
```
kubectl get pod
```
Example Output:
```
NAME      READY   STATUS    RESTARTS   AGE
pvc-pod   1/1     Running   0          1m
```
Notice that the STATUS column confirms that the Pod deployed and should be running. The next stage is to test that the Persistent Volume is mounted and accessible.

(Optional) If the Pod deploys successfully, the Persistent Volume Claim should have claimed the Persistent Volume.

kubectl get pv

Example Output:

[oracle@ocne ~]$ kubectl get pv
NAME      CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM              STORAGECLASS   VOLUMEATTRIBUTESCLASS   REASON   AGE
test-pv   1Gi        RWO            Retain           Bound    default/test-pvc   oci-bv         <unset>                          67s

kubectl get pvc

Example Output:

[oracle@ocne ~]$ kubectl get pvc
NAME       STATUS   VOLUME    CAPACITY   ACCESS MODES   STORAGECLASS   VOLUMEATTRIBUTESCLASS   AGE
test-pvc   Bound    test-pv   1Gi        RWO            oci-bv         <unset>                 2m17s

Notice that the STATUS column in the Persistent Volume and the Persistent Volume Claim confirm they are Bound. Additionally, the CLAIM column for the Persistent Volume shows the Persistent Volume Claim (test-pvc) you created earlier has Bound to the Persistent Volume.

Log into the deployed Pod and test the Persistent Volume

These steps show how to get a session shell into the running container and test that writing data stores that data outside of the Pod. Then, you will delete the Pod and recreate it. If it works, the data will remain accessible to the freshly deployed Pod.

Get a shell inside the Pod.
```
kubectl exec -it pod/pvc-pod -- /bin/bash
```
The command sends you to a shell prompt inside the Pod you just deployed.
Check whether there are any files in the mounted volume yet.
```
ls /tmp/test-pv
```
As expected, nothing returns.
Write a file to the volume.
```
echo "bar" > /tmp/test-pv/foo
```
Confirm the file shows in the volume.
```
ls /tmp/test-pv
```
The output shows the file foo exists.
Exit the Pod shell environment.
```
exit
```
This command exits the Pods shell and returns to the host Oracle Linux node.
The file you created in the volume is now stored independently of the Pod. Next, you will confirm this by deleting and recreating the Pod.
Delete the Pod.
```
kubectl delete pod/pvc-pod
```
Deploy the Pod again.
```
kubectl apply -f pod.yaml
```

Shell into the Pod again.

kubectl exec -it pod/pvc-pod -- /bin/bash

Confirm the foo file you created earlier is still accessible.
```
cat /tmp/test-pv/foo
```
The output displays the contents of the foo file.
Exit the Pod shell environment.
```
exit
```
This command exits the Pods shell and returns to the host Oracle Linux node.

Where is the Data Written?

A primary reason to store files outside of any container is that the information contained in them is valuable to your business or you personally. Therefore, you need to know where to locate the data to manage that information, for example, to ensure backups work. The following steps will show how to find the data on the free lab deployment.

The Pod deployment executes in this tutorial on one of three worker nodes and saves any files to the local disk on that node. The next steps show how to confirm that they are stored there.

Get details of the cluster nodes.

kubectl get nodes -A

Example Output:

[oracle@ocne ~]$ kubectl get nodes -A
NAME                            STATUS   ROLES           AGE   VERSION
mycluster-control-plane-6vqbp   Ready    control-plane   18m   v1.30.10+1.el8
mycluster-md-0-ks8px-9rvwk      Ready    <none>          17m   v1.30.10+1.el8
mycluster-md-0-ks8px-clhw6      Ready    <none>          17m   v1.30.10+1.el8
mycluster-md-0-ks8px-mkzjd      Ready    <none>          16m   v1.30.10+1.el8

Confirm where the pod deployed.

kubectl get pods -o wide

If you have more than one worker node, the output will indicate where the worker node the pod deployed.

Example Output:

[oracle@ocne ~]$ kubectl get pods -o wide
NAME      READY   STATUS    RESTARTS   AGE   IP           NODE                         NOMINATED NODE   READINESS GATES
pvc-pod   1/1     Running   0          89m   10.244.2.3   mycluster-md-0-ks8px-9rvwk   <none>           <none>

Access the node's root directory directly.
```
ocne cluster console --node <WORKER-NODE-WHERE-POD-DEPLOYED> -d
```
Note: Update with the worker node name returned in the previous step.
Confirm the foo file holds the same content you entered from inside the Pod.
```
cat /tmp/test-pv/foo
```
The output showing bar confirms that the content exists and that the Pod saves to the external `foo' file you created from within the Pod, and the Persistent Volume and Persistent Volume Claim provided the mechanism for the Pod to do it.

Next Steps

Confirming the existence of that data concludes the walkthrough of introducing Persistent Volumes and Persistent Volume Claims and how to use them to provide some flexibility over how you manage your data.