Scale an Oracle Cloud Native Environment Cluster

0
0
Send lab feedback

Scale an Oracle Cloud Native Environment Cluster

Introduction

Scaling up a Kubernetes cluster means adding nodes; likewise, scaling down occurs by removing nodes. These nodes can be either control plane or worker nodes. Oracle recommends against scaling the cluster up and down simultaneously but instead performing a scale up and down in two separate commands.

Scaling the Kubernetes cluster control plane or worker nodes in odd numbers is recommended to avoid split-brain scenarios and maintain the quorum. For example, 3, 5, or 7 control plane nodes or worker nodes ensure the cluster's reliability.

Objectives

In this tutorial, you'll learn how to use kubectl scale and kubectl edit to:

  • Add a new control plane and worker nodes to a Kubernetes cluster
  • Remove control plane nodes and worker nodes from a Kubernetes cluster

Prerequisites

  • Minimum of one Oracle Linux instance

    • Runs ocne and libvirt ephemeral cluster

  • OCI cluster creation requires access to the following resources in an Oracle Cloud Infrastructure tenancy:

    • Virtual cloud network with four subnets
    • Network load balancer
    • Object Storage bucket with minimum 5 GiB available
    • Compute Custom Image
    • Compute Arm Shape for the control plane node
      • VM.Standard.A1.Flex with two OCPU and 12 memory
    • Compute for each additional control plane and worker node
      • VM.Standard.E4.Flex with four OCPU and 64 memory

Configure Oracle Cloud Native Environment

Note: If running in your tenancy, read the linux-virt-labs GitHub project README.md and complete the prerequisites before deploying the lab environment.

  1. Open a terminal on the Luna Desktop.

  2. Clone the linux-virt-labs GitHub project.

    git clone https://github.com/oracle-devrel/linux-virt-labs.git

  3. Change into the working directory.

    cd linux-virt-labs/ocne2

  4. Install the required collections.

    ansible-galaxy collection install -r requirements.yml

  5. Increase the Boot volume size.

    cat << EOF | tee instances.yml > /dev/null
compute_instances:
  1:
    instance_name: "ocne"
    type: "server"
    boot_volume_size_in_gbs: 128
install_ocne_rpm: true
install_type: "oci"
ocne_cluster_name: "mycluster"
create_ocne_oci_cluster: true
EOF

  6. Deploy the lab environment.

    ansible-playbook create_instance.yml -e localhost_python_interpreter="/usr/bin/python3.6" -e "@instances.yml"

    The free lab environment requires the extra variable local_python_interpreter, which sets ansible_python_interpreter for plays running on localhost. This variable is needed because the environment installs the RPM package for the Oracle Cloud Infrastructure SDK for Python, located under the python3.6 modules.

    The default deployment shape uses the AMD CPU and Oracle Linux 8. To use an Intel CPU or Oracle Linux 9, add -e instance_shape="VM.Standard3.Flex" or -e os_version="9" to the deployment command.

    Important: Wait for the playbook to run successfully and reach the pause task. At this stage of the playbook, the installation of the Oracle Cloud Native Environment is complete, and the instances are ready. Take note of the previous play, which prints the public and private IP addresses of the nodes it deploys and any other deployment information needed while running the lab.

Confirm the Cluster is Running

  1. Open a terminal and connect via SSH to the ocne instance.

    ssh oracle@<ip_address_of_node>

  2. Get a list of known clusters using the CLI.

    ocne cluster list

    Example Output:

    [oracle@ocne ~]$ ocne cluster list
ocne-ephemeral
mycluster

    The output shows two clusters are present, the one you just created (mycluster) and the bootstrap cluster (ocne-ephemeral).

  3. Get the location of the kube configuration.

    ocne cluster show -C mycluster

    We use the -C to specify a specific cluster from the cluster list.

  4. Set the KUBECONFIG environment variable.

    export KUBECONFIG=$(ocne cluster show -C mycluster)

  5. Wait for the cluster to stabilize and all nodes to report in a ready state.

    watch kubectl get nodes -A

    Example Output:

    Every 2.0s: kubectl get nodes -A                               ocne: Wed Nov 13 16:40:08 2024

NAME                            STATUS   ROLES           AGE     VERSION
mycluster-control-plane-8dq98   Ready    control-plane   11m     v1.30.3+1.el8
mycluster-control-plane-dkscq   Ready    control-plane   8m      v1.30.3+1.el8
mycluster-control-plane-jr9gz   Ready    control-plane   9m      v1.30.3+1.el8
mycluster-md-0-8jr9k-8gnvw	    Ready    <none>          9m      v1.30.3+1.el8
mycluster-md-0-8jr9k-fn9mk	    Ready    <none>          9m      v1.30.3+1.el8
mycluster-md-0-8jr9k-kr9p6	    Ready    <none>          9m      v1.30.3+1.el8

    Wait for the three worker and control plane nodes to show a STATUS of Ready, then type ctrl-c to exit the watch command.

Use the Scale Command to Scale the Control Plane and Worker Nodes

Kubernetes administrators typically use the kubectl scale command to scale resources deployed onto your Kubernetes cluster (e.g.: Deployments, StatefulSets, ReplicaSets, etc.). However, it can also scale the number of control and worker nodes in your Kubernetes cluster.

Scale Up the Control Plane and Worker Nodes

  1. Confirm the bootstrap cluster is present.

    sudo virsh list

    Example Output:

    [oracle@ocne ~]$ sudo virsh list
 Id   Name                             State
------------------------------------------------
 1    ocne-ephemeral-control-plane-1   running

  2. Set the kubectl context to reference the bootstrap cluster.

    export KUBECONFIG=~/.kube/kubeconfig.ocne-ephemeral.local

  3. Use the Cluster API Provider for OCI to confirm the number of nodes present.

    watch kubectl get ocimachine -n ocne

    Example Output:

    Every 2.0s: kubectl get ocimachine -n ocne                                ocne: Wed Dec 18 11:29:08 2024

[oracle@ocne ~]$ kubectl get ocimachine -n ocne
NAME                            AGE
mycluster-control-plane-wl7n8   9m58s
mycluster-control-plane-dkscq   12m
mycluster-control-plane-jr9gz   15m
mycluster-md-0-fvtvm-6nzwb      10m
mycluster-md-0-fvtvm-qs4dl      10m
mycluster-md-0-fvtvm-ttpl5      10m

    Wait until the output matches the cluster's structure from the kubectl get nodes -A command. Then type ctrl-c to exit the watch command.

  4. Get the name of the machinedeployments Custom Resource.

    kubectl get machinedeployments -n ocne

    Example Output:

    [oracle@ocne ~]$ kubectl get machinedeployments -n ocne
NAME             CLUSTER     REPLICAS   READY   UPDATED   UNAVAILABLE   PHASE     AGE    VERSION
mycluster-md-0   mycluster   3          3       3         0             Running   10m    v1.30.3

    The NAME column shows the MachineDeployments, which reflect Kubernetes objects representing the worker nodes in the cluster.

  5. Scale up a worker node from 3 to 4.

    kubectl scale machinedeployment mycluster-md-0 --replicas=4 --namespace ocne

  6. Scale up the control plane nodes from 3 to 5.

    kubectl scale kubeadmcontrolplane mycluster-control-plane --replicas=5 --namespace ocne

  7. Set the context back to the Workload cluster and confirm the new control plane nodes are present.

    export KUBECONFIG=$(ocne cluster show -C mycluster)
watch kubectl get nodes -A

    Wait for this to complete, but check for the existence of five control plane nodes and four worker nodes. Once they show as running, type ctrl-c to exit the watch command.

Scale Down the Control Plane and Worker Nodes

  1. Set the kubectl context to reference the bootstrap cluster.

    export KUBECONFIG=~/.kube/kubeconfig.ocne-ephemeral.local

  2. Scale down a worker node.

    kubectl scale machinedeployment mycluster-md-0 --replicas=3 --namespace ocne

  3. Scale down the control plane nodes.

    kubectl scale kubeadmcontrolplane mycluster-control-plane --replicas=3 --namespace ocne

  4. Set the context back to the Workload cluster and confirm the new control plane nodes are present.

    export KUBECONFIG=$(ocne cluster show -C mycluster)
watch kubectl get nodes -A

    Wait for the cluster to stabilize and the scale down to complete by leaving only three worker and three control plane nodes running. Then type ctrl-c to exit the watch command.

Use the Edit Command to Scale the Control Plane and Worker Nodes

The kubectl edit command provides both Kubernetes administrators and developers with a way to apply real-time direct edits and modification to a deployed Kubernetes API configuration or resource. By default, it opens the referenced Kubernetes API resource in vi on Linux or notepad on Windows. To use your favorite editor, set the KUBE_EDITOR or EDITOR variables within the same shell as you run the edit command. However, it is generally not recommended for use on production systems to avoid altering something accidentally. It is, however, a flexible way to test changes in development or test environments.

You can use the edit command to accomplish the same scale up and down actions as the scale command performs.

Scale Up a Worker Node

  1. Confirm the bootstrap cluster is present.

    sudo virsh list

    Example Output:

    [oracle@ocne ~]$ sudo virsh list
 Id   Name                             State
------------------------------------------------
 1    ocne-ephemeral-control-plane-1   running

  2. Set the kubectl context to reference the bootstrap cluster.

    export KUBECONFIG=~/.kube/kubeconfig.ocne-ephemeral.local

  3. Use the Cluster API Provider for OCI to confirm the number of nodes present.

    kubectl get ocimachine -n ocne

    Example Output:

    [oracle@ocne ~]$ kubectl get ocimachine -n ocne
NAME                            AGE
mycluster-control-plane-dbrkt   5m45s
mycluster-control-plane-nrlh9   4m13s
mycluster-control-plane-wm7rg   8m34s
mycluster-md-0-9cczf-kspf9      9m11s
mycluster-md-0-9cczf-ngjgk      9m11s
mycluster-md-0-9cczf-z62zl      9m11s

  4. Get the name of the machinedeployments Custom Resource.

    kubectl get machinedeployments -n ocne

    Example Output:

    [oracle@ocne ~]$ kubectl get machinedeployments -n ocne
NAME             CLUSTER     REPLICAS   READY   UPDATED   UNAVAILABLE   PHASE     AGE    VERSION
mycluster-md-0   mycluster   3          3       3         0             Running   10m    v1.30.3

  5. Edit the Custom Resource.

    kubectl edit machinedeployments --namespace ocne mycluster-md-0

    This command opens the Custom Resource for editing.

  6. Set the Replicas field to the number of the worker nodes you want to deploy.

    For this example, change the replicas: 3 field to: replicas: 4. Then save and exit. You enter the total number of worker nodes, and the cluster will scale up or down accordingly.

    Example Output:

    # Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be
# reopened with the relevant failures.
#
apiVersion: cluster.x-k8s.io/v1beta1
kind: MachineDeployment
metadata:
  annotations:
...
...
spec:
  clusterName: mycluster
  minReadySeconds: 0
  progressDeadlineSeconds: 600
  replicas: 3
  revisionHistoryLimit: 1
...
...

    Note: This opens the Custom Resource in vi. Use the standard vi keystrokes to enter text, and to save the change and exit editor.

  7. Confirm the creation of the new worker node.

    kubectl get ocimachine --namespace ocne

    Example Output:

    [oracle@ocne ~]$ kubectl get ocimachine --namespace ocne
NAME                            AGE
mycluster-control-plane-lj9m9   11m
mycluster-control-plane-t6gc8   14m
mycluster-control-plane-w427z   12m
mycluster-md-0-fvtvm-bzfnc      8s
mycluster-md-0-fvtvm-qs4dl      38m
mycluster-md-0-fvtvm-ttpl5      38m
mycluster-md-0-fvtvm-xf4ch      22m

  8. Set the context back to the Workload cluster and confirm the new worker node is present.

    export KUBECONFIG=$(ocne cluster show -C mycluster)
watch kubectl get nodes -A

    Wait for the new worker node to show a STATUS of Ready. Then type ctrl-c to exit the watch command.

Scale Up Control Plane Nodes

  1. Set the kubectl context to reference the bootstrap cluster.

    export KUBECONFIG=~/.kube/kubeconfig.ocne-ephemeral.local

  2. Get the name of the kubeadmcontrolplane Custom Resource.

    kubectl get kubeadmcontrolplane -n ocne

    Example Output:

    [oracle@ocne ~]$ kubectl get kubeadmcontrolplane -n ocne
NAME                      CLUSTER     INITIALIZED   API SERVER AVAILABLE   REPLICAS   READY   UPDATED   UNAVAILABLE   AGE   VERSION
mycluster-control-plane   mycluster   true          true                   3          3       3         0             13m   v1.30.3

  3. Edit the Custom Resource.

    kubectl edit kubeadmcontrolplane --namespace ocne mycluster-control-plane

  4. Set the Replicas field to the new number of control plane nodes to deploy.

    Change the replicas: 3 field to: replicas: 5, then save and exit.

    Example Output:

    # Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be   
# reopened with the relevant failures.
#
apiVersion: controlplane.cluster.x-k8s.io/v1beta1
kind: KubeadmControlPlane
metadata:
  creationTimestamp: "2024-11-05T17:03:52Z"
  finalizers:
  - kubeadm.controlplane.cluster.x-k8s.io
  generation: 3
  labels:
    cluster.x-k8s.io/cluster-name: mycluster
...
...
  machineTemplate:
   infrastructureRef:
      apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
      kind: OCIMachineTemplate
      name: mycluster-control-plane
      namespace: ocne
    metadata: {}
  replicas: 5
  rolloutStrategy:
...
...

    Note: This opens the Custom Resource in vi. Use the standard vi keystrokes to enter text, and to save the change and exit editor.

    Important: The number of control plane nodes set must always be an odd number or the scale request will not be completed.

  5. Confirm the new control plane nodes deployed.

    watch kubectl get ocimachine --namespace ocne

    Example Output:

    Every 2.0s: kubectl get ocimachine --namespace ocne                    ocne: Wed Dec 18 11:15:03 2024

NAME                            AGE
mycluster-control-plane-8wq8l   16s
mycluster-control-plane-lj9m9   16m
mycluster-control-plane-qzwmb   97s
mycluster-control-plane-t6gc8   19m
mycluster-control-plane-w427z   18m
mycluster-md-0-fvtvm-bzfnc      5m14s
mycluster-md-0-fvtvm-qs4dl      43m
mycluster-md-0-fvtvm-ttpl5      43m
mycluster-md-0-fvtvm-xf4ch      27m

    Note: It will take a few minutes for both of the new control plane nodes to show. Then type ctrl-c to exit the watch command.

  6. Set the context back to the Workload cluster and confirm the new control plane nodes are present.

    export KUBECONFIG=$(ocne cluster show -C mycluster)
watch kubectl get nodes -A

    Wait for both of the new control plane nodes to display under the NAME column with a ready STATUS. Then type ctrl-c to exit the watch command.

Scale Down a Worker Node

  1. Set the kubectl context to reference the bootstrap cluster.

    export KUBECONFIG=~/.kube/kubeconfig.ocne-ephemeral.local

  2. Edit the Custom Resource.

    kubectl edit machinedeployments --namespace ocne mycluster-md-0

  3. Set the Replicas field to reduce the number of worker nodes.

    Change the replicas: 4 field to: replicas: 3. Then save and exit.

    Example Output:

    # Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be
# reopened with the relevant failures.
#
apiVersion: cluster.x-k8s.io/v1beta1
kind: MachineDeployment
metadata:
  annotations:
...
...
spec:
  clusterName: mycluster
  minReadySeconds: 0
  progressDeadlineSeconds: 600
  replicas: 3
  revisionHistoryLimit: 1
...
...

    Note: This opens the Custom Resource in vi. Use the standard vi keystrokes to enter text, and to save the change and exit editor.

Scale Down the Control Plane Nodes

  1. Edit the Custom Resource (using the NAME noted in the previous step).

    kubectl edit kubeadmcontrolplane --namespace ocne mycluster-control-plane

  2. Set the Replicas field to a lower number to reduce the number of control plane nodes.

    Change the replicas: 5 field to: replicas: 3. Then save and exit.

    Example Output:

    # Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be   
# reopened with the relevant failures.
#
apiVersion: controlplane.cluster.x-k8s.io/v1beta1
kind: KubeadmControlPlane
metadata:
  creationTimestamp: "2024-11-05T17:03:52Z"
  finalizers:
  - kubeadm.controlplane.cluster.x-k8s.io
  generation: 3
  labels:
    cluster.x-k8s.io/cluster-name: mycluster
...
...
  machineTemplate:
   infrastructureRef:
      apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
      kind: OCIMachineTemplate
      name: mycluster-control-plane
      namespace: ocne
    metadata: {}
  replicas: 5
  rolloutStrategy:
...
...

    Note: This opens the Custom Resource in vi. Use the standard vi keystrokes to enter text, and to save the change and exit editor.

    Important: The number of control plane nodes set must always be an odd number or the scale request will not be completed.

  3. Confirm the removal of the control plane nodes.

    watch kubectl get ocimachine --namespace ocne

    Example Output:

    Every 2.0s: kubectl get ocimachine --namespace ocne                    ocne: Wed Dec 18 11:35:43 2024

NAME                            AGE
mycluster-control-plane-8wq8l   10m
mycluster-control-plane-qzwmb   12m
mycluster-control-plane-t6gc8   30m
mycluster-md-0-fvtvm-qs4dl	    54m
mycluster-md-0-fvtvm-ttpl5	    54m
mycluster-md-0-fvtvm-xf4ch	    37m

    When the removal process is complete, you can type ctrl-c to exit the watch command.

  4. Set the context back to the Workload cluster and confirm the removal of the control plane nodes.

    export KUBECONFIG=$(ocne cluster show -C mycluster)
kubectl get nodes -A

Next Steps

That completes the demonstration of adding and removing nodes from a Kubernetes cluster. While this exercise demonstrated updating the control plane and worker nodes simultaneously, this is not the recommended approach to scaling up or down an Oracle Cloud Native Environment Kubernetes cluster. In production environments, administrators should undertake these tasks separately.

SSR