Deploy an External Load Balancer with Oracle Cloud Native Environment

0
0
Send lab feedback

Deploy an External Load Balancer with Oracle Cloud Native Environment

Introduction

Oracle Cloud Native Environment is a fully integrated suite for the development and management of cloud native applications. The Kubernetes module is the core module. It is used to deploy and manage containers and also automatically installs and configures CRI-O, runC and Kata Containers. CRI-O manages the container runtime for a Kubernetes cluster. The runtime may be either runC or Kata Containers.

Objectives

This tutorial/lab demonstrates how to:

  • Configure the Kubernetes cluster with the Oracle Cloud Infrastructure load balancer to enable high availability
  • Configure Oracle Cloud Native Environment on a 5-node cluster
  • Verify Load Balancer failover between the control plane nodes completes successfully

Support Note: Using an external load balancer such as the Oracle Cloud Instructure Load Balancer IS recommended for production deployments.

Prerequisites

The host systems to perform the steps in this tutorial are listed in this section. You need:

  • 6 Oracle Linux systems to use as:

    • Operator node (ocne-operator)
    • 3 Kubernetes control plane nodes (ocne-control01, ocne-control02, ocne-control03 )
    • 2 Kubernetes worker nodes (ocne-worker01, ocne-worker02)
    • 1 Oracle Linux system with kubectl installed (sysop01)

    Note: In a production environment, it is recommended that you have a cluster with at least five control plane nodes and at least three worker nodes.

  • Each system should have a minimum of the following installed:

    • Latest Oracle Linux 8 (x86_64) installed and running the Unbreakable Enterprise Kernel Release 6 (UEK R6)
  • This environment is pre-configured with the following:

    • Created an oracle user account (used during the install)
    • Granted the oracle account sudo access
    • Set up key-based SSH, also known as passwordless SSH, between the instances
    • Oracle Cloud Native Environment Release 1.5 installed (but no environment created)

Set Up Lab Environment

Note: When using the free lab environment, see Oracle Linux Lab Basics for connection and other usage instructions.

This lab involves multiple systems, each of which requires different steps to be performed. It is recommended to start by opening a terminal window or tab and connecting to the following nodes. This avoids you having to repeatedly log in and out. The nodes are:

  • ocne-operator
  • Any one of:
    • ocne-control01
    • ocne-control02
    • ocne-control03
  • sysop01
  1. Open a terminal and connect via ssh to each of the nodes.

    ssh oracle@<ip_address_of_ol_node>

Set up Load Balancer (Optional)

Note: If using the free lab environment, the steps presented in the section are not required because the initial lab deployment phase completed them - instead Luna lab users should proceed directly to the next page: Create a Platform CLI Configuration File and continue from there. Otherwise, the details of how to configure this are provided for users who wish to replicate these steps on their own Oracle Cloud Infrastructure tenancy.

Create an Oracle Cloud Infrastructure Load Balancer

  1. Login to the Cloud Console.

    public_ip_address

  2. Click on the hamburger menu (top-left), then Networking and Load Balancers.

    public_ip_address

  3. The Load Balancers page displays.

    public_ip_address

  4. Locate the Compartment being used from the drop-down list.

    public_ip_address

  5. Click on the Create Load Balancer button.

    public_ip_address

  6. In the pop-up dialog box, click the Create Load Balancer button to select the default Load Balancer type.

    public_ip_address

Update Load Balancer Details

  1. Locate the default Visibility type section and click the Private option.

    public_ip_address

  2. Scroll further down the page to the Choose Networking section.

  3. Select the value provided in the drop-down list-box for Visual Cloud Network and Subnet.

    Note: the values shown in the image will be different each time the Lab is initiated.

    public_ip_address

  4. Click Next to move to the next step.

Set Load Balancer Policy and Protocol

  1. Now set the Load Balancing Policy and Health Check Protocol.

    • Accept the default Load Balancing Policy which is Weighted Round Robin
  2. In the Specify Health Check Policy section enter the settings shown.

    • Under Protocol select TCP
    • Change the Port value from 80 to 6443

    public_ip_address

Add Backend Nodes

  1. Click the Add Backends button, and a window will open.

    public_ip_address

  2. Select the following nodes, and click the Add Selected Backends button.

    • ocne-control01
    • ocne-control02
    • ocne-control03
  3. Update the Port column for each of the newly selected backend servers from the default value of 80 to 6443.

    public_ip_address

  4. Click the Next button to proceed to the next step.

Configure the Load Balancer Listener

  1. Select the TCP button

  2. Change the Port used for ingress traffic from 443 to 6443

    The entered values should look like this image.

    public_ip_address

  3. Click the Next button.

Configure Load Balancer Logging

  1. The final step during the Setup process, is the Manage Logging options.

    public_ip_address

    Because no changes are required for this scenario, click the Submit button to create the Load Balancer.

Load Balancer Information

  1. Once created, an overview page for the newly created Load Balancer is displayed.

    public_ip_address

    Note: The Overall Health and Backend Sets Health sections may not diplay as OK (Green). This is because Oracle Cloud Native Environment has not yet been created on the Kubernetes control plane and worker nodes. The next steps will address this.

Create a Platform CLI Configuration File

Administrators can use a configuration file to simplify creating and managing environments and modules. The configuration file, written in valid YAML syntax, includes all information about the environments and modules to create. Using a configuration file saves repeated entries of Platform CLI command options.

Note: If more than one control plane node is entered in the myenvironment.yaml used to configure the Oracle Cloud Native Environment, then olcnectl requires that details of a load balancer (in our case this is the Oracle Cloud Infrastructure load balancer) are entered into the myenvironment.yaml file. Achieve this via entering a new argument (load-balancer: <enter-your-OCI-LB-ip-here>) into the myenvironment.yaml.

Example Configuration File:

environments:
  - environment-name: myenvironment
    globals:
      api-server: 127.0.0.1:8091
      secret-manager-type: file
      olcne-ca-path: /etc/olcne/configs/certificates/production/ca.cert
      olcne-node-cert-path: /etc/olcne/configs/certificates/production/node.cert
      olcne-node-key-path:  /etc/olcne/configs/certificates/production/node.key
    modules:
      - module: kubernetes
        name: mycluster
        args:
          container-registry: container-registry.oracle.com/olcne
          load-balancer: 10.0.0.<XXX>:6443
          master-nodes: ocne-control01:8090,ocne-control02:8090,ocne-control03:8090
          worker-nodes: ocne-worker01:8090,ocne-worker02:8090
          selinux: enforcing
          restrict-service-externalip: true
          restrict-service-externalip-ca-cert: /etc/olcne/configs/certificates/restrict_external_ip/production/ca.cert
          restrict-service-externalip-tls-cert: /etc/olcne/configs/certificates/restrict_external_ip/production/node.cert
          restrict-service-externalip-tls-key: /etc/olcne/configs/certificates/restrict_external_ip/production/node.key

During lab deployment, a configuration file is automatically generated and ready to use in the exercise. More information on manually creating a configuration file is in the documentation at Using a Configuration File .

Update the Configuration File

  1. (On ocne-operator) View the configuration file contents.

    cat ~/myenvironment.yaml
  2. (On ocne-operator) Add the load balancer IPv4 address value to the myenvironment.yaml file.

    LB_IP=$(oci lb load-balancer list --auth instance_principal --compartment-id $COMPARTMENT_OCID | jq -r '.data[]."ip-addresses"[]."ip-address"')
    sed -i "14i\          load-balancer: $LB_IP:6443" ~/myenvironment.yaml

    For the above command to succeed, the following preparatory steps must be done on the operator node. These steps are done for you as part of the free lab deployment.

    • Installed Oracle Cloud Infrastructure CLI.

    • Lookup the OCID of the Compartment where deploying Oracle Cloud Native Environment.

    • Added these environment variables to the users ~/.bashrc file.

      • COMPARTMENT_OCID=<compartment_ocid>
      • LC_ALL=C.UTF-8
      • LANG=C.UTF-8
  3. (On ocne-operator) Confirm the load balancer value has been added to the myenviroment.yaml file.

    cat ~/myenvironment.yaml

Create the Environment and Kubernetes Module

  1. (On ocne-operator) Create the environment.

    cd ~
    olcnectl environment create --config-file myenvironment.yaml

    Example Output:

    [oracle@ocne-operator ~]$ olcnectl environment create --config-file myenvironment.yaml
    Environment myenvironment created.
  2. (On ocne-operator) Create the Kubernetes module.

    olcnectl module create --config-file myenvironment.yaml

    Example Output:

    [oracle@ocne-operator ~]$ olcnectl module create --config-file myenvironment.yaml
    Modules created successfully.
  3. (On ocne-operator) Validate the Kubernetes module.

    olcnectl module validate --config-file myenvironment.yaml

    Example Output:

    [oracle@ocne-operator ~]$ olcnectl module validate --config-file myenvironment.yaml
    Validation of module mycluster succeeded.

    In this free lab environment, there are no validation errors. If there are any errors, the commands required to fix the nodes are provided as output of this command.

  4. (On ocne-operator) Install the Kubernetes module.

    olcnectl module install --config-file myenvironment.yaml

    Note: The deployment of Kubernetes to the nodes will take several minutes to complete.

    Example Output:

    [oracle@ocne-operator ~]$ olcnectl module install --config-file myenvironment.yaml
    Modules installed successfully.
  5. (On ocne-operator) Verify the deployment of the Kubernetes module.

    olcnectl module instances --config-file myenvironment.yaml

    Example Output:

    [oracle@ocne-operator ~]$ olcnectl module instances --config-file myenvironment.yaml
    INSTANCE           	MODULE    	STATE    
    mycluster          	kubernetes	installed
    ocne-control01:8090	node      	installed
    ocne-control02:8090	node      	installed
    ocne-control03:8090	node      	installed
    ocne-worker01:8090 	node      	installed
    ocne-worker02:8090 	node      	installed
    [oracle@ocne-operator ~]$

Set up kubectl

  1. (On ocne-controlxx node) Set up the kubectl command.

    mkdir -p $HOME/.kube
    sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
    sudo chown $(id -u):$(id -g) $HOME/.kube/config
    export KUBECONFIG=$HOME/.kube/config
    echo 'export KUBECONFIG=$HOME/.kube/config' >> $HOME/.bashrc
  2. (On sysop01) Set up the kubectl command.

    1. Copy the config file from one of the control plane nodes.

      mkdir -p $HOME/.kube
      ssh -o StrictHostKeyChecking=no 10.0.0.150 "sudo cat /etc/kubernetes/admin.conf" > $HOME/.kube/config
    2. Export the configuration for use by the kubectl command.

      sudo chown $(id -u):$(id -g) $HOME/.kube/config
      export KUBECONFIG=$HOME/.kube/config
      echo 'export KUBECONFIG=$HOME/.kube/config' >> $HOME/.bashrc
  3. (On sysop01) Verify kubectl works.

    kubectl get nodes

    Example Output:

    [oracle@sysop01 ~]$ kubectl get nodes
    NAME             STATUS   ROLES                  AGE     VERSION
    ocne-control01   Ready    control-plane,master   5m10s   v1.23.7+1.el8
    ocne-control02   Ready    control-plane,master   3m33s   v1.23.7+1.el8
    ocne-control03   Ready    control-plane,master   4m29s   v1.23.7+1.el8
    ocne-worker01    Ready    <none>                 3m15s   v1.23.7+1.el8
    ocne-worker02    Ready    <none>                 3m8s    v1.23.7+1.el8
    [oracle@sysop01 ~]$

Confirm the Load Balancer Manages a Control Plane Node Outage

The installation of Oracle Cloud Native Environment with three control plane nodes behind an external load balancer is complete.

The next section confirms that the external load balancer will detect when a control plane node fails and remove it from the Round Robin traffic distribution policy. The final testing step demonstrates that when the 'missing' node recovers, it automatically rejoins the cluster and becomes available to handle cluster-based traffic once again.

Confirm the Load Balancer is Active

If using the free lab environment, sign into the Cloud Console as shown in the Oracle Linux Lab Basics documentation.

Access the Load Balancer Details

  1. If not already connected, login to the Cloud Console.

    public_ip_address

  2. Click on the hamburger menu (top-left), then Networking and Load Balancers.

    public_ip_address

  3. This displays the Load Balancers page.

    public_ip_address

  4. Locate the Compartment being used from the drop-down list.

    public_ip_address

    If needing instruction on this step, see Oracle Linux Lab Basics for details.

  5. The pre-created Load Balancer summary details should look similar to this image.

    public_ip_address

  6. Click on the ocne-load-balancer link.

  7. Scroll down to the Backend Sets link (on the left-hand side of the browser under the Resources heading).

  8. Click on the Backend Sets link.

    public_ip_address

  9. A link to the existing Backend Set is displayed. Click the link (in this example it is called ocne-lb-backend-set).

    public_ip_address

  10. Click on the Backends link (on the left-hand side of the browser under the Resources heading).

    public_ip_address

  11. The screen confirms there are three healthy nodes present.

    public_ip_address

Stop One of the Control Plane Node Instances

  1. Click on the hamburger menu (top left), and navigate to Compute and then Instances.

    public_ip_address

  2. The Instances page displays.

    public_ip_address

  3. Click on the one of the control plane nodes listed (for example: ocne-control01).

  4. The Instance details displays.

    public_ip_address

  5. Click on the *Stop button.

  6. In the pop-up dialog box, select the Force stop the instance by immediately powering off check-box, and then click the Force stop instance button.

    Note: Do NOT do this on a production system as it may incur data loss, corruption or worse to the entire system.

    public_ip_address

  7. Wait until the Instance details page confirms the instance is Stopped.

    public_ip_address

Verify the Oracle Cloud Infrastructure Load Balancer Registers the Control Plane Node Failure

  1. Navigate to Networking > Load Balancers.

  2. Click the ocne-load-balancer link.

  3. Under Resources on the lower left panel, click on Backend Sets.

  4. Then click on the displayed name in the Backend Sets table (example: ocne-lb-backend-set).

  5. Click Backends to display the nodes.

    The status on this page should update automatically. It usually takes 2-3 minutes for the displayed status to alter.

  6. The initial reported stage displayed should be a Warning status.

    public_ip_address

  7. A few minutes later (~2-3 minutes) the status will update to Critical status. This means that the node is confirmed to be unresponsive by the Oracle Cloud Infrastructure load balancer process and therefore the load balancer will no longer forward incoming requests to the unavailable backend control plan node.

    public_ip_address

Confirm the Oracle Cloud Native Environment Cluster Responds

Given there are at least two active members remaining in the cluster, the active control plane nodes should respond to kubectl commands.

  1. (On sysop01) Verify kubectl responds and reports the control plane node just stopped as NotReady (Unavailable).

    kubectl get nodes

    Example Output:

    [oracle@sysop01 ~]$ kubectl get nodes
    NAME             STATUS     ROLES                  AGE   VERSION
    ocne-control01   NotReady   control-plane,master   26m   v1.23.7+1.el8
    ocne-control02   Ready      control-plane,master   25m   v1.23.7+1.el8
    ocne-control03   Ready      control-plane,master   24m   v1.23.7+1.el8
    ocne-worker01    Ready      <none>                 24m   v1.23.7+1.el8
    ocne-worker02    Ready      <none>                 26m   v1.23.7+1.el8
    [oracle@sysop01 ~]$

    *Note: It is likely to be 2-3 minutes before the NotReady status is displayed. Please repeat the kubectl get nodes command periodically until the status changes from Ready to NotReady.

Restart the Stopped Control Plane Node

  1. Navigate to Compute > Instances.

  2. Click on the control plane node that was previously stopped.

  3. Start the instance by clicking the Start button.

    Wait until the Status section turns Green and confirms the Instance is Running.

    public_ip_address

Verify the Control Plane Node Rejoins the Oracle Cloud Infrastructure Load Balancer Cluster

  1. Navigate to Networking > Load Balancers.

  2. Click the ocne-load-balancer link.

  3. Under Resources on the lower left panel, click on Backend Sets.

  4. Then click on the displayed name in the Backend Sets table (example: ocne-lb-backend-set).

  5. Click Backends to display the nodes.

    The status on this page should update automatically. It usually takes 2-3 minutes for the displayed status to alter.

  6. The Overall Health will show a Warning status until the restarted node is detected.

    public_ip_address

  7. Once detected, the Overall Health will report as OK.

    public_ip_address

At this stage the control node has rejoined the cluster, and all three control plane nodes are participating in the round robin apportioning of incoming traffic to the cluster.

Get the Control Plane Node Status

  1. (On sysop01) Verify kubectl responds and reports the control plane node which restarted has a status of Ready (Available).

    kubectl get nodes

    Example Output:

    [oracle@sysop01 ~]$ kubectl get nodes
    NAME             STATUS     ROLES                  AGE   VERSION
    ocne-control01   Ready      control-plane,master   26m   v1.23.7+1.el8
    ocne-control02   Ready      control-plane,master   25m   v1.23.7+1.el8
    ocne-control03   Ready      control-plane,master   24m   v1.23.7+1.el8
    ocne-worker01    Ready      <none>                 24m   v1.23.7+1.el8
    ocne-worker02    Ready      <none>                 26m   v1.23.7+1.el8
    [oracle@sysop01 ~]$

    *Note: Status changes can take 2-3 minutes before updated status is displayed. Please repeat the kubectl get nodes command periodically until the status changes.

    This confirms that load balancer has been configured correctly and is accepting requests succesfully.

For More Information

SSR