Use MetalLB with Oracle Cloud Native Environment

0
0
Send lab feedback

Use MetalLB with Oracle Cloud Native Environment

Introduction

Network load balancers provide a method of externally exposing Kubernetes applications. A Kubernetes LoadBalancer service is used to create a network load balancer that provides and exposes an external IP address that can be used to connect to an application from outside the cluster.

MetalLB is a network load balancer for Kubernetes applications deployed on Oracle Cloud Native Environment running on bare metal hosts. MetalLB allows you to use Kubernetes LoadBalancer services, which traditionally make use of a cloud provider network load balancer, in a bare metal environment.

Objectives

This document outlines how to setup and use the MetalLB module for Kubernetes applications using MetalLB with Oracle Cloud Native Environment.

Prerequisites

The host systems to perform the steps in this tutorial are listed in this section. To be successful requires:

  • 7 Oracle Linux systems to use as:

    • Operator node (ocne-operator)
    • 3 Kubernetes control plane nodes (ocne-control01, ocne-control02, ocne-control03)
    • 3 Kubernetes worker nodes (ocne-worker01, ocne-worker02, ocne-worker03)
  • A virtual IP address for the primary control plane node. This IP address should not be in use on any node, and is assigned dynamically to the control plane node assigned as the primary controller by the load balancer.

    Oracle Support Disclaimer: If you are deploying to Oracle Cloud Infrastructure, your tenancy requires enabling a new feature introduced in OCI: Layer 2 Networking for VLANs within your virtual cloud networks (VCNs). The OCI Layer 2 Networking feature is not generally available, although the tenancy for the free lab environment does have this feature enabled.

    If you have a use case, please work with your technical team to get your tenancy listed to use this feature.

  • Each system should have a minimum of the following installed:

    • Latest Oracle Linux 8 (x86_64) installed and running the Unbreakable Enterprise Kernel Release 6 (UEK R6)
  • The pre-configured setup on these systems is:

    • An oracle user account with sudo privileges
    • Passwordless SSH between each node
    • Oracle Cloud Native Environment installed and configured

Set Up Lab Environment

Note: When using the free lab environment, see Oracle Linux Lab Basics for connection and other usage instructions.

This lab involves multiple systems, each of which requires different steps to be performed. It is recommended to start by opening seven terminal windows or tabs and connecting to each node. This avoids you having to repeatedly log in and out. The nodes are:

  • ocne-control01
  • ocne-control02
  • ocne-control03
  • ocne-operator
  • ocne-worker01
  • ocne-worker02
  • ocne-worker03

Important: The free lab environment deploys a fully installed Oracle Cloud Native Environment across the provided nodes. This deployment takes approximately 25-30 minutes to finish after launch. Therefore, you might want to step away while this runs and then return to complete the lab.

  1. Open a terminal and connect via ssh to each node.

    ssh oracle@<ip_address_of_ol_node>

Validate the Kubernetes Environment

  1. (On any control plane node) Verify kubectl works.

    kubectl get nodes

    Example Output:

    [oracle@ocne-control01 ~]$ kubectl get nodes
    NAME             STATUS   ROLES                  AGE   VERSION
    ocne-control01   Ready    control-plane,master   22m   v1.23.7+1.el8
    ocne-control02   Ready    control-plane,master   21m   v1.23.7+1.el8
    ocne-control03   Ready    control-plane,master   20m   v1.23.7+1.el8
    ocne-worker01    Ready    <none>                 20m   v1.23.7+1.el8
    ocne-worker02    Ready    <none>                 19m   v1.23.7+1.el8
    ocne-worker03    Ready    <none>                 19m   v1.23.7+1.el8
    [oracle@ocne-control01 ~]$

Configure the Worker Nodes

(On all control plane and worker nodes) Set up the Network Ports.

sudo firewall-cmd --zone=public --add-port=7946/tcp --permanent
sudo firewall-cmd --zone=public --add-port=7946/udp --permanent
sudo firewall-cmd --reload

Install the MetalLB Module

Next install and validate the MetalLB module.

On the ocne-operator node:

  1. Create the MetalLB configuration file.

    cat << 'EOF' | tee metallb-config.yaml
    address-pools:
    - name: default
      protocol: layer2
      addresses:
      - 10.0.12.240-10.0.12.250
    EOF
  2. View the configuration file contents.

    cat ~/myenvironment.yaml
  3. Add a Helm and MetalLB module to the Oracle Cloud Native Environment configuration file.

    cat << 'EOF' | tee -a ~/myenvironment.yaml
          - module: helm
            name: myhelm
            args:
              helm-kubernetes-module: mycluster
          - module: metallb
            name: mymetallb
            args: 
              metallb-helm-module: myhelm 
              helm-kubernetes-module: mycluster  
              metallb-config: /home/oracle/metallb-config.yaml             
    EOF
  4. Confirm the Helm and MetalLB modules have been added to the myenviroment.yaml file.

    cat ~/myenvironment.yaml
  5. Create the modules.

    olcnectl module create --config-file myenvironment.yaml

    Example Output:

    [oracle@ocne-operator ~]$ olcnectl module create --config-file myenvironment.yaml
    Modules created successfully.
    Modules created successfully.
    Modules created successfully.
    [oracle@ocne-operator ~]$
  6. Validate the modules.

    olcnectl module validate --config-file myenvironment.yaml

    Example Output:

    [oracle@ocne-operator ~]$ olcnectl module validate --config-file myenvironment.yaml
    Validation of module mycluster succeeded.
    Validation of module myhelm succeeded.
    Validation of module mymetallb succeeded.
    [oracle@ocne-operator ~]$
  7. Install the modules.

    olcnectl module install --config-file myenvironment.yaml

    Note: This may take a few minutes to complete.

    Example Output:

    [oracle@ocne-operator ~]$ olcnectl module install --config-file myenvironment.yaml
    Modules installed successfully.
    Modules installed successfully.
    Modules installed successfully.
    [oracle@ocne-operator ~]$
  8. Show the installed modules.

    olcnectl module instances --config-file myenvironment.yaml

    Example Output:

    [oracle@ocne-operator ~]$ olcnectl module instances --config-file myenvironment.yaml
    INSTANCE       	MODULE    	STATE    
    10.0.12.11:8090	node      	installed
    10.0.12.12:8090	node      	installed
    10.0.12.13:8090	node      	installed
    10.0.12.21:8090	node      	installed
    10.0.12.22:8090	node      	installed
    10.0.12.23:8090	node      	installed
    mycluster      	kubernetes	installed
    myhelm         	helm      	installed
    mymetallb      	metallb   	installed
    [oracle@ocne-operator ~]$

Create a Kubernetes Application

In this section, you create a Kubernetes application that uses a LoadBalancer service.

On any control plane node:

  1. Create a Kubernetes application.

    tee echo-oci-lb.yml > /dev/null << 'EOF'
    ---
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: echo-deployment
      labels:
        app: echo1
    spec:
      replicas: 2
      selector:
        matchLabels:
          app: echo1
      template:
        metadata:
          labels:
            app: echo1
        spec:
          containers:
          - name: echoserver
            image: k8s.gcr.io/echoserver:1.4
            ports:
            - containerPort: 80
    ---
    kind: Service
    apiVersion: v1
    metadata:
      name: echo-lb-service
    spec:
      selector:
        app: echo1
      type: LoadBalancer
      ports:
      - name: http
        port: 80
        targetPort: 8080
    EOF
  2. Create the service.

    kubectl create -f echo-oci-lb.yml

    Example Output:

    [oracle@ocne-control01 ~]$ kubectl create -f echo-oci-lb.yml
    deployment.apps/echo-deployment created
    service/echo-lb-service created
    [oracle@ocne-control01 ~]$ 
  3. Confirm the Kubernetes deployment is running.

    kubectl get deployments

    Example Output:

    [oracle@ocne-control01 ~]$ kubectl get deployment      
    NAME              READY   UP-TO-DATE   AVAILABLE   AGE
    echo-deployment   2/2     2            2           3m15s
    [oracle@ocne-control01~]$
  4. Show the Kubernetes service is running.

    kubectl get svc

    Example Output:

    [oracle@ocne-control01 ~]$ kubectl get svc
    NAME              TYPE           CLUSTER-IP     EXTERNAL-IP   PORT(S)        AGE
    echo-lb-service   LoadBalancer   10.111.72.49   10.0.12.240   80:31727/TCP   4m47s
    kubernetes        ClusterIP      10.96.0.1      <none>        443/TCP        58m
    [oracle@ocne-control01 ~]$

    Notice the EXTERNAL-IP for the echo-lb-service LoadBalancer has an IP address of 10.0.12.240. This IP address is provided by MetalLB and is the external IP address that you can use to connect to the application.

Testing the Deployment

The next step is to test the newly deployed application. As the EXTERNAL-IP value is dynamically provisioned by MetalLB (in this scenario between the range 10.0.12.240-10.0.12.250), the first two steps store this dynamic value as operating system variables.

  1. (On any control plane node) Capture the assigned EXTERNAL-IP address.

    LB=$(kubectl get svc -o jsonpath="{.status.loadBalancer.ingress[0].ip}" echo-lb-service)
  2. Capture the port number assigned.

    LBPORT=$(kubectl get svc -o jsonpath="{.spec.ports[0].port}" echo-lb-service)
  3. Confirm these are stored as environment variables.

    echo $LB
    echo $LBPORT

    Example Output:

    [oracle@ocne-control01 ~]$ echo $LB
    10.0.12.240
    [oracle@ocne-control01 ~]$ echo $LBPORT
    80
    [oracle@ocne-control01 ~]$
  4. Use curl to connect to the deployed application.

    curl -i -w "\n" $LB:$LBPORT

    Example Output:

    [oracle@ocne-control01 ~]$ curl -i -w "\n" $LB:$LBPORT
    HTTP/1.1 200 OK
    Server: nginx/1.10.0
    Date: Wed, 10 Aug 2022 10:52:10 GMT
    Content-Type: text/plain
    Transfer-Encoding: chunked
    Connection: keep-alive
    
    CLIENT VALUES:
    client_address=10.244.2.0
    command=GET
    real path=/
    query=nil
    request_version=1.1
    request_uri=http://10.0.12.240:8080/
    
    SERVER VALUES:
    server_version=nginx: 1.10.0 - lua: 10001
    
    HEADERS RECEIVED:
    accept=*/*
    host=10.0.12.240
    user-agent=curl/7.61.1
    BODY:
    -no body in request-
    [oracle@ocne-control01 ~]$

Test From a Non-Kubernetes Host

  1. (On ocne-operator) This last test confirms that MetalLB is responding to external requests.

    curl -v http://10.0.12.240:80

    Example Output:

    [oracle@ocne-operator ~]$ curl -v http://10.0.12.240:80
    * Rebuilt URL to: http://10.0.12.240:80/
    *   Trying 10.0.12.240...
    * TCP_NODELAY set
    * Connected to 10.0.12.240 (10.0.12.240) port 80 (#0)
    > GET / HTTP/1.1
    > Host: 10.0.12.240
    > User-Agent: curl/7.61.1
    > Accept: */*
    > 
    < HTTP/1.1 200 OK
    < Server: nginx/1.10.0
    < Date: Wed, 10 Aug 2022 11:38:08 GMT
    < Content-Type: text/plain
    < Transfer-Encoding: chunked
    < Connection: keep-alive
    < 
    CLIENT VALUES:
    client_address=10.244.0.0
    command=GET
    real path=/
    query=nil
    request_version=1.1
    request_uri=http://10.0.12.240:8080/
    
    SERVER VALUES:
    server_version=nginx: 1.10.0 - lua: 10001
    
    HEADERS RECEIVED:
    accept=*/*
    host=10.0.12.240
    user-agent=curl/7.61.1
    BODY:
    * Connection #0 to host 10.0.12.240 left intact
    [oracle@ocne-operator ~]$

This confirms that MetalLB has been configured correctly, an application deployed and is accepting requests successfully.

For More Information

2022-12-01T07:57:24.810Z