Use MetalLB with Oracle Cloud Native Environment

0
0
Send lab feedback

Use MetalLB with Oracle Cloud Native Environment

Introduction

Network load balancers provide a method of externally exposing Kubernetes applications. A Kubernetes LoadBalancer service is used to create a network load balancer that provides and exposes an external IP address that can be used to connect to an application from outside the cluster.

MetalLB is a network load balancer for Kubernetes applications deployed on Oracle Cloud Native Environment running on bare metal hosts. MetalLB allows you to use Kubernetes LoadBalancer services, which traditionally make use of a cloud provider network load balancer, in a bare metal environment.

Objectives

This document outlines how to setup and use the MetalLB module for Kubernetes applications using MetalLB with Oracle Cloud Native Environment.

Prerequisites

The host systems to perform the steps in this tutorial are listed in this section. To be successful requires:

  • 7 Oracle Linux systems to use as:

    • Operator node (ocne-operator)
    • 3 Kubernetes control plane nodes (ocne-control01, ocne-control02, ocne-control03)
    • 3 Kubernetes worker nodes (ocne-worker01, ocne-worker02, ocne-worker03)

  • A virtual IP address for the primary control plane node. This IP address should not be in use on any node, and is assigned dynamically to the control plane node assigned as the primary controller by the load balancer.

    Oracle Support Disclaimer: If you are deploying to Oracle Cloud Infrastructure, your tenancy requires enabling a new feature introduced in OCI: Layer 2 Networking for VLANs within your virtual cloud networks (VCNs). The OCI Layer 2 Networking feature is not generally available, although the tenancy for the free lab environment does have this feature enabled.

    If you have a use case, please work with your technical team to get your tenancy listed to use this feature.

  • Each system should have a minimum of the following installed:

    • Latest Oracle Linux 8 (x86_64) installed and running the Unbreakable Enterprise Kernel Release 6 (UEK R6)

  • The pre-configured setup on these systems is:

    • An oracle user account with sudo privileges
    • Passwordless SSH between each node
    • Oracle Cloud Native Environment installed and configured

Set Up Lab Environment

Note: When using the free lab environment, see Oracle Linux Lab Basics for connection and other usage instructions.

This lab involves multiple systems, each of which requires different steps to be performed. It is recommended to start by opening seven terminal windows or tabs and connecting to each node. This avoids you having to repeatedly log in and out. The nodes are:

  • ocne-control01
  • ocne-control02
  • ocne-control03
  • ocne-operator
  • ocne-worker01
  • ocne-worker02
  • ocne-worker03

Important: The free lab environment deploys a fully installed Oracle Cloud Native Environment across the provided nodes. This deployment takes approximately 25-30 minutes to finish after launch. Therefore, you might want to step away while this runs and then return to complete the lab.

  1. Open a terminal and connect via ssh to each node.

    ssh oracle@<ip_address_of_ol_node>

Validate the Kubernetes Environment

  1. (On any control plane node) Verify kubectl works.

    kubectl get nodes

    Example Output:

    [oracle@ocne-control01 ~]$ kubectl get nodes
NAME             STATUS   ROLES                  AGE   VERSION
ocne-control01   Ready    control-plane,master   22m   v1.23.7+1.el8
ocne-control02   Ready    control-plane,master   21m   v1.23.7+1.el8
ocne-control03   Ready    control-plane,master   20m   v1.23.7+1.el8
ocne-worker01    Ready    <none>                 20m   v1.23.7+1.el8
ocne-worker02    Ready    <none>                 19m   v1.23.7+1.el8
ocne-worker03    Ready    <none>                 19m   v1.23.7+1.el8
[oracle@ocne-control01 ~]$

Configure the Worker Nodes

(On all control plane and worker nodes) Set up the Network Ports.

sudo firewall-cmd --zone=public --add-port=7946/tcp --permanent
sudo firewall-cmd --zone=public --add-port=7946/udp --permanent
sudo firewall-cmd --reload

Install the MetalLB Module

Next install and validate the MetalLB module.

On the ocne-operator node:

  1. Create the MetalLB configuration file.

    cat << 'EOF' | tee metallb-config.yaml
address-pools:
- name: default
  protocol: layer2
  addresses:
  - 10.0.12.240-10.0.12.250
EOF

  2. View the configuration file contents.

    cat ~/myenvironment.yaml

  3. Add a Helm and MetalLB module to the Oracle Cloud Native Environment configuration file.

    cat << 'EOF' | tee -a ~/myenvironment.yaml
      - module: helm
        name: myhelm
        args:
          helm-kubernetes-module: mycluster
      - module: metallb
        name: mymetallb
        args: 
          metallb-helm-module: myhelm 
          helm-kubernetes-module: mycluster  
          metallb-config: /home/oracle/metallb-config.yaml             
EOF

  4. Confirm the Helm and MetalLB modules have been added to the myenviroment.yaml file.

    cat ~/myenvironment.yaml

  5. Create the modules.

    olcnectl module create --config-file myenvironment.yaml

    Example Output:

    [oracle@ocne-operator ~]$ olcnectl module create --config-file myenvironment.yaml
Modules created successfully.
Modules created successfully.
Modules created successfully.
[oracle@ocne-operator ~]$

  6. Validate the modules.

    olcnectl module validate --config-file myenvironment.yaml

    Example Output:

    [oracle@ocne-operator ~]$ olcnectl module validate --config-file myenvironment.yaml
Validation of module mycluster succeeded.
Validation of module myhelm succeeded.
Validation of module mymetallb succeeded.
[oracle@ocne-operator ~]$

  7. Install the modules.

    olcnectl module install --config-file myenvironment.yaml

    Note: This may take a few minutes to complete.

    Example Output:

    [oracle@ocne-operator ~]$ olcnectl module install --config-file myenvironment.yaml
Modules installed successfully.
Modules installed successfully.
Modules installed successfully.
[oracle@ocne-operator ~]$

  8. Show the installed modules.

    olcnectl module instances --config-file myenvironment.yaml

    Example Output:

    [oracle@ocne-operator ~]$ olcnectl module instances --config-file myenvironment.yaml
INSTANCE       	MODULE    	STATE    
10.0.12.11:8090	node      	installed
10.0.12.12:8090	node      	installed
10.0.12.13:8090	node      	installed
10.0.12.21:8090	node      	installed
10.0.12.22:8090	node      	installed
10.0.12.23:8090	node      	installed
mycluster      	kubernetes	installed
myhelm         	helm      	installed
mymetallb      	metallb   	installed
[oracle@ocne-operator ~]$

Create a Kubernetes Application

In this section, you create a Kubernetes application that uses a LoadBalancer service.

On any control plane node:

  1. Create a Kubernetes application.

    tee echo-oci-lb.yml > /dev/null << 'EOF'
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: echo-deployment
  labels:
    app: echo1
spec:
  replicas: 2
  selector:
    matchLabels:
      app: echo1
  template:
    metadata:
      labels:
        app: echo1
    spec:
      containers:
      - name: echoserver
        image: k8s.gcr.io/echoserver:1.4
        ports:
        - containerPort: 80
---
kind: Service
apiVersion: v1
metadata:
  name: echo-lb-service
spec:
  selector:
    app: echo1
  type: LoadBalancer
  ports:
  - name: http
    port: 80
    targetPort: 8080
EOF

  2. Create the service.

    kubectl create -f echo-oci-lb.yml

    Example Output:

    [oracle@ocne-control01 ~]$ kubectl create -f echo-oci-lb.yml
deployment.apps/echo-deployment created
service/echo-lb-service created
[oracle@ocne-control01 ~]$

  3. Confirm the Kubernetes deployment is running.

    kubectl get deployments

    Example Output:

    [oracle@ocne-control01 ~]$ kubectl get deployment      
NAME              READY   UP-TO-DATE   AVAILABLE   AGE
echo-deployment   2/2     2            2           3m15s
[oracle@ocne-control01~]$

  4. Show the Kubernetes service is running.

    kubectl get svc

    Example Output:

    [oracle@ocne-control01 ~]$ kubectl get svc
NAME              TYPE           CLUSTER-IP     EXTERNAL-IP   PORT(S)        AGE
echo-lb-service   LoadBalancer   10.111.72.49   10.0.12.240   80:31727/TCP   4m47s
kubernetes        ClusterIP      10.96.0.1      <none>        443/TCP        58m
[oracle@ocne-control01 ~]$

    Notice the EXTERNAL-IP for the echo-lb-service LoadBalancer has an IP address of 10.0.12.240. This IP address is provided by MetalLB and is the external IP address that you can use to connect to the application.

Testing the Deployment

The next step is to test the newly deployed application. As the EXTERNAL-IP value is dynamically provisioned by MetalLB (in this scenario between the range 10.0.12.240-10.0.12.250), the first two steps store this dynamic value as operating system variables.

  1. (On any control plane node) Capture the assigned EXTERNAL-IP address.

    LB=$(kubectl get svc -o jsonpath="{.status.loadBalancer.ingress[0].ip}" echo-lb-service)

  2. Capture the port number assigned.

    LBPORT=$(kubectl get svc -o jsonpath="{.spec.ports[0].port}" echo-lb-service)

  3. Confirm these are stored as environment variables.

    echo $LB
echo $LBPORT

    Example Output:

    [oracle@ocne-control01 ~]$ echo $LB
10.0.12.240
[oracle@ocne-control01 ~]$ echo $LBPORT
80
[oracle@ocne-control01 ~]$

  4. Use curl to connect to the deployed application.

    curl -i -w "\n" $LB:$LBPORT

    Example Output:

    [oracle@ocne-control01 ~]$ curl -i -w "\n" $LB:$LBPORT
HTTP/1.1 200 OK
Server: nginx/1.10.0
Date: Wed, 10 Aug 2022 10:52:10 GMT
Content-Type: text/plain
Transfer-Encoding: chunked
Connection: keep-alive

CLIENT VALUES:
client_address=10.244.2.0
command=GET
real path=/
query=nil
request_version=1.1
request_uri=http://10.0.12.240:8080/

SERVER VALUES:
server_version=nginx: 1.10.0 - lua: 10001

HEADERS RECEIVED:
accept=*/*
host=10.0.12.240
user-agent=curl/7.61.1
BODY:
-no body in request-
[oracle@ocne-control01 ~]$

Test From a Non-Kubernetes Host

  1. (On ocne-operator) This last test confirms that MetalLB is responding to external requests.

    curl -v http://10.0.12.240:80

    Example Output:

    [oracle@ocne-operator ~]$ curl -v http://10.0.12.240:80
* Rebuilt URL to: http://10.0.12.240:80/
*   Trying 10.0.12.240...
* TCP_NODELAY set
* Connected to 10.0.12.240 (10.0.12.240) port 80 (#0)
> GET / HTTP/1.1
> Host: 10.0.12.240
> User-Agent: curl/7.61.1
> Accept: */*
> 
< HTTP/1.1 200 OK
< Server: nginx/1.10.0
< Date: Wed, 10 Aug 2022 11:38:08 GMT
< Content-Type: text/plain
< Transfer-Encoding: chunked
< Connection: keep-alive
< 
CLIENT VALUES:
client_address=10.244.0.0
command=GET
real path=/
query=nil
request_version=1.1
request_uri=http://10.0.12.240:8080/

SERVER VALUES:
server_version=nginx: 1.10.0 - lua: 10001

HEADERS RECEIVED:
accept=*/*
host=10.0.12.240
user-agent=curl/7.61.1
BODY:
* Connection #0 to host 10.0.12.240 left intact
[oracle@ocne-operator ~]$

This confirms that MetalLB has been configured correctly, an application deployed and is accepting requests successfully.

For More Information

SSR