Use OpenCost with Oracle Cloud Native Environment

0
0
Send lab feedback

Use OpenCost with Oracle Cloud Native Environment

Introduction

Measuring the real-time costs of running and deploying applications on your Oracle Cloud Native Environment (Oracle CNE) install helps you manage them. OpenCost is a vendor-neutral open-source project that records container and cloud infrastructure costs to facilitate real-time business cost monitoring. You can use the real-time costs it reports to monitor both in-cluster costs, such as CPU, GPU, and memory, and Cloud provider-based costs, such as storage.

Objectives

In this tutorial, you will learn:

  • How to install OpenCost
  • How to install the OpenCost plugin using Krew
  • How to use OpenCost to monitor container and cloud infrastructure costs

Prerequisites

  • Minimum of one Oracle Linux instance

  • Each system should have Oracle Linux installed and configured with:

    • An Oracle user account (used during the installation) with sudo access
    • Key-based SSH, also known as password-less SSH, between the hosts

  • OCI cluster creation requires access to the following resources in an Oracle Cloud Infrastructure tenancy:

    • Virtual cloud network with four subnets
    • Network load balancer
    • Object Storage bucket with minimum 5 GiB available
    • Compute Custom Image
    • Compute Arm Shape for the control plane node
      • VM.Standard.A1.Flex with two OCPU and 12 memory
    • Compute for each additional control plane and worker node
      • VM.Standard.E4.Flex with four OCPU and 64 memory

Configure Oracle CNE

Note: If running in your own tenancy, read the linux-virt-labs GitHub project README.md and complete the prerequisites before deploying the lab environment.

  1. Open a terminal on the Luna Desktop.

  2. Clone the linux-virt-labs GitHub project.

    git clone https://github.com/oracle-devrel/linux-virt-labs.git

  3. Change into the working directory.

    cd linux-virt-labs/ocne2

  4. Install the required collections.

    ansible-galaxy collection install -r requirements.yml

  5. Increase the Boot volume size.

    cat << EOF | tee instances.yml > /dev/null
compute_instances:
  1:
    instance_name: "ocne"
    type: "server"
    boot_volume_size_in_gbs: 128
ocne_type: "oci"
install_ocne_rpm: true
create_ocne_oci_cluster: true
ocne_cluster_name: "mycluster"
num_cp_nodes: 1
num_wk_nodes: 3
update_all: true
EOF

  6. Deploy the lab environment.

    ansible-playbook create_instance.yml -e localhost_python_interpreter="/usr/bin/python3.6" -e "@instances.yml"

    The free lab environment requires the extra variable local_python_interpreter, which sets ansible_python_interpreter for plays running on localhost. This variable is needed because the environment installs the RPM package for the Oracle Cloud Infrastructure SDK for Python, located under the python3.6 modules.

    The default deployment shape uses the AMD CPU and Oracle Linux 8. To use an Intel CPU or Oracle Linux 9, add -e instance_shape="VM.Standard3.Flex" or -e os_version="9" to the deployment command.

    Important: Wait for the playbook to run successfully and reach the pause task. At this stage of the playbook, the installation of the Oracle CNE is complete, and the instances are ready. Take note of the previous play, which prints the public and private IP addresses of the nodes it deploys and any other deployment information needed while running the lab.

Access the Kubernetes Cluster

  1. Open a terminal and connect via SSH to the ocne instance.

    ssh oracle@<ip_address_of_instance>

  2. Get a list of known clusters using the CLI.

    ocne cluster list

  3. Get the location of the kube configuration.

    ocne cluster show -C mycluster

    We use the -C to specify a specific cluster from the cluster list.

  4. Set the KUBECONFIG environment variable.

    export KUBECONFIG=$(ocne cluster show -C mycluster)

  5. Wait for the cluster to stabilize and all pods to report in a running state.

    watch kubectl get pods -A

    Once all the pods show a STATUS of Running, type ctrl-c to exit the watch command.

  6. Confirm how many nodes are present.

    kubectl get nodes

Install Helm

Helm provides an easy way to manage applications installed on your Oracle CNE cluster - in this example, you use it to install OpenCost.

  1. Install Git.

    sudo dnf install -y git

  2. Download the latest version of Helm and install it locally.

    curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3
chmod 700 get_helm.sh
./get_helm.sh

    Note: It is not usually recommended to execute a script without reviewing and understanding it before you execute it. However, this script is widely used and well-documented. Oracle recommends you review it before executing it on your installs.

Install Prometheus

OpenCost uses Prometheus to store scraped metrics for the Kubernetes cluster.

  1. Add the Prometheus Community Helm repo.

    helm repo add prometheus-community https://prometheus-community.github.io/helm-charts

  2. Update the Prometheus Helm repo.

    helm repo update

  3. Install Prometheus and configure it for OpenCost.

    You will install Prometheus in the prometheus-system namespace using the default OpenCost settings.

    helm install prometheus --repo https://prometheus-community.github.io/helm-charts prometheus --namespace prometheus-system --create-namespace --set prometheus-pushgateway.enabled=false --set alertmanager.enabled=false -f https://raw.githubusercontent.com/opencost/opencost/develop/kubernetes/prometheus/extraScrapeConfigs.yaml

    Note: You can use an existing Prometheus instance, please reference the OpenCost documentation for details.

Install OpenCost

  1. Create the OpenCost namespace.

    kubectl create namespace opencost

  2. Add the Helm OpenCost repo.

    helm repo add opencost https://opencost.github.io/opencost-helm-chart

  3. Update the Helm repo.

    helm repo update

  4. Download a default OpenCost values definition file.

    You use the values.yaml file to configure the OpenCost configuration setting and to set the default costings used in the tutorial.

    curl -o local.yaml https://raw.githubusercontent.com/opencost/opencost-helm-chart/refs/heads/main/charts/opencost/values.yaml

  5. Install the OpenCost Helm chart.

    helm install opencost --repo https://opencost.github.io/opencost-helm-chart opencost --namespace opencost -f local.yaml

  6. Port forward the OpenCost UI.

    kubectl port-forward --namespace opencost service/opencost 9003:9090

    Note: If you receive an error notification, this is because Helm has not fully deployed OpenCost. Wait a few seconds and repeat the port-forward command.

    Important: Using a port forward should only be used for internal testing. Production deployments should use an Ingress rule to provide access to OpenCost.

Access the OpenCost UI

  1. Open a new terminal window and configure an SSH tunnel.

    ssh -L 9333:localhost:9003 oracle@<ip_address_of_node>

  2. Open a web browser and enter the URL.

    Hint: You may need to refresh the browser a few times before OpenCost has collected enough data to be displayed.

    http://localhost:9333/allocation?window=48h

    opencost-start

    Where the parameters available are:

    • Date Range - Use this to choose a default period to display from the defaults listed. It is also possible to use the "Date Picker" to determine a custom date range.
    • Breakdown - Use this to choose which Kubernetes resource type to display the associated cost. For example, Cluster, Node, Namespace, Pod, Deployment, etc.
    • Resolution - Determines how the graph displays any returned data. The choice is either per day or cumulatively.
    • Currency - Sets the currency denominator displayed.

    Note: If requested, approve the security warning based on the browser used. For Chrome, click the Advanced button and then the Proceed to localhost (unsafe) link. Also, notice that the Cloud Costs and External Costs links on the webpage are disabled. This behavior happens because you do not integrate OpenCost with an OCI Tenancy in this tutorial. If you need this functionality for your deployment, please reference the OpenCost documentation for details on configuring your OCI tenancy to work with OpenCost.

Install Krew

Krew is a package manager for kubectl plugins, which makes it easy to discover and install them on your system.

  1. Open a new terminal and connect via SSH to the ocne instance.

    ssh oracle@<ip_address_of_instance>

  2. Set the KUBECONFIG environment variable.

    export KUBECONFIG=$(ocne cluster show -C mycluster)

  3. Install the git package.

    sudo dnf install -y git

  4. Download and install the krew plugin.

    (
set -x; cd "$(mktemp -d)" &&
OS="$(uname | tr '[:upper:]' '[:lower:]')" &&
ARCH="$(uname -m | sed -e 's/x86_64/amd64/' -e 's/\(arm\)\(64\)\?.*/\1\2/' -e 's/aarch64$/arm64/')" &&
KREW="krew-${OS}_${ARCH}" &&
curl -fsSLO "https://github.com/kubernetes-sigs/krew/releases/latest/download/${KREW}.tar.gz" &&
tar zxvf "${KREW}.tar.gz" &&
./"${KREW}" install krew
)

  5. Add the krew binary to your PATH environment variable.

    export PATH="${KREW_ROOT:-$HOME/.krew}/bin:$PATH"

    To make this change permanent, update your .bashrc file and restart your shell by logging out and back into the system.

  6. Verify the installation.

    kubectl krew

    Example Output:

    [oracle@ocne ~]$ kubectl krew
krew is the kubectl plugin manager.
You can invoke krew through kubectl: "kubectl krew [command]..."

Usage:
  kubectl krew [command]

Available Commands:
  help        Help about any command
  index       Manage custom plugin indexes
  info        Show information about an available plugin
  install     Install kubectl plugins
  list        List installed kubectl plugins
  search      Discover kubectl plugins
  uninstall   Uninstall plugins
  update      Update the local copy of the plugin index
  upgrade     Upgrade installed plugins to newer versions
  version     Show krew version and diagnostics

Flags:
  -h, --help      help for krew
  -v, --v Level   number for the log level verbosity

Use "kubectl krew [command] --help" for more information about a command.

Install the OpenCost CLI Plugin

  1. Install the plugin.

    kubectl krew install cost

    Example Output:

    [oracle@ocne ~]$ kubectl krew install cost
Updated the local copy of plugin index.
Installing plugin: cost
Installed plugin: cost
\
 | Use this plugin:
 | 	kubectl cost
 | Documentation:
 | 	https://github.com/kubecost/kubectl-cost
 | Caveats:
 | \
 |  | Requires Kubecost (a cluster-side daemon) to be installed in your cluster.
 |  | See https://www.kubecost.com/install for installation instructions.
 | /
/
WARNING: You installed plugin "cost" from the krew-index plugin repository.
   These plugins are not audited for security by the Krew maintainers.
   Run them at your own risk.

  2. Verify the plugin installation.

    kubectl cost version
kubectl cost --help

Using OpenCost CLI Plugin

The basic syntax structure used by kubectl cost maps to the standard Kubernetes concepts of namespace, label, deployment, StatefulSet, etc primarily. It uses the Kubecost APIs to return useful cost data related to the workloads and deployments on your Oracle CNE cluster. The tool is flexible, so let's start with a few examples.

  1. Show the monthly cost projection for all the namespaces in the cluster.

    kubectl cost namespace --opencost true

    Example Output:

    [oracle@ocne ~]$ kubectl cost namespace --opencost true
+-----------------+-------------------+--------------------+-----------------+
| CLUSTER         | NAMESPACE         | MONTHLY RATE (ALL) | COST EFFICIENCY |
+-----------------+-------------------+--------------------+-----------------+
| __idle__        | __idle__          |        1100.228400 |        0.000000 |
| default-cluster | kube-system       |          23.187600 |        0.470162 |
|                 | monitoring        |          11.530800 |        0.296739 |
|                 | kube-flannel      |           9.702000 |        0.031735 |
|                 | prometheus-system |           1.309216 |        1.000000 |
|                 | opencost          |           0.800139 |        0.303188 |
|                 | ocne-system       |           0.079200 |        1.000000 |
+-----------------+-------------------+--------------------+-----------------+
| SUMMED          |                   |        1146.837356 |                 |
+-----------------+-------------------+--------------------+-----------------+

    Where:

    • --opencost true = When defined as true, it configures OpenCost to use the parameters according to the OpenCost default specification. Which is equivalent to providing the command-line options: --service-port 9003 --service-name opencost --kubecost-namespace opencost --allocation-path /allocation/compute.

  2. Show the total cost for the past five days by all deployments with the "app" label.

    kubectl cost label --historical -l app --opencost true

    Example Output:

    [oracle@ocne ~]$ kubectl cost label --historical -l app --opencost true
+-----------------+--------------------------------+--------------------+-----------------+
| CLUSTER         | LABEL:APP                      | MONTHLY RATE (ALL) | COST EFFICIENCY |
+-----------------+--------------------------------+--------------------+-----------------+
| __idle__        | __idle__                       |        1100.224800 |        0.000000 |
| default-cluster | __unallocated__                |          35.937616 |        0.367372 |
|                 | flannel                        |           9.702000 |        0.031831 |
|                 | csi-oci-node                   |           0.450000 |        1.000000 |
|                 | csi-oci-controller             |           0.370800 |        1.000000 |
|                 | kube-prometheus-stack-operator |           0.122400 |        1.000000 |
+-----------------+--------------------------------+--------------------+-----------------+
| SUMMED          |                                |        1146.807616 |                 |
+-----------------+--------------------------------+--------------------+-----------------+

  3. Show the projected monthly cost by deployment, including CPU and CPU cost efficiency details.

    kubectl cost deployment --show-cpu --opencost true

    Example Output:

    [oracle@ocne ~]$ kubectl cost deployment --show-cpu --opencost true
+-----------------+-------------------+------------------------------------+------------+----------+--------------------+-----------------+
| CLUSTER         | NAMESPACE         | DEPLOYMENT                         | CPU        | CPU EFF. | MONTHLY RATE (ALL) | COST EFFICIENCY |
+-----------------+-------------------+------------------------------------+------------+----------+--------------------+-----------------+
| __idle__        | __idle__          | __idle__                           | 528.487200 | 0.000000 |        1100.224800 |        0.000000 |
| default-cluster | kube-system       | __unallocated__                    | 14.860800  | 0.008692 |          17.499600 |        1.359666 |
|                 | monitoring        |                                    | 9.342000   | 0.002225 |          11.311200 |        0.241625 |
|                 | kube-flannel      |                                    | 9.104400   | 0.008750 |           9.702000 |        0.031878 |
|                 | kube-system       | coredns                            | 4.550400   | 0.000750 |           5.313600 |        0.023865 |
|                 | prometheus-system | prometheus-server                  | 0.165600   | 1.000000 |           1.219216 |        1.000000 |
|                 | opencost          | opencost                           | 0.454539   | 0.008000 |           0.800139 |        0.303828 |
|                 | kube-system       | csi-oci-controller                 | 0.075600   | 1.000000 |           0.370800 |        1.000000 |
|                 | monitoring        | promstack-kube-prometheus-operator | 0.010800   | 1.000000 |           0.122400 |        1.000000 |
|                 | prometheus-system | prometheus-kube-state-metrics      | 0.032400   | 1.000000 |           0.097200 |        1.000000 |
|                 | monitoring        | promstack-kube-state-metrics       | 0.028800   | 1.000000 |           0.097200 |        1.000000 |
|                 | ocne-system       | ui                                 | 0.007200   | 1.000000 |           0.054000 |        1.000000 |
|                 |                   | ocne-catalog                       | 0.000000   | 0.000000 |           0.025200 |        1.000000 |
+-----------------+-------------------+------------------------------------+------------+----------+--------------------+-----------------+
| SUMMED          |                   |                                    | 567.119739 |          |        1146.837356 |                 |
+-----------------+-------------------+------------------------------------+------------+----------+--------------------+-----------------+

  4. Show the cost of every pod deployed, including CPU-specific costs.

    kubectl cost pod --show-cpu --opencost true

    Example Output:

    oracle@ocne ~]$ kubectl cost pod --show-cpu --opencost true
+-----------------+-------------------+-------------------------------------------------------+------------+----------+--------------------+-----------------+
| CLUSTER         | NAMESPACE         | POD                                                   | CPU        | CPU EFF. | MONTHLY RATE (ALL) | COST EFFICIENCY |
+-----------------+-------------------+-------------------------------------------------------+------------+----------+--------------------+-----------------+
| __idle__        | __idle__          | __idle__                                              | 540.291130 | 0.000000 |        1124.022678 |        0.000000 |
| default-cluster | kube-system       | kube-apiserver-mycluster-control-plane-q8qjn          | 5.689252   | 0.024160 |           7.020939 |        0.209251 |
|                 | monitoring        | prometheus-promstack-kube-prometheus-prometheus-0     | 4.767026   | 0.007700 |           5.893983 |        1.301700 |
|                 |                   | alertmanager-promstack-kube-prometheus-alertmanager-0 | 4.560417   | 0.000300 |           5.306087 |        0.022176 |
|                 | kube-system       | kube-controller-manager-mycluster-control-plane-q8qjn | 4.552904   | 0.007350 |           4.752000 |        0.048939 |
|                 |                   | coredns-7dcb9db977-4hwdv                              | 2.276452   | 0.001300 |           2.657739 |        0.025254 |
|                 |                   | coredns-7dcb9db977-zfdbf                              | 2.276452   | 0.001400 |           2.657739 |        0.024453 |
|                 |                   | etcd-mycluster-control-plane-q8qjn                    | 2.276452   | 0.024300 |           2.575096 |        0.125228 |
|                 | kube-flannel      | kube-flannel-ds-vl5qd                                 | 2.276452   | 0.017500 |           2.424835 |        0.040974 |
|                 |                   | kube-flannel-ds-7tw4k                                 | 2.276452   | 0.017300 |           2.424835 |        0.041796 |
|                 |                   | kube-flannel-ds-48qk4                                 | 2.276452   | 0.017200 |           2.424835 |        0.040679 |
|                 |                   | kube-flannel-ds-cn46r                                 | 2.276452   | 0.015000 |           2.424835 |        0.033543 |
|                 | kube-system       | kube-scheduler-mycluster-control-plane-q8qjn          | 2.276452   | 0.003100 |           2.340313 |        0.030303 |
|                 | prometheus-system | prometheus-server-6c586d99cd-vcxfq                    | 0.137113   | 1.000000 |           1.209669 |        1.000000 |
|                 | opencost          | opencost-6b8cb599b7-m6pk2                             | 0.455040   | 0.013000 |           0.802560 |        0.310859 |
|                 | kube-system       | csi-oci-controller-7ff494cb89-4ncws                   | 0.075130   | 1.000000 |           0.368139 |        1.000000 |
|                 |                   | csi-oci-node-cqhfh                                    | 0.001878   | 1.000000 |           0.142748 |        1.000000 |
|                 |                   | csi-oci-node-mhts8                                    | 0.001878   | 1.000000 |           0.123965 |        1.000000 |
|                 | monitoring        | promstack-kube-prometheus-operator-7dc5684785-tgdj4   | 0.009391   | 1.000000 |           0.122087 |        1.000000 |
|                 | kube-system       | csi-oci-node-54cz9                                    | 0.001878   | 1.000000 |           0.099548 |        1.000000 |
|                 | monitoring        | promstack-kube-state-metrics-5bd74cc8d9-jch68         | 0.030052   | 1.000000 |           0.099548 |        1.000000 |
|                 | prometheus-system | prometheus-kube-state-metrics-65846b5c64-k6qbf        | 0.031930   | 1.000000 |           0.097670 |        1.000000 |
|                 | kube-system       | oci-cloud-controller-manager-6n4n2                    | 0.031930   | 1.000000 |           0.095791 |        1.000000 |
|                 |                   | kube-proxy-k2tqn                                      | 0.011270   | 1.000000 |           0.090157 |        1.000000 |
|                 |                   | csi-oci-node-l7g44                                    | 0.000000   | 0.000000 |           0.080765 |        1.000000 |
|                 |                   | kube-proxy-rrzls                                      | 0.007513   | 1.000000 |           0.067617 |        1.000000 |
|                 | monitoring        | promstack-prometheus-node-exporter-4jqk7              | 0.020661   | 1.000000 |           0.065739 |        1.000000 |
|                 | kube-system       | kube-proxy-8t48n                                      | 0.005635   | 1.000000 |           0.063861 |        1.000000 |
|                 | ocne-system       | ui-5f59d8454b-p9tz4                                   | 0.007513   | 1.000000 |           0.052591 |        1.000000 |
|                 | kube-system       | kube-proxy-4q5gk                                      | 0.003757   | 1.000000 |           0.046957 |        1.000000 |
|                 | monitoring        | promstack-prometheus-node-exporter-z5vfx              | 0.001878   | 1.000000 |           0.033809 |        1.000000 |
|                 | ocne-system       | ocne-catalog-8c94cc49f-vnhpx                          | 0.000000   | 0.000000 |           0.026296 |        1.000000 |
|                 | monitoring        | promstack-prometheus-node-exporter-q7hm7              | 0.000000   | 0.000000 |           0.026296 |        1.000000 |
|                 |                   | promstack-prometheus-node-exporter-dfj4k              | 0.000000   | 0.000000 |           0.022539 |        1.000000 |
+-----------------+-------------------+-------------------------------------------------------+------------+----------+--------------------+-----------------+
| SUMMED          |                   |                                                       | 578.906797 |          |        1170.664263 |                 |
+-----------------+-------------------+-------------------------------------------------------+------------+----------+--------------------+-----------------+

Next Steps

This tutorial introduced you to using OpenCost with Oracle CNE. It only scratches the surface of how it can help you manage the costs of your Oracle CNE infrastructure and application deployment. Refer to the Opencost documentation for more details.

SSR