Use a Container to Create a DNF or ULN Repo Mirror

Introduction

Oracle Linux 8 includes a fully functional reposync tool for DNF, that makes it easy to create a mirror of any yum repository. This facility can be extended to also mirror ULN channels for environments where the majority of your systems do not have direct access to the internet. By creating a yum mirror of the yum repositories and ULN channels that your organization uses, you can reduce network overhead and improve yum performance across your environment. Yum mirrors are also useful if you are configuring other services for your environment such as offline Ksplice.

While previous releases of Oracle Linux included a uln-yum-mirror package that could be used to perform mirroring services, this was not particularly efficient and was relatively complicated to set up.

A ULN or yum mirror service is a typical example of a service that is best run within a set of containers. By using either Docker or Podman, you can quickly and easily deploy a container that uses the oraclelinux:8-slim image to handle scheduled synchronization of the yum repositories or ULN channels that you use within your organization. You can also deploy a container that handles the provisioning of the mirrored repositories within a web service that client systems are able to access.

An opensource GitHub project provides the Dockerfiles, scripts and instructions to do set up this kind of service at https://github.com/Djelibeybi/oraclelinux-reposync .

Objectives

In this lab, you'll learn to:

Install container-tools
Build a container with the required software
Create a mirror service using reposync within the container

Prerequisites

A system with Oracle Linux 7 or Oracle Linux 8 installed and with access to the internet.
This tutorial assumes that you are using Oracle Linux 8, so package install commands use dnf, if you are using Oracle Linux 7, substitute these command instructions with yum.
In this tutorial, we use Podman and Buildah to handle the build and running of containers, but if you are using Oracle Linux 7, you can equally use Docker to perform these tasks.

Install required packages

If not already connected, open a terminal and connect via ssh to the ol-node01 system:

ssh oracle@<ip_address_of_ol-node01>

Install git so that you can clone the container-reposync repository:

sudo dnf install -y git

Install podman and related utilities:

sudo dnf module install -y container-tools:ol8

Note: If you are using Oracle Linux 7, the buildah and skopeo packages are only available via the unsupported EPEL repositories. For this reason, you may choose to use Docker for the purpose of building and running your container images. More details on using Docker are found in the opensource project README.md .

Clone the container-reposync repository

Clone the repo:

git clone https://github.com/Djelibeybi/oraclelinux-reposync.git
cd oraclelinux-reposync

Build the images

Build the two container images included in the repo:

buildah build-using-dockerfile -t ol-repo-sync .
buildah build-using-dockerfile -t ol-repo-web -f Dockerfile.nginx .

Verify the build:

podman images

Create additional storage directories

Two additional directories need to be created for the container-reposync service to function correctly:

rhn: If you intend to mirror ULN channels you must create a directory to store your ULN registration information to make it persistent across subsequent container restarts. You can map this directory into your container whenever you run it, so that it has access to ULN registration data.
repo: A directory must be created to store all of the packages and metadata that you mirror from ULN or the yum server. The file system hosting this based directory needs enough available disk space to cater for all of the repositories and channels that you intend to mirror. You may choose to mount dedicated storage to this location or to map to an alternate location if requred.

These directories can be located wherever you like, but it is advised that you create them alongside the configuration information for this container. For example:

mkdir rhn repo

Set the configuration variables for reposync

The ol-repo-sync image depends on configuration information that is stored in various configuration files stored in the config directory.

config/uln.conf: This file stores ULN access credentials such as Oracle SSO credentials and an active CSI. The file permissions must be set to restrict access. You do not need this file if you don't intend to mirror from ULN.
config/repo-map.json: This file is used to construct repository URLs and to identify how to access different ULN channels and yum repositories. You should not edit this file, but you can refer to it to see what channel names are available for mirroring. Note that this file may need to be updated from time to time to account for new ULN channels. Furthermore, some entries in this file are only available in ULN. including all repos that contain the word base, patch, ksplice, JavaSE and Exadata. These repos may be further restricted to specific CSIs.
config/repos.json: This file is used to identify which ULN channels or yum repositories should be mirrored. It is separated into two sections: the first listing ULN channels and the second listing yum repositories. If you only intend to mirror yum repositories, remove the ULN section entirely. The ULN section should only contain channels that are not available as repositories on https://yum.oracle.com as it is more eefficient to mirror directly from the yum server. The majority of channels are also available on the Oracle Linux yum server, so use this section to list your channels whenever possible. If you add entries to the ULN section, you must have a ULN account and you must register the container image with ULN.

Note: An Oracle Linux support subscription is required to sync from ULN. If you do not have a support subscription, remove the uln array completely from the config/repos.json.

For the best sync performance, use the yum source instead of uln wherever possible, as yum.oracle.com leverages the Akamai CDN and will almost always have much higher download speeds than ULN.

(Optional) If you have an active Oracle Linux support subscription:
```
cp config/uln.sample.conf config/uln.conf
```
Replace the placeholders with Oracle SSO credentials and an active CSI. To protect the content of this file, run:
```
chmod 400 config/uln.conf
```
This prevents anyone except yourself from access.
Create a config/repo-map.json file by running the following command:
```
podman run --rm -it \
  --name ol-repo-sync \
  -v ${PWD}/config:/config:z \
  -v ${PWD}/repo:/repo:z \
  ol-repo-sync update
```
This command can be run again at any time if you want to update the config/repo-map.json file with the latest repo configuration. The command should at least be run whenever a new update or major version is released so that the new repos are available for syncing.
Copy repos.json
```
cp config/repos.sample.json config/repos.json
```
Add all the repos you want to sync to either the uln or yum array.
Example:
Here is a script that syncs the Oracle Linux 8 Ksplice aware userspace packages from ULN and the Oracle Linux Automation Manager packages from yum.oracle.com. If you do not have a ULN account during this lab, remove the entire uln block.
```
echo '{
   "uln": [
        "ol8_x86_64_userspace_ksplice",
        "ol8_aarch64_userspace_ksplice"
    ],   
    "yum": [
        "ol8_x86_64_automation"
    ]
}' | tee config/repos.json
```

(Optional) Register your container with ULN

If you do not intend to mirror any channels from ULN, you do not need to register your container. If you have entered your ULN credentials into the ULN configuration file and you have created a directory to contain your ULN registration data and you have configured at least one ULN channel in the repo configuration file, you must register the container.

Registration can be performed by running:

podman run --rm -it \
  -v ${PWD}/rhn:/etc/sysconfig/rhn:z \
  -v ${PWD}/config:/config:z \
  -v ${PWD}/repo:/repo:z \
  ol-repo-sync register

Note:* This will take a few minutes with no output to the terminal but should return to the command prompt when completed. The rhn, and config directories in the current working directory are mapped into the container. You only need to perform registration once for the container as long as the rhn directory is mapped to /etc/sysconfig/rhn for each subsequent container that you run.

Populate your mirror repository

To populate the mirror repository with packages from the configured repositories and channels, run:

podman run --rm -it \
  -v ${PWD}/rhn:/etc/sysconfig/rhn \
  -v ${PWD}/config:/config \
  -v ${PWD}/repo:/repo:z \
  ol-repo-sync

The container automatically adds and subscribes each channel configured in config/repos.json and create an identical hierarchy to that used by the Oracle Linux yum server.

This command can be scheduled to run on a recurring schedule using a cronjob or systemd timer.

Note: This step takes a while to complete as the packages are all downloaded local to your system.

Serve the local yum mirror to client systems

Use the ol-repo-web container image to serve the yum repositories to your client systems. This container can run permanently and can be configured to start at boot:

podman run --detach --name yum-server \
  -p 8080:80 \
  -v ${PWD}/repo:/var/www/html/repo:ro \
  ol-repo-web

Note: The repo directory is mapped into the yum-server container with read-only permissions to allow the container to continue to run and serve clients while the mirrored repositories and channels are updated.

Verify the local yum mirror

Create a new dnf repository entry with the following content:

echo '[ol_automation_http_repo]
name=OL_automation_x86_64_HTTP
baseurl="http://localhost:8080/repo/OracleLinux/OL8/automation/$basearch/"
gpgcheck=0' | sudo tee /etc/yum.repo.d/ol-local.repo

Note: If exposing to external systems, change the baseurl above to the IP address or hostname of the system running the container.
Also open the firewall to allow access to port 8080.
sudo firewall-cmd --permanent --add-port=8080/tcp
sudo firewall-cmd --reload

Then confirm the new entry works:

dnf repolist

dnf info ansible

For More Information

See other related resources: