Use a Container to Create a DNF or ULN Repository Mirror

Introduction

Oracle Linux includes a fully functional reposync tool for DNF, making creating a mirror of any yum repository easy. We can also extend this facility to mirror ULN channels for environments where the majority of your systems do not have direct access to the internet. By creating a yum mirror of your organization's yum repositories and ULN channels, you can reduce network overhead and improve yum performance across your environment. Yum mirrors are also useful if you are configuring other services for your environment, such as offline Ksplice.

While previous releases of Oracle Linux included a uln-yum-mirror package that you could use to perform mirroring services, this was not particularly efficient and was relatively complicated to set up.

A ULN or yum mirror service is a typical example of a service that is best run within a set of containers. Using Podman, you can quickly and easily deploy a container that uses the Oracle Linux slim image to handle scheduled synchronization of the yum repositories or ULN channels that you use within your organization. You can also deploy a container that handles the provisioning of the mirrored repositories within a web service that client systems are able to access.

An opensource GitHub project provides the Dockerfiles, scripts, and instructions to set up this kind of service at https://github.com/Djelibeybi/oraclelinux-reposync .

Objectives

In this tutorial, you'll learn to:

Install container-tools
Build a container with the required software
Create a mirror service using reposync within the container

Prerequisites

Minimum of a single Oracle Linux system
Each system should have Oracle Linux installed and configured with:
- A non-root user account with sudo access
- Access to the Internet

If you are using Oracle Linux 7, substitute these dnf command instructions with yum, and use Docker to perform those tasks referencing podman or buildah.

Deploy Oracle Linux

Note: If running in your own tenancy, read the linux-virt-labs GitHub project README.md and complete the prerequisites before deploying the lab environment.

Open a terminal on the Luna Desktop.

Clone the linux-virt-labs GitHub project.

git clone https://github.com/oracle-devrel/linux-virt-labs.git

Change into the working directory.
```
cd linux-virt-labs/ol
```

Install the required collections.

ansible-galaxy collection install -r requirements.yml

Deploy the lab environment.
```
ansible-playbook create_instance.yml -e localhost_python_interpreter="/usr/bin/python3.6"
```
The free lab environment requires the extra variable local_python_interpreter, which sets ansible_python_interpreter for plays running on localhost. This variable is needed because the environment installs the RPM package for the Oracle Cloud Infrastructure SDK for Python, located under the python3.6 modules.
The default deployment shape uses the AMD CPU and Oracle Linux 8. To use an Intel CPU or Oracle Linux 9, add -e instance_shape="VM.Standard3.Flex" or -e os_version="9" to the deployment command.
Important: Wait for the playbook to run successfully and reach the pause task. At this stage of the playbook, the installation of Oracle Linux is complete, and the instances are ready. Take note of the previous play, which prints the public and private IP addresses of the nodes it deploys and any other deployment information needed while running the lab.

Install Required Packages

Open a terminal and connect via SSH to the ol-node-01 instance:

ssh oracle@<ip_address_of_instance>

Install Git so that you can clone the container-reposync repository:

sudo dnf install -y git

Install podman and related utilities:

Oracle Linux 8:

sudo dnf module install -y container-tools:ol8

Oracle Linux 9:

sudo dnf install -y container-tools

Note: If you are using Oracle Linux 7, the buildah and skopeo packages are only available via the unsupported EPEL repositories. For this reason, you may choose to use Docker for the purpose of building and running your container images. More details on using Docker are found in the opensource project README.md .

Clone the container-reposync Repository

Clone the repository:

git clone https://github.com/Djelibeybi/oraclelinux-reposync.git
cd oraclelinux-reposync

Build the Images

Build the two container images included in the repo:

buildah build-using-dockerfile -t ol-repo-sync .
buildah build-using-dockerfile -t ol-repo-web -f Dockerfile.nginx .

Verify the build:

podman images

Create Additional Storage Directories

You need to create two additional directories for the container-reposync service to function correctly:

rhn: If you intend to mirror ULN channels, you must create a directory to store your ULN registration information to make it persistent across subsequent container restarts. You can map this directory into your container whenever you run it so that it has access to ULN registration data.
repo: You must create a directory to store all the packages and metadata you mirror from ULN or the yum server. The file system hosting this based directory needs enough available disk space to cater to all of the repositories and channels that you intend to mirror. You may mount dedicated storage to this location or map to an alternate location if required.

The location of these directories can be wherever you like, but we advise you to create them alongside the configuration information for this container. For example:

mkdir rhn repo

Set the Configuration Variables for Reposync

The ol-repo-sync image depends on configuration information stored in various configuration files in the config directory.

config/uln.conf: This file stores ULN access credentials such as Oracle SSO credentials and an active CSI. You must set the file permissions to restrict access. You do not need this file if you don't intend to mirror from ULN.
config/repo-map.json: The image uses this file to construct repository URLs and to identify how to access different ULN channels and yum repositories. You should not edit this file, but you can refer to it to see what channel names are available for mirroring. Note that this file may need to be updated from time to time to account for new ULN channels. Furthermore, some entries in this file are only available in ULN. It includes all repos containing the words base, patch, ksplice, JavaSE, and Exadata. ULN may further restrict these repos to specific CSIs.
config/repos.json: This file identifies which ULN channels or yum repositories you'll mirror. It contains two sections: the first listing ULN channels and the second listing yum repositories. If you only intend to mirror yum repositories, remove the ULN section entirely. The ULN section should only contain channels not available as repositories on https://yum.oracle.com as it is more efficient to mirror directly from the yum server. The majority of channels are also available on the Oracle Linux yum server, so use this section to list your channels whenever possible. If you add entries to the ULN section, you must have a ULN account and register the container image with ULN.

Note: An Oracle Linux support subscription is required to sync from ULN. If you do not have a support subscription, remove the uln array completely from the config/repos.json.

For the best sync performance, use the yum source instead of ULN wherever possible, as yum.oracle.com leverages the Akamai CDN and will almost always have much higher download speeds than ULN.

(Optional) If you have an active Oracle Linux support subscription:
```
cp config/uln.sample.conf config/uln.conf
```
Replace the placeholders with Oracle SSO credentials and an active CSI. To protect the content of this file, run:
```
chmod 400 config/uln.conf
```
This setting prevents anyone except yourself from access.
Create a config/repo-map.json file by running the following command:
```
podman run --rm -it \
  --name ol-repo-sync \
  -v ${PWD}/config:/config:z \
  -v ${PWD}/repo:/repo:z \
  ol-repo-sync update
```
This command can be run again at any time to update the config/repo-map.json file with the latest repo configuration. You should at least run it whenever a new update or major version is released to make the new repos available for syncing.
Copy repos.json
```
cp config/repos.sample.json config/repos.json
```
Add all the repos you want to sync to the uln or yum array.
Example:
Here is a script that syncs the Oracle Linux 8 Ksplice aware userspace packages from ULN and the Oracle Linux Automation Manager packages from yum.oracle.com. If you do not have a ULN account during this lab, remove the entire uln block.
```
echo '{
   "uln": [
        "ol8_x86_64_userspace_ksplice",
        "ol8_aarch64_userspace_ksplice"
    ],   
    "yum": [
        "ol8_x86_64_automation"
    ]
}' | tee config/repos.json
```

(Optional) Register Your Container with ULN

If you do not intend to mirror any channels from ULN, you do not need to register your container. However, if you have entered your ULN credentials into the ULN configuration file, created a directory to contain your ULN registration data, and configured at least one ULN channel in the repo configuration file, you must register the container.

Perform the registration by running the following:

podman run --rm -it \
  -v ${PWD}/rhn:/etc/sysconfig/rhn:z \
  -v ${PWD}/config:/config:z \
  -v ${PWD}/repo:/repo:z \
  ol-repo-sync register

Note:* This will take a few minutes without output to the terminal but should return to the command prompt when completed. Podman maps the rhn and config directories from the current working directory into the container. You only need to perform container registration once so long as the rhn directory is mapped to /etc/sysconfig/rhn for each subsequent time you run the container.

Populate Your Mirror Repository

To populate the mirror repository with packages from the configured repositories and channels, run the following:

podman run --rm -it \
  -v ${PWD}/rhn:/etc/sysconfig/rhn \
  -v ${PWD}/config:/config \
  -v ${PWD}/repo:/repo:z \
  ol-repo-sync

The container automatically adds and subscribes to each channel configured in config/repos.json and creates an identical hierarchy to that used by the Oracle Linux yum server.

You can schedule this command to run on a recurring schedule using a cronjob or systemd timer.

Note: This step takes a while as the packages are downloaded locally to your system.

Serve the Local Yum Mirror to Client Systems

Use the ol-repo-web container image to serve the yum repositories to your client systems. This container can run permanently, and you can configure it to start at boot:

podman run --detach --name yum-server \
  -p 8080:80 \
  -v ${PWD}/repo:/var/www/html/repo:ro \
  ol-repo-web

Note: The repo directory is mapped into the yum-server container with read-only permissions, allowing the container to continue running and serving clients while the mirrored repositories and channels are updated.

Verify the Local Yum Mirror

Create a new dnf repository entry with the following content:

cat << 'EOF' | sudo tee /etc/yum.repos.d/ol-local.repo > /dev/null
[ol_automation_http_repo]
name=OL_automation ($basearch)
baseurl=http://localhost:8080/repo/OracleLinux/OL8/automation/$basearch/
gpgcheck=0
EOF

Note: If exposing to external systems, change the baseurl above to the IP address or hostname of the system running the container.
Also, open the firewall to allow access to port 8080.
sudo firewall-cmd --permanent --add-port=8080/tcp
sudo firewall-cmd --reload

Then confirm the new entry works:

dnf repolist

dnf info ansible

Next Steps

Using reposync makes local mirroring of remote repositories easier. By wrapping it in a container just makes it even easier. With your new-found skills, set up a mirror of your own and begin leveraging its benefits. Check out our content on the Oracle Linux Training Station for more learning opportunities related to DNF and containers.