System Logging on Oracle Linux

Introduction

In this tutorial, you configure system logging, use rsyslog templates to format log messages, install and run logwatch, view the journald journal, and configure persistent journald storage.

Background

System log files contain messages about the system, kernel, services, and applications. The journald logging daemon, which is part of systemd, records system messages in non-persistent journal logs in memory and in the /run/log/journal directory. journald forwards messages to the system logging daemon, rsyslog. As files in /run are volatile, the log data is lost after a reboot unless you create the directory /var/log/journal. You can use the journalctl command to query the journal logs.

Objectives

Explore the rsylog.conf file
Explore the logrotate.conf file
Configure and use rsyslog templates
Install logwatch and run the logwatch utility
Explore and use journald

What Do You Need?

A fully patched Oracle Linux or later system.

Note: When using the free lab environment, see Oracle Linux Lab Basics for connection and other usage instructions.

Explore and Modify the `rsyslog.conf` File

Oracle Linux relies on the service, rsyslogd, to facilitate logging. This daemon permits the processing and forwarding of logs on your system. /etc/rsyslog.conf file is the main configuration file for system logging.

Watch the video below for an overview on rsyslog.

Use the less command to explore the system logging configuration file.
```
less /etc/rsyslog.conf
```
Use the up-arrow and down-arrow keys to view the various sections of the file.
- The MODULES section uses the $ModLoad directive to load the modules.
- The GLOBAL DIRECTIVES section specifies configuration options.
- The RULES section defines a facility (facility.priority) and an action.
Press the q key to exit the less command.
Use the ls command to list the cron* files in the /var/log directory.
```
ls /var/log/cron*
```
You might not have any other log files at this time, or files with a date stamp.
Use the vi editor to change cron logging to use a new log file: /var/log/cron_new.
```
vi /etc/rsyslog.conf
```
- Search for cron stuff in the file. You can do that by typing /; followed by, cron stuff.
```
cron stuff
```
- Enter insert mode by typing i.
- On the next line, change /var/log/cron to /var/log/cron_new, as shown:
```
/var/log/cron_new
```
- Save and close the file by typing ESC, :wq! and ENTER.
Use the systemctl command to restart the rsyslog service.
```
systemctl restart rsyslog
```
Use the crontab -e command to create a cron job that runs the ls command every minute.
```
crontab -e
```
- Enter insert mode by typing i.
- Enter the following:
```
* * * * * ls
```
Type ESC, :wq! and ENTER to save and close the file.
Use the ls command to list the cron files in the /var/log directory.
```
ls /var/log/cron*
```
Note: The command shows the new cron log file.
Use the tail command to view the last entries in the cron log file.
```
tail -2 /var/log/cron
```
Notice from the date_time stamps of the log entries.
Use the tail command to view the lastest entries in the cron_new log file.
```
tail -2 /var/log/cron_new
```
Notice from the date_time stamps that the new log entries are being written to cron_new.
Use the vi editor to change cron logging back to /var/log/cron.
```
vi /etc/rsyslog.conf
```
- Enter the following:
```
/var/log/cron
```
Use the systemctl command to restart the rsyslog service.
```
systemctl restart rsyslog
```
Use the tail command to ensure that cron is now logging events to /var/log/cron. Compare the output from running the command on the following files:
```
tail -2 /var/log/cron
```
```
tail -2 /var/log/cron_new
```
Notice that the date_time stamp for cron is a later time stamp than cron_new.
Use the rm command to delete the cron_new log file.
```
rm /var/log/cron_new
```
Use crontab –r to remove the crontab.
```
crontab -r
```
Use the vi command to modify the system logging configuration file by adding an entry at the bottom of the file to log all debug messages to /var/log/debug.
```
vi /etc/rsyslog.conf
```
- Enter insert mode.
- Enter the following:
```
*.debug             /var/log/debug
```
Run the command systemctl to restart the rsyslog service.
```
systemctl restart rsyslog
```
Use the logger command to generate an informational log message. The logger command logs enteries in the system log.
```
logger -p info "This is an info-priority message"
```
The logger command is a quick way to add messages to the /var/log/syslog file from the command line`
Run the tail command to view the log files.
```
tail /var/log/messages
```
```
tail /var/log/debug
```
The message appears in both log files.
Use the logger command to generate a debug-level log message.
```
logger -p debug "This is an debug-priority message"
```
Run the tail command to view the log files.
```
tail /var/log/messages
```
```
tail /var/log/debug
```
The message appears in only the /var/log/debug log file.

Explore and Modify Log Rotation

Oracle linux installs the logrotate utility to rotate logs. logrotate places a logrotate.conf configuration file the /etc directory at install time. The logwatch configuration file contains the directives for the default rotation frequency. You can modify the frequency of the log rotation by editing this configuration file.

Use the ls command to view represtations of rotated logs in the /var/log directory.
```
ls /var/log/messages*
```
```
ls /var/log/maillog*
```
```
ls /var/log/cron*
```
- In a production system, files with numbers at the end of the file name represent rotated logs with time stamps added to the log file names.
- The rotated log files provide a history of events, and the files without time stamps provide a record of the most current events.
- The number of rotated files depend on the how long the system has been running.
Use the vi editor to modify the /etc/logrotate.conf configuration file to change the frequency of the default log file rotation from weekly to daily.
```
vi /etc/logrotate.conf
```
- Enter the following:
```
daily
```
You can create separate configuration files for any specific log file in the /etc/logrotate.d directory. These files can define any of the configuration options. These options override the global options in /etc/logrotate.conf. Oracle Linux provides a few separate configuration files by default.
Use the cat command to explore the options used in these files.
```
cat /etc/logrotate.d/chrony
```
```
cat /etc/logrotate.d/syslog
```
```
cat /etc/logrotate.d/firewalld
```
The options in the /etc/logrotate.d/configuration files are described as follows:
- missingok: If the log file is missing, do not issue an error message.
- nocreate: New log files are not created.
- postrotate/endscript:The lines between these directives are executed after the log file is rotated.
- sharedscripts:The postrotatescript runs only once, not once for each log that is rotated.
Note: For a full list of directives and configuration options, refer to the logrotate(8)man page.

Use `rsyslog` Templates

Templates allow you to specify and format rsyslogd output the way a user might want. A template consists of a template directive, a name, the actual template text and optional options.

A sample syntax is:

$template TEMPLATE_NAME,"text %PROPERTY% text", [OPTION]

The fields are:

$template: Directive that notifies rsyslog this line is a template
TEMPLATE_NAME: Defines the name of the template
"text": Actual template text surrounded by quotation marks
%PROPERTY%: Specific message content surrounded by percent signs
OPTION: Specifies options that modify the template functionality

Use the command vi /etc/rsyslog.conf to define a template.
- Add the template definition line shown in the code box to the bottom of the file.
- This entry creates a template named lab.
- Do not exit the vi editor.
```
vi /etc/rsyslog.conf
```
- Enter the following:
```
$template lab,"Message: %msg%\n"
```
Continue editing /etc/rsyslog.conf and create a log file that uses the template.
- Add a new line after the template definition line.
- This entry writes all messages to the /var/log/lab.log file and formats the entries by using the log template.
- Exit the vi editor and save the file after adding the new line.
- Enter the following:
```
*.*     /var/log/lab.log;lab
```
After saving the changes to /etc/rsyslog.conf, use the systemctl command to restart the rsyslog service.
```
systemctl restart rsyslog
```
Use the cat command to view the /var/log/lab.log file.
```
cat /var/log/lab.log
```
The lab.log file contains all entries preceded by the text “Message:” followed by the actual message, as defined in the lab template.

Use the vi editor and change the lab template definition in /etc/rsyslog.conf, as shown:

vi /etc/rsyslog.conf

Enter the following:

$template lab, "Time: %timestamp%, Facility: %syslogfacility-text%, Priority: %syslogpriority-text%, Hostname: %hostname%, Message: %msg%\n"

After saving the changes, restart the rsyslog service.
```
systemctl restart rsyslog
```
Rerun the cat command and view the entries now in lab.log.
The newest entries now include the Time, Facility, Priority, Hostname, and Message properties, as defined in the class template.

Install and Use `logwatch`

logwatch is a customizable log monitoring system. It analyzes system logs for a given time period and reports on specific areas of interest.

It might be necessary to install the logwatch packages. After installing it, logwatch is configured by default to run each night as defined in /etc/cron.daily/0logwatch and email a report to the root user.

Watch the video below for an overview on logwatch.

Run the command dnf list logwatch to verify its availability.
```
dnf list logwatch
```
The output lists the logwatch packages as Available Pachages only, which means you need to install the utiltiy. The packages are availabe in the ol8_baseos_latest repository located in the /etc/yum.repos.d/oracle-linux-ol8.repo file. This repository is enabled by default.
Run the command dnf install logwatch -y to install the logwatch packages.
```
dnf install logwatch -y
```
Use the find command to list all logwatch files.
```
find / -name "*logwatch*"
```
The output lists several files:
- The main configuration file is /usr/share/logwatch/default.conf/logwatch.conf.
- Local configuration options can be set in /etc/logwatch/conf/logwatch.conf.
- AHOWTO-Customize-Logwatchfile file exists in the /usr/share/doc/logwatch/ directory. This file describes the structure of the Logwatch files in this version, how to modify the configuration files for your system, and how to create new service filters.
Use the less command to view the logwatch configuration file.
```
less /usr/share/logwatch/default.conf/logwatch.conf
```
Review this file and note the various configuarable items, such as:
- LogDir: Is the default log directory, and all files are placed relative to this directory
- TmpDir: Is a temporary directory you define to override the default /tmp directory
- MailTo: Is the user to mail the reports to. root is the default recipient.
- MailFrom: Is the default person to mail reports from. Logwatch is the default sender.
- Range: Is the default time range for the report. yesterday is the default range.
- Detail: Is the default detail level for the report. Detail levels can be set to: Low or O, Med or 5, or High or 10. Low is the default detail level.
- Service: Is the default services to report on as defined in /usr/share/logwatch/scripts/services/. All services is the default. You can also disable certain services even when specifying All.
Use the less command to view the logwatch cron file.
```
less /etc/cron.daily/0logwatch
```
Run logwatch --help command to view the logwatch help.
```
logwatch --help
```

Run logwatch with the --range option and a date_range parameter of today. This will process log data from the current day. Sample output is displayed.

logwatch --range today

Sample output:

 ################### Logwatch 7.4.3 (04/27/16) #################### 
      Processing Initiated: Thu Oct 27 14:55:54 2021
      Date Range Processed: today
                           ( 2021-Oct-27 )
                           Period is day.
      Detail Level of Output: 0
      Type of Output/Format: stdout / text
      Logfiles for Host: ol-serverver
 ##################################################################

Use `journald`

Log files can also be managed by the journald daemon, which is part of systemd. The full service name is systemd-journald.service, and the full daemon name is systemd-journald.

You use journalctl to query the systemd journal logs. By default, the listed entries include a time stamp, the host name, the application involved, and the given message.

The main configuration file for systemd-journald is /etc/systemd/journald.conf.

In this practice, you use the journalctl command to query the systemd journal, view journald metadata, and enable persistent journald storage. Your log output might vary.

Watch the video below for an overview on journald and journalctl.

Use the journalctl command with no options or arguments to query the systemd journal.
```
journalctl
```
Note the following:
- Enter q to exit journalctl.
- All log data is displayed, including rotated logs.
- The beginning of the boot process is indicated with a special entry.
- Entries with error priority and higher are in red.
- Entries with notice and warning priority are in bold font (which might not be visible in your terminal window; opening a terminal window in the GNOME GUI will show the bold font).
Run the journalctl -h command to display usage and query options.
```
journalctl -h
```
Run the journalctl -r command to display the newest log entries first.
```
journalctl -r
```
Run the journalctl –n 3 command to display the three newest log entries.
```
journalctl -n 3
```
The -n option displays the <number> of the most recent log entries specified.
Run the journalctl with the --since option to view log entries in the last 10 minutes.
```
journalctl --since  "5 minutes ago"
```
The output displays log enteries within the specified range.
You can use the -u option to display log entries for any systemd unit.
```
journalctl -u crond
```

Type the journalctl command, add a space, and then press the Tab key twice to display the metadata fields.

journalctl <TAB> <TAB>

Sample output:

_AUDIT_LOGINUID=              JOB_RESULT=                   SSSD_DOMAIN=
_AUDIT_SESSION=               JOB_TYPE=                     _STREAM_ID=
AVAILABLE=                    JOURNAL_NAME=                 SYSLOG_FACILITY=
AVAILABLE_PRETTY=             JOURNAL_PATH=                 SYSLOG_IDENTIFIER=

Run the journalctl -o verbose command to display log entries in verbose format.
- Verbose format shows the metadata fields and values for all journal entries.
- You can use the -o option to display log entries in any supported format. Supported format options are short, short-iso, short-precise, short-monotonic, verbose, export, json, json-pretty, and cat.
```
journalctl -o verbose
```
Run the journalctl -f command to display log entries as they are being added.
```
journalctl -f
```
Enter CTRL + C to terminate the command.
Run the journalctl –p crit command to display log entries with a priority of crit, alert, or emerg.
- It is likely your lab system has not logged entries with any of these priorities.
- You can use the -p option to display log entries of any priority. Valid priorities are debug, info, notice, warning, err, crit, alert, and emerg.
- Without specifying a range with the -p option, log entries at the given level and those with greater significance are shown (a range has the form FROM..TO, for example, crit..alert).
```
journalctl -p crit
```
Run the journalctl -p warning command to view the entries reported.
```
journalctl -p warning
```
Run the journalctl –F _UID command to display unique values for the _UID metadata field.
```
journalctl -F _UID
```
Run the journalctl _UID=<value> command to show only log entries that match the condition. The example uses the _UID=0
```
journalctl _UID=0
```
Use the mount command, pipe the output to grep, and search for the string “run”.
```
mount | grep run
```
The output shows that the file system type mounted on /run is tmpfs.
Use the ls -l command to view the /run/log/journal/ directory.
```
ls -l /run/log/journal/
```
By default, journal data is stored in this directory. Also notice the date_time stamp on this directory.
Use the mkdir -p command to make the /var/log/journal directory.
```
mkdir -p /var/log/journal
```
Use the systemctl command to restart the systemd-journald service.
```
systemctl restart systemd-journald
```
Use the ls -l command to view the /var/log/journal directory.
```
ls -l /var/log/journal/
```
Journal data is now stored in this directory. Also notice the date_time stamp on this directory is more recent than the date_time stamp on /run/log/journal.