Work With Performance Co-Pilot on Oracle Linux

Introduction

Performance Co-Pilot (PCP) collects OS and network metrics that you can use to diagnose performance issues. PCP is a distributed architecture with collector hosts and monitor hosts.

PCP collector hosts are hosts being monitored. These hosts require a Performance Metrics Domain Agent (PMDA) which is responsible for collecting performance measurements. All PMDAs are controlled by the Performance Metrics Collector Daemon (pmcd) on the same hosts.

PCP monitor hosts display data collected from hosts, or archives, that have the PCP collector installed. Many monitor tools are included with the core PCP package.

Objectives

In this lab, you will learn:

Configure one Oracle Linux host as a PCP collector and monitor
Use PCP monitoring tools on a single host
Configure remote monitoring of a 2nd Oracle Linux host
Monitor performance with pmchart GUI

Prerequisites

Two (2) Oracle Linux 9 systems.

Setup Lab Environment

Note: When using the free lab environment, see Oracle Linux Lab Basics for connection and other usage instructions.

Open a terminal and connect via SSH to the ol-node-01 instance.
```
ssh oracle@<ip_address_of_ol-node-01>
```
Install the pcp, pcp-system-tools, and pcp-gui packages on the ol-node-01 instance.
```
sudo dnf install pcp pcp-system-tools pcp-gui
```
The pcp package provides the basic PCP tools and services for collecting performance data. The pcp-system-tools and pcp-gui packages are optionally installed on the monitoring host to extend the set of monitoring tools from the base pcp package.
Enable and start the Performance Metrics Collector Daemon (pmcd) and the Performance Metrics Logger (pmlogger) collector host services.
```
sudo systemctl enable --now pmcd pmlogger
```

Use PCP monitoring tools on a single host

Run the pcp utility on the ol-node-01 instance.

sudo pcp

The default output summarizes the PCP installation on the local hosts.

Example Output:

[oracle@ol-node-01 ~]$ sudo pcp
Performance Co-Pilot configuration on ol-node-01:

 platform: Linux ol-node-01 5.15.0-204.147.6.2.el9uek.x86_64 #2 SMP Thu Mar 7 22:09:50 PST 2024 x86_64
 hardware: 4 cpus, 1 disk, 1 node, 31828MB RAM
 timezone: GMT
services: pmcd
    pmcd: Version 6.2.0-1, 12 agents, 6 clients
    pmda: root pmcd proc pmproxy xfs linux nfsclient mmv kvm jbd2
          dm openmetrics
pmlogger: primary logger: /var/log/pcp/pmlogger/ol-node-01/20240508.12.41
    pmie: primary engine: /var/log/pcp/pmie/ol-node-01/pmie.log

Alternatively, you can pass a command to pcp to run, for example, pcp free.

sudo pcp free

Example Output:

[oracle@ol-node-01 ~]$ sudo pcp free
               total        used        free      shared  buff/cache   available
Mem:        32592652     1357328    23443732       27712     7791592    30732656
Swap         6291452           0     6291452

When pcp is invoked with a command to run, for example free, it runs pcp-free which is located in the /usr/libexec/pcp/bin/ directory.

List commands in the /usr/libexec/pcp/bin/ directory that begin with pcp-.

sudo ls /usr/libexec/pcp/bin/pcp-*

Example Output:

[oracle@ol-node-01 ~]$ sudo ls /usr/libexec/pcp/bin/pcp-*
/usr/libexec/pcp/bin/pcp-atop        /usr/libexec/pcp/bin/pcp-pidstat
/usr/libexec/pcp/bin/pcp-atopsar     /usr/libexec/pcp/bin/pcp-ps
/usr/libexec/pcp/bin/pcp-buddyinfo   /usr/libexec/pcp/bin/pcp-python
/usr/libexec/pcp/bin/pcp-dmcache     /usr/libexec/pcp/bin/pcp-reboot-init
/usr/libexec/pcp/bin/pcp-dstat       /usr/libexec/pcp/bin/pcp-shping
/usr/libexec/pcp/bin/pcp-free        /usr/libexec/pcp/bin/pcp-slabinfo
/usr/libexec/pcp/bin/pcp-htop        /usr/libexec/pcp/bin/pcp-ss
/usr/libexec/pcp/bin/pcp-iostat      /usr/libexec/pcp/bin/pcp-summary
/usr/libexec/pcp/bin/pcp-ipcs        /usr/libexec/pcp/bin/pcp-tapestat
/usr/libexec/pcp/bin/pcp-lvmcache    /usr/libexec/pcp/bin/pcp-uptime
/usr/libexec/pcp/bin/pcp-meminfo     /usr/libexec/pcp/bin/pcp-verify
/usr/libexec/pcp/bin/pcp-mpstat      /usr/libexec/pcp/bin/pcp-vmstat
/usr/libexec/pcp/bin/pcp-netstat     /usr/libexec/pcp/bin/pcp-zoneinfo
/usr/libexec/pcp/bin/pcp-numastat

Explore by running pcp and pass different commands, for example, run pcp atop.
Note: Some of these next command examples require you to press CTRL-C to exit.
```
sudo pcp atop
```
Run pcp atopsar.
```
sudo pcp atopsar
```
Run pcp buddyinfo.
```
sudo pcp buddyinfo
```
Run pcp dstat.
```
sudo pcp dstat
```
Note: Each of these pcp- commands has a man page which describes the command and provides usage.
View the man page for different pcp- commands, for example, run man pcp-atop.
```
man pcp-atop
```
Run man pcp-atopsar.
```
man pcp-atopsar
```
Run man pcp-buddyinfo.
```
man pcp-buddyinfo
```
Run man pcp-dstat.
```
man pcp-dstat
```

In addition to the pcp- commands in the /usr/libexec/pcp/bin/ directory, this directory contains many other files provided by the pcp package. Run ls /usr/libexec/pcp/bin/ to view all of the files in this directory.

ls /usr/libexec/pcp/bin

Example Output:

[oracle@ol-node-01 ~]$ ls /usr/libexec/pcp/bin
chkhelp         pcp-numastat      pmgetopt                pmlogger_rewrite
discover        pcp-pidstat       pmhostname              pmlogredact
find-filter     pcp-ps            pmie_check              pmlogreduce
install-sh      pcp-python        pmie_daily              pmlogrewrite
mkaf            pcp-reboot-init   pmie_dump_stats         pmnewlog
newhelp         pcp-shping        pmie_email              pmnsadd
pcp-atop        pcp-slabinfo      pmie_farm               pmnsdel
pcp-atopsar     pcp-ss            pmiestatus              pmnsmerge
pcp-buddyinfo   pcp-summary       pmie_webhook            pmpause
pcp-dmcache     pcp-tapestat      pmlock                  pmpost
pcp-dstat       pcp-uptime        pmlogconf               pmproxy
pcp-free        pcp-verify        pmlogextract            pmsignal
pcp-htop        pcp-vmstat        pmlogger                pmsleep
pcp-iostat      pcp-zoneinfo      pmlogger_check          pmsnap
pcp-ipcs        pmcd              pmlogger_daily          pmwtf
pcp-lvmcache    pmcd_wait         pmlogger_daily_report   runaspcp
pcp-meminfo     pmconfig          pmlogger_farm           telnet-probe
pcp-mpstat      pmcpp             pmlogger_janitor
pcp-netstat     pmfind_check      pmlogger_merge

Note: A man page also exists for many of these commands.

Use PCP monitoring tools from `/usr/bin` on a single host

The pcp package installs many performance monitoring utilities in the /usr/bin directory.

Use the rpm command to list files included in the PCP package that are installed in the /usr/bin directory.

rpm -ql pcp | grep /usr/bin

Example Output:

[oracle@ol-node-01 ~]$ rpm -ql pcp | grep /usr/bin
/usr/bin/dbpmda
/usr/bin/pcp
/usr/bin/pmafm
/usr/bin/pmdate
/usr/bin/pmdiff
/usr/bin/pmdumplog
/usr/bin/pmevent
/usr/bin/pmfind
/usr/bin/pmgenmap
/usr/bin/pmie
/usr/bin/pmie2col
/usr/bin/pmieconf
/usr/bin/pmiectl
/usr/bin/pminfo
/usr/bin/pmjson
/usr/bin/pmlc
/usr/bin/pmlogcheck
/usr/bin/pmlogconf
/usr/bin/pmlogctl
/usr/bin/pmlogdump
/usr/bin/pmlogextract
/usr/bin/pmlogger
/usr/bin/pmloglabel
/usr/bin/pmlogmv
/usr/bin/pmlogpaste
/usr/bin/pmlogredact
/usr/bin/pmlogreduce
/usr/bin/pmlogrewrite
/usr/bin/pmlogsize
/usr/bin/pmlogsummary
/usr/bin/pmprobe
/usr/bin/pmpython
/usr/bin/pmrepconf
/usr/bin/pmsearch
/usr/bin/pmseries
/usr/bin/pmsocks
/usr/bin/pmstat
/usr/bin/pmstore
/usr/bin/pmtrace
/usr/bin/pmval

Explore by running of few of these commands, for example, run pminfo -t to list all of the enabled performance metrics and include a short description of each.

pminfo -t

Example Output:

[oracle@ol-node-01 ~]$ pminfo -t
jbd2.njournals [Count of active JBD2 (Journal Block Device v2) devices]
jbd2.transaction.count [Total transactions committed per journal]
jbd2.transaction.requested [Total journal transactions requested per journal]
jbd2.transaction.max_blocks [Maximum transaction blocks (buffers) per journal]
jbd2.transaction.total.blocks [Total transaction blocks per journal]
jbd2.transaction.total.blocks_logged [Total logged blocks per journal]
jbd2.transaction.total.handles [Total handle count per journal]
jbd2.transaction.total.time.waiting [Total time waiting per journal]
jbd2.transaction.total.time.request_delay [Total request delay per journal]
jbd2.transaction.total.time.running [Total running time per journal]
...
disk.wwid.avg_rqsz [average I/O request size for both reads and writes]
disk.wwid.r_avg_rqsz [average I/O request size for reads to the device]
disk.wwid.w_avg_rqsz [average I/O request size for writes to the device]
disk.wwid.d_avg_rqsz [average I/O request size for discards to the device]
disk.wwid.util [percentage of time device was busy processing requests]
proc.psinfo.age [time in seconds since process was started]
proc.io.total_bytes [total bytes read and written by process]
proc.hog.cpu [average percentage CPU utilization of each process]
proc.hog.mem [sum of resident and swapped memory used by the process]
proc.hog.disk [average I/O rate of each process since it was started]

Note: Some of these next command examples require you to press CTRL-C to exit.

Run the pmrep command to view network interface information.

pmrep network.interface.hw_addr

Example Output:

[oracle@ol-node-01 ~]$ pmrep network.interface.hw_addr
  n.i.hw_addr  n.i.hw_addr
           lo         ens3
                          
  00:00:00:00  02:00:17:0b
  00:00:00:00  02:00:17:0b
  00:00:00:00  02:00:17:0b
^C

Run the pmrep command to view outgoing metrics from the ens3 network interface in real time.

pmrep -i ens3 -v network.interface.out

Example Output:

[oracle@ol-node-01 ~]$ pmrep -i ens3 -v network.interface.out
  n.i.o.bytes  n.i.o.packets  n.i.o.errors  n.i.o.drops  n.i.o.fifo  n.i.o.carrier  n.i.o.compressed
         ens3           ens3          ens3         ens3        ens3           ens3              ens3
       byte/s        count/s       count/s      count/s     count/s        count/s           count/s
          N/A            N/A           N/A          N/A         N/A            N/A               N/A
      279.594          1.997         0.000        0.000       0.000          0.000             0.000
^C

Use the pmval command to view live disk write operations per partition.

pmval disk.partitions.write

Example Output:

[oracle@ol-node-01 ~]$ pmval disk.partitions.write

metric:    disk.partitions.write
host:      ol-node-01
semantics: cumulative counter (converting to rate)
units:     count (converting to count / sec)
samples:   all

              sda1                  sda2                  sda3    
               0.0                   0.0                   0.0    
               0.0                   0.0                   0.0    
               0.0                   0.0                   0.9993 
^C

Run PCP tools against PCP archives

The pmlogger service generates archives of performance metrics. PCP archives are located in /var/log/pcp/pmlogger/<hostname>.

View the archives in the /var/log/pcp/pmlogger/ol-node-01 directory.

ls -l /var/log/pcp/pmlogger/ol-node-01/

Example Output:

[oracle@ol-node-01 ~]$ ls -l /var/log/pcp/pmlogger/ol-node-01/
total 47076
-rw-r--r--. 1 pcp pcp 36889492 May  8 19:12 20240508.18.31.0
-rw-r--r--. 1 pcp pcp     3492 May  8 19:12 20240508.18.31.index
-rw-r--r--. 1 pcp pcp   666790 May  8 19:12 20240508.18.31.meta
-rw-r--r--. 1 pcp pcp      277 May  8 18:31 Latest
-rw-r--r--. 1 pcp pcp    28609 May  8 18:31 pmlogger.log

Note: In this example, the archive name is 20240508.18.31.0. Substitute the actual archive name for your environment in the following steps.

Use the pmdumplog command to view the host, timezone, and time period of the archive.

pmdumplog -L /var/log/pcp/pmlogger/ol-node-01/<your-archive-name>

Example Output:

[oracle@ol-node-01 ~]$ pmdumplog -L /var/log/pcp/pmlogger/ol-node-01/<your-archive-name>
Log Label (Log Format Version 2)
Performance metrics from host ol-node-01
    commencing Wed May  8 18:31:28.901169 2024
    ending     Wed May  8 19:17:58.936852 2024
Archive timezone: GMT
PID for pmlogger: 133063

You can run almost any PCP command against the archive by including the -a option and providing the archive name as an argument.

Run the pcp utility on the archive.

sudo pcp -a /var/log/pcp/pmlogger/ol-node-01/<your-archive-name>

Example Output:

[oracle@ol-node-01 ~]$ sudo pcp -a /var/log/pcp/pmlogger/ol-node-01/<your-archive-name>
Performance Co-Pilot configuration on ol-node-01:

archive: /var/log/pcp/pmlogger/ol-node-01/<your-archive-name>
platform: 
 hardware: 4 cpus, 1 disk, 1 node, 31828MB RAM
 timezone: GMT
pmcd: Version unknown,

List all performance metrics enabled at the time the archive was created.
```
pminfo -a /var/log/pcp/pmlogger/ol-node-01/<your-archive-name>
```
View disk write operations per partition from the archive. Pipe the command to less to show header information.
```
pmval disk.partitions.write -a /var/log/pcp/pmlogger/ol-node-01/<your-archive-name> | less
```
View outgoing metrics from the ens3 network interface from the archive. Pipe the command to less to show header information.
```
pmrep -i ens3 -v network.interface.out -a /var/log/pcp/pmlogger/ol-node-01/<your-archive-name> | less
```

Configure remote monitoring of a 2nd PCP collector host

In this section, you configure the ol-node-02 instance as a PCP collector. You perform additional steps to allow the ol-node-01 instance to monitor performance metrics on the remote host.

Open a new terminal and connect via SSH to the ol-node-02 instance.
```
ssh oracle@<ip_address_of_ol-node-02>
```
Install the pcp package on the ol-node-02 instance.
```
sudo dnf install pcp
```
Enable and start the Performance Metrics Collector Daemon (pmcd) and the Performance Metrics Logger (pmlogger) collector host services.
```
sudo systemctl enable --now pmcd pmlogger
```
Enable the PCP client tools to connect to a remote PMCD by editing the /etc/sysconfig/pmcd file. Change PMCD_LOCAL=1 to PMCD_LOCAL=0. Save the file and exit the vi editor.
```
cd /etc/sysconfig
sudo vi pmcd
```
Run the following command to restart the PMCD.
```
sudo /usr/libexec/pcp/lib/pmcd start
```
The PCP client tools communicate with PMCD over the network using TCP port 44321 by default. Use the firewall-cmd command to add port 44321/tcp.
```
sudo firewall-cmd --add-port=44321/tcp
```
Return to the terminal for the ol-node-01 instance.
Note: All remaining tasks in this lab are performed from the ol-node-01 instance.
Use the firewall-cmd command to add port 44321/tcp.
```
sudo firewall-cmd --add-port=44321/tcp
```
To enable centralized archive collection on the monitoring host, configure pmlogger to fetch performance metrics from collector hosts. Add the ol-node-02 instance to the pmlogger configuration file, /etc/pcp/pmlogger/control and then restart the pmlogger service on the monitoring host.
Use the vi editor to edit /etc/pcp/pmlogger/control.
```
sudo vi /etc/pcp/pmlogger/control
```

While in vi, locate the following section:

# remote host
#remote         n   n   PCP_ARCHIVE_DIR/remote          -r -T24h10m -c config.remote

Uncomment the line beginning with #remote and replace each occurrence of remote in the line with ol-node-02 as follows:
```
ol-node-02         n   n   PCP_ARCHIVE_DIR/ol-node-02          -r -T24h10m -c config.ol-node-02
```
Save the file and exit the vi editor.
Restart the pmlogger service.
```
sudo systemctl restart pmlogger
```

Verify that you now have ol-node-01 and ol-node-02 directories in /var/log/pcp/pmlogger directory.

ls -l /var/log/pcp/pmlogger

Example Output:

[oracle@ol-node-01 ~]$ ls -l /var/log/pcp/pmlogger
total 12
drwxrwxr-x. 2 pcp pcp 4096 May  8 20:26 ol-node-01
drwxrwxr-x. 2 pcp pcp  119 May  8 20:26 ol-node-02
-rw-r--r--. 1 pcp pcp  131 May  8 20:26 pmlogger_check.log
-rw-r--r--. 1 pcp pcp  315 May  8 18:31 pmlogger_check.log.prev

Verify that you can view PCP services on remote host ol-node-02.

sudo pcp -h ol-node-02

Example Output:

[oracle@ol-node-01 ~]$ sudo pcp -h ol-node-02
Performance Co-Pilot configuration on ol-node-02:

platform: Linux ol-node-02 5.15.0-204.147.6.2.el9uek.x86_64 #2 SMP Thu Mar 7 22:09:50 PST 2024 x86_64
 hardware: 4 cpus, 1 disk, 1 node, 31828MB RAM
 timezone: GMT
 services: pmcd
     pmcd: Version 6.2.0-2, 12 agents, 6 clients
     pmda: root pmcd proc pmproxy xfs linux nfsclient mmv kvm jbd2
           dm openmetrics
pmlogger: primary logger: /var/log/pcp/pmlogger/ol-node-02/20240509.12.46
     pmie: primary engine: /var/log/pcp/pmie/ol-node-02/pmie.log

Verify that you can view the PCP services from the local archive for host ol-node-02. View the archives in the /var/log/pcp/pmlogger/ol-node-02 directory.

ls -l /var/log/pcp/pmlogger/ol-node-02/

Example Output:

[oracle@ol-node-01 ~]$ ls -l /var/log/pcp/pmlogger/ol-node-02/
total 3500
-rw-r--r--. 1 pcp pcp 2700040 May  9 13:04 20240509.12.50.0
-rw-r--r--. 1 pcp pcp     492 May  9 13:04 20240509.12.50.index
-rw-r--r--. 1 pcp pcp  374876 May  9 13:04 20240509.12.50.meta
-rw-r--r--. 1 pcp pcp     277 May  9 12:50 Latest
-rw-r--r--. 1 pcp pcp   34923 May  9 12:50 pmlogger.log

Note: In this example, the archive name is 20240509.12.50.0. Substitute the actual archive name for your environment in the following command.

View PCP services from the archive for remote host ol-node-02. Substitute the actual archive name for your environment in the following command.

sudo pcp -a /var/log/pcp/pmlogger/ol-node-02/<your-archive-name>

Example Output:

[oracle@ol-node-01 ~]$ sudo pcp -a /var/log/pcp/pmlogger/ol-node-02/<your-archive-name>
Performance Co-Pilot configuration on ol-node-02:

  archive: /var/log/pcp/pmlogger/ol-node-02/<your-archive-name>
 platform: Linux ol-node-02 5.15.0-204.147.6.2.el9uek.x86_64 #2 SMP Thu Mar 7 22:09:50 PST 2024 x86_64
 hardware: 4 cpus, 1 disk, 1 node, 31828MB RAM
 timezone: GMT
 services: pmcd
     pmcd: Version 6.2.0-2, 12 agents, 4 clients
     pmda: root pmcd proc pmproxy xfs linux nfsclient mmv kvm jbd2
           dm openmetrics
     pmie: primary engine: /var/log/pcp/pmie/ol-node-02/pmie.log

You can run almost any PCP command against the remote host by including the -h option and providing the host name as an argument.

Run the pmstat command for remote host ol-node-02.

pmstat -h ol-node-02

Example Output:

[oracle@ol-node-01 ~]$ pmstat -h ol-node-02
@ Thu May  9 13:19:35 2024
 loadavg                      memory      swap        io    system         cpu
   1 min   swpd   free   buff  cache   pi   po   bi   bo   in   cs  us  sy  id
    0.00      0 29667m   4204  1539m    0    0    0   17  114  170   0   0 100
    0.00      0 29666m   4204  1540m    0    0    0    4  117  164   0   0 100
^C

Run the pmval command to view live disk write operations per partition for remote host ol-node-02.

pmval disk.partitions.write -h ol-node-02

Example Output:

[oracle@ol-node-01 ~]$    pmval disk.partitions.write -h ol-node-02

metric:    disk.partitions.write
host:      ol-node-02
semantics: cumulative counter (converting to rate)
units:     count (converting to count / sec)
samples:   all

              sda1                  sda2                  sda3    
               0.0                   0.0                   1.999  
               0.0                   0.0                   0.0    
               0.0                   0.0                   0.0    
               0.0                   0.0                   0.0    
^C

Monitor performance with pmchart GUI

pmchart is a graphical utility provided by the pcp-gui package. pmchart allows you to plot PCP performance metrics from a live host or from PCP archives. Multiple charts can be displayed simultaneously and many aspects of pmchart can be customized through the interface.

This section of the lab assumes you have two (2) terminal windows open, and that you are logged into the ol-node-01 instance from one terminal window, and that you are logged into the ol-node-02 instance from the second terminal windows.

To begin this lab, logoff from both Oracle Linux instances so that you are at the [luna.user@lunabox Desktop]$ prompt in both terminal windows.
```
exit
```
From one terminal, access the vnc session on the ol-node-01 instance using a SSH tunnel.
```
ssh -L 5910:localhost:5901 oracle@<ip_address_of_ol-node-01>
```
From the 2nd terminal, open vncviewer. Provide the password of oracle when prompted.
```
vncviewer localhost:10
```
The Oracle Linux desktop appears on the ol-node-01 instance. Click on Activities and open a Terminal window.
Enter the command pmchart to open the graphical interface.
```
pmchart
```
From the File menu select the ‘New Chart’ option.
The New Chart dialog box appears. The three tabs on the right side of the window provide the following settings:
- Chart: Properties related to the entire chart (Title, Legend settings, Y-Axis scaling, etc.)
- Metrics: Available metrics (performance data) for plotting in the chart
- Plots: Properties related to each individual chart plot (color, label)
From the Metrics tab, select a performance metric, for example kernel.cpu.util.sys, by drilling down as shown.
Click OK and after a few seconds, data will begin to appear on the chart.
Click the 'Zoom In' button to change the scale for better viewing.
Add the ol-node-02 instance by clicking 'New Chart', then click the 'Add a host' button on the New Chart dialog box. Enter ol-node-02 as the Hostname and click OK.
You can now select available performance metrics from ol-node-02. You can scroll down to view available metrics for ol-node-02, or you can double-click the ol-node-01 entry from the Metrics tab to collapse the metrics for ol-node-01 and view metrics for ol-node-02.
From the Metrics tab, select kernel.cpu.util.sys for ol-node-02 and click OK. You now have a 2nd chart showing the same performance metrics for each host.

Summary

The Performance Co-Pilot (PCP) is a toolkit designed for monitoring and managing system-level performance. Oracle Linux hosts can be configured as PCP collectors, those that collect and extract performance data from various sources, and can be configured as PCP monitors, those that display data collected from hosts or archives that have the PCP collector installed.