Linux I/O Monitoring and Analyze
There is a couple of tools available that allows you to monitor and analyze the disk I/O performance of your Linux driven system. Here are listed few of them and also how to install and examples of their basic usage.
The htop command
If a newer version of htop is available at your distribution, there is available an additional tab that shows the I/O metrics of the instance – Screen 1. Here is how to check the available version and install htop.
sudo apt show htop 2>/dev/null | grep '^Version'
sudo apt install htop
Install the latest version of htop 3.2.1–1 on Ubuntu Server 22.04.1 from a .deb package.
cd /tmp
wget --no-check-certificate https://http.us.debian.org/debian/pool/main/h/htop/htop_3.2.1-1_amd64.deb
sudo apt install ./htop_3.2.1-1_amd64.deb
To be able to see all data in most cases you need to run the tool as root:
sudo htop
htop (v 3.2+). Use Tab to switch to the I/O tab, then use F6 to open the Sort by menu, and sort by IO_WRITE_RATE. The screenshot is taken on Kali Linux 2022. 
The iostat command
…
The iotop command
…
The dstat command
dstat is a versatile tool for generating system resource statistics, it is a versatile replacement for vmstat, iostat and ifstat. Dstat is unique in letting you aggregate block device throughput for a certain diskset or network bandwidth for a group of interfaces, ie. you can see the throughput for all the block devices that make up a single filesystem or storage system.
sudo apt install dstat
There is an uncountable multitude of options and plugins available for dstat. Here is one example of usage – Screen 3 – where are applied the following options.
-D sdc– adds column that reports the I/O rate of/dev/sdc.-t,–time– enable time/date output.-a,–all– equals to-cdngy(-ccpu,-ddisk;-nenable network stats;-genable page stats;-yenable system stats).–top-io– show most expensive I/O process.–top-bio– show most expensive block I/O process.–top-mem– show process using the most memory.
sudo dstat -D sda -ta --top-io --top-bio --top-mem
dstat command. 
Here is another example that will output the average I/O rate per minute.
dstat -tdD total 60
----system--------dsk/total--
time | read writ
¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯
29-08 08:40:13 | 1138M 1782M
29-08 08:41:13 | 234k 744k
29-08 08:42:13 | 293k 171k
29-08 08:43:13 | 268k 113k
29-08 08:44:13 | 1100k 129k
The vmstat command
vmstat – virtual memory statistics – reports information about processes, memory, paging, block IO, traps, disks and cpu activity. The first report produced gives averages since the last reboot. Additional reports give information on a sampling period of length delay. The process and memory reports are instantaneous in either case. Here is how to get statistics about the block devices.
sudo apt install sysstat
sudo sed -i 's/ENABLED="false"/ENABLED="true"/' /etc/default/sysstat
sudo systemctl enable --now sysstat.service
The sar command
The sar command is part of the package sysstat. It outputs the contents of selected cumulative activity counters in the operating system. The activities are collected by the sysstat.service. After installing the package we need to enable the collector service and wait until some statistics are collected.
sudo apt install sysstat
sudo sed -i 's/ENABLED="false"/ENABLED="true"/' /etc/default/sysstat
sudo systemctl enable --now sysstat.service
systemctl cat sysstat-collect.timer
# /lib/systemd/system/sysstat-collect.timer
# /lib/systemd/system/sysstat-collect.timer
# (C) 2014 Tomasz Torcz <tomek@pipebreaker.pl>
#
# sysstat-12.5.2 systemd unit file:
# Activates activity collector every 10 minutes
[Unit]
Description=Run system activity accounting tool every 10 minutes
[Timer]
OnCalendar=*:00/10
[Install]
WantedBy=sysstat.service
sar
Linux 5.15.39-4-pve (ubuntu-lxc-pve) 08/28/22 _x86_64_ (24 CPU)
20:41:48 LINUX RESTART (24 CPU)
20:50:05 CPU %user %nice %system %iowait %steal %idle
21:00:00 all 1.66 0.00 0.26 0.02 0.00 98.06
21:10:10 all 2.66 0.00 0.27 0.03 0.00 97.03
21:20:13 all 1.92 0.00 0.29 0.02 0.00 97.76
Average: all 2.09 0.00 0.27 0.03 0.00 97.62
Monitor the Files Size Changes Recursively
By the following command we can monitor which are the most written files for the past 10 minutes, larger than 800 Kb. This is done recursively for the directories /var/lib and /var/log. The output of the command is shown at Screen 3.
sudo watch -n 3 -d \
"find /var/lib /var/log -type f -size +800k -mmin -10 -printf '%-30s \t %t %p\n' | grep -Pv '\.(gz|[0-9])$'"
watch and find to monitor file change in real time. 
Here is an advanced version :) which outputs also an additional data generated by iostat:
sudo watch -n 3 -d \
"find /var/lib /var/log -type f -size +800k -mmin -10 -printf '%-30s \t %t %p\n' | grep -Pv '\.(gz|[0-9])$';
echo;
iostat /dev/sda2"
Every 3.0s: find /var/lib /var/log -type f -size +800k -mmin -10 -printf '%s \t %t %p\n'...; iostat /dev/sda2...
4362053 Mon Aug 29 08:25:15.1573410540 2022 /var/lib/redis/dump.rdb
67108864 Mon Aug 29 08:29:34.6284410810 2022 /var/lib/mysql/undo_001
3276800 Mon Aug 29 08:29:36.1604593970 2022 /var/lib/mysql/#innodb_redo/#ib_redo10127
83886080 Mon Aug 29 08:29:33.6604295080 2022 /var/lib/mysql/mysql.ibd
31459279 Mon Aug 29 08:29:34.2284362990 2022 /var/lib/mysql/binlog.005694
5242880 Mon Aug 29 08:29:33.6284291260 2022 /var/lib/mysql/SCloud/oc_authtoken.ibd
6291456 Mon Aug 29 08:29:34.6284410810 2022 /var/lib/mysql/SCloud/oc_jobs.ibd
50331648 Mon Aug 29 08:29:34.6284410810 2022 /var/lib/mysql/undo_002
79691776 Mon Aug 29 08:29:34.6284410810 2022 /var/lib/mysql/ibdata1
11873325 Mon Aug 29 08:29:01.5600457430 2022 /var/log/syslog
31441772 Mon Aug 29 08:29:08.0921238280 2022 /var/log/auth.log
1504046 Mon Aug 29 08:25:15.2373420090 2022 /var/log/redis/redis-server.log
8388608 Mon Aug 29 08:27:04.5226471030 2022 /var/log/journal/e8dsfe54457bd2f6a44344e1/user-1000.journal
33554432 Mon Aug 29 08:29:08.5241289930 2022 /var/log/journal/e8dsfe54457bd2f6a44344e1/system.journal
1183995 Mon Aug 29 08:25:17.2973666000 2022 /var/log/apache2/wiki.error.log
1170720 Mon Aug 29 08:25:17.3013666480 2022 /var/log/apache2/wiki.access.log
1166389 Mon Aug 29 08:25:14.0613279700 2022 /var/log/apache2/cloud.access.log
1243871 Mon Aug 29 08:25:13.9613267760 2022 /var/log/apache2/bg.mirror.access.log
Linux 5.15.0-46-generic (szs.space) 08/29/22 _x86_64_ (16 CPU)
avg-cpu: %user %nice %system %iowait %steal %idle
2.50 0.00 0.73 1.06 0.00 95.71
Device tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd
sda2 55.71 454.89 741.44 421.11 32361849 52747168 29958424
References
- Htop: GitHub | Home page [
htop] - Unix and Linux: How can I monitor disk I/O? [
sar,iostat,iotop] - Unix and Linux: How do I measure total disk I/O per hour? [
dstat,iostat] - Proxmox Forum: Disk prevent from spinning down because of pvestatd [
dstat] - PVE HDD Sleep (Suspend)
- Preload Tool for Better System Performance
- Linux Swap and Swapfile
