Tmpfs Tools Profile Sync Daemon (PSD): Difference between revisions

From WikiMLT
m (Text replacement - "I/O Monitoring and Analyze" to "Linux I/O Monitoring and Analyze")
 
(8 intermediate revisions by the same user not shown)
Line 4: Line 4:
It is designed to work on desktop systems, but by creating custom fake profiles it could be used for non production servers :) Currently I'm using it to minimize [[Kali Linux Install Brave browser|Brave browser]] disk writings. More information about the PSD's Brave profile could be foun in the references section below.
It is designed to work on desktop systems, but by creating custom fake profiles it could be used for non production servers :) Currently I'm using it to minimize [[Kali Linux Install Brave browser|Brave browser]] disk writings. More information about the PSD's Brave profile could be foun in the references section below.


== Preparation ==
{{collapse/begin}}
Identify which directories has high [[I/O Monitoring and Analyze|I/O rate]] by the following command.<syntaxhighlight lang="shell" line="1" class="force-prompt">
== Identify the Problem ==
{{collapse/div|#Preparation details}}
Identify which directories has high [[Linux I/O Monitoring and Analyze|I/O rate]] by the following command.<syntaxhighlight lang="shell" line="1" class="force-prompt">
sudo watch -d -n 1 "find ~/.cache ~/.config -type f -size +80k -mmin -10 -printf '%-30s \t %t %p\n'"
sudo watch -d -n 1 "find ~/.cache ~/.config -type f -size +80k -mmin -10 -printf '%-30s \t %t %p\n'"
</syntaxhighlight>Check the current size of the identified directories.<syntaxhighlight lang="shell" line="1" class="force-prompt">
</syntaxhighlight>Check the current size of the identified directories.<syntaxhighlight lang="shell" line="1" class="force-prompt">
Line 12: Line 14:
iostat -h /dev/nvme0n1 -d 60 -t
iostat -h /dev/nvme0n1 -d 60 -t
</syntaxhighlight>
</syntaxhighlight>
{{collapse/end}}


== Install log2ram ==
== Install PSD ==
<syntaxhighlight lang="shell" line="1" class="force-prompt">
The PSD tool is available as [https://github.com/graysky2/profile-sync-daemon#installation-from-distro-packages package] for many Linux distributions. Here is how to install on Debian/Kali.<syntaxhighlight lang="shell" line="1" class="force-prompt">
echo "deb [signed-by=/usr/share/keyrings/azlux-archive-keyring.gpg] http://packages.azlux.fr/debian/ bullseye main" | sudo tee /etc/apt/sources.list.d/azlux.list
sudo apt install profile-sync-daemon
sudo wget -O /usr/share/keyrings/azlux-archive-keyring.gpg  https://azlux.fr/repo.gpg
</syntaxhighlight>However the Debian's package is little bit older but works stable. If you want the latest version [https://github.com/graysky2/profile-sync-daemon/blob/master/INSTALL install] it from its GitHub repository by performing the following steps.<syntaxhighlight lang="shell" line="1" class="force-prompt">
sudo apt update
cd ~/Git # or ~/Downloads or where you prefer
sudo apt install log2ram
git clone https://github.com/graysky2/profile-sync-daemon.git
cd profile-sync-daemon
make
sudo make install
</syntaxhighlight>
== Setup PSD ==
Within the latest version on GitHub is available a [https://github.com/graysky2/profile-sync-daemon/blob/master/contrib/brave profile for Brave browser] also there is a [https://github.com/graysky2/profile-sync-daemon/pull/240 discussion] about it. However I prefer to create two custom profiles: 1) the first will handle <code>~/.config/BraveSoftware/Brave-Browser</code>, and 2) the second will handle <code>~/.cache/BraveSoftware/Brave-Browser</code>.<syntaxhighlight lang="shell" line="1">
echo -e 'DIRArr[0]="$XDG_CONFIG_HOME/BraveSoftware/Brave-Browser"\nPSNAME="brave"' | \
sudo tee /usr/share/psd/browsers/brave
echo -e 'DIRArr[0]="'"$HOME"'/.cache/BraveSoftware/Brave-Browser"\nPSNAME="brave-cache"' | \
sudo tee /usr/share/psd/browsers/brave-cache
</syntaxhighlight>The next step is to add brave and brave-cache to the PSD user's configuration file - <code>psd.conf</code>.<syntaxhighlight lang="shell" line="1">
echo -e '\nBROWSERS="brave brave-cache"' >> ~/.config/psd/psd.conf
</syntaxhighlight>I prefer also to enable the use of <code>overlayfs</code> mode to improve sync speed and to use a smaller memory footprint. The first thing according to that is to add entry in the <code>psd.conf</code> file, and also an entry into the <code>sudoers.d</code> directory, because this feature requires more permissions. <syntaxhighlight lang="shell" line="1">
echo -e '\nUSE_OVERLAYFS="yes"' >> ~/.config/psd/psd.conf
echo "$USER ALL=(ALL) NOPASSWD: /usr/bin/psd-overlay-helper" | \
sudo tee -a /etc/sudoers.d/psd-overlay-helper
</syntaxhighlight>
</syntaxhighlight>


== Setup log2ram ==
* Note it is better and save to use just <code>echo "$USER ALL=(ALL) NOPASSWD: /usr/bin/psd-overlay-helper"</code> to generate the line and the use <code>sudo visudo /etc/sudoers.d/psd-overlay-helper</code>.
<syntaxhighlight lang="shell" line="1" class="force-prompt">
In addition you can enable the suspend-sync feature.<syntaxhighlight lang="shell" line="1">
sudo nano /etc/log2ram.conf
echo -e '\nUSE_SUSPSYNC="yes"' >> ~/.config/psd/psd.conf
</syntaxhighlight><syntaxhighlight lang="bash" line="1">
</syntaxhighlight>This feature requires <code>glib2</code> library, it could be installed on nowadays Debian/Ubuntu/Kali/etc. via the following [https://askubuntu.com/a/58357/566421 command].<syntaxhighlight lang="shell" line="1">
SIZE=2G
sudo apt install libglib2.0-dev
MAIL=true
</syntaxhighlight>
PATH_DISK="/var/log"
== Usage ==
ZL2R=false
You can use <code>man psd</code> to get detailed information. However here are few essential commands.
COMP_ALG=lz4
LOG_DISK_SIZE=100M


</syntaxhighlight>Modify the default systemd timer unit, if you want to log more or less frequently than one time per day.<syntaxhighlight lang="shell" line="1" class="force-prompt">
'''PREVIEW MODE.''' The preview option can be called to show users exactly what <code>psd</code> will do/is doing based on the entries in the config file. It will also provide useful information such as profile size, paths, and if any recovery snapshots have been created.<syntaxhighlight lang="shell" line="1">
sudo systemctl edit log2ram-daily.timer
psd p
</syntaxhighlight>Reboot the system. After the reboot check [https://github.com/azlux/log2ram#is-it-working does it working] and use the <code>iostat</code> command to take a new statistic.
</syntaxhighlight><syntaxhighlight lang="terraform">
Profile-sync-daemon v6.45
 
systemd service: active
resync-timer:    active
sync on sleep:  disabled
use overlayfs:  enabled
 
Psd will manage the following per /home/<user>/.config/psd/.psd.conf:
 
browser/psname:  brave/brave
owner/group id:  <user>/1000
sync target:    /home/<user>/.config/BraveSoftware/Brave-Browser
tmpfs dir:      /run/user/1000/psd/<user>-brave
profile size:    420M
overlayfs size:  85M
recovery dirs:  none
 
browser/psname:  brave-cache/brave-cache
owner/group id:  <user>/1000
sync target:     /home/<user>/.cache/BraveSoftware/Brave-Browser
tmpfs dir:      /run/user/1000/psd/<user>-brave-cache
profile size:    277M
overlayfs size:  21M
recovery dirs:  none
 
</syntaxhighlight>'''START AND STOP PSD.''' Psd  ships  with a systemd user service to start or stop it (<code>psd.service</code>). Additionally, a provided <code>resync-timer</code> will run an hourly <code>resync</code> from <code>tmpfs</code> back to the disk. The <code>resync-timer</code> is started automatically with <code>psd.service</code> so there is no need to start the timer; only start <code>psd.service</code>.<syntaxhighlight lang="shell" line="1">
systemctl --user status psd.service
systemctl --user start psd.service
systemctl --user stop psd.service
systemctl --user enable psd.service
systemctl --user disable psd.service
</syntaxhighlight>Tweak the <code>rsync-timer</code>.<syntaxhighlight lang="shell" line="1">
systemctl --user cat psd-resync.timer
systemctl --user edit psd-resync.timer
</syntaxhighlight>'''CLEAN MODE.''' The clean mode will delete ALL recovery snapshots that have accumulated. Only run it when sure these are no longer needed.<syntaxhighlight lang="shell" line="1">
psd c
</syntaxhighlight>


== References ==
== References ==
 
* GitHub: [https://github.com/graysky2/profile-sync-daemon Profile-sync-daemon (psd)] | [https://github.com/graysky2/profile-sync-daemon/pull/240 Add Brave Browser #240]
* GitHub: [https://github.com/azlux/log2ram log2ram]
* Arch Linux Wiki: [https://wiki.archlinux.org/title/Profile-sync-daemon Profile-sync-daemon]
* ArchLinux Wiki: [https://wiki.archlinux.org/title/Firefox/Profile_on_RAM Firefox/Profile on RAM]
* Arch Linux Wiki: [https://wiki.archlinux.org/title/Firefox/Profile_on_RAM Firefox/Profile on RAM]
* Linuxhowto.net: [https://www.linuxhowto.net/how-to-write-log-files-in-ram-using-log2ram-in-linux/ Write Log Files In RAM Using Log2ram In Linux]<br />
* Arch Linux Wiki: [https://wiki.archlinux.org/title/Chromium#Cache_in_tmpfs Chromium/Cache in tmpfs]
* Arch Linux Wiki: [https://wiki.archlinux.org/title/XDG_Base_Directory XDG Base Directory]<br />


<noinclude>
<noinclude>
Line 47: Line 102:
  | Прндл  = Linux Desktop
  | Прндл  = Linux Desktop
  | Прндл1 = Linux Server
  | Прндл1 = Linux Server
  | Стадий = 3
  | Стадий = 6
  | Фаза  = Разработване
  | Фаза  = Утвърждаване
  | Статус = Разработван
  | Статус = Утвърден
  | ИдтПт  = Spas
  | ИдтПт  = Spas
  | РзбПт  = {{REVISIONUSER}}
  | РзбПт = Spas
| АвтПт  = Spas
| УтвПт = {{REVISIONUSER}}
  | ИдтДт  = 31.08.2022
  | ИдтДт  = 31.08.2022
  | РзбДт  = {{Today}}
  | РзбДт = 31.08.2022
| АвтДт  = 31.08.2022
| УтвДт = {{Today}}
  | ИдтРв  = [[Special:Permalink/30872|30872]]
  | ИдтРв  = [[Special:Permalink/30872|30872]]
  | РзбРв  = {{REVISIONID}}
  | РзбРв = [[Special:Permalink/30883|30883]]
| АвтРв  = [[Special:Permalink/30884|30884]]
| УтвРв = {{REVISIONID}}
}}
}}
</div>
</div>
</noinclude>
</noinclude>

Latest revision as of 10:18, 3 September 2022

Pro­file-sync-dae­mon (psd) is a tiny pseu­do-dae­mon de­signed to man­age your browser's pro­file in tmpfs and to pe­ri­od­i­cal­ly sync it back to your phys­i­cal disc (HDD/SSD). This is ac­com­plished via a sym­link­ing step and an in­no­v­a­tive use of rsync to main­tain back-up and syn­chro­niza­tion be­tween the two. One of the ma­jor de­sign goals of psd is a com­plete­ly trans­par­ent user ex­pe­ri­ence.

It is de­signed to work on desk­top sys­tems, but by cre­at­ing cus­tom fake pro­files it could be used for non pro­duc­tion servers :) Cur­rent­ly I'm us­ing it to min­i­mize Brave brows­er disk writ­ings. More in­for­ma­tion about the PSD's Brave pro­file could be foun in the ref­er­ences sec­tion be­low.

Iden­ti­fy the Prob­lem

#Prepa­ra­tion de­tails
Iden­ti­fy which di­rec­to­ries has high I/O rate by the fol­low­ing com­mand.
sudo watch -d -n 1 "find ~/.cache ~/.config -type f -size +80k -mmin -10 -printf '%-30s \t %t %p\n'"
Check the cur­rent size of the iden­ti­fied di­rec­to­ries.
du -hs ~/.cache ~/.config
Take a sta­tis­tic for the av­er­age disk write per minute be­fore the set­up.
iostat -h /dev/nvme0n1 -d 60 -t

In­stall PSD

The PSD tool is avail­able as pack­age for many Lin­ux dis­tri­b­u­tions. Here is how to in­stall on Debian/​​​Kali.

sudo apt install profile-sync-daemon

How­ev­er the Debian's pack­age is lit­tle bit old­er but works sta­ble. If you want the lat­est ver­sion in­stall it from its GitHub repos­i­to­ry by per­form­ing the fol­low­ing steps.

cd ~/Git # or ~/Downloads or where you prefer
git clone https://github.com/graysky2/profile-sync-daemon.git
cd profile-sync-daemon
make
sudo make install

Set­up PSD

With­in the lat­est ver­sion on GitHub is avail­able a pro­file for Brave brows­er al­so there is a dis­cus­sion about it. How­ev­er I pre­fer to cre­ate two cus­tom pro­files: 1) the first will han­dle ~/.config/BraveSoftware/Brave-Browser, and 2) the sec­ond will han­dle ~/.cache/BraveSoftware/Brave-Browser.

echo -e 'DIRArr[0]="$XDG_CONFIG_HOME/BraveSoftware/Brave-Browser"\nPSNAME="brave"' | \
sudo tee /usr/share/psd/browsers/brave
echo -e 'DIRArr[0]="'"$HOME"'/.cache/BraveSoftware/Brave-Browser"\nPSNAME="brave-cache"' | \
sudo tee /usr/share/psd/browsers/brave-cache

The next step is to add brave and brave-cache to the PSD user's con­fig­u­ra­tion file – psd.conf.

echo -e '\nBROWSERS="brave brave-cache"' >> ~/.config/psd/psd.conf

I pre­fer al­so to en­able the use of over­layfs mode to im­prove sync speed and to use a small­er mem­o­ry foot­print. The first thing ac­cord­ing to that is to add en­try in the psd.conf file, and al­so an en­try in­to the sudoers.d di­rec­to­ry, be­cause this fea­ture re­quires more per­mis­sions.

echo -e '\nUSE_OVERLAYFS="yes"' >> ~/.config/psd/psd.conf
echo "$USER ALL=(ALL) NOPASSWD: /usr/bin/psd-overlay-helper" | \
sudo tee -a /etc/sudoers.d/psd-overlay-helper
  • Note it is bet­ter and save to use just echo "$USER ALL=(ALL) NOPASS­WD: /us­r/bin/psd-over­lay-helper" to gen­er­ate the line and the use su­do vi­su­do /etc/sudoers.d/psd-overlay-helper.

In ad­di­tion you can en­able the sus­pend-sync fea­ture.

echo -e '\nUSE_SUSPSYNC="yes"' >> ~/.config/psd/psd.conf

This fea­ture re­quires glib2 li­brary, it could be in­stalled on nowa­days Debian/​​​Ubuntu/​​​Kali/​​​etc. via the fol­low­ing com­mand.

sudo apt install libglib2.0-dev

Us­age

You can use man psd to get de­tailed in­for­ma­tion. How­ev­er here are few es­sen­tial com­mands.

PRE­VIEW MODE. The pre­view op­tion can be called to show users ex­act­ly what psd will do/​​​is do­ing based on the en­tries in the con­fig file. It will al­so pro­vide use­ful in­for­ma­tion such as pro­file size, paths, and if any re­cov­ery snap­shots have been cre­at­ed.

psd p
Profile-sync-daemon v6.45

 systemd service: active
 resync-timer:    active
 sync on sleep:   disabled
 use overlayfs:   enabled

Psd will manage the following per /home/<user>/.config/psd/.psd.conf:

 browser/psname:  brave/brave
 owner/group id:  <user>/1000
 sync target:     /home/<user>/.config/BraveSoftware/Brave-Browser
 tmpfs dir:       /run/user/1000/psd/<user>-brave
 profile size:    420M
 overlayfs size:  85M
 recovery dirs:   none

 browser/psname:  brave-cache/brave-cache
 owner/group id:  <user>/1000
 sync target:     /home/<user>/.cache/BraveSoftware/Brave-Browser
 tmpfs dir:       /run/user/1000/psd/<user>-brave-cache
 profile size:    277M
 overlayfs size:  21M
 recovery dirs:   none

START AND STOP PSD. Psd  ships  with a sys­temd user ser­vice to start or stop it (psd.service). Ad­di­tion­al­ly, a pro­vid­ed re­sync-timer will run an hourly re­sync from tmpfs back to the disk. The re­sync-timer is start­ed au­to­mat­i­cal­ly with psd.service so there is no need to start the timer; on­ly start psd.service.

systemctl --user status psd.service
systemctl --user start psd.service
systemctl --user stop psd.service
systemctl --user enable psd.service
systemctl --user disable psd.service

Tweak the rsync-timer.

systemctl --user cat psd-resync.timer
systemctl --user edit psd-resync.timer

CLEAN MODE. The clean mode will delete ALL re­cov­ery snap­shots that have ac­cu­mu­lat­ed. On­ly run it when sure these are no longer need­ed.

psd c

Ref­er­ences