SSH Persistent Tunnel and SSHFS Mount via "systemd" units

Lin­ux Serv­er

De­vOps and SRE

Net­work and Se­cu­ri­ty

In this tu­to­r­i­al we will cre­ate a sys­temd ser­vice that will use the tool au­tossh to cre­ate a per­sis­tent tun­nel in or­der to make a ser­vice run­ning on a lo­cal host avail­able on a re­mote host. The steps pro­vid­ed be­low are test­ed with­in the cir­cum­stances of Ubun­tu Serv­er 22.04.

Set­up SSH con­nec­tion and Port for­ward­ing

In the cur­rent sce­nario, so called lo­cal host will es­tab­lish SSH con­nec­tion to the re­mote host. So we will for­ward a re­mote port to lis­ten at our lo­cal port – oth­er­wise said we will bind the re­mote port to the lo­cal port. The ser­vice that I want to for­ward by this tech­nique is Elas­tic­search which cur­rent­ly lis­ten on 127.0.0.1:9200 at the lo­cal host.

Cre­ate ded­i­cat­ed sys­tem user(s)

Here we will cre­ate a sys­tem user named ssh­fwd, ded­i­cat­ed for the ser­vice cre­at­ed in the fol­low­ing step. We will use the same user­name on bot lo­cal and re­mote hosts, – so run the fol­low­ing com­mand at both hosts.

sudo useradd -r -s /bin/false sshfwd

id sshfwd

uid=975(sshfwd) gid=975(sshfwd) groups=975(sshfwd)

• -r – cre­ate sys­tem user (note there is not spec­i­fied home di­rec­to­ry be­cause we do need it),
• -s /​​​bin/​​​false – this will be the de­fault shell for the user.

Gen­er­ate SSH key pair

It is prefer­able to use ED25519 based key, be­cause it is more se­cure and al­so it is faster be­cause is much short­en es­pe­cial­ly than 4096 bit RSA key.

#Ref­er­ences

• SSH Key: Ed25519 vs RSA
• Is it bad that my ed25519 key is so short com­pared to a RSA key?
• Up­grade Your SSH Key to Ed25519
• How to se­cure your SSH serv­er with pub­lic key Ed25519 El­lip­tic Curve Cryp­tog­ra­phy
• The pre­vi­ous ver­sion of the an­swer which us­es RSA key.

Gen­er­ate pub­lic and pri­vate SSH keys with­in your user's ~/.ssh di­rec­to­ry at the lo­cal host. The pub­lic key must be trans­ferred to the re­mote host. The pri­vate key will be used at the lo­cal host.

mkdir -m700 ~/.ssh/sshfwd
ssh-keygen -t ed25519 -C 'sshfwd@local.host' -f ~/.ssh/sshfwd/id_ed25519_to_hostname -q -N ''

ls -la ~/.ssh/sshfwd

total 16
drwx------  2 <user> <user> 4096 Jul 20 21:36 .
drwx------ 12 <user> <user> 4096 Jul 20 21:28 ..
-rw-------  1 <user> <user> 3381 Jul 20 21:36 id_ed25519_to_hostname
-rw-r--r--  1 <user> <user>  742 Jul 20 21:36 id_ed25519_to_hostname.pub

• -q – si­lence; -N '' – emp­ty (with­out) passphrase.

Set­up the Lo­cal host

1. Cre­ate a di­rec­to­ry where the pri­vate key will be ac­ces­si­ble for the ssh­fwd user and copy it there.

mkdir -m700 /etc/ssh/sshfwd
sudo cp ~/.ssh/sshfwd/id_ed25519_to_hostname /etc/ssh/sshfwd/
sudo chown -R sshfwd:sshfwd /etc/ssh/sshfwd

2. Cre­ate the fol­low­ing con­fig­u­ra­tion file.

sudo nano /etc/ssh/ssh_config.d/sshfwd-to-hostname.conf

# This file is loaded by a directive in /etc/ssh/ssh_config
Host sshfwd-to-hostname
    HostName 192.168.2.200
    IdentityFile /etc/ssh/sshfwd/id_ed25519_to_hostname
    User sshfwd
    Port 22
    RemoteForward 172.17.201.1:9200 127.0.0.1:9200
    StrictHostKeyChecking no
    GatewayPorts yes
    Compression yes

Note the val­ue ssh­fwd-to-host­name of the di­rec­tive Host is an alias for the ac­tu­al host­name or IP ad­dress pro­vid­ed by the Host­Name di­rec­tive. In the Ex­am­ple above 192.168.2.200 is the IP ad­dress of the re­mote host.

#SSH Con­fig Ex­pla­na­tion

The di­rec­tive Re­mote­For­ward 172.17.201.1:9200 127.0.0.1:9200 will bind the re­mote port 9200 on the re­mote in­ter­face 172.17.201.1 to the lo­cal port 9200 on the loop­back in­ter­face 127.0.0.1 where our (Elas­tic­search) ser­vice ac­tu­al­ly lis­ten. Sim­pli­fied ver­sion of this di­rec­tive could be Re­mote­For­ward 9200 127.0.0.1:9200 which means the re­mote 0.0.0.0:9200 will be bind to the lo­cal 127.0.0.1:9200.

How­ev­er, In my case, I want to for­ward port 9200 on­ly to a spe­cif­ic (docker's) re­mote in­ter­face and al­so I need to share it with the rest hosts that com­mu­ni­cate with that in­ter­face – oth­er­wise said the re­mote SSH serv­er should be con­fig­ured to op­er­ate as gate­way, this will be done in the fol­low­ing steps.

The di­rec­tive Stric­tHostK­ey­Check­ing no is equiv­a­lent to the CLI op­tion -oStrictHostKeyChecking=no and will cause an au­to­mat­ic ac­cept of the re­mote host's keys.

Set­up the Re­mote host

0. The first thing you need to do is to trans­fer the file id_​​​ed25519_​​​to_​​​hostname.pub gen­er­at­ed above to the re­mote host. In my case I al­ready have an op­er­a­tional SSH con­nec­tion, so rsync is my first choice. Then log-in to the re­mote host – the fol­low­ing steps of this sec­tion are per­formed there.

rsync ~/.ssh/sshfwd/id_ed25519_to_hostname.pub remote.host.com:/tmp/

ssh remote.host.com

1. On the re­mote host, cre­ate a di­rec­to­ry where authorized_​​​keys file for the ssh­fwd user will be stored and use the the pub­lic key copied in the step above to cre­ate it.

mkdir -m700 /etc/ssh/sshfwd
sudo mv /tmp/id_ed25519_to_hostname.pub /etc/ssh/sshfwd/authorized_keys
sudo chown -R sshfwd:sshfwd /etc/ssh/sshfwd

2. Ed­it /​​​etc/​​​ssh/​​​sshd_​​​config file on the re­mote host, and add the fol­low­ing match sec­tion at very bot­tom of the file.

sudo nano /etc/ssh/sshd_config

Match User sshfwd
    AuthorizedKeysFile /etc/ssh/%u/authorized_keys
    MaxSessions 0
    GatewayPorts clientspecified

Ex­pla­na­tion about the above di­rec­tives could be found at man sshd_​​​config.

#SSHD Di­rec­tives Ex­pla­na­tion

• Match – In­tro­duces a con­di­tion­al block.  If all of the cri­te­ria on the Match line are sat­is­fied, the key­words on the fol­low­ing lines over­ride those set in the glob­al sec­tion of the con­fig file, un­til ei­ther an­oth­er Match line or the end of the file. If a key­word ap­pears in mul­ti­ple Match blocks that are sat­is­fied, on­ly the first in­stance of the key­word is ap­plied.
• Au­tho­rized­KeysFile – ac­cepts the to­kens %% (A lit­er­al ‘%’), %h (the home di­rec­to­ry of the user), %U (the nu­mer­ic user ID of the tar­get user), and %u (the user­name).
• MaxSes­sions – Spec­i­fies the max­i­mum num­ber of open shell, lo­gin or sub­sys­tem (e.g. sftp) ses­sions per­mit­ted per net­work con­nec­tion.  Mul­ti­ple ses­sions may be es­tab­lished by clients that sup­port con­nec­tion mul­ti­plex­ing.  Set­ting MaxSes­sions to 1 will ef­fec­tive­ly dis­able ses­sion mul­ti­plex­ing, where­as set­ting it to 0 will pre­vent all shell, lo­gin and sub­sys­tem ses­sions while still per­mit­ting for­ward­ing.  The de­fault is 10.
• Gate­way­Ports – Spec­i­fies whether re­mote hosts are al­lowed to con­nect to ports for­ward­ed for the client.  By de­fault, sshd(8) binds re­mote port for­ward­ings to the loop­back ad­dress. This pre­vents oth­er re­mote hosts from con­nect­ing to for­ward­ed ports.  Gate­way­Ports can be used to spec­i­fy that sshd should al­low re­mote port for­ward­ings to bind to non-loop­back ad­dress­es, thus al­low­ing oth­er hosts to con­nect.  The ar­gu­ment may be no to force re­mote port for­ward­ings to be avail­able to the lo­cal host on­ly, yes to force re­mote port for­ward­ings to bind to the wild­card ad­dress, or clientspec­i­fied to al­low the client to se­lect the ad­dress to which the for­ward­ing is bound.  The de­fault is no.

• Note when you for­ward­ing HTTP/HTTPS ports, which will be ac­ces­si­ble via Re­vers Proxy you need to add Gate­way­Ports yes in­to the server's /​​​etc/​​​ssh/​​​sshd_​​​config.

Test the con­nec­tion

At this stage we should be able to es­tab­lish a con­nec­tion be­tween the two hosts by us­ing the new key pair. Note there is not lo­gin shell avail­able for the ssh­fwd user, for that rea­son if we at­tempt to just SSH in­to the re­mote host the con­nec­tion will be im­me­di­ate­ly closed af­ter it is suc­cess­ful­ly es­tab­lished. So we can add the op­tions -T and -N to the ssh com­mand as fol­low.

sudo -u sshfwd ssh sshfwd-to-hostname -TN

Could not create directory '/home/sshfwd/.ssh' (No such file or directory).
Failed to add the host to the list of known hosts (/home/sshfwd/.ssh/known_hosts).
_

With this com­mand we will es­tab­lish an SSH con­nec­tion on the name of the user ssh­fwd. The con­nec­tion will hang on the state shown at the gray box above. We can ap­pend the op­tion -f to our com­mand in or­der to push the con­nec­tion to the back­ground.

sudo -u sshfwd ssh sshfwd-to-hostname -fTN

Ex­pla­na­tion about the above di­rec­tives could be found at man ssh and man ssh_​​​config.

#SSH Op­tions

• -f – Re­quests ssh to go to back­ground just be­fore com­mand ex­e­cu­tion. This is use­ful if ssh is go­ing to ask for pass­words or passphras­es, but the user wants it in the back­ground…
• -T – Dis­able pseu­do-ter­mi­nal al­lo­ca­tion.
• -N – Do not ex­e­cute a re­mote com­mand. This is use­ful for just for­ward­ing ports. Re­fer to the de­scrip­tion of Ses­sion­Type in ssh_​​​config.

Lat­er we can ter­mi­nate the con­nec­tion by killing all process­es that be­longs to the user ssh­fwd.

sudo pkill -9 -u sshfwd

Be­fore killing the con­nec­tion you can log-in to the re­mote host and test whether the port is for­ward­ed.

sudo netstat -pnat | grep sshfwd

tcp        0      0 172.17.201.1:9200       0.0.0.0:*               LISTEN      317057/sshd: sshfwd
tcp        0      0 192.168.2.200:22        192.168.1.110:33948     ESTABLISHED 316980/sshd: sshfwd

Or as it is in my case test does the for­ward­ed (Elas­tic­search) ser­vice re­sponse cor­rect­ly.

/usr/bin/curl 'http://172.17.201.1:9200'

#Elas­tic­search Re­sponse

{
  "name" : "i2IO0Ft",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "UIIVp_38SVdt38NaDsawUA",
  "version" : {
    "number" : "5.6.16",
    "build_hash" : "3a740d1",
    "build_date" : "2019-03-13T15:33:36.565Z",
    "build_snapshot" : false,
    "lucene_version" : "6.6.1"
  },
  "tagline" : "You Know, for Search"
}

In­stall Au­tossh

The ser­vice be­low us­es the com­mand au­tossh which is wrap­per of the ssh com­mand. So if you don't want to in­stall ad­di­tion­al pack­ages just mod­i­fy the Ex­ec­Start di­rec­tive of the ser­vice, oth­er­wise you will need to in­stall it.

sudo apt update && sudo apt install autossh

Cre­ate "sys­temd" ser­vice for Port for­ward­ing

sudo nano /etc/systemd/system/sshfwd-to-hostname.service

[Unit]
Description=Forward ports to <hostname>
After=network-online.target

[Service]
User=sshfwd
Environment=AUTOSSH_GATETIME=0
ExecStart=/usr/bin/autossh -M 0 -TN -q -o "ServerAliveInterval 60" -o "ServerAliveCountMax 3" sshfwd-to-hostname
ExecStop=/usr/bin/pkill -9 -u sshfwd # kill all processes owned by sshfwd
ExecReload=/bin/kill -s HUP $MAINPID
Restart=always
RestartSec=3

[Install]
WantedBy=multi-user.target

sudo systemctl daemon-reload
sudo systemctl enable sshfwd-to-hostname.service
sudo systemctl start sshfwd-to-hostname.service

In­stall SSHFS

SSHFS is a filesys­tem client based on SSH. SSHFS al­lows you to mount a re­mote filesys­tem us­ing SSH (more pre­cise­ly, the SFTP sub­sys­tem). Most SSH servers sup­port and en­able this SFTP ac­cess by de­fault, so SSHFS is very sim­ple to use – there's noth­ing to do on the serv­er-side.

By de­fault, file per­mis­sions are ig­nored by SSHFS. Any user that can ac­cess the filesys­tem will be able to per­form any op­er­a­tion that the re­mote serv­er per­mits – based on the cre­den­tials that were used to con­nect to the serv­er. If this is un­de­sired, lo­cal per­mis­sion check­ing can be en­abled with -o default_​​​permissions.

By de­fault, on­ly the mount­ing user will be able to ac­cess the filesys­tem. Ac­cess for oth­er users can be en­abled by pass­ing -o allow_​​​other. In this case you most like­ly al­so want to use -o default_​​​permissions.

It is rec­om­mend­ed to run SSHFS as reg­u­lar user (not as root). For this to work the mount­point must be owned by the user. If user­name is omit­ted SSHFS will use the lo­cal user­name. If the di­rec­to­ry is omit­ted, SSHFS will mount the (re­mote) home di­rec­to­ry. If you need to en­ter a pass­word sshfs will ask for it (ac­tu­al­ly it just runs ssh which ask for the pass­word if need­ed).

In most cas­es it is not avail­able by de­fault and must be in­stalled.

sudo apt install sshfs

Here is a ba­sic ex­am­ple how to do mount via sshfs at the com­mand line.

sshfs hostname:/remote/path ~/sshfs-mount-point

• In this ex­am­ple host­name i de­fined with­in the file ~/.ssh/config or with­in a glob­al con­fig file – i.e. /etc­/­ssh­/­ssh­_­config.d­/­sshfwd­-­to­-­hostname­.conf.

SSHFS can be used al­so with­in /​​​etc/​​​fstab as it is de­scribed in its man page, or can be used with­in sys­temd mount unit as it is shown in the next sec­tion.

Cre­ate "sys­temd" ser­vice for SSHFS mount­ing

sudo nano /etc/systemd/system/mount-remote-fs.mount

[Unit]
Description=Mount a remote directory to /mnt/home.network.com/user-downloads

[Mount]
What=user@home.network.com:/home/user/Downloads
Where=/mnt/home.network.com/user-downloads
Type=fuse.sshfs
Options=_netdev,allow_other,port=22,default_permissions,IdentityFile=/etc/ssh/sshfwd/id_ed25519_to_hostname,reconnect,ServerAliveInterval=30,ServerAliveCountMax=5,x-systemd.automount,uid=33,gid=33
TimeoutSec=60
#Restart=always
#RestartSec=3

[Install]
WantedBy=multi-user.target

sudo systemctl daemon-reload
sudo systemctl enable mount-remote-fs.mount

systemctl start mount-remote-fs.mount    # to mount the remote directory
systemctl stop mount-remote-fs.mount     # to unmount the remote directory
systemctl status mount-remote-fs.mount   # to check the status

In the above ex­am­ple we have mount­ed a di­rec­to­ry from a re­mote in­stance for the user www-da­ta, prob­a­bly some web ser­vice as NextCloud will ac­cess this di­rec­to­ry :) This is a kin of spe­cial case, in most cas­es we will mount a di­rec­to­ry for the cur­rent user in use as it is shown in this guide: Mount Re­mote Filesys­tems Us­ing SSHFS and sys­temd.

Ref­er­ences

• Freedesktop​.org: Sys­temd Man­u­al
• Lin­ux­ize: Us­ing the SSH Con­fig File
• GitHub Gist: Thomasfr/autossh.service
• Coder­wall: Al­low for se­lect­ing in­ter­face when port for­ward­ing
• Eric's blog: Cre­at­ing a per­sis­tent SSH tun­nel in Ubun­tu
• Bug­gy­coder: Mount Re­mote Filesys­tems Us­ing SSHFS and sys­temd,
• Github Gist: proprietary/mnt-mymountpoint.mount
• Ask Ubun­tu: Ac­cess re­mote mul­ti­ple servers be­hind NAT
• Ask Ubun­tu: Us­ing SSHFS on C ac­cessed via A and B
• Ask Ubun­tu: How do I set­up SSH key based au­then­ti­ca­tion for GitHub by us­ing ~/.ssh/config file?
• Ask Ubun­tu: How to copy (or move) files from re­mote ma­chine to lo­cal ma­chine?
• Ask Ubun­tu: How to share files/​​​folders be­tween two dif­fer­ent Ubun­tu com­put­ers and are on dif­fer­ent net­work?
• Ask Ubun­tu: Filezil­la or Win­SCP al­ter­na­tive for Ubun­tu
• GitHub: a2proxy – col­lec­tion of Apache2 Re­verse Proxy ex­am­ple con­fig­u­ra­tions and gen­er­a­tor script

Reg­is­ter: Lin­ux Serv­er

Идентифициране

Разработване

Авторизиране

Утвърждаване

Spas

Spas

Spas

Spas

19.07.2022

8.09.2022

8.09.2022

26.09.2022

28889

31247

31377 30784

31957 30786