SSH Persistent Tunnel and SSHFS Mount via "systemd" units
Template:ContentArticleHeader/Network Security
In this tutorial we will create a systemd service that will use the tool autossh to create a persistent tunnel in order to make a service running on a local host available on a remote host. The steps provided below are tested within the circumstances of Ubuntu Server 22.04.
In the current scenario, so called local host will establish SSH connection to the remote host. So we will forward a remote port to listen at our local port – otherwise said we will bind the remote port to the local port. The service that I want to forward by this technique is Elasticsearch which currently listen on 127.0.0.1:9200 at the local host.
Setup SSH Connection and port forwarding
Create a system user
Here we will create a system user named sshfwd, dedicated for the service created in the following step. We will use the same username on bot local and remote hosts, – so run the following command at both hosts.
sudo useradd -r -s /bin/false sshfwd
id sshfwd
uid=975(sshfwd) gid=975(sshfwd) groups=975(sshfwd)
-r– create system user (note there is not specified home directory because we do need it),-s /bin/false– this will be the default shell for the user.
Generate SSK key pair
Generate public and private SSH keys within your user's ~/.ssh directory at the local host. The public key must be transferred to the remote host. The private key will be used at the local host.
mkdir -m700 ~/.ssh/sshfwd
ssh-keygen -t rsa -b 4096 -C 'sshfwd@local.host' -f ~/.ssh/sshfwd/id_rsa_sshfwd-to-hostname -q -N ''
ls -la ~/.ssh/sshfwd
total 16
drwx------ 2 <user> <user> 4096 Jul 20 21:36 .
drwx------ 12 <user> <user> 4096 Jul 20 21:28 ..
-rw------- 1 <user> <user> 3381 Jul 20 21:36 id_rsa_sshfwd-to-hostname
-rw-r--r-- 1 <user> <user> 742 Jul 20 21:36 id_rsa_sshfwd-to-hostname.pub
-q– silence;-N ''– empty (without) passphrase.
Setup the Local host
1. Create a directory where the private key will be accessible for the sshfwd user and copy it there.
mkdir -m700 /etc/ssh/sshfwd
sudo cp ~/.ssh/sshfwd/id_rsa_sshfwd-to-hostname /etc/ssh/sshfwd/
sudo chown -R sshfwd:sshfwd /etc/ssh/sshfwd
2. Create the following configuration file.
sudo nano /etc/ssh/ssh_config.d/sshfwd-to-hostname.conf
# This file is loaded by a directive in /etc/ssh/ssh_config
Host sshfwd-to-hostname
HostName 192.168.2.200
IdentityFile /etc/ssh/sshfwd/id_rsa_sshfwd-to-hostname
User sshfwd
Port 22
RemoteForward 172.17.201.1:9200 127.0.0.1:9200
StrictHostKeyChecking no
GatewayPorts yes
Compression yes
Note the value sshfwd-to-hostname of the directive Host is an alias for the actual hostname or IP address provided by the HostName directive. In the Example above 192.168.2.200 is the IP address of the remote host.
The directive RemoteForward 172.17.201.1:9200 127.0.0.1:9200 will bind the remote port 9200 on the remote interface 172.17.201.1 to the local port 9200 on the loopback interface 127.0.0.1 where our (Elasticsearch) service actually listen. Simplified version of this directive could be RemoteForward 9200 127.0.0.1:9200 which means the remote 0.0.0.0:9200 will be bind to the local 127.0.0.1:9200.
However, In my case, I want to forward port 9200 only to a specific (docker's) remote interface and also I need to share it with the rest hosts that communicate with that interface – otherwise said the remote SSH server should be configured to operate as gateway, this will be done in the following steps.
The directive StrictHostKeyChecking no is equivalent to the CLI option -oStrictHostKeyChecking=no and will cause an automatic accept of the remote host's keys.
Setup the Remote host
0. The first thing you need to do is to transfer the file id_rsa_sshfwd-to-hostname.pub generated above to the remote host. In my case I already have an operational SSH connection, so rsync is my first choice. Then log-in to the remote host – the following steps of this section are performed there.
rsync ~/.ssh/sshfwd/id_rsa_sshfwd-to-hostname.pub remote.host.com:/tmp/
ssh remote.host.com
1. On the remote host, create a directory where authorized_keys file for the sshfwd user will be stored and use the the public key copied in the step above to create it.
mkdir -m700 /etc/ssh/sshfwd
sudo mv /tmp/id_rsa_sshfwd-to-hostname.pub /etc/ssh/sshfwd/authorized_keys
sudo chown -R sshfwd:sshfwd /etc/ssh/sshfwd
2. Edit /etc/ssh/sshd_config file on the remote host, and add the following match section at very bottom of the file.
sudo nano /etc/ssh/sshd_config
Match User sshfwd
AuthorizedKeysFile /etc/ssh/%u/authorized_keys
MaxSessions 0
GatewayPorts clientspecified
Explanation about the above directives could be found at man sshd_config.
Match– Introduces a conditional block. If all of the criteria on the Match line are satisfied, the keywords on the following lines override those set in the global section of the config file, until either another Match line or the end of the file. If a keyword appears in multiple Match blocks that are satisfied, only the first instance of the keyword is applied.AuthorizedKeysFile– accepts the tokens%%(A literal ‘%’),%h(the home directory of the user),%U(the numeric user ID of the target user), and%u(the username).MaxSessions– Specifies the maximum number of open shell, login or subsystem (e.g. sftp) sessions permitted per network connection. Multiple sessions may be established by clients that support connection multiplexing. Setting MaxSessions to 1 will effectively disable session multiplexing, whereas setting it to 0 will prevent all shell, login and subsystem sessions while still permitting forwarding. The default is 10.GatewayPorts– Specifies whether remote hosts are allowed to connect to ports forwarded for the client. By default, sshd(8) binds remote port forwardings to the loopback address. This prevents other remote hosts from connecting to forwarded ports.GatewayPortscan be used to specify that sshd should allow remote port forwardings to bind to non-loopback addresses, thus allowing other hosts to connect. The argument may benoto force remote port forwardings to be available to the local host only,yesto force remote port forwardings to bind to the wildcard address, orclientspecifiedto allow the client to select the address to which the forwarding is bound. The default isno.
- Note when you forwarding HTTP/HTTPS ports, which will be accessible via Revers Proxy you need to add
GatewayPorts yesinto the server's/etc/ssh/sshd_config.
Test the connection
At this stage we should be able to establish a connection between the two hosts by using the new key pair. Note there is not login shell available for the sshfwd user, for that reason if we attempt to just SSH into the remote host the connection will be immediately closed after it is successfully established. So we can add the options -T and -N to the ssh command as follow.
sudo -u sshfwd ssh sshfwd-to-hostname -TN
Could not create directory '/home/sshfwd/.ssh' (No such file or directory).
Failed to add the host to the list of known hosts (/home/sshfwd/.ssh/known_hosts).
_
With this command we will establish an SSH connection on the name of the user sshfwd. The connection will hang on the state shown at the gray box above. We can append the option -f to our command in order to push the connection to the background.
sudo -u sshfwd ssh sshfwd-to-hostname -fTN
Explanation about the above directives could be found at man ssh and man ssh_config.
-f– Requestssshto go to background just before command execution. This is useful ifsshis going to ask for passwords or passphrases, but the user wants it in the background…-T– Disable pseudo-terminal allocation.-N– Do not execute a remote command. This is useful for just forwarding ports. Refer to the description ofSessionTypeinssh_config.
Later we can terminate the connection by killing all processes that belongs to the user sshfwd.
sudo pkill -9 -u sshfwd
Before killing the connection you can log-in to the remote host and test whether the port is forwarded.
sudo netstat -pnat | grep sshfwd
tcp 0 0 172.17.201.1:9200 0.0.0.0:* LISTEN 317057/sshd: sshfwd
tcp 0 0 192.168.2.200:22 192.168.1.110:33948 ESTABLISHED 316980/sshd: sshfwd
Or as it is in my case test does the forwarded (Elasticsearch) service response correctly.
/usr/bin/curl 'http://172.17.201.1:9200'
{
"name" : "i2IO0Ft",
"cluster_name" : "elasticsearch",
"cluster_uuid" : "UIIVp_38SVdt38NaDsawUA",
"version" : {
"number" : "5.6.16",
"build_hash" : "3a740d1",
"build_date" : "2019-03-13T15:33:36.565Z",
"build_snapshot" : false,
"lucene_version" : "6.6.1"
},
"tagline" : "You Know, for Search"
}
Create Systemd service
sudo nano /etc/systemd/system/sshfwd-to-hostname.service
[Unit]
Description=Forward ports to <hostname>
After=network-online.target
[Service]
User=sshfwd
Environment=AUTOSSH_GATETIME=0
ExecStart=/usr/bin/autossh -M 0 -TN -q -o "ServerAliveInterval 60" -o "ServerAliveCountMax 3" sshfwd-to-hostname
ExecStop=/usr/bin/pkill -9 -u sshfwd # kill all processes owned by sshfwd
ExecReload=/bin/kill -s HUP $MAINPID
Restart=always
RestartSec=3
[Install]
WantedBy=multi-user.target
sudo systemctl daemon-reload
sudo systemctl enable sshfwd-to-hostname.service
sudo systemctl start sshfwd-to-hostname.service
Install Autossh
The service above uses the command autossh which is wraper of the ssh command. So if you don't want to install additional packages just modify the ExecStart directive of the service, otherwise you will need to install it.
sudo apt update && sudo apt install autossh
References
- Freedesktop.org: Systemd Manual
- Linuxize: Using the SSH Config File
- GitHub Gist: Thomasfr/autossh.service
- Coderwall: Allow for selecting interface when port forwarding
- Eric's blog: Creating a persistent SSH tunnel in Ubuntu
- Access remote multiple servers behind NAT
- Using SSHFS on C accessed via A and B
