Add support for using multiple connections

The -o max_conns=N option causes multiple SSH processes and response processing threads to
be created. This means that e.g. reading a large file no longer blocks all access to the
filesystem.

The connection is chosen by checking the per-connection statistics:

  1. Choose connection with least outstanding requests; if it's a tie,
  2. choose connection with least directory handles; if it's a tie,
  3. choose connection with least file handles; if it's a tie,
  4. choose connection which has already been established.

The implementation assumes that the max_conns values will be small; it
uses linear search.

Example benchmark:

With single connection:

$ sshfs -o max_conns=1,workaround=nobuflimit ebox: mnt
$ cat mnt/tmp/bigfile > /dev/null &
$ time find mnt > /dev/null

real	1m50.432s
user	0m0.133s
sys	0m0.467s

With multiple connections:

$ ~/in-progress/sshfs/build/sshfs -o max_conns=5,workaround=nobuflimit ebox: mnt
$ cat mnt/tmp/bigfile > /dev/null &
$ time find mnt > /dev/null

real	1m15.338s
user	0m0.142s
sys	0m0.491s

This feature was implemented to large extend by Timo Savola <timo.savola@iki.fi>. Thanks
to CEA.fr for sponsoring the remaining work to complete this feature and integrate it into
SSHFS!

ChangeLog.rst

@@ -1,3 +1,10 @@
+Unreleased Changes
+------------------
+
+* New max_conns option enables the use of multiple connections to improve responsiveness
+  during large file transfers. Thanks to Timo Savola for doing most of the implementation
+  work, and thanks to CEA.fr for sponsoring remaining bugfixes and cleanups!
+
 Release 3.6.0 (2019-11-03)
 --------------------------
 

sshfs.rst

@@ -218,6 +218,14 @@ Options
       for example if the file size is not known in advance (before reading it).
       e.g. /proc filesystem 
 
+-o max_conns=N
+   sets the maximum number of simultaneous SSH connections
+   to use. Each connection is established with a separate SSH process.
+   The primary purpose of this feature is to improve the responsiveness of the
+   file system during large file transfers. When using more than once
+   connection, the *password_stdin* and *slave* options can not be
+   used, and the *buflimit* workaround is not supported/
+
 In addition, SSHFS accepts several options common to all FUSE file
 systems. These are described in the `mount.fuse` manpage (look
 for "general", "libfuse specific", and "high-level API" options).

test/test_sshfs.py

@@ -33,7 +33,8 @@ def name_generator(__ctr=[0]):
 @pytest.mark.parametrize("debug", (False, True))
 @pytest.mark.parametrize("cache_timeout", (0,1))
 @pytest.mark.parametrize("sync_rd", (True, False))
-def test_sshfs(tmpdir, debug, cache_timeout, sync_rd, capfd):
+@pytest.mark.parametrize("multiconn", (True,False))
+def test_sshfs(tmpdir, debug, cache_timeout, sync_rd, multiconn, capfd):
     
     # Avoid false positives from debug messages
     #if debug:
@@ -77,6 +78,9 @@ def test_sshfs(tmpdir, debug, cache_timeout, sync_rd, capfd):
     cmdline += [ '-o', 'entry_timeout=0',
                  '-o', 'attr_timeout=0' ]
 
+    if multiconn:
+        cmdline += [ '-o', 'max_conns=3',
+                     '-o', 'workaround=nobuflimit' ]
     
     new_env = dict(os.environ) # copy, don't modify