Using Filespooler over NNCP

NNCP is a powerful tool for building Asynchronous Communication networks. It features end-to-end Encryption as well as all sorts of other features; see my NNCP Concepts page for some more ideas.

NNCP has two fundamental communcation types:

  • File transfer
  • Remote execution request

You might say, “this sounds a lot like Filespooler”, and you’re right. I designed Filespooler to be a generic tool, and it was specifically designed because of some pain points I had with NNCP. Specifically:

  • nncp-exec can execute jobs remotely, but it makes no attempt to do so in order.
  • nncp-exec has one thing it does when a job fails - retry indefinitely - and this is sometimes sub-optimal.

Filespooler has neither of those limitations. It makes a perfect companion to NNCP, in fact. Let’s take a look at how.

Ideas for getting data to a remote queue

There are two easy enough ways to get data to a Filespooler queue using NNCP:

  1. nncp-file can directly write it into the queue’s jobs directory, as described at Guidelines for Writing To Filespooler Queues Without Using Filespooler.
  2. Or, you can pipe the prepared job to nncp-exec that will call fspl queue-write on the remote.

Since nncp-file’s writing algorithm is safe and compatible with Filespooler, you can really choose either option. I tend to prefer the second just because it frees me from ever having to generate filenames and such.

Using Filespooler with nncp-exec

First, we need to grant permission to run fspl queue-write. On the receiver, we’ll edit nncp.hjson to allow a specific sender to run this command. In the neigh block for that sender, add something like this:

    exec: {
     enqueue: ["fspl", "queue-write", "/srv/nncp/queue"]
    }

Of course, use whatever path is appropriate there, taking care to have correct permissions (or use sudo). You can easily support multiple queues this way by giving them each a unique NNCP command name and path.

Now, on the sender, you would do something like this:

echo hi | fspl prepare -s ~/seqfile -i - | \
   nncp-exec othernode enqueue

And that’s it! Simple!

Using Filespooler with nncp-file

This is also pretty easy. To make this work, on the receiver, the incoming path has to include the queue (or, at the very least, the jobs subdirectory within it). You get a little less control over things this way.

On the receiver side, set incoming within the neigh section to an appropriate path.

On the sending side, you can use something like this:

echo hi | fspl prepare -s ~/seqfile -i | \
  nncp-file - "othernode:queuepath/jobs/$(fspl gen-filename)"

The fspl gen-filename command will print a random filename that mathes the Filespooler pattern to stdout. This allows us to use it as a target filename. Since nncp-toss will write the file in a safe manner, this is all it takes to deliver a queue file to NNCP with Filespooler!

No-Queue Filespooler to Enhancs nncp-exec

Another way to use Filespooler is to allow you to pass environment variables along with nncp-exec requests, even if you never use a queue. See Using Filespooler Without Queues to Pass More Metadata for more info on this technique.

I loaded up this title with buzzwords. The basic idea is that IM systems shouldn’t have to only use the Internet. Why not let them be carried across LoRa radios, USB sticks, local Wifi networks, and yes, the Internet? I’ll first discuss how, and then why.

One frustration people sometimes have with ssh or NNCP is that they’d like to pass along a lot of metadata to the receiving end. Both ssh and nncp-exec allow you to pass along command-line parameters, but neither of them permit passing along more than that. What if you have a whole host of data to pass? Maybe a dozen things, some of them optional? It would be very nice if you could pass along the environment.

Filespooler provides the fspl queue-write command to easily add files to a queue. However, the design of Filespooler intentionally makes it easy to add files to the queue by some other command. For instance, Using Filespooler over Syncthing has Syncthing do the final write, the nncp-file (but not the nncp-exec) method in Using Filespooler over NNCP had NNCP do it, and so forth.

It seems that lately I’ve written several shell implementations of a simple queue that enforces ordered execution of jobs that may arrive out of order. After writing this for the nth time in bash, I decided it was time to do it properly. But first, a word on the why of it all.

In some cases, you may want to use Filespooler to send the data from one machine to many others. An example of this could be using gitsync-nncp over Filespooler where you would like to propagate the changes to many computers.

Filespooler lets you request the remote execution of programs, including stdin and environment. It can use tools such as S3, Dropbox, Syncthing, NNCP, ssh, UUCP, USB drives, CDs, etc. as transport; basically, a filesystem is the network for Filespooler. Filespooler is particularly suited to distributed and Asynchronous Communication.

NNCP lets you securely send files, or request remote execution, between systems. It uses asynchronous communication, so the source and destination need never be online simultaneously. NNCP can route requests via intermediate devices – other NNCP nodes, USB sticks, tapes, radios, phones, cloud services, whatever – leading to a network that is highly resilient and flexible. NNCP makes it much easier to communicate with devices that lack Internet connectivity, or have poor Internet.