NNCP Concepts

This page describes some basic concepts of NNCP.

What NNCP can do

NNCP is a versatile communications layer; think of it as an Asynchronous Communication version of ssh. See my NNCP page for ideas and links. NNCP can effectively route over anything from AWS instances to USB sticks and CD-Rs, and supports file transport and remote execution.

Everything in NNCP is protected by end-to-end encryption and good security.

A key property of an NNCP network is that the two endpoints of communication need not be online simultaneously.

Basic Concepts of Packets

NNCP works with “packets”. A packet is something that carries data. It is stored as a file in a queue on disk in your local NNCP instance. A packet can be an arbitrary size; from just a few bytes to multiple TB.

So, the NNCP commands are concerned with:

  • Creating packets
  • Transporting packets
  • Processing packets
    • And, as a subset of processing, routing packets.

NNCP operations

I will go into the details of how packets are processed in a bit, but first I need to give you some background on the big picture of the concepts these packets provide. After that, I’ll explain how these concepts map to the three categories of commands I outlined above.

The basic operations within NNCP are file transmission and remote command execution.

File transmission

With file transmission, you can either send a file to a remote node (using nncp-file) or request a remote node send a file to you (nncp-freq). For instance:

nncp-file /bin/bash remotenode:bash2

This command creates a packet, encrypted to remotenode, using /bin/bash as the content. It adds some metadata specifying that the file should be named bash2 on the remote end. It writes this packet as a file in the local queue bound for transmission to remotenode.

The remotenode’s configuration specifies whether this node is allowed to send it files, and if so, where to put them.

To request a file from the remote node:

nncp-freq remotenode:README.txt

This command creates a (very small) packet, encrypted to remotenode, saying “Hey, send me README.txt”. remotenode checks to see if the request is authorized, and if so, what directory to find the file in. If it finds one, it does something very similar to the nncp-file command above, generating a reply packet that is basically a file transmission.

Remote execution

Let’s say that on node bob, I authorized node alice to run a command I call “tarextract”, defined like this:

"tarextract": ["/usr/bin/tar", "-xpf", "-"]

If you think this has a parallel to ssh’s authorized_keys, you’re absolutely right.

Now on alice, I can say this:

tar -cpf - /usr/local | nncp-exec bob tarextract -C /var/local/backups

So what does nncp-exec do? It reads stdin and encrypts it to node bob. It adds metadata, saying it’s requesting the “tarextract” command, and adding two arguments to it (the -C /var/local/backups). It writes this all into a packet file in the transmission queue for bob.

When bob processes the packet, it spins up /usr/bin/tar -xpf - -C /var/local/backups, piping to it the data that had originally been piped to nncp-exec.

Email can be sent over NNCP; for instance, to use tools originating in the UUCP days and which still work today, one might pipe an email message to nncp-exec mailserver rmail foo@example.com. The NNCP manual has details on how to set this up.

The commands

Now let’s talk about the NNCP commands and how they relate to the operations we’ve identified so far.

Commands to create packets

We’ve already covered the three main commands to create packets: nncp-file, nncp-freq, and nncp-exec. All of these commands take some sort of input – a file, a command to run, or just the name of a file to request – and create a packet, encrypted with the destination’s public key, which they write into the outgoing queue for the destination’s node. (Except for when we route via an intermediary; more on that below.)

Commands to transport packets

Now, there are three main ways to transport packets between NNCP nodes:

  • Over a mountable filesystem - could be anything from USB sticks to Dropbox or Syncthing (with nncp-xfer)
  • Over a tar-like stream (with nncp-bundle)
  • Over the network (using nncp-daemon and either nncp-call or nncp-caller)

Let’s take a look at these in more detail.

NNCP over a mountable filesystem

Since NNCP packets are fundamentally just files, these files can be transported to the remote over anything that can store… well, files. nncp-xfer does just that. You can specify a destination node, and it will extract packets destined for that node from the NNCP queue, and copy (or move) them to the target path. It makes a simple directory structure on the target path.

Then, at the remote, you run nncp-xfer in receive mode. It does just the opposite; scans the target path for packets meant for it, and moves those into the incoming queue for later processing.

Since the directory tree is very basic, it can be just about anything. A USB stick is common, or a shared folder with Syncthing or Dropbox, or a mounted S3 folder – whatever you like.

You can, of course, use the same filesystem to both send and receive packets from a given node.

Over a tar-like stream

Sometimes it is handy to be able to pipe a stream to somewhere. The target might be a tape drive, a process that burns the stream to a DVD+R, an ssh pipe to a remote, or all sorts of other things you can pipe stuff to in Unix.

That’s what nncp-bundle does. Conceptually, it’s very similar to nncp-xfer; it’s just that the interchange format is a stream (or single file) instead of a directory tree populated with files. nncp-bundle in transmission mode generates a stream for a remote, filled with all the packets in the transmission queue for that node. In receive mode, it receives a stream, and copies any packets meant for the local node into its incoming queue.

Over the network

Most people will probably have a transmission over the network at some point. Although all NNCP nodes are peers, in TCP, you need someone to be the server and someone to be a client. So, the server is nncp-daemon and the client is nncp-call or nncp-caller.

Why two clients? Both do conceptually the same thing, and speak the same protocol. The difference is that nncp-call is designed to be run as a standalone command (“connect to the server for node foo RIGHT NOW”), while nncp-caller is designed to be a long-running service, that either maintains long-lasting connections to the remote nncp-daemon, or periodically establishes connections to it, or both. Functionally they are the same.

The network communication with NNCP is quite versatile. It is bidirectional, can send packets in an order given by their user-defined prioritization, can resume interrupted transfers, and can even auto-detect peers on the local network. It is also, of course, encrypted and authenticated.

More fun with transports

My page at Tunneling NNCP over other transports shows some more advanced ways to integrate these fundamental commands with things like ssh, Syncthing, and tor.

Commands to process packets

So far, I’ve talked a lot about queues. The commands that create packets (nncp-file, freq, and exec) put the packets in an outbound queue. There is also an inbound queue. The commands to transport packets exchange data from those queues. That leaves us with one more piece: how do we handle packets in the inbound queue?

That’s the job of nncp-toss. It literally processes packets from incoming queues. It verifies their integrity, decrypts them, and handles them as needed: running commands for packets generated by nncp-exec, depositing files sent by nncp-file, answering requests from nncp-freq, and also routing packets on to their destination (more on that below).

nncp-toss can be run as a standalone command, from a cron job, or as a daemon with the -cycle parameter.

There is also a command nncp-stat that shows you the status of all your queues, incoming and outgoing.

Next Steps

Take a look at my other NNCP resources, particularly Getting Started with NNCP.

Advanced Routing

You can skip this section for now; it covers some more advanced topics.

Thus far, I have made an assumption: that when you send a packet from node alice to node bob, alice and bob are “directly” connected. That is, they can speak to each other, or you can move a USB stick between them, or some such scenario.

Now imagine this scenario: you have three nodes. Alice can communicate with bob, and bob can communicate with charlie, but alice and charlie have no way of communicating directly. Wouldn’t it be nice to use bob as an intermediary between alice and charlie? In fact, with NNCP, you can do just that!

For instance, on alice you could say:

nncp-file -via bob /bin/bash charlie:bash2

So, here alice sends a packet to charlie via bob! What exactly does that do? It uses onion encryption, like Tor. Conceptually, it works like this:

  • First, alice generates the packet for charlie, encrypted – like usual – using charlie’s public key.
  • Then it generates a packet for bob. The type of this packet, rather than asking bob to receive a file or run a command, says “send this encrypted payload on to charlie.” It then encrypts this outer packet using Bob’s public key, and places it in the queue for transmission to bob.
  • When bob receives the packet, it decrypts the outer layer of encryption, and discovers it needs to pass the payload on to charlie. It puts this packet in its transmission queue for charlie. Note that because the packet bound for charlie is encrypted using charlie’s key, bob can’t see what’s in it; only where it’s going.
  • By the time charlie receives the packet, all the outer layers of encryption are gone, and it can decrypt the payload and process it like usual.

(In practice, this is much more efficient and optimized than you might guess from reading this description.)

You can, in fact, have a via path that’s arbitrarily long; for instance, -via alpha,bravo,charlie. This kind of via routing is used in a number of scenarios: for instance, the quux.org NNCP public relay is using it. Also note that each hop along the via path can transport packets using any of NNCP’s methods. So you could, for instance, build a very secure machine to receive backups. So secure, in fact, that it has no network access.

You could have one of the machines on your network as the dedicated “hub”; all the machines could send their backups to the backuphost via the hub. Then periodically you write the backups to a USB drive, unplug it from the hub, plug it into the backup server, and ingest them there.

You can specify a default via path for each of the nodes you communicate with in the configuration file.

Multicasting

NNCP also supports what it calls multicast areas. The idea here is when you want to send packet to more than one destination; for instance, to a whole handful of nodes in your NNCP network. It facilitates what amounts to an Asynchronous Mesh Network. I use it in my gitsync-nncp software.

NNCP is a powerful tool for building Asynchronous Communication networks. It features end-to-end Encryption as well as all sorts of other features; see my NNCP Concepts page for some more ideas.

I sometimes see people read about NNCP and wonder “This sounds great! But… what can I do with it?” This page aims to answer those questions.

The care and feeding of an NNCP installation.

This page describes the basic installation and configuration of NNCP.

Usenet, of course, originally ran over UUCP in quite a few cases. Since NNCP is quite similar to UUCP – in fact, you can map UUCP commands to NNCP ones – it is quite possible, and not all that hard, to run Usenet over NNCP. In fact, in a number of ways, it works better than Usenet over UUCP!

According to the NNCP documentation, NNCP is intended to help build up small size ad-hoc friend-to-friend (F2F) statically routed darknet delay-tolerant networks for fire-and-forget secure reliable files, file requests, Internet Email and commands transmission. All packets are integrity checked, end-to-end Encrypted, explicitly authenticated by known participants public keys. Onion encryption is applied to relayed packets. Each node acts both as a client and server, can use push and poll behaviour model. Also there is multicasting area support.

NNCP lets you securely send files, or request remote execution, between systems. It uses asynchronous communication, so the source and destination need never be online simultaneously. NNCP can route requests via intermediate devices – other NNCP nodes, USB sticks, tapes, radios, phones, cloud services, whatever – leading to a network that is highly resilient and flexible. NNCP makes it much easier to communicate with devices that lack Internet connectivity, or have poor Internet.