Purpose of Append-Only Queues
In the Using Filespooler over Syncthing example, we synced the entire b64queue directory. This is unnecessary.
It would be better to sync only the jobs subdirectory of it, to prevent it looking like a valid queue for processing on the sender.
Implications of Append-Only Queues
The Filespooler Reference discusses append-only queues. An append-only queue will allow commands that append (fspl queue-write) and commands that inspect (for instance, fspl queue-ls) to succeed. Commands that require a sequence number, such as fspl queue-process, will fail.
The idea is that often times, the jobs directory will be synced. It would be convenient to be able to write to this directory on the sender side, while still preventing the sender from accidentally trying to process the queue. That is exactly the point of an append-only queue.
Set up
- On the sender, use
fspl queue-init --append-onlyto create an append-only queue. - On the receiver, use
fspl queue-initwithout--append-onlyto create a regular queue. - Both queues will contain a
jobssubdirectory. Tell your software to sync just that directory.
Links to this note
You can use Filespooler with a number of other filesystems and storage options. s3fs, for instance, lets you mount S3 filesystems locally. I can’t possibly write about every such option, so I’ll write about one: rclone.
Filespooler is a way to execute commands in strict order on a remote machine, and its communication method is by files. This is a perfect mix for Syncthing (and others, but this page is about Filespooler and Syncthing).
Filespooler lets you request the remote execution of programs, including stdin and environment. It can use tools such as S3, Dropbox, Syncthing, NNCP, ssh, UUCP, USB drives, CDs, etc. as transport; basically, a filesystem is the network for Filespooler. Filespooler is particularly suited to distributed and Asynchronous Communication.