Handling Filespooler Command Output

By default, Filespooler doesn’t do anything special with the output from the commands that fspl queue-process executes. If they write to stdout or stderr, you’ll see this on the controlling terminal or wherever you have piped or redirected it.

Since fspl queue-process, by default, will process as many jobs as it can with each invocation, this may result in the output from multiple jobs being concatenated together. In some cases, this may be undesirable. Here are some ways to work around this situation:

  1. You can use --maxjobs 1 to fspl queue-process to ensure that a given invocation always processes either 0 or 1 jobs.
  2. You can use --output-to SaveBoth to fspl queue-process, causing Filespooler to write the output of each job to its own dedicated file (see the Filespooler Reference for more details)
  3. You can process output internally in the program or script that fspl queue-process is running. One particular option here would be to pipe the output of the executed command to logger, which will then log it to syslog.

Filespooler is designed to work well in automated situations, including when started from cron or systemd. It is a fairly standard program in that way. I’ll discuss a few thoughts here that may help you architect your system.

It seems that lately I’ve written several shell implementations of a simple queue that enforces ordered execution of jobs that may arrive out of order. After writing this for the nth time in bash, I decided it was time to do it properly. But first, a word on the why of it all.

Filespooler is a way to execute commands in strict order on a remote machine, and its communication method is by files. This is a perfect mix for Syncthing (and others, but this page is about Filespooler and Syncthing).

Filespooler lets you request the remote execution of programs, including stdin and environment. It can use tools such as S3, Dropbox, Syncthing, NNCP, ssh, UUCP, USB drives, CDs, etc. as transport; basically, a filesystem is the network for Filespooler. Filespooler is particularly suited to distributed and Asynchronous Communication.