filed
Job queue using FUSE
git clone git://mccd.space/filed
| Log | Files | Refs | README | LICENSE |
commit d74b297abbb03a27868516e5f2fd5b54798261f1 parent 351d5e5baf2bd7ee38146e6b9da424c32e85ba71 Author: Marc Coquand <marc@coquand.email> Date: Mon, 15 Dec 2025 21:53:47 +0100 docs Diffstat:
| M | README.md | | | 15 | +++++++++++++++ |
| M | qj.1.scd | | | 4 | ++-- |
2 files changed, 17 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md @@ -62,6 +62,20 @@ And you can restart a job by moving the job back to pending: $ mv /tmp/qj-jobs/failed/1 /tmp/qj-jobs/pending ``` +## Design & Motivation + +If you're building any kind of web application, at some point you probably need a job queue. Whether that's for sending an email confirmation or batch process some files, many apps need it. + +Often these jobs can fail, whether that's due to network errors, memory issues or something else, so some retry mechanism is necessary. Being able to easily inspect the jobs and rerun them as an admin is also very important, so that the admin can fix whatever issue caused it to fail in the first place. + +I wanted a tool that I could incorporate and use with whatever programming language I desired, and that makes it easy to understand when a job fail and rerun jobs if there is an error. `qj` is very intuitive to build an integration for: just write a file telling it what to execute. + +I also wanted a tool that made it simple to inspect, without needing to expose a web portal or have set up separate auth. `qj` allows you to inspect and operate the queue just by SSHing into the server, and reuses battle-proven identity system already built into Linux. + +The simple file-based API `qj`, inspired by plan9, allows me to slim down the amount of code needed considerably, while still exposing a very scriptable and easy-to-understand interface. + +I've tried a few other queue tools: sqs/sns, rabbitmq, bull, systemd-run. The first two felt heavyweight, and required setting up a lot of infrastructure, especially if you want to rerun and inspect jobs. It felt like far too much work for a simple app. Bull was more in line with what I wanted, but I think operating on files is simpler to build custom automation, and easier to secure. Systemd-run lacked the retry functionality and the interface was rather clunky. + ## TODO - [x] Support chmod and chown @@ -75,6 +89,7 @@ $ mv /tmp/qj-jobs/failed/1 /tmp/qj-jobs/pending - [nq] - `nq` is simpler and not a persistent process, but does not feature retries. They serve different purposes: `nq` for ad-hoc queuing of command lines. `qj` serves well as a job manager for your server, where you want admins to see jobs and be able to rerun them. - [task-spooler] - `ts` has better control over how you want the task executed (GPU or CPU), and a lot of other features. It does (AFAIK) not support retries, which are supported in `qj`. - [bull] - `bull` is only for node and javascript. It features a graphical UI, and a few other features not found in `qj`. `qj` eschews a GUI in favor of simple files, allowing it to better interoperate with other systems, and allows it to use regular unix permissions for access management. +- sqs - requires you to setup most infrastructure around retries yourself. sqs is far more complex, more focused on message passing, harder to inspect, but far more flexible. Sqs scales better and fits more workloads. [nq]: https://github.com/leahneukirchen/nq [task-spooler]: https://github.com/justanhduc/task-spooler diff --git a/qj.1.scd b/qj.1.scd @@ -10,8 +10,8 @@ qj - queue jobs utility # DESCRIPTION -qj is a job queue that operates on files. It mounts a directory to -_mountpoint_ that is used to inspect and run jobs. +qj is an inspectable job queue that operates on files with retries. It mounts +a directory to _mountpoint_ that is used to inspect and run jobs. qj exposes 4 directories, where each directory contains zero or more _jobs_. Job names must be unique across all four directories. The directories are: