filed
Job queue using FUSE
git clone git://mccd.space/filed
| Log | Files | Refs | README | LICENSE |
commit 7a885c232ae6ca4dab8097e9cad9811d6e280cc9 parent 0378369b3b7709033e8e6ce888c332d60c71789e Author: Marc Coquand <marc@coquand.email> Date: Tue, 16 Dec 2025 16:59:27 +0100 Docs Diffstat:
| M | README.md | | | 32 | ++++++++++++++------------------ |
1 file changed, 14 insertions(+), 18 deletions(-) diff --git a/README.md b/README.md @@ -1,12 +1,14 @@ # File d'attente -File d'attente (acronym filed) is a file-base job queue. It is intended for trusted, single-server workloads, as a companion queue to another application, while also allowing admins to easily inspect and rerun jobs that have failed. +*File d'attente* (acr. filed, French for queue) is a file-base job queue, written in Go. -File d'attente uses files and directories for manipulation. Creating a job is as simple as adding it to `/pending`, viewing running jobs is as simple as running `ls /active`, and restarting a failed job is as simple as `mv /failed/job /pending`. +It is intended for trusted, single-server workloads, as a companion queue to another application, that makes it easy to inspect and rerun jobs that have failed. It comes with automatic retries, timeout, backoff built-in. -File d'attente aims to be a good unix citizen, and works well with other tools in the ecosystem, for example: +File d'attente uses files and directories for manipulation. Create a job is by `printf myjob > /pending/$id`, view running jobs with `ls /active`, and restarting a failed job with `mv /failed/$id /pending`. -- Periodic jobs using cron +File d'attente works well with other tools in the ecosystem, for example: + +- Create periodic jobs using cron - Authnz using built-in unix users ## Installation @@ -27,11 +29,11 @@ $ scdoc < filed.1.scd > filed.1 # mv filed.1 /usr/local/man/man1 ``` -## Basic Principles +## Getting started -File d'attente installs a CLI tool called `filed`. The job queue operates using files. Each job corresponds to one file, and the directories indicate the job state. It is recommended to read the [man pages] for more complete documentation and security considerations. +It is recommended to read the [man pages] for more complete documentation and security considerations, but below is a small example to get you started. -`filed` requires a job directory and a state file location (defaulting to `XDG_DATA_HOME`). +`filed` requires a job directory and a state file location (defaulting to `XDG_DATA_HOME`). Afterward, you can start the daemon: ``` $ mkdir /tmp/filed-jobs @@ -40,7 +42,7 @@ $ filed /tmp/filed-jobs `filed` mounts the directory `filed-jobs` and exposes a few files and directories. -A job can then be added by creating a file in the pending directory: +A job can then be added by creating a file in the newly available pending directory: ``` $ printf "echo 'hello world'" > /tmp/filed-jobs/pending/1 @@ -52,7 +54,7 @@ If all went well, you can see the job output in `/complete`: $ cat /tmp/filed-jobs/complete/1 ``` -By default, a job retries 3 times, and if unsuccessful, it is moved to the `failed` directory. You can inspect the logs to see what went wrong: +By default, a job retries 3 times, and if unsuccessful, gets moved to `/failed`. You can inspect the logs to see what went wrong: ``` $ cat /tmp/filed-jobs/failed/2 @@ -77,17 +79,11 @@ $ rm /tmp/filed-jobs/failed/2 ## Design & Motivation -If you're building any kind of web application, at some point you probably need a job queue. Whether that's for sending an email confirmation or batch process some files, many apps need it. - -Often these jobs can fail, whether that's due to network errors, memory issues or something else, so some retry mechanism is necessary. Being able to easily inspect the jobs and rerun them as an admin is also very important, so that the admin can fix whatever issue caused it to fail in the first place. - -I wanted a tool that I could incorporate and use with whatever programming language I desired, and that makes it easy to understand when a job fail and rerun jobs if there is an error. `filed` is very intuitive to build an integration for: just write a file telling it what to execute. - -I also wanted a tool that made it simple to inspect, without needing to expose a admin portal with separate sign in. `filed` allows you to inspect and operate the queue just by SSHing into the server, and reuses the decades old identity system already built into Linux. +If you're building any kind of web application, at some point you probably need a job queue, whether that's for sending an email confirmation or batch processing some files async. -The simple file-based API of File d'attente, also allows me to slim down the amount of code needed to write considerably, while still exposing a very scriptable and easy-to-understand interface. +Since I often use languages with a small ecosystem, I wanted to build a tool that was easy to integrate with, with just enough features, regardless of what language I use. I also wanted to make it easy for admins to understand why a job fails, and to rerun jobs if there is an error. I was inspired by plan9, and files proved to be a great abstraction. File d'attente makes it very easy to inspect the state, without needing to expose a admin portal with separate sign in. Instead, all admin operations can be done by just SSHing into the server, and the operations are very intuitive. -I've tried a few other queue tools: sqs/sns, rabbitmq, bull, systemd-run. The first two felt heavyweight, and required setting up a lot of infrastructure, especially if you want to rerun and inspect jobs. It felt like far too much work for a simple app. Bull was more in line with what I wanted, but I think operating on files is simpler for building custom automation, and easier to secure. Systemd-run lacked the retry functionality and the interface was rather clunky. +The simple file-based API also allows me to slim down the amount of code needed to write considerably, while still exposing a very scriptable and easy-to-understand interface. ## TODO