filed
Job queue using FUSE
git clone git://mccd.space/filed
| Log | Files | Refs | README | LICENSE |
commit c50459b579f1246acc81f98b35bf00f7dfef51d3 parent 7a885c232ae6ca4dab8097e9cad9811d6e280cc9 Author: Marc Coquand <marc@coquand.email> Date: Tue, 16 Dec 2025 17:13:13 +0100 Docs Diffstat:
| M | README.md | | | 40 | ++++++++++++++++++++-------------------- |
1 file changed, 20 insertions(+), 20 deletions(-) diff --git a/README.md b/README.md @@ -1,10 +1,10 @@ # File d'attente -*File d'attente* (acr. filed, French for queue) is a file-base job queue, written in Go. +*File d'attente* (queue in French) is a file-based job queue, written in Go. -It is intended for trusted, single-server workloads, as a companion queue to another application, that makes it easy to inspect and rerun jobs that have failed. It comes with automatic retries, timeout, backoff built-in. +The tool is intended for trusted, single-server workloads, as a companion queue to another application. File d'attente makes it easy to inspect and rerun jobs that have failed, and comes with automatic retries, timeout, backoff built-in. -File d'attente uses files and directories for manipulation. Create a job is by `printf myjob > /pending/$id`, view running jobs with `ls /active`, and restarting a failed job with `mv /failed/$id /pending`. +File d'attente uses files and directories for manipulation. Create a job with `printf cmd > /pending/$id`, view running jobs with `ls /active`, and restart a failed job with `mv /failed/$id /pending`. File d'attente works well with other tools in the ecosystem, for example: @@ -15,16 +15,16 @@ File d'attente works well with other tools in the ecosystem, for example: File d'attente is built in Go and depends on sqlite and fuse (make sure fusermount is available in path). -``` -git clone https://sr.ht/~marcc/filed/ -cd filed -go build -go install +```sh +$ git clone https://sr.ht/~marcc/filed/ +$ cd filed +$ go build +$ go install ``` To build the docs you need [scdoc] -``` +```sh $ scdoc < filed.1.scd > filed.1 # mv filed.1 /usr/local/man/man1 ``` @@ -35,7 +35,7 @@ It is recommended to read the [man pages] for more complete documentation and se `filed` requires a job directory and a state file location (defaulting to `XDG_DATA_HOME`). Afterward, you can start the daemon: -``` +```sh $ mkdir /tmp/filed-jobs $ filed /tmp/filed-jobs ``` @@ -44,19 +44,21 @@ $ filed /tmp/filed-jobs A job can then be added by creating a file in the newly available pending directory: -``` +```sh $ printf "echo 'hello world'" > /tmp/filed-jobs/pending/1 ``` If all went well, you can see the job output in `/complete`: -``` +```sh $ cat /tmp/filed-jobs/complete/1 ``` By default, a job retries 3 times, and if unsuccessful, gets moved to `/failed`. You can inspect the logs to see what went wrong: -``` +```sh +$ printf "ech this-will-fail" > /tmp/filed-jobs/pending/2 +# Wait for a bit until it finishes retrying $ cat /tmp/filed-jobs/failed/2 >>> ech this-will-fail sh: 1: ech: not found @@ -65,25 +67,23 @@ sh: 1: ech: not found [System Error]: exit status 127 ``` -And you can restart a job by moving the job back to pending: +You can restart a job by moving the job back to pending: -``` +```sh $ mv /tmp/filed-jobs/failed/2 /tmp/filed-jobs/pending ``` Finally, if you want to remove a completed or failed jobs: -``` +```sh $ rm /tmp/filed-jobs/failed/2 ``` ## Design & Motivation -If you're building any kind of web application, at some point you probably need a job queue, whether that's for sending an email confirmation or batch processing some files async. - -Since I often use languages with a small ecosystem, I wanted to build a tool that was easy to integrate with, with just enough features, regardless of what language I use. I also wanted to make it easy for admins to understand why a job fails, and to rerun jobs if there is an error. I was inspired by plan9, and files proved to be a great abstraction. File d'attente makes it very easy to inspect the state, without needing to expose a admin portal with separate sign in. Instead, all admin operations can be done by just SSHing into the server, and the operations are very intuitive. +I wanted to create a queue that would be easy to use for self-hosted web applications, that could be used by any programming language. I also wanted to make it easy for admins to understand why a job fails, and to rerun jobs if there is an error. -The simple file-based API also allows me to slim down the amount of code needed to write considerably, while still exposing a very scriptable and easy-to-understand interface. +I was inspired by 9p, and files proved to be a great abstraction since directories model state transitions quite well. File d'attente makes it very easy to inspect the state, without needing to build an admin portal with separate sign in. Instead, all admin operations can be done by just SSHing into the server, and the operations for manipulating, securing and automating the system become very intuitive. The source code can then be very slimmed down, while still packing a lot of features. ## TODO