filed
Job queue using FUSE
git clone git://mccd.space/filed
| Log | Files | Refs | README | LICENSE |
commit c459f45b02772a01c7b08a70a446cd29206391b7 parent 8b2b7c98bbed7b7acdf825b5167daed0a7537a36 Author: Marc <marc@coquand.email> Date: Mon, 29 Jun 2026 16:05:45 +0200 Change README Diffstat:
| A | README | | | 179 | +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ |
| D | README.md | | | 181 | ------------------------------------------------------------------------------- |
2 files changed, 179 insertions(+), 181 deletions(-)
diff --git a/README b/README
@@ -0,0 +1,179 @@
+FILE D'ATTENTE
+
+ *File d'attente* (queue in French) is a concurrent, file-based
+ job queue, written in Go.
+
+ File d'attente uses files and directories for queue manipulation.
+ Create a job with "`printf cmd > /pending/$id`", view running
+ jobs with "`ls /active`", and restart a failed job with "`mv
+ /failed/$id /pending`".
+
+ The tool is intended for single-server workloads, as a companion
+ queue to another application. File d'attente comes with
+ sandboxing, automatic retries, timeout, and backoff built-in.
+
+INSTALLATION
+
+ File d'attente is built in Go and depends on sqlite and fuse
+ (make sure fusermount is available in path).
+
+ $ git clone https://sr.ht/~marcc/filed/
+ $ cd filed
+ $ go install
+ $ go install cmd/filed-launch.go
+
+ To build the docs you need [scdoc]
+
+ $ for f in filed*.scd; do
+ scdoc < "$f" > "${f%.scd}"
+ done
+ # mv filed.5 /usr/local/man/man5
+ # mv filed.config.5 /usr/local/man/man5
+ # mv filed-launch.1 /usr/local/man/man1
+
+
+GETTING STARTED
+
+ It is recommended to read the [man pages] for more complete
+ documentation and security considerations, but below is a small
+ example to get you started.
+
+ `filed` requires a job directory and a state file location (defaulting
+ to `XDG_DATA_HOME`). Afterward, you can start the daemon:
+
+ ```sh
+ $ mkdir /tmp/filed-jobs
+ $ filed -rof "/usr/bin/echo" -ro "/lib" /tmp/filed-jobs
+ ```
+
+ `filed` mounts the directory `filed-jobs` and exposes a few files and
+ directories. With the above script, each job will launch in a
+ sandboxed-mode and only have access to `echo` and `lib`.
+
+ A job can then be added by creating a file in the newly available
+ pending directory:
+
+ ```sh
+ $ printf "echo 'hello world'" > /tmp/filed-jobs/pending/1
+ ```
+
+ If all went well, you can see the job output in `/complete`:
+
+ ```sh
+ $ cat /tmp/filed-jobs/complete/1
+ >>> echo 'hello world'
+ hello world
+ ```
+
+ By default, a job retries 3 times, and if unsuccessful, gets moved to
+ `/failed`. You can inspect the logs to see what went wrong:
+
+ ```sh
+ $ printf "ech this-will-fail" > /tmp/filed-jobs/pending/2
+ # Wait for a bit until it finishes retrying
+ $ cat /tmp/filed-jobs/failed/2
+ >>> ech this-will-fail
+ sh: 1: ech: not found
+
+
+ [System Error]: exit status 127
+ ```
+
+ You can restart a job by moving the job back to pending:
+
+ ```sh
+ $ mv /tmp/filed-jobs/failed/2 /tmp/filed-jobs/pending
+ ```
+
+ Finally, if you want to remove completed or failed jobs:
+
+ ```sh
+ $ rm /tmp/filed-jobs/failed/2
+ ```
+
+DOCUMENTATION, SECURITY CONSIDERATIONS, MAINTENANCE
+
+ Available in the manpages:
+
+ - [filed.5]
+ - [filed.config.5]
+ - [filed-launch.1]
+
+DESIGN & MOTIVATION
+
+ I wanted to create a queue that would be easy to use for
+ self-hosted web applications, that could be used by any
+ programming language. I also wanted to make it easy for admins
+ to understand why a job fails, and to rerun jobs if there is an
+ error.
+
+ I was inspired by 9p, and files proved to be a great
+ abstraction since directories model state transitions quite
+ well. File d'attente makes it very easy to inspect the state,
+ without needing to build an admin portal with separate sign in.
+ Instead, all admin operations can be done by just SSHing into
+ the server, and the operations for manipulating, securing and
+ automating the system become very intuitive. The source code
+ can then be very slimmed down, while still packing a lot of
+ features.
+
+TODO
+ - [x] Support chmod and chown
+ - [x] State is configured via environment variable
+ - [x] Customizable backoff and timeout before retries
+ - [x] Last modified and created at are correctly rendered for jobs
+ - [x] "Landlock"-mode for sandboxing
+ - [x] Add filed-launch - a script that can be used to restrict
+ command access
+ - [x] Add command arguments to filed to lock it down, but still
+ allow it access to state files, and remove that access in
+ filed-launch
+ - [ ] Support landlock cli to only take -ro or -rw, use stat to determine if it's a file.
+ - [ ] Support network restrictions
+ - [ ] A reusable systemd unit file
+ - [ ] Notification on failure. Unfortunately [inotify does not work
+ with fuse], which would have been elegant otherwise.
+ - [ ] Notify forget and other updates.
+ - [ ] Package for Alpine Linux (with reusable openrc script)
+ - [ ] Add support for removing/moving active jobs
+ - [ ] When moved to failed, the job should be killed immediately
+ - [x] When removed, the job should be killed immediately
+
+CONTRIBUTING
+
+ bugs/patches can be submitted by email to ~marcc/public-inbox@lists.sr.ht
+
+STATUS
+
+ File d'attente is tested, but not battle-tested. There are probably
+ quite a few warts and inefficiencies.
+
+ALTERNATIVES
+
+ - [nq] - `nq` is simpler and not a persistent process, but does not
+ feature retries. They serve different purposes: `nq` for ad-hoc
+ queuing of command lines. `filed` serves well as a job manager for
+ your server, where you want admins to see jobs and be able to rerun
+ them.
+ - [task-spooler] - `ts` has better control over how you want the task
+ executed (GPU or CPU), and a lot of other features. It does (AFAIK)
+ not support retries, which are supported in `filed`.
+ - [bull] - `bull` is only for node and javascript. It features a
+ graphical UI, and a few other features not found in `filed`. `filed`
+ eschews a GUI in favor of simple files, allowing it to better
+ interoperate with other systems, and allows it to use regular unix
+ permissions for access management.
+ - sqs - requires you to setup most infrastructure around retries
+ yourself. sqs is far more complex, more focused on message passing,
+ harder to inspect, but far more flexible. Sqs scales better and fits
+ more workloads.
+
+[nq]: https://github.com/leahneukirchen/nq
+[task-spooler]: https://github.com/justanhduc/task-spooler
+[bull]: https://www.npmjs.com/package/bull
+[man pages]: https://git.sr.ht/~marcc/filed/tree/main/item/filed.5.scd
+[scdoc]: https://git.sr.ht/~sircmpwn/scdoc
+[inotify does not work with fuse]: https://github.com/bazil/fuse/issues/188
+[filed.5]: https://git.sr.ht/~marcc/filed/tree/main/item/filed.5.scd
+[filed.config.5]: https://git.sr.ht/~marcc/filed/tree/main/item/filed.config.5.scd
+[filed-launch.1]: https://git.sr.ht/~marcc/filed/tree/main/item/filed-launch.1.scd
diff --git a/README.md b/README.md
@@ -1,181 +0,0 @@
-# File d'attente
-
-*File d'attente* (queue in French) is a concurrent, file-based job
-queue, written in Go.
-
-File d'attente uses files and directories for queue manipulation.
-Create a job with "`printf cmd > /pending/$id`", view running jobs
-with "`ls /active`", and restart a failed job with "`mv /failed/$id
-/pending`".
-
-The tool is intended for single-server workloads, as a companion queue
-to another application. File d'attente comes with sandboxing,
-automatic retries, timeout, and backoff built-in.
-
-## Installation
-
-File d'attente is built in Go and depends on sqlite and fuse (make
-sure fusermount is available in path).
-
-```sh
-$ git clone https://sr.ht/~marcc/filed/
-$ cd filed
-$ go install
-$ go install cmd/filed-launch.go
-```
-
-To build the docs you need [scdoc]
-
-```sh
-$ for f in filed*.scd; do
- scdoc < "$f" > "${f%.scd}"
-done
-# mv filed.5 /usr/local/man/man5
-# mv filed.config.5 /usr/local/man/man5
-# mv filed-launch.1 /usr/local/man/man1
-```
-
-## Getting started
-
-It is recommended to read the [man pages] for more complete
-documentation and security considerations, but below is a small
-example to get you started.
-
-`filed` requires a job directory and a state file location (defaulting
-to `XDG_DATA_HOME`). Afterward, you can start the daemon:
-
-```sh
-$ mkdir /tmp/filed-jobs
-$ filed -rof "/usr/bin/echo" -ro "/lib" /tmp/filed-jobs
-```
-
-`filed` mounts the directory `filed-jobs` and exposes a few files and
-directories. With the above script, each job will launch in a
-sandboxed-mode and only have access to `echo` and `lib`.
-
-A job can then be added by creating a file in the newly available
-pending directory:
-
-```sh
-$ printf "echo 'hello world'" > /tmp/filed-jobs/pending/1
-```
-
-If all went well, you can see the job output in `/complete`:
-
-```sh
-$ cat /tmp/filed-jobs/complete/1
->>> echo 'hello world'
-hello world
-```
-
-By default, a job retries 3 times, and if unsuccessful, gets moved to
-`/failed`. You can inspect the logs to see what went wrong:
-
-```sh
-$ printf "ech this-will-fail" > /tmp/filed-jobs/pending/2
-# Wait for a bit until it finishes retrying
-$ cat /tmp/filed-jobs/failed/2
->>> ech this-will-fail
-sh: 1: ech: not found
-
-
-[System Error]: exit status 127
-```
-
-You can restart a job by moving the job back to pending:
-
-```sh
-$ mv /tmp/filed-jobs/failed/2 /tmp/filed-jobs/pending
-```
-
-Finally, if you want to remove completed or failed jobs:
-
-```sh
-$ rm /tmp/filed-jobs/failed/2
-```
-
-## Documentation, security considerations, maintenance etc.
-
-Available in the manpages:
-
-- [filed.5]
-- [filed.config.5]
-- [filed-launch.1]
-
-## Design & Motivation
-
-I wanted to create a queue that would be easy to use for self-hosted
-web applications, that could be used by any programming language. I
-also wanted to make it easy for admins to understand why a job fails,
-and to rerun jobs if there is an error.
-
-I was inspired by 9p, and files proved to be a great abstraction since
-directories model state transitions quite well. File d'attente makes
-it very easy to inspect the state, without needing to build an admin
-portal with separate sign in. Instead, all admin operations can be
-done by just SSHing into the server, and the operations for
-manipulating, securing and automating the system become very
-intuitive. The source code can then be very slimmed down, while still
-packing a lot of features.
-
-## TODO
-
-- [x] Support chmod and chown
-- [x] State is configured via environment variable
-- [x] Customizable backoff and timeout before retries
-- [x] Last modified and created at are correctly rendered for jobs
-- [x] "Landlock"-mode for sandboxing
- - [x] Add filed-launch - a script that can be used to restrict
- command access
- - [x] Add command arguments to filed to lock it down, but still
- allow it access to state files, and remove that access in
- filed-launch
-- [ ] Support landlock cli to only take -ro or -rw, use stat to determine if it's a file.
-- [ ] Support network restrictions
-- [ ] A reusable systemd unit file
-- [ ] Notification on failure. Unfortunately [inotify does not work
- with fuse], which would have been elegant otherwise.
-- [ ] Notify forget and other updates.
-- [ ] Package for Alpine Linux (with reusable openrc script)
-- [ ] Add support for removing/moving active jobs
- - [ ] When moved to failed, the job should be killed immediately
- - [x] When removed, the job should be killed immediately
-
-## Contributing
-
-bugs/patches can be submitted by email to ~marcc/public-inbox@lists.sr.ht
-
-## Status
-
-File d'attente is tested, but not battle-tested. There are probably
-quite a few warts and inefficiencies.
-
-## Alternatives
-
-- [nq] - `nq` is simpler and not a persistent process, but does not
- feature retries. They serve different purposes: `nq` for ad-hoc
- queuing of command lines. `filed` serves well as a job manager for
- your server, where you want admins to see jobs and be able to rerun
- them.
-- [task-spooler] - `ts` has better control over how you want the task
- executed (GPU or CPU), and a lot of other features. It does (AFAIK)
- not support retries, which are supported in `filed`.
-- [bull] - `bull` is only for node and javascript. It features a
- graphical UI, and a few other features not found in `filed`. `filed`
- eschews a GUI in favor of simple files, allowing it to better
- interoperate with other systems, and allows it to use regular unix
- permissions for access management.
-- sqs - requires you to setup most infrastructure around retries
- yourself. sqs is far more complex, more focused on message passing,
- harder to inspect, but far more flexible. Sqs scales better and fits
- more workloads.
-
-[nq]: https://github.com/leahneukirchen/nq
-[task-spooler]: https://github.com/justanhduc/task-spooler
-[bull]: https://www.npmjs.com/package/bull
-[man pages]: https://git.sr.ht/~marcc/filed/tree/main/item/filed.5.scd
-[scdoc]: https://git.sr.ht/~sircmpwn/scdoc
-[inotify does not work with fuse]: https://github.com/bazil/fuse/issues/188
-[filed.5]: https://git.sr.ht/~marcc/filed/tree/main/item/filed.5.scd
-[filed.config.5]: https://git.sr.ht/~marcc/filed/tree/main/item/filed.config.5.scd
-[filed-launch.1]: https://git.sr.ht/~marcc/filed/tree/main/item/filed-launch.1.scd