filed

Job queue using FUSE

git clone git://mccd.space/filed

commit 3f109ffe1503c796a634b0bf00a0174b14820680
parent a74e244a301be15ef78d6aa7a0afe12a88459b57
Author: Marc Coquand <marc@coquand.email>
Date:   Tue, 16 Dec 2025 11:50:03 +0100

*

Diffstat:

M.gitignore | 4++--


MREADME.md | 50+++++++++++++++++++++++++-------------------------


Mconfig.go | 2+-


Afiled.1.scd | 123+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++


Mgo.mod | 2+-


Mjobdir.go | 2+-


Mmain.go | 10+++++-----


Mmanager.go | 2+-


Mnewid.go | 2+-


Mpendingdir.go | 2+-


Dqj.1.scd | 122-------------------------------------------------------------------------------


Msetfattr.go | 2+-


12 files changed, 162 insertions(+), 161 deletions(-)
diff --git a/.gitignore b/.gitignore
@@ -1,2 +1,2 @@
-qj
-qj.1
+filed
+filed.1
diff --git a/README.md b/README.md
@@ -1,71 +1,71 @@
-# qj - queue jobs and inspect
+# File d'attente - Simple Job queue
 
-`qj` is a simple job queue. It is intended for trusted, single-server workloads, and aims to work as a companion queue to another application, while also allowing admins to easily inspect and rerun jobs that have failed.
+File d'attente (acronym filed) is a simple job queue. It is intended for trusted, single-server workloads, as a companion queue to another application, while also allowing admins to easily inspect and rerun jobs that have failed.
 
-`qj` uses files and directories for manipulation. Creating a job is as simple as adding it to `/pending`, viewing running jobs is as simple as running `ls /active`, and restarting a failed job is as simple as `mv /failed/job /pending`.
+File d'attente uses files and directories for manipulation. Creating a job is as simple as adding it to `/pending`, viewing running jobs is as simple as running `ls /active`, and restarting a failed job is as simple as `mv /failed/job /pending`.
 
-`qj` aims to be a good unix citizen, and works well with other tools in the ecosystem, for example:
+File d'attente aims to be a good unix citizen, and works well with other tools in the ecosystem, for example:
 
 - Periodic jobs using cron
 - Authnz using built-in unix users
 
 ## Installation
 
-`qj` is built in Go and depends on sqlite and fuse (make sure fusermount is available in path).
+File d'attente is built in Go and depends on sqlite and fuse (make sure fusermount is available in path).
 
 ```
-git clone https://sr.ht/~marcc/qj/
-cd qj
+git clone https://sr.ht/~marcc/filed/
+cd filed
 go build
 ```
 
 To build the docs you need scdoc
 
 ```
-scdoc < qj.1.scd > qj.1
+scdoc < filed.1.scd > filed.1
 ```
 
 ## Basic Principles
 
-`qj` is a job queue that operates using files. Each job corresponds to one file, and the directories indicate the job state. It is recommended to read the [man pages] for more complete documentation and security considerations.
+File d'attente installs a CLI tool called `filed`. The job queue operates using files. Each job corresponds to one file, and the directories indicate the job state. It is recommended to read the [man pages] for more complete documentation and security considerations.
 
-`qj` requires a job directory and a state file location (defaulting to `XDG_DATA_HOME`).
+`filed` requires a job directory and a state file location (defaulting to `XDG_DATA_HOME`).
 
 ```
-$ mkdir /tmp/qj-jobs
-$ qj /tmp/qj-jobs 
+$ mkdir /tmp/filed-jobs
+$ filed /tmp/filed-jobs
 ```
 
-Once run, `qj` will set up a directory in `qj-jobs` that contains a few files and directories.
+Once run, `filed` will set up a directory in `filed-jobs` that contains a few files and directories.
 
 A job can easily be added by just creating a file in the pending directory:
 
 ```
-$ printf "echo 'hello world'" > /tmp/qj-jobs/pending/1
+$ printf "echo 'hello world'" > /tmp/filed-jobs/pending/1
 ```
 
 If all went well, you can see the job output:
 
 ```
-$ cat /tmp/qj-jobs/complete/1
+$ cat /tmp/filed-jobs/complete/1
 ```
 
 By default, a job retries 3 times, and if unsuccessful is moved to the `failed` directory. You can inspect the logs to see what went wrong:
 
 ```
-$ cat /tmp/qj-jobs/failed/1
+$ cat /tmp/filed-jobs/failed/1
 ```
 
 And you can restart a job by moving the job back to pending:
 
 ```
-$ mv /tmp/qj-jobs/failed/1 /tmp/qj-jobs/pending
+$ mv /tmp/filed-jobs/failed/1 /tmp/filed-jobs/pending
 ```
 
 Finally, if you want to remove a completed or failed jobs:
 
 ```
-$ rm /tmp/qj-jobs/failed/1
+$ rm /tmp/filed-jobs/failed/1
 ```
 
 ## Design & Motivation
@@ -74,11 +74,11 @@ If you're building any kind of web application, at some point you probably need 
 
 Often these jobs can fail, whether that's due to network errors, memory issues or something else, so some retry mechanism is necessary. Being able to easily inspect the jobs and rerun them as an admin is also very important, so that the admin can fix whatever issue caused it to fail in the first place.
 
-I wanted a tool that I could incorporate and use with whatever programming language I desired, and that makes it easy to understand when a job fail and rerun jobs if there is an error. `qj` is very intuitive to build an integration for: just write a file telling it what to execute.
+I wanted a tool that I could incorporate and use with whatever programming language I desired, and that makes it easy to understand when a job fail and rerun jobs if there is an error. `filed` is very intuitive to build an integration for: just write a file telling it what to execute.
 
-I also wanted a tool that made it simple to inspect, without needing to expose a web portal or set up separate auth system. `qj` allows you to inspect and operate the queue just by SSHing into the server, and reuses the decades old proven identity system already built into Linux.
+I also wanted a tool that made it simple to inspect, without needing to expose a web portal or set up separate auth system. `filed` allows you to inspect and operate the queue just by SSHing into the server, and reuses the decades old proven identity system already built into Linux.
 
-The simple file-based API `qj`, inspired by plan9, also allows me to slim down the amount of code needed considerably, while still exposing a very scriptable and easy-to-understand interface.
+The simple file-based API of File d'attente, inspired by plan9, also allows me to slim down the amount of code needed considerably, while still exposing a very scriptable and easy-to-understand interface.
 
 I've tried a few other queue tools: sqs/sns, rabbitmq, bull, systemd-run. The first two felt heavyweight, and required setting up a lot of infrastructure, especially if you want to rerun and inspect jobs. It felt like far too much work for a simple app. Bull was more in line with what I wanted, but I think operating on files is simpler for building custom automation, and easier to secure. Systemd-run lacked the retry functionality and the interface was rather clunky.
 
@@ -94,12 +94,12 @@ I've tried a few other queue tools: sqs/sns, rabbitmq, bull, systemd-run. The fi
 
 ## Alternatives
 
-- [nq] - `nq` is simpler and not a persistent process, but does not feature retries. They serve different purposes: `nq` for ad-hoc queuing of command lines. `qj` serves well as a job manager for your server, where you want admins to see jobs and be able to rerun them.
-- [task-spooler] - `ts` has better control over how you want the task executed (GPU or CPU), and a lot of other features. It does (AFAIK) not support retries, which are supported in `qj`.
-- [bull] - `bull` is only for node and javascript. It features a graphical UI, and a few other features not found in `qj`. `qj` eschews a GUI in favor of simple files, allowing it to better interoperate with other systems, and allows it to use regular unix permissions for access management.
+- [nq] - `nq` is simpler and not a persistent process, but does not feature retries. They serve different purposes: `nq` for ad-hoc queuing of command lines. `filed` serves well as a job manager for your server, where you want admins to see jobs and be able to rerun them.
+- [task-spooler] - `ts` has better control over how you want the task executed (GPU or CPU), and a lot of other features. It does (AFAIK) not support retries, which are supported in `filed`.
+- [bull] - `bull` is only for node and javascript. It features a graphical UI, and a few other features not found in `filed`. `filed` eschews a GUI in favor of simple files, allowing it to better interoperate with other systems, and allows it to use regular unix permissions for access management.
 - sqs - requires you to setup most infrastructure around retries yourself. sqs is far more complex, more focused on message passing, harder to inspect, but far more flexible. Sqs scales better and fits more workloads.
 
 [nq]: https://github.com/leahneukirchen/nq
 [task-spooler]: https://github.com/justanhduc/task-spooler
 [bull]: https://www.npmjs.com/package/bull
-[man pages]: https://git.sr.ht/~marcc/qj/tree/main/item/qj.1.scd
+[man pages]: https://git.sr.ht/~marcc/filed/tree/main/item/filed.1.scd
diff --git a/config.go b/config.go
@@ -3,9 +3,9 @@ package main
 import (
 	"context"
 	"encoding/json"
+	"filed/store"
 	"log/slog"
 	"os"
-	"qj/store"
 	"syscall"
 
 	"bazil.org/fuse"
diff --git a/filed.1.scd b/filed.1.scd
@@ -0,0 +1,123 @@
+FILED(1)
+
+# NAME
+
+filed - queue jobs utility
+
+# SYNOPSIS
+
+*filed* _mountpoint_
+
+# DESCRIPTION
+
+filed (file d'attente) is an inspectable job queue that operates on files
+with retries. It mounts a directory to _mountpoint_ that is used to inspect
+and run jobs.
+
+filed exposes 4 directories, where each directory contains zero or more _jobs_.
+Job names must be unique across all four directories. The directories are:
+
+	*pending* - jobs to be run. To create a new job, create a file 
+	here with the command to run.
+
+	*active* - currently running jobs. It is possible to access logs of
+	the running jobs by inspecting the files.
+
+	*failed* - jobs that exceeded retry count. You can retry a job by
+	moving them back to pending. You can also safely remove jobs here.
+
+	*complete* - jobs that succeeded, with content being the job
+	output. Jobs here can safely be removed.
+
+filed exposes 2 files: 
+
+	*new-id* contains a short unique id that can be sampled for job
+	name entropy. The id is guaranteed to be unique at creation.
+
+	*config.json* provides various settings. Changes made to this file 
+	are applied immediately. See *CONFIGURATION* for full list of options.
+
+# SECURITY
+
+All commands are executed using the same user as the main process. Access
+is configured the same way you configure other files on Unix systems.
+
+By default, all write operations are allowed by the executing user, and all
+read operations by the executing user's group. All other users have zero 
+access to the system.
+
+It is recommended for the admin to update the users and group to apply
+principle of least access. 
+
+Importantly, the system is intended for only trusted scripts: the job user
+has access to the state, and is thus able to rewrite access rights. It is
+recommended for the running scripts to use _namespaces(7)_ or _Landlock(7)_
+to drop privileges. More security features are coming in the future.
+
+Access rights can be modified using _CHOWN(1)_ and _CHMOD(1)_. 
+
+# MAINTENANCE
+
+filed never removes logs of completed and failed jobs. It is therefore recommended
+that the admin adds a cron job to clean up old jobs. The cron job could
+look like this, for example:
+
+	```
+	0 2 * * * find /path/to/filed/{complete,failed} -type f -mtime +7 -delete
+	```
+
+# CONFIGURATION
+
+## Max retries
+
+Maximum amount of retries before moving the job to failed. 
+
+## Max job count
+
+Maximum amount of concurrent jobs
+
+## Backoff mult and backoff base
+
+The time to wait before retrying. The formula is 
+
+	```
+	base * mult^attempts
+	```
+
+## Timeout 
+
+Time before the job will be killed by a signal.
+
+# ENVIRONMENT
+
+## FILED_STATE_FILE
+
+Path to store sqlite state. File is created if it does not
+exist. Defaults to $XDG_DATA_HOME/filed.db, or exit(1) if XDG_DATA_HOME is unset.
+
+# EXAMPLE
+
+Create a new job with a unique id, that echoes hello world:
+
+	$ printf "echo helloworld" > "/var/filed/pending/$(< /var/filed/new-id)"
+
+Retry a job that failed:
+
+	$ mv /var/filed/failed/myjob /var/filed/pending
+
+Inspect a currently running job:
+
+	$ cat /var/filed/active/myjob
+
+
+# SEE ALSO
+
+Periodic jobs can be set up using _CRON(8)_.
+
+# AUTHORS
+
+Maintained by Marc Coquand <marc@coquand.email>. Up-to-date sources can be
+found at https://git.sr.ht/~marcc/filed and bugs/patches can be submitted by
+email to ~marcc/public-inbox@lists.sr.ht.
+
+
diff --git a/go.mod b/go.mod
@@ -1,4 +1,4 @@
-module qj
+module filed
 
 go 1.24.4
 
diff --git a/jobdir.go b/jobdir.go
@@ -2,10 +2,10 @@ package main
 
 import (
 	"context"
+	"filed/store"
 	"log/slog"
 	"os"
 	"os/exec"
-	"qj/store"
 	"runtime"
 	"syscall"
 
diff --git a/main.go b/main.go
@@ -2,11 +2,11 @@ package main
 
 import (
 	"context"
+	"filed/store"
 	"flag"
 	"fmt"
 	"log/slog"
 	"os"
-	"qj/store"
 	"syscall"
 
 	"bazil.org/fuse"
@@ -27,15 +27,15 @@ func main() {
 		usage()
 		os.Exit(2)
 	}
-	dbPath := os.Getenv("QJ_STATE_FILE")
+	dbPath := os.Getenv("FILED_STATE_FILE")
 	if dbPath == "" {
 		xdg_home := os.Getenv("XDG_DATA_HOME")
 		if xdg_home == "" {
-			fmt.Fprintf(os.Stderr, "QJ_STATE_FILE environment variable needs to be set.\n")
+			fmt.Fprintf(os.Stderr, "FILED_STATE_FILE environment variable needs to be set.\n")
 			usage()
 			os.Exit(1)
 		}
-		dbPath = fmt.Sprintf("%s/qj.db", xdg_home)
+		dbPath = fmt.Sprintf("%s/filed.db", xdg_home)
 	}
 
 	userUid := uint32(os.Getuid())
@@ -57,7 +57,7 @@ func main() {
 	slog.Info("Mounting filesystem", "mountpoint", mountpoint)
 	c, err := fuse.Mount(
 		mountpoint,
-		fuse.FSName("qj"),
+		fuse.FSName("filed"),
 		fuse.AllowOther(),
 		fuse.DefaultPermissions(),
 	)
diff --git a/manager.go b/manager.go
@@ -3,11 +3,11 @@ package main
 import (
 	"bytes"
 	"context"
+	"filed/store"
 	"fmt"
 	"log/slog"
 	"math"
 	"os/exec"
-	"qj/store"
 	"sync"
 	"time"
 )
diff --git a/newid.go b/newid.go
@@ -2,10 +2,10 @@ package main
 
 import (
 	"context"
+	"filed/store"
 	"log/slog"
 	"math/rand"
 	"os"
-	"qj/store"
 	"syscall"
 	"time"
 
diff --git a/pendingdir.go b/pendingdir.go
@@ -3,9 +3,9 @@ package main
 import (
 	"bytes"
 	"context"
+	"filed/store"
 	"log/slog"
 	"os"
-	"qj/store"
 	"syscall"
 
 	"bazil.org/fuse"
diff --git a/qj.1.scd b/qj.1.scd
@@ -1,122 +0,0 @@
-QJ(1)
-
-# NAME
-
-qj - queue jobs utility
-
-# SYNOPSIS
-
-*qj* _mountpoint_
-
-# DESCRIPTION
-
-qj is an inspectable job queue that operates on files with retries. It mounts
-a directory to _mountpoint_ that is used to inspect and run jobs.
-
-qj exposes 4 directories, where each directory contains zero or more _jobs_. 
-Job names must be unique across all four directories. The directories are:
-
-	*pending* - jobs to be run. To create a new job, create a file 
-	here with the command to run.
-
-	*active* - currently running jobs. It is possible to access logs of
-	the running jobs by inspecting the files.
-
-	*failed* - jobs that exceeded retry count. You can retry a job by
-	moving them back to pending. You can also safely remove jobs here.
-
-	*complete* - jobs that succeeded, with content being the job
-	output. Jobs here can safely be removed.
-
-qj exposes 2 files: 
-
-	*new-id* contains a short unique id that can be sampled for job
-	name entropy. The id is guaranteed to be unique at creation.
-
-	*config.json* provides various settings. Changes made to this file 
-	are applied immediately. See *CONFIGURATION* for full list of options.
-
-# SECURITY
-
-All commands are executed using the same user as the main process. Access
-is configured the same way you configure other files on Unix systems.
-
-By default, all write operations are allowed by the executing user, and all
-read operations by the executing user's group. All other users have zero 
-access to the system.
-
-It is recommended for the admin to update the users and group to apply
-principle of least access. 
-
-Importantly, the system is intended for only trusted scripts: the job user
-has access to the state, and is thus able to rewrite access rights. It is
-recommended for the running scripts to use _namespaces(7)_ or _Landlock(7)_
-to drop privileges. More security features are coming in the future.
-
-Access rights can be modified using _CHOWN(1)_ and _CHMOD(1)_. 
-
-# MAINTENANCE
-
-qj never removes logs of completed and failed jobs. It is therefore recommended
-that the admin adds a cron job to clean up old jobs. The cron job could
-look like this, for example:
-
-	```
-	0 2 * * * find /path/to/qj/{complete,failed} -type f -mtime +7 -delete
-	```
-
-# CONFIGURATION
-
-## Max retries
-
-Maximum amount of retries before moving the job to failed. 
-
-## Max job count
-
-Maximum amount of concurrent jobs
-
-## Backoff mult and backoff base
-
-The time to wait before retrying. The formula is 
-
-	```
-	base * mult^attempts
-	```
-
-## Timeout 
-
-Time before the job will be killed by a signal.
-
-# ENVIRONMENT
-
-## QJ_STATE_FILE
-
-Path to store sqlite state. File is created if it does not
-exist. Defaults to $XDG_DATA_HOME/qj.db, or exit(1) if XDG_DATA_HOME is unset.
-
-# EXAMPLE
-
-Create a new job with a unique id, that echoes hello world:
-
-	$ printf "echo helloworld" > "/var/qj/pending/$(< /var/qj/new-id)"
-
-Retry a job that failed:
-
-	$ mv /var/qj/failed/myjob /var/qj/pending
-
-Inspect a currently running job:
-
-	$ cat /var/qj/active/myjob
-
-
-# SEE ALSO
-
-Periodic jobs can be set up using _CRON(8)_.
-
-# AUTHORS
-
-Maintained by Marc Coquand <marc@coquand.email>. Up-to-date sources can be
-found at https://git.sr.ht/~marcc/qj and bugs/patches can be submitted by
-email to ~marcc/public-inbox@lists.sr.ht.
-
-
diff --git a/setfattr.go b/setfattr.go
@@ -2,9 +2,9 @@ package main
 
 import (
 	"context"
+	"filed/store"
 	"log/slog"
 	"os"
-	"qj/store"
 	"syscall"
 
 	"bazil.org/fuse"