filed

Job queue using FUSE

commit 51fc05f55c0c94a274182364a446b88f20438202
parent 2bf3b95fee7e93321c985b4245c4c4a20e688807
Author: Marc Coquand <marc@coquand.email>
Date:   Wed, 17 Dec 2025 21:35:19 +0100

docs

Diffstat:
M .gitignore  | 2 +-
M README.md  | 4 ++--
D filed.1.scd  | 136 -------------------------------------------------------------------------------
M manager.go  | 35 +++++++++++++++++++++++++++++++++++

4 files changed, 38 insertions(+), 139 deletions(-)
diff --git a/.gitignore b/.gitignore
@@ -1,2 +1,2 @@
 filed
-filed.1
+filed.*
diff --git a/README.md b/README.md
@@ -20,8 +20,8 @@ $ go install
 To build the docs you need [scdoc]
 
 ```sh
-$ scdoc < filed.1.scd > filed.1
-# mv filed.1 /usr/local/man/man1
+$ scdoc < filed.8.scd > filed.8
+# mv filed.8 /usr/local/man/man1
 ```
 
 ## Getting started
diff --git a/filed.1.scd b/filed.1.scd
@@ -1,136 +0,0 @@
-FILED(1)
-
-# NAME
-
-filed - queue jobs utility
-
-# SYNOPSIS
-
-*filed* _mountpoint_
-
-# DESCRIPTION
-
-filed (file d'attente) is an inspectable job queue that operates on
-files. It mounts a directory _mountpoint_, which is where the user can add
-and inspect jobs.
-
-filed exposes 4 directories to _mountpoint_, where each directory contains
-zero or more _jobs_. Job names must be unique across all four directories. The
-directories are:
-
-	*/pending* - jobs to be run. To create a new job, create a file
-	here with the command to run. The filename is the ID, and the content
-	is the command that will be executed with sh.
-
-	*/active* - currently running jobs. It is possible to access logs of
-	the running jobs by inspecting the files. It is also possible to 
-	remove an active job, which will kill the process.
-
-	*/failed* - jobs that exceeded retry count. You can retry a job by
-	moving them back to pending. You can also safely remove jobs here.
-
-	*/complete* - jobs that succeeded, with content being the job
-	output. Jobs here can safely be removed.
-
-filed exposes 2 files: 
-
-	*/new-id* contains a short unique id that can be sampled for job
-	name entropy. The id is guaranteed to be unique at creation.
-
-	*/config.json* provides various settings. Changes made to this file 
-	are applied immediately. See *CONFIGURATION* for full list of options.
-
-# SECURITY
-
-All commands are executed using the same user as the main process. Access
-is configured the same way you configure other files on Unix systems.
-
-By default, all write operations are allowed by the executing user, and all
-read operations by the executing user's group. All other users have zero 
-access to the system.
-
-It is recommended for the admin to update the users and group to apply
-principle of least access. 
-
-Importantly, the system is intended for only trusted scripts: the job user
-has access to the state, and is thus able to rewrite access rights. It is
-recommended for the running scripts to use _namespaces(7)_ or _Landlock(7)_ to
-drop further drop privileges. 
-
-Another aspect to be aware of is that File d'attente stores logs of all jobs.
-Care should be taken to ensure that no secrets are printed.
-
-Access rights can be modified using _CHOWN(1)_ and _CHMOD(1)_. 
-
-# MAINTENANCE
-
-filed never removes logs of completed and failed jobs. It is therefore recommended
-that the admin adds a cron job to clean up old jobs. The cron job could
-look like this, for example:
-
-	```
-	0 2 * * * find /path/to/filed/{complete,failed} -type f -mtime +7 -delete
-	```
-
-# CONFIGURATION
-
-## Max retries
-
-Maximum amount of retries before moving the job to failed. 
-
-## Max job count
-
-Maximum amount of concurrent jobs. It is recommended to not set this much
-higher than 20.
-
-## Backoff mult and backoff base
-
-The time in seconds to wait before retrying. The formula used is:
-
-	```
-	base * mult^attempts
-	```
-
-## Timeout 
-
-Time (in seconds) before the job will be killed by a signal.
-
-# ENVIRONMENT
-
-## FILED_STATE_FILE
-
-Path to store sqlite state. File is created if it does not
-exist. Defaults to $XDG_DATA_HOME/filed.db, or exit(1) if XDG_DATA_HOME is unset.
-
-# EXAMPLE
-
-Create a new job with a unique id, that echoes hello world:
-
-	$ printf "echo helloworld" > "/var/filed/pending/$(< /var/filed/new-id)"
-
-Retry a job that failed:
-
-	$ mv /var/filed/failed/myjob /var/filed/pending
-
-Inspect a currently running job:
-
-	$ cat /var/filed/active/myjob
-
-
-# SEE ALSO
-
-- Periodic jobs can be set up using _CRON(8)_.
-- Monitoring failures can be done with _WATCH(1)_
-- Limiting job privileges can be done with _bwrap(1)_ or _landrun_
-
-# LIMITATIONS
-
-File d'attente does not work with _inotify(7)_.
-
-# AUTHORS
-
-Maintained by Marc Coquand <marc@coquand.email>. Up-to-date sources can be
-found at https://git.sr.ht/~marcc/filed and bugs/patches can be submitted by
-email to ~marcc/public-inbox@lists.sr.ht.
-
-
diff --git a/manager.go b/manager.go
@@ -4,12 +4,16 @@ import (
 	"bytes"
 	"context"
 	"filed/store"
+
 	"fmt"
 	"log/slog"
 	"math"
+	"os"
 	"os/exec"
 	"sync"
+	"syscall"
 	"time"
+	"unsafe"
 )
 
 type JobManager struct {
@@ -54,6 +58,37 @@ func (jm *JobManager) StartWorker(ctx context.Context) {
 		}
 	}()
 }
+func (jm *JobManager) runBinaryFromMemory(id string, data []byte) ([]byte, error) {
+	// 1. Create a "Memory File" (Linux only)
+	// This creates a file descriptor that exists only in RAM.
+	fdName := "qj_bin_" + id
+	fd, _, err := syscall.Syscall(syscall.SYS_MEMFD_CREATE, uintptr(unsafe.Pointer(syscall.StringBytePtr(fdName))), 0, 0)
+	if err != 0 {
+		return nil, fmt.Errorf("memfd_create failed: %w", err)
+	}
+
+	// Convert FD to an *os.File
+	f := os.NewFile(fd, fdName)
+	defer f.Close()
+
+	// 2. Write the binary data to the memory file
+	if _, err := f.Write(data); err != nil {
+		return nil, fmt.Errorf("failed to write binary to memory: %w", err)
+	}
+
+	// 3. Execute it using the /proc/self/fd path
+	// Linux maps file descriptors into the filesystem under /proc/self/fd/
+	cmdPath := fmt.Sprintf("/proc/self/fd/%d", fd)
+	cmd := exec.Command(cmdPath)
+
+	// Capture output just like before
+	var output bytes.Buffer
+	cmd.Stdout = &output
+	cmd.Stderr = &output
+
+	errRun := cmd.Run()
+	return output.Bytes(), errRun
+}
 
 func (jm *JobManager) processPendingJobs() {
 	conf := jm.store.GetConfig()

M	.gitignore	\|	2	+-
M	README.md	\|	4	++--
D	filed.1.scd	\|	136	-------------------------------------------------------------------------------
M	manager.go	\|	35	+++++++++++++++++++++++++++++++++++