filed

Job queue using FUSE
filed.5.scd (4212B)
      1 FILED(5)
      2 
      3 # NAME
      4 
      5 filed - queue jobs utility
      6 
      7 # SYNOPSIS
      8 
      9 *filed* [_option_]... _mdir_
     10 
     11 # DESCRIPTION
     12 
     13 filed (file d'attente) is a pseudo-filesystem which provides an inspectable
     14 job queue that operates on files. It mounts a directory _mdir_, which is
     15 where the user can add and inspect jobs.
     16 
     17 All jobs are executed with *filed-launch*(1), allowing you to restrict job
     18 accesses. If one or more _options_ are supplied, filed will launch with 
     19 *landlock*(7) sandbox, restrict itself to only the necessary directories and
     20 files for operations along with the supplied _options_. Jobs thereafter will
     21 have their access further dropped to only access _options_.
     22 
     23 If no _option_ is supplied, *filed* will launch with access unrestricted.
     24 
     25 ## Overview
     26 
     27 Underneath _mdir_, there are the following groups of files and subdirectories:
     28 
     29 _mdir_/pending/\*  
     30 	Jobs to be run. To create a new job, create a file here with
     31 	the command to run. The filename is the ID, and the content is
     32 	the command that will be executed with sh.
     33 
     34 _mdir_/active/\* 
     35 	Currently running jobs. It is possible to access logs of
     36 	the running jobs by inspecting the files. It is also possible to 
     37 	remove an active job, which will kill the process.
     38 
     39 _mdir_/failed/\* 
     40 	Jobs that exceeded retry count. You can retry a job by
     41 	moving them back to pending. You can also safely remove jobs here.
     42 
     43 _mdir_/complete/\* 
     44 	Jobs that succeeded, with content being the job	output. Jobs
     45 	here can safely be removed.
     46 
     47 _mdir_/_new-id_ 
     48 	Contains a short unique id that can be sampled for job name
     49 	entropy. The id is guaranteed to be unique at creation.
     50 
     51 _mdir_/config.json 
     52 	Provides various settings. Changes made to this file are applied
     53 	immediately. See *filed.config*(5) for full list of options.  
     54 
     55 # OPTIONS
     56 
     57 Options for file restrictions can be used multiple times.
     58 
     59 *-rwf* _file_
     60 	Give read, execute and write access to file.
     61 
     62 *-rof* _file_
     63 	Give read and execute access to file.
     64 
     65 *-rw* _dir_
     66 	Give read and execute access to directory.
     67 
     68 *-ro* _dir_
     69 	Give read and execute access to directory.
     70 
     71 # SECURITY
     72 
     73 All commands are executed using the same user as the main process. Access
     74 is configured the same way you configure other files on Unix systems.
     75 
     76 By default, all write operations are allowed by the executing user, and all
     77 read operations by the executing user's group. All other users have zero 
     78 access to the system.
     79 
     80 It is recommended for the admin to update the users and group to apply
     81 principle of least access. 
     82 
     83 It is also recommended to supply _options_ so that *filed* and jobs have
     84 restricted access to the file system.
     85 
     86 Another aspect to be aware of is that File d'attente stores logs of all jobs.
     87 Care should be taken to ensure that no secrets are printed.
     88 
     89 Access rights can be modified using *chown*(1) and *chmod*(1). 
     90 
     91 # MAINTENANCE
     92 
     93 filed never removes logs of completed and failed jobs. It is therefore recommended
     94 that the admin adds a cron job to clean up old jobs. The cron job could
     95 look like this, for example:
     96 
     97 	```
     98 	0 2 * * * find /path/to/filed/{complete,failed} -type f -mtime +7 -delete
     99 	```
    100 
    101 # ENVIRONMENT
    102 
    103 ## FILED_STATE_FILE
    104 
    105 Path to store sqlite state. File is created if it does not
    106 exist. Defaults to $XDG_DATA_HOME/filed.db, or exit(1) if XDG_DATA_HOME is unset.
    107 
    108 # EXAMPLE
    109 
    110 *filed* -rof "/usr/bin/cat" -ro "/lib" -ro "$HOME/some-dir" /var/filed
    111 	Mount *filed* pseudo-filesystem to /var/filed. Restrict scripts that
    112 	can be launched to cat and only allow jobs to access files in some-dir.
    113 
    114 printf "echo helloworld" > "/var/filed/pending/$(< /var/filed/new-id)"
    115 	Create a new job with a unique id sampled from _new-id_, that echoes
    116 	hello world.
    117 
    118 mv /var/filed/failed/myjob /var/filed/pending
    119 	Retry a job that failed.
    120 
    121 cat /var/filed/active/myjob
    122 	Inspect a currently running job.
    123 
    124 # SEE ALSO
    125 
    126 *filed.config*(5) *filed-launch*(1) *landlock*(7)
    127 
    128 - Periodic jobs can be set up using *cron*(8).
    129 - Monitoring failures can be done with *watch*(1)
    130 
    131 # LIMITATIONS
    132 
    133 File d'attente does not work with *inotify*(7).
    134 
    135 # AUTHORS
    136 
    137 Maintained by Marc Coquand <marc@coquand.email>. Up-to-date sources can be
    138 found at https://git.sr.ht/~marcc/filed and bugs/patches can be submitted by
    139 email to ~marcc/public-inbox@lists.sr.ht.
    140