GNU ELPA - el-job

I may write yet another variant.

Something that came with experience is that it's best to make a new variant library for a narrow use-case, rather than complicate one library with different code flows. When it comes to this type of library, you really want to keep it easy to reason about!

Ideas as of [2025-10-06 Mon]:

File IPC

Sending input and output by writing and reading files, instead of through the pipe connection.

Theory: performance can sometimes be a lot better, with large inputs and outputs. I would guess it depends a lot on the machine.

Drawback: if the data sent is sensitive, those files probably should be encrypted, and that could negate any performance benefit.

Worker daemons

Keeping subprocesses alive forever, so they are available at beck and call – think worker daemons.

That's basically implemented in "el-job-old", and partly why it got so hairy, but it could be remade to put this usage front-and-center, with a "happy path" UX.

At this time, I do not have a demanding use-case to experiment with, to discover what that happy path should be, how the affordances should be.

If you need this now, I recommend async.el.

I wanted to shorten the round-trip as much as possible, between the start of an async task and having the results.

For example, say you have some lisp that collects completion candidates, and you want to run it asynchronously because the lisp you wrote isn't always fast enough to avoid the user's notice, but you'd still like it to return as soon as possible.

• Jobs must now have :inputs. If :inputs nil and there was nothing queued, el-job-launch will no-op and return the symbol inputs-were-empty.

• Some renames to follow Elisp convention
  • el-job:timestamps and friends now el-job-timestamps.

• DROP SUPPORT Emacs 28
  • It likely has not been working for a while anyway. Maybe works on the v0.3 branch, from 0.3.26+.

• Jobs must now have :id (no more anonymous jobs).
• Pruned many code paths.

• Changed internals so that all builds of Emacs can be expected to perform similarly well.

• No longer keeps processes alive forever. All jobs are kept alive for up to 30 seconds of disuse, then reaped.
• Pruned many code paths.
• Many arguments changed, and a few were removed. Consult the docstring of el-job-launch again.

1. The return value from the :funcall-per-input function must always be a list with a fixed length, where the elements are also lists.

   For example, org-mem passes :funcall-per-input #'org-mem-parser--parse-file to el-job, and if you look in org-mem-parser.el for the defun of org-mem-parser--parse-file, it always returns a list of 5 items:

   (list (if missing-file (list missing-file)) ; List of 0 or 1 item
         (if file-mtime (list file-mtime))     ; List of 0 or 1 item
         found-entries                         ; List of many items
         org-node-parser--found-links          ; List of many items
         (if problem (list problem))))         ; List of 0 or 1 item
   

   It may look clunky to return sub-lists of only one item, but you could consider it a minor expense in exchange for simpler library code.

2. Some data types cannot be exchanged with the children: those whose printed form look like #<...>. For example, #<buffer notes.org>, #<obarray n=94311>, #<marker at 3102 in README.org>.

   IIUC, this sort of data only has meaning within the current process – so even if you could send it, it would not be usable by the recipient anyway.

3. For now, this library tends to be applicable only to a narrow set of use-cases, since you can only pass one :inputs list which would tend to contain a single kind of thing, e.g. it could be a list of files to visit, to be split between child processes. In many potential use-cases, you'd actually want multiple input lists and split them differently, and that's not supported yet.