Skip to content

Worker lifecycle

When a worker is born, what it costs while alive, and every way it dies -- including what happens to it when ComfyUI itself stops. The policy is ADR-0019; this page is the plain-language tour, with the crash cases spelled out.

The one-sentence contract, from the ADR: a worker is disposable and its replacement is invisible -- spawned lazily on first use, verified before trusted, killed without ceremony, replaced behind a generation counter. Everything resident in a worker is a cache that must be rebuildable, never the only copy of anything.

Birth: lazy, verified, then trusted

Nothing spawns at ComfyUI startup. Browsing the node menu, loading workflows, even /object_info never spawn a worker -- proxies answer from the metadata snapshot. They may talk to one, though: a proxy's INPUT_TYPES calls _refresh_combo_options, which asks this env's worker on its side lane (send_side) if it is alive, busy or not, and falls back to the cached options when the worker is dead, was never started, or has not yet imported the pack through a real call. A worker comes into existence only once a node from its env actually executes.

The first call pays for: materializing the worker source into a temp dir (ADR-0006), launching the env's interpreter (which imports torch -- the dominant cost, seconds to tens of seconds), the auth handshake, the config push, and the transport canary (the spawn-time channel). A worker whose CPU-tier canary fails verification is refused, not used (verify_transport raises). A GPU-tier canary failure, or a device-UUID mismatch between parent and worker, is a demotion: the worker is still used, with gpu_zero_copy_ok cleared so its CUDA tensors take the CPU shared-memory path.

Life: warm, single-file, and paid for

Once up, the worker stays alive across executions -- that is the point: the second call skips the torch import entirely. The standing cost of a warm worker is ~180-550 MB host RAM plus a CUDA context (and VRAM for whatever models it holds).

That VRAM is released two ways, not one. A host-side idle sweep (a 10 s daemon timer in pool.py) sends full_release to workers that have sat idle long enough, and the worker gives back everything it holds, and ComfyUI's own eviction reaches into the worker from outside: comfy-env registers a stand-in for each of the worker's models in current_loaded_models, so upstream's free_memory can unload it exactly as it unloads a host model (comfy-env's memory management, ADR-0038). An earlier draft of this line said outside eviction no longer happens; that describes a design that was reversed before it shipped.

  • One call at a time, on the main lane. A worker serves a single in-flight call (ADR-0020). Eviction commands are a partial exception: they are answered mid-call only while the worker is blocked in _call_parent waiting on its own callback (progress, VRAM budget). A worker in pure compute answers nothing on that lane, and a parent thread trying to send_command to it gives up after _COMMAND_LOCK_TIMEOUT (30 s) rather than block ComfyUI.
  • Cheap questions have their own lane. A second connection, read by a daemon thread, serves ping, refresh_input_types and fingerprint while the main thread is inside a node, in that node's GIL gaps. It serves nothing else (a release mid-forward would drop pages a kernel is reading), never imports a pack module, and never writes the main socket; a side request that gets no reply within a second is abandoned and the worker is left alone for five seconds. Liveness is still judged on the main lane: a side pong from a worker wedged in a CUDA kernel proves nothing.
  • Health checks are idle-only. A worker idle for more than 60 s gets a ping before its next call; a busy worker is never pestered.

  • What accumulates inside -- loaded models, the object cache, JIT state -- is all rebuildable. That invariant is what makes every death below survivable.

  • Warm is not forever. After the release sweep, the same timer reaps: a worker that is alive, not mid-call, holding nothing, idle past COMFY_ENV_IDLE_REAP_SECONDS (30 minutes by default, 0 disables) and not the running prompt's worker has its process exited, and its next call is a cold start. One class of worker is never reaped: one with a model stand-in registered in current_loaded_models. The loader node's cached output is that stand-in, and only that process can serve it; killing it would turn the next cache-hit prompt into "please reload the model node", which ComfyUI never considered an evictable outcome. Decided as direction in ADR-0019, built 2026-09-13.

Two cold envs referenced by one prompt start in parallel. The pool lock guards membership of the pool only; the spawn itself (process start, ready frame, transport canary) runs with it released, and a second caller for the same env waits on the first's future rather than starting a twin. Until 2026-09-13 the lock was held across the whole spawn, so envs started one after the other and a call to a warm env could not even look its worker up while another env was starting (six envs: 6.3 s serial, 2.9 s parallel).

Death, all six ways

# Trigger What happens
1 Crash or timeout (segfault in a native lib, ADR-0018 kill) The in-flight call fails with a named error; the worker object is permanently retired. The next call gets a fresh worker with a bumped generation -- the caller never sees a dead worker, only a new one. The crash costs the call and the worker's caches, nothing the parent holds. Every kill path -- this one, the health-check restart, the accept timeout after spawn, and shutdown -- goes through one _kill_tree(): the worker is spawned in its own process group, so the kill reaches the Python under the pixi run wrapper, not just the wrapper (until 2026-09-12 it reached only the wrapper, and the real worker lived on with its VRAM, parent pid 1).
2 Clean ComfyUI stop (Ctrl-C, normal exit) An atexit hook sends every worker a shutdown frame, waits 5 s, then kills; temp dirs are removed. Workers die with the parent.
3 ComfyUI killed hard, worker idle (SIGKILL, crash, OOM -- atexit never runs) The idle worker is blocked reading its socket; the parent's death closes it, the read fails, and the worker's own loop exits promptly. No parent needed.
4 ComfyUI killed hard, worker mid-computation The worker is not reading the socket, so it does not notice. It finishes the running call for nobody -- holding its RAM and VRAM the whole time -- and only exits when it tries to send its reply: transport.send raises on the dead socket, the error handler tries to send an error frame, that raises again unhandled, and the process dies. A worker deep in a 30-minute bake outlives its parent by up to 30 minutes.
5 The sweep at next startup The backstop for anything left behind: the next register_nodes() runs _cleanup_stale_workers, which kills persistent_worker.py processes whose parent pid no longer exists, unlinks dead-owner socket files (macOS only -- the unix:// filename embeds the owning pid; Linux uses the abstract namespace and leaves no file), and removes comfyui_pvenv_* temp dirs no live process has in its cwd or command line. Orphans are recognised by the host pid baked into the worker's temp-dir name (comfyui_pvenv_<hostpid>_…), not by the immediate parent: under pixi run a worker whose wrapper died has parent pid 1 and one whose host died has a live wrapper, so a parent test caught neither. A worker whose host is gone is killed as a group. Workers from an older comfy-env, with no host pid in the name, fall back to the parent test.
6 The idle reaper A deliberate death through the crash path (case 1): the record is retired, the process killed as a group, stand-ins answer eviction as already offloaded. Only a worker holding nothing and serving no stand-in qualifies, so nothing is lost that the next call cannot rebuild.

One row per process: the worker record

Since 2026-09-13 the pool holds a WorkerRecord per env, not a bare (worker, generation) tuple. The row carries everything the parent knows about one worker process: the SubprocessWorker, its generation, the reserve high-water it has been charged (held), when it last did anything (last_activity, what the idle sweep reads), the last prompt it ran, and whether its memory manager has been reported. It still unpacks as worker, gen = entry, so nothing that read the tuple had to change.

Two rules make the row safe without a lock of its own. It is born in one place, after the transport canary passes, and replaced, never mutated in place, when the process is replaced: a reader that took a snapshot of the pool keeps a consistent pair, and a late write into a retired row is lost, which is right for held (the process is gone) and harmless for last_activity. _POOL_LOCK guards membership of the pool only; the fields are read lock-free by the reserve arithmetic and the budget callback, which run on other workers' call threads and must never wait behind a spawn.

The row also gives replacement a single exit. Three paths swap a process out (the crash removal, the dead-worker branch on the next call, and the socket-unhealthy restart that keeps the SubprocessWorker object), and before the record each cleared a different subset of the per-process state. The dead branch left the old process's reserve charge on the new one; the restart path left the last VRAM report on the worker object; the last prompt and the pin-regression flag were never cleared, so a replacement under the same key inherited a stale figure and skipped its first regression line. All three now go through _retire_worker_state, which forgets everything per-process: the reserve high-water (the one place a shrink is allowed), the pin and overhead reports (the key must be absent from the allocator's input, not kept at zero), the activity clock, the manager flag, and the harvest fields on the worker object. Patchers are the caller's business, because the paths differ on whether they must stay alive (next section).

The subtlest rule: what replacement must preserve

When a worker is replaced (case 1), its SubprocessModelPatchers are deregistered but deliberately kept alive, because the restart can fire inside ComfyUI's free_memory iteration -- mutating the model list or letting the old patchers be garbage-collected mid-iteration would corrupt upstream's loop. Stale patchers are quarantined as already-offloaded and released on the next registration. ADR-0019 records both halves; any change to restart handling must keep them.

Where to go next