Skip to content

Custom wire types

Nodes in ComfyUI exchange objects along edges.

A socket type is defined by just a string (IMAGE, VIDEO, LATENT) and vanilla ComfyUI never looks inside the object. The same instance is handed from one node to the next, in one process.

ComfyUI ships ~85 built-in types (comfy_api/latest/_io.py). By the shape of the Python object behind them:

  1. Primitives -- INT, FLOAT, STRING, BOOLEAN, COMBO.
  2. Tensors -- IMAGE and MASK are a bare torch.Tensor.
  3. Dicts of tensors -- LATENT ({"samples": tensor, ...}), AUDIO ({"waveform": tensor, "sample_rate": int}).
  4. Nested containers -- CONDITIONING is a list of [tensor, dict] pairs.
  5. Model handles -- MODEL, CLIP, VAE, CONTROL_NET, STYLE_MODEL: ComfyUI wrapper objects around weights, usually multi-GB and GPU-resident.
  6. Class-based payloads -- newer additions like MESH, VOXEL, SPLAT (comfy_api/latest/_util/geometry_types.py) and VIDEO. Ordinary Python classes whose fields happen to be tensors.

How comfy-env moves the standard types

The comfy-env inter process transport walks the object structurally: it dispatches on what the value is, never on the socket string:

Shape How it crosses
Primitives (1) inline in the JSON message
Tensors (2) the serialization ladder: CUDA IPC where available, else shared memory. No copy of the bulk
Dicts and lists (3, 4) walked recursively; every tensor inside takes the tensor path, so LATENT costs the same as the tensor it wraps
Model handles (5) the rule is that they do not cross — in either direction (ADR-0040). Worker→host: the model stays resident in the worker and the parent gets a SubprocessModelPatcher duck-type so ComfyUI's VRAM manager can still see and evict it (ADR-0035). Host→worker: nothing in the walker recognises a model today. A MODEL, CLIP, VAE or CONTROL_NET handed to an isolated node falls to the pickle rung like any unknown class: a ModelPatcher pickles successfully into a divergent copy in the worker (patches applied there never reach the host's), and a VAE/CONTROL_NET fails with a misleading "register a serializer" error. The named refusal ADR-0040 decides is planned, not shipped — see the ADR

That covers most of ComfyUI's vocabulary for free, because most of it is tensors, dicts, lists and primitives all the way down.

Where that stops working

Group 6 is the exception. The walker dispatches on type, does not recognise an arbitrary class, and falls to its last rung: the object is pickled into shared memory. It still crosses, and for something small that is fine. Otherwise it costs you:

  • Tensors inside get pickled too -- they never reach the tensor path.
  • Both sides need the class importable, or the value arrives as an opaque receipt instead of a real object.
  • Version skew becomes your problem -- trimesh 7.x pickling into a trimesh 8.x env.

comfy-env registers codecs for MESH, VOXEL and SPLAT, so those are already handled; other core types may follow. MODEL, CLIP and VAE deliberately never will -- weights stay in the worker that owns them.

Pickling that fails outright raises rather than dropping the value; the exact guarantee is practical rule 2.

Types from custom nodepacks

Sometimes an author of a nodepack might define their own type: TRIMESH, POINTCLOUD, SKELETON...

comfy-env has never seen the class and cannot guess which fields are bulk and which are metadata, so these land on the pickle rung above -- and pay all three of its costs.

Declaring your wire types (ADR-0015, mechanism in ADR-0014) is how you tell it: decompose the type into schema + arrays, so the bulk rides the shared-memory tensor path and no pickle is involved.

The recipe

The worked example throughout: ComfyUI-GeometryPack moves trimesh.Trimesh (the type behind its TRIMESH sockets) as shared-memory arrays.

1. Declare your sockets in the pack root comfy-env-root.toml:

[types]
TRIMESH    = "custom"     # serialize/deserialize code in ./serialization.py
SKELETON   = "builtin"    # dict of arrays -- automatic transport
INTRINSICS = "builtin"

A pack whose types are all dicts/arrays/tensors needs no [types] table at all. "builtin" entries are documentation with teeth (comfy-test can diff declared vs observed); "custom" entries require step 2.

[types] does not route anything

Routing is decided by the serializer registry, looked up by Python type at wire time. [types] is a declaration: comfy-env reads it once at startup and, for every socket marked "custom", refuses to start the pack unless serialization.py exists, imports, and registers something. A socket marked "builtin" changes no behaviour at all. Typos fail at parse time.

So declaring a type does not make it fast -- registering a serializer does. The declaration is what stops you shipping a pack whose serializer file quietly went missing.

2. serialization.py at your pack root (that exact name -- it is loaded by file path under a per-pack mangled module name, so every pack can use it without collisions). Top-level imports must be stdlib/numpy/comfy_env only; heavy libraries are imported inside the functions, so every process -- including a bare host with none of your deps -- can read the file:

try:    # parent process (comfy-env installed)
    from comfy_env.isolation.workers._ipc_shared import register_serializer
except ImportError:  # worker process (standalone copied module)
    from _ipc_shared import register_serializer

def _serialize_trimesh(mesh, recurse):
    payload = {
        "vertices": recurse(mesh.vertices),   # shared-memory arrays
        "faces": recurse(mesh.faces),
    }
    visual = getattr(mesh, "visual", None)
    if type(visual).__name__ == "TextureVisuals":
        uv = getattr(visual, "uv", None)
        if uv is not None and len(uv):
            payload["uv"] = recurse(uv)
    return payload

def _deserialize_trimesh(payload, recurse):
    import trimesh
    mesh = trimesh.Trimesh(
        vertices=recurse(payload["vertices"]),
        faces=recurse(payload["faces"]),
        process=False,                        # exact round-trip, no merging
    )
    if payload.get("uv") is not None:
        from trimesh.visual import TextureVisuals
        mesh.visual = TextureVisuals(uv=recurse(payload["uv"]))
    return mesh

# Register deserialize only where the library exists: a side without it
# holds the value as a materialized OpaquePayload receipt instead.
try:
    import trimesh  # noqa: F401
    _DESER = _deserialize_trimesh
except ImportError:
    _DESER = None

register_serializer(
    "Trimesh", _serialize_trimesh, _DESER,
    tag="trimesh.Trimesh",   # type identity -- see tag rules below
)

serialize(obj, recurse) returns a JSON-safe dict; anything passed through recurse re-enters the transport, so arrays and tensors take the shared-memory path. deserialize(payload, recurse) gets the payload raw and calls recurse on the parts it reconstructs. Registration matches by class name, then by MRO -- registering a base class covers its subclasses.

That's it. comfy-env loads the file parent-side at register_nodes() and worker-side at startup (via COMFY_ENV_SERIALIZER_FILES).

Tag rules (ADR-0015)

  • Shared library types tag by type identity: trimesh.Trimesh, not geompack.Trimesh. Two packs that both declare trimesh.Trimesh interoperate by construction -- each side rebuilds with its own registered functions; nobody executes another pack's code.
  • Pack-private types take a pack prefix (trellis2.ShapeSLAT), where collision is impossible by construction.
  • Payload ground rules: arrays, JSON primitives, and bytes -- never nested pickles. Raw arrays are what make version-skewed envs (py3.11/trimesh 7.x <-> py3.13/trimesh 8.x) interoperate: the library version never touches the wire.

What happens when a side can't reconstruct your type

Nothing breaks. That side holds the value as a materialized OpaquePayload -- every frame is copied into receiver-owned memory on receipt, so the receipt survives worker restarts and TTL expiry, and re-serializing emits fresh frames for the next hop. The bare ComfyUI host (which installs only comfy-env, per the host-env principle) forwards your objects between workers without ever understanding them. If some other pack installs your library into the host, the conditional registration above picks it up and the host reconstructs real objects instead -- native-node interop with zero configuration.

Practical rules (learned the hard way)

  • Only recurse long-lived arrays. Pass mesh.vertices directly -- do not wrap in np.asarray(...) or otherwise create temporaries. The transport's dedup map is keyed by id(); a temporary that gets garbage-collected mid-walk can hand its id to your next array, which then receives the wrong frame (observed in the wild: faces deserialized as vertices). Accessors that synthesize arrays per call (e.g. trimesh's vertex_colors) are unsafe to recurse for the same reason.
  • Unserializable values raise loudly. If recurse cannot encode an object, the transport raises a TypeError naming the type and the underlying cause (it previously leaked the raw object into the JSON message and crashed two layers away). Wrap optional-fidelity parts in try/except if you'd rather drop them than fail the call.
  • Never serialize objects with back-references to your bulk data. A trimesh visual holds a reference to its mesh -- recursing it whole would re-serialize the entire geometry. Decompose by field instead.
  • Degrade on fidelity, not on geometry: materials and metadata are try/except candidates (a missing dependency costs fidelity); the core arrays are not.

The full production module, with all of the above applied: serialization.py.