The build process¶

From a YAML config to a wheel a resolver can find: what runs, where it lands, and what to check when it breaks.

Where do the files live?¶

cuda-wheels/                          (the Comfy-Forge line's layout)
├── defaults/            WHAT the farm builds, farm-wide
│   ├── python_cuda_torch_os_policy.yml  the PCTO axes: owned policy
│   │                                    (platforms, python bounds,
│   │                                    supported_cudas, defaults) + the
│   │                                    GENERATED combinations rows
│   ├── arch_policy.yml                  the owned arch policy + exceptions,
│   │                                    read at BUILD time (CW-ADR-0012)
│   ├── scraped_torch_matrix.json        what upstream ships (the committed
│   │                                    PCWM snapshot -- derivation input)
│   ├── phantom_combos.json              cells upstream never shipped
│   │                                    (GENERATED by derive_defaults.py)
├── packages/            one FOLDER per package = the complete unit
│   ├── flash_attn/
│   │   ├── package.yml          source repo, tag, build knobs
│   │   ├── arch_override.yml    arch_list / arch_list_by_cuda (optional)
│   │   └── patches/             pre-build source patches (optional)
│   ├── natten/
│   │   ├── package.yml
│   │   ├── pcto_override.yml    build_matrix / min_pytorch / links_torch
│   │   ├── arch_override.yml
│   │   ├── patches/natten.py
│   │   └── README.md            this package's quirks (rendered in-folder)
│   └── ...                      one folder per package
│
├── scripts/             the machinery
│   ├── package_loader.py            ONE loader: folders + overrides ->
│   │                                the same flat config dict everywhere
│   ├── generate_matrix.py           configs  -> CI job matrix
│   ├── derive_defaults.py           scraped matrix + policy -> generated rows
│   ├── torch_watch.py               daily upstream-combo watcher (reports)
│   ├── fetch_torch_matrix.py        scrape upstream / render the matrix page
│   ├── generate_index.py            releases -> PEP 503 index (into _site/)
│   ├── generate_dashboard.py        releases -> dashboard page
│   ├── verify_wheel.py              pre-publication gate: every wheel is
│   │                                verified in the build container
│   │                                BEFORE upload (import test, arch
│   │                                assert, ABI/glibc audits)
│   └── audit.py                     one audit command, three lenses:
│                                    --gaps / --naming / --archs
│
├── .github/
│   ├── workflows/
│   │   ├── build.yml                the build entry point (workflow_dispatch)
│   │   ├── update-index.yml         build _site/ + deploy to gh-pages
│   │   ├── torch-matrix.yml         refresh the PCWM snapshot + dry-run
│   │   │                            the grid derivation (issue on rot)
│   │   ├── torch-watch.yml          daily: report new upstream combos
│   │   └── get-sources.yml          publish patched sources for inspection
│   └── actions/
│       ├── setup-cuda/              install + cache a CUDA toolkit
│       ├── setup-build-env/         python, torch, build deps
│       └── build-wheel/             checkout, patch, compile, repair, rename
│
└── README.md

The website is not in main. The PEP 503 index, dashboard and matrix page are built into _site/ at deploy time and exist only on the gh-pages branch; the shorter-index guard compares against a checkout of the live branch, and the matrix page renders from the committed scraped_torch_matrix.json without touching upstream.

Two rules keep this tidy, and both are load-bearing:

packages/ is declarative. A package is data, never a shell script (CW-ADR-0001).
A package's patches/ is the only place source is modified, as idempotent Python scripts co-located with the config. Upstream source is never forked to fix a build.

How does a package become build jobs?¶

The job list is a subtraction chain:

Upstream truth — fetch_torch_matrix.py snapshots every combo torch actually ships.
The shared grid — defaults/python_cuda_torch_os_policy.yml, derived from that snapshot.
Package overrides narrow it (combinations, platforms, min_pytorch).
Minus phantom combos — cells upstream never shipped (CW-ADR-0007).
Minus already built — we check whether the wheel is already in the release and skip it. This makes builds resumable and incremental: re-dispatching builds only what is missing; --overwrite skips the check.

What happens after a build?¶

Every finished wheel is uploaded straight to its package's rolling GitHub release (<pkg>-latest). That's the storage, and it updates the moment a job succeeds.

The pip index is a separate, deliberate step.

Dispatching Update Index (update-index.yml) rebuilds the whole site from the Releases API and deploys it to gh-pages. A build run never touches the live index on its own unless dispatched with update_index=true, so half-finished build waves can't publish a half-updated index.

What do the wheel names mean?¶

<pkg>-<version>+cu<CCC>torch<M.m>-cp<PY>-cp<PY>-<platform>.whl

flash_attn-2.8.3+cu124torch2.4-cp311-cp311-win_amd64.whl
gsplat-1.5.3+cu124torch2.4-cp310-cp310-manylinux_2_28_x86_64.whl

The local version tag +cu128torch2.9 encodes the CUDA/torch combo.
The wheel's internal METADATA version is patched to match the filename so pip/uv see a consistent version (CW-ADR-0004).
Linux wheels go through auditwheel repair to manylinux_2_28 (the glibc floor is policy: it tracks PyTorch's manylinux baseline, enforced by building in the manylinux_2_28 container), excluding libcuda/libtorch -- those must come from the host.
Builds pin exactly torch==<ver>+cu<short> from PyTorch's own index, so every wheel is tied to a torch family -- the same pin comfy-env replicates into its generated environments.

The glibc floor: 2.28¶

glibc is Linux's C runtime; a binary runs only on systems with a glibc at least as new as the one it was compiled against, and it cannot be upgraded without upgrading the OS. manylinux_2_28 in a wheel's platform tag declares that floor: glibc ≥ 2.28, i.e. AlmaLinux/RHEL 8, Debian 10, Ubuntu 18.10 or newer — effectively every distro still receiving updates.

The farm gets that floor by building inside the quay.io/pypa/manylinux_2_28 container (AlmaLinux 8), not by trusting the runner image: a build on stock ubuntu-22.04 would silently inherit a glibc-2.35 floor and lock out older systems. The number is policy, not accident — it tracks PyTorch's own manylinux baseline (torch ships manylinux_2_28 wheels today; if torch moves to 2_34, we move). A wheel and the torch it links must sit on the same side of that baseline anyway, so there is nothing to gain by diverging in either direction.

Sequential and sharded compiles¶

GitHub runners have a hard 6-hour job limit; some compiles (flash_attn, flashinfer, llama_cpp_python) go overboard. Three escape hatches (CW-ADR-0006):

Disk freeing -- preinstalled runner images deleted up front.
Sharding (sharding: N) -- translation units are split across N parallel jobs; a link job unions their compiler caches, links one ordinary wheel, and fails below a 100% cache hit rate. Not a cpp_extension-only mechanism: there are two partitioning schemes selected by shard_filter, and the two heaviest sharded packages in the farm are a cmake build (natten) and a pccm build (spconv). Picking the wrong filter is a silent slow build, and the shard-stage link is expected to fail. Sharding is the page; CW-ADR-0014 is the original decision and now covers only the seat half. (A third hatch -- a sequential-checkpoint chain -- existed briefly and was removed 2026-08-21: measurement showed no package needed it. The "multi-hour" compiles were artifacts of bad parallelism settings; at sane knobs the heaviest package, nunchaku, builds in 99 minutes. If a package ever truly outgrows 6h, sharding is the answer -- cmake-style packages included, via the CUDA_WHEELS_SHARD_INDEX/COUNT env filter natten's patch demonstrates.)

Config fields¶

Required (the loader hard-errors without them):

Field	Purpose
`name`	the package/dist name — keys the release tag, wheel prefix and index entry
`source_repo` / `source_tag`	where the source comes from. Always pinned to a commit or tag -- the loader refuses `main`/`master`/`HEAD`, because a floating ref means the wheels in one release need not come from the same source
`links_torch`	`true` = one wheel per (cuda × torch); `false` = never links libtorch, built once per (cuda, python, platform) and listed under every torch (CW-ADR-0011)

Optional — in package.yml:

Field	Purpose
`version`	pin the wheel version when upstream detection fails
`build_subdir`	build from a subdirectory, for extensions inside a larger repo
`patch_script`	Python run against the checked-out source before building
`pre_build_script`	shell run before the compile (e.g. export CMAKE_ARGS)
`clone_recursive`	clone submodules
`extra_deps`	extra pip build dependencies
`extra_cuda_components`	additional CUDA toolkit packages (cufft, nvtx, ...)
`nvcc_flags`	appended to the nvcc command line (trailing flags win)
`max_jobs`	cap parallel compile jobs — see nvcc builds
`sharding: N`	split one cell's compile across N parallel jobs + a link job -- read Sharding before setting it
`shard_filter`	`seat` (default, nvcc-wrapper hash partition) or `source` (the package's own patch deletes out-of-slice TUs). Wrong choice = silent slow build
`sharding_platforms`	restrict sharding to specific lanes
`requires_dist`	curated `Requires-Dist`: REPLACES the wheel's upstream dependency metadata wholesale (CW-ADR-0004). PEP 508 strings; `{LOCAL}` expands to the wheel's own local tag, `{VER:<folder>}` to that package's pinned `version` — together they make exact sibling pins (`cumesh==0.0.1+cu128torch2.8`) that only our index can satisfy. Declare it only for packages whose upstream list is wrong (build-tool leakage, sibling mis-pins); the gate's C2 check asserts the wheel carries exactly the expanded list. Never pin `torch==X.Y.Z` — consumer envs pin torch at major.minor deliberately

Optional — in override files (each requires an explaining README.md):

Field	File	Purpose
`arch_list` / `arch_list_by_cuda`	`arch_override.yml`	override the inherited GPU architectures (x86 lanes)
`arch_list_aarch64` / `arch_list_by_cuda_aarch64`	`arch_override.yml`	same, for the aarch64 lane — the x86 fields are deliberately not consulted there; declare these when a kernel gap is platform-independent (e.g. sageattention ships no sm_100 anywhere)
`min_pytorch`	`pcto_override.yml`	floor, for packages that do not support older torch
`build_matrix` (`combinations` / `platforms`)	`pcto_override.yml`	own cell grid or platform restriction

packages/README.md is the authoritative reference; each package folder's README.md collects per-package quirks.