Automated PR - 2026-07-07

2026-07-07 16:57:50 +00:00
parent 780984275f
commit 63fd9a4f86
157 changed files with 15976 additions and 5043 deletions
@@ -0,0 +1,22 @@
+# Conditioning Types
+
+Pipelines use different conditioning methods from [`ltx-core`](../../ltx-core/) for controlling generation. See the [ltx-core conditioning documentation](../../ltx-core/README.md#conditioning--control) for details.
+
+## Image Conditioning
+
+All pipelines support image conditioning, but with different methods:
+
+- **Replacing Latents** ([`image_conditionings_by_replacing_latent`](../src/ltx_pipelines/utils/helpers.py)):
+  - Replaces the latent at a specific frame with the encoded image
+  - Strong control over specific frames
+
+- **Guiding Latents** ([`image_conditionings_by_adding_guiding_latent`](../src/ltx_pipelines/utils/helpers.py)):
+  - Adds the image as a guiding signal rather than replacing
+  - Better for smooth interpolation between keyframes
+
+## Video Conditioning
+
+- **Video Conditioning** (ICLoraPipeline only):
+  - Conditions on entire reference videos
+  - Useful for video-to-video transformations
+  - Uses `VideoConditionByKeyframeIndex` from [`ltx-core`](../../ltx-core/)
@@ -0,0 +1,64 @@
+# Installation & Usage
+
+## Installation
+
+```bash
+# From the repository root
+uv sync --frozen
+
+# Or install as a package
+pip install -e packages/ltx-pipelines
+```
+
+## Requirements
+
+- **LTX-2 Model Checkpoint** - Local `.safetensors` file
+- **Gemma Text Encoder** - Local Gemma model directory
+- **Spatial Upscaler** - Required by two-stage pipelines, for the upsampling stage
+- **Distilled LoRA** - Required by two-stage non-distilled pipelines, used for the stage-2 refinement
+
+## Running Pipelines
+
+All pipelines can be run directly from the command line. Each pipeline module is executable:
+
+```bash
+# Run a pipeline (example: two-stage text-to-video)
+python -m ltx_pipelines.ti2vid_two_stages \
+    --checkpoint-path path/to/checkpoint.safetensors \
+    --distilled-lora path/to/distilled_lora.safetensors 0.8 \
+    --spatial-upsampler-path path/to/upsampler.safetensors \
+    --gemma-root path/to/gemma \
+    --prompt "A beautiful sunset over the ocean" \
+    --output-path output.mp4
+
+# View all available options for any pipeline
+python -m ltx_pipelines.ti2vid_two_stages --help
+```
+
+### Available pipeline modules
+
+- `ltx_pipelines.ti2vid_two_stages` - Two-stage text/image-to-video (recommended). ([docs](pipelines.md#1-ti2vidtwostagespipeline), [source](../src/ltx_pipelines/ti2vid_two_stages.py))
+- `ltx_pipelines.ti2vid_two_stages_hq` - Two-stage text/image-to-video (different sampler, better quality). ([docs](pipelines.md#2-ti2vidtwostageshqpipeline), [source](../src/ltx_pipelines/ti2vid_two_stages_hq.py))
+- `ltx_pipelines.ti2vid_one_stage` - Single-stage text/image-to-video. ([docs](pipelines.md#3-ti2vidonestagepipeline), [source](../src/ltx_pipelines/ti2vid_one_stage.py))
+- `ltx_pipelines.t2a_one_stage` - Single-stage text-to-audio (audio-only output). ([docs](pipelines.md#11-t2aonestagepipeline), [source](../src/ltx_pipelines/t2a_one_stage.py))
+- `ltx_pipelines.distilled` - Fast text/image-to-video pipeline using only the distilled model. ([docs](pipelines.md#4-distilledpipeline), [source](../src/ltx_pipelines/distilled.py))
+- `ltx_pipelines.ic_lora` - Video-to-video with IC-LoRA. ([docs](pipelines.md#5-iclorapipeline), [source](../src/ltx_pipelines/ic_lora.py))
+- `ltx_pipelines.keyframe_interpolation` - Keyframe interpolation. ([docs](pipelines.md#6-keyframeinterpolationpipeline), [source](../src/ltx_pipelines/keyframe_interpolation.py))
+- `ltx_pipelines.a2vid_two_stage` - Audio-to-video generation conditioned on an input audio. ([docs](pipelines.md#7-a2vidpipelinetwostage), [source](../src/ltx_pipelines/a2vid_two_stage.py))
+- `ltx_pipelines.retake` - Regenerate a time region of an existing video. ([docs](pipelines.md#8-retakepipeline), [source](../src/ltx_pipelines/retake.py))
+- `ltx_pipelines.hdr_ic_lora` - Video-to-video with HDR output (linear float via LogC3 inverse decode). ([docs](pipelines.md#9-hdriclorapipeline), [source](../src/ltx_pipelines/hdr_ic_lora.py))
+- `ltx_pipelines.lipdub` - Lip dubbing / re-voicing with IC-LoRA and audio reference conditioning. ([docs](pipelines.md#10-lipdubpipeline), [source](../src/ltx_pipelines/lipdub.py))
+
+Use `--help` with any pipeline module to see all available options and parameters.
+
+## Common CLI flags
+
+These flags are shared across the pipeline CLIs (they come from a common base parser); run a module with `--help` for its full set.
+
+- `--seed <int>` - random seed for reproducible generation (default 10).
+- `--offload {none,cpu,disk}` - offload transformer weights to reduce peak GPU memory. `cpu` holds them in system RAM; `disk` streams them from disk when RAM is also limited (slower). Default `none`.
+- `--quantization {fp8-cast,fp8-scaled-mm}` - run the transformer in FP8 to cut memory. `fp8-cast` downcasts a bf16 checkpoint on the fly (any FP8-capable GPU); `fp8-scaled-mm` expects an fp8 checkpoint and native FP8 support (best on Hopper+).
+- `--max-batch-size <int>` - max batch per transformer forward pass (default 1). Higher values reduce layer-streaming transfers at the cost of peak memory.
+- `--compile [key=value ...]` - enable `torch.compile`, optionally overriding the compilation config.
+- `--lora <path> [strength]` - apply a LoRA (repeatable; default strength 1.0).
+- `--enhance-prompt` - rewrite the prompt with the built-in enhancer before generation.
@@ -0,0 +1,78 @@
+# Multi-GPU Inference
+
+Run LTX-2 pipelines across several GPUs on a single machine.
+
+> ## ⚠️ Important
+>
+> **Multi-GPU (MGPU) is a latency tool, not a memory tool.** It is designed to reduce
+> the latency of a single generation on multi-GPU servers (H100, B200) by splitting
+> each denoising step and the VAE decode across GPUs.
+>
+> **MGPU is not a way to fit a bigger model.** The mutable **working copy** of the
+> transformer is a **full replica on every GPU** (each rank builds the whole model;
+> LoRAs are fused into it in place). MGPU therefore cannot make a checkpoint that
+> doesn't fit on one GPU suddenly fit — for that use FP8 quantization and weight
+> offloading (see [Optimization Tips](../optimization.md)).
+>
+> Each rank *also* holds a second, immutable copy of the **clean (pre-LoRA)
+> weights** — kept for LoRA hot-swap (reset + broadcast) — but that copy is
+> **sharded** across GPUs (`ShardedSD`, ~1/world_size per rank), not replicated.
+> Sequence parallelism additionally splits **activation** memory across ranks. See
+> the [weight tracker](pipeline-setup.md#transformerweighttracker--working-copy--sharded-clean-weights)
+> for the exact layout. The headline purpose is **latency**, not memory.
+>
+> **Single machine only.** One process per GPU, `MASTER_ADDR=localhost`, one rank
+> per GPU. No multi-node.
+
+## Requirements
+
+- **Linux** -- NCCL and CUDA-IPC peer buffers are Linux-only (no macOS/Windows).
+- **>=2 CUDA GPUs on a single node** with P2P access (NVLink/PCIe). No multi-node.
+- **PyTorch with CUDA.**
+- **`ltx-kernels` built** -- the SP all2all kernel is mandatory. Build with
+  `uv sync --group kernels` (needs a CUDA toolkit / nvcc and a C++ compiler, gcc or
+  clang). See the root README.
+
+## Capabilities
+
+| Technique | Purpose |
+| --------- | ------- |
+| [Sequence parallelism (SP)](sequence-parallel.md) | Split the token sequence across GPUs; faithful — numerically equivalent to single-GPU |
+| [Tiled data parallelism (TDP)](tiled-data-parallel.md) | One spatial (height x width) tile per GPU; for resolutions outside the training distribution. **Upscale only** |
+| [Distributed decoder](distributed-decoder.md) | Decode latent tiles in parallel, assemble on the driver |
+| [Distributed Gemma](gemma.md) | Shard Gemma across GPUs via Accelerate `device_map`, or replicate + split prompts |
+| [MGPU controller](controller.md) | Persistent worker fleet; dispatch a job, stream results |
+| [Pipeline setup](pipeline-setup.md) | Swap single-GPU builders for MGPU builders; share one weights registry |
+
+## Architecture overview
+
+The [`MGPUController`](controller.md) spawns one worker process per GPU and runs a
+user-defined **runner** (a subclass of `MGPURunner`) in SPMD lockstep. A runner's
+`setup()` builds a standard pipeline, then **swaps** each block's builder for an MGPU
+builder (SP / TDP / distributed decoder / distributed Gemma). All builders share one
+`StateDictRegistry` so the checkpoint loads from disk once per process.
+
+Two runners are provided, each with a CLI:
+
+- [`ltx_pipelines.ti2vid_two_stages_mgpu`](../../src/ltx_pipelines/ti2vid_two_stages_mgpu.py) — SP stage 1 + TDP stage 2 + Accelerate Gemma + distributed VAE.
+- [`ltx_pipelines.distilled_mgpu`](../../src/ltx_pipelines/distilled_mgpu.py) — SP (shared stage) + Accelerate Gemma + distributed VAE.
+
+```bash
+# Two-stage on all visible GPUs
+python -m ltx_pipelines.ti2vid_two_stages_mgpu \
+    --checkpoint-path path/to/checkpoint.safetensors \
+    --distilled-lora path/to/distilled_lora.safetensors 1.0 \
+    --spatial-upsampler-path path/to/upsampler.safetensors \
+    --gemma-root path/to/gemma \
+    --prompt "A beautiful sunset over the ocean" \
+    --output-path output.mp4
+```
+
+## Pages
+
+- **[Controller](controller.md)** — `MGPUController` / `MGPURunner` / `Stream`, lifecycle, one-job-at-a-time contract, threading, error handling.
+- **[Pipeline setup](pipeline-setup.md)** — swapping builders, the shared weights registry, the LoRA-hot-swap weight tracker.
+- **[Sequence parallelism](sequence-parallel.md)** — faithful token-dim split, the all2all kernels, `AttentionManager`, `SequenceParallelBuilder`.
+- **[Tiled data parallelism](tiled-data-parallel.md)** — out-of-distribution resolutions, position normalization, shared negative (reference) positions, `TiledDataParallelBuilder`.
+- **[Distributed decoder](distributed-decoder.md)** — inter-GPU vs intra-GPU tiling, `DistributedDecoderBuilder`.
+- **[Gemma](gemma.md)** — `AccelerateGemmaBuilder` (Accelerate `device_map` sharding) and `BatchParallelGemmaBuilder` (replicated; not for the distilled pipeline).
@@ -0,0 +1,115 @@
+# MGPU Controller
+
+**Source**: [`multigpu/controller.py`](../../src/ltx_pipelines/multigpu/controller.py), [`multigpu/runner.py`](../../src/ltx_pipelines/multigpu/runner.py)
+
+The controller is a persistent, one-job-at-a-time GPU fleet. It spawns one worker
+process per GPU, runs a user-defined **runner** in SPMD lockstep, and streams
+results back.
+
+## Public classes
+
+### `MGPUController`
+
+```python
+MGPUController(
+    runner_cls: type[MGPURunner],
+    *,
+    num_gpus: int | None = None,      # GPUs 0..num_gpus-1 (default: all visible)
+    devices: Sequence[int] | None = None,  # place on specific physical GPUs, e.g. [2, 3]
+    logs_specs: LogsSpecs | None = None,
+)
+```
+
+`num_gpus` and `devices` are mutually exclusive. `devices=[2, 3]` puts rank r on
+`cuda:devices[r]`, so two controllers can share one machine on disjoint GPU sets.
+
+Lifecycle:
+
+| Method | What it does |
+| ------ | ------------ |
+| `start(*, timeout=30min, **setup_kwargs)` | Spawn the fleet, run `setup(**setup_kwargs)` on every rank, block until all report ready. `timeout` bounds NCCL init + CUDA init + `setup()`; it must exceed the slowest model load. |
+| `stream(*, timeout=None, **kwargs) -> Stream` | Dispatch one job and return **immediately**. Iterate the returned `Stream` to collect. |
+| `shutdown(*, graceful_timeout=60.0)` | Tear down the fleet; also force-terminates it — safe to call from another thread to recover a job that cannot be drained. |
+| `is_alive` (property) | True while the fleet is up and unpoisoned. |
+
+### `MGPURunner`
+
+`MGPURunner` is the abstract base class implemented per pipeline. The controller
+ships the subclass to every worker by value (a runner defined in `__main__` or a test
+module is supported), builds one instance per worker, injects the NCCL groups, calls
+`setup()` once, then invokes the instance per job.
+
+```python
+class MyRunner(MGPURunner):
+    @torch.inference_mode()
+    def setup(self, *, checkpoint_path: str, ...) -> None:
+        # build the pipeline + swap in MGPU builders (see pipeline-setup.md)
+        ...
+
+    @torch.inference_mode()
+    def __call__(self, *, prompt: str, ...) -> Iterator[...]:
+        video, audio = self._pipeline(...)
+        yield output_path   # __call__ MUST be a generator (use `yield`, even once)
+```
+
+- `setup()` and `__call__()` run on **every** rank. `self.groups` gives the
+  per-component `NCCLGroups` (`gemma_group`, `transformer_group`, `vae_group`).
+- The framework does **not** apply inference mode — decorate `setup`/`__call__` explicitly.
+
+## Usage
+
+```python
+from ltx_pipelines.multigpu import MGPUController
+
+controller = MGPUController(MyRunner, num_gpus=8)
+controller.start(checkpoint_path="...", gemma_root="...")   # setup kwargs
+stream = controller.stream(prompt="a cat", seed=42)
+try:
+    for item in stream:      # one element per yield, as it arrives (NOT gathered across ranks)
+        show(item)
+finally:
+    stream.drain()           # free the controller even on early exit
+controller.shutdown()
+```
+
+### Passing tensors
+
+Tensors are transparent:
+
+- **Inputs.** Pass them as **top-level** kwargs (`stream(latent=t, steps=30)`) and
+  the relay (rank 0) broadcasts them to every rank over NCCL — `__call__` receives
+  them already on the local GPU. An input tensor nested inside a list/dict kwarg is
+  **not** broadcast; it falls back to the (slower) pickle path.
+- **Outputs.** Yield tensors back (including nested inside a dict) and they return
+  via the result queue by shared memory / CUDA IPC — no pickling, regardless of
+  nesting.
+
+Everything else must be picklable and small.
+
+## Contract and limitations
+
+- **Single machine only.** `MASTER_ADDR=localhost`, `RANK == LOCAL_RANK`, one rank per GPU.
+- **One job at a time.** No job queue, no pipelining. Consume the `Stream` to the
+  end before the next `stream()`. Abandoning it is **not** cleaned up: the next
+  `stream()` raises `ControllerBusyError` until `stream.drain()` or `shutdown()` is
+  called. The recommended pattern is `try: ... finally: stream.drain()`.
+- **SPMD lockstep.** Yields are forwarded individually (in result-queue order), not
+  gathered. Only per-rank terminals are collected to end the stream.
+- **Thread ownership (baton-lock).** Any thread may call `stream()`. Each job
+  belongs to its **dispatching** thread — only that
+  thread may iterate or `drain()` its `Stream` (enforced in `Stream.__next__`). A
+  single lock guards only the in-flight check-and-set: among concurrent `stream()`
+  callers one proceeds and the rest raise `ControllerBusyError`.
+
+## Error handling
+
+| Situation | Outcome |
+| --------- | ------- |
+| Runner raises an **unexpected** exception | **Fatal** — a desynced NCCL collective cannot be unwound. The controller is poisoned and a new one must be constructed. |
+| Runner raises `RunnerError` (or `ValueError`, auto-converted) **identically on every rank** | Recoverable. Iterating the `Stream` re-raises `SymmetricRunnerError`; the fleet survives — fix the input and retry. Raise it outside any collective (e.g. validating broadcast kwargs before the first one). |
+| Some ranks raise `RunnerError`, others finish clean | `AsymmetricRunnerError` — surfaced prominently (a latent hang risk), but does not terminate the fleet. |
+| Worker death / exceeded per-job `timeout` | Surfaced when the `Stream` is next iterated; the controller is poisoned. |
+
+The public API (`from ltx_pipelines.multigpu import ...`) exports `MGPUController`,
+`MGPURunner`, `Stream`, `RunnerError`, `SymmetricRunnerError`,
+`AsymmetricRunnerError`, `ControllerBusyError`, and `NCCLGroups`.
@@ -0,0 +1,72 @@
+# Distributed VAE Decoder
+
+**Source**: [`ltx_core/multigpu/vae/distributed_decoder.py`](../../../ltx-core/src/ltx_core/multigpu/vae/distributed_decoder.py), [`multigpu/vae_builders.py`](../../src/ltx_pipelines/multigpu/vae_builders.py)
+
+## What it is
+
+VAE decode is expensive and embarrassingly parallel over space/time tiles. The
+distributed decoder splits the latent into tiles, assigns them **round-robin** to
+ranks (the tile count may exceed the GPU count), and every rank decodes its tiles in
+parallel.
+Workers ship their decoded tiles to the **driver rank** over an `mp.Queue` (CUDA
+IPC — zero-copy handle sharing); the driver blends overlaps and yields the assembled
+frames as temporal batches spread across the GPUs.
+
+## Inter-GPU tiling vs intra-GPU tiling
+
+Two independent tilings, commonly conflated:
+
+| | Controls | Config | Set by |
+| --- | --- | --- | --- |
+| **Inter-GPU** (MGPU) | Which rank decodes which tile — **parallelism** | `vae_tiling: TileCountConfig` (at build time) | `DistributedDecoderBuilder` |
+| **Intra-GPU** (SGPU) | Chunking *within* a rank's tile to bound **VRAM** | `tiling_config: TilingConfig` (per call) | the pipeline's usual tiling kwarg |
+
+They compose — a rank can further chunk its assigned tile for VRAM — with one
+guard: **multi-GPU temporal tiling and single-GPU temporal tiling cannot both be
+on.** If `vae_tiling.frames.num_tiles > 1` and `tiling_config.temporal_config` is
+set, `decode_video` raises, because two causal temporal splits would conflict.
+
+## API
+
+### `DistributedDecoderBuilder`
+
+```python
+from ltx_pipelines.multigpu.vae_builders import DistributedDecoderBuilder
+from ltx_core.tiling import TileCountConfig, DimensionTilingConfig
+
+DistributedDecoderBuilder(
+    inner: BuilderProtocol,        # the block's single-GPU decoder builder
+    queue: Queue,                  # spawn-context mp.Queue, shared across ranks (CUDA IPC)
+    vae_group: dist.ProcessGroup,  # self.groups.vae_group
+    vae_tiling: TileCountConfig,   # inter-GPU split
+    driver_rank: int,              # rank that collects + assembles (usually 0)
+    registry: Registry,
+)
+```
+
+`build()` returns a `DistributedVideoDecoder`; its `decode_video(latent,
+tiling_config=None, ...)` returns an iterator of temporal batches on the driver, and
+an empty iterator on workers (they only `put` their tiles).
+
+## Usage
+
+```python
+# The queue is created once, in the CLI __main__, and passed to controller.start(...)
+# as a setup kwarg so every worker shares it:
+vae_queue = torch.multiprocessing.get_context("spawn").SimpleQueue()
+
+# inside runner.setup():
+vae_tiling = TileCountConfig(height=DimensionTilingConfig(num_tiles=8, overlap=4))
+pipeline.video_decoder._decoder_builder = DistributedDecoderBuilder(
+    inner=pipeline.video_decoder._decoder_builder,
+    queue=vae_queue,
+    vae_group=self.groups.vae_group,
+    vae_tiling=vae_tiling,
+    driver_rank=0,
+    registry=registry,
+)
+```
+
+The shipped runners tile the VAE across **height** (8 tiles, overlap 4). Only the
+driver ends up with the assembled video, which is why the runner's `__call__`
+encodes the file on `driver_rank` and yields `None` on the others.
@@ -0,0 +1,80 @@
+# Gemma Text Encoder (Multi-GPU)
+
+**Source**: [`multigpu/gemma_builders.py`](../../src/ltx_pipelines/multigpu/gemma_builders.py), [`multigpu/bp_gemma_builder.py`](../../src/ltx_pipelines/multigpu/bp_gemma_builder.py)
+
+Two ways to run the Gemma text encoder across the fleet. Both swap in for
+`pipeline.prompt_encoder._text_encoder_builder` and broadcast the resulting
+embeddings to every rank (so the transformer ranks all have them).
+
+## `AccelerateGemmaBuilder` (Accelerate `device_map` — the default)
+
+Loads Gemma **once, on the source rank**, with Accelerate `device_map="auto"`, which
+**shards Gemma's layers across the available GPUs**. Non-source ranks receive a
+lightweight `AccelerateGemmaWrapper` stub that receives the encoded embeddings over
+NCCL. The source rank fuses all prompts into one Gemma call, then broadcasts each
+output.
+
+The first `build()` loads via HuggingFace `from_pretrained` and caches the full
+state dict (including non-persistent buffers) in the registry; later builds recreate
+the model from cache and reinstall the dispatch hooks — no disk I/O.
+
+```python
+from ltx_pipelines.multigpu.gemma_builders import AccelerateGemmaBuilder
+
+AccelerateGemmaBuilder(
+    gemma_root_path: str,
+    gemma_group: dist.ProcessGroup | None,      # self.groups.gemma_group
+    broadcast_group: dist.ProcessGroup | None,  # self.groups.transformer_group
+    registry: Registry,
+    *,
+    src_rank: int,                              # rank that loads + encodes (usually 0)
+    dtype: torch.dtype = torch.bfloat16,
+)
+```
+
+Usage (in `runner.setup()`):
+
+```python
+pipeline.prompt_encoder._text_encoder_builder = AccelerateGemmaBuilder(
+    gemma_root_path=gemma_root,
+    gemma_group=self.groups.gemma_group,
+    broadcast_group=self.groups.transformer_group,
+    registry=registry,
+    src_rank=0,
+    dtype=pipeline.dtype,
+)
+```
+
+The shipped runners (`ti2vid_two_stages_mgpu`, `ti2vid_two_stages_hq_mgpu`, `distilled_mgpu`) use this builder.
+
+## `BatchParallelGemmaBuilder` (replicated — data-parallel over prompts)
+
+Every rank materialises a **full** `GemmaTextEncoder` on its own GPU via the standard
+`SingleGPUModelBuilder` path (no Accelerate, no `device_map`, no per-layer dispatch
+hooks). The wrapper (`BatchParallelGemmaWrapper`) then **partitions the prompt list
+across ranks** in `encode` and broadcasts each prompt's output, so the forwards run
+concurrently on different GPUs. Non-deterministic prompt enhancement
+(`enhance_t2v` / `enhance_i2v`) is routed through a single `src_rank`.
+
+```python
+from ltx_pipelines.multigpu.bp_gemma_builder import BatchParallelGemmaBuilder
+
+BatchParallelGemmaBuilder(
+    gemma_root_path: str,
+    broadcast_group: dist.ProcessGroup | None,
+    registry: Registry,
+    *,
+    src_rank: int,
+    dtype: torch.dtype = torch.bfloat16,
+)
+```
+
+### Not for the distilled pipeline
+
+Batch-parallel is beneficial only when there is **more than one prompt to encode** —
+the typical CFG case, positive + negative (B=2 on 2 ranks = one prompt per rank, both
+forwards concurrent). The **distilled** pipeline runs **without CFG**: its `__call__`
+accepts a single `prompt` and no `negative_prompt`, so there is only one prompt to
+encode and no work to partition; batch-parallel provides no speedup in that case. Use
+`AccelerateGemmaBuilder` for the distilled pipeline (as the shipped `distilled` runner
+does).
@@ -0,0 +1,111 @@
+# Setting Up an MGPU Pipeline
+
+**Source**: [`ti2vid_two_stages_mgpu.py`](../../src/ltx_pipelines/ti2vid_two_stages_mgpu.py), [`multigpu/weight_tracker.py`](../../src/ltx_pipelines/multigpu/weight_tracker.py)
+
+An MGPU pipeline **is** a single-GPU pipeline with its per-block builders swapped
+for MGPU builders. Build the standard pipeline, then replace each block's
+`_transformer_builder` / `_text_encoder_builder` / `_decoder_builder`.
+
+## The pattern
+
+This is performed inside a runner's `setup()` (which runs on every rank — see
+[Controller](controller.md)).
+
+```python
+from ltx_core.loader.registry import StateDictRegistry
+from ltx_pipelines.ti2vid_two_stages import TI2VidTwoStagesPipeline
+from ltx_pipelines.multigpu.sp_builder import SequenceParallelBuilder
+from ltx_pipelines.multigpu.tdp_builder import TiledDataParallelBuilder
+from ltx_pipelines.multigpu.gemma_builders import AccelerateGemmaBuilder
+from ltx_pipelines.multigpu.vae_builders import DistributedDecoderBuilder
+from ltx_pipelines.multigpu.weight_tracker import TransformerWeightTracker
+from ltx_core.multigpu.transformer.attention import AttentionManager
+
+# 1. ONE shared registry for every builder in this process.
+registry = StateDictRegistry()
+
+# 2. Build the normal pipeline, handing it the registry.
+pipeline = TI2VidTwoStagesPipeline(
+    checkpoint_path=..., distilled_lora=..., spatial_upsampler_path=...,
+    gemma_root=..., loras=[], registry=registry, quantization=...,
+)
+
+# 3. One weight tracker per transformer process group (shared by the stages).
+tracker = TransformerWeightTracker(group=self.groups.transformer_group)
+
+# 4. Swap each block's builder.
+pipeline.stage_1._transformer_builder = SequenceParallelBuilder(
+    inner=pipeline.stage_1._transformer_builder, attn_mgr=attn_mgr,
+    registry=registry, tracker=tracker,
+)
+pipeline.stage_2._transformer_builder = TiledDataParallelBuilder(
+    inner=pipeline.stage_2._transformer_builder, group=self.groups.transformer_group,
+    tiling=tdp_tiling, registry=registry, tracker=tracker,
+)
+pipeline.prompt_encoder._text_encoder_builder = AccelerateGemmaBuilder(...)
+pipeline.video_decoder._decoder_builder = DistributedDecoderBuilder(...)
+```
+
+Each MGPU builder **wraps** the block's existing single-GPU builder (`inner=...`),
+so it inherits the checkpoint path, quantization, compilation, and LoRA config —
+only the parallelism is added. See the per-technique pages for each builder's
+constructor.
+
+> **`with_builder` vs direct assignment.** `DiffusionStage.with_builder(builder)`
+> returns a *new* stage with the builder swapped (functional, never mutates). The
+> runners assign `stage._transformer_builder = ...` directly because they mutate the
+> pipeline once, in place, during `setup()`. Both reach the same builder slot.
+
+## The shared weights registry
+
+`StateDictRegistry` is an in-process cache of loaded state dicts, keyed by
+`(resolved paths, sd_ops name)`. Passing **one** registry to every builder means:
+
+- The transformer checkpoint is read from disk **once per process**, even though
+  stage 1 (SP) and stage 2 (TDP) are separate builders on the same file.
+- Gemma and the VAE cache their weights the same way (rebuild the module tree from
+  the cached tensors, skip disk I/O).
+
+The registry is **per process** — it is not shared across ranks. Each worker loads
+its own copy, so the full checkpoint is resident on every GPU (see the
+[memory disclaimer](README.md)).
+
+## `TransformerWeightTracker` — working copy + sharded clean weights
+
+```python
+TransformerWeightTracker(group: dist.ProcessGroup, bucket_mb=256, no_lora_swap=False)
+```
+
+The tracker is shared by the transformer stage builders that operate on the same
+checkpoint. It does **not** own weights — it references the tensors in the registry
+and receives a builder at `build()` time. Two copies of the weights exist per rank,
+and they are **not** the same shape of memory:
+
+- **Working copy** — the model the builder returns, backed by the registry's
+  tensors. This is a **full replica on every GPU**. LoRAs are fused into it
+  **in place**; `broadcast_sd` (a zero-copy `ShardedSD` view over these tensors)
+  broadcasts each owner rank's freshly fused shards — bucketed, `bucket_mb` at a
+  time — so all ranks converge on identical working weights.
+- **Clean weights** (`stored_sd`) — an immutable, cloned backup of the original
+  (pre-LoRA) weights, held so the working copy can be reset before a different LoRA
+  set is applied. This copy is **sharded** across ranks (deterministic
+  `md5(key) % world_size` ownership): each rank stores only its ~1/world_size slice,
+  not a full clone.
+
+So per-GPU transformer memory is one full working model **plus** a ~1/world_size
+clean-weights shard — the clean backup is distributed, the working copy is not.
+
+This allows a two-stage pipeline to apply the distilled LoRA to stage 2 and reset it
+for stage 1 without reloading the checkpoint. Pass `no_lora_swap=True` when the
+LoRA set is fixed (none, or one set for the whole run): the clean-weights clone is
+skipped (`stored_sd` becomes a zero-copy view) and any swap/reset raises — saves the
+~1/N shard, and guards against accidental swaps.
+
+## Full example
+
+The shipped runners are the reference: read
+[`ti2vid_two_stages_mgpu.py`](../../src/ltx_pipelines/ti2vid_two_stages_mgpu.py)
+(`setup()` lines ~54–132) and
+[`distilled_mgpu.py`](../../src/ltx_pipelines/distilled_mgpu.py). Each ends with a
+`__main__` block wiring the runner into an `MGPUController` behind the standard
+two-stage CLI parser.
@@ -0,0 +1,100 @@
+# Sequence Parallelism (SP)
+
+**Source**: [`ltx_core/multigpu/transformer/sequence_parallel.py`](../../../ltx-core/src/ltx_core/multigpu/transformer/sequence_parallel.py), [`multigpu/sp_builder.py`](../../src/ltx_pipelines/multigpu/sp_builder.py)
+
+## What it is
+
+SP splits the **token (sequence) dimension** of the video across GPUs. Each rank
+holds a slice of the tokens, runs the transformer forward on its slice, and the
+outputs are gathered back to all ranks. Self-attention still needs every token to
+see every other token, so the Q/K/V heads are exchanged across ranks with a custom
+**all2all** kernel: each rank ends up with all tokens for a subset of heads, does
+local attention, then the results are shuffled back.
+
+**SP is faithful — numerically equivalent to single-GPU inference.** Attention stays
+global (all2all preserves the full token interaction); only the floating-point
+reduction order changes. The all2all kernels move bytes only — the round-trip
+`gather(send(x)) == x` is byte-exact. SP is the appropriate choice whenever the
+single-GPU result is required at lower latency.
+
+This is the default for **stage 1** (`ti2vid_two_stages_mgpu`) and the **shared stage**
+(`distilled_mgpu`, where one SP wrapping covers both the half-res and full-res calls).
+
+## How the forward pass works
+
+Per denoising step, [`SequenceParallelModelWrapper`](../../../ltx-core/src/ltx_core/multigpu/transformer/sequence_parallel.py):
+
+1. Pads the video seq dim up to a multiple of `world_size` (padded keys are masked
+   out; padded rows sliced off after the gather) so every rank gets an equal shard.
+2. Tiles latent/timesteps/positions to this rank's slice.
+3. Runs the model — video self-attention (`attn1`) and video→audio cross-attention
+   are patched to route Q/K/V through the all2all kernel.
+4. `all_gather`s the output tokens back to full length on every rank and unpads.
+
+## The all2all kernels (`ltx-kernels`)
+
+The custom op is `ltx_kernels.All2All` (from the `ltx-kernels` package); the CUDA
+kernels use CUDA-IPC peer buffers to exchange tokens directly between ranks' GPUs.
+**`ltx-kernels` must be installed** — the SP builder imports it.
+
+## API
+
+### `AttentionManager`
+
+```python
+from ltx_core.multigpu.transformer.attention import AttentionManager
+
+attn_mgr = AttentionManager(
+    max_tokens: int,           # upper bound on total video tokens (raises above it)
+    num_heads: int,            # transformer.num_attention_heads
+    head_dim: int,             # transformer.attention_head_dim
+    tensor_dtype: torch.dtype,
+    group: dist.ProcessGroup,  # self.groups.transformer_group
+    copy_out_: bool = False,
+)
+```
+
+Owns the all2all buffers (sized `ceil(max_tokens / world_size)` tokens per rank) and, per step,
+`set_seqlen_all2all(...)` updates the per-rank token counts. `num_heads` must be
+divisible by `world_size`.
+
+### `SequenceParallelBuilder`
+
+```python
+from ltx_pipelines.multigpu.sp_builder import SequenceParallelBuilder
+
+SequenceParallelBuilder(
+    inner: ModelBuilderProtocol,   # the stage's single-GPU transformer builder
+    attn_mgr: AttentionManager,
+    registry: Registry,
+    tracker: TransformerWeightTracker,
+)
+```
+
+Wraps a `SingleGPUModelBuilder` (raises otherwise), injects the all2all attention
+module-ops, and `build()` returns a `SequenceParallelModelWrapper`.
+
+## Usage
+
+```python
+# inside runner.setup(), per stage:
+model_cfg = pipeline.stage_1._transformer_builder.model_config().get("transformer", {})
+attn_mgr = AttentionManager(
+    max_tokens=32768,
+    num_heads=model_cfg["num_attention_heads"],
+    head_dim=model_cfg["attention_head_dim"],
+    tensor_dtype=pipeline.dtype,
+    group=self.groups.transformer_group,
+)
+pipeline.stage_1._transformer_builder = SequenceParallelBuilder(
+    inner=pipeline.stage_1._transformer_builder,
+    attn_mgr=attn_mgr,
+    registry=registry,
+    tracker=tracker,
+)
+```
+
+`max_tokens` must cover the largest step. Reference: stage 1 at 512x768x121 is
+~6k video tokens; the distilled shared stage's full-res call (1024x1536x121) is
+~24k — both ship with `sp_max_tokens=32768`. Exceeding it raises with a clear
+"use a smaller resolution or fewer frames" message.
@@ -0,0 +1,130 @@
+# Tiled Data Parallelism (TDP)
+
+**Source**: [`ltx_core/multigpu/transformer/tiled_data_parallel.py`](../../../ltx-core/src/ltx_core/multigpu/transformer/tiled_data_parallel.py), [`multigpu/tdp_builder.py`](../../src/ltx_pipelines/multigpu/tdp_builder.py)
+
+## What it is
+
+TDP splits the patchified `(frames, height, width)` latent into **tiles** and gives
+each tile to a GPU. Every rank runs the full transformer on its own tile(s),
+overlapping regions are blended with trapezoidal masks, and a single `all_reduce`
+sums the blended tiles into the final result (masks sum to 1 globally). Tiles are
+assigned **round-robin**, so the tile count may exceed the GPU count (16 tiles on
+4 GPUs = 4 tiles/rank). Audio is processed untiled on every tile forward and averaged
+across tiles.
+
+Unlike [sequence parallelism](sequence-parallel.md), TDP is **not** bit-faithful to
+single-GPU: each tile is denoised with only local context and blended, so it is an
+approximation.
+
+> ## ⚠️ Do not use the TDP stage's audio output
+>
+> Audio is **not** tiled. It is denoised on **every** tile's forward pass — each with
+> a different, partial video context — and those results are **averaged across all
+> tiles**. That average is not a meaningful audio latent. Take the final audio from
+> the first (SP) stage and keep it frozen through the TDP upscale; treat the TDP
+> stage's audio only as the video-conditioning context it needs internally, never as
+> output.
+
+## When to use it
+
+TDP is an **upscaler**. It produces video at resolutions the model never saw during
+training by running each tile at a resolution the model handles well and blending the
+results. This is why the shipped two-stage runner uses TDP for **stage 2** (the
+high-resolution upscale).
+
+TDP can also be **faster** than running the whole frame on one GPU: self-attention is
+quadratic in the token count, so splitting `N` tokens into `T` tiles drops per-tile
+attention cost from `O(N^2)` to `O((N/T)^2)`.
+
+> **Do not run TDP as the first stage.** Starting from pure noise (a high first
+> sigma), each tile denoises independently and produces **unrelated content** — the
+> tiles never converge on a single coherent video. Generate the first stage with
+> [SP](sequence-parallel.md) (faithful, full-frame), then **upscale** that result
+> with TDP.
+>
+> Even as the upscale stage, tiles can **drift** apart, and the drift grows with the
+> **first sigma** of the TDP stage (more noise re-injected means more freedom per
+> tile). For consistency, either condition on the stage-1 result with
+> **negative-index image conditioning** (for i2v), or use a **smaller first sigma**.
+
+## Position normalization
+
+A tile's tokens must carry positions in the range the model was trained on — not the
+global positions of a tile in the corner of a large frame, which the model was never
+trained to handle. With `normalize_positions=True` (default), each tile's positions
+are shifted so the tile's **generated** tokens start at zero in every dimension:
+
+```python
+offset = gen_pos[..., 0].amin(dim=2, keepdim=True)...  # min start per (batch, dim)
+positions = positions - offset                          # shift generated + conditioning tokens
+```
+
+Interval widths are preserved (only the origin moves), so RoPE sees a valid,
+in-distribution position grid per tile.
+
+## Shared negative (reference) positions
+
+Conditioning tokens are appended after the generated tokens. A tile keeps a
+conditioning token when its `[start, end)` interval overlaps the tile in all three
+dimensions — **or** when it has a **negative time coordinate**. Negative-time tokens
+are **reference tokens** (e.g. reference frames / audio references): they are kept by
+**every** tile so all tiles share the same reference context. To avoid
+double-counting a token kept by several tiles, its blend weight is `1 / (number of
+tiles that kept it)`.
+
+## API
+
+### Tiling config (`ltx_core.tiling`)
+
+```python
+from ltx_core.tiling import TileCountConfig, DimensionTilingConfig
+
+TileCountConfig(
+    frames: DimensionTilingConfig = DimensionTilingConfig(num_tiles=1, overlap=0),
+    height: DimensionTilingConfig = DimensionTilingConfig(num_tiles=1, overlap=0),
+    width:  DimensionTilingConfig = DimensionTilingConfig(num_tiles=1, overlap=0),
+)
+DimensionTilingConfig(num_tiles: int, overlap: int = 0)   # counts, not sizes; overlap in latent grid units
+```
+
+`TileCountConfig` specifies tile **counts** per dimension (contrast the single-GPU
+VAE `TilingConfig`, which specifies tile **sizes**).
+
+### `TiledDataParallelBuilder`
+
+```python
+from ltx_pipelines.multigpu.tdp_builder import TiledDataParallelBuilder
+
+TiledDataParallelBuilder(
+    inner: ModelBuilderProtocol,   # the stage's single-GPU transformer builder
+    group: dist.ProcessGroup,      # self.groups.transformer_group
+    tiling: TileCountConfig,
+    registry: Registry,
+    tracker: TransformerWeightTracker,
+    normalize_positions: bool = True,
+)
+```
+
+Wraps a `SingleGPUModelBuilder`. Its `build()` requires a `video_tools` kwarg (the
+`VideoLatentTools` for the target shape) so the wrapper can compute tiles — the
+pipeline passes this through automatically.
+
+## Usage
+
+```python
+# inside runner.setup(), stage 2 -- balanced 2D spatial (height x width) grid over the group:
+from ltx_core.tiling import TileCountConfig, DimensionTilingConfig, balanced_tile_split
+
+h_tiles, w_tiles = balanced_tile_split(dist.get_world_size(self.groups.transformer_group))
+tdp_tiling = TileCountConfig(
+    height=DimensionTilingConfig(num_tiles=h_tiles, overlap=5),
+    width=DimensionTilingConfig(num_tiles=w_tiles, overlap=5),
+)
+pipeline.stage_2._transformer_builder = TiledDataParallelBuilder(
+    inner=pipeline.stage_2._transformer_builder,
+    group=self.groups.transformer_group,
+    tiling=tdp_tiling,
+    registry=registry,
+    tracker=tracker,
+)
+```
@@ -0,0 +1,52 @@
+# 🎛️ Multimodal Guidance
+
+LTX-2 pipelines use **multimodal guidance** to steer the diffusion process for both video and audio modalities. Each modality (video, audio) has its own guider with independent parameters, allowing fine-grained control over generation quality and adherence to prompts.
+
+## Guidance Parameters
+
+The `MultiModalGuiderParams` dataclass controls guidance behavior:
+
+| Parameter | Description |
+| --------- | ----------- |
+| `cfg_scale` | **Classifier-Free Guidance** scale. Higher values make the output adhere more strongly to the text prompt. Typical values: 2.0–5.0. Set to **1.0** to disable. |
+| `stg_scale` | **Spatio-Temporal Guidance** scale. Controls perturbation-based guidance for improved temporal coherence. Typical values: 0.5–1.5. Set to **0.0** to disable. |
+| `stg_blocks` | Which transformer blocks to perturb for STG (e.g., `[29]` for the last block). Set to **`[]`** to disable STG. |
+| `rescale_scale` | Rescales the guided prediction to match the variance of the conditional prediction. Helps prevent over-saturation. Typical values: 0.5–0.7. Set to **0.0** to disable. |
+| `modality_scale` | **Modality CFG** scale. Steers the model away from unsynced video and audio results, improving audio-visual coherence. Set to **1.0** to disable. |
+| `skip_step` | Skip guidance every N steps. Can speed up inference with minimal quality loss. Set to **0** to disable (never skip). |
+
+## How It Works
+
+The multimodal guider combines three guidance signals during each denoising step:
+
+1. **CFG (Text Guidance)**: Steers generation toward the text prompt by computing `(cond - uncond_text)`.
+2. **STG (Perturbation Guidance)**: Improves structural coherence by perturbing specific transformer blocks and steering away from the perturbed prediction.
+3. **Modality CFG**: For joint audio-video generation, steers the model away from unsynced video and audio results.
+
+## Example Configuration
+
+```python
+from ltx_core.components.guiders import MultiModalGuiderParams
+
+# Video guider: moderate CFG, STG enabled, modality isolation
+video_guider_params = MultiModalGuiderParams(
+    cfg_scale=3.0,
+    stg_scale=1.0,
+    rescale_scale=0.7,
+    modality_scale=3.0,
+    stg_blocks=[29],
+)
+
+# Audio guider: higher CFG for stronger prompt adherence
+audio_guider_params = MultiModalGuiderParams(
+    cfg_scale=7.0,
+    stg_scale=1.0,
+    rescale_scale=0.7,
+    modality_scale=3.0,
+    stg_blocks=[29],
+)
+```
+
+> **Tip:** Start with the default values from [`constants.py`](../src/ltx_pipelines/utils/constants.py) and adjust based on your use case. Higher `cfg_scale` = stronger prompt adherence but potentially less natural motion; higher `stg_scale` = better temporal coherence but slower inference (requires extra forward passes).
+>
+> **Tip:** When generating video with audio, set `modality_scale` > 1.0 (e.g., 3.0) to improve audio-visual sync. If generating video-only, set it to 1.0 to disable.
@@ -0,0 +1,149 @@
+# ⚡ Optimization Tips
+
+## Memory Optimization
+
+### FP8 Quantization (Lower Memory Footprint)
+
+For smaller GPU memory footprint, use the `--quantization` flag and set `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True`.
+
+Two quantization policies are available:
+
+| Policy | CLI Flag | Description |
+| ------ | -------- | ----------- |
+| **FP8 Cast** | `--quantization fp8-cast` | Downcasts transformer linear weights to FP8 during loading; upcasts on the fly during inference. No extra dependencies. |
+| **FP8 Scaled MM** | `--quantization fp8-scaled-mm` | Uses FP8 scaled matrix multiplication via PyTorch's `torch._scaled_mm`. Best performance on Hopper+ GPUs with native FP8 support. |
+
+**CLI:**
+
+```bash
+# FP8 Cast (works on any GPU with FP8 support)
+PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True python -m ltx_pipelines.ti2vid_two_stages \
+    --quantization fp8-cast --checkpoint-path=...
+
+# FP8 Scaled MM (no extra deps, best on Hopper+ GPUs)
+PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True python -m ltx_pipelines.ti2vid_two_stages \
+    --quantization fp8-scaled-mm --checkpoint-path=...
+```
+
+**Programmatically:**
+
+When authoring custom scripts, pass a `QuantizationPolicy` to pipeline classes:
+
+```python
+from ltx_core.quantization.fp8_cast import build_policy as build_fp8_cast_policy
+# Alternative:
+# from ltx_core.quantization.fp8_scaled_mm import build_policy as build_fp8_scaled_mm_policy
+
+pipeline = TI2VidTwoStagesPipeline(
+    checkpoint_path=ltx_model_path,
+    distilled_lora=distilled_lora,
+    spatial_upsampler_path=upsampler_path,
+    gemma_root=gemma_root_path,
+    loras=[],
+    quantization=build_fp8_cast_policy(ltx_model_path),
+)
+pipeline(...)
+```
+
+You still need to use `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True` when launching:
+
+```bash
+PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True python my_denoising_pipeline.py
+```
+
+### Memory Cleanup Between Stages
+
+By default, pipelines clean GPU memory (especially transformer weights) between stages. If you have enough memory, you can skip this cleanup to reduce running time:
+
+```python
+# In pipeline implementations, memory cleanup happens automatically
+# between stages. For custom pipelines, you can skip:
+# utils.cleanup_memory()  # Comment out if you have enough VRAM
+```
+
+## Compilation (`torch.compile`)
+
+Compiling the transformer blocks with `torch.compile` speeds up inference. It is **opt-in and off by default**. The blocks are compiled shape-polymorphically (the sequence dimension is marked dynamic), so one compiled artifact serves any token count without recompiling.
+
+**CLI** — the `--compile` flag maps directly to `CompilationConfig`:
+
+| Form | Result |
+| ---- | ------ |
+| *(flag absent)* | eager, no compilation |
+| `--compile` | compile with defaults |
+| `--compile KEY=VALUE ...` | compile, overriding individual fields |
+
+```bash
+# Defaults
+python -m ltx_pipelines.ti2vid_two_stages --compile --checkpoint-path=...
+
+# reduce-overhead captures CUDA graphs -- the main latency lever for the denoising loop.
+# Off by default because graph capture reserves static memory pools (extra VRAM), so it
+# trades memory for speed; enable it when you have headroom.
+python -m ltx_pipelines.ti2vid_two_stages --compile mode=reduce-overhead --checkpoint-path=...
+
+# Several overrides at once
+python -m ltx_pipelines.ti2vid_two_stages \
+    --compile mode=max-autotune fullgraph=true dynamic=true --checkpoint-path=...
+```
+
+| Field | Values | Default | Notes |
+| ----- | ------ | ------- | ----- |
+| `mode` | `none`, `reduce-overhead`, `max-autotune`, … | `none` | `reduce-overhead`/`max-autotune` enable CUDA graphs |
+| `backend` | `inductor`, `eager`, … | `inductor` | |
+| `fullgraph` | `true`/`false` | `false` | |
+| `dynamic` | `auto`/`true`/`false` | `auto` | the seq dim is marked dynamic regardless |
+| `inductor_config` | JSON object or path to a `.json` | `{}` | `torch._inductor.config` overrides |
+| `dynamo_config` | JSON object or path to a `.json` | `{"inline_inbuilt_nn_modules": true, "cache_size_limit": 256}` | `torch._dynamo.config` overrides |
+
+**Controlling inductor / dynamo configs.** `inductor_config` and `dynamo_config` take either an inline JSON object or a path to a `.json` file, applied via `torch._inductor.config.patch(...)` / `torch._dynamo.config.patch(...)` around the compiled forward. They **replace the defaults wholesale — they do not merge**, so when overriding `dynamo_config` re-include any defaults you want to keep:
+
+```bash
+python -m ltx_pipelines.ti2vid_two_stages \
+    --compile 'inductor_config={"max_autotune": true}' \
+              'dynamo_config={"inline_inbuilt_nn_modules": true, "cache_size_limit": 256, "recompile_limit": 32}' \
+    --checkpoint-path=...
+```
+
+**Programmatically**, pass a `CompilationConfig` to the pipeline:
+
+```python
+from ltx_core.model.transformer.compiling import CompilationConfig
+
+pipeline = TI2VidTwoStagesPipeline(
+    ...,
+    compilation_config=CompilationConfig(mode="reduce-overhead"),
+)
+```
+
+**Faster cache loads: `unsafe_skip_cache_dynamic_shape_guards` (unsafe, opt-in).** Inductor's FX-graph cache re-checks the dynamic-shape guards stored with each entry on every lookup. Setting this flag skips that re-check (every entry is treated as a guard hit), which speeds up warm and cross-process cache loads. It is **not enabled by default** because it is a correctness hazard: a kernel first compiled at a small sequence length keeps int32 address arithmetic, and reusing it at a larger sequence length (roughly **>58k tokens/rank**) overflows int32 and reads out of bounds — surfacing as a CUDA illegal memory access or silently corrupted output. Only enable it when your token counts stay within the range the cached kernels were compiled for:
+
+```bash
+python -m ltx_pipelines.ti2vid_two_stages \
+    --compile 'inductor_config={"unsafe_skip_cache_dynamic_shape_guards": true}' \
+    --checkpoint-path=...
+```
+
+## Denoising Loop Optimization
+
+**Gradient Estimation Denoising Loop:**
+
+Instead of the standard Euler denoising loop, you can use gradient estimation for fewer steps (~20-30 instead of 40):
+
+```python
+from ltx_pipelines.utils import gradient_estimating_euler_denoising_loop
+
+# Use gradient estimation denoising loop
+def denoising_loop(sigmas, video_state, audio_state, stepper):
+    return gradient_estimating_euler_denoising_loop(
+        sigmas=sigmas,
+        video_state=video_state,
+        audio_state=audio_state,
+        stepper=stepper,
+        transformer=transformer,
+        denoiser=denoiser,
+        ge_gamma=2.0,  # Gradient estimation coefficient
+    )
+```
+
+This allows you to use **20-30 steps instead of 40** while maintaining quality. The gradient estimation function is defined in [`samplers.py`](../src/ltx_pipelines/utils/samplers.py).
@@ -0,0 +1,48 @@
+# Pipeline Selection Guide
+
+## Quick Decision Tree
+
+```text
+Do you have an existing video to modify?
+├─ YES → Use RetakePipeline (regenerate a specific time region)
+│
+Do you have an audio file to drive generation?
+├─ YES → Use A2VidPipelineTwoStage (audio-to-video)
+│
+Do you need HDR output (linear float frames for EXR / tonemapping)?
+├─ YES → Use HDRICLoraPipeline (video-to-video with LogC3 inverse decode)
+│
+Do you need to condition on existing images/videos?
+├─ YES → Do you have reference videos for video-to-video?
+│  ├─ YES → Use ICLoraPipeline
+│  └─ NO → Do you have multiple keyframe images to interpolate?
+│     ├─ YES → Use KeyframeInterpolationPipeline
+│     └─ NO → Use TI2VidTwoStagesPipeline (image conditioning only)
+│
+└─ NO → Text-to-video only
+   ├─ Do you need best quality?
+   │  └─ YES → Use TI2VidTwoStagesPipeline (recommended for production)
+   │
+   └─ Do you need fastest inference?
+      └─ YES → Use DistilledPipeline (with 8 predefined sigmas)
+```
+
+> **Note:** [`TI2VidOneStagePipeline`](../src/ltx_pipelines/ti2vid_one_stage.py) is primarily for educational purposes. For best quality, use two-stage pipelines ([`TI2VidTwoStagesPipeline`](../src/ltx_pipelines/ti2vid_two_stages.py), [`TI2VidTwoStagesHQPipeline`](../src/ltx_pipelines/ti2vid_two_stages_hq.py), [`ICLoraPipeline`](../src/ltx_pipelines/ic_lora.py), [`KeyframeInterpolationPipeline`](../src/ltx_pipelines/keyframe_interpolation.py), [`A2VidPipelineTwoStage`](../src/ltx_pipelines/a2vid_two_stage.py), or [`DistilledPipeline`](../src/ltx_pipelines/distilled.py)). For editing existing videos, use [`RetakePipeline`](../src/ltx_pipelines/retake.py).
+
+## Features Comparison
+
+| Pipeline | Stages | [Multimodal Guidance](multimodal-guidance.md) | Upsampling | Conditioning | Best For |
+| -------- | ------ | --- | ---------- | ------------- | -------- |
+| [**TI2VidTwoStagesPipeline**](pipelines.md#1-ti2vidtwostagespipeline) | 2 | ✅ | ✅ | Image | **Production quality** (recommended) |
+| [**TI2VidTwoStagesHQPipeline**](pipelines.md#2-ti2vidtwostageshqpipeline) | 2 | ✅ | ✅ | Image | Same as above, res_2s sampler (higher quality) |
+| [**TI2VidOneStagePipeline**](pipelines.md#3-ti2vidonestagepipeline) | 1 | ✅ | ❌ | Image | Educational, prototyping |
+| [**DistilledPipeline**](pipelines.md#4-distilledpipeline) | 2 | ❌ | ✅ | Image | Fastest inference (8 sigmas) |
+| [**ICLoraPipeline**](pipelines.md#5-iclorapipeline) | 2 | ✅ | ✅ | Image + Video | Video-to-video transformations |
+| [**KeyframeInterpolationPipeline**](pipelines.md#6-keyframeinterpolationpipeline) | 2 | ✅ | ✅ | Keyframes | Animation, interpolation |
+| [**A2VidPipelineTwoStage**](pipelines.md#7-a2vidpipelinetwostage) | 2 | ✅ | ✅ | Audio + Image | Audio-driven video generation |
+| [**RetakePipeline**](pipelines.md#8-retakepipeline) | 1 | ✅ | ❌ | Source Video | Regenerating a time region of a video |
+| [**HDRICLoraPipeline**](pipelines.md#9-hdriclorapipeline) | 2 | ❌ | ✅ | Video | HDR video-to-video (linear float output for EXR) |
+| [**LipDubPipeline**](pipelines.md#10-lipdubpipeline) | 2 | ✅ | ✅ | Video + Audio | Lip dubbing with audio ref conditioning |
+| [**T2AOneStagePipeline**](pipelines.md#11-t2aonestagepipeline) | 1 | Audio only | ❌ | None (text) | Text-to-audio (audio-only output, no video) |
+
+See [Available Pipelines](pipelines.md) for a full description of each.
@@ -0,0 +1,151 @@
+# Available Pipelines
+
+Full reference for each pipeline. See the [Pipeline Selection Guide](pipeline-selection.md) to pick one.
+
+---
+
+## 1. TI2VidTwoStagesPipeline
+
+**Best for:** High-quality text/image-to-video generation with upsampling. **Recommended for production use.**
+
+**Source**: [`src/ltx_pipelines/ti2vid_two_stages.py`](../src/ltx_pipelines/ti2vid_two_stages.py)
+
+Two-stage generation: Stage 1 generates low-resolution video with [multimodal guidance](multimodal-guidance.md), Stage 2 upsamples to 2x resolution with distilled LoRA refinement. Supports image conditioning. Highest quality output, slower than one-stage but significantly better quality.
+
+**Use when:** Production-quality video generation, higher resolution needed, quality over speed, text-to-video with image conditioning.
+
+---
+
+## 2. TI2VidTwoStagesHQPipeline
+
+**Best for:** Same two-stage text/image-to-video as TI2VidTwoStagesPipeline but with a different sampler and step count.
+
+**Source**: [`src/ltx_pipelines/ti2vid_two_stages_hq.py`](../src/ltx_pipelines/ti2vid_two_stages_hq.py)
+
+Uses the **res_2s** second-order sampler instead of Euler. Same stage structure (stage 1 at target resolution with CFG, stage 2 upsampling with distilled LoRA) and image conditioning support. Typically allows fewer steps for comparable quality; trade-offs differ from the default Euler-based pipeline.
+
+**Use when:** You want the same two-stage workflow with fewer steps or prefer the res_2s sampling behavior.
+
+---
+
+## 3. TI2VidOneStagePipeline
+
+**Best for:** Educational purposes and quick prototyping.
+
+**Source**: [`src/ltx_pipelines/ti2vid_one_stage.py`](../src/ltx_pipelines/ti2vid_one_stage.py)
+
+> **⚠️ Important:** This pipeline is primarily for educational purposes. For production-quality results, use `TI2VidTwoStagesPipeline` or other two-stage pipelines.
+
+Single-stage generation (no upsampling) with [multimodal guidance](multimodal-guidance.md) and image conditioning support. Faster inference but lower resolution output (typically 512x768).
+
+**Use when:** Learning how the pipeline works, quick prototyping, testing, or when high resolution is not needed.
+
+---
+
+## 4. DistilledPipeline
+
+**Best for:** Fastest inference with good quality using a distilled model with predefined sigma schedule.
+
+**Source**: [`src/ltx_pipelines/distilled.py`](../src/ltx_pipelines/distilled.py)
+
+Two-stage generation with 8 predefined sigmas (8 steps in stage 1, 4 steps in stage 2). No guidance required. Fastest inference among all pipelines. Supports image conditioning. Requires spatial upsampler.
+
+**Use when:** Fastest inference is critical, batch processing many videos, or when you have a distilled model checkpoint.
+
+---
+
+## 5. ICLoraPipeline
+
+**Best for:** Video-to-video and image-to-video transformations using IC-LoRA.
+
+**Source**: [`src/ltx_pipelines/ic_lora.py`](../src/ltx_pipelines/ic_lora.py)
+
+Two-stage generation with IC-LoRA support. Can condition on reference videos (video-to-video) or images at specific frames. CFG guidance in stage 1, upsampling in stage 2. Requires IC-LoRA trained model.
+
+**Note:** ICLoraPipeline can only be used with a distilled model.
+
+**Use when:** Video-to-video transformations, image-to-video with strong control, or when you have reference videos to guide generation.
+
+---
+
+## 6. KeyframeInterpolationPipeline
+
+**Best for:** Generating videos by interpolating between keyframe images.
+
+**Source**: [`src/ltx_pipelines/keyframe_interpolation.py`](../src/ltx_pipelines/keyframe_interpolation.py)
+
+Two-stage generation with keyframe interpolation. Uses guiding latents (additive conditioning) instead of replacing latents for smoother transitions. [Multimodal guidance](multimodal-guidance.md) in stage 1, upsampling in stage 2.
+
+**Use when:** You have keyframe images and want to interpolate between them, creating smooth transitions, or animation/motion interpolation tasks.
+
+---
+
+## 7. A2VidPipelineTwoStage
+
+**Best for:** Generating video driven by an input audio.
+
+**Source**: [`src/ltx_pipelines/a2vid_two_stage.py`](../src/ltx_pipelines/a2vid_two_stage.py)
+
+Two-stage audio-to-video generation. Stage 1 generates video at half resolution with audio conditioning (video-only denoising with the audio frozen), then Stage 2 upsamples by 2x and refines the video while keeping the audio fixed, using a distilled LoRA. The input audio is encoded via the audio VAE and used as the initial audio latent, but the original audio waveform is passed through and returned in the output to preserve fidelity. Supports image conditioning and prompt enhancement.
+
+**Extra CLI arguments:** `--audio-path` (required), `--audio-start-time`, `--audio-max-duration`.
+
+**Use when:** You have an audio clip and want to generate a matching video, audio-reactive video generation, or music visualization.
+
+---
+
+## 8. RetakePipeline
+
+**Best for:** Regenerating a specific time region of an existing video while keeping the rest unchanged.
+
+**Source**: [`src/ltx_pipelines/retake.py`](../src/ltx_pipelines/retake.py)
+
+Single-stage generation that encodes the source video and audio into latents, applies a temporal region mask to mark `[start_time, end_time]` for regeneration, and denoises only the masked region from a text prompt. Content outside the time window is preserved. Supports independent control over video and audio regeneration (`regenerate_video`, `regenerate_audio` flags), and can use either the full model with CFG guidance or the distilled model with a fixed sigma schedule.
+
+**Extra CLI arguments:** `--video-path` (required), `--start-time` (required), `--end-time` (required).
+
+**Constraints:** Source video frame count must satisfy the 8k+1 format (e.g. 97, 193) and resolution must be multiples of 32.
+
+**Use when:** You want to re-do a specific section of a generated video (e.g. fix a bad segment), selectively regenerate audio or video in a time window, or iterate on part of a result without re-generating the entire clip.
+
+---
+
+## 9. HDRICLoraPipeline
+
+**Best for:** Video-to-video generation with HDR output for EXR export and offline tonemapping.
+
+**Source**: [`src/ltx_pipelines/hdr_ic_lora.py`](../src/ltx_pipelines/hdr_ic_lora.py)
+
+Two-stage video-to-video on the distilled model with an HDR IC-LoRA. Decoded latents pass through an HDR inverse transform (ARRI LogC3, auto-detected from LoRA metadata) to produce a **linear HDR float** tensor `[f, h, w, c]`. Video-only (audio skipped). Text embeddings are pre-computed externally and loaded from a `.safetensors` file. Tonemapping and EXR saving are the caller's responsibility. LoRA and embeddings: [`Lightricks/LTX-2.3-22b-IC-LoRA-HDR`](https://huggingface.co/Lightricks/LTX-2.3-22b-IC-LoRA-HDR).
+
+**Extra CLI arguments:** `--input` (mp4 or directory, required), `--output-dir` (required), `--hdr-lora` (required), `--text-embeddings` (pre-computed `.safetensors`, required), `--num-frames`, `--spatial-tile` (tiled VAE decode tile size; reduce on lower-VRAM GPUs), `--skip-mp4` (EXR only, no H.264 preview), `--exr-half` (float16 EXR), `--high-quality` (generates 2x frames internally for smoother output, ~2x slower), `--offload {none,cpu,disk}` (weight offloading; disables FP8 quantization when not `none`).
+
+**Use when:** You need linear HDR float output for EXR export, color grading, or custom tonemapping workflows.
+
+---
+
+## 10. LipDubPipeline
+
+**Best for:** Lip dubbing, rephrasing while keeping the same speaker identity and matching lip movements to new audio.
+
+**Source**: [`src/ltx_pipelines/lipdub.py`](../src/ltx_pipelines/lipdub.py)
+
+Uses IC-LoRA on a **distilled** checkpoint with a **single** lip-dub IC-LoRA applied in **both** stages. The reference clip provides video and audio reference tokens whose VAE latents are appended to the target audio sequence as frozen reference tokens. The frame count and frame rate are derived from the reference video (frame count is silently snapped to the nearest `8k+1`), so the CLI does not accept `--num-frames` or `--frame-rate`. Required: `--reference-video`. Optional: `--reference-strength`. LoRA: [`Lightricks/LTX-2.3-22b-IC-LoRA-LipDub`](https://huggingface.co/Lightricks/LTX-2.3-22b-IC-LoRA-LipDub).
+
+**Note:** Requires a distilled model checkpoint and one lip-dub IC-LoRA (`--lora` exactly once).
+
+**Use when:** Dubbing, rephrasing with matched lips and speaker identity.
+
+---
+
+## 11. T2AOneStagePipeline
+
+**Best for:** Text-to-audio — generating speech/audio only (no video) from a text prompt, e.g. driving an audio-style LoRA such as an accent LoRA.
+
+**Source**: [`src/ltx_pipelines/t2a_one_stage.py`](../src/ltx_pipelines/t2a_one_stage.py)
+
+Single-stage, **audio-only** generation: the video branch is absent (`video=None`), so only the audio modality is denoised and decoded through the audio VAE + vocoder, producing a wave file. Audio duration is derived from `--num-frames` / `--frame-rate` (the same `8k+1` frame convention as video). Audio guidance (CFG/STG) is optional — the `--audio-*` flags default to the model's values; the video→audio cross-modal guidance is disabled since there is no video modality.
+
+**Extra CLI arguments (all optional, with sensible defaults):** `--num-frames`, `--frame-rate`, `--negative-prompt`, `--audio-cfg-guidance-scale`, `--audio-stg-guidance-scale`, `--audio-stg-blocks`, `--audio-rescale-scale`, `--audio-skip-step`. No `--height/--width/--image` (audio has no spatial dimensions).
+
+**Use when:** You need speech/audio from text alone, or to evaluate an audio-only LoRA (accent, voice style) without generating video.