Automated PR - 2026-03-04
This commit is contained in:
sync-bot
@@ -12,7 +12,7 @@ LTX-2 Pipelines provides production-ready implementations that abstract away the
|
||||
|
||||
**Key Features:**
|
||||
|
||||
- 🎬 **Multiple Pipeline Types**: Text-to-video, image-to-video, video-to-video, and keyframe interpolation
|
||||
- 🎬 **Multiple Pipeline Types**: Text-to-video, image-to-video, video-to-video, audio-to-video, keyframe interpolation, and retake
|
||||
- ⚡ **Optimized Performance**: Support for FP8 transformers, gradient estimation, and memory optimization
|
||||
- 🎯 **Production Ready**: Two-stage pipelines for best quality output
|
||||
- 🔧 **LoRA Support**: Easy integration with trained LoRA adapters
|
||||
@@ -23,7 +23,7 @@ LTX-2 Pipelines provides production-ready implementations that abstract away the
|
||||
|
||||
## 🚀 Quick Start
|
||||
|
||||
`ltx-pipelines` provides ready-made inference pipelines for text-to-video, image-to-video, video-to-video, and keyframe interpolation. Built using building blocks from [`ltx-core`](../ltx-core/), these pipelines handle the complete inference flow including model loading, encoding, decoding, and file I/O.
|
||||
`ltx-pipelines` provides ready-made inference pipelines for text-to-video, image-to-video, video-to-video, audio-to-video, keyframe interpolation, and retake. Built using building blocks from [`ltx-core`](../ltx-core/), these pipelines handle the complete inference flow including model loading, encoding, decoding, and file I/O.
|
||||
|
||||
## 🔧 Installation
|
||||
|
||||
@@ -56,10 +56,13 @@ python -m ltx_pipelines.ti2vid_two_stages --help
|
||||
Available pipeline modules:
|
||||
|
||||
- `ltx_pipelines.ti2vid_two_stages` - Two-stage text/image-to-video (recommended).
|
||||
- `ltx_pipelines.ti2vid_two_stages_res2s` - Two-stage text/image-to-video (use 2 times less steps).
|
||||
- `ltx_pipelines.ti2vid_one_stage` - Single-stage text/image-to-video.
|
||||
- `ltx_pipelines.distilled` - Fast text/image-to-video pipeline using only the distilled model.
|
||||
- `ltx_pipelines.ic_lora` - Video-to-video with IC-LoRA.
|
||||
- `ltx_pipelines.keyframe_interpolation` - Keyframe interpolation.
|
||||
- `ltx_pipelines.a2vid_two_stage` - Audio-to-video generation conditioned on an input audio.
|
||||
- `ltx_pipelines.retake` - Regenerate a time region of an existing video.
|
||||
|
||||
Use `--help` with any pipeline module to see all available options and parameters.
|
||||
|
||||
@@ -70,6 +73,12 @@ Use `--help` with any pipeline module to see all available options and parameter
|
||||
### Quick Decision Tree
|
||||
|
||||
```text
|
||||
Do you have an existing video to modify?
|
||||
├─ YES → Use RetakePipeline (regenerate a specific time region)
|
||||
│
|
||||
Do you have an audio file to drive generation?
|
||||
├─ YES → Use A2VidPipelineTwoStage (audio-to-video)
|
||||
│
|
||||
Do you need to condition on existing images/videos?
|
||||
├─ YES → Do you have reference videos for video-to-video?
|
||||
│ ├─ YES → Use ICLoraPipeline
|
||||
@@ -85,17 +94,20 @@ Do you need to condition on existing images/videos?
|
||||
└─ YES → Use DistilledPipeline (with 8 predefined sigmas)
|
||||
```
|
||||
|
||||
> **Note:** [`TI2VidOneStagePipeline`](src/ltx_pipelines/ti2vid_one_stage.py) is primarily for educational purposes. For best quality, use two-stage pipelines ([`TI2VidTwoStagesPipeline`](src/ltx_pipelines/ti2vid_two_stages.py), [`ICLoraPipeline`](src/ltx_pipelines/ic_lora.py), [`KeyframeInterpolationPipeline`](src/ltx_pipelines/keyframe_interpolation.py), or [`DistilledPipeline`](src/ltx_pipelines/distilled.py)).
|
||||
> **Note:** [`TI2VidOneStagePipeline`](src/ltx_pipelines/ti2vid_one_stage.py) is primarily for educational purposes. For best quality, use two-stage pipelines ([`TI2VidTwoStagesPipeline`](src/ltx_pipelines/ti2vid_two_stages.py), [`TI2VidTwoStagesRes2sPipeline`](src/ltx_pipelines/ti2vid_two_stages_res2s.py), [`ICLoraPipeline`](src/ltx_pipelines/ic_lora.py), [`KeyframeInterpolationPipeline`](src/ltx_pipelines/keyframe_interpolation.py), [`A2VidPipelineTwoStage`](src/ltx_pipelines/a2vid_two_stage.py), or [`DistilledPipeline`](src/ltx_pipelines/distilled.py)). For editing existing videos, use [`RetakePipeline`](src/ltx_pipelines/retake.py).
|
||||
|
||||
### Features Comparison
|
||||
|
||||
| Pipeline | Stages | [Multimodal Guidance](#%EF%B8%8F-multimodal-guidance) | Upsampling | Conditioning | Best For |
|
||||
| -------- | ------ | --- | ---------- | ------------- | -------- |
|
||||
| **TI2VidTwoStagesPipeline** | 2 | ✅ | ✅ | Image | **Production quality** (recommended) |
|
||||
| **TI2VidTwoStagesRes2sPipeline** | 2 | ✅ | ✅ | Image | Same as above, res_2s sampler (fewer steps) |
|
||||
| **TI2VidOneStagePipeline** | 1 | ✅ | ❌ | Image | Educational, prototyping |
|
||||
| **DistilledPipeline** | 2 | ❌ | ✅ | Image | Fastest inference (8 sigmas) |
|
||||
| **ICLoraPipeline** | 2 | ✅ | ✅ | Image + Video | Video-to-video transformations |
|
||||
| **KeyframeInterpolationPipeline** | 2 | ✅ | ✅ | Keyframes | Animation, interpolation |
|
||||
| **A2VidPipelineTwoStage** | 2 | ✅ | ✅ | Audio + Image | Audio-driven video generation |
|
||||
| **RetakePipeline** | 1 | ✅ | ❌ | Source Video | Regenerating a time region of a video |
|
||||
|
||||
---
|
||||
|
||||
@@ -113,7 +125,19 @@ Two-stage generation: Stage 1 generates low-resolution video with [multimodal gu
|
||||
|
||||
---
|
||||
|
||||
### 2. TI2VidOneStagePipeline
|
||||
### 2. TI2VidTwoStagesRes2sPipeline
|
||||
|
||||
**Best for:** Same two-stage text/image-to-video as TI2VidTwoStagesPipeline but with a different sampler and step count.
|
||||
|
||||
**Source**: [`src/ltx_pipelines/ti2vid_two_stages_res2s.py`](src/ltx_pipelines/ti2vid_two_stages_res2s.py)
|
||||
|
||||
Uses the **res_2s** second-order sampler instead of Euler. Same stage structure (stage 1 at target resolution with CFG, stage 2 upsampling with distilled LoRA) and image conditioning support. Typically allows fewer steps for comparable quality; trade-offs differ from the default Euler-based pipeline.
|
||||
|
||||
**Use when:** You want the same two-stage workflow with fewer steps or prefer the res_2s sampling behavior.
|
||||
|
||||
---
|
||||
|
||||
### 3. TI2VidOneStagePipeline
|
||||
|
||||
**Best for:** Educational purposes and quick prototyping.
|
||||
|
||||
@@ -127,7 +151,7 @@ Single-stage generation (no upsampling) with [multimodal guidance](#%EF%B8%8F-mu
|
||||
|
||||
---
|
||||
|
||||
### 3. DistilledPipeline
|
||||
### 4. DistilledPipeline
|
||||
|
||||
**Best for:** Fastest inference with good quality using a distilled model with predefined sigma schedule.
|
||||
|
||||
@@ -139,7 +163,7 @@ Two-stage generation with 8 predefined sigmas (8 steps in stage 1, 4 steps in st
|
||||
|
||||
---
|
||||
|
||||
### 4. ICLoraPipeline
|
||||
### 5. ICLoraPipeline
|
||||
|
||||
**Best for:** Video-to-video and image-to-video transformations using IC-LoRA.
|
||||
|
||||
@@ -147,11 +171,13 @@ Two-stage generation with 8 predefined sigmas (8 steps in stage 1, 4 steps in st
|
||||
|
||||
Two-stage generation with IC-LoRA support. Can condition on reference videos (video-to-video) or images at specific frames. CFG guidance in stage 1, upsampling in stage 2. Requires IC-LoRA trained model.
|
||||
|
||||
**Note:** ICLoraPipeline can only be used with a distilled model.
|
||||
|
||||
**Use when:** Video-to-video transformations, image-to-video with strong control, or when you have reference videos to guide generation.
|
||||
|
||||
---
|
||||
|
||||
### 5. KeyframeInterpolationPipeline
|
||||
### 6. KeyframeInterpolationPipeline
|
||||
|
||||
**Best for:** Generating videos by interpolating between keyframe images.
|
||||
|
||||
@@ -163,6 +189,36 @@ Two-stage generation with keyframe interpolation. Uses guiding latents (additive
|
||||
|
||||
---
|
||||
|
||||
### 7. A2VidPipelineTwoStage
|
||||
|
||||
**Best for:** Generating video driven by an input audio.
|
||||
|
||||
**Source**: [`src/ltx_pipelines/a2vid_two_stage.py`](src/ltx_pipelines/a2vid_two_stage.py)
|
||||
|
||||
Two-stage audio-to-video generation. Stage 1 generates video at half resolution with audio conditioning (video-only denoising with the audio frozen), then Stage 2 upsamples by 2x and refines the video while keeping the audio fixed, using a distilled LoRA. The input audio is encoded via the audio VAE and used as the initial audio latent, but the original audio waveform is passed through and returned in the output to preserve fidelity. Supports image conditioning and prompt enhancement.
|
||||
|
||||
**Extra CLI arguments:** `--audio-path` (required), `--audio-start-time`, `--audio-max-duration`.
|
||||
|
||||
**Use when:** You have an audio clip and want to generate a matching video, audio-reactive video generation, or music visualization.
|
||||
|
||||
---
|
||||
|
||||
### 8. RetakePipeline
|
||||
|
||||
**Best for:** Regenerating a specific time region of an existing video while keeping the rest unchanged.
|
||||
|
||||
**Source**: [`src/ltx_pipelines/retake.py`](src/ltx_pipelines/retake.py)
|
||||
|
||||
Single-stage generation that encodes the source video and audio into latents, applies a temporal region mask to mark `[start_time, end_time]` for regeneration, and denoises only the masked region from a text prompt. Content outside the time window is preserved. Supports independent control over video and audio regeneration (`regenerate_video`, `regenerate_audio` flags), and can use either the full model with CFG guidance or the distilled model with a fixed sigma schedule.
|
||||
|
||||
**Extra CLI arguments:** `--video-path` (required), `--start-time` (required), `--end-time` (required).
|
||||
|
||||
**Constraints:** Source video frame count must satisfy the 8k+1 format (e.g. 97, 193) and resolution must be multiples of 32.
|
||||
|
||||
**Use when:** You want to re-do a specific section of a generated video (e.g. fix a bad segment), selectively regenerate audio or video in a time window, or iterate on part of a result without re-generating the entire clip.
|
||||
|
||||
---
|
||||
|
||||
## 🎨 Conditioning Types
|
||||
|
||||
Pipelines use different conditioning methods from [`ltx-core`](../ltx-core/) for controlling generation. See the [ltx-core conditioning documentation](../ltx-core/README.md#conditioning--control) for details.
|
||||
@@ -250,28 +306,43 @@ audio_guider_params = MultiModalGuiderParams(
|
||||
|
||||
### Memory Optimization
|
||||
|
||||
**FP8 Transformer (Lower Memory Footprint):**
|
||||
**FP8 Quantization (Lower Memory Footprint):**
|
||||
|
||||
For smaller GPU memory footprint, use the `enable-fp8` flag and use the `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True` environment variable.
|
||||
For smaller GPU memory footprint, use the `--quantization` flag and set `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True`.
|
||||
|
||||
Two quantization policies are available:
|
||||
|
||||
| Policy | CLI Flag | Description |
|
||||
| ------ | -------- | ----------- |
|
||||
| **FP8 Cast** | `--quantization fp8-cast` | Downcasts transformer linear weights to FP8 during loading; upcasts on the fly during inference. No extra dependencies. |
|
||||
| **FP8 Scaled MM** | `--quantization fp8-scaled-mm` | Uses FP8 scaled matrix multiplication via TensorRT-LLM (`tensorrt_llm` must be installed). Best performance on Hopper GPUs. |
|
||||
|
||||
**CLI:**
|
||||
|
||||
```bash
|
||||
PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True python -m ltx_pipelines.ti2vid_one_stage --enable-fp8 --checkpoint-path=...
|
||||
# FP8 Cast (works on any GPU with FP8 support)
|
||||
PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True python -m ltx_pipelines.ti2vid_two_stages \
|
||||
--quantization fp8-cast --checkpoint-path=...
|
||||
|
||||
# FP8 Scaled MM (requires tensorrt_llm, best on Hopper GPUs)
|
||||
PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True python -m ltx_pipelines.ti2vid_two_stages \
|
||||
--quantization fp8-scaled-mm --checkpoint-path=...
|
||||
```
|
||||
|
||||
**Programmatically:**
|
||||
|
||||
When authoring custom scripts, pass the `fp8transformer` flag to pipeline classes or construct your own by analogy:
|
||||
When authoring custom scripts, pass a `QuantizationPolicy` to pipeline classes:
|
||||
|
||||
```python
|
||||
from ltx_core.quantization import QuantizationPolicy
|
||||
|
||||
pipeline = TI2VidTwoStagesPipeline(
|
||||
checkpoint_path=ltx_model_path,
|
||||
distilled_lora=distilled_lora,
|
||||
spatial_upsampler_path=upsampler_path,
|
||||
gemma_root=gemma_root_path,
|
||||
loras=[],
|
||||
fp8transformer=True,
|
||||
quantization=QuantizationPolicy.fp8_cast(), # or QuantizationPolicy.fp8_scaled_mm()
|
||||
)
|
||||
pipeline(...)
|
||||
```
|
||||
@@ -299,7 +370,7 @@ By default, pipelines clean GPU memory (especially transformer weights) between
|
||||
Instead of the standard Euler denoising loop, you can use gradient estimation for fewer steps (~20-30 instead of 40):
|
||||
|
||||
```python
|
||||
from ltx_pipelines.utils.helpers import gradient_estimating_euler_denoising_loop
|
||||
from ltx_pipelines.utils import gradient_estimating_euler_denoising_loop
|
||||
|
||||
# Use gradient estimation denoising loop
|
||||
def denoising_loop(sigmas, video_state, audio_state, stepper):
|
||||
@@ -379,7 +450,7 @@ pipeline(
|
||||
num_inference_steps=40,
|
||||
video_guider_params=video_guider_params,
|
||||
audio_guider_params=audio_guider_params,
|
||||
images=[("input_image.jpg", 0, 1.0)], # Image at frame 0, strength 1.0
|
||||
images=[ImageConditioningInput("input_image.jpg", 0, 1.0, 33)], # Image at frame 0, strength 1.0, CRF 33
|
||||
)
|
||||
```
|
||||
|
||||
|
||||
Reference in New Issue
Block a user