Automated PR - 2026-01-05
This commit is contained in:
sync-bot
@@ -0,0 +1,368 @@
|
||||
# Configuration Reference
|
||||
|
||||
The trainer uses structured Pydantic models for configuration, making it easy to customize training parameters.
|
||||
This guide covers all available configuration options and their usage.
|
||||
|
||||
## 📋 Overview
|
||||
|
||||
The main configuration class is [`LtxTrainerConfig`](../src/ltx_trainer/config.py), which includes the following
|
||||
sub-configurations:
|
||||
|
||||
- **ModelConfig**: Base model and training mode settings
|
||||
- **LoraConfig**: LoRA training parameters
|
||||
- **TrainingStrategyConfig**: Training strategy settings (text-to-video or video-to-video)
|
||||
- **OptimizationConfig**: Learning rate, batch sizes, and scheduler settings
|
||||
- **AccelerationConfig**: Mixed precision and quantization settings
|
||||
- **DataConfig**: Data loading parameters
|
||||
- **ValidationConfig**: Validation and inference settings
|
||||
- **CheckpointsConfig**: Checkpoint saving frequency and retention settings
|
||||
- **HubConfig**: Hugging Face Hub integration settings
|
||||
- **WandbConfig**: Weights & Biases logging settings
|
||||
- **FlowMatchingConfig**: Timestep sampling parameters
|
||||
|
||||
## 📄 Example Configuration Files
|
||||
|
||||
Check out our example configurations in the `configs` directory:
|
||||
|
||||
- 📄 [Audio-Video LoRA Training](../configs/ltx2_av_lora.yaml) - Joint audio-video to generation training
|
||||
- 📄 [IC-LoRA Training](../configs/ltx2_v2v_ic_lora.yaml) - Video-to-video transformation training
|
||||
|
||||
## ⚙️ Configuration Sections
|
||||
|
||||
### ModelConfig
|
||||
|
||||
Controls the base model and training mode settings.
|
||||
|
||||
```yaml
|
||||
model:
|
||||
model_path: "/path/to/ltx-2-model.safetensors" # Local path to model checkpoint
|
||||
text_encoder_path: "/path/to/gemma-model" # Path to Gemma text encoder directory
|
||||
training_mode: "lora" # "lora" or "full"
|
||||
load_checkpoint: null # Path to checkpoint to resume from
|
||||
```
|
||||
|
||||
**Key parameters:**
|
||||
|
||||
| Parameter | Description |
|
||||
|---------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------|
|
||||
| `model_path` | **Required.** Local path to the LTX-2 model checkpoint (`.safetensors` file). URLs are not supported. |
|
||||
| `text_encoder_path` | **Required.** Path to the Gemma text encoder model directory. Download from [HuggingFace](https://huggingface.co/google/gemma-3-12b-it-qat-q4_0-unquantized/). |
|
||||
| `training_mode` | Training approach - `"lora"` for LoRA training or `"full"` for full-rank fine-tuning. |
|
||||
| `load_checkpoint` | Optional path to resume training from a checkpoint file or directory. |
|
||||
|
||||
> [!NOTE]
|
||||
> LTX-2 requires both a model checkpoint and a Gemma text encoder. Both must be local paths.
|
||||
|
||||
### LoraConfig
|
||||
|
||||
LoRA-specific fine-tuning parameters (only used when `training_mode: "lora"`).
|
||||
|
||||
```yaml
|
||||
lora:
|
||||
rank: 32 # LoRA rank (higher = more parameters)
|
||||
alpha: 32 # LoRA alpha scaling factor
|
||||
dropout: 0.0 # Dropout probability (0.0-1.0)
|
||||
target_modules: # Modules to apply LoRA to
|
||||
- "to_k"
|
||||
- "to_q"
|
||||
- "to_v"
|
||||
- "to_out.0"
|
||||
```
|
||||
|
||||
**Key parameters:**
|
||||
|
||||
| Parameter | Description |
|
||||
|------------------|---------------------------------------------------------------------------------|
|
||||
| `rank` | LoRA rank - higher values mean more trainable parameters (typical range: 8-128) |
|
||||
| `alpha` | Alpha scaling factor - typically set equal to rank |
|
||||
| `dropout` | Dropout probability for regularization |
|
||||
| `target_modules` | List of transformer modules to apply LoRA adapters to (see below) |
|
||||
|
||||
#### Understanding Target Modules
|
||||
|
||||
The LTX-2 transformer has separate attention and feed-forward blocks for video and audio, as well as cross-attention
|
||||
modules that enable the two modalities to exchange information. Choosing the right `target_modules` is critical for
|
||||
achieving good results, especially when training with audio.
|
||||
|
||||
**Video-only modules:**
|
||||
|
||||
| Module Pattern | Description |
|
||||
|------------------------------------------------------------|---------------------------------|
|
||||
| `attn1.to_k`, `attn1.to_q`, `attn1.to_v`, `attn1.to_out.0` | Video self-attention |
|
||||
| `attn2.to_k`, `attn2.to_q`, `attn2.to_v`, `attn2.to_out.0` | Video cross-attention (to text) |
|
||||
| `ff.net.0.proj`, `ff.net.2` | Video feed-forward network |
|
||||
|
||||
**Audio-only modules:**
|
||||
|
||||
| Module Pattern | Description |
|
||||
|------------------------------------------------------------------------------------|---------------------------------|
|
||||
| `audio_attn1.to_k`, `audio_attn1.to_q`, `audio_attn1.to_v`, `audio_attn1.to_out.0` | Audio self-attention |
|
||||
| `audio_attn2.to_k`, `audio_attn2.to_q`, `audio_attn2.to_v`, `audio_attn2.to_out.0` | Audio cross-attention (to text) |
|
||||
| `audio_ff.net.0.proj`, `audio_ff.net.2` | Audio feed-forward network |
|
||||
|
||||
**Audio-video cross-attention modules:**
|
||||
|
||||
These modules enable bidirectional information flow between the audio and video modalities:
|
||||
|
||||
| Module Pattern | Description |
|
||||
|--------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------|
|
||||
| `audio_to_video_attn.to_k`, `audio_to_video_attn.to_q`, `audio_to_video_attn.to_v`, `audio_to_video_attn.to_out.0` | Video attends to audio (Q from video, K/V from audio) |
|
||||
| `video_to_audio_attn.to_k`, `video_to_audio_attn.to_q`, `video_to_audio_attn.to_v`, `video_to_audio_attn.to_out.0` | Audio attends to video (Q from audio, K/V from video) |
|
||||
|
||||
**Recommended configurations:**
|
||||
|
||||
For **video-only training**, target the video attention layers:
|
||||
|
||||
```yaml
|
||||
target_modules:
|
||||
- "attn1.to_k"
|
||||
- "attn1.to_q"
|
||||
- "attn1.to_v"
|
||||
- "attn1.to_out.0"
|
||||
- "attn2.to_k"
|
||||
- "attn2.to_q"
|
||||
- "attn2.to_v"
|
||||
- "attn2.to_out.0"
|
||||
```
|
||||
|
||||
For **audio-video training**, use patterns that match both branches:
|
||||
|
||||
```yaml
|
||||
target_modules:
|
||||
- "to_k"
|
||||
- "to_q"
|
||||
- "to_v"
|
||||
- "to_out.0"
|
||||
```
|
||||
|
||||
> [!NOTE]
|
||||
> Using shorter patterns like `"to_k"` will match all attention modules including `attn1.to_k`, `audio_attn1.to_k`,
|
||||
> `audio_to_video_attn.to_k`, and `video_to_audio_attn.to_k`, effectively training video, audio, and cross-modal
|
||||
> attention branches together.
|
||||
|
||||
> [!TIP]
|
||||
> You can also target the feed-forward (FFN) modules (`ff.net.0.proj`, `ff.net.2` for video,
|
||||
> `audio_ff.net.0.proj`, `audio_ff.net.2` for audio) to increase the LoRA's capacity and potentially
|
||||
> help it capture the target distribution better.
|
||||
|
||||
### TrainingStrategyConfig
|
||||
|
||||
Configures the training strategy. The trainer includes two built-in strategies described below.
|
||||
For custom use cases, see [Implementing Custom Training Strategies](custom-training-strategies.md).
|
||||
|
||||
#### Text-to-Video Strategy
|
||||
|
||||
```yaml
|
||||
training_strategy:
|
||||
name: "text_to_video"
|
||||
first_frame_conditioning_p: 0.1 # Probability of first-frame conditioning
|
||||
with_audio: false # Enable joint audio-video training
|
||||
audio_latents_dir: "audio_latents" # Directory for audio latents (when with_audio: true)
|
||||
```
|
||||
|
||||
#### Video-to-Video Strategy (IC-LoRA)
|
||||
|
||||
```yaml
|
||||
training_strategy:
|
||||
name: "video_to_video"
|
||||
first_frame_conditioning_p: 0.1
|
||||
reference_latents_dir: "reference_latents" # Directory for reference video latents
|
||||
```
|
||||
|
||||
**Key parameters:**
|
||||
|
||||
| Parameter | Description |
|
||||
|------------------------------|------------------------------------------------------------------|
|
||||
| `name` | Strategy type: `"text_to_video"` or `"video_to_video"` |
|
||||
| `first_frame_conditioning_p` | Probability of using first frame as conditioning (0.0-1.0) |
|
||||
| `with_audio` | (text_to_video only) Enable joint audio-video training |
|
||||
| `audio_latents_dir` | (text_to_video only) Directory name for audio latents |
|
||||
| `reference_latents_dir` | (video_to_video only) Directory name for reference video latents |
|
||||
|
||||
### OptimizationConfig
|
||||
|
||||
Training optimization parameters including learning rates, batch sizes, and schedulers.
|
||||
|
||||
```yaml
|
||||
optimization:
|
||||
learning_rate: 1e-4 # Learning rate
|
||||
steps: 2000 # Total training steps
|
||||
batch_size: 1 # Batch size per GPU
|
||||
gradient_accumulation_steps: 1 # Steps to accumulate gradients
|
||||
max_grad_norm: 1.0 # Gradient clipping threshold
|
||||
optimizer_type: "adamw" # "adamw" or "adamw8bit"
|
||||
scheduler_type: "linear" # Scheduler type
|
||||
scheduler_params: { } # Additional scheduler parameters
|
||||
enable_gradient_checkpointing: true # Memory optimization
|
||||
```
|
||||
|
||||
**Key parameters:**
|
||||
|
||||
| Parameter | Description |
|
||||
|---------------------------------|----------------------------------------------------------------------------------------------|
|
||||
| `learning_rate` | Learning rate for optimization (typical range: 1e-5 to 1e-3) |
|
||||
| `steps` | Total number of training steps |
|
||||
| `batch_size` | Batch size per GPU (reduce if running out of memory) |
|
||||
| `gradient_accumulation_steps` | Accumulate gradients over multiple steps |
|
||||
| `scheduler_type` | LR scheduler: `"constant"`, `"linear"`, `"cosine"`, `"cosine_with_restarts"`, `"polynomial"` |
|
||||
| `enable_gradient_checkpointing` | Trade training speed for GPU memory savings (recommended for large models) |
|
||||
|
||||
### AccelerationConfig
|
||||
|
||||
Hardware acceleration and compute optimization settings.
|
||||
|
||||
```yaml
|
||||
acceleration:
|
||||
mixed_precision_mode: "bf16" # "no", "fp16", or "bf16"
|
||||
quantization: null # Quantization options
|
||||
load_text_encoder_in_8bit: false # Load text encoder in 8-bit
|
||||
```
|
||||
|
||||
**Key parameters:**
|
||||
|
||||
| Parameter | Description |
|
||||
|-----------------------------|------------------------------------------------------------------------------------|
|
||||
| `mixed_precision_mode` | Precision mode - `"bf16"` recommended for modern GPUs |
|
||||
| `quantization` | Model quantization: `null`, `"int8-quanto"`, `"int4-quanto"`, `"fp8-quanto"`, etc. |
|
||||
| `load_text_encoder_in_8bit` | Load the Gemma text encoder in 8-bit to save GPU memory |
|
||||
|
||||
### DataConfig
|
||||
|
||||
Data loading and processing configuration.
|
||||
|
||||
```yaml
|
||||
data:
|
||||
preprocessed_data_root: "/path/to/preprocessed/data" # Path to precomputed dataset
|
||||
num_dataloader_workers: 2 # Background data loading workers
|
||||
```
|
||||
|
||||
**Key parameters:**
|
||||
|
||||
| Parameter | Description |
|
||||
|--------------------------|--------------------------------------------------------------------------------------------|
|
||||
| `preprocessed_data_root` | Path to your preprocessed dataset (contains `latents/`, `conditions/`, etc.) |
|
||||
| `num_dataloader_workers` | Number of parallel data loading processes (0 = synchronous loading, useful when debugging) |
|
||||
|
||||
### ValidationConfig
|
||||
|
||||
Validation and inference settings for monitoring training progress.
|
||||
|
||||
```yaml
|
||||
validation:
|
||||
prompts: # Validation prompts
|
||||
- "A cat playing with a ball"
|
||||
- "A dog running in a field"
|
||||
negative_prompt: "worst quality, inconsistent motion, blurry, jittery, distorted"
|
||||
images: null # Optional image paths for image-to-video
|
||||
reference_videos: null # Reference video paths (IC-LoRA only)
|
||||
video_dims: [ 576, 576, 89 ] # Video dimensions [width, height, frames]
|
||||
frame_rate: 25.0 # Frame rate for generated videos
|
||||
seed: 42 # Random seed for reproducibility
|
||||
inference_steps: 30 # Number of inference steps
|
||||
interval: 100 # Steps between validation runs
|
||||
videos_per_prompt: 1 # Videos generated per prompt
|
||||
guidance_scale: 4.0 # CFG guidance strength
|
||||
stg_scale: 1.0 # STG guidance strength (0.0 to disable)
|
||||
stg_blocks: [ 29 ] # Transformer blocks to perturb for STG
|
||||
stg_mode: "stg_av" # "stg_av" or "stg_v" (video only)
|
||||
generate_audio: true # Whether to generate audio
|
||||
skip_initial_validation: false # Skip validation at step 0
|
||||
include_reference_in_output: false # Include reference video side-by-side (IC-LoRA)
|
||||
```
|
||||
|
||||
**Key parameters:**
|
||||
|
||||
| Parameter | Description |
|
||||
|-------------------------------|--------------------------------------------------------------------------------------------------------------------------|
|
||||
| `prompts` | List of text prompts for validation video generation |
|
||||
| `images` | List of image paths for image-to-video validation (must match number of prompts) |
|
||||
| `reference_videos` | List of reference video paths for IC-LoRA validation (must match number of prompts) |
|
||||
| `video_dims` | Output dimensions `[width, height, frames]`. Width/height must be divisible by 32, frames must satisfy `frames % 8 == 1` |
|
||||
| `interval` | Steps between validation runs (set to `null` to disable) |
|
||||
| `guidance_scale` | CFG (Classifier-Free Guidance) scale. Recommended: 4.0 |
|
||||
| `stg_scale` | STG (Spatio-Temporal Guidance) scale. 0.0 disables STG. Recommended: 1.0 |
|
||||
| `stg_blocks` | Transformer blocks to perturb for STG. Recommended: `[29]` (single block) |
|
||||
| `stg_mode` | STG mode: `"stg_av"` perturbs both audio and video, `"stg_v"` perturbs video only |
|
||||
| `generate_audio` | Whether to generate audio in validation samples |
|
||||
| `include_reference_in_output` | For IC-LoRA: concatenate reference video side-by-side with output |
|
||||
|
||||
### CheckpointsConfig
|
||||
|
||||
Model checkpointing configuration.
|
||||
|
||||
```yaml
|
||||
checkpoints:
|
||||
interval: 250 # Steps between checkpoint saves (null = disabled)
|
||||
keep_last_n: 3 # Number of recent checkpoints to retain
|
||||
```
|
||||
|
||||
**Key parameters:**
|
||||
|
||||
| Parameter | Description |
|
||||
|---------------|------------------------------------------------------------------------|
|
||||
| `interval` | Steps between intermediate checkpoint saves (set to `null` to disable) |
|
||||
| `keep_last_n` | Number of most recent checkpoints to keep (-1 = keep all) |
|
||||
|
||||
### HubConfig
|
||||
|
||||
Hugging Face Hub integration for automatic model uploads.
|
||||
|
||||
```yaml
|
||||
hub:
|
||||
push_to_hub: false # Enable Hub uploading
|
||||
hub_model_id: "username/model-name" # Hub repository ID
|
||||
```
|
||||
|
||||
**Key parameters:**
|
||||
|
||||
| Parameter | Description |
|
||||
|----------------|------------------------------------------------------------------|
|
||||
| `push_to_hub` | Whether to automatically push trained models to Hugging Face Hub |
|
||||
| `hub_model_id` | Repository ID in format `"username/repository-name"` |
|
||||
|
||||
### WandbConfig
|
||||
|
||||
Weights & Biases logging configuration.
|
||||
|
||||
```yaml
|
||||
wandb:
|
||||
enabled: false # Enable W&B logging
|
||||
project: "ltx-2-trainer" # W&B project name
|
||||
entity: null # W&B username or team
|
||||
tags: [ ] # Tags for the run
|
||||
log_validation_videos: true # Log validation videos to W&B
|
||||
```
|
||||
|
||||
**Key parameters:**
|
||||
|
||||
| Parameter | Description |
|
||||
|-------------------------|--------------------------------------------------|
|
||||
| `enabled` | Whether to enable W&B logging |
|
||||
| `project` | W&B project name |
|
||||
| `entity` | W&B username or team (null uses default account) |
|
||||
| `log_validation_videos` | Whether to log validation videos to W&B |
|
||||
|
||||
### FlowMatchingConfig
|
||||
|
||||
Flow matching training configuration for timestep sampling.
|
||||
|
||||
```yaml
|
||||
flow_matching:
|
||||
timestep_sampling_mode: "shifted_logit_normal" # Timestep sampling strategy
|
||||
timestep_sampling_params: { } # Additional sampling parameters
|
||||
```
|
||||
|
||||
**Key parameters:**
|
||||
|
||||
| Parameter | Description |
|
||||
|----------------------------|------------------------------------------------------------|
|
||||
| `timestep_sampling_mode` | Sampling strategy: `"uniform"` or `"shifted_logit_normal"` |
|
||||
| `timestep_sampling_params` | Additional parameters for the sampling strategy |
|
||||
|
||||
## 🚀 Next Steps
|
||||
|
||||
Once you've configured your training parameters:
|
||||
|
||||
- Set up your dataset using [Dataset Preparation](dataset-preparation.md)
|
||||
- Choose your training approach in [Training Modes](training-modes.md)
|
||||
- Start training with the [Training Guide](training-guide.md)
|
||||
@@ -0,0 +1,509 @@
|
||||
# Implementing Custom Training Strategies
|
||||
|
||||
This guide explains how to implement your own training strategy for specialized use cases like audio-only training,
|
||||
video inpainting, or other custom training recipes.
|
||||
|
||||
## 📋 Overview
|
||||
|
||||
The trainer uses the **Strategy Pattern** to separate training logic from the core training loop. Each strategy defines:
|
||||
|
||||
1. **What data is needed** - Which preprocessed data directories to load
|
||||
2. **How to prepare inputs** - Transform batch data into model inputs
|
||||
3. **How to compute loss** - Calculate the training objective
|
||||
|
||||
This architecture lets you implement new training modes without modifying the core trainer code.
|
||||
|
||||
### When You Need a Custom Strategy
|
||||
|
||||
Consider implementing a custom strategy when you need:
|
||||
|
||||
- **Different input modalities** (e.g., audio-only, audio-to-video conditioning)
|
||||
- **Additional conditioning signals** (e.g., masks for inpainting, depth maps)
|
||||
- **Custom loss computation** (e.g., weighted losses, auxiliary losses)
|
||||
- **Different noise application patterns** (e.g., partial masking)
|
||||
|
||||
## 🏗️ Architecture Overview
|
||||
|
||||
### How Strategies Fit Into the Trainer
|
||||
|
||||
The trainer delegates all training-mode-specific logic to the strategy:
|
||||
|
||||
1. **Initialization** — The trainer calls `get_data_sources()` to determine which preprocessed data directories to load
|
||||
2. **Each training step:**
|
||||
- Calls `prepare_training_inputs()` to transform the raw batch into model-ready inputs
|
||||
- Runs the transformer forward pass
|
||||
- Calls `compute_loss()` to compute the training objective
|
||||
|
||||
The trainer handles everything else: optimization, checkpointing, validation, and distributed training.
|
||||
|
||||
### Key Components
|
||||
|
||||
| Component | Purpose |
|
||||
|-----------------------------------------------------------------------------------------|--------------------------------------------------------------|
|
||||
| [`TrainingStrategyConfigBase`](../src/ltx_trainer/training_strategies/base_strategy.py) | Base class for strategy configuration (Pydantic model) |
|
||||
| [`TrainingStrategy`](../src/ltx_trainer/training_strategies/base_strategy.py) | Abstract base class defining the strategy interface |
|
||||
| [`ModelInputs`](../src/ltx_trainer/training_strategies/base_strategy.py) | Dataclass containing prepared inputs for the transformer |
|
||||
| [`Modality`](../../ltx-core/src/ltx_core/model/transformer/modality.py) | ltx-core dataclass representing video or audio modality data |
|
||||
|
||||
## 📝 Step-by-Step Implementation
|
||||
|
||||
### Step 1: Plan Your Strategy
|
||||
|
||||
Before writing code, answer these questions:
|
||||
|
||||
1. **What additional data does your strategy need?**
|
||||
- Example: Inpainting needs mask latents alongside video latents
|
||||
- Example: Audio-to-video needs reference audio embeddings
|
||||
|
||||
2. **What does conditioning look like?**
|
||||
- Which tokens should be noised vs. kept clean?
|
||||
- How should conditioning tokens be structured (e.g., first frame, reference video, mask)?
|
||||
|
||||
3. **How should loss be computed?**
|
||||
- Which tokens contribute to the loss?
|
||||
- Are there multiple loss terms to combine?
|
||||
|
||||
### Step 2: Extend Data Preprocessing (If Needed)
|
||||
|
||||
If your strategy requires additional preprocessed data beyond video latents, audio latents, and text embeddings, you'll
|
||||
need to extend the preprocessing pipeline.
|
||||
|
||||
#### Option A: Modify `process_dataset.py`
|
||||
|
||||
For integrated preprocessing, add new arguments and processing steps to the main script. For example, to add mask
|
||||
preprocessing:
|
||||
|
||||
```python
|
||||
# In process_dataset.py, add a new argument
|
||||
@app.command()
|
||||
def main(
|
||||
# ... existing arguments ...
|
||||
mask_column: str | None = typer.Option(
|
||||
default=None,
|
||||
help="Column name containing mask video paths (for inpainting)",
|
||||
),
|
||||
) -> None:
|
||||
# ... existing processing ...
|
||||
|
||||
# Process masks if provided
|
||||
if mask_column:
|
||||
logger.info("Processing mask videos for inpainting training...")
|
||||
mask_latents_dir = output_base / "mask_latents"
|
||||
|
||||
compute_latents(
|
||||
dataset_file=dataset_path,
|
||||
video_column=mask_column,
|
||||
resolution_buckets=parsed_resolution_buckets,
|
||||
output_dir=str(mask_latents_dir),
|
||||
model_path=model_path,
|
||||
# ... other args ...
|
||||
)
|
||||
```
|
||||
|
||||
#### Option B: Create a Standalone Script
|
||||
|
||||
For complex preprocessing that doesn't fit naturally into the existing pipeline, create a dedicated script
|
||||
(e.g., `scripts/process_masks.py`). Use [`scripts/compute_reference.py`](../scripts/compute_reference.py) as a
|
||||
template - it shows how to process paired data and update the dataset JSON.
|
||||
|
||||
#### Expected Output Structure
|
||||
|
||||
Your preprocessing should create a directory structure that the strategy can reference:
|
||||
|
||||
```
|
||||
preprocessed_data_root/
|
||||
├── latents/ # Video latents (standard)
|
||||
├── conditions/ # Text embeddings (standard)
|
||||
├── audio_latents/ # Audio latents (if with_audio)
|
||||
├── mask_latents/ # Your custom data directory
|
||||
└── reference_latents/ # Reference videos (for IC-LoRA)
|
||||
```
|
||||
|
||||
### Step 3: Create the Strategy Configuration
|
||||
|
||||
Create a new file for your strategy (e.g., `src/ltx_trainer/training_strategies/inpainting.py`):
|
||||
|
||||
```python
|
||||
"""Inpainting training strategy.
|
||||
|
||||
This strategy implements video inpainting training where:
|
||||
- Mask latents indicate which regions to inpaint
|
||||
- Loss is computed only on masked (inpainted) regions
|
||||
"""
|
||||
|
||||
from typing import Any, Literal
|
||||
|
||||
import torch
|
||||
from pydantic import Field
|
||||
from torch import Tensor
|
||||
|
||||
from ltx_core.model.transformer.modality import Modality
|
||||
from ltx_trainer.timestep_samplers import TimestepSampler
|
||||
from ltx_trainer.training_strategies.base_strategy import (
|
||||
ModelInputs,
|
||||
TrainingStrategy,
|
||||
TrainingStrategyConfigBase,
|
||||
)
|
||||
|
||||
|
||||
class InpaintingConfig(TrainingStrategyConfigBase):
|
||||
"""Configuration for inpainting training strategy."""
|
||||
|
||||
# The 'name' field acts as a discriminator for the config union
|
||||
name: Literal["inpainting"] = "inpainting"
|
||||
|
||||
mask_latents_dir: str = Field(
|
||||
default="mask_latents",
|
||||
description="Directory name for mask latents",
|
||||
)
|
||||
|
||||
# Add any strategy-specific parameters
|
||||
mask_threshold: float = Field(
|
||||
default=0.5,
|
||||
description="Threshold for binary mask conversion",
|
||||
ge=0.0,
|
||||
le=1.0,
|
||||
)
|
||||
```
|
||||
|
||||
**Key points:**
|
||||
|
||||
- Inherit from `TrainingStrategyConfigBase`
|
||||
- Use `Literal["your_strategy_name"]` for the `name` field - this enables automatic strategy selection
|
||||
- Use Pydantic `Field` for validation and documentation
|
||||
|
||||
### Step 4: Implement the Strategy Class
|
||||
|
||||
```python
|
||||
class InpaintingStrategy(TrainingStrategy):
|
||||
"""Inpainting training strategy.
|
||||
|
||||
Trains the model to fill in masked regions of videos while
|
||||
keeping unmasked regions as conditioning.
|
||||
"""
|
||||
|
||||
config: InpaintingConfig
|
||||
|
||||
def __init__(self, config: InpaintingConfig):
|
||||
super().__init__(config)
|
||||
|
||||
@property
|
||||
def requires_audio(self) -> bool:
|
||||
"""Whether this strategy requires audio components."""
|
||||
return False # Set to True if your strategy needs audio
|
||||
|
||||
def get_data_sources(self) -> dict[str, str]:
|
||||
"""Define which data directories to load.
|
||||
|
||||
Returns a mapping of directory names to batch keys.
|
||||
The trainer will load .pt files from each directory and
|
||||
make them available in the batch under the specified key.
|
||||
"""
|
||||
return {
|
||||
"latents": "latents", # -> batch["latents"]
|
||||
"conditions": "conditions", # -> batch["conditions"]
|
||||
self.config.mask_latents_dir: "masks", # -> batch["masks"]
|
||||
}
|
||||
|
||||
def prepare_training_inputs(
|
||||
self,
|
||||
batch: dict[str, Any],
|
||||
timestep_sampler: TimestepSampler,
|
||||
) -> ModelInputs:
|
||||
"""Transform batch data into model inputs.
|
||||
|
||||
This is where the core training logic lives:
|
||||
1. Extract and patchify latents
|
||||
2. Sample noise and apply it appropriately
|
||||
3. Create conditioning masks
|
||||
4. Build Modality objects for the transformer
|
||||
"""
|
||||
# Get video latents [B, C, F, H, W]
|
||||
latents_data = batch["latents"]
|
||||
video_latents = latents_data["latents"]
|
||||
|
||||
# Get dimensions
|
||||
num_frames = latents_data["num_frames"][0].item()
|
||||
height = latents_data["height"][0].item()
|
||||
width = latents_data["width"][0].item()
|
||||
|
||||
# Patchify: [B, C, F, H, W] -> [B, seq_len, C]
|
||||
video_latents = self._video_patchifier.patchify(video_latents)
|
||||
|
||||
batch_size, seq_len, _ = video_latents.shape
|
||||
device = video_latents.device
|
||||
dtype = video_latents.dtype
|
||||
|
||||
# Get mask latents and process them
|
||||
mask_data = batch["masks"]
|
||||
mask_latents = mask_data["latents"]
|
||||
mask_latents = self._video_patchifier.patchify(mask_latents)
|
||||
|
||||
# Create binary mask: True = inpaint this region, False = keep original
|
||||
inpaint_mask = mask_latents.mean(dim=-1) > self.config.mask_threshold
|
||||
|
||||
# Sample noise and sigmas
|
||||
sigmas = timestep_sampler.sample_for(video_latents)
|
||||
noise = torch.randn_like(video_latents)
|
||||
|
||||
# Apply noise only to inpaint regions
|
||||
sigmas_expanded = sigmas.view(-1, 1, 1)
|
||||
noisy_latents = (1 - sigmas_expanded) * video_latents + sigmas_expanded * noise
|
||||
|
||||
# Keep original latents for non-inpaint regions (conditioning)
|
||||
inpaint_mask_expanded = inpaint_mask.unsqueeze(-1)
|
||||
noisy_latents = torch.where(inpaint_mask_expanded, noisy_latents, video_latents)
|
||||
|
||||
# Create per-token timesteps
|
||||
# Conditioning tokens (non-inpaint) get timestep=0
|
||||
# Inpaint tokens get the sampled sigma
|
||||
timesteps = self._create_per_token_timesteps(~inpaint_mask, sigmas.squeeze())
|
||||
|
||||
# Compute targets (velocity prediction: noise - clean)
|
||||
targets = noise - video_latents
|
||||
|
||||
# Get text embeddings
|
||||
conditions = batch["conditions"]
|
||||
video_prompt_embeds = conditions["video_prompt_embeds"]
|
||||
prompt_attention_mask = conditions["prompt_attention_mask"]
|
||||
|
||||
# Generate position embeddings
|
||||
positions = self._get_video_positions(
|
||||
num_frames=num_frames,
|
||||
height=height,
|
||||
width=width,
|
||||
batch_size=batch_size,
|
||||
fps=24.0, # Or get from latents_data
|
||||
device=device,
|
||||
dtype=dtype,
|
||||
)
|
||||
|
||||
# Create video Modality
|
||||
video_modality = Modality(
|
||||
enabled=True,
|
||||
latent=noisy_latents,
|
||||
timesteps=timesteps,
|
||||
positions=positions,
|
||||
context=video_prompt_embeds,
|
||||
context_mask=prompt_attention_mask,
|
||||
)
|
||||
|
||||
# Loss mask: only compute loss on inpaint regions
|
||||
loss_mask = inpaint_mask
|
||||
|
||||
return ModelInputs(
|
||||
video=video_modality,
|
||||
audio=None,
|
||||
video_targets=targets,
|
||||
audio_targets=None,
|
||||
video_loss_mask=loss_mask,
|
||||
audio_loss_mask=None,
|
||||
)
|
||||
|
||||
def compute_loss(
|
||||
self,
|
||||
video_pred: Tensor,
|
||||
audio_pred: Tensor | None,
|
||||
inputs: ModelInputs,
|
||||
) -> Tensor:
|
||||
"""Compute training loss on inpaint regions only."""
|
||||
# MSE loss
|
||||
loss = (video_pred - inputs.video_targets).pow(2)
|
||||
|
||||
# Apply loss mask
|
||||
loss_mask = inputs.video_loss_mask.unsqueeze(-1).float()
|
||||
loss = loss.mul(loss_mask).div(loss_mask.mean() + 1e-8)
|
||||
|
||||
return loss.mean()
|
||||
```
|
||||
|
||||
### Step 5: Register the Strategy
|
||||
|
||||
You need to register your strategy in two places:
|
||||
|
||||
**1. Update [`src/ltx_trainer/training_strategies/__init__.py`](../src/ltx_trainer/training_strategies/__init__.py):**
|
||||
|
||||
```python
|
||||
# Add import for your strategy
|
||||
from ltx_trainer.training_strategies.inpainting import InpaintingConfig, InpaintingStrategy
|
||||
|
||||
# Add to the TrainingStrategyConfig type alias
|
||||
TrainingStrategyConfig = TextToVideoConfig | VideoToVideoConfig | InpaintingConfig
|
||||
|
||||
# Add to __all__
|
||||
__all__ = [
|
||||
# ... existing exports ...
|
||||
"InpaintingConfig",
|
||||
"InpaintingStrategy",
|
||||
]
|
||||
|
||||
|
||||
# Add case in get_training_strategy()
|
||||
def get_training_strategy(config: TrainingStrategyConfig) -> TrainingStrategy:
|
||||
match config:
|
||||
# ... existing cases ...
|
||||
case InpaintingConfig():
|
||||
strategy = InpaintingStrategy(config)
|
||||
```
|
||||
|
||||
**2. Update [`src/ltx_trainer/config.py`](../src/ltx_trainer/config.py):**
|
||||
|
||||
```python
|
||||
# Add import
|
||||
from ltx_trainer.training_strategies.inpainting import InpaintingConfig
|
||||
|
||||
# Add to the TrainingStrategyConfig union with a Tag matching your strategy name
|
||||
TrainingStrategyConfig = Annotated[
|
||||
Annotated[TextToVideoConfig, Tag("text_to_video")]
|
||||
| Annotated[VideoToVideoConfig, Tag("video_to_video")]
|
||||
| Annotated[InpaintingConfig, Tag("inpainting")], # Add your config
|
||||
Discriminator(_get_strategy_discriminator),
|
||||
]
|
||||
```
|
||||
|
||||
### Step 6: Create a Configuration File
|
||||
|
||||
Create an example config in `configs/`:
|
||||
|
||||
```yaml
|
||||
# configs/ltx2_inpainting_lora.yaml
|
||||
|
||||
model:
|
||||
model_path: "/path/to/ltx2.safetensors"
|
||||
text_encoder_path: "/path/to/gemma"
|
||||
training_mode: "lora"
|
||||
|
||||
training_strategy:
|
||||
name: "inpainting" # Must match your Literal type
|
||||
mask_latents_dir: "mask_latents"
|
||||
mask_threshold: 0.5
|
||||
|
||||
lora:
|
||||
rank: 32
|
||||
alpha: 32
|
||||
target_modules:
|
||||
- "to_k"
|
||||
- "to_q"
|
||||
- "to_v"
|
||||
- "to_out.0"
|
||||
|
||||
data:
|
||||
preprocessed_data_root: "/path/to/preprocessed/dataset"
|
||||
|
||||
optimization:
|
||||
learning_rate: 1e-4
|
||||
steps: 2000
|
||||
batch_size: 1
|
||||
|
||||
# ... other config sections ...
|
||||
```
|
||||
|
||||
## 🔧 Helper Methods Reference
|
||||
|
||||
The base `TrainingStrategy` class provides these helper methods:
|
||||
|
||||
| Method | Purpose |
|
||||
|----------------------------------------------|-------------------------------------------------|
|
||||
| `_video_patchifier.patchify(latents)` | Convert `[B, C, F, H, W]` → `[B, seq_len, C]` |
|
||||
| `_audio_patchifier.patchify(latents)` | Convert `[B, C, T, F]` → `[B, T, C*F]` |
|
||||
| `_get_video_positions(...)` | Generate position embeddings for video |
|
||||
| `_get_audio_positions(...)` | Generate position embeddings for audio |
|
||||
| `_create_per_token_timesteps(mask, sigma)` | Create timesteps with 0 for conditioning tokens |
|
||||
| `_create_first_frame_conditioning_mask(...)` | Create mask for first-frame conditioning |
|
||||
|
||||
## 📊 Understanding ModelInputs
|
||||
|
||||
The `ModelInputs` dataclass contains everything needed for the forward pass and loss computation:
|
||||
|
||||
```python
|
||||
@dataclass
|
||||
class ModelInputs:
|
||||
video: Modality # Video modality data
|
||||
audio: Modality | None # Audio modality (None if video-only)
|
||||
|
||||
video_targets: Tensor # Target values for loss (velocity)
|
||||
audio_targets: Tensor | None
|
||||
|
||||
video_loss_mask: Tensor # Boolean: True = compute loss for this token
|
||||
audio_loss_mask: Tensor | None
|
||||
|
||||
ref_seq_len: int | None = None # For IC-LoRA: reference sequence length
|
||||
```
|
||||
|
||||
## 📊 Understanding Modality
|
||||
|
||||
The `Modality` dataclass (from ltx-core) represents a single modality's data:
|
||||
|
||||
```python
|
||||
@dataclass(frozen=True)
|
||||
class Modality:
|
||||
enabled: bool # Whether this modality is active
|
||||
latent: Tensor # [B, seq_len, C] - the latent tokens
|
||||
timesteps: Tensor # [B, seq_len] - per-token timesteps (sigmas)
|
||||
positions: Tensor # [B, dims, seq_len, 2] - position bounds
|
||||
context: Tensor # [B, ctx_len, C] - text embeddings
|
||||
context_mask: Tensor # [B, ctx_len] - attention mask for context
|
||||
```
|
||||
|
||||
> [!NOTE]
|
||||
> **Per-token timesteps:** Each token in the sequence has its own timestep. Conditioning tokens—those that should remain
|
||||
> un-noised—must have `timestep=0`. This is how the model distinguishes clean reference tokens from tokens to denoise. Use
|
||||
`_create_per_token_timesteps(conditioning_mask, sigma)` to set this up correctly.
|
||||
|
||||
> [!NOTE]
|
||||
> `Modality` is immutable (frozen dataclass). Use `dataclasses.replace()` to create modified copies.
|
||||
|
||||
## ✅ Testing Your Strategy
|
||||
|
||||
1. **Verify your training configuration is valid:**
|
||||
```bash
|
||||
uv run python -c "
|
||||
from ltx_trainer.config import LtxTrainerConfig
|
||||
import yaml
|
||||
|
||||
with open('configs/ltx2_inpainting_lora.yaml') as f:
|
||||
config = LtxTrainerConfig(**yaml.safe_load(f))
|
||||
print(f'Strategy: {config.training_strategy.name}')
|
||||
"
|
||||
```
|
||||
|
||||
2. **Test strategy instantiation:**
|
||||
```bash
|
||||
uv run python -c "
|
||||
from ltx_trainer.training_strategies import get_training_strategy
|
||||
from ltx_trainer.training_strategies.inpainting import InpaintingConfig
|
||||
|
||||
config = InpaintingConfig()
|
||||
strategy = get_training_strategy(config)
|
||||
print(f'Data sources: {strategy.get_data_sources()}')
|
||||
"
|
||||
```
|
||||
|
||||
3. **Run a short training test:**
|
||||
```bash
|
||||
uv run python scripts/train.py configs/ltx2_inpainting_lora.yaml
|
||||
```
|
||||
|
||||
## 💡 Tips and Best Practices
|
||||
|
||||
### Debugging
|
||||
|
||||
- Set `data.num_dataloader_workers: 0` to get clearer error messages
|
||||
- Use a small dataset and few steps for initial testing
|
||||
- Check tensor shapes at each step with print statements
|
||||
|
||||
## 🔗 Related Documentation
|
||||
|
||||
- [Training Modes](training-modes.md) - Overview of built-in training modes
|
||||
- [Configuration Reference](configuration-reference.md) - All configuration options
|
||||
- [Dataset Preparation](dataset-preparation.md) - Preprocessing workflow
|
||||
- [ltx-core Documentation](../../ltx-core/README.md) - Core model components
|
||||
|
||||
## 📚 Reference: Existing Strategies
|
||||
|
||||
Study these implementations for guidance:
|
||||
|
||||
| Strategy | Complexity | Key Features |
|
||||
|------------------------------------------------------------------------------------|------------|------------------------------------------------|
|
||||
| [`TextToVideoStrategy`](../src/ltx_trainer/training_strategies/text_to_video.py) | Simple | First-frame conditioning, optional audio |
|
||||
| [`VideoToVideoStrategy`](../src/ltx_trainer/training_strategies/video_to_video.py) | Medium | Reference video concatenation, split loss mask |
|
||||
@@ -0,0 +1,342 @@
|
||||
# Dataset Preparation Guide
|
||||
|
||||
This guide covers the complete workflow for preparing and preprocessing your dataset for training.
|
||||
|
||||
## 📋 Overview
|
||||
|
||||
The general dataset preparation workflow is:
|
||||
|
||||
1. **(Optional)** Split long videos into scenes using `split_scenes.py`
|
||||
2. **(Optional)** Generate captions for your videos using `caption_videos.py`
|
||||
3. **Preprocess your dataset** using `process_dataset.py` to compute and cache video/audio latents and text embeddings
|
||||
4. **Run the trainer** with your preprocessed dataset
|
||||
|
||||
## 🎬 Step 1: Split Scenes
|
||||
|
||||
If you're starting with raw, long-form videos (e.g., downloaded from YouTube), you should first split them into shorter, coherent scenes.
|
||||
|
||||
```bash
|
||||
uv run python scripts/split_scenes.py input.mp4 scenes_output_dir/ \
|
||||
--filter-shorter-than 5s
|
||||
```
|
||||
|
||||
This will create multiple video clips in `scenes_output_dir`.
|
||||
These clips will be the input for the captioning step, if you choose to use it.
|
||||
|
||||
The script supports many configuration options for scene detection (detector algorithms, thresholds, minimum scene lengths, etc.):
|
||||
|
||||
```bash
|
||||
uv run python scripts/split_scenes.py --help
|
||||
```
|
||||
|
||||
## 📝 Step 2: Caption Videos
|
||||
|
||||
If your dataset doesn't include captions, you can automatically generate them using multimodal models that understand both video and audio.
|
||||
|
||||
```bash
|
||||
uv run python scripts/caption_videos.py scenes_output_dir/ \
|
||||
--output scenes_output_dir/dataset.json
|
||||
```
|
||||
|
||||
If you're running into VRAM issues, try enabling 8-bit quantization to reduce memory usage:
|
||||
|
||||
```bash
|
||||
uv run python scripts/caption_videos.py scenes_output_dir/ \
|
||||
--output scenes_output_dir/dataset.json \
|
||||
--use-8bit
|
||||
```
|
||||
|
||||
This will create a `dataset.json` file containing video paths and their captions.
|
||||
|
||||
**Captioning options:**
|
||||
|
||||
| Option | Description |
|
||||
|--------|-------------|
|
||||
| `--captioner-type` | `qwen_omni` (default, local) or `gemini_flash` (API) |
|
||||
| `--use-8bit` | Enable 8-bit quantization for lower VRAM usage |
|
||||
| `--no-audio` | Disable audio processing (video-only captions) |
|
||||
| `--override` | Re-caption files that already have captions |
|
||||
| `--api-key` | API key for Gemini Flash (or set `GOOGLE_API_KEY` env var) |
|
||||
|
||||
**Caption format:**
|
||||
|
||||
The captioner produces structured captions with sections for:
|
||||
- **Visual content**: People, objects, actions, settings, colors, movements
|
||||
- **Speech transcription**: Word-for-word transcription of spoken content
|
||||
- **Sounds**: Music, ambient sounds, sound effects
|
||||
- **On-screen text**: Any visible text overlays
|
||||
|
||||
> [!NOTE]
|
||||
> The automatically generated captions may contain inaccuracies or hallucinated content.
|
||||
> We recommend reviewing and correcting the generated captions in your `dataset.json` file before proceeding to preprocessing.
|
||||
|
||||
## ⚡ Step 3: Dataset Preprocessing
|
||||
|
||||
This step preprocesses your video dataset by:
|
||||
|
||||
1. Resizing and cropping videos to fit specified resolution buckets
|
||||
2. Computing and caching video latent representations
|
||||
3. Computing and caching text embeddings for captions
|
||||
4. (Optional) Computing and caching audio latents
|
||||
|
||||
> [!WARNING]
|
||||
> Very large videos (especially high spatial resolution and/or many frames) can cause GPU out-of-memory (OOM)
|
||||
> during preprocessing/encoding.
|
||||
> The simplest fix is to reduce the target resolution (spatially: width/height) and/or the number of frames
|
||||
> (temporally) by using `--resolution-buckets` with smaller dimensions (lower width/height and/or fewer frames).
|
||||
|
||||
### Basic Usage
|
||||
|
||||
```bash
|
||||
uv run python scripts/process_dataset.py dataset.json \
|
||||
--resolution-buckets "960x544x49" \
|
||||
--model-path /path/to/ltx-2-model.safetensors \
|
||||
--text-encoder-path /path/to/gemma-model
|
||||
```
|
||||
|
||||
### With Audio Processing
|
||||
|
||||
For audio-video training, add the `--with-audio` flag:
|
||||
|
||||
```bash
|
||||
uv run python scripts/process_dataset.py dataset.json \
|
||||
--resolution-buckets "960x544x49" \
|
||||
--model-path /path/to/ltx-2-model.safetensors \
|
||||
--text-encoder-path /path/to/gemma-model \
|
||||
--with-audio
|
||||
```
|
||||
|
||||
### 📊 Dataset Format
|
||||
|
||||
The trainer supports either videos or single images.
|
||||
Note that your dataset must be homogeneous - either all videos or all images, mixing is not supported.
|
||||
|
||||
> [!TIP]
|
||||
> **Image Datasets:** When using images, follow the same preprocessing steps and format requirements as with videos,
|
||||
> but use `1` for the frame count in the resolution bucket (e.g., `960x544x1`).
|
||||
|
||||
The dataset must be a CSV, JSON, or JSONL metadata file with columns for captions and video paths:
|
||||
|
||||
**JSON format example:**
|
||||
|
||||
```json
|
||||
[
|
||||
{
|
||||
"caption": "A cat playing with a ball of yarn",
|
||||
"media_path": "videos/cat_playing.mp4"
|
||||
},
|
||||
{
|
||||
"caption": "A dog running in the park",
|
||||
"media_path": "videos/dog_running.mp4"
|
||||
}
|
||||
]
|
||||
```
|
||||
|
||||
**JSONL format example:**
|
||||
|
||||
```jsonl
|
||||
{"caption": "A cat playing with a ball of yarn", "media_path": "videos/cat_playing.mp4"}
|
||||
{"caption": "A dog running in the park", "media_path": "videos/dog_running.mp4"}
|
||||
```
|
||||
|
||||
**CSV format example:**
|
||||
|
||||
```csv
|
||||
caption,media_path
|
||||
"A cat playing with a ball of yarn","videos/cat_playing.mp4"
|
||||
"A dog running in the park","videos/dog_running.mp4"
|
||||
```
|
||||
|
||||
### 📐 Resolution Buckets
|
||||
|
||||
Videos are organized into "buckets" of specific dimensions (width × height × frames).
|
||||
Each video is assigned to the nearest matching bucket.
|
||||
You can preprocess with one or multiple resolution buckets.
|
||||
When training with multiple resolution buckets, you must use a batch size of 1.
|
||||
|
||||
The dimensions of each bucket must follow these constraints due to LTX-2's VAE architecture:
|
||||
|
||||
- **Spatial dimensions** (width and height) must be multiples of 32
|
||||
- **Number of frames** must satisfy `frames % 8 == 1` (e.g., 1, 9, 17, 25, 33, 41, 49, 57, 65, 73, 81, 89, 97, 121, etc.)
|
||||
|
||||
**Guidelines for choosing training resolution:**
|
||||
|
||||
- For high-quality, detailed videos: use larger spatial dimensions (e.g. 768x448) with fewer frames (e.g. 89)
|
||||
- For longer, motion-focused videos: use smaller spatial dimensions (512×512) with more frames (121)
|
||||
- Memory usage increases with both spatial and temporal dimensions
|
||||
|
||||
**Example usage:**
|
||||
|
||||
```bash
|
||||
uv run python scripts/process_dataset.py dataset.json \
|
||||
--resolution-buckets "960x544x49" \
|
||||
--model-path /path/to/ltx-2-model.safetensors \
|
||||
--text-encoder-path /path/to/gemma-model
|
||||
```
|
||||
|
||||
Multiple buckets are supported by separating entries with `;`:
|
||||
|
||||
```bash
|
||||
uv run python scripts/process_dataset.py dataset.json \
|
||||
--resolution-buckets "960x544x49;512x512x49" \
|
||||
--model-path /path/to/ltx-2-model.safetensors \
|
||||
--text-encoder-path /path/to/gemma-model
|
||||
```
|
||||
|
||||
**Video processing workflow:**
|
||||
|
||||
1. Videos are **resized** maintaining aspect ratio until either width or height matches the target
|
||||
2. The larger dimension is **center cropped** to match the bucket's dimensions
|
||||
3. Only the **first X frames are taken** to match the bucket's frame count, remaining frames are ignored
|
||||
|
||||
> [!NOTE]
|
||||
> The sequence length processed by the transformer model can be calculated as:
|
||||
>
|
||||
> ```
|
||||
> sequence_length = (H/32) * (W/32) * ((F-1)/8 + 1)
|
||||
> ```
|
||||
>
|
||||
> Where:
|
||||
> - H = Height of video
|
||||
> - W = Width of video
|
||||
> - F = Number of frames
|
||||
> - 32 = VAE's spatial downsampling factor
|
||||
> - 8 = VAE's temporal downsampling factor
|
||||
>
|
||||
> For example, a 768×448×89 video would have sequence length:
|
||||
> ```
|
||||
> (768/32) * (448/32) * ((89-1)/8 + 1) = 24 * 14 * 12 = 4,032
|
||||
> ```
|
||||
>
|
||||
> Keep this in mind when choosing video dimensions, as longer sequences require more GPU memory.
|
||||
|
||||
> [!WARNING]
|
||||
> When training with multiple resolution buckets, you must use a batch size of 1
|
||||
> (i.e., set `optimization.batch_size: 1` in your training config).
|
||||
|
||||
### 📁 Output Structure
|
||||
|
||||
The preprocessed data is saved in a `.precomputed` directory:
|
||||
|
||||
```
|
||||
dataset/
|
||||
└── .precomputed/
|
||||
├── latents/ # Cached video latents
|
||||
├── conditions/ # Cached text embeddings
|
||||
├── audio_latents/ # (only if --with-audio) Cached audio latents
|
||||
└── reference_latents/ # (only for IC-LoRA) Cached reference video latents
|
||||
```
|
||||
|
||||
## 🪄 IC-LoRA Reference Video Preprocessing
|
||||
|
||||
For IC-LoRA training, you need to preprocess datasets that include reference videos.
|
||||
Reference videos provide the conditioning input while target videos represent the desired transformed output.
|
||||
|
||||
### Dataset Format with Reference Videos
|
||||
|
||||
**JSON format:**
|
||||
|
||||
```json
|
||||
[
|
||||
{
|
||||
"caption": "A cat playing with a ball of yarn",
|
||||
"media_path": "videos/cat_playing.mp4",
|
||||
"reference_path": "references/cat_playing_depth.mp4"
|
||||
}
|
||||
]
|
||||
```
|
||||
|
||||
**JSONL format:**
|
||||
|
||||
```jsonl
|
||||
{"caption": "A cat playing with a ball of yarn", "media_path": "videos/cat_playing.mp4", "reference_path": "references/cat_playing_depth.mp4"}
|
||||
{"caption": "A dog running in the park", "media_path": "videos/dog_running.mp4", "reference_path": "references/dog_running_depth.mp4"}
|
||||
```
|
||||
|
||||
### Preprocessing with Reference Videos
|
||||
|
||||
To preprocess a dataset with reference videos, add the `--reference-column` argument specifying the name of the field
|
||||
in your dataset JSON/JSONL/CSV that contains the reference video paths:
|
||||
|
||||
```bash
|
||||
uv run python scripts/process_dataset.py dataset.json \
|
||||
--resolution-buckets "960x544x49" \
|
||||
--model-path /path/to/ltx-2-model.safetensors \
|
||||
--text-encoder-path /path/to/gemma-model \
|
||||
--reference-column "reference_path"
|
||||
```
|
||||
|
||||
This will create an additional `reference_latents/` directory containing the preprocessed reference video latents.
|
||||
|
||||
|
||||
### Generating Reference Videos
|
||||
|
||||
**Dataset Requirements for IC-LoRA:**
|
||||
|
||||
- Your dataset must contain paired videos where each target video has a corresponding reference video
|
||||
- Reference and target videos must have *identical* resolution and length
|
||||
- Both reference and target videos should be preprocessed together using the same resolution buckets
|
||||
|
||||
We provide an example script, [`scripts/compute_reference.py`](../scripts/compute_reference.py), to generate reference
|
||||
videos for a given dataset. The default implementation generates Canny edge reference videos.
|
||||
|
||||
```bash
|
||||
uv run python scripts/compute_reference.py scenes_output_dir/ \
|
||||
--output scenes_output_dir/dataset.json
|
||||
```
|
||||
|
||||
The script accepts a JSON file as the dataset configuration and updates it in-place by adding the filenames of the generated reference videos.
|
||||
|
||||
If you want to generate a different type of condition (depth maps, pose skeletons, etc.), modify or replace the `compute_reference()` function within this script.
|
||||
|
||||
### Example Dataset
|
||||
|
||||
For reference, see our **[Canny Control Dataset](https://huggingface.co/datasets/Lightricks/Canny-Control-Dataset)** which demonstrates proper IC-LoRA dataset structure with paired videos and Canny edge maps.
|
||||
|
||||
|
||||
## 🎯 LoRA Trigger Words
|
||||
|
||||
When training a LoRA, you can specify a trigger token that will be prepended to all captions:
|
||||
|
||||
```bash
|
||||
uv run python scripts/process_dataset.py dataset.json \
|
||||
--resolution-buckets "960x544x49" \
|
||||
--model-path /path/to/ltx-2-model.safetensors \
|
||||
--text-encoder-path /path/to/gemma-model \
|
||||
--lora-trigger "MYTRIGGER"
|
||||
```
|
||||
|
||||
This acts as a trigger word that activates the LoRA during inference when you include the same token in your prompts.
|
||||
|
||||
> [!NOTE]
|
||||
> There is no need to manually insert the trigger word into your dataset JSON/JSONL/CSV file.
|
||||
> The trigger word specified with `--lora-trigger` is automatically prepended to each caption during preprocessing.
|
||||
|
||||
## 🔍 Decoding Videos for Verification
|
||||
|
||||
If you add the `--decode` flag, the script will VAE-decode the precomputed latents and save the resulting videos
|
||||
in `.precomputed/decoded_videos`. When audio preprocessing is enabled (`--with-audio`), audio latents will also be
|
||||
decoded and saved to `.precomputed/decoded_audio`. This allows you to visually and audibly inspect the processed data.
|
||||
|
||||
```bash
|
||||
uv run python scripts/process_dataset.py dataset.json \
|
||||
--resolution-buckets "960x544x49" \
|
||||
--model-path /path/to/ltx-2-model.safetensors \
|
||||
--text-encoder-path /path/to/gemma-model \
|
||||
--decode
|
||||
```
|
||||
|
||||
For single-frame images, the decoded latents will be saved as PNG files rather than MP4 videos.
|
||||
|
||||
## 🚀 Next Steps
|
||||
|
||||
Once your dataset is preprocessed, you can proceed to:
|
||||
|
||||
- Configure your training parameters in [Configuration Reference](configuration-reference.md)
|
||||
- Choose your training approach in [Training Modes](training-modes.md)
|
||||
- Start training with the [Training Guide](training-guide.md)
|
||||
|
||||
> [!TIP]
|
||||
> If your training recipe requires additional preprocessed data (e.g., masks, conditioning signals), see
|
||||
> [Implementing Custom Training Strategies](custom-training-strategies.md) for guidance on extending the
|
||||
> preprocessing pipeline.
|
||||
@@ -0,0 +1,128 @@
|
||||
# Quick Start Guide
|
||||
|
||||
Get up and running with LTX-2 training in just a few steps!
|
||||
|
||||
## 📋 Prerequisites
|
||||
|
||||
Before you begin, ensure you have:
|
||||
|
||||
1. **LTX-2 Model Checkpoint** - A local `.safetensors` file containing the LTX-2 model weights.
|
||||
Download `ltx-2-19b-dev.safetensors` from: [HuggingFace Hub](https://huggingface.co/Lightricks/LTX-2)
|
||||
2. **Gemma Text Encoder** - A local directory containing the Gemma model (required for LTX-2).
|
||||
Download from: [HuggingFace Hub](https://huggingface.co/google/gemma-3-12b-it-qat-q4_0-unquantized/)
|
||||
3. **Linux with CUDA** - The trainer requires `triton` which is Linux-only
|
||||
4. **GPU with sufficient VRAM** - 80GB recommended. Lower VRAM may work with gradient checkpointing and lower
|
||||
resolutions
|
||||
|
||||
## ⚡ Installation
|
||||
|
||||
First, install [uv](https://docs.astral.sh/uv/getting-started/installation/) if you haven't already.
|
||||
Then clone the repository and install the dependencies:
|
||||
|
||||
```bash
|
||||
git clone https://github.com/Lightricks/LTX-2
|
||||
```
|
||||
|
||||
The `ltx-trainer` package is part of the `LTX-2` monorepo. Install the dependencies from the repository root,
|
||||
then navigate to the trainer package:
|
||||
|
||||
```bash
|
||||
# From the repository root
|
||||
uv sync
|
||||
cd packages/ltx-trainer
|
||||
```
|
||||
|
||||
> [!NOTE]
|
||||
> The trainer depends on [`ltx-core`](../../ltx-core/) and [`ltx-pipelines`](../../ltx-pipelines/)
|
||||
> packages which are automatically installed from the monorepo.
|
||||
|
||||
## 🏋 Training Workflow
|
||||
|
||||
### 1. Prepare Your Dataset
|
||||
|
||||
Organize your videos and captions, then preprocess them:
|
||||
|
||||
```bash
|
||||
# Split long videos into scenes (optional)
|
||||
uv run python scripts/split_scenes.py input.mp4 scenes_output_dir/ --filter-shorter-than 5s
|
||||
|
||||
# Generate captions for videos (optional)
|
||||
uv run python scripts/caption_videos.py scenes_output_dir/ --output dataset.json
|
||||
|
||||
# Preprocess the dataset (compute latents and embeddings)
|
||||
uv run python scripts/process_dataset.py dataset.json \
|
||||
--resolution-buckets "960x544x49" \
|
||||
--model-path /path/to/ltx-2-model.safetensors \
|
||||
--text-encoder-path /path/to/gemma-model
|
||||
```
|
||||
|
||||
See [Dataset Preparation](dataset-preparation.md) for detailed instructions.
|
||||
|
||||
### 2. Configure Training
|
||||
|
||||
Create or modify a configuration YAML file. Start with one of the example configs:
|
||||
|
||||
- [`configs/ltx2_av_lora.yaml`](../configs/ltx2_av_lora.yaml) - Audio-video LoRA training
|
||||
- [`configs/ltx2_v2v_ic_lora.yaml`](../configs/ltx2_v2v_ic_lora.yaml) - IC-LoRA video-to-video
|
||||
|
||||
Key settings to update:
|
||||
|
||||
```yaml
|
||||
model:
|
||||
model_path: "/path/to/ltx-2-model.safetensors"
|
||||
text_encoder_path: "/path/to/gemma-model"
|
||||
|
||||
data:
|
||||
preprocessed_data_root: "/path/to/preprocessed/data"
|
||||
|
||||
output_dir: "outputs/my_training_run"
|
||||
```
|
||||
|
||||
See [Configuration Reference](configuration-reference.md) for all available options.
|
||||
|
||||
### 3. Start Training
|
||||
|
||||
```bash
|
||||
uv run python scripts/train.py configs/ltx2_av_lora.yaml
|
||||
```
|
||||
|
||||
For multi-GPU training:
|
||||
|
||||
```bash
|
||||
uv run accelerate launch scripts/train.py configs/ltx2_av_lora.yaml
|
||||
```
|
||||
|
||||
See [Training Guide](training-guide.md) for distributed training and advanced options.
|
||||
|
||||
## 🎯 Training Modes
|
||||
|
||||
The trainer supports several training modes:
|
||||
|
||||
| Mode | Description | Config Example |
|
||||
|----------------------|--------------------------------|--------------------------------------------|
|
||||
| **LoRA** | Efficient adapter training | `training_strategy.name: "text_to_video"` |
|
||||
| **Audio-Video LoRA** | Joint audio-video training | `training_strategy.with_audio: true` |
|
||||
| **IC-LoRA** | Video-to-video transformations | `training_strategy.name: "video_to_video"` |
|
||||
| **Full Fine-tuning** | Full model training | `model.training_mode: "full"` |
|
||||
|
||||
See [Training Modes](training-modes.md) for detailed explanations,
|
||||
or [Custom Training Strategies](custom-training-strategies.md) if you need to implement your own training recipe.
|
||||
|
||||
## Next Steps
|
||||
|
||||
Once you've completed your first training run, you can:
|
||||
|
||||
- **Use your trained LoRA for inference** - The [`ltx-pipelines`](../../ltx-pipelines/) package provides
|
||||
production-ready inference
|
||||
pipelines for various use cases (T2V, I2V, IC-LoRA, etc.). See the package documentation for details.
|
||||
- Learn more about [Dataset Preparation](dataset-preparation.md) for advanced preprocessing
|
||||
- Explore different [Training Modes](training-modes.md) (LoRA, Audio-Video, IC-LoRA)
|
||||
- Dive deeper into [Training Configuration](configuration-reference.md)
|
||||
- Understand the model architecture in [LTX-Core Documentation](../../ltx-core/README.md)
|
||||
|
||||
## Need Help?
|
||||
|
||||
If you run into issues at any step, see the [Troubleshooting Guide](troubleshooting.md) for solutions to common
|
||||
problems.
|
||||
|
||||
Join our [Discord community](https://discord.gg/ltxplatform) for real-time help and discussion!
|
||||
@@ -0,0 +1,203 @@
|
||||
# Training Guide
|
||||
|
||||
This guide covers how to run training jobs, from basic single-GPU training to advanced distributed setups and automatic
|
||||
model uploads.
|
||||
|
||||
## ⚡ Basic Training (Single GPU)
|
||||
|
||||
After preprocessing your dataset and preparing a configuration file, you can start training using the trainer script:
|
||||
|
||||
```bash
|
||||
uv run python scripts/train.py configs/ltx2_av_lora.yaml
|
||||
```
|
||||
|
||||
The trainer will:
|
||||
|
||||
1. **Load your configuration** and validate all parameters
|
||||
2. **Initialize models** and apply optimizations
|
||||
3. **Run the training loop** with progress tracking
|
||||
4. **Generate validation videos** (if configured)
|
||||
5. **Save the trained weights** in your output directory
|
||||
|
||||
### Output Files
|
||||
|
||||
**For LoRA training:**
|
||||
|
||||
- `lora_weights.safetensors` - Main LoRA weights file
|
||||
- `training_config.yaml` - Copy of training configuration
|
||||
- `validation_samples/` - Generated validation videos (if enabled)
|
||||
|
||||
**For full model fine-tuning:**
|
||||
|
||||
- `model_weights.safetensors` - Full model weights
|
||||
- `training_config.yaml` - Copy of training configuration
|
||||
- `validation_samples/` - Generated validation videos (if enabled)
|
||||
|
||||
## 🖥️ Distributed / Multi-GPU Training
|
||||
|
||||
We use Hugging Face 🤗 [Accelerate](https://huggingface.co/docs/accelerate/index) for multi-GPU DDP and FSDP.
|
||||
|
||||
### Configure Accelerate
|
||||
|
||||
Run the interactive wizard once to set up your environment (DDP / FSDP, GPU count, etc.):
|
||||
|
||||
```bash
|
||||
uv run accelerate config
|
||||
```
|
||||
|
||||
This stores your preferences in `~/.cache/huggingface/accelerate/default_config.yaml`.
|
||||
|
||||
### Use the Provided Accelerate Configs (Recommended)
|
||||
|
||||
We include ready-to-use Accelerate config files in `configs/accelerate/`:
|
||||
|
||||
- [ddp.yaml](../configs/accelerate/ddp.yaml) — Standard DDP
|
||||
- [ddp_compile.yaml](../configs/accelerate/ddp_compile.yaml) — DDP with `torch.compile` (Inductor)
|
||||
- [fsdp.yaml](../configs/accelerate/fsdp.yaml) — Standard FSDP (auto-wraps `BasicAVTransformerBlock`)
|
||||
- [fsdp_compile.yaml](../configs/accelerate/fsdp_compile.yaml) — FSDP with `torch.compile` (Inductor)
|
||||
|
||||
Launch with a specific config using `--config_file`:
|
||||
|
||||
```bash
|
||||
# DDP (2 GPUs shown as example)
|
||||
CUDA_VISIBLE_DEVICES=0,1 \
|
||||
uv run accelerate launch --config_file configs/accelerate/ddp.yaml \
|
||||
scripts/train.py configs/ltx2_av_lora.yaml
|
||||
|
||||
# DDP + torch.compile
|
||||
CUDA_VISIBLE_DEVICES=0,1 \
|
||||
uv run accelerate launch --config_file configs/accelerate/ddp_compile.yaml \
|
||||
scripts/train.py configs/ltx2_av_lora.yaml
|
||||
|
||||
# FSDP (4 GPUs shown as example)
|
||||
CUDA_VISIBLE_DEVICES=0,1,2,3 \
|
||||
uv run accelerate launch --config_file configs/accelerate/fsdp.yaml \
|
||||
scripts/train.py configs/ltx2_av_lora.yaml
|
||||
|
||||
# FSDP + torch.compile
|
||||
CUDA_VISIBLE_DEVICES=0,1,2,3 \
|
||||
uv run accelerate launch --config_file configs/accelerate/fsdp_compile.yaml \
|
||||
scripts/train.py configs/ltx2_av_lora.yaml
|
||||
```
|
||||
|
||||
**Notes:**
|
||||
|
||||
- The number of processes is taken from the Accelerate config (`num_processes`). Override with `--num_processes X` or
|
||||
restrict GPUs with `CUDA_VISIBLE_DEVICES`.
|
||||
- The compile variants enable `torch.compile` with the Inductor backend via Accelerate's `dynamo_config`.
|
||||
- FSDP configs auto-wrap the transformer blocks (`fsdp_transformer_layer_cls_to_wrap: BasicAVTransformerBlock`).
|
||||
|
||||
### Launch with Your Default Accelerate Config
|
||||
|
||||
If you prefer to use your default Accelerate profile:
|
||||
|
||||
```bash
|
||||
# Use settings from your default accelerate config
|
||||
uv run accelerate launch scripts/train.py configs/ltx2_av_lora.yaml
|
||||
|
||||
# Override number of processes on the fly (e.g., 2 GPUs)
|
||||
uv run accelerate launch --num_processes 2 scripts/train.py configs/ltx2_av_lora.yaml
|
||||
|
||||
# Select specific GPUs
|
||||
CUDA_VISIBLE_DEVICES=0,1 uv run accelerate launch scripts/train.py configs/ltx2_av_lora.yaml
|
||||
```
|
||||
|
||||
> [!TIP]
|
||||
> You can disable the in-terminal progress bars with `--disable-progress-bars` flag in the trainer CLI if desired.
|
||||
|
||||
### Benefits of Distributed Training
|
||||
|
||||
- **Faster training**: Distribute workload across multiple GPUs
|
||||
- **Larger effective batch sizes**: Combine gradients from multiple GPUs
|
||||
- **Memory efficiency**: Each GPU handles a portion of the batch
|
||||
|
||||
> [!NOTE]
|
||||
> Distributed training requires that all GPUs have sufficient memory for the model and batch size. The effective batch
|
||||
> size becomes `batch_size × num_processes`.
|
||||
|
||||
## 🤗 Pushing Models to Hugging Face Hub
|
||||
|
||||
You can automatically push your trained models to the Hugging Face Hub by adding the following to your configuration:
|
||||
|
||||
```yaml
|
||||
hub:
|
||||
push_to_hub: true
|
||||
hub_model_id: "your-username/your-model-name"
|
||||
```
|
||||
|
||||
### Prerequisites
|
||||
|
||||
Before pushing, make sure you:
|
||||
|
||||
1. **Have a Hugging Face account** - Sign up at [huggingface.co](https://huggingface.co)
|
||||
2. **Are logged in** via `huggingface-cli login` or have set the `HUGGING_FACE_HUB_TOKEN` environment variable
|
||||
3. **Have write access** to the specified repository (it will be created if it doesn't exist)
|
||||
|
||||
### Login Options
|
||||
|
||||
**Option 1: Interactive login**
|
||||
|
||||
```bash
|
||||
uv run huggingface-cli login
|
||||
```
|
||||
|
||||
**Option 2: Environment variable**
|
||||
|
||||
```bash
|
||||
export HUGGING_FACE_HUB_TOKEN="your_token_here"
|
||||
```
|
||||
|
||||
### What Gets Uploaded
|
||||
|
||||
The trainer will automatically:
|
||||
|
||||
- **Create a model card** with training details and sample outputs
|
||||
- **Upload model weights**
|
||||
- **Push sample videos as GIFs** in the model card
|
||||
- **Include training configuration and prompts**
|
||||
|
||||
## 📊 Weights & Biases Logging
|
||||
|
||||
Enable experiment tracking with W&B by adding to your configuration:
|
||||
|
||||
```yaml
|
||||
wandb:
|
||||
enabled: true
|
||||
project: "ltx-2-trainer"
|
||||
entity: null # Your W&B username or team
|
||||
tags: [ "ltx2", "lora" ]
|
||||
log_validation_videos: true
|
||||
```
|
||||
|
||||
This will log:
|
||||
|
||||
- Training loss and learning rate
|
||||
- Validation videos
|
||||
- Model configuration
|
||||
- Training progress
|
||||
|
||||
## 🚀 Next Steps
|
||||
|
||||
After training completes:
|
||||
|
||||
- **Run inference with your trained LoRA** - The [`ltx-pipelines`](../../ltx-pipelines/) package provides
|
||||
production-ready inference
|
||||
pipelines that support loading custom LoRAs. Available pipelines include text-to-video, image-to-video,
|
||||
IC-LoRA video-to-video, and more. See the [`ltx-pipelines`](../../ltx-pipelines/) package for usage details.
|
||||
- **Test your model** with validation prompts
|
||||
- **Iterate and improve** based on validation results
|
||||
- **Share your results** by pushing to Hugging Face Hub
|
||||
|
||||
## 💡 Tips for Successful Training
|
||||
|
||||
- **Start small**: Begin with a small dataset and a few hundred steps to verify everything works
|
||||
- **Monitor validation**: Keep an eye on validation samples to catch overfitting
|
||||
- **Adjust learning rate**: Lower learning rates often produce better results
|
||||
- **Use gradient checkpointing**: Essential for training with limited GPU memory
|
||||
- **Save checkpoints**: Regular checkpoints help recover from interruptions
|
||||
|
||||
## Need Help?
|
||||
|
||||
If you encounter issues during training, see the [Troubleshooting Guide](troubleshooting.md).
|
||||
|
||||
Join our [Discord community](https://discord.gg/ltxplatform) for real-time help!
|
||||
@@ -0,0 +1,221 @@
|
||||
# Training Modes Guide
|
||||
|
||||
The trainer supports several training modes, each suited for different use cases and requirements.
|
||||
|
||||
## 🎯 Standard LoRA Training (Video-Only)
|
||||
|
||||
Standard LoRA (Low-Rank Adaptation) training fine-tunes the model by adding small, trainable adapter layers while
|
||||
keeping the base model frozen. This approach:
|
||||
|
||||
- **Requires significantly less memory and compute** than full fine-tuning
|
||||
- **Produces small, portable weight files** (typically a few hundred MB)
|
||||
- **Is ideal for learning specific styles, effects, or concepts**
|
||||
- **Can be easily combined with other LoRAs** during inference
|
||||
|
||||
Configure standard LoRA training with:
|
||||
|
||||
```yaml
|
||||
model:
|
||||
training_mode: "lora"
|
||||
|
||||
training_strategy:
|
||||
name: "text_to_video"
|
||||
first_frame_conditioning_p: 0.1
|
||||
with_audio: false # Video-only training
|
||||
```
|
||||
|
||||
## 🔊 Audio-Video LoRA Training
|
||||
|
||||
LTX-2 supports joint audio-video generation. You can train LoRA adapters that affect both video and audio output:
|
||||
|
||||
- **Synchronized audio-video generation** - Audio matches the visual content
|
||||
- **Same efficient LoRA approach** - Just enable audio training
|
||||
- **Requires audio latents** - Dataset must include preprocessed audio
|
||||
|
||||
Configure audio-video training with:
|
||||
|
||||
```yaml
|
||||
model:
|
||||
training_mode: "lora"
|
||||
|
||||
training_strategy:
|
||||
name: "text_to_video"
|
||||
first_frame_conditioning_p: 0.1
|
||||
with_audio: true # Enable audio training
|
||||
audio_latents_dir: "audio_latents" # Directory containing audio latents
|
||||
```
|
||||
|
||||
**Example configuration file:**
|
||||
|
||||
- 📄 [Audio-Video LoRA Training](../configs/ltx2_av_lora.yaml)
|
||||
|
||||
**Dataset structure for audio-video training:**
|
||||
|
||||
```
|
||||
preprocessed_data_root/
|
||||
├── latents/ # Video latents
|
||||
├── conditions/ # Text embeddings
|
||||
└── audio_latents/ # Audio latents (required when with_audio: true)
|
||||
```
|
||||
|
||||
> [!IMPORTANT]
|
||||
> When training audio-video LoRAs, ensure your `target_modules` configuration captures video, audio, and
|
||||
> cross-modal attention branches. Use patterns like `"to_k"` instead of `"attn1.to_k"` to match:
|
||||
> - Video modules: `attn1.to_k`, `attn2.to_k`
|
||||
> - Audio modules: `audio_attn1.to_k`, `audio_attn2.to_k`
|
||||
> - Cross-modal modules: `audio_to_video_attn.to_k`, `video_to_audio_attn.to_k`
|
||||
>
|
||||
> The cross-modal attention modules (`audio_to_video_attn` and `video_to_audio_attn`) enable bidirectional
|
||||
> information flow between audio and video, which is critical for synchronized audiovisual generation.
|
||||
> See [Understanding Target Modules](configuration-reference.md#understanding-target-modules) for detailed guidance.
|
||||
|
||||
> [!NOTE]
|
||||
> You can generate audio during validation even if you're not training the audio branch.
|
||||
> Set `validation.generate_audio: true` independently of `training_strategy.with_audio`.
|
||||
|
||||
## 🔥 Full Model Fine-tuning
|
||||
|
||||
Full model fine-tuning updates all parameters of the base model, providing maximum flexibility but
|
||||
requiring substantial computational resources and larger training datasets:
|
||||
|
||||
- **Offers the highest potential quality and capability improvements**
|
||||
- **Requires multiple GPUs** and distributed training techniques (e.g., FSDP)
|
||||
- **Produces large checkpoint files** (several GB)
|
||||
- **Best for major model adaptations** or when LoRA limitations are reached
|
||||
|
||||
Configure full fine-tuning with:
|
||||
|
||||
```yaml
|
||||
model:
|
||||
training_mode: "full"
|
||||
|
||||
training_strategy:
|
||||
name: "text_to_video"
|
||||
first_frame_conditioning_p: 0.1
|
||||
```
|
||||
|
||||
> [!IMPORTANT]
|
||||
> Full fine-tuning of LTX-2 requires multiple high-end GPUs (e.g., 4-8× H100 80GB) and distributed
|
||||
> training with FSDP. See [Training Guide](training-guide.md) for multi-GPU setup instructions.
|
||||
|
||||
## 🔄 In-Context LoRA (IC-LoRA) Training
|
||||
|
||||
IC-LoRA is a specialized training mode for video-to-video transformations.
|
||||
Unlike standard training modes that learn from individual videos, IC-LoRA learns transformations from pairs of videos.
|
||||
IC-LoRA enables a wide range of advanced video-to-video applications, such as:
|
||||
|
||||
- **Control adapters** (e.g., Depth, Pose): Learn to map from a control signal (like a depth map or pose skeleton) to a
|
||||
target video
|
||||
- **Video deblurring**: Transform blurry input videos into sharp, high-quality outputs
|
||||
- **Style transfer**: Apply the style of a reference video to a target video sequence
|
||||
- **Colorization**: Convert grayscale reference videos into colorized outputs
|
||||
- **Restoration and enhancement**: Denoise, upscale, or restore old or degraded videos
|
||||
|
||||
By providing paired reference and target videos, IC-LoRA can learn complex transformations that go beyond caption-based conditioning.
|
||||
|
||||
IC-LoRA training fundamentally differs from standard LoRA and full fine-tuning:
|
||||
|
||||
- **Reference videos** provide clean, unnoised conditioning input showing the "before" state
|
||||
- **Target videos** are noised during training and represent the desired "after" state
|
||||
- **The model learns transformations** from reference videos to target videos
|
||||
- **Loss is applied only to the target portion**, not the reference
|
||||
- **Training and inference time increase significantly** due to the doubled sequence length
|
||||
|
||||
To enable IC-LoRA training, configure your YAML file with:
|
||||
|
||||
```yaml
|
||||
model:
|
||||
training_mode: "lora" # Required: IC-LoRA uses LoRA mode
|
||||
|
||||
training_strategy:
|
||||
name: "video_to_video"
|
||||
first_frame_conditioning_p: 0.1
|
||||
reference_latents_dir: "reference_latents" # Directory for reference video latents
|
||||
```
|
||||
|
||||
**Example configuration file:**
|
||||
|
||||
- 📄 [IC-LoRA Training](../configs/ltx2_v2v_ic_lora.yaml) - Video-to-video transformation training
|
||||
|
||||
### Dataset Requirements for IC-LoRA
|
||||
|
||||
- Your dataset must contain **paired videos** where each target video has a corresponding reference video
|
||||
- Reference and target videos must have **identical resolution and length**
|
||||
- Both reference and target videos should be **preprocessed together** using the same resolution buckets
|
||||
|
||||
**Dataset structure for IC-LoRA training:**
|
||||
|
||||
```
|
||||
preprocessed_data_root/
|
||||
├── latents/ # Target video latents (what the model learns to generate)
|
||||
├── conditions/ # Text embeddings for each video
|
||||
└── reference_latents/ # Reference video latents (conditioning input)
|
||||
```
|
||||
|
||||
### Generating Reference Videos
|
||||
|
||||
We provide an example script to generate reference videos (e.g., Canny edge maps) for a given dataset.
|
||||
The script takes a JSON file as input (e.g., output of `caption_videos.py`) and updates it with the generated reference
|
||||
video paths.
|
||||
|
||||
```bash
|
||||
uv run python scripts/compute_reference.py scenes_output_dir/ \
|
||||
--output scenes_output_dir/dataset.json
|
||||
```
|
||||
|
||||
To compute a different condition (depth maps, pose skeletons, etc.), modify the `compute_reference()` function in the
|
||||
script.
|
||||
|
||||
### Configuration Requirements for IC-LoRA
|
||||
|
||||
- You **must** provide `reference_videos` in your validation configuration when using IC-LoRA training
|
||||
- The number of reference videos must match the number of validation prompts
|
||||
|
||||
Example validation configuration for IC-LoRA:
|
||||
|
||||
```yaml
|
||||
validation:
|
||||
prompts:
|
||||
- "First prompt describing the desired output"
|
||||
- "Second prompt describing the desired output"
|
||||
reference_videos:
|
||||
- "/path/to/reference1.mp4"
|
||||
- "/path/to/reference2.mp4"
|
||||
include_reference_in_output: true # Show reference side-by-side with output
|
||||
```
|
||||
|
||||
## 📊 Training Mode Comparison
|
||||
|
||||
| Aspect | LoRA | Audio-Video LoRA | Full Fine-tuning | IC-LoRA |
|
||||
|----------------------|------------|------------------|------------------|----------------|
|
||||
| **Memory Usage** | Low | Low-Medium | High | Medium |
|
||||
| **Training Speed** | Fast | Fast | Slow | Medium |
|
||||
| **Output Size** | 100MB-few GB (depends on rank) | 100MB-few GB (depends on rank) | Tens of GB | 100MB-few GB (depends on rank) |
|
||||
| **Flexibility** | Medium | Medium | High | Specialized |
|
||||
| **Audio Support** | Optional | Yes | Optional | No |
|
||||
| **Reference Videos** | No | No | No | Yes (required) |
|
||||
|
||||
## 🎬 Using Trained Models for Inference
|
||||
|
||||
After training, use the [`ltx-pipelines`](../../ltx-pipelines/) package for production inference with your trained LoRAs:
|
||||
|
||||
| Training Mode | Recommended Pipeline |
|
||||
|---------------|---------------------|
|
||||
| LoRA / Audio-Video LoRA | `TI2VidOneStagePipeline` or `TI2VidTwoStagesPipeline` |
|
||||
| IC-LoRA | `ICLoraPipeline` |
|
||||
|
||||
All pipelines support loading custom LoRAs via the `loras` parameter. See the [`ltx-pipelines`](../../ltx-pipelines/) package
|
||||
documentation for detailed usage instructions.
|
||||
|
||||
## 🚀 Next Steps
|
||||
|
||||
Once you've chosen your training mode:
|
||||
|
||||
- Set up your dataset using [Dataset Preparation](dataset-preparation.md)
|
||||
- Configure your training parameters in [Configuration Reference](configuration-reference.md)
|
||||
- Start training with the [Training Guide](training-guide.md)
|
||||
|
||||
> [!TIP]
|
||||
> Need a training mode that's not covered here? See [Implementing Custom Training Strategies](custom-training-strategies.md)
|
||||
> to learn how to create your own strategy for specialized use cases like video inpainting, audio-only training, or
|
||||
> custom conditioning.
|
||||
@@ -0,0 +1,295 @@
|
||||
# Troubleshooting Guide
|
||||
|
||||
This guide covers common issues and solutions when training with the LTX-2 trainer.
|
||||
|
||||
## 🔧 VRAM and Memory Issues
|
||||
|
||||
Memory management is crucial for successful training with LTX-2.
|
||||
|
||||
### Memory Optimization Techniques
|
||||
|
||||
#### 1. Enable Gradient Checkpointing
|
||||
|
||||
Gradient checkpointing trades training speed for memory savings. **Highly recommended** for most training runs:
|
||||
|
||||
```yaml
|
||||
optimization:
|
||||
enable_gradient_checkpointing: true
|
||||
```
|
||||
|
||||
#### 2. Enable 8-bit Text Encoder
|
||||
|
||||
Load the Gemma text encoder in 8-bit precision to save GPU memory:
|
||||
|
||||
```yaml
|
||||
acceleration:
|
||||
load_text_encoder_in_8bit: true
|
||||
```
|
||||
|
||||
#### 3. Reduce Batch Size
|
||||
|
||||
Lower the batch size if you encounter out-of-memory errors:
|
||||
|
||||
```yaml
|
||||
optimization:
|
||||
batch_size: 1 # Start with 1 and increase gradually
|
||||
```
|
||||
|
||||
Use gradient accumulation to maintain a larger effective batch size:
|
||||
|
||||
```yaml
|
||||
optimization:
|
||||
batch_size: 1
|
||||
gradient_accumulation_steps: 4 # Effective batch size = 4
|
||||
```
|
||||
|
||||
#### 4. Use Lower Resolution
|
||||
|
||||
Reduce spatial or temporal dimensions to save memory:
|
||||
|
||||
```bash
|
||||
# Smaller spatial resolution
|
||||
uv run python scripts/process_dataset.py dataset.json \
|
||||
--resolution-buckets "512x512x49" \
|
||||
--model-path /path/to/model.safetensors \
|
||||
--text-encoder-path /path/to/gemma
|
||||
|
||||
# Fewer frames
|
||||
uv run python scripts/process_dataset.py dataset.json \
|
||||
--resolution-buckets "960x544x25" \
|
||||
--model-path /path/to/model.safetensors \
|
||||
--text-encoder-path /path/to/gemma
|
||||
```
|
||||
|
||||
#### 5. Enable Model Quantization
|
||||
|
||||
Use quantization to reduce memory usage:
|
||||
|
||||
```yaml
|
||||
acceleration:
|
||||
quantization: "int8-quanto" # Options: int8-quanto, int4-quanto, fp8-quanto
|
||||
```
|
||||
|
||||
#### 6. Use 8-bit Optimizer
|
||||
|
||||
The 8-bit AdamW optimizer uses less memory:
|
||||
|
||||
```yaml
|
||||
optimization:
|
||||
optimizer_type: "adamw8bit"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## ⚠️ Common Usage Issues
|
||||
|
||||
### Issue: "No module named 'ltx_trainer'" Error
|
||||
|
||||
**Solution:**
|
||||
Ensure you've installed the dependencies and are using `uv run` to execute scripts:
|
||||
|
||||
```bash
|
||||
# From the repository root
|
||||
uv sync
|
||||
cd packages/ltx-trainer
|
||||
uv run python scripts/train.py configs/ltx2_av_lora.yaml
|
||||
```
|
||||
|
||||
> [!TIP]
|
||||
> Always use `uv run` to execute Python scripts. This automatically uses the correct virtual environment
|
||||
> without requiring manual activation.
|
||||
|
||||
### Issue: "Gemma model path is not a directory" Error
|
||||
|
||||
**Solution:**
|
||||
The `text_encoder_path` must point to a directory containing the Gemma model, not a file:
|
||||
|
||||
```yaml
|
||||
model:
|
||||
model_path: "/path/to/ltx-2-model.safetensors" # File path
|
||||
text_encoder_path: "/path/to/gemma-model/" # Directory path
|
||||
```
|
||||
|
||||
### Issue: "Model path does not exist" Error
|
||||
|
||||
**Solution:**
|
||||
LTX-2 requires local model paths. URLs are not supported:
|
||||
|
||||
```yaml
|
||||
# ✅ Correct - local path
|
||||
model:
|
||||
model_path: "/path/to/ltx-2-model.safetensors"
|
||||
|
||||
# ❌ Wrong - URL not supported
|
||||
model:
|
||||
model_path: "https://huggingface.co/..."
|
||||
```
|
||||
|
||||
### Issue: "Frames must satisfy frames % 8 == 1" Error
|
||||
|
||||
**Solution:**
|
||||
LTX-2 requires the number of frames to satisfy `frames % 8 == 1`:
|
||||
|
||||
- ✅ Valid: 1, 9, 17, 25, 33, 41, 49, 57, 65, 73, 81, 89, 97, 121
|
||||
- ❌ Invalid: 24, 32, 48, 64, 100
|
||||
|
||||
### Issue: Slow Training Speed
|
||||
|
||||
**Optimizations:**
|
||||
|
||||
1. **Disable gradient checkpointing** (if you have enough VRAM):
|
||||
|
||||
```yaml
|
||||
optimization:
|
||||
enable_gradient_checkpointing: false
|
||||
```
|
||||
|
||||
|
||||
2. **Use torch.compile** via Accelerate:
|
||||
|
||||
```bash
|
||||
uv run accelerate launch --config_file configs/accelerate/ddp_compile.yaml \
|
||||
scripts/train.py configs/ltx2_av_lora.yaml
|
||||
```
|
||||
|
||||
### Issue: Poor Quality Validation Outputs
|
||||
|
||||
**Solutions:**
|
||||
|
||||
1. **Use Image-to-Video Validation:**
|
||||
For more reliable validation, use image-to-video (first-frame conditioning) rather than pure text-to-video:
|
||||
|
||||
```yaml
|
||||
validation:
|
||||
prompts:
|
||||
- "a professional portrait video of a person"
|
||||
images:
|
||||
- "/path/to/first_frame.png" # One image per prompt
|
||||
```
|
||||
|
||||
2. **Increase inference steps:**
|
||||
|
||||
```yaml
|
||||
validation:
|
||||
inference_steps: 50 # Default is 30
|
||||
```
|
||||
|
||||
3. **Adjust guidance settings:**
|
||||
|
||||
```yaml
|
||||
validation:
|
||||
guidance_scale: 4.0 # CFG scale (recommended: 4.0)
|
||||
stg_scale: 1.0 # STG scale for temporal coherence (recommended: 1.0)
|
||||
stg_blocks: [29] # Transformer block to perturb
|
||||
```
|
||||
|
||||
4. **Check caption quality:**
|
||||
Review and manually edit captions for accuracy if using auto-generated captions.
|
||||
LTX-2 prefers long, detailed captions that describe both visual content and audio (e.g., ambient sounds, speech,
|
||||
music).
|
||||
|
||||
5. **Check target modules:**
|
||||
Ensure your `target_modules` configuration matches your training goals. For audio-video training,
|
||||
use patterns that match both branches (e.g., `"to_k"` instead of `"attn1.to_k"`).
|
||||
See [Understanding Target Modules](configuration-reference.md#understanding-target-modules) for details.
|
||||
|
||||
6. **Adjust LoRA rank:**
|
||||
Try higher values for more capacity:
|
||||
|
||||
```yaml
|
||||
lora:
|
||||
rank: 64 # Or 128 for more capacity
|
||||
```
|
||||
|
||||
7. **Increase training steps:**
|
||||
|
||||
```yaml
|
||||
optimization:
|
||||
steps: 3000
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🔍 Debugging Tools
|
||||
|
||||
### Monitor GPU Memory Usage
|
||||
|
||||
Track memory usage during training:
|
||||
|
||||
```bash
|
||||
# Watch GPU memory in real-time
|
||||
watch -n 1 nvidia-smi
|
||||
|
||||
# Log memory usage to file
|
||||
nvidia-smi --query-gpu=memory.used,memory.total --format=csv --loop=5 > memory_log.csv
|
||||
```
|
||||
|
||||
### Verify Preprocessed Data
|
||||
|
||||
Decode latents to visualize the preprocessed videos:
|
||||
|
||||
```bash
|
||||
uv run python scripts/decode_latents.py dataset/.precomputed/latents debug_output \
|
||||
--model-path /path/to/model.safetensors
|
||||
```
|
||||
|
||||
To also decode audio latents, add the `--with-audio` flag:
|
||||
|
||||
```bash
|
||||
uv run python scripts/decode_latents.py dataset/.precomputed/latents debug_output \
|
||||
--model-path /path/to/model.safetensors \
|
||||
--with-audio
|
||||
```
|
||||
|
||||
Compare decoded videos and audio with originals to ensure quality.
|
||||
|
||||
---
|
||||
|
||||
## 💡 Best Practices
|
||||
|
||||
### Before Training
|
||||
|
||||
- [ ] Test preprocessing with a small subset first
|
||||
- [ ] Verify all video files are accessible
|
||||
- [ ] Check available GPU memory
|
||||
- [ ] Review configuration against hardware capabilities
|
||||
- [ ] Ensure model and text encoder paths are correct
|
||||
|
||||
### During Training
|
||||
|
||||
- [ ] Monitor GPU memory usage
|
||||
- [ ] Check loss convergence regularly
|
||||
- [ ] Review validation samples periodically
|
||||
- [ ] Save checkpoints frequently
|
||||
|
||||
### After Training
|
||||
|
||||
- [ ] Test trained model with diverse prompts
|
||||
- [ ] Document training parameters and results
|
||||
- [ ] Archive training data and configs
|
||||
|
||||
## 🆘 Getting Help
|
||||
|
||||
If you're still experiencing issues:
|
||||
|
||||
1. **Check logs:** Review console output for error details
|
||||
2. **Search issues:** Look through GitHub issues for similar problems
|
||||
3. **Provide details:** When reporting issues, include:
|
||||
- Hardware specifications (GPU model, VRAM)
|
||||
- Configuration file used
|
||||
- Complete error message
|
||||
- Steps to reproduce the issue
|
||||
|
||||
---
|
||||
|
||||
## 🤝 Join the Community
|
||||
|
||||
Have questions, want to share your results, or need real-time help?
|
||||
Join our [community Discord server](https://discord.gg/ltxplatform)
|
||||
to connect with other users and the development team!
|
||||
|
||||
- Get troubleshooting help
|
||||
- Share your training results and workflows
|
||||
- Stay up to date with announcements and updates
|
||||
|
||||
We look forward to seeing you there!
|
||||
@@ -0,0 +1,274 @@
|
||||
# Utility Scripts Reference
|
||||
|
||||
This guide covers the various utility scripts available for preprocessing, conversion, and debugging tasks.
|
||||
|
||||
## 🎬 Dataset Processing Scripts
|
||||
|
||||
### Video Scene Splitting
|
||||
|
||||
The `scripts/split_scenes.py` script automatically splits long videos into shorter, coherent scenes.
|
||||
|
||||
```bash
|
||||
# Basic scene splitting
|
||||
uv run python scripts/split_scenes.py input.mp4 output_dir/ --filter-shorter-than 5s
|
||||
```
|
||||
|
||||
**Key features:**
|
||||
|
||||
- **Automatic scene detection**: Uses PySceneDetect for intelligent splitting
|
||||
- **Multiple algorithms**: Content-based, adaptive, threshold, and histogram detection
|
||||
- **Filtering options**: Remove scenes shorter than specified duration
|
||||
- **Customizable parameters**: Thresholds, window sizes, and detection modes
|
||||
|
||||
**Common options:**
|
||||
|
||||
```bash
|
||||
# See all available options
|
||||
uv run python scripts/split_scenes.py --help
|
||||
|
||||
# Use adaptive detection with custom threshold
|
||||
uv run python scripts/split_scenes.py video.mp4 scenes/ --detector adaptive --threshold 30.0
|
||||
|
||||
# Limit to maximum number of scenes
|
||||
uv run python scripts/split_scenes.py video.mp4 scenes/ --max-scenes 50
|
||||
```
|
||||
|
||||
### Automatic Video Captioning
|
||||
|
||||
The `scripts/caption_videos.py` script generates captions for videos (with audio) using multimodal models.
|
||||
|
||||
```bash
|
||||
# Generate captions for all videos in a directory (uses Qwen2.5-Omni by default)
|
||||
uv run python scripts/caption_videos.py videos_dir/ --output dataset.json
|
||||
|
||||
# Use 8-bit quantization to reduce VRAM usage
|
||||
uv run python scripts/caption_videos.py videos_dir/ --output dataset.json --use-8bit
|
||||
|
||||
# Use Gemini Flash API instead (requires API key)
|
||||
uv run python scripts/caption_videos.py videos_dir/ --output dataset.json \
|
||||
--captioner-type gemini_flash --api-key YOUR_API_KEY
|
||||
|
||||
# Caption without audio processing (video-only)
|
||||
uv run python scripts/caption_videos.py videos_dir/ --output dataset.json --no-audio
|
||||
|
||||
# Force re-caption all files
|
||||
uv run python scripts/caption_videos.py videos_dir/ --output dataset.json --override
|
||||
```
|
||||
|
||||
**Key features:**
|
||||
|
||||
- **Audio-visual captioning**: Processes both video and audio content, including speech transcription
|
||||
- **Multiple backends**:
|
||||
- `qwen_omni` (default): Local Qwen2.5-Omni model - processes video + audio locally
|
||||
- `gemini_flash`: Google Gemini Flash API - cloud-based, requires API key
|
||||
- **Structured output**: Captions include visual description, speech transcription, sounds, and on-screen text
|
||||
- **Memory optimization**: 8-bit quantization option for limited VRAM
|
||||
- **Incremental processing**: Skips already-captioned files by default
|
||||
- **Multiple output formats**: JSON, JSONL, CSV, or TXT
|
||||
|
||||
**Caption format:**
|
||||
|
||||
The captioner produces structured captions with four sections:
|
||||
- `[VISUAL]`: Detailed description of visual content
|
||||
- `[SPEECH]`: Word-for-word transcription of spoken content
|
||||
- `[SOUNDS]`: Description of music, ambient sounds, sound effects
|
||||
- `[TEXT]`: Any on-screen text visible in the video
|
||||
|
||||
**Environment variables (for Gemini Flash):**
|
||||
|
||||
Set one of these to use Gemini Flash without passing `--api-key`:
|
||||
- `GOOGLE_API_KEY`
|
||||
- `GEMINI_API_KEY`
|
||||
|
||||
### Dataset Preprocessing
|
||||
|
||||
The `scripts/process_dataset.py` script processes videos and caches latents for training.
|
||||
|
||||
```bash
|
||||
# Basic preprocessing
|
||||
uv run python scripts/process_dataset.py dataset.json \
|
||||
--resolution-buckets "960x544x49" \
|
||||
--model-path /path/to/ltx-2-model.safetensors \
|
||||
--text-encoder-path /path/to/gemma-model
|
||||
|
||||
# With audio processing
|
||||
uv run python scripts/process_dataset.py dataset.json \
|
||||
--resolution-buckets "960x544x49" \
|
||||
--model-path /path/to/ltx-2-model.safetensors \
|
||||
--text-encoder-path /path/to/gemma-model \
|
||||
--with-audio
|
||||
|
||||
# With video decoding for verification
|
||||
uv run python scripts/process_dataset.py dataset.json \
|
||||
--resolution-buckets "960x544x49" \
|
||||
--model-path /path/to/ltx-2-model.safetensors \
|
||||
--text-encoder-path /path/to/gemma-model \
|
||||
--decode
|
||||
```
|
||||
|
||||
Multiple resolution buckets can be specified, separated by `;`:
|
||||
|
||||
```bash
|
||||
uv run python scripts/process_dataset.py dataset.json \
|
||||
--resolution-buckets "960x544x49;512x512x81" \
|
||||
--model-path /path/to/ltx-2-model.safetensors \
|
||||
--text-encoder-path /path/to/gemma-model
|
||||
```
|
||||
|
||||
> [!NOTE]
|
||||
> When training with multiple resolution buckets, set `optimization.batch_size: 1`.
|
||||
|
||||
For detailed usage, see the [Dataset Preparation Guide](dataset-preparation.md).
|
||||
|
||||
### Reference Video Generation
|
||||
|
||||
The `scripts/compute_reference.py` script provides a template for creating reference videos needed for IC-LoRA training.
|
||||
The default implementation generates Canny edge reference videos.
|
||||
|
||||
```bash
|
||||
# Generate Canny edge reference videos
|
||||
uv run python scripts/compute_reference.py videos_dir/ --output dataset.json
|
||||
```
|
||||
|
||||
**Key features:**
|
||||
|
||||
- **Canny edge detection**: Creates edge-based reference videos
|
||||
- **In-place editing**: Updates existing dataset JSON files
|
||||
- **Customizable**: Modify the `compute_reference()` function for different conditions (depth, pose, etc.)
|
||||
|
||||
> [!TIP]
|
||||
> You can edit this script to generate other types of reference videos for IC-LoRA training,
|
||||
> such as depth maps, segmentation masks, or any custom video transformation.
|
||||
|
||||
## 🔍 Debugging and Verification Scripts
|
||||
|
||||
### Latents Decoding
|
||||
|
||||
The `scripts/decode_latents.py` script decodes precomputed video latents back into video files for visual inspection.
|
||||
|
||||
```bash
|
||||
# Basic usage
|
||||
uv run python scripts/decode_latents.py /path/to/latents/dir \
|
||||
--output-dir /path/to/output \
|
||||
--model-path /path/to/ltx-2-model.safetensors
|
||||
|
||||
# With VAE tiling for large videos
|
||||
uv run python scripts/decode_latents.py /path/to/latents/dir \
|
||||
--output-dir /path/to/output \
|
||||
--model-path /path/to/ltx-2-model.safetensors \
|
||||
--vae-tiling
|
||||
|
||||
# Decode both video and audio latents
|
||||
uv run python scripts/decode_latents.py /path/to/latents/dir \
|
||||
--output-dir /path/to/output \
|
||||
--model-path /path/to/ltx-2-model.safetensors \
|
||||
--with-audio
|
||||
```
|
||||
|
||||
**The script will:**
|
||||
|
||||
1. **Load the VAE model** from the specified path
|
||||
2. **Process all `.pt` latent files** in the input directory
|
||||
3. **Decode each latent** back into a video using the VAE
|
||||
4. **Save resulting videos** as MP4 files in the output directory
|
||||
|
||||
**When to use:**
|
||||
|
||||
- **Verify preprocessing quality**: Check that your videos were encoded correctly
|
||||
- **Debug training data**: Visualize what the model actually sees during training
|
||||
- **Quality assessment**: Ensure latent encoding preserves important visual details
|
||||
|
||||
|
||||
### Inference Script
|
||||
|
||||
The `scripts/inference.py` script runs inference with a trained model.
|
||||
|
||||
> [!TIP]
|
||||
> For production inference, consider using the [`ltx-pipelines`](../../ltx-pipelines/) package which provides optimized,
|
||||
> feature-rich pipelines for various use cases:
|
||||
> - **Text/Image-to-Video**: `TI2VidOneStagePipeline`, `TI2VidTwoStagesPipeline`
|
||||
> - **Distilled (fast) inference**: `DistilledPipeline`
|
||||
> - **IC-LoRA video-to-video**: `ICLoraPipeline`
|
||||
> - **Keyframe interpolation**: `KeyframeInterpolationPipeline`
|
||||
>
|
||||
> All pipelines support loading custom LoRAs trained with this trainer.
|
||||
|
||||
```bash
|
||||
# Text-to-video inference (with audio by default)
|
||||
# By default, uses CFG scale 4.0 and STG scale 1.0 with block 29
|
||||
uv run python scripts/inference.py \
|
||||
--checkpoint /path/to/model.safetensors \
|
||||
--text-encoder-path /path/to/gemma \
|
||||
--prompt "A cat playing with a ball" \
|
||||
--output output.mp4
|
||||
|
||||
# Video-only (skip audio generation)
|
||||
uv run python scripts/inference.py \
|
||||
--checkpoint /path/to/model.safetensors \
|
||||
--text-encoder-path /path/to/gemma \
|
||||
--prompt "A cat playing with a ball" \
|
||||
--skip-audio \
|
||||
--output output.mp4
|
||||
|
||||
# Image-to-video with conditioning image
|
||||
uv run python scripts/inference.py \
|
||||
--checkpoint /path/to/model.safetensors \
|
||||
--text-encoder-path /path/to/gemma \
|
||||
--prompt "A cat walking" \
|
||||
--condition-image first_frame.png \
|
||||
--output output.mp4
|
||||
|
||||
# Custom guidance settings
|
||||
uv run python scripts/inference.py \
|
||||
--checkpoint /path/to/model.safetensors \
|
||||
--text-encoder-path /path/to/gemma \
|
||||
--prompt "A cat playing with a ball" \
|
||||
--guidance-scale 4.0 \
|
||||
--stg-scale 1.0 \
|
||||
--stg-blocks 29 \
|
||||
--output output.mp4
|
||||
|
||||
# Disable STG (CFG only)
|
||||
uv run python scripts/inference.py \
|
||||
--checkpoint /path/to/model.safetensors \
|
||||
--text-encoder-path /path/to/gemma \
|
||||
--prompt "A cat playing with a ball" \
|
||||
--stg-scale 0.0 \
|
||||
--output output.mp4
|
||||
```
|
||||
|
||||
**Guidance parameters:**
|
||||
|
||||
| Parameter | Default | Description |
|
||||
|-----------|---------|-------------|
|
||||
| `--guidance-scale` | 4.0 | CFG (Classifier-Free Guidance) scale |
|
||||
| `--stg-scale` | 1.0 | STG (Spatio-Temporal Guidance) scale. 0.0 disables STG |
|
||||
| `--stg-blocks` | 29 | Transformer block(s) to perturb for STG |
|
||||
| `--stg-mode` | stg_av | `stg_av` perturbs both audio and video, `stg_v` video only |
|
||||
|
||||
## 🚀 Training Scripts
|
||||
|
||||
### Basic and Distributed Training
|
||||
|
||||
Use `scripts/train.py` for both single GPU and multi-GPU runs:
|
||||
|
||||
```bash
|
||||
# Single-GPU training
|
||||
uv run python scripts/train.py configs/ltx2_av_lora.yaml
|
||||
|
||||
# Multi-GPU (uses your accelerate config)
|
||||
uv run accelerate launch scripts/train.py configs/ltx2_av_lora.yaml
|
||||
|
||||
# Override number of processes
|
||||
uv run accelerate launch --num_processes 4 scripts/train.py configs/ltx2_av_lora.yaml
|
||||
```
|
||||
|
||||
For detailed usage, see the [Training Guide](training-guide.md).
|
||||
|
||||
## 💡 Tips for Using Utility Scripts
|
||||
|
||||
- **Start with `--help`**: Always check available options for each script
|
||||
- **Test on small datasets**: Verify workflows with a few files before processing large datasets
|
||||
- **Use decode verification**: Always decode a few samples to verify preprocessing quality
|
||||
- **Monitor VRAM usage**: Use `--use-8bit` or quantization flags when running into memory issues
|
||||
- **Keep backups**: Make copies of important dataset files before running conversion scripts
|
||||
Reference in New Issue
Block a user