indigo/LTX-2
1
0
Fork 0
Code Issues Pull Requests 1 Actions Packages Projects Releases Wiki Activity

Automated PR - 2026-06-17

Browse Source
This commit is contained in:
github-actions[bot]
2026-06-17 14:06:32 +00:00
parent d6053703e0
commit 0d3d3a3855
90 changed files with 8265 additions and 4511 deletions
Download Patch File Download Diff File Expand all files Collapse all files
packages/ltx-trainer/docs/configuration-reference.md
+153 -62
View File
@@ -10,7 +10,7 @@ sub-configurations:
- **ModelConfig**: Base model and training mode settings
- **LoraConfig**: LoRA training parameters
- **TrainingStrategyConfig**: Training strategy settings (text-to-video or video-to-video)
- **TrainingStrategyConfig**: Training strategy settings (flexible conditioning framework)
- **OptimizationConfig**: Learning rate, batch sizes, and scheduler settings
- **AccelerationConfig**: Mixed precision and quantization settings
- **DataConfig**: Data loading parameters
@@ -24,13 +24,29 @@ sub-configurations:
Check out our example configurations in the `configs` directory:
- 📄 [Audio-Video LoRA Training](../configs/ltx2_av_lora.yaml) - Joint audio-video generation training
- 📄 [Audio-Video LoRA Training (Low VRAM)](../configs/ltx2_av_lora_low_vram.yaml) - Memory-optimized config for 32GB
GPUs (uses 8-bit optimizer, INT8 quantization, and reduced LoRA rank)
- 📄 [IC-LoRA Training](../configs/ltx2_v2v_ic_lora.yaml) - Video-to-video transformation training
- 📄 [Text-to-Video LoRA](../configs/t2v_lora.yaml) - Text-to-video LoRA training
- 📄 [Image-to-Video LoRA](../configs/i2v_lora.yaml) - Image-to-video LoRA training
- 📄 [IC-LoRA Video-to-Video](../configs/v2v_ic_lora.yaml) - IC-LoRA video-to-video training
- 📄 [Audio-to-Video LoRA](../configs/a2v_lora.yaml) - Audio-to-video LoRA training
- 📄 [Video-to-Audio LoRA](../configs/v2a_lora.yaml) - Video-to-audio (Foley) LoRA training
- 📄 [Video Extension LoRA](../configs/video_extend_lora.yaml) - Video extension (forward) LoRA training
- 📄 [Video Suffix LoRA](../configs/video_suffix_lora.yaml) - Video extension (backward) LoRA training
- 📄 [Video Inpainting LoRA](../configs/video_inpainting_lora.yaml) - Video inpainting LoRA training
- 📄 [Video Outpainting LoRA](../configs/video_outpainting_lora.yaml) - Video outpainting (spatial crop) LoRA training
- 📄 [Text-to-Audio LoRA](../configs/t2a_lora.yaml) - Text-to-audio LoRA training
- 📄 [Audio Extension LoRA](../configs/audio_extend_lora.yaml) - Audio extension (forward) LoRA training
- 📄 [Audio Suffix LoRA](../configs/audio_suffix_lora.yaml) - Audio extension (backward) LoRA training
- 📄 [Audio Inpainting LoRA](../configs/audio_inpainting_lora.yaml) - Audio inpainting LoRA training
- 📄 [Audio-to-Audio IC-LoRA](../configs/a2a_ic_lora.yaml) - Audio IC-LoRA transformation training
- 📄 [AV2AV IC-LoRA](../configs/av2av_ic_lora.yaml) - Audio+video IC-LoRA transformation training
- 📄 [T2V LoRA (Low VRAM)](../configs/t2v_lora_low_vram.yaml) - Memory-optimized config for 32GB GPUs
## ⚙️ Configuration Sections
> [!NOTE]
> The YAML snippets below show **recommended starting values**, not necessarily the code defaults.
> Fields you omit from your config file will use the code defaults from [`config.py`](../src/ltx_trainer/config.py).
### ModelConfig
Controls the base model and training mode settings.
@@ -149,37 +165,60 @@ target_modules:
### TrainingStrategyConfig
Configures the training strategy. The trainer includes two built-in strategies described below.
For custom use cases, see [Implementing Custom Training Strategies](custom-training-strategies.md).
Configures the training strategy. The recommended strategy is `"flexible"`, which supports all conditioning scenarios through configuration.
#### Text-to-Video Strategy
#### Flexible Strategy
The flexible strategy provides a unified conditioning framework. Each modality (video, audio) is configured
independently with its own latents directory, generation flag, and list of conditions.
```yaml
training_strategy:
name: "text_to_video"
first_frame_conditioning_p: 0.1 # Probability of first-frame conditioning
with_audio: false # Enable joint audio-video training
audio_latents_dir: "audio_latents" # Directory for audio latents (when with_audio: true)
name: "flexible"
video:
is_generated: true # Video is denoised during training
latents_dir: "latents" # Directory containing precomputed video latents
conditions:
- type: first_frame # Use first frame as conditioning
probability: 0.5 # Apply this condition 50% of the time
audio:
is_generated: true # Audio is denoised during training
latents_dir: "audio_latents" # Directory containing precomputed audio latents
conditions: [] # No additional audio conditions (text-only)
```
#### Video-to-Video Strategy (IC-LoRA)
**ModalityConfig parameters:**
```yaml
training_strategy:
name: "video_to_video"
first_frame_conditioning_p: 0.1
reference_latents_dir: "reference_latents" # Directory for reference video latents
```
| Parameter | Description |
|----------------|------------------------------------------------------------------------------------------------------------------|
| `is_generated` | `true` = modality is denoised (contributes to loss). `false` = frozen conditioning (sigma=0, no loss). |
| `latents_dir` | Directory name within `preprocessed_data_root` containing precomputed latents for this modality. |
| `conditions` | List of conditioning configs applied during training (see condition types below). Text conditioning is implicit. |
**Key parameters:**
**Condition types:**
| Parameter | Description |
|------------------------------|------------------------------------------------------------------|
| `name` | Strategy type: `"text_to_video"` or `"video_to_video"` |
| `first_frame_conditioning_p` | Probability of using first frame as conditioning (0.0-1.0) |
| `with_audio` | (text_to_video only) Enable joint audio-video training |
| `audio_latents_dir` | (text_to_video only) Directory name for audio latents |
| `reference_latents_dir` | (video_to_video only) Directory name for reference video latents |
| Type | Parameters | Description |
|----------------|-----------------------------------------------------|---------------------------------------------------------------------------------------|
| `first_frame` | `probability` | First latent frame is clean, excluded from loss. **Video only.** |
| `prefix` | `temporal_boundary`, `probability` | First N latent temporal units are clean. For extension forward. |
| `suffix` | `temporal_boundary`, `probability` | Last N latent temporal units are clean. For extension backward. |
| `spatial_crop` | `spatial_region` (y1, x1, y2, x2 in px), `probability` | Rectangular region is clean, excluded from loss. For outpainting. **Video only.** |
| `mask` | `mask_dir`, `probability` | Per-sample mask directory. Masks are thresholded at `0.5`; `1` means conditioning, `0` means generate. |
| `reference` | `latents_dir`, `probability` | IC-LoRA style concatenation. Reference tokens are prepended, clean (timestep=0), no loss. |
> [!NOTE]
> The `prefix`, `suffix`, `mask`, and `reference` condition types work on both video and audio modalities —
> place them in the `video.conditions` or `audio.conditions` list as appropriate.
> `first_frame` and `spatial_crop` are video-only conditions.
> [!NOTE]
> Training conditions reference **directories** of precomputed data (within `preprocessed_data_root`),
> while validation conditions reference **individual files** (images, videos, masks) that are encoded
> on-the-fly during validation. The condition `type` names are the same, but the fields differ.
> [!NOTE]
> The legacy `text_to_video` and `video_to_video` strategies are deprecated but remain forward-compatible.
> New configs should use `name: "flexible"`.
### OptimizationConfig
@@ -206,7 +245,7 @@ optimization:
| `steps` | Total number of training steps |
| `batch_size` | Batch size per GPU (reduce if running out of memory) |
| `gradient_accumulation_steps` | Accumulate gradients over multiple steps |
| `scheduler_type` | LR scheduler: `"constant"`, `"linear"`, `"cosine"`, `"cosine_with_restarts"`, `"polynomial"` |
| `scheduler_type` | LR scheduler: `"constant"`, `"linear"`, `"cosine"`, `"cosine_with_restarts"`, `"polynomial"`, `"step"` |
| `enable_gradient_checkpointing` | Trade training speed for GPU memory savings (recommended for large models) |
### AccelerationConfig
@@ -226,7 +265,7 @@ acceleration:
| Parameter | Description |
|---------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `mixed_precision_mode` | Precision mode - `"bf16"` recommended for modern GPUs |
| `quantization` | Model quantization: `null`, `"int8-quanto"`, `"int4-quanto"`, `"fp8-quanto"`, etc. |
| `quantization` | Model quantization: `null`, `"int8-quanto"`, `"int4-quanto"`, `"int2-quanto"`, `"fp8-quanto"`, or `"fp8uz-quanto"` |
| `load_text_encoder_in_8bit` | Load the Gemma text encoder in 8-bit to save GPU memory |
| `offload_optimizer_during_validation` | Move optimizer state to CPU before validation video sampling and back afterwards. Useful when validation OOMs because VAE decoder + transformer + optimizer state can't coexist on the GPU (full fine-tune, high-rank LoRA). No effect for FSDP. |
@@ -244,50 +283,84 @@ data:
| Parameter | Description |
|--------------------------|--------------------------------------------------------------------------------------------|
| `preprocessed_data_root` | Path to your preprocessed dataset (contains `latents/`, `conditions/`, etc.) |
| `preprocessed_data_root` | Path to your preprocessed dataset directory produced by `process_dataset.py` (contains `latents/`, `conditions/`, etc.) |
| `num_dataloader_workers` | Number of parallel data loading processes (0 = synchronous loading, useful when debugging) |
### ValidationConfig
Validation and inference settings for monitoring training progress.
Validation and inference settings for monitoring training progress. Validation samples use a self-describing
format where each sample specifies its own prompt and conditions.
```yaml
validation:
prompts: # Validation prompts
- "A cat playing with a ball"
- "A dog running in a field"
negative_prompt: "worst quality, inconsistent motion, blurry, jittery, distorted"
images: null # Optional image paths for image-to-video
reference_videos: null # Reference video paths (IC-LoRA only)
video_dims: [ 576, 576, 89 ] # Video dimensions [width, height, frames]
frame_rate: 25.0 # Frame rate for generated videos
seed: 42 # Random seed for reproducibility
inference_steps: 30 # Number of inference steps
interval: 100 # Steps between validation runs
guidance_scale: 4.0 # CFG guidance strength
stg_scale: 1.0 # STG guidance strength (0.0 to disable)
stg_blocks: [ 29 ] # Transformer blocks to perturb for STG
stg_mode: "stg_av" # "stg_av" or "stg_v" (video only)
generate_audio: true # Whether to generate audio
skip_initial_validation: false # Skip validation at step 0
include_reference_in_output: false # Include reference video side-by-side (IC-LoRA)
samples:
- prompt: "A cat playing with a ball"
conditions:
- type: first_frame
image_or_video: "/path/to/image.png"
- prompt: "A dog running in a field"
video_dims: [576, 576, 89] # Output dimensions: [width, height, frames]
negative_prompt: "worst quality, inconsistent motion, blurry, jittery, distorted" # Negative prompt for all samples
frame_rate: 25.0 # Output video frame rate (fps)
seed: 42 # Random seed for reproducibility
inference_steps: 30 # Number of denoising steps
interval: 100 # Run validation every N steps (null to disable)
guidance_scale: 4.0 # CFG scale (higher = stronger prompt adherence)
stg_scale: 1.0 # STG scale (0.0 to disable)
stg_blocks: [29] # Transformer blocks to apply STG perturbation
stg_mode: "stg_av" # STG mode: "stg_av" (audio+video) or "stg_v" (video only)
generate_audio: true # Whether to generate audio during validation
generate_video: true # Whether to generate video during validation
skip_initial_validation: false # Skip validation at step 0
```
**Key parameters:**
| Parameter | Description |
|-------------------------------|--------------------------------------------------------------------------------------------------------------------------|
| `prompts` | List of text prompts for validation video generation |
| `images` | List of image paths for image-to-video validation (must match number of prompts) |
| `reference_videos` | List of reference video paths for IC-LoRA validation (must match number of prompts) |
| `video_dims` | Output dimensions `[width, height, frames]`. Width/height must be divisible by 32, frames must satisfy `frames % 8 == 1` |
| `interval` | Steps between validation runs (set to `null` to disable) |
| `guidance_scale` | CFG (Classifier-Free Guidance) scale. Recommended: 4.0 |
| `stg_scale` | STG (Spatio-Temporal Guidance) scale. 0.0 disables STG. Recommended: 1.0 |
| `stg_blocks` | Transformer blocks to perturb for STG. Recommended: `[29]` (single block) |
| `stg_mode` | STG mode: `"stg_av"` perturbs both audio and video, `"stg_v"` perturbs video only |
| `generate_audio` | Whether to generate audio in validation samples |
| `include_reference_in_output` | For IC-LoRA: concatenate reference video side-by-side with output |
| Parameter | Description |
|--------------------------|--------------------------------------------------------------------------------------------------------------------------|
| `samples` | List of `ValidationSample` objects (see below). Replaces the legacy `prompts`/`images`/`reference_videos` fields. |
| `video_dims` | Output dimensions `[width, height, frames]`. Width/height must be divisible by 32, frames must satisfy `frames % 8 == 1` |
| `interval` | Steps between validation runs (set to `null` to disable) |
| `guidance_scale` | CFG (Classifier-Free Guidance) scale. Recommended: 4.0 |
| `stg_scale` | STG (Spatio-Temporal Guidance) scale. 0.0 disables STG. Recommended: 1.0 |
| `stg_blocks` | Transformer blocks to perturb for STG. Recommended: `[29]` (single block) |
| `stg_mode` | STG mode: `"stg_av"` perturbs both audio and video, `"stg_v"` perturbs video only |
| `generate_audio` | Whether to generate audio in validation samples |
| `generate_video` | Whether to generate video in validation samples. Set to `false` for V2A (video-to-audio) validation. Default: `true` |
| `skip_initial_validation`| Skip validation video sampling at step 0 (beginning of training) |
#### ValidationSample
Each sample in the `samples` list has:
| Field | Description |
|--------------|-------------------------------------------------------------------------------------------------|
| `prompt` | Text prompt for this validation sample. |
| `conditions` | List of validation conditions (see types below). Empty list = text-only generation. |
| `video_dims` | Optional per-sample override for `(width, height, frames)`. Inherits from `ValidationConfig` if not set. |
| `seed` | Optional per-sample override for random seed. Inherits from `ValidationConfig` if not set. |
#### Validation Condition Types
| Type | Parameters | Description |
|------------------|------------------------------------------------------------|-------------------------------------------------------------------------|
| `first_frame` | `image_or_video` (path) | Use the first frame of the image/video as conditioning. |
| `prefix` | `video` or `audio` (path), optional `num_frames`/`duration`| Use a video/audio clip as temporal prefix (for extension forward). |
| `suffix` | `video` or `audio` (path), optional `num_frames`/`duration`| Use a video/audio clip as temporal suffix (for extension backward). |
| `spatial_crop` | `video` (path), `spatial_region` (y1, x1, y2, x2) | Provide spatial context for outpainting. Video only. |
| `mask` | `video` or `audio` (path), `mask` (path) | Mask-based inpainting with a binary mask file. |
| `reference` | `video` or `audio` (path), optional video-reference `downscale_factor`, `temporal_scale_factor`, `include_in_output` | IC-LoRA style reference conditioning. |
| `video_to_audio` | `video` (path) | Freeze video, generate audio. For Foley/V2A tasks. |
| `audio_to_video` | `audio` (path) | Freeze audio, generate video. For audio-driven generation. |
For video `reference` validation conditions, `downscale_factor` is the spatial reference scale and
`temporal_scale_factor` is the temporal reference scale. Set both to match the factors used when
preprocessing video reference latents for training; validation media is encoded on the fly and cannot infer
those factors from the training dataset.
> [!NOTE]
> The legacy fields `prompts`, `images`, and `reference_videos` are deprecated but auto-converted to `samples`
> internally. New configs should use the `samples` format.
### CheckpointsConfig
@@ -298,6 +371,8 @@ checkpoints:
interval: 250 # Steps between checkpoint saves (null = disabled)
keep_last_n: 3 # Number of recent checkpoints to retain
precision: bfloat16 # Precision for saved weights (bfloat16 or float32)
no_resume: false # Ignore saved state, start from step 0
save_training_state: "minimal" # "full", "minimal", or "off"
```
**Key parameters:**
@@ -307,6 +382,8 @@ checkpoints:
| `interval` | Steps between intermediate checkpoint saves (set to `null` to disable) |
| `keep_last_n` | Number of most recent checkpoints to keep (-1 = keep all) |
| `precision` | Precision for saved checkpoint weights: `"bfloat16"` (default) or `"float32"` |
| `no_resume` | When `true`, ignore saved training state and start from step 0. Model weights from `load_checkpoint` are still loaded. |
| `save_training_state` | Save training state for resume: `"full"` (optimizer + scheduler + RNG), `"minimal"` (scheduler + RNG only, sufficient for LoRA), `"off"` (no resume). |
### HubConfig
@@ -364,6 +441,20 @@ flow_matching:
| `timestep_sampling_mode` | Sampling strategy: `"uniform"` or `"shifted_logit_normal"` |
| `timestep_sampling_params` | Additional parameters for the sampling strategy |
### General Configuration
Top-level settings for the training run.
```yaml
seed: 42 # Random seed for reproducibility
output_dir: "outputs/my_training_run" # Directory for outputs (checkpoints, validation videos, logs)
```
| Parameter | Description |
|--------------|----------------------------------------------------------|
| `seed` | Random seed for reproducibility (default: `42`) |
| `output_dir` | Directory to save outputs (default: `"outputs"`) |
## 🚀 Next Steps
Once you've configured your training parameters: