github-actions[bot]
161 lines
7.7 KiB
Markdown
161 lines
7.7 KiB
Markdown
# Quick Start Guide
|
|
|
|
Get up and running with LTX-2 training in just a few steps!
|
|
|
|
## 📋 Prerequisites
|
|
|
|
Before you begin, ensure you have:
|
|
|
|
1. **LTX-2 Model Checkpoint** - A local `.safetensors` file containing the LTX-2 model weights.
|
|
Download `ltx-2.3-22b-dev.safetensors` from: [HuggingFace Hub](https://huggingface.co/Lightricks/LTX-2.3)
|
|
The trainer supports LTX-2 and LTX-2.3 checkpoints through the same configuration API; version-specific components
|
|
are detected from the checkpoint.
|
|
2. **Gemma Text Encoder** - A local directory containing the Gemma model (required for LTX-2).
|
|
Download from: [HuggingFace Hub](https://huggingface.co/google/gemma-3-12b-it-qat-q4_0-unquantized/)
|
|
3. **Linux with CUDA** - The trainer requires `triton` which is Linux-only; CUDA 13+ is recommended
|
|
4. **GPU with sufficient VRAM** - 80GB recommended for the standard config. For GPUs with 32GB VRAM (e.g., RTX 5090),
|
|
use the [low VRAM config](../configs/t2v_lora_low_vram.yaml) which enables INT8 quantization and other
|
|
memory optimizations
|
|
|
|
## ⚡ Installation
|
|
|
|
First, install [uv](https://docs.astral.sh/uv/getting-started/installation/) if you haven't already.
|
|
Then clone the repository and install the dependencies:
|
|
|
|
```bash
|
|
git clone https://github.com/Lightricks/LTX-2
|
|
```
|
|
|
|
The `ltx-trainer` package is part of the `LTX-2` monorepo. Install the dependencies from the repository root,
|
|
then navigate to the trainer package:
|
|
|
|
```bash
|
|
# From the repository root
|
|
uv sync
|
|
cd packages/ltx-trainer
|
|
```
|
|
|
|
> [!NOTE]
|
|
> The trainer depends on [`ltx-core`](../../ltx-core/) and [`ltx-pipelines`](../../ltx-pipelines/)
|
|
> packages which are automatically installed from the monorepo.
|
|
|
|
## 🏋 Training Workflow
|
|
|
|
If you are using an agent-enabled environment with repository skills, you can ask for the
|
|
[`train-model`](../../../.claude/skills/train-model/SKILL.md) skill to run this workflow with you.
|
|
It creates a run workspace, confirms the training mode, prepares data, preprocesses latents,
|
|
launches training, and monitors the run while stopping for approval before expensive steps.
|
|
|
|
### 1. Choose a Training Mode
|
|
|
|
Start with [`t2v_lora.yaml`](../configs/t2v_lora.yaml) for a first run with videos and captions. For modes such as
|
|
IC-LoRA, inpainting, or outpainting, check [Training Modes](training-modes.md) first because your metadata needs extra
|
|
columns such as `reference_video`, `video_mask`, or `audio_mask` before preprocessing.
|
|
|
|
### 2. Prepare Your Dataset
|
|
|
|
Organize your videos and captions, then preprocess them:
|
|
|
|
```bash
|
|
# Split long videos into scenes (optional)
|
|
uv run python scripts/split_scenes.py input.mp4 scenes_output_dir/ --filter-shorter-than 5s
|
|
|
|
# Generate captions for videos (optional)
|
|
uv run python scripts/caption_videos.py scenes_output_dir/ --output dataset.json
|
|
|
|
# Preprocess the dataset (compute latents and embeddings)
|
|
uv run python scripts/process_dataset.py dataset.json \
|
|
--resolution-buckets "960x544x49" \
|
|
--model-path /path/to/ltx-2-model.safetensors \
|
|
--text-encoder-path /path/to/gemma-model
|
|
```
|
|
|
|
By default, preprocessing writes to `.precomputed/`. Use that directory as `data.preprocessed_data_root`
|
|
in your training config.
|
|
|
|
See [Dataset Preparation](dataset-preparation.md) for detailed instructions.
|
|
|
|
### 3. Configure Training
|
|
|
|
Create or modify a configuration YAML file. Start with one of the example configs:
|
|
|
|
- [`configs/t2v_lora.yaml`](../configs/t2v_lora.yaml) - Text-to-video LoRA
|
|
- [`configs/t2v_lora_low_vram.yaml`](../configs/t2v_lora_low_vram.yaml) - Same as above, tuned for ~32GB VRAM (INT8 quantization and memory optimizations)
|
|
- [`configs/v2v_ic_lora.yaml`](../configs/v2v_ic_lora.yaml) - IC-LoRA video-to-video
|
|
|
|
Key settings to update:
|
|
|
|
```yaml
|
|
model:
|
|
model_path: "/path/to/ltx-2-model.safetensors"
|
|
text_encoder_path: "/path/to/gemma-model"
|
|
|
|
data:
|
|
preprocessed_data_root: "/path/to/preprocessed/data"
|
|
|
|
output_dir: "outputs/my_training_run"
|
|
```
|
|
|
|
See [Configuration Reference](configuration-reference.md) for all available options.
|
|
|
|
### 4. Start Training
|
|
|
|
```bash
|
|
uv run python scripts/train.py configs/t2v_lora.yaml
|
|
```
|
|
|
|
For multi-GPU training:
|
|
|
|
```bash
|
|
uv run accelerate launch scripts/train.py configs/t2v_lora.yaml
|
|
```
|
|
|
|
See [Training Guide](training-guide.md) for distributed training and advanced options.
|
|
|
|
## 🎯 Training Modes
|
|
|
|
> [!TIP]
|
|
> **First time?** Start with [`t2v_lora.yaml`](../configs/t2v_lora.yaml) — it's the simplest mode
|
|
> and only requires videos with captions. You can explore other modes once you've confirmed your
|
|
> setup works.
|
|
|
|
The trainer supports several training modes:
|
|
|
|
| Mode | Description | Example Config |
|
|
|-----------------------|--------------------------------------------|-------------------------------------------------------------------|
|
|
| **Text-to-Video** | Generate video+audio from text prompts | [`t2v_lora.yaml`](../configs/t2v_lora.yaml) |
|
|
| **Image-to-Video** | Animate from a starting image | [`i2v_lora.yaml`](../configs/i2v_lora.yaml) |
|
|
| **Video Extension** | Extend videos temporally (forward/backward)| [`video_extend_lora.yaml`](../configs/video_extend_lora.yaml), [`video_suffix_lora.yaml`](../configs/video_suffix_lora.yaml) |
|
|
| **IC-LoRA (V2V)** | Video-to-video transformations | [`v2v_ic_lora.yaml`](../configs/v2v_ic_lora.yaml) |
|
|
| **Audio-to-Video** | Generate video conditioned on audio | [`a2v_lora.yaml`](../configs/a2v_lora.yaml) |
|
|
| **Video-to-Audio** | Generate audio/foley from video | [`v2a_lora.yaml`](../configs/v2a_lora.yaml) |
|
|
| **Video Inpainting** | Fill in masked regions of video | [`video_inpainting_lora.yaml`](../configs/video_inpainting_lora.yaml) |
|
|
| **Video Outpainting** | Extend video spatially | [`video_outpainting_lora.yaml`](../configs/video_outpainting_lora.yaml) |
|
|
| **Text-to-Audio** | Generate audio from text prompts | [`t2a_lora.yaml`](../configs/t2a_lora.yaml) |
|
|
| **Audio Extension** | Extend audio temporally | [`audio_extend_lora.yaml`](../configs/audio_extend_lora.yaml), [`audio_suffix_lora.yaml`](../configs/audio_suffix_lora.yaml) |
|
|
| **Audio Inpainting** | Fill in masked regions of audio | [`audio_inpainting_lora.yaml`](../configs/audio_inpainting_lora.yaml) |
|
|
| **IC-LoRA (A2A)** | Audio-to-audio transformations | [`a2a_ic_lora.yaml`](../configs/a2a_ic_lora.yaml) |
|
|
| **AV2AV IC-LoRA** | Audio+video IC-LoRA transformations | [`av2av_ic_lora.yaml`](../configs/av2av_ic_lora.yaml) |
|
|
| **Full Fine-tuning** | Full model training (any mode above) | Set `model.training_mode: "full"` |
|
|
|
|
See [Training Modes](training-modes.md) for detailed explanations of each mode.
|
|
|
|
## Next Steps
|
|
|
|
Once you've completed your first training run, you can:
|
|
|
|
- **Use your trained LoRA for inference** - The [`ltx-pipelines`](../../ltx-pipelines/) package provides
|
|
production-ready inference
|
|
pipelines for various use cases (T2V, I2V, IC-LoRA, etc.). See the package documentation for details.
|
|
- Learn more about [Dataset Preparation](dataset-preparation.md) for advanced preprocessing
|
|
- Explore different [Training Modes](training-modes.md)
|
|
- Dive deeper into [Training Configuration](configuration-reference.md)
|
|
- Understand the model architecture in [LTX-Core Documentation](../../ltx-core/README.md)
|
|
|
|
## Need Help?
|
|
|
|
If you run into issues at any step, see the [Troubleshooting Guide](troubleshooting.md) for solutions to common
|
|
problems.
|
|
|
|
Join our [Discord community](https://discord.gg/ltxplatform) for real-time help and discussion!
|