indigo/LTX-2
1
0
Fork 0
Code Issues Pull Requests 1 Actions Packages Projects Releases Wiki Activity

Automated PR - 2026-06-17

Browse Source
This commit is contained in:
github-actions[bot]
2026-06-17 14:21:07 +00:00
parent d6053703e0
commit f4b06fb977
103 changed files with 12887 additions and 3665 deletions
Download Patch File Download Diff File Expand all files Collapse all files
packages/ltx-trainer/docs/utility-scripts.md
+51 -132
View File
@@ -35,75 +35,53 @@ uv run python scripts/split_scenes.py video.mp4 scenes/ --max-scenes 50
### Automatic Video Captioning
The `scripts/caption_videos.py` script generates captions for videos (with audio) using multimodal models.
The `scripts/caption_videos.py` script generates a single, detailed combined audio-visual
caption per video as a continuous paragraph of prose. Two backends are available:
- **`qwen_omni` (default)** — Qwen3-Omni-30B-A3B-Thinking served via a local
[vLLM](https://docs.vllm.ai/) HTTP server (~1-3 s/video on H100). Highest quality, runs
fully offline once the model is downloaded.
- **`gemini_flash`** — Google Gemini (cloud, `gemini-3.5-flash`). No GPU required. Auth is
automatic: set `GEMINI_API_KEY` (or `GOOGLE_API_KEY`) for the Developer API, or just have
Google Cloud credentials available (`gcloud auth` / an attached service account) and it
uses Vertex AI with no extra setup.
**Step 1 — launch the captioner server** (`qwen_omni` only, one-time).
`scripts/serve_captioner.py` runs vLLM in an isolated environment via `uvx`, so vLLM's heavy
CUDA dependencies never touch the trainer's venv. It defaults to dynamic FP8 quantization
(~31 GiB weights, fits on 40 GB GPUs, same speed as BF16 on H100):
```bash
# Generate captions for all videos in a directory (uses Qwen2.5-Omni by default)
uv run python scripts/caption_videos.py videos_dir/ --output dataset.json
# Terminal 1 - stays running
uv run python packages/ltx-trainer/scripts/serve_captioner.py
# Use 8-bit quantization to reduce VRAM usage
uv run python scripts/caption_videos.py videos_dir/ --output dataset.json --use-8bit
# Use Gemini Flash API instead (requires API key)
uv run python scripts/caption_videos.py videos_dir/ --output dataset.json \
--captioner-type gemini_flash --api-key YOUR_API_KEY
# Use Gemini Flash with parallel workers for faster throughput
uv run python scripts/caption_videos.py videos_dir/ --output dataset.json \
--captioner-type gemini_flash --num-workers 5
# Caption without audio processing (video-only)
uv run python scripts/caption_videos.py videos_dir/ --output dataset.json --no-audio
# Force re-caption all files
uv run python scripts/caption_videos.py videos_dir/ --output dataset.json --override
# Useful variants:
# --print-cmd show the vLLM command without running it
# --quantization bf16 use BF16 instead (needs ~66 GiB free VRAM)
# --hf-home /mnt/disk override where the ~65 GB model is downloaded
```
**Key features:**
- **Audio-visual captioning**: Processes both video and audio content, including speech transcription
- **Multiple backends**:
- `qwen_omni` (default): Local Qwen2.5-Omni model - processes video + audio locally
- `gemini_flash`: Google Gemini Flash API - cloud-based, requires API key
- **Parallel captioning** (Gemini Flash only): Use `--num-workers` to run multiple API calls concurrently for faster throughput on large datasets
- **Structured output**: Captions include visual description, speech transcription, sounds, and on-screen text
- **Memory optimization**: 8-bit quantization option for limited VRAM
- **Incremental processing**: Skips already-captioned files by default; progress is saved every 5 videos
- **Multiple output formats**: JSON, JSONL, CSV, or TXT
**Caption format:**
The captioner produces structured captions with four sections:
- `[VISUAL]`: Detailed description of visual content
- `[SPEECH]`: Word-for-word transcription of spoken content
- `[SOUNDS]`: Description of music, ambient sounds, sound effects
- `[TEXT]`: Any on-screen text visible in the video
**Parallel captioning with Gemini Flash:**
When using `--captioner-type gemini_flash`, you can speed up large dataset captioning by running multiple API calls at the same time using `--num-workers` (accepts 1–10, default is 1):
**Step 2 — caption your videos.**
```bash
export GEMINI_API_KEY="your-key-here"
# Terminal 2 - default backend talks to the server above
uv run python packages/ltx-trainer/scripts/caption_videos.py videos_dir/ --output dataset.json
# Caption a large dataset with 5 workers running concurrently
uv run python scripts/caption_videos.py videos_dir/ \
--output dataset.json \
--captioner-type gemini_flash \
--num-workers 5
# Remote server: --vllm-url http://other-host:8001/v1
# Gemini (gemini-3.5-flash): --captioner-type gemini_flash (uses GEMINI_API_KEY, else gcloud/Vertex)
# Gemini, parallel calls: --captioner-type gemini_flash --num-workers 5
# Re-caption everything: --override
```
> [!NOTE]
> `--num-workers` is only supported with `gemini_flash`. Using it with `qwen_omni` or any other local model will raise an error, because local GPU models are not thread-safe.
Captioning is incremental (already-captioned files are skipped, progress saves every 5 videos)
and writes JSON, JSONL, CSV, or TXT based on the output extension.
> [!TIP]
> Keep `--num-workers` between 3–5 for most use cases. Very high values (8–10) may hit Gemini API rate limits depending on your quota tier.
Qwen3-Omni-Thinking can optionally emit a `<think>...</think>` chain-of-thought before the
caption (`--enable-thinking`). It is off by default, which is recommended for bulk captioning
(thinking is slower as it generates the reasoning trace first).
**Environment variables (for Gemini Flash):**
Set one of these to use Gemini Flash without passing `--api-key`:
- `GOOGLE_API_KEY`
- `GEMINI_API_KEY`
For Gemini, keep `--num-workers` at 3-5 (higher values may hit API rate limits).
### Dataset Preprocessing
@@ -116,13 +94,6 @@ uv run python scripts/process_dataset.py dataset.json \
--model-path /path/to/ltx-2-model.safetensors \
--text-encoder-path /path/to/gemma-model
# With audio processing
uv run python scripts/process_dataset.py dataset.json \
--resolution-buckets "960x544x49" \
--model-path /path/to/ltx-2-model.safetensors \
--text-encoder-path /path/to/gemma-model \
--with-audio
# With video decoding for verification
uv run python scripts/process_dataset.py dataset.json \
--resolution-buckets "960x544x49" \
@@ -186,6 +157,10 @@ uv run python scripts/compute_reference.py videos_dir/ --output dataset.json
> You can edit this script to generate other types of reference videos for IC-LoRA training,
> such as depth maps, segmentation masks, or any custom video transformation.
> [!NOTE]
> `compute_reference.py` writes generated references to the `reference_video` column, which
> `process_dataset.py` detects automatically.
## 🔍 Debugging and Verification Scripts
### Latents Decoding
@@ -224,73 +199,17 @@ uv run python scripts/decode_latents.py /path/to/latents/dir \
- **Debug training data**: Visualize what the model actually sees during training
- **Quality assessment**: Ensure latent encoding preserves important visual details
### Inference with Trained Models
### Inference Script
For inference with trained LoRAs, use the [`ltx-pipelines`](../../ltx-pipelines/) package which provides
production-ready pipelines:
The `scripts/inference.py` script runs inference with a trained model.
- **Text/Image-to-Video**: `TI2VidOneStagePipeline`, `TI2VidTwoStagesPipeline`
- **Distilled (fast) inference**: `DistilledPipeline`
- **IC-LoRA video-to-video**: `ICLoraPipeline`
- **Keyframe interpolation**: `KeyframeInterpolationPipeline`
> [!TIP]
> For production inference, consider using the [`ltx-pipelines`](../../ltx-pipelines/) package which provides optimized,
> feature-rich pipelines for various use cases:
> - **Text/Image-to-Video**: `TI2VidOneStagePipeline`, `TI2VidTwoStagesPipeline`
> - **Distilled (fast) inference**: `DistilledPipeline`
> - **IC-LoRA video-to-video**: `ICLoraPipeline`
> - **Keyframe interpolation**: `KeyframeInterpolationPipeline`
>
> All pipelines support loading custom LoRAs trained with this trainer.
```bash
# Text-to-video inference (with audio by default)
# By default, uses CFG scale 4.0 and STG scale 1.0 with block 29
uv run python scripts/inference.py \
--checkpoint /path/to/model.safetensors \
--text-encoder-path /path/to/gemma \
--prompt "A cat playing with a ball" \
--output output.mp4
# Video-only (skip audio generation)
uv run python scripts/inference.py \
--checkpoint /path/to/model.safetensors \
--text-encoder-path /path/to/gemma \
--prompt "A cat playing with a ball" \
--skip-audio \
--output output.mp4
# Image-to-video with conditioning image
uv run python scripts/inference.py \
--checkpoint /path/to/model.safetensors \
--text-encoder-path /path/to/gemma \
--prompt "A cat walking" \
--condition-image first_frame.png \
--output output.mp4
# Custom guidance settings
uv run python scripts/inference.py \
--checkpoint /path/to/model.safetensors \
--text-encoder-path /path/to/gemma \
--prompt "A cat playing with a ball" \
--guidance-scale 4.0 \
--stg-scale 1.0 \
--stg-blocks 29 \
--output output.mp4
# Disable STG (CFG only)
uv run python scripts/inference.py \
--checkpoint /path/to/model.safetensors \
--text-encoder-path /path/to/gemma \
--prompt "A cat playing with a ball" \
--stg-scale 0.0 \
--output output.mp4
```
**Guidance parameters:**
| Parameter | Default | Description |
|-----------|---------|-------------|
| `--guidance-scale` | 4.0 | CFG (Classifier-Free Guidance) scale |
| `--stg-scale` | 1.0 | STG (Spatio-Temporal Guidance) scale. 0.0 disables STG |
| `--stg-blocks` | 29 | Transformer block(s) to perturb for STG |
| `--stg-mode` | stg_av | `stg_av` perturbs both audio and video, `stg_v` video only |
All pipelines support loading custom LoRAs trained with this trainer.
## 🚀 Training Scripts
@@ -300,13 +219,13 @@ Use `scripts/train.py` for both single GPU and multi-GPU runs:
```bash
# Single-GPU training
uv run python scripts/train.py configs/ltx2_av_lora.yaml
uv run python scripts/train.py configs/t2v_lora.yaml
# Multi-GPU (uses your accelerate config)
uv run accelerate launch scripts/train.py configs/ltx2_av_lora.yaml
uv run accelerate launch scripts/train.py configs/t2v_lora.yaml
# Override number of processes
uv run accelerate launch --num_processes 4 scripts/train.py configs/ltx2_av_lora.yaml
uv run accelerate launch --num_processes 4 scripts/train.py configs/t2v_lora.yaml
```
For detailed usage, see the [Training Guide](training-guide.md).
@@ -316,5 +235,5 @@ For detailed usage, see the [Training Guide](training-guide.md).
- **Start with `--help`**: Always check available options for each script
- **Test on small datasets**: Verify workflows with a few files before processing large datasets
- **Use decode verification**: Always decode a few samples to verify preprocessing quality
- **Monitor VRAM usage**: Use `--use-8bit` or quantization flags when running into memory issues
- **Monitor VRAM usage**: Reach for quantization or lower-memory settings (e.g. FP8 for the captioner server) when running into memory issues
- **Keep backups**: Make copies of important dataset files before running conversion scripts