indigo
110adc781e
Port mechanism 2 of SCAIL-2 (arXiv:2606.10804) to LTX-2: extra per-token in-context conditioning channels (1 environment switch + K=6 character binding slots) concatenated onto the latent before the first projection. Faithful temporal encoding for LTX's VAE (temporal factor 8): each latent frame stacks its 8 pixel sub-frames along the channel dim, giving 8*(K+1)=56 channels (vs the paper's 4*(K+1)=28 on Wan 2.1). Plumbing (backward compatible; cond_channels=None leaves every existing pipeline unchanged): - LatentState/Modality gain an optional cond_channels field (patchified [B,T,C]). - LTXModel(mask_conditioning_channels=0) config-gates a widened patchify_proj; TransformerArgsPreprocessor concatenates cond_channels (or zero-pads) before it. - tools.clear_conditioning trims it; token-appending conditioning items (reference video/audio, driving, keyframe) extend it via extend_cond_channels. New: - VideoConditionByMaskChannels: encodes (K+1) pixel masks -> 56 channels, written onto the trailing driving tokens (noisy target stays all-zero, per the paper). - widen_patchify_proj_for_mask_channels: zero-init checkpoint surgery so a converted model reproduces the base output exactly until finetuned. Verified (CPU, random weights): backward compat, zero-init widened forward is bit-identical to baseline for any cond_channels, and the driving+mask pipeline forwards without crashing with correct placement/clipping. Visual quality requires Phase 3 finetuning; Replacement-mode z_ref height shift still deferred. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
LTX-2 Pipelines
High-level pipeline implementations for generating audio-video content with Lightricks' LTX-2 model. This package provides ready-to-use pipelines for text-to-video, image-to-video, video-to-video, audio-to-video, keyframe interpolation, and retake tasks.
Pipelines are built using building blocks from ltx-core (schedulers, guiders, noisers, patchifiers) and handle the complete inference flow including model loading, encoding, decoding, and file I/O.
Key Features:
- 🎬 Multiple Pipeline Types: Text-to-video, image-to-video, video-to-video, audio-to-video, keyframe interpolation, and retake
- ⚡ Optimized Performance: Support for FP8 transformers, gradient estimation, and memory optimization
- 🎯 Production Ready: Two-stage pipelines for best quality output
- 🔧 LoRA Support: Easy integration with trained LoRA adapters
- 📦 Self-Contained: Handles model loading, encoding, decoding, and file I/O
- 🚀 CLI Support: All pipelines can be run as command-line scripts
Quick Start
# From the repository root
uv sync --frozen
# Run a pipeline (example: two-stage text-to-video)
python -m ltx_pipelines.ti2vid_two_stages \
--checkpoint-path path/to/checkpoint.safetensors \
--distilled-lora path/to/distilled_lora.safetensors 0.8 \
--spatial-upsampler-path path/to/upsampler.safetensors \
--gemma-root path/to/gemma \
--prompt "A beautiful sunset over the ocean" \
--output-path output.mp4
See Installation & Usage for full setup, CLI modules, and shared flags.
📚 Documentation
| Topic | Description |
|---|---|
| Installation & Usage | Install, requirements, running pipelines from the CLI, common flags |
| Pipeline Selection Guide | Decision tree + feature comparison to pick the right pipeline |
| Available Pipelines | Full reference for all 11 pipelines |
| Conditioning Types | Image and video conditioning methods |
| Multimodal Guidance | CFG / STG / modality guidance parameters and tuning |
| Optimization Tips | FP8 quantization, torch.compile, gradient estimation |
| Multi-GPU Inference | Run a single generation across GPUs for latency (SP, TDP, distributed VAE, distributed Gemma) |
🔗 Related Projects
- LTX-Core - Core model implementation and inference components (schedulers, guiders, noisers, patchifiers)
- LTX-Trainer - Training and fine-tuning tools