Automated PR - 2026-01-29

2026-01-29 18:42:17 +00:00
parent 727c43e998
commit ca1623ad2a
31 changed files with 1723 additions and 663 deletions
@@ -111,7 +111,8 @@ IC-LoRA enables a wide range of advanced video-to-video applications, such as:
 - **Colorization**: Convert grayscale reference videos into colorized outputs
 - **Restoration and enhancement**: Denoise, upscale, or restore old or degraded videos

-By providing paired reference and target videos, IC-LoRA can learn complex transformations that go beyond caption-based conditioning.
+By providing paired reference and target videos, IC-LoRA can learn complex transformations that go beyond caption-based
+conditioning.

 IC-LoRA training fundamentally differs from standard LoRA and full fine-tuning:

@@ -140,8 +141,10 @@ training_strategy:
 ### Dataset Requirements for IC-LoRA

 - Your dataset must contain **paired videos** where each target video has a corresponding reference video
- Reference and target videos must have **identical resolution and length**
- Both reference and target videos should be **preprocessed together** using the same resolution buckets
+- Reference and target videos must have the **same frame count** (length)
+- Reference videos can optionally be at **lower spatial resolution** than target videos (
+  see [Scaled Reference Conditioning](#scaled-reference-conditioning) below)
+- Both reference and target videos should be **preprocessed** before training

 **Dataset structure for IC-LoRA training:**

@@ -181,30 +184,82 @@ validation:
  reference_videos:
    - "/path/to/reference1.mp4"
    - "/path/to/reference2.mp4"
+  reference_downscale_factor: 1  # Set to match preprocessing (e.g., 2 for half resolution)
  include_reference_in_output: true  # Show reference side-by-side with output
 ```

+### Scaled Reference Conditioning
+
+For more efficient training and inference, you can use **downscaled reference videos** while keeping target videos at
+full resolution. This reduces the number of conditioning tokens, leading to:
+
+- **Faster training** due to shorter sequence lengths
+- **Faster inference** with reduced memory usage
+- **Same aspect ratio** maintained between reference and target
+
+#### How It Works
+
+When the reference video has resolution `H/n × W/n` and the target video has resolution `H × W`, the trainer
+automatically detects this scale factor `n` and adjusts the positional encodings so that the reference positions
+map to the correct locations in the target coordinate space.
+
+#### Preprocessing Datasets with Scaled References
+
+Use the `--reference-downscale-factor` option when running `process_dataset.py`:
+
+```bash
+# Process dataset with scaled reference videos (half resolution)
+uv run python scripts/process_dataset.py dataset.json \
+    --resolution-buckets 768x768x25 \
+    --model-path /path/to/ltx2.safetensors \
+    --text-encoder-path /path/to/gemma \
+    --reference-column "reference_path" \
+    --reference-downscale-factor 2
+```
+
+This will:
+
+- Process target videos at 768×768 resolution
+- Process reference videos at 384×384 resolution (768 / 2)
+- The trainer will automatically infer the scale factor from the dimension ratio
+
+**Important**: Set `reference_downscale_factor: 2` in your validation configuration to match the preprocessing:
+
+```yaml
+validation:
+  reference_downscale_factor: 2  # Must match the preprocessing factor
+  reference_videos:
+    - "/path/to/reference1.mp4"
+    - "/path/to/reference2.mp4"
+```
+
+> [!NOTE]
+> The scale factor must be a positive integer, and all dimensions must be divisible by 32.
+> Common scale factors are 1 (no scaling), 2 (half resolution), or 4 (quarter resolution).
+
 ## 📊 Training Mode Comparison

-| Aspect               | LoRA       | Audio-Video LoRA | Full Fine-tuning | IC-LoRA        |
-|----------------------|------------|------------------|------------------|----------------|
-| **Memory Usage**     | Low        | Low-Medium       | High             | Medium         |
-| **Training Speed**   | Fast       | Fast             | Slow             | Medium         |
-| **Output Size**      | 100MB-few GB (depends on rank) | 100MB-few GB (depends on rank) | Tens of GB | 100MB-few GB (depends on rank) |
-| **Flexibility**      | Medium     | Medium           | High             | Specialized    |
-| **Audio Support**    | Optional   | Yes              | Optional         | No             |
-| **Reference Videos** | No         | No               | No               | Yes (required) |
+| Aspect               | LoRA                           | Audio-Video LoRA               | Full Fine-tuning | IC-LoRA                        |
+|----------------------|--------------------------------|--------------------------------|------------------|--------------------------------|
+| **Memory Usage**     | Low                            | Low-Medium                     | High             | Medium                         |
+| **Training Speed**   | Fast                           | Fast                           | Slow             | Medium                         |
+| **Output Size**      | 100MB-few GB (depends on rank) | 100MB-few GB (depends on rank) | Tens of GB       | 100MB-few GB (depends on rank) |
+| **Flexibility**      | Medium                         | Medium                         | High             | Specialized                    |
+| **Audio Support**    | Optional                       | Yes                            | Optional         | No                             |
+| **Reference Videos** | No                             | No                             | No               | Yes (required)                 |

 ## 🎬 Using Trained Models for Inference

-After training, use the [`ltx-pipelines`](../../ltx-pipelines/) package for production inference with your trained LoRAs:
+After training, use the [`ltx-pipelines`](../../ltx-pipelines/) package for production inference with your trained
+LoRAs:

-| Training Mode | Recommended Pipeline |
-|---------------|---------------------|
+| Training Mode           | Recommended Pipeline                                  |
+|-------------------------|-------------------------------------------------------|
 | LoRA / Audio-Video LoRA | `TI2VidOneStagePipeline` or `TI2VidTwoStagesPipeline` |
-| IC-LoRA | `ICLoraPipeline` |
+| IC-LoRA                 | `ICLoraPipeline`                                      |

-All pipelines support loading custom LoRAs via the `loras` parameter. See the [`ltx-pipelines`](../../ltx-pipelines/) package
+All pipelines support loading custom LoRAs via the `loras` parameter. See the [`ltx-pipelines`](../../ltx-pipelines/)
+package
 documentation for detailed usage instructions.

 ## 🚀 Next Steps
@@ -216,6 +271,7 @@ Once you've chosen your training mode:
 - Start training with the [Training Guide](training-guide.md)

 > [!TIP]
-> Need a training mode that's not covered here? See [Implementing Custom Training Strategies](custom-training-strategies.md)
+> Need a training mode that's not covered here?
+> See [Implementing Custom Training Strategies](custom-training-strategies.md)
 > to learn how to create your own strategy for specialized use cases like video inpainting, audio-only training, or
 > custom conditioning.