Add SCAIL-2 training integration (Phase 3, ltx-trainer FlexibleStrategy)
Wire SCAIL-2 driving + in-context mask conditioning into the trainer via the unified FlexibleStrategy, so the widened patchify_proj (Phase 2) can be trained. - flexible.py: new DrivingConditionConfig (driving-latent concat with a RoPE width offset ΔW) and MaskChannelsConditionConfig (semantic masks -> per-token channels), added to the condition union and get_data_sources. Driving is prepended (cond-first, target stays at the tail for loss slicing); mask channels are written onto the driving tokens via Modality.cond_channels, reusing ltx-core encode_mask_channels. The noisy target keeps a zero mask. - model_loader.load_transformer gains mask_conditioning_channels, widening the video patchify_proj with zero-init columns via a new live-module helper (widen_module_patchify_proj_for_mask_channels). ModelConfig exposes the field. - trainer unfreezes patchify_proj in LoRA mode when mask channels are active (the new input columns are new base params LoRA cannot reach). - configs/scail_animation_lora.yaml plus README / training-modes table rows. Verified on CPU (verify_phase3_trainer.py): config round-trips, prepare_training _inputs builds cond_channels [B,T,56] with the mask on the driving tokens, a tiny widened model forwards and compute_loss returns a finite [B] loss, and the widen helper is output-preserving at zero init. Real training needs Linux+GPU+checkpoint; dataset preprocessing (driving latents + semantic masks) and validation-runner wiring are left for later. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
@@ -252,6 +252,14 @@ class ModelConfig(ConfigBaseModel):
|
||||
description="Training mode - either LoRA fine-tuning or full model fine-tuning",
|
||||
)
|
||||
|
||||
mask_conditioning_channels: int = Field(
|
||||
default=0,
|
||||
ge=0,
|
||||
description="SCAIL-2 in-context mask channels. If > 0, the video patchify_proj is widened by this "
|
||||
"many zero-init input columns at load time. Use with a 'driving' + 'mask_channels' condition and, "
|
||||
"in LoRA mode, patchify_proj is additionally unfrozen so the new columns can train. 0 disables.",
|
||||
)
|
||||
|
||||
load_checkpoint: str | Path | None = Field(
|
||||
default=None,
|
||||
description="Path to a checkpoint file or directory to load from. "
|
||||
|
||||
Reference in New Issue
Block a user