Add SCAIL-2 training integration (Phase 3, ltx-trainer FlexibleStrategy)

Wire SCAIL-2 driving + in-context mask conditioning into the trainer via the
unified FlexibleStrategy, so the widened patchify_proj (Phase 2) can be trained.

- flexible.py: new DrivingConditionConfig (driving-latent concat with a RoPE
  width offset ΔW) and MaskChannelsConditionConfig (semantic masks -> per-token
  channels), added to the condition union and get_data_sources. Driving is
  prepended (cond-first, target stays at the tail for loss slicing); mask
  channels are written onto the driving tokens via Modality.cond_channels,
  reusing ltx-core encode_mask_channels. The noisy target keeps a zero mask.
- model_loader.load_transformer gains mask_conditioning_channels, widening the
  video patchify_proj with zero-init columns via a new live-module helper
  (widen_module_patchify_proj_for_mask_channels). ModelConfig exposes the field.
- trainer unfreezes patchify_proj in LoRA mode when mask channels are active
  (the new input columns are new base params LoRA cannot reach).
- configs/scail_animation_lora.yaml plus README / training-modes table rows.

Verified on CPU (verify_phase3_trainer.py): config round-trips, prepare_training
_inputs builds cond_channels [B,T,56] with the mask on the driving tokens, a tiny
widened model forwards and compute_loss returns a finite [B] loss, and the widen
helper is output-preserving at zero init. Real training needs Linux+GPU+checkpoint;
dataset preprocessing (driving latents + semantic masks) and validation-runner
wiring are left for later.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
2026-07-09 10:09:40 +08:00
parent 110adc781e
commit e03cc62548
10 changed files with 398 additions and 7 deletions
@@ -22,6 +22,41 @@ _VIDEO_PROJ_SUFFIX = "patchify_proj.weight"
_AUDIO_PROJ_SUFFIX = "audio_patchify_proj.weight"
def widen_module_patchify_proj_for_mask_channels(model: torch.nn.Module, mask_channels: int) -> torch.nn.Module:
"""In place, widen a live ``LTXModel``'s video ``patchify_proj`` to accept ``mask_channels`` extra inputs.
Replaces ``model.patchify_proj`` with a wider ``Linear`` whose original input columns are copied and
whose ``mask_channels`` new columns are zero (so the model reproduces its pre-widening output until
those columns are trained). Also sets ``model.mask_conditioning_channels`` for metadata/consistency.
Idempotent guard: raises if the model is already widened to a different value.
"""
if mask_channels < 0:
raise ValueError(f"mask_channels must be non-negative, got {mask_channels}")
proj = getattr(model, "patchify_proj", None)
if not isinstance(proj, torch.nn.Linear):
raise AttributeError("model has no linear 'patchify_proj' to widen")
already = int(getattr(model, "mask_conditioning_channels", 0))
if mask_channels in (0, already):
model.mask_conditioning_channels = mask_channels
return model
if already != 0:
raise ValueError(f"patchify_proj already widened for {already} mask channels, refusing to re-widen")
new_in = proj.in_features + mask_channels
wider = torch.nn.Linear(new_in, proj.out_features, bias=proj.bias is not None)
wider = wider.to(device=proj.weight.device, dtype=proj.weight.dtype)
with torch.no_grad():
wider.weight.zero_()
wider.weight[:, : proj.in_features].copy_(proj.weight)
if proj.bias is not None:
wider.bias.copy_(proj.bias)
model.patchify_proj = wider
model.mask_conditioning_channels = mask_channels
return model
def widen_patchify_proj_for_mask_channels(
state_dict: dict[str, torch.Tensor],
mask_channels: int,