Add SCAIL-2 training integration (Phase 3, ltx-trainer FlexibleStrategy)

Wire SCAIL-2 driving + in-context mask conditioning into the trainer via the
unified FlexibleStrategy, so the widened patchify_proj (Phase 2) can be trained.

- flexible.py: new DrivingConditionConfig (driving-latent concat with a RoPE
  width offset ΔW) and MaskChannelsConditionConfig (semantic masks -> per-token
  channels), added to the condition union and get_data_sources. Driving is
  prepended (cond-first, target stays at the tail for loss slicing); mask
  channels are written onto the driving tokens via Modality.cond_channels,
  reusing ltx-core encode_mask_channels. The noisy target keeps a zero mask.
- model_loader.load_transformer gains mask_conditioning_channels, widening the
  video patchify_proj with zero-init columns via a new live-module helper
  (widen_module_patchify_proj_for_mask_channels). ModelConfig exposes the field.
- trainer unfreezes patchify_proj in LoRA mode when mask channels are active
  (the new input columns are new base params LoRA cannot reach).
- configs/scail_animation_lora.yaml plus README / training-modes table rows.

Verified on CPU (verify_phase3_trainer.py): config round-trips, prepare_training
_inputs builds cond_channels [B,T,56] with the mask on the driving tokens, a tiny
widened model forwards and compute_loss returns a finite [B] loss, and the widen
helper is output-preserving at zero init. Real training needs Linux+GPU+checkpoint;
dataset preprocessing (driving latents + semantic masks) and validation-runner
wiring are left for later.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
2026-07-09 10:09:40 +08:00
parent 110adc781e
commit e03cc62548
10 changed files with 398 additions and 7 deletions
+6 -2
View File
@@ -45,8 +45,12 @@
- `VideoConditionByMaskChannels`(K+1) 語意 mask → 空間下採樣 + 時間 8× 堆疊 → 寫入尾端 driving tokentarget 保持零,符合論文)。
- 驗證通過(`verify_mask_channels.py`):向後相容、zero-init 加寬 forward == baseline、mask pipeline 不 crash。**僅驗證 plumbing,畫質需 Phase 3 微調。**
### Phase 3 — 訓練整合(ltx-trainer⬜ 未開始
- dataset 產出 (target, driving, mask),接上 Phase 1/2 conditioning,設微調 loss 與凍結策略
### Phase 3 — 訓練整合(ltx-trainer✅ 已完成(程式碼路徑;實訓需 GPU)
- **決策**:完整整合到 `FlexibleStrategy`;訓練 = LoRA + 解凍 `patchify_proj`(新 mask 欄位無法純 LoRA 訓練)
-`DrivingConditionConfig` + `MaskChannelsConditionConfig``flexible.py`);`_apply_driving_condition`(cond-first concat + ΔW) + `_build_mask_channels`(重用 `encode_mask_channels`) → `Modality.cond_channels`
- `load_transformer(mask_conditioning_channels=)``widen_module_patchify_proj_for_mask_channels` 加寬;`ModelConfig.mask_conditioning_channels`trainer 在 LoRA 模式解凍 patchify_proj。
- `configs/scail_animation_lora.yaml` + docs。CPU 單元驗證通過(`verify_phase3_trainer.py`)。
- **本機無 GPU/Linux/checkpoint → 未跑實機訓練**dataset 前處理(driving latents + 語意 mask)與 validation runner 接線未做。
### Phase 4 — Pipeline + CLI 包裝 ⬜ 未開始
- 仿 `lipdub.py``scail_animation.py` pipeline + arg parser。