Add SCAIL-2 training integration (Phase 3, ltx-trainer FlexibleStrategy)
Wire SCAIL-2 driving + in-context mask conditioning into the trainer via the unified FlexibleStrategy, so the widened patchify_proj (Phase 2) can be trained. - flexible.py: new DrivingConditionConfig (driving-latent concat with a RoPE width offset ΔW) and MaskChannelsConditionConfig (semantic masks -> per-token channels), added to the condition union and get_data_sources. Driving is prepended (cond-first, target stays at the tail for loss slicing); mask channels are written onto the driving tokens via Modality.cond_channels, reusing ltx-core encode_mask_channels. The noisy target keeps a zero mask. - model_loader.load_transformer gains mask_conditioning_channels, widening the video patchify_proj with zero-init columns via a new live-module helper (widen_module_patchify_proj_for_mask_channels). ModelConfig exposes the field. - trainer unfreezes patchify_proj in LoRA mode when mask channels are active (the new input columns are new base params LoRA cannot reach). - configs/scail_animation_lora.yaml plus README / training-modes table rows. Verified on CPU (verify_phase3_trainer.py): config round-trips, prepare_training _inputs builds cond_channels [B,T,56] with the mask on the driving tokens, a tiny widened model forwards and compute_loss returns a finite [B] loss, and the widen helper is output-preserving at zero init. Real training needs Linux+GPU+checkpoint; dataset preprocessing (driving latents + semantic masks) and validation-runner wiring are left for later. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
+6
-2
@@ -45,8 +45,12 @@
|
||||
- `VideoConditionByMaskChannels`:(K+1) 語意 mask → 空間下採樣 + 時間 8× 堆疊 → 寫入尾端 driving token(target 保持零,符合論文)。
|
||||
- 驗證通過(`verify_mask_channels.py`):向後相容、zero-init 加寬 forward == baseline、mask pipeline 不 crash。**僅驗證 plumbing,畫質需 Phase 3 微調。**
|
||||
|
||||
### Phase 3 — 訓練整合(ltx-trainer)⬜ 未開始
|
||||
- dataset 產出 (target, driving, mask),接上 Phase 1/2 conditioning,設微調 loss 與凍結策略。
|
||||
### Phase 3 — 訓練整合(ltx-trainer)✅ 已完成(程式碼路徑;實訓需 GPU)
|
||||
- **決策**:完整整合到 `FlexibleStrategy`;訓練 = LoRA + 解凍 `patchify_proj`(新 mask 欄位無法純 LoRA 訓練)。
|
||||
- 新 `DrivingConditionConfig` + `MaskChannelsConditionConfig`(`flexible.py`);`_apply_driving_condition`(cond-first concat + ΔW) + `_build_mask_channels`(重用 `encode_mask_channels`) → `Modality.cond_channels`。
|
||||
- `load_transformer(mask_conditioning_channels=)` 用 `widen_module_patchify_proj_for_mask_channels` 加寬;`ModelConfig.mask_conditioning_channels`;trainer 在 LoRA 模式解凍 patchify_proj。
|
||||
- `configs/scail_animation_lora.yaml` + docs。CPU 單元驗證通過(`verify_phase3_trainer.py`)。
|
||||
- **本機無 GPU/Linux/checkpoint → 未跑實機訓練**;dataset 前處理(driving latents + 語意 mask)與 validation runner 接線未做。
|
||||
|
||||
### Phase 4 — Pipeline + CLI 包裝 ⬜ 未開始
|
||||
- 仿 `lipdub.py` 寫 `scail_animation.py` pipeline + arg parser。
|
||||
|
||||
+11
-2
@@ -30,10 +30,19 @@
|
||||
|
||||
## Phase 3 — 訓練整合(ltx-trainer)
|
||||
|
||||
> **決策**:完整整合到 `FlexibleStrategy`;訓練方式 = **LoRA + 解凍 patchify_proj**(新 mask 欄位無法純 LoRA 訓練)。**本機無 GPU/Linux/checkpoint,只做到 CPU 單元驗證**,實訓需在 GPU 機器跑。
|
||||
|
||||
| # | 任務 | 狀態 | 備註 |
|
||||
|---|---|---|---|
|
||||
| 3.1 | dataset 產出 (target, driving, mask) | ⬜ | |
|
||||
| 3.2 | 接上 conditioning + 微調 loss / 凍結策略 | ⬜ | |
|
||||
| 3.1 | 新增 Driving/MaskChannels ConditionConfig | ✅ | `flexible.py`:`DrivingConditionConfig`(latents_dir/mode/width_offset) + `MaskChannelsConditionConfig`(mask_dir/num_slots),加入 union + `get_data_sources` |
|
||||
| 3.2 | strategy 接線 driving concat + cond_channels | ✅ | `_apply_driving_condition`(cond-first concat + ΔW width 偏移) + `_build_mask_channels`(重用 `encode_mask_channels`,寫前 N driving token) → `Modality.cond_channels` |
|
||||
| 3.3 | model_loader + ModelConfig 支援加寬 | ✅ | `widen_module_patchify_proj_for_mask_channels`(live module zero-init) + `load_transformer(mask_conditioning_channels=)` + `ModelConfig.mask_conditioning_channels` |
|
||||
| 3.4 | LoRA 模式解凍 patchify_proj | ✅ | `trainer._unfreeze_patchify_proj`:mask_channels>0 時把 video patchify_proj 設 trainable,讓新欄位隨 LoRA 一起訓 |
|
||||
| 3.5 | 範例 config + docs | ✅ | `configs/scail_animation_lora.yaml`;`configs/README.md` 與 `docs/training-modes.md` 表格 row |
|
||||
| 3.6 | CPU 單元驗證 | ✅ | `verify_phase3_trainer.py`:config round-trip、prepare_training_inputs 建 cond_channels[B,T,56]、driving 前置/mask placement、widened model forward + compute_loss finite、widen helper zero-init 等價 |
|
||||
| 3.7 | dataset 前處理(產 driving latents + 語意 mask) | ⬜ | 需 process_dataset 產出 `driving_latents/`(同 target 形狀)與 `char_masks/`(mask=[K+1,F_pix,H,W]);語意 mask 需分割模型,屬資料工程,未做 |
|
||||
| 3.8 | validation runner 接 driving/mask | ⬜ | 驗證期取樣尚未接 SCAIL 條件(config 內 validation 先停用),與 Phase 4 一起 |
|
||||
| 3.9 | 實機訓練跑通 | ⬜ | 需 Linux + GPU + checkpoint,本機無法 |
|
||||
|
||||
## Phase 4 — Pipeline + CLI
|
||||
|
||||
|
||||
Reference in New Issue
Block a user