indigo
110adc781e
Port mechanism 2 of SCAIL-2 (arXiv:2606.10804) to LTX-2: extra per-token in-context conditioning channels (1 environment switch + K=6 character binding slots) concatenated onto the latent before the first projection. Faithful temporal encoding for LTX's VAE (temporal factor 8): each latent frame stacks its 8 pixel sub-frames along the channel dim, giving 8*(K+1)=56 channels (vs the paper's 4*(K+1)=28 on Wan 2.1). Plumbing (backward compatible; cond_channels=None leaves every existing pipeline unchanged): - LatentState/Modality gain an optional cond_channels field (patchified [B,T,C]). - LTXModel(mask_conditioning_channels=0) config-gates a widened patchify_proj; TransformerArgsPreprocessor concatenates cond_channels (or zero-pads) before it. - tools.clear_conditioning trims it; token-appending conditioning items (reference video/audio, driving, keyframe) extend it via extend_cond_channels. New: - VideoConditionByMaskChannels: encodes (K+1) pixel masks -> 56 channels, written onto the trailing driving tokens (noisy target stays all-zero, per the paper). - widen_patchify_proj_for_mask_channels: zero-init checkpoint surgery so a converted model reproduces the base output exactly until finetuned. Verified (CPU, random weights): backward compat, zero-init widened forward is bit-identical to baseline for any cond_channels, and the driving+mask pipeline forwards without crashing with correct placement/clipping. Visual quality requires Phase 3 finetuning; Replacement-mode z_ref height shift still deferred. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
3.7 KiB
3.7 KiB
SCAIL-2 → LTX-2 移植任務追蹤
狀態圖例：✅ 完成 ｜ 🔄 進行中 ｜ ⬜ 未開始 ｜ ⏸️ 暫緩
相關計畫見 plan.md。
Phase 1 — Driving 串接 item（推論期 PoC）
| # | 任務 | 狀態 | 產出 / 備註 |
|---|---|---|---|
| 1.1 | 新增 VideoConditionByDrivingLatent + DrivingMode |
✅ | packages/ltx-core/src/ltx_core/conditioning/types/driving_video_cond.py，以 reference_video_cond.py 為藍本，實作 ΔW width 偏移、時間對齊 target、max_pos 防呆、token 數防呆 |
| 1.2 | 匯出新 conditioning 類別 | ✅ | conditioning/types/__init__.py、conditioning/__init__.py |
| 1.3 | 免-GPU 資料流驗證腳本 | ✅ | 序列長度、ΔW 不重疊、時間對齊、frozen denoise_mask、attention_mask=None、clear_conditioning 剝除、超界/token 數防呆 — 全通過；ruff clean |
| 1.4 | （選配）端到端 smoke run | ✅ | 本機 CPU-only、無 checkpoint，改用小型真實 LTXModel（隨機權重）跑 pipeline 實走路徑：create_noised_state（含 driving cond）→ modality_from_latent_state → 真 transformer forward（seq 160=target 80+driving 80）→ clear_conditioning（→80）。ANIMATION/REPLACEMENT 皆通過：不 crash、輸出 finite、driving frozen 且正確剝除。僅驗證整合，不評估畫質 |
Phase 2 — In-context mask channel（checkpoint 手術）
決策：mask channel 時間編碼採忠實堆疊——每 latent 幀對應 8 個 pixel 子幀沿 channel 堆疊，
8×(K+1)=56channel（LTX 時間因子 8，K=6）。非 Wan 的 4×(K+1)=28。
| # | 任務 | 狀態 | 備註 |
|---|---|---|---|
| 2.1 | patchify_proj 加寬（config-gated） |
✅ | LTXModel(mask_conditioning_channels=0)；>0 時 patchify_proj=Linear(in+mask, inner)。預設不變 |
| 2.2 | LatentState/Modality 帶 cond_channels 欄位 |
✅ | types.py、modality.py 加 cond_channels: Tensor|None=None（patchified [B,T,C]）；tools.clear_conditioning 裁切；helpers.modality_from_latent_state 帶入 |
| 2.3 | 投影前 concat/zero-pad cond_channels | ✅ | transformer_args.py _apply_patchify_proj：寬度不足時用 cond_channels 補齊，None 則補零 |
| 2.4 | checkpoint zero-init 加寬轉換 | ✅ | mask_channels_checkpoint.py widen_patchify_proj_for_mask_channels：尾端補零欄，載入舊權重行為不變 |
| 2.5 | VideoConditionByMaskChannels（56ch） |
✅ | mask_channels_cond.py：1 環境開關 + K=6 綁定槽 → 空間下採樣 + 時間 8× 堆疊(causal 首幀複製)=56ch，寫入尾端 driving token，target 保持零 |
| 2.6 | token-append conditioning 延伸 cond_channels | ✅ | cond_channels.py extend_cond_channels；reference_video/reference_audio/driving/keyframe 皆接上 |
| 2.7 | Phase 2 驗證（免訓練） | ✅ | verify_mask_channels.py：mask=0 向後相容；zero-init 加寬 forward == baseline（任意 cond_channels）；driving+mask pipeline forward 不 crash、cond_channels 形狀/placement 正確、clear_conditioning 剝除 |
| 2.8 | Replacement Mode z_ref 高度位移 ΔH_ref | ⬜ | 仍暫緩（Phase 1 divergence，需獨立 reference token group） |
Phase 3 — 訓練整合（ltx-trainer）
| # | 任務 | 狀態 | 備註 |
|---|---|---|---|
| 3.1 | dataset 產出 (target, driving, mask) | ⬜ | |
| 3.2 | 接上 conditioning + 微調 loss / 凍結策略 | ⬜ |
Phase 4 — Pipeline + CLI
| # | 任務 | 狀態 | 備註 |
|---|---|---|---|
| 4.1 | scail_animation.py pipeline + arg parser |
⬜ | 仿 lipdub.py |
決議紀錄
- 範圍：先只做 Phase 1（推論期 PoC）。Phase 2+ 待 Phase 1 驗證後再討論。
- 架構前提：SCAIL-2 建構於 Wan 2.1，本移植為跨架構移植。