Add SCAIL-2 driving-latent conditioning (Phase 1, inference PoC)

Port mechanisms 1+3 of SCAIL-2 (arXiv:2606.10804) to LTX-2: concatenate a
driving video latent directly into the DiT token sequence with a width-axis
RoPE offset (ΔW) so driving coords stay detached from the target video.

- New VideoConditionByDrivingLatent + DrivingMode in ltx-core conditioning,
  modeled on VideoConditionByReferenceLatent (patchify -> positions -> append
  -> attention mask). Applies ΔW width shift, aligns time to the target, and
  guards against RoPE wrap (max_pos) and target/driving shape mismatch.
- Export both from conditioning packages.
- docs/plan.md and docs/tasks.md track the phased port.

Inference-only: no weight changes. Mechanism 2 (in-context mask channels,
patchify_proj widening) and training are deferred to Phase 2+. Validated via
plumbing checks and a real (random-weight) transformer forward smoke run;
visual quality is not validated (requires Phase 2/3 finetuning).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
2026-07-09 09:06:35 +08:00
parent 9377758131
commit baa6646fd1
5 changed files with 270 additions and 0 deletions
+42
View File
@@ -0,0 +1,42 @@
# SCAIL-2 → LTX-2 移植任務追蹤
狀態圖例:✅ 完成 | 🔄 進行中 | ⬜ 未開始 | ⏸️ 暫緩
相關計畫見 [`plan.md`](./plan.md)。
## Phase 1 — Driving 串接 item(推論期 PoC
| # | 任務 | 狀態 | 產出 / 備註 |
|---|---|---|---|
| 1.1 | 新增 `VideoConditionByDrivingLatent` + `DrivingMode` | ✅ | `packages/ltx-core/src/ltx_core/conditioning/types/driving_video_cond.py`,以 `reference_video_cond.py` 為藍本,實作 ΔW width 偏移、時間對齊 target、`max_pos` 防呆、token 數防呆 |
| 1.2 | 匯出新 conditioning 類別 | ✅ | `conditioning/types/__init__.py``conditioning/__init__.py` |
| 1.3 | 免-GPU 資料流驗證腳本 | ✅ | 序列長度、ΔW 不重疊、時間對齊、frozen denoise_mask、attention_mask=None、`clear_conditioning` 剝除、超界/token 數防呆 — 全通過;ruff clean |
| 1.4 | (選配)端到端 smoke run | ✅ | 本機 CPU-only、無 checkpoint,改用小型真實 `LTXModel`(隨機權重)跑 pipeline 實走路徑:`create_noised_state`(含 driving cond)→ `modality_from_latent_state`**真 transformer forward**seq 160=target 80+driving 80)→ `clear_conditioning`(→80)。ANIMATION/REPLACEMENT 皆通過:不 crash、輸出 finite、driving frozen 且正確剝除。**僅驗證整合,不評估畫質** |
## Phase 2 — In-context mask channelcheckpoint 手術)
| # | 任務 | 狀態 | 備註 |
|---|---|---|---|
| 2.1 | `patchify_proj` 加寬 `128 → 128+28` | ⬜ | `model.py:158` `_init_video``proj_out` 不動 |
| 2.2 | `LatentState` 帶額外 conditioning-channel 欄位 | ⬜ | 同步改 `tools.py` patchify/unpatchify/clear、`Modality` |
| 2.3 | 投影前 concat mask channel 到 `x` | ⬜ | `transformer_args.py:209` |
| 2.4 | checkpoint zero-init 轉換 script | ⬜ | 新輸入欄位 zero-init,載入舊權重行為不變 |
| 2.5 | 產生 28-channel mask 的 conditioning item | ⬜ | 環境開關 + K=6 角色綁定槽,`4(K+1)=28` |
| 2.6 | Replacement Mode z_ref 高度位移 ΔH_ref | ⬜ | Phase 1 暫緩項 |
## Phase 3 — 訓練整合(ltx-trainer
| # | 任務 | 狀態 | 備註 |
|---|---|---|---|
| 3.1 | dataset 產出 (target, driving, mask) | ⬜ | |
| 3.2 | 接上 conditioning + 微調 loss / 凍結策略 | ⬜ | |
## Phase 4 — Pipeline + CLI
| # | 任務 | 狀態 | 備註 |
|---|---|---|---|
| 4.1 | `scail_animation.py` pipeline + arg parser | ⬜ | 仿 `lipdub.py` |
## 決議紀錄
- **範圍**:先只做 Phase 1(推論期 PoC)。Phase 2+ 待 Phase 1 驗證後再討論。
- **架構前提**SCAIL-2 建構於 Wan 2.1,本移植為跨架構移植。