indigo
baa6646fd1
Port mechanisms 1+3 of SCAIL-2 (arXiv:2606.10804) to LTX-2: concatenate a driving video latent directly into the DiT token sequence with a width-axis RoPE offset (ΔW) so driving coords stay detached from the target video. - New VideoConditionByDrivingLatent + DrivingMode in ltx-core conditioning, modeled on VideoConditionByReferenceLatent (patchify -> positions -> append -> attention mask). Applies ΔW width shift, aligns time to the target, and guards against RoPE wrap (max_pos) and target/driving shape mismatch. - Export both from conditioning packages. - docs/plan.md and docs/tasks.md track the phased port. Inference-only: no weight changes. Mechanism 2 (in-context mask channels, patchify_proj widening) and training are deferred to Phase 2+. Validated via plumbing checks and a real (random-weight) transformer forward smoke run; visual quality is not validated (requires Phase 2/3 finetuning). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
43 lines
2.6 KiB
Markdown
43 lines
2.6 KiB
Markdown
# SCAIL-2 → LTX-2 移植任務追蹤
|
||
|
||
狀態圖例：✅ 完成 ｜ 🔄 進行中 ｜ ⬜ 未開始 ｜ ⏸️ 暫緩
|
||
|
||
相關計畫見 [`plan.md`](./plan.md)。
|
||
|
||
## Phase 1 — Driving 串接 item（推論期 PoC）
|
||
|
||
| # | 任務 | 狀態 | 產出 / 備註 |
|
||
|---|---|---|---|
|
||
| 1.1 | 新增 `VideoConditionByDrivingLatent` + `DrivingMode` | ✅ | `packages/ltx-core/src/ltx_core/conditioning/types/driving_video_cond.py`，以 `reference_video_cond.py` 為藍本，實作 ΔW width 偏移、時間對齊 target、`max_pos` 防呆、token 數防呆 |
|
||
| 1.2 | 匯出新 conditioning 類別 | ✅ | `conditioning/types/__init__.py`、`conditioning/__init__.py` |
|
||
| 1.3 | 免-GPU 資料流驗證腳本 | ✅ | 序列長度、ΔW 不重疊、時間對齊、frozen denoise_mask、attention_mask=None、`clear_conditioning` 剝除、超界/token 數防呆 — 全通過；ruff clean |
|
||
| 1.4 | （選配）端到端 smoke run | ✅ | 本機 CPU-only、無 checkpoint，改用小型真實 `LTXModel`（隨機權重）跑 pipeline 實走路徑：`create_noised_state`（含 driving cond）→ `modality_from_latent_state` → **真 transformer forward**（seq 160=target 80+driving 80）→ `clear_conditioning`（→80）。ANIMATION/REPLACEMENT 皆通過：不 crash、輸出 finite、driving frozen 且正確剝除。**僅驗證整合，不評估畫質** |
|
||
|
||
## Phase 2 — In-context mask channel（checkpoint 手術）
|
||
|
||
| # | 任務 | 狀態 | 備註 |
|
||
|---|---|---|---|
|
||
| 2.1 | `patchify_proj` 加寬 `128 → 128+28` | ⬜ | `model.py:158` `_init_video`；`proj_out` 不動 |
|
||
| 2.2 | `LatentState` 帶額外 conditioning-channel 欄位 | ⬜ | 同步改 `tools.py` patchify/unpatchify/clear、`Modality` |
|
||
| 2.3 | 投影前 concat mask channel 到 `x` | ⬜ | `transformer_args.py:209` |
|
||
| 2.4 | checkpoint zero-init 轉換 script | ⬜ | 新輸入欄位 zero-init，載入舊權重行為不變 |
|
||
| 2.5 | 產生 28-channel mask 的 conditioning item | ⬜ | 環境開關 + K=6 角色綁定槽，`4(K+1)=28` |
|
||
| 2.6 | Replacement Mode z_ref 高度位移 ΔH_ref | ⬜ | Phase 1 暫緩項 |
|
||
|
||
## Phase 3 — 訓練整合（ltx-trainer）
|
||
|
||
| # | 任務 | 狀態 | 備註 |
|
||
|---|---|---|---|
|
||
| 3.1 | dataset 產出 (target, driving, mask) | ⬜ | |
|
||
| 3.2 | 接上 conditioning + 微調 loss / 凍結策略 | ⬜ | |
|
||
|
||
## Phase 4 — Pipeline + CLI
|
||
|
||
| # | 任務 | 狀態 | 備註 |
|
||
|---|---|---|---|
|
||
| 4.1 | `scail_animation.py` pipeline + arg parser | ⬜ | 仿 `lipdub.py` |
|
||
|
||
## 決議紀錄
|
||
- **範圍**：先只做 Phase 1（推論期 PoC）。Phase 2+ 待 Phase 1 驗證後再討論。
|
||
- **架構前提**：SCAIL-2 建構於 Wan 2.1，本移植為跨架構移植。
|