indigo/LTX-2
1
0
Fork 0
Code Issues Pull Requests 1 Actions Packages Projects Releases Wiki Activity

Add SCAIL-2 animation inference pipeline (Phase 4, ltx-pipelines)

Browse Source 
Two-stage distilled inference pipeline that animates a character from a driving
video, wiring the Phase 1-3 SCAIL-2 conditioning into a runnable CLI.

- scail_animation.py: ScailAnimationPipeline (mirrors distilled.py) plus a pure,
  testable build_scail_conditionings that assembles VideoConditionByDrivingLatent
  + VideoConditionByMaskChannels (driving appended, mask on the trailing driving
  tokens), and a load_masks helper. main() + scail_animation_arg_parser add
  --driving-video / --mask-path / --mode / --driving-strength on top of the
  standard two-stage distilled parser.
- LTXModelConfigurator now reads config `mask_conditioning_channels`, so a
  SCAIL-trained checkpoint whose config declares it builds the widened
  patchify_proj automatically — no runtime widening wrapper needed at inference.
- CLAUDE.md pipeline table row.

Verified on CPU (verify_phase4_pipeline.py): the module imports, the CLI parses
the SCAIL flags, build_scail_conditionings grows the sequence and places the mask
channels on the driving tokens (target stays zero), driving-only leaves
cond_channels None, and the configurator honors mask_conditioning_channels
(patchify_proj widened from config). End-to-end runs still need a GPU and a
SCAIL-trained checkpoint.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
indigo
2026-07-09 10:22:35 +08:00
parent e03cc62548
commit 06c0870bbb
6 changed files with 400 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/plan.md
+6 -2
View File
@@ -52,8 +52,12 @@
- `configs/scail_animation_lora.yaml` + docs。CPU 單元驗證通過（`verify_phase3_trainer.py`）。
- **本機無 GPU/Linux/checkpoint → 未跑實機訓練**；dataset 前處理（driving latents + 語意 mask）與 validation runner 接線未做。
### Phase 4 — Pipeline + CLI 包裝 ⬜ 未開始
- 仿 `lipdub.py` 寫 `scail_animation.py` pipeline + arg parser。
### Phase 4 — Pipeline + CLI 包裝 ✅ 已完成（程式碼路徑；實跑需 GPU）
- `LTXModelConfigurator` 讀 `mask_conditioning_channels` → SCAIL checkpoint 自描述、載入自動加寬（不需 runtime wrapper）。
- `scail_animation.py`：`ScailAnimationPipeline`（仿 `distilled.py` 兩階段）+ 可測純函式 `build_scail_conditionings`（driving + mask 條件組裝）+ `load_masks` + `main()`。
- `utils/args.py` `scail_animation_arg_parser`（`--driving-video/--mask-path/--mode/--driving-strength`）。
- CPU 驗證通過（`verify_phase4_pipeline.py`）；`ltx-pipelines/CLAUDE.md` 表格 row。
- **實機端到端需 GPU + SCAIL-trained checkpoint**（config 含 `mask_conditioning_channels` + 訓練好的 patchify_proj + LoRA）。
## Phase 1 簡化取捨（記錄，Phase 2 需回頭處理）
- (a) driving 時間座標直接複製 target 的（token-wise），故 driving 需與 target 同 F/H/W。
docs/tasks.md
+5 -1
View File
@@ -48,7 +48,11 @@
| # | 任務 | 狀態 | 備註 |
|---|---|---|---|
| 4.1 | `scail_animation.py` pipeline + arg parser | ⬜ | 仿 `lipdub.py` |
| 4.1 | configurator 讀 `mask_conditioning_channels` | ✅ | `LTXModelConfigurator` 兩分支 `config.get("mask_conditioning_channels",0)` → SCAIL checkpoint 自描述、載入自動加寬（不需 runtime widen wrapper） |
| 4.2 | `ScailAnimationPipeline` + `build_scail_conditionings` | ✅ | `scail_animation.py`：仿 `distilled.py` 兩階段，注入 SCAIL driving+mask conditioning；`build_scail_conditionings` 為可測純函式；`load_masks` helper |
| 4.3 | `scail_animation_arg_parser` + `main()` | ✅ | `utils/args.py`：在 `default_2_stage_distilled_arg_parser` 上加 `--driving-video/--mask-path/--mode/--driving-strength` |
| 4.4 | CPU 驗證 + docs | ✅ | `verify_phase4_pipeline.py`：import、CLI round-trip、conditioning assembly(driving append + mask 在尾端)、configurator 自描述加寬；`ltx-pipelines/CLAUDE.md` pipeline 表格 row |
| 4.5 | 實機端到端跑通 | ⬜ | 需 GPU + SCAIL-trained checkpoint（含 `mask_conditioning_channels` config + 訓練好的 patchify_proj + LoRA） |
## 決議紀錄
- **範圍**：先只做 Phase 1（推論期 PoC）。Phase 2+ 待 Phase 1 驗證後再討論。