Add char_masks preprocessing script for SCAIL-2 training (task 3.7)
scripts/process_char_masks.py turns per-sample character label-map videos/images
(integer pixel labels: 0 = environment, 1..K = characters -> binding slots) into
the pixel-space semantic-mask tensors the SCAIL training/inference path consumes:
{"mask": [K+1, F_pix, H_pix, W_pix]} (ch0 = environment switch, ch1..K = slots).
- Aligns to the target video's latent grid read from the saved latent metadata
(F_pix=(F-1)*8+1, H*32, W*32), so char_masks/ lines up file-for-file with
latents/ / driving_latents/ for PrecomputedDataset.
- Nearest-neighbour resize so integer labels are never blended; labels > K are
dropped with a warning; ch0 filled uniformly with --environment-switch.
- Reuses process_videos.py helpers (naming, atomic save, VAE factors) and matches
its typer CLI conventions.
Verified on CPU: a synthetic 2-character label map (plus an out-of-range id)
produces mask (7,17,128,128) with ch0 uniform, slots placed correctly, id>K
dropped, and feeds encode_mask_channels to the 8*(K+1)=56 channels. README +
docs/tasks.md 3.7 updated (upstream label-map generation via SAM/tracking is
dataset-specific and still out of scope).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
+1
-1
@@ -40,7 +40,7 @@
|
||||
| 3.4 | LoRA 模式解凍 patchify_proj | ✅ | `trainer._unfreeze_patchify_proj`:mask_channels>0 時把 video patchify_proj 設 trainable,讓新欄位隨 LoRA 一起訓 |
|
||||
| 3.5 | 範例 config + docs | ✅ | `configs/scail_animation_lora.yaml`;`configs/README.md` 與 `docs/training-modes.md` 表格 row |
|
||||
| 3.6 | CPU 單元驗證 | ✅ | `verify_phase3_trainer.py`:config round-trip、prepare_training_inputs 建 cond_channels[B,T,56]、driving 前置/mask placement、widened model forward + compute_loss finite、widen helper zero-init 等價 |
|
||||
| 3.7 | dataset 前處理(產 driving latents + 語意 mask) | ⬜ | 需 process_dataset 產出 `driving_latents/`(同 target 形狀)與 `char_masks/`(mask=[K+1,F_pix,H,W]);語意 mask 需分割模型,屬資料工程,未做 |
|
||||
| 3.7 | dataset 前處理(產 driving latents + 語意 mask) | 🟡 | `char_masks/` 前處理已做:`scripts/process_char_masks.py`(label-map 影片/圖 → `[K+1,F_pix,H,W]`,對齊 target latent,nearest 保留整數 label,ch0 環境開關)。`driving_latents/` 沿用既有 `process_videos.py`(driving 影片走 video latent 路徑,同 target 形狀)。**仍缺**:從原始影片產生 label-map 的分割/追蹤步驟(SAM 等,資料集特定,未含) |
|
||||
| 3.8 | validation runner 接 driving/mask | ⬜ | 驗證期取樣尚未接 SCAIL 條件(config 內 validation 先停用),與 Phase 4 一起 |
|
||||
| 3.9 | 實機訓練跑通 | ⬜ | 需 Linux + GPU + checkpoint,本機無法 |
|
||||
|
||||
|
||||
Reference in New Issue
Block a user