Add char_masks preprocessing script for SCAIL-2 training (task 3.7)

scripts/process_char_masks.py turns per-sample character label-map videos/images
(integer pixel labels: 0 = environment, 1..K = characters -> binding slots) into
the pixel-space semantic-mask tensors the SCAIL training/inference path consumes:
{"mask": [K+1, F_pix, H_pix, W_pix]} (ch0 = environment switch, ch1..K = slots).

- Aligns to the target video's latent grid read from the saved latent metadata
  (F_pix=(F-1)*8+1, H*32, W*32), so char_masks/ lines up file-for-file with
  latents/ / driving_latents/ for PrecomputedDataset.
- Nearest-neighbour resize so integer labels are never blended; labels > K are
  dropped with a warning; ch0 filled uniformly with --environment-switch.
- Reuses process_videos.py helpers (naming, atomic save, VAE factors) and matches
  its typer CLI conventions.

Verified on CPU: a synthetic 2-character label map (plus an out-of-range id)
produces mask (7,17,128,128) with ch0 uniform, slots placed correctly, id>K
dropped, and feeds encode_mask_channels to the 8*(K+1)=56 channels. README +
docs/tasks.md 3.7 updated (upstream label-map generation via SAM/tracking is
dataset-specific and still out of scope).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
2026-07-13 09:49:25 +08:00
parent b69eedbd54
commit 5594d49c76
3 changed files with 230 additions and 3 deletions
+12 -2
View File
@@ -86,8 +86,18 @@ char_masks/ # per-sample "mask" = [K+1, F_pix, H, W] (ch0 env switch, 1.
`latents/`, `conditions/`, and `driving_latents/` come from the existing
`packages/ltx-trainer/scripts/process_dataset.py` (run it once per video set).
`char_masks/` still needs a segmentation step (e.g. SAM) to produce the semantic
masks — that preprocessing is not implemented yet (see `docs/tasks.md` 3.7).
`char_masks/` is produced by `packages/ltx-trainer/scripts/process_char_masks.py`
from per-sample **label-map** videos/images (integer pixel labels: `0` =
environment, `1..K` = characters → binding slots):
```bash
python packages/ltx-trainer/scripts/process_char_masks.py dataset.csv \
--mask-column char_labels --latents-dir ./latents \
--output-dir ./char_masks --num-slots 6 --main-media-column media_path
```
You still need a segmentation/tracking model (e.g. SAM) to *produce* those label
maps from raw video — that upstream step is dataset-specific and not included.
## Cost notes