Add char_masks preprocessing script for SCAIL-2 training (task 3.7)
scripts/process_char_masks.py turns per-sample character label-map videos/images
(integer pixel labels: 0 = environment, 1..K = characters -> binding slots) into
the pixel-space semantic-mask tensors the SCAIL training/inference path consumes:
{"mask": [K+1, F_pix, H_pix, W_pix]} (ch0 = environment switch, ch1..K = slots).
- Aligns to the target video's latent grid read from the saved latent metadata
(F_pix=(F-1)*8+1, H*32, W*32), so char_masks/ lines up file-for-file with
latents/ / driving_latents/ for PrecomputedDataset.
- Nearest-neighbour resize so integer labels are never blended; labels > K are
dropped with a warning; ch0 filled uniformly with --environment-switch.
- Reuses process_videos.py helpers (naming, atomic save, VAE factors) and matches
its typer CLI conventions.
Verified on CPU: a synthetic 2-character label map (plus an out-of-range id)
produces mask (7,17,128,128) with ch0 uniform, slots placed correctly, id>K
dropped, and feeds encode_mask_channels to the 8*(K+1)=56 channels. README +
docs/tasks.md 3.7 updated (upstream label-map generation via SAM/tracking is
dataset-specific and still out of scope).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
+12
-2
@@ -86,8 +86,18 @@ char_masks/ # per-sample "mask" = [K+1, F_pix, H, W] (ch0 env switch, 1.
|
||||
|
||||
`latents/`, `conditions/`, and `driving_latents/` come from the existing
|
||||
`packages/ltx-trainer/scripts/process_dataset.py` (run it once per video set).
|
||||
`char_masks/` still needs a segmentation step (e.g. SAM) to produce the semantic
|
||||
masks — that preprocessing is not implemented yet (see `docs/tasks.md` 3.7).
|
||||
`char_masks/` is produced by `packages/ltx-trainer/scripts/process_char_masks.py`
|
||||
from per-sample **label-map** videos/images (integer pixel labels: `0` =
|
||||
environment, `1..K` = characters → binding slots):
|
||||
|
||||
```bash
|
||||
python packages/ltx-trainer/scripts/process_char_masks.py dataset.csv \
|
||||
--mask-column char_labels --latents-dir ./latents \
|
||||
--output-dir ./char_masks --num-slots 6 --main-media-column media_path
|
||||
```
|
||||
|
||||
You still need a segmentation/tracking model (e.g. SAM) to *produce* those label
|
||||
maps from raw video — that upstream step is dataset-specific and not included.
|
||||
|
||||
## Cost notes
|
||||
|
||||
|
||||
Reference in New Issue
Block a user