scripts/process_char_masks.py turns per-sample character label-map videos/images
(integer pixel labels: 0 = environment, 1..K = characters -> binding slots) into
the pixel-space semantic-mask tensors the SCAIL training/inference path consumes:
{"mask": [K+1, F_pix, H_pix, W_pix]} (ch0 = environment switch, ch1..K = slots).
- Aligns to the target video's latent grid read from the saved latent metadata
(F_pix=(F-1)*8+1, H*32, W*32), so char_masks/ lines up file-for-file with
latents/ / driving_latents/ for PrecomputedDataset.
- Nearest-neighbour resize so integer labels are never blended; labels > K are
dropped with a warning; ch0 filled uniformly with --environment-switch.
- Reuses process_videos.py helpers (naming, atomic save, VAE factors) and matches
its typer CLI conventions.
Verified on CPU: a synthetic 2-character label map (plus an out-of-range id)
produces mask (7,17,128,128) with ch0 uniform, slots placed correctly, id>K
dropped, and feeds encode_mask_channels to the 8*(K+1)=56 channels. README +
docs/tasks.md 3.7 updated (upstream label-map generation via SAM/tracking is
dataset-specific and still out of scope).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The [tool.ruff] target-version was accidentally set to the package version
"1.1.7", which made ruff fail to parse the whole package's pyproject and
silently skip linting. Set it to "py310" to match requires-python >=3.10.
With ruff working again it flagged a too-many-branches finding in the Phase 3
SCAIL wiring: extract the driving + mask-channel loops from _process_modality
into a new _apply_scail_conditions helper. Behavior is unchanged (Phase 3 CPU
verification still passes); full `ruff check .` on the trainer now passes.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Wire SCAIL-2 driving + in-context mask conditioning into the trainer via the
unified FlexibleStrategy, so the widened patchify_proj (Phase 2) can be trained.
- flexible.py: new DrivingConditionConfig (driving-latent concat with a RoPE
width offset ΔW) and MaskChannelsConditionConfig (semantic masks -> per-token
channels), added to the condition union and get_data_sources. Driving is
prepended (cond-first, target stays at the tail for loss slicing); mask
channels are written onto the driving tokens via Modality.cond_channels,
reusing ltx-core encode_mask_channels. The noisy target keeps a zero mask.
- model_loader.load_transformer gains mask_conditioning_channels, widening the
video patchify_proj with zero-init columns via a new live-module helper
(widen_module_patchify_proj_for_mask_channels). ModelConfig exposes the field.
- trainer unfreezes patchify_proj in LoRA mode when mask channels are active
(the new input columns are new base params LoRA cannot reach).
- configs/scail_animation_lora.yaml plus README / training-modes table rows.
Verified on CPU (verify_phase3_trainer.py): config round-trips, prepare_training
_inputs builds cond_channels [B,T,56] with the mask on the driving tokens, a tiny
widened model forwards and compute_loss returns a finite [B] loss, and the widen
helper is output-preserving at zero init. Real training needs Linux+GPU+checkpoint;
dataset preprocessing (driving latents + semantic masks) and validation-runner
wiring are left for later.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
indigo
github-actions[bot]
sync-bot