indigo
5594d49c76
scripts/process_char_masks.py turns per-sample character label-map videos/images
(integer pixel labels: 0 = environment, 1..K = characters -> binding slots) into
the pixel-space semantic-mask tensors the SCAIL training/inference path consumes:
{"mask": [K+1, F_pix, H_pix, W_pix]} (ch0 = environment switch, ch1..K = slots).
- Aligns to the target video's latent grid read from the saved latent metadata
(F_pix=(F-1)*8+1, H*32, W*32), so char_masks/ lines up file-for-file with
latents/ / driving_latents/ for PrecomputedDataset.
- Nearest-neighbour resize so integer labels are never blended; labels > K are
dropped with a warning; ch0 filled uniformly with --environment-switch.
- Reuses process_videos.py helpers (naming, atomic save, VAE factors) and matches
its typer CLI conventions.
Verified on CPU: a synthetic 2-character label map (plus an out-of-range id)
produces mask (7,17,128,128) with ch0 uniform, slots placed correctly, id>K
dropped, and feeds encode_mask_channels to the 8*(K+1)=56 channels. README +
docs/tasks.md 3.7 updated (upstream label-map generation via SAM/tracking is
dataset-specific and still out of scope).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
112 lines
4.4 KiB
Markdown
112 lines
4.4 KiB
Markdown
# SCAIL-2 on Modal — free testing environment
|
|
|
|
Test the SCAIL-2 LTX-2 training/inference **code path** on Modal without a local
|
|
GPU. The free path uses tiny random-init models + synthetic data (no checkpoint,
|
|
no dataset), so it validates that the SCAIL wiring runs on real Linux/GPU — not
|
|
model quality.
|
|
|
|
> ⚠️ The real 19B LTX-2 model is **not** free to train/run. The `verify`/`smoke`
|
|
> targets are near-free; `train` on a real checkpoint uses a paid GPU.
|
|
|
|
## 1. One-time setup
|
|
|
|
1. Sign up at [modal.com](https://modal.com) (the free Starter plan includes a
|
|
monthly credit allowance — enough for many `verify`/`smoke` runs).
|
|
2. Install and authenticate:
|
|
```bash
|
|
pip install modal
|
|
modal setup # opens a browser to link your account / token
|
|
```
|
|
|
|
## 2. Free / near-free checks
|
|
|
|
From the repo root:
|
|
|
|
```bash
|
|
# CPU: SCAIL-2 Phase 1-4 plumbing (driving concat, mask channels, zero-init widen,
|
|
# a FlexibleStrategy training step + loss, inference conditioning assembly).
|
|
modal run modal/app.py::verify
|
|
|
|
# Same checks on a T4 GPU (validates the CUDA path). ~cents.
|
|
modal run modal/app.py::smoke
|
|
```
|
|
|
|
The first run builds the image (installs the `ltx-core`/`ltx-pipelines`/`ltx-trainer`
|
|
workspace via `uv sync`). `ltx-kernels` (CUDA-compiled) is intentionally skipped —
|
|
attention falls back to PyTorch SDPA, so no CUDA toolchain is required.
|
|
|
|
Expected tail:
|
|
```
|
|
[OK] Phase 1 driving concat: seq 48 -> 96
|
|
[OK] Phase 2 zero-init widen: in_features=72, output preserved
|
|
[OK] Phase 3 training step: cond_channels (1, 96, 56), loss ...
|
|
[OK] Phase 4 build_scail_conditionings: [driving, mask_channels]
|
|
All SCAIL-2 checks passed on cuda # (or cpu)
|
|
```
|
|
|
|
## 3. Scaling up to real training (paid)
|
|
|
|
`train` runs `packages/ltx-trainer/scripts/train.py` against a real checkpoint.
|
|
You must supply the weights + data via the persistent `scail-data` Volume.
|
|
|
|
1. Create/populate the Volume (checkpoint, Gemma encoder, preprocessed latents):
|
|
```bash
|
|
modal volume create scail-data # if not auto-created
|
|
modal volume put scail-data /local/ltx-2-model.safetensors /model/ltx-2.safetensors
|
|
modal volume put scail-data /local/gemma /model/gemma
|
|
modal volume put scail-data /local/preprocessed /data/preprocessed
|
|
```
|
|
2. Copy `configs/scail_animation_lora.yaml`, and point its paths at the mounted
|
|
Volume (everything lands under `/data` in the container):
|
|
```yaml
|
|
model:
|
|
model_path: "/data/model/ltx-2.safetensors"
|
|
text_encoder_path: "/data/model/gemma"
|
|
mask_conditioning_channels: 56 # widens patchify_proj at load
|
|
data:
|
|
preprocessed_data_root: "/data/preprocessed"
|
|
```
|
|
(Put your edited config on the Volume too, or bake it into the repo.)
|
|
3. Launch (pick a GPU big enough for the model — the 19B needs an A100):
|
|
```bash
|
|
# edit gpu="A10G" -> "A100" in modal/app.py::train for the full model
|
|
modal run modal/app.py::train --config-rel /data/scail_animation_lora.yaml
|
|
```
|
|
|
|
### Dataset preprocessing (not yet automated for SCAIL)
|
|
|
|
The SCAIL training config expects, under `preprocessed_data_root/`:
|
|
|
|
```
|
|
latents/ # target video latents
|
|
conditions/ # text embeddings
|
|
driving_latents/ # driving video latents (same F/H/W as target)
|
|
char_masks/ # per-sample "mask" = [K+1, F_pix, H, W] (ch0 env switch, 1..K binding slots)
|
|
```
|
|
|
|
`latents/`, `conditions/`, and `driving_latents/` come from the existing
|
|
`packages/ltx-trainer/scripts/process_dataset.py` (run it once per video set).
|
|
`char_masks/` is produced by `packages/ltx-trainer/scripts/process_char_masks.py`
|
|
from per-sample **label-map** videos/images (integer pixel labels: `0` =
|
|
environment, `1..K` = characters → binding slots):
|
|
|
|
```bash
|
|
python packages/ltx-trainer/scripts/process_char_masks.py dataset.csv \
|
|
--mask-column char_labels --latents-dir ./latents \
|
|
--output-dir ./char_masks --num-slots 6 --main-media-column media_path
|
|
```
|
|
|
|
You still need a segmentation/tracking model (e.g. SAM) to *produce* those label
|
|
maps from raw video — that upstream step is dataset-specific and not included.
|
|
|
|
## Cost notes
|
|
|
|
| Target | GPU | Rough cost | Use |
|
|
|---------|-------|-----------|-----|
|
|
| `verify`| none | ~free | validate code path on CPU |
|
|
| `smoke` | T4 | cents | validate CUDA path |
|
|
| `train` | A10G/A100 | paid | real fine-tuning (needs checkpoint + data) |
|
|
|
|
Free credits are best spent on `verify`/`smoke` to catch integration issues before
|
|
committing a paid GPU to a real run. Watch usage in the Modal dashboard.
|