LTX-2/modal/README.md

SCAIL-2 on Modal — free testing environment

Test the SCAIL-2 LTX-2 training/inference code path on Modal without a local GPU. The free path uses tiny random-init models + synthetic data (no checkpoint, no dataset), so it validates that the SCAIL wiring runs on real Linux/GPU — not model quality.

⚠️ The real 19B LTX-2 model is not free to train/run. The verify/smoke targets are near-free; train on a real checkpoint uses a paid GPU.

1. One-time setup

1. Sign up at modal.com (the free Starter plan includes a monthly credit allowance — enough for many verify/smoke runs).
2. Install and authenticate:

   pip install modal
   modal setup          # opens a browser to link your account / token
   

2. Free / near-free checks

From the repo root:

# CPU: SCAIL-2 Phase 1-4 plumbing (driving concat, mask channels, zero-init widen,
# a FlexibleStrategy training step + loss, inference conditioning assembly).
modal run modal/app.py::verify

# Same checks on a T4 GPU (validates the CUDA path). ~cents.
modal run modal/app.py::smoke

The first run builds the image (installs the ltx-core/ltx-pipelines/ltx-trainer workspace via uv sync). ltx-kernels (CUDA-compiled) is intentionally skipped — attention falls back to PyTorch SDPA, so no CUDA toolchain is required.

Expected tail:

[OK] Phase 1 driving concat: seq 48 -> 96
[OK] Phase 2 zero-init widen: in_features=72, output preserved
[OK] Phase 3 training step: cond_channels (1, 96, 56), loss ...
[OK] Phase 4 build_scail_conditionings: [driving, mask_channels]
All SCAIL-2 checks passed on cuda   # (or cpu)

3. Scaling up to real training (paid)

train runs packages/ltx-trainer/scripts/train.py against a real checkpoint. You must supply the weights + data via the persistent scail-data Volume.

1. Create/populate the Volume (checkpoint, Gemma encoder, preprocessed latents):

   modal volume create scail-data          # if not auto-created
   modal volume put scail-data /local/ltx-2-model.safetensors /model/ltx-2.safetensors
   modal volume put scail-data /local/gemma /model/gemma
   modal volume put scail-data /local/preprocessed /data/preprocessed
   

2. Copy configs/scail_animation_lora.yaml, and point its paths at the mounted Volume (everything lands under /data in the container):

   model:
     model_path: "/data/model/ltx-2.safetensors"
     text_encoder_path: "/data/model/gemma"
     mask_conditioning_channels: 56       # widens patchify_proj at load
   data:
     preprocessed_data_root: "/data/preprocessed"
   

   (Put your edited config on the Volume too, or bake it into the repo.)
3. Launch (pick a GPU big enough for the model — the 19B needs an A100):

   # edit gpu="A10G" -> "A100" in modal/app.py::train for the full model
   modal run modal/app.py::train --config-rel /data/scail_animation_lora.yaml
   

Dataset preprocessing (not yet automated for SCAIL)

The SCAIL training config expects, under preprocessed_data_root/:

latents/           # target video latents
conditions/        # text embeddings
driving_latents/   # driving video latents (same F/H/W as target)
char_masks/        # per-sample "mask" = [K+1, F_pix, H, W]  (ch0 env switch, 1..K binding slots)

latents/, conditions/, and driving_latents/ come from the existing packages/ltx-trainer/scripts/process_dataset.py (run it once per video set). char_masks/ is produced by packages/ltx-trainer/scripts/process_char_masks.py from per-sample label-map videos/images (integer pixel labels: 0 = environment, 1..K = characters → binding slots):

python packages/ltx-trainer/scripts/process_char_masks.py dataset.csv \
    --mask-column char_labels --latents-dir ./latents \
    --output-dir ./char_masks --num-slots 6 --main-media-column media_path

You still need a segmentation/tracking model (e.g. SAM) to produce those label maps from raw video — that upstream step is dataset-specific and not included.

Cost notes

Target	GPU	Rough cost	Use
verify	none	~free	validate code path on CPU
smoke	T4	cents	validate CUDA path
train	A10G/A100	paid	real fine-tuning (needs checkpoint + data)

Free credits are best spent on verify/smoke to catch integration issues before committing a paid GPU to a real run. Watch usage in the Modal dashboard.