Add Modal test environment for SCAIL-2 training/inference

A free-tier-friendly Modal app to validate the SCAIL-2 code path on real
Linux/GPU without a local GPU or the 19B checkpoint.

- modal/checks.py: device-aware (CPU/GPU auto) consolidation of the Phase 1-4
  plumbing checks using tiny random-init models + synthetic data (no checkpoint,
  no dataset): driving concat, zero-init patchify_proj widening (output-preserving),
  a FlexibleStrategy driving+mask training step + loss, and inference conditioning
  assembly. Passes on CPU locally.
- modal/app.py: builds the workspace via `uv sync` (skips CUDA-only ltx-kernels;
  attention falls back to SDPA). Functions: verify (CPU, ~free), smoke (T4, cents),
  train (A10G/A100, paid — runs the real trainer against a checkpoint + data on the
  scail-data Volume).
- modal/README.md: free-tier setup (modal setup), run commands, cost table, the
  scale-up path, and the expected preprocessed dataset layout.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
2026-07-12 09:37:27 +08:00
parent a598f89d99
commit b69eedbd54
3 changed files with 344 additions and 0 deletions
+101
View File
@@ -0,0 +1,101 @@
# SCAIL-2 on Modal — free testing environment
Test the SCAIL-2 LTX-2 training/inference **code path** on Modal without a local
GPU. The free path uses tiny random-init models + synthetic data (no checkpoint,
no dataset), so it validates that the SCAIL wiring runs on real Linux/GPU — not
model quality.
> ⚠️ The real 19B LTX-2 model is **not** free to train/run. The `verify`/`smoke`
> targets are near-free; `train` on a real checkpoint uses a paid GPU.
## 1. One-time setup
1. Sign up at [modal.com](https://modal.com) (the free Starter plan includes a
monthly credit allowance — enough for many `verify`/`smoke` runs).
2. Install and authenticate:
```bash
pip install modal
modal setup # opens a browser to link your account / token
```
## 2. Free / near-free checks
From the repo root:
```bash
# CPU: SCAIL-2 Phase 1-4 plumbing (driving concat, mask channels, zero-init widen,
# a FlexibleStrategy training step + loss, inference conditioning assembly).
modal run modal/app.py::verify
# Same checks on a T4 GPU (validates the CUDA path). ~cents.
modal run modal/app.py::smoke
```
The first run builds the image (installs the `ltx-core`/`ltx-pipelines`/`ltx-trainer`
workspace via `uv sync`). `ltx-kernels` (CUDA-compiled) is intentionally skipped —
attention falls back to PyTorch SDPA, so no CUDA toolchain is required.
Expected tail:
```
[OK] Phase 1 driving concat: seq 48 -> 96
[OK] Phase 2 zero-init widen: in_features=72, output preserved
[OK] Phase 3 training step: cond_channels (1, 96, 56), loss ...
[OK] Phase 4 build_scail_conditionings: [driving, mask_channels]
All SCAIL-2 checks passed on cuda # (or cpu)
```
## 3. Scaling up to real training (paid)
`train` runs `packages/ltx-trainer/scripts/train.py` against a real checkpoint.
You must supply the weights + data via the persistent `scail-data` Volume.
1. Create/populate the Volume (checkpoint, Gemma encoder, preprocessed latents):
```bash
modal volume create scail-data # if not auto-created
modal volume put scail-data /local/ltx-2-model.safetensors /model/ltx-2.safetensors
modal volume put scail-data /local/gemma /model/gemma
modal volume put scail-data /local/preprocessed /data/preprocessed
```
2. Copy `configs/scail_animation_lora.yaml`, and point its paths at the mounted
Volume (everything lands under `/data` in the container):
```yaml
model:
model_path: "/data/model/ltx-2.safetensors"
text_encoder_path: "/data/model/gemma"
mask_conditioning_channels: 56 # widens patchify_proj at load
data:
preprocessed_data_root: "/data/preprocessed"
```
(Put your edited config on the Volume too, or bake it into the repo.)
3. Launch (pick a GPU big enough for the model — the 19B needs an A100):
```bash
# edit gpu="A10G" -> "A100" in modal/app.py::train for the full model
modal run modal/app.py::train --config-rel /data/scail_animation_lora.yaml
```
### Dataset preprocessing (not yet automated for SCAIL)
The SCAIL training config expects, under `preprocessed_data_root/`:
```
latents/ # target video latents
conditions/ # text embeddings
driving_latents/ # driving video latents (same F/H/W as target)
char_masks/ # per-sample "mask" = [K+1, F_pix, H, W] (ch0 env switch, 1..K binding slots)
```
`latents/`, `conditions/`, and `driving_latents/` come from the existing
`packages/ltx-trainer/scripts/process_dataset.py` (run it once per video set).
`char_masks/` still needs a segmentation step (e.g. SAM) to produce the semantic
masks — that preprocessing is not implemented yet (see `docs/tasks.md` 3.7).
## Cost notes
| Target | GPU | Rough cost | Use |
|---------|-------|-----------|-----|
| `verify`| none | ~free | validate code path on CPU |
| `smoke` | T4 | cents | validate CUDA path |
| `train` | A10G/A100 | paid | real fine-tuning (needs checkpoint + data) |
Free credits are best spent on `verify`/`smoke` to catch integration issues before
committing a paid GPU to a real run. Watch usage in the Modal dashboard.