SCAIL-2 character animation port (Phases 1-4) #1

2026-07-09T19:48:51Z

indigo commented

2026-07-09 19:48:51 +00:00

Port the SCAIL-2 (arXiv:2606.10804) end-to-end character-animation method to LTX-2. SCAIL-2 was built on Wan 2.1, so this is a cross-architecture port. All four phases are implemented and pass CPU-level plumbing/integration verification; real training and end-to-end inference still need a GPU + a SCAIL-trained checkpoint.

Phases

Phase 1 — Driving-latent concatenation + mode-specific RoPE (baa6646)

VideoConditionByDrivingLatent + DrivingMode in ltx-core: concatenate the driving latent into the DiT token sequence with a RoPE width offset (ΔW), time-aligned to the target, with wrap/shape guards. No weight changes.

Phase 2 — In-context mask channels + zero-init checkpoint surgery (110adc7)

Optional cond_channels on LatentState/Modality, concatenated onto the latent before a config-gated widened patchify_proj.
VideoConditionByMaskChannels: (K+1) semantic masks → 8*(K+1)=56 channels (faithful LTX temporal-factor-8 stacking) written onto the driving tokens; target stays zero.
widen_patchify_proj_for_mask_channels zero-init surgery — a converted checkpoint reproduces the base output exactly until finetuned.

Phase 3 — Training integration (e03cc62)

DrivingConditionConfig + MaskChannelsConditionConfig in the trainer's FlexibleStrategy; load_transformer(mask_conditioning_channels=) widens at load; LoRA mode additionally unfreezes patchify_proj (the new columns can't be reached by LoRA).
configs/scail_animation_lora.yaml + docs.

Phase 4 — Inference pipeline + CLI (06c0870)

ScailAnimationPipeline + build_scail_conditionings + scail_animation_arg_parser.
LTXModelConfigurator reads mask_conditioning_channels, so a SCAIL-trained checkpoint is self-describing (auto-widens on load — no runtime wrapper).

Fix — ltx-trainer ruff config (a598f89)

[tool.ruff] target-version was accidentally the package version "1.1.7", which made ruff skip the whole package. Set to py310; fixed the hidden too-many-branches finding it then surfaced.

Verification

Each phase has a CPU verification (no GPU/checkpoint): sequence growth, ΔW non-overlap, attention/denoise masks, zero-init equivalence (bit-identical output), training prepare_training_inputs + loss step, CLI round-trip, and self-describing config. ruff check passes across the changed packages.

Not included (data/GPU work)

Dataset preprocessing (driving latents + semantic masks, needs a segmentation model), validation-runner wiring, real training/inference on GPU, and the Replacement-mode z_ref height shift (needs a separate reference token group).

🤖 Generated with Claude Code

Port the SCAIL-2 (arXiv:2606.10804) end-to-end character-animation method to LTX-2. SCAIL-2 was built on Wan 2.1, so this is a cross-architecture port. All four phases are implemented and pass CPU-level plumbing/integration verification; real training and end-to-end inference still need a GPU + a SCAIL-trained checkpoint. ## Phases **Phase 1 — Driving-latent concatenation + mode-specific RoPE** (`baa6646`) - `VideoConditionByDrivingLatent` + `DrivingMode` in ltx-core: concatenate the driving latent into the DiT token sequence with a RoPE width offset (ΔW), time-aligned to the target, with wrap/shape guards. No weight changes. **Phase 2 — In-context mask channels + zero-init checkpoint surgery** (`110adc7`) - Optional `cond_channels` on `LatentState`/`Modality`, concatenated onto the latent before a config-gated widened `patchify_proj`. - `VideoConditionByMaskChannels`: (K+1) semantic masks → `8*(K+1)=56` channels (faithful LTX temporal-factor-8 stacking) written onto the driving tokens; target stays zero. - `widen_patchify_proj_for_mask_channels` zero-init surgery — a converted checkpoint reproduces the base output exactly until finetuned. **Phase 3 — Training integration** (`e03cc62`) - `DrivingConditionConfig` + `MaskChannelsConditionConfig` in the trainer's `FlexibleStrategy`; `load_transformer(mask_conditioning_channels=)` widens at load; LoRA mode additionally unfreezes `patchify_proj` (the new columns can't be reached by LoRA). - `configs/scail_animation_lora.yaml` + docs. **Phase 4 — Inference pipeline + CLI** (`06c0870`) - `ScailAnimationPipeline` + `build_scail_conditionings` + `scail_animation_arg_parser`. - `LTXModelConfigurator` reads `mask_conditioning_channels`, so a SCAIL-trained checkpoint is self-describing (auto-widens on load — no runtime wrapper). **Fix — ltx-trainer ruff config** (`a598f89`) - `[tool.ruff] target-version` was accidentally the package version `"1.1.7"`, which made ruff skip the whole package. Set to `py310`; fixed the hidden too-many-branches finding it then surfaced. ## Verification Each phase has a CPU verification (no GPU/checkpoint): sequence growth, ΔW non-overlap, attention/denoise masks, zero-init equivalence (bit-identical output), training `prepare_training_inputs` + loss step, CLI round-trip, and self-describing config. `ruff check` passes across the changed packages. ## Not included (data/GPU work) Dataset preprocessing (driving latents + semantic masks, needs a segmentation model), validation-runner wiring, real training/inference on GPU, and the Replacement-mode `z_ref` height shift (needs a separate reference token group). 🤖 Generated with [Claude Code](https://claude.com/claude-code)

indigo added 5 commits 2026-07-09 19:48:54 +00:00

Add SCAIL-2 driving-latent conditioning (Phase 1, inference PoC) baa6646fd1

Port mechanisms 1+3 of SCAIL-2 (arXiv:2606.10804) to LTX-2: concatenate a
driving video latent directly into the DiT token sequence with a width-axis
RoPE offset (ΔW) so driving coords stay detached from the target video.

- New VideoConditionByDrivingLatent + DrivingMode in ltx-core conditioning,
  modeled on VideoConditionByReferenceLatent (patchify -> positions -> append
  -> attention mask). Applies ΔW width shift, aligns time to the target, and
  guards against RoPE wrap (max_pos) and target/driving shape mismatch.
- Export both from conditioning packages.
- docs/plan.md and docs/tasks.md track the phased port.

Inference-only: no weight changes. Mechanism 2 (in-context mask channels,
patchify_proj widening) and training are deferred to Phase 2+. Validated via
plumbing checks and a real (random-weight) transformer forward smoke run;
visual quality is not validated (requires Phase 2/3 finetuning).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Add SCAIL-2 in-context mask channels (Phase 2, plumbing + zero-init surgery) 110adc781e

Port mechanism 2 of SCAIL-2 (arXiv:2606.10804) to LTX-2: extra per-token
in-context conditioning channels (1 environment switch + K=6 character binding
slots) concatenated onto the latent before the first projection.

Faithful temporal encoding for LTX's VAE (temporal factor 8): each latent frame
stacks its 8 pixel sub-frames along the channel dim, giving 8*(K+1)=56 channels
(vs the paper's 4*(K+1)=28 on Wan 2.1).

Plumbing (backward compatible; cond_channels=None leaves every existing pipeline
unchanged):
- LatentState/Modality gain an optional cond_channels field (patchified [B,T,C]).
- LTXModel(mask_conditioning_channels=0) config-gates a widened patchify_proj;
  TransformerArgsPreprocessor concatenates cond_channels (or zero-pads) before it.
- tools.clear_conditioning trims it; token-appending conditioning items
  (reference video/audio, driving, keyframe) extend it via extend_cond_channels.

New:
- VideoConditionByMaskChannels: encodes (K+1) pixel masks -> 56 channels, written
  onto the trailing driving tokens (noisy target stays all-zero, per the paper).
- widen_patchify_proj_for_mask_channels: zero-init checkpoint surgery so a
  converted model reproduces the base output exactly until finetuned.

Verified (CPU, random weights): backward compat, zero-init widened forward is
bit-identical to baseline for any cond_channels, and the driving+mask pipeline
forwards without crashing with correct placement/clipping. Visual quality
requires Phase 3 finetuning; Replacement-mode z_ref height shift still deferred.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Add SCAIL-2 training integration (Phase 3, ltx-trainer FlexibleStrategy) e03cc62548

Wire SCAIL-2 driving + in-context mask conditioning into the trainer via the
unified FlexibleStrategy, so the widened patchify_proj (Phase 2) can be trained.

- flexible.py: new DrivingConditionConfig (driving-latent concat with a RoPE
  width offset ΔW) and MaskChannelsConditionConfig (semantic masks -> per-token
  channels), added to the condition union and get_data_sources. Driving is
  prepended (cond-first, target stays at the tail for loss slicing); mask
  channels are written onto the driving tokens via Modality.cond_channels,
  reusing ltx-core encode_mask_channels. The noisy target keeps a zero mask.
- model_loader.load_transformer gains mask_conditioning_channels, widening the
  video patchify_proj with zero-init columns via a new live-module helper
  (widen_module_patchify_proj_for_mask_channels). ModelConfig exposes the field.
- trainer unfreezes patchify_proj in LoRA mode when mask channels are active
  (the new input columns are new base params LoRA cannot reach).
- configs/scail_animation_lora.yaml plus README / training-modes table rows.

Verified on CPU (verify_phase3_trainer.py): config round-trips, prepare_training
_inputs builds cond_channels [B,T,56] with the mask on the driving tokens, a tiny
widened model forwards and compute_loss returns a finite [B] loss, and the widen
helper is output-preserving at zero init. Real training needs Linux+GPU+checkpoint;
dataset preprocessing (driving latents + semantic masks) and validation-runner
wiring are left for later.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Add SCAIL-2 animation inference pipeline (Phase 4, ltx-pipelines) 06c0870bbb

Two-stage distilled inference pipeline that animates a character from a driving
video, wiring the Phase 1-3 SCAIL-2 conditioning into a runnable CLI.

- scail_animation.py: ScailAnimationPipeline (mirrors distilled.py) plus a pure,
  testable build_scail_conditionings that assembles VideoConditionByDrivingLatent
  + VideoConditionByMaskChannels (driving appended, mask on the trailing driving
  tokens), and a load_masks helper. main() + scail_animation_arg_parser add
  --driving-video / --mask-path / --mode / --driving-strength on top of the
  standard two-stage distilled parser.
- LTXModelConfigurator now reads config `mask_conditioning_channels`, so a
  SCAIL-trained checkpoint whose config declares it builds the widened
  patchify_proj automatically — no runtime widening wrapper needed at inference.
- CLAUDE.md pipeline table row.

Verified on CPU (verify_phase4_pipeline.py): the module imports, the CLI parses
the SCAIL flags, build_scail_conditionings grows the sequence and places the mask
channels on the driving tokens (target stays zero), driving-only leaves
cond_channels None, and the configurator honors mask_conditioning_channels
(patchify_proj widened from config). End-to-end runs still need a GPU and a
SCAIL-trained checkpoint.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Fix ltx-trainer ruff target-version and a hidden lint finding a598f89d99

The [tool.ruff] target-version was accidentally set to the package version
"1.1.7", which made ruff fail to parse the whole package's pyproject and
silently skip linting. Set it to "py310" to match requires-python >=3.10.

With ruff working again it flagged a too-many-branches finding in the Phase 3
SCAIL wiring: extract the driving + mask-channel loops from _process_modality
into a new _apply_scail_conditions helper. Behavior is unchanged (Phase 3 CPU
verification still passes); full `ruff check .` on the trainer now passes.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

indigo added 1 commit 2026-07-12 01:37:36 +00:00

Add Modal test environment for SCAIL-2 training/inference b69eedbd54

A free-tier-friendly Modal app to validate the SCAIL-2 code path on real
Linux/GPU without a local GPU or the 19B checkpoint.

- modal/checks.py: device-aware (CPU/GPU auto) consolidation of the Phase 1-4
  plumbing checks using tiny random-init models + synthetic data (no checkpoint,
  no dataset): driving concat, zero-init patchify_proj widening (output-preserving),
  a FlexibleStrategy driving+mask training step + loss, and inference conditioning
  assembly. Passes on CPU locally.
- modal/app.py: builds the workspace via `uv sync` (skips CUDA-only ltx-kernels;
  attention falls back to SDPA). Functions: verify (CPU, ~free), smoke (T4, cents),
  train (A10G/A100, paid — runs the real trainer against a checkpoint + data on the
  scail-data Volume).
- modal/README.md: free-tier setup (modal setup), run commands, cost table, the
  scale-up path, and the expected preprocessed dataset layout.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

indigo added 1 commit 2026-07-13 01:49:30 +00:00

Add char_masks preprocessing script for SCAIL-2 training (task 3.7) 5594d49c76

scripts/process_char_masks.py turns per-sample character label-map videos/images
(integer pixel labels: 0 = environment, 1..K = characters -> binding slots) into
the pixel-space semantic-mask tensors the SCAIL training/inference path consumes:
{"mask": [K+1, F_pix, H_pix, W_pix]} (ch0 = environment switch, ch1..K = slots).

- Aligns to the target video's latent grid read from the saved latent metadata
  (F_pix=(F-1)*8+1, H*32, W*32), so char_masks/ lines up file-for-file with
  latents/ / driving_latents/ for PrecomputedDataset.
- Nearest-neighbour resize so integer labels are never blended; labels > K are
  dropped with a warning; ch0 filled uniformly with --environment-switch.
- Reuses process_videos.py helpers (naming, atomic save, VAE factors) and matches
  its typer CLI conventions.

Verified on CPU: a synthetic 2-character label map (plus an out-of-range id)
produces mask (7,17,128,128) with ch0 uniform, slots placed correctly, id>K
dropped, and feeds encode_mask_channels to the 8*(K+1)=56 channels. README +
docs/tasks.md 3.7 updated (upstream label-map generation via SAM/tracking is
dataset-specific and still out of scope).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

This pull request can be merged automatically.

You are not authorized to merge this pull request.

View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.

git fetch -u origin scail2-driving-cond-phase1:scail2-driving-cond-phase1

git checkout scail2-driving-cond-phase1

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: indigo/LTX-2#1