Two-stage distilled inference pipeline that animates a character from a driving
video, wiring the Phase 1-3 SCAIL-2 conditioning into a runnable CLI.
- scail_animation.py: ScailAnimationPipeline (mirrors distilled.py) plus a pure,
testable build_scail_conditionings that assembles VideoConditionByDrivingLatent
+ VideoConditionByMaskChannels (driving appended, mask on the trailing driving
tokens), and a load_masks helper. main() + scail_animation_arg_parser add
--driving-video / --mask-path / --mode / --driving-strength on top of the
standard two-stage distilled parser.
- LTXModelConfigurator now reads config `mask_conditioning_channels`, so a
SCAIL-trained checkpoint whose config declares it builds the widened
patchify_proj automatically — no runtime widening wrapper needed at inference.
- CLAUDE.md pipeline table row.
Verified on CPU (verify_phase4_pipeline.py): the module imports, the CLI parses
the SCAIL flags, build_scail_conditionings grows the sequence and places the mask
channels on the driving tokens (target stays zero), driving-only leaves
cond_channels None, and the configurator honors mask_conditioning_channels
(patchify_proj widened from config). End-to-end runs still need a GPU and a
SCAIL-trained checkpoint.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Wire SCAIL-2 driving + in-context mask conditioning into the trainer via the
unified FlexibleStrategy, so the widened patchify_proj (Phase 2) can be trained.
- flexible.py: new DrivingConditionConfig (driving-latent concat with a RoPE
width offset ΔW) and MaskChannelsConditionConfig (semantic masks -> per-token
channels), added to the condition union and get_data_sources. Driving is
prepended (cond-first, target stays at the tail for loss slicing); mask
channels are written onto the driving tokens via Modality.cond_channels,
reusing ltx-core encode_mask_channels. The noisy target keeps a zero mask.
- model_loader.load_transformer gains mask_conditioning_channels, widening the
video patchify_proj with zero-init columns via a new live-module helper
(widen_module_patchify_proj_for_mask_channels). ModelConfig exposes the field.
- trainer unfreezes patchify_proj in LoRA mode when mask channels are active
(the new input columns are new base params LoRA cannot reach).
- configs/scail_animation_lora.yaml plus README / training-modes table rows.
Verified on CPU (verify_phase3_trainer.py): config round-trips, prepare_training
_inputs builds cond_channels [B,T,56] with the mask on the driving tokens, a tiny
widened model forwards and compute_loss returns a finite [B] loss, and the widen
helper is output-preserving at zero init. Real training needs Linux+GPU+checkpoint;
dataset preprocessing (driving latents + semantic masks) and validation-runner
wiring are left for later.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Port mechanism 2 of SCAIL-2 (arXiv:2606.10804) to LTX-2: extra per-token
in-context conditioning channels (1 environment switch + K=6 character binding
slots) concatenated onto the latent before the first projection.
Faithful temporal encoding for LTX's VAE (temporal factor 8): each latent frame
stacks its 8 pixel sub-frames along the channel dim, giving 8*(K+1)=56 channels
(vs the paper's 4*(K+1)=28 on Wan 2.1).
Plumbing (backward compatible; cond_channels=None leaves every existing pipeline
unchanged):
- LatentState/Modality gain an optional cond_channels field (patchified [B,T,C]).
- LTXModel(mask_conditioning_channels=0) config-gates a widened patchify_proj;
TransformerArgsPreprocessor concatenates cond_channels (or zero-pads) before it.
- tools.clear_conditioning trims it; token-appending conditioning items
(reference video/audio, driving, keyframe) extend it via extend_cond_channels.
New:
- VideoConditionByMaskChannels: encodes (K+1) pixel masks -> 56 channels, written
onto the trailing driving tokens (noisy target stays all-zero, per the paper).
- widen_patchify_proj_for_mask_channels: zero-init checkpoint surgery so a
converted model reproduces the base output exactly until finetuned.
Verified (CPU, random weights): backward compat, zero-init widened forward is
bit-identical to baseline for any cond_channels, and the driving+mask pipeline
forwards without crashing with correct placement/clipping. Visual quality
requires Phase 3 finetuning; Replacement-mode z_ref height shift still deferred.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Port mechanisms 1+3 of SCAIL-2 (arXiv:2606.10804) to LTX-2: concatenate a
driving video latent directly into the DiT token sequence with a width-axis
RoPE offset (ΔW) so driving coords stay detached from the target video.
- New VideoConditionByDrivingLatent + DrivingMode in ltx-core conditioning,
modeled on VideoConditionByReferenceLatent (patchify -> positions -> append
-> attention mask). Applies ΔW width shift, aligns time to the target, and
guards against RoPE wrap (max_pos) and target/driving shape mismatch.
- Export both from conditioning packages.
- docs/plan.md and docs/tasks.md track the phased port.
Inference-only: no weight changes. Mechanism 2 (in-context mask channels,
patchify_proj widening) and training are deferred to Phase 2+. Validated via
plumbing checks and a real (random-weight) transformer forward smoke run;
visual quality is not validated (requires Phase 2/3 finetuning).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
indigo