Add SCAIL-2 in-context mask channels (Phase 2, plumbing + zero-init surgery)
Port mechanism 2 of SCAIL-2 (arXiv:2606.10804) to LTX-2: extra per-token in-context conditioning channels (1 environment switch + K=6 character binding slots) concatenated onto the latent before the first projection. Faithful temporal encoding for LTX's VAE (temporal factor 8): each latent frame stacks its 8 pixel sub-frames along the channel dim, giving 8*(K+1)=56 channels (vs the paper's 4*(K+1)=28 on Wan 2.1). Plumbing (backward compatible; cond_channels=None leaves every existing pipeline unchanged): - LatentState/Modality gain an optional cond_channels field (patchified [B,T,C]). - LTXModel(mask_conditioning_channels=0) config-gates a widened patchify_proj; TransformerArgsPreprocessor concatenates cond_channels (or zero-pads) before it. - tools.clear_conditioning trims it; token-appending conditioning items (reference video/audio, driving, keyframe) extend it via extend_cond_channels. New: - VideoConditionByMaskChannels: encodes (K+1) pixel masks -> 56 channels, written onto the trailing driving tokens (noisy target stays all-zero, per the paper). - widen_patchify_proj_for_mask_channels: zero-init checkpoint surgery so a converted model reproduces the base output exactly until finetuned. Verified (CPU, random weights): backward compat, zero-init widened forward is bit-identical to baseline for any cond_channels, and the driving+mask pipeline forwards without crashing with correct placement/clipping. Visual quality requires Phase 3 finetuning; Replacement-mode z_ref height shift still deferred. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
@@ -269,6 +269,7 @@ def modality_from_latent_state(
|
||||
context=context,
|
||||
context_mask=None,
|
||||
attention_mask=state.attention_mask,
|
||||
cond_channels=state.cond_channels,
|
||||
)
|
||||
|
||||
|
||||
|
||||
Reference in New Issue
Block a user