Automated PR - 2026-06-17

.claude/skills/train-model/references/config-patching.md

@@ -0,0 +1,73 @@
+# Config Patching
+
+How to safely produce `<workspace>/<run-name>/config.yaml` from an example in `packages/ltx-trainer/configs/`. The trainer's config schema is Pydantic with `extra="forbid"` — unknown fields are rejected. Full field reference: [`packages/ltx-trainer/docs/configuration-reference.md`](../../../../packages/ltx-trainer/docs/configuration-reference.md).
+
+## Workflow
+
+1. Copy the example config matching the selected mode (see `mode-selector.md`) to `<workspace>/<run-name>/config.yaml`.
+2. Patch fields as described below. Preserve YAML comments where possible — they help the user audit the run later.
+3. **Never** edit the example config in `packages/ltx-trainer/configs/`. That's the user's reference library.
+
+## Required Patches (every run)
+
+| Field | Value |
+|-------|-------|
+| `model.model_path` | Absolute path to local `.safetensors` (from probe or user). |
+| `model.text_encoder_path` | Absolute path to local Gemma directory (from probe or user). |
+| `data.preprocessed_data_root` | `<workspace>/<run-name>/dataset/.precomputed` (absolute). |
+| `output_dir` | `<workspace>/<run-name>/outputs` (absolute). |
+
+## Hardware-Driven Patches
+
+Apply per the matched VRAM tier in `references/hardware-profiles.md`. After autotune (Phase 6), patch the winning trial's deltas in.
+
+## Schema Constraints (validate before launch)
+
+These will cause Pydantic errors or runtime failures; check before invoking the trainer.
+
+- **Frame count:** `validation.video_dims[2]` must satisfy `frames % 8 == 1` (1, 9, 17, 25, 33, 41, 49, 57, 65, 73, 81, 89, ...).
+- **Resolution:** `validation.video_dims[0]` and `[1]` must be divisible by 32.
+- **Multi-bucket training:** if dataset uses multiple resolution buckets, set `optimization.batch_size: 1`.
+- **At least one generated modality:** `training_strategy` must have at least one of `video.is_generated` or `audio.is_generated` set to `true`.
+- **Audio condition restrictions:** the audio modality cannot use `first_frame` or `spatial_crop` conditions.
+- **Strategy name:** prefer `training_strategy.name: "flexible"`. `text_to_video` and `video_to_video` still work but emit deprecation warnings.
+
+## LoRA Patches
+
+For style/concept LoRAs:
+
+- `lora.rank` and `lora.alpha`: set from the matched VRAM tier and use case. **32GB tier** pins rank 16 per `t2v_lora_low_vram.yaml`; **80GB+ tier** uses rank 32 per `t2v_lora.yaml`. Keep `alpha == rank`. See `mode-selector.md` for use-case-driven rank guidance.
+- `lora.target_modules`: short patterns like `"to_k"`, `"to_q"`, `"to_v"`, `"to_out.0"` match all attention modules (video + audio + cross-modal). Add `"ff.net.0.proj"`, `"ff.net.2"` only if user explicitly wants higher capacity.
+- **Audio-only LoRA targets** (T2A, audio inpainting): use `"audio_attn1.to_*"`, `"audio_attn2.to_*"` patterns to avoid touching video weights. See `configs/t2a_lora.yaml` for the exact list.
+
+## Validation Sample Prompts
+
+The example configs ship with placeholder validation prompts. Validation condition fields are documented in
+[`configuration-reference.md#validation-condition-types`](../../../../packages/ltx-trainer/docs/configuration-reference.md#validation-condition-types).
+For style/concept LoRAs:
+- Replace at least one `validation.samples[].prompt` with a prompt that uses the user's trigger word or describes the target concept. Tells the user something useful at the first validation interval.
+- Keep `validation.video_dims` consistent with the training resolution to make samples comparable.
+- **Describe the audio, for any run with a generated audio modality** (joint audio+video, T2A, V2A, etc.). The validation prompts must describe the audio the **same way the training captions do** — if the training captions transcribe speech or characterise sound (e.g. *"he says: ‘…’"*, *"calm spoken voice, quiet room tone"*, *"upbeat acoustic guitar"*), the validation prompts must include comparable audio direction. A prompt with no audio description gives the model no guidance for the audio branch and the generated audio comes out poor. This is not speech-specific — any audio (music, ambience, foley) needs describing. Mirror the structure/level of audio detail found in the dataset captions (inspect a few before writing the prompts).
+
+## W&B Patches
+
+- If the W&B credential check passes (`uv run python -c "import wandb; print(bool(wandb.Api().api_key))"` → `True`): `wandb.enabled: true`, `wandb.project` = `ltx2-<mode>`, `wandb.tags` includes the mode. (Do not use `wandb status` — it falsely reports `api_key: null` when logged in via netrc.)
+- If `False`: `wandb.enabled: false`. Surface in plan: *"Not logged in to W&B — run `wandb login` before training to enable tracking."* If the check errored/was ambiguous, ask the user rather than assuming off.
+
+## Output Dir Behaviour
+
+The trainer resumes optimizer/scheduler/step state **only when `model.load_checkpoint` is set** to a checkpoint file; it then looks for a matching `training_state_step_*.pt` next to that file. It does **not** auto-detect prior checkpoints in `output_dir/checkpoints/`. To resume, patch `model.load_checkpoint` to the latest checkpoint. To load weights but skip state restore: `checkpoints.no_resume: true`.
+
+For the orchestrator's resume flow: when resuming an interrupted run, patch `model.load_checkpoint` to the latest checkpoint under `output_dir/checkpoints/`. Leaving it unset starts a fresh run from step 0 even if checkpoints exist on disk.
+
+## Self-Check Before Launch
+
+Before any `python scripts/train.py` invocation:
+
+1. All `model_path`, `text_encoder_path`, `preprocessed_data_root` exist on disk.
+2. Frame and resolution constraints satisfied (see above).
+3. Generated modalities have matching latents directories under `.precomputed/`.
+4. For modes with `reference` condition: `reference_latents/` (and/or `reference_audio_latents/`) exists.
+5. For modes with `mask` condition: `video_masks/` (and/or `audio_masks/`) exists.
+
+A failed check at this point is much cheaper than a failed training start.

.claude/skills/train-model/references/hardware-profiles.md

@@ -0,0 +1,98 @@
+# VRAM Tiers
+
+Map probed GPU(s) to a starting training config. The autotune sweep in Phase 6 then empirically improves on this baseline. Use these **tier names** in `plan.md` and user-facing chat — not letter codes.
+
+Source of truth: the two configs shipped in the trainer repo —
+`packages/ltx-trainer/configs/t2v_lora.yaml` (standard) and
+`packages/ltx-trainer/configs/t2v_lora_low_vram.yaml` (low VRAM).
+Per `packages/ltx-trainer/docs/quick-start.md`, the trainer documents
+**80GB recommended** and **32GB minimum**. Anything below 32GB is
+unsupported by the project.
+
+## Probe
+
+```bash
+nvidia-smi --query-gpu=name,memory.total --format=csv,noheader
+```
+
+Pick the **smallest** VRAM tier across the visible GPUs. Multi-GPU only adds throughput at the same per-GPU memory budget — it doesn't relax per-GPU limits.
+
+## Minimum Gate
+
+If per-GPU VRAM is **< 32 GB**, stop the run. Surface to the user:
+
+> "This GPU has <N>GB VRAM. The LTX-2 trainer requires a minimum of 32GB (see `packages/ltx-trainer/docs/quick-start.md`). Training is unlikely to fit even with maximum memory savings, and we don't ship a tested config below 32GB. Options: (a) abort, (b) try anyway with the low-VRAM config and accept it may OOM — purely at your own risk."
+
+Do not invent a sub-32GB tier. The trainer team doesn't ship one.
+
+## 32GB tier — low-VRAM config
+
+**VRAM range:** 32 GB per GPU (trainer minimum).
+
+**Typical GPUs:** RTX 5090, V100 32GB.
+
+Start from `packages/ltx-trainer/configs/t2v_lora_low_vram.yaml` verbatim. Key choices already in that file (do not re-specify in `<workspace>/<run-name>/config.yaml` — copy the file and patch only the paths from `references/config-patching.md`):
+
+- `optimizer_type: "adamw8bit"`
+- `enable_gradient_checkpointing: true`
+- `batch_size: 1`, `gradient_accumulation_steps: 1`
+- `quantization: "int8-quanto"`
+- `load_text_encoder_in_8bit: true`
+- `offload_optimizer_during_validation: true`
+- `lora.rank: 16`, `lora.alpha: 16`
+
+Autotune (Phase 6) will sweep `quantization` off, `optimizer_type` → adamw, and `batch_size` up — but at 32GB the sweep often hits OOM on trial 2 or 3. That's fine; the conservative baseline still works.
+
+## 80GB+ tier — standard config
+
+**VRAM range:** 80 GB per GPU and above (trainer recommended).
+
+**Typical GPUs:** A100 80GB, H100 80GB, H200, B200.
+
+Start from `packages/ltx-trainer/configs/t2v_lora.yaml` verbatim. Key choices already in that file:
+
+- `optimizer_type: "adamw"`
+- `enable_gradient_checkpointing: true` (autotune may turn it off if headroom allows)
+- `batch_size: 1`, `gradient_accumulation_steps: 1`
+- `quantization: null`
+- `load_text_encoder_in_8bit: false`
+- `lora.rank: 32`, `lora.alpha: 32`
+
+On the 80GB+ tier, the autotune baseline already equals this config (adamw, no quantization), so the quantization/optimizer trials are no-ops; the only real lever is gradient checkpointing off — but for the 22B model that **usually OOMs even with tens of GB of apparent headroom**, so treat a win there as unlikely. The trainer reports its own step-time and peak-VRAM at the end of each run — use those rather than an external timer.
+
+For ≥140GB GPUs (H200, B200), the same 80GB+ tier baseline applies. FA3/FA4 attention backends are viable on Hopper/Blackwell and can speed up training, but they're optional — the trainer's defaults work on PyTorch SDPA without extra setup.
+
+## 40–60GB tier — mid-range (autotune from low-VRAM)
+
+**VRAM range:** 40–60 GB per GPU. The trainer doesn't ship a tested config for this range.
+
+**Typical GPUs:** A40, A6000 48GB, L40, RTX 6000 Ada.
+
+Start from the **32GB tier** (low-VRAM config) and let autotune relax `quantization`, `optimizer_type`, and `batch_size` based on actual headroom. Don't pre-bake intermediate YAML values that haven't been measured. Surface this as **40–60GB tier** in the plan.
+
+## Multi-GPU
+
+If `nvidia-smi` reports N ≥ 2 GPUs of the same model:
+
+- Launch with `uv run accelerate launch scripts/train.py <config>`.
+- Use `packages/ltx-trainer/configs/accelerate/fsdp.yaml` for full fine-tune.
+- DDP (default `accelerate launch` without a config file) is fine for LoRA.
+- Effective batch = `batch_size * gradient_accumulation_steps * num_gpus`. Reduce `gradient_accumulation_steps` proportionally to keep the effective batch consistent with the plan.
+
+## Full Fine-Tune
+
+If the user chose full fine-tune (`model.training_mode: "full"`):
+
+- Require multi-GPU + FSDP on 80GB+ tier GPUs. Otherwise warn in the plan that single-GPU full FT is unlikely to fit and propose LoRA instead.
+- Set `acceleration.offload_optimizer_during_validation: true` always (optimizer state is huge under full FT).
+
+## Model Path Constraints
+
+- `model.model_path`: local `.safetensors` only. No URLs.
+- `model.text_encoder_path`: local Gemma model directory. No URLs.
+
+If probe didn't find these in conventional locations (`/models/`, `~/models/`, `$LTX_MODELS_DIR`), ask the user in Phase 3 (or offer to download per `references/onboarding.md`).
+
+## Notes on Loss-of-Generality
+
+The two anchor tiers (32GB and 80GB+) correspond directly to the two configs the trainer ships. The autotune sweep is the empirical layer — if a particular GPU consistently lands on a different stable config, **update the relevant trainer config first**, not this file. This skill follows the trainer's choices, not the other way around.

.claude/skills/train-model/references/mode-selector.md

@@ -0,0 +1,74 @@
+# Mode Selector
+
+Map the user's stated intent to a `flexible`-strategy configuration. All modes are supported via a single strategy (`training_strategy.name: "flexible"`); the difference is which modality is generated and which `conditions` are attached.
+
+> Full reference: [`packages/ltx-trainer/docs/training-modes.md`](../../../../packages/ltx-trainer/docs/training-modes.md). This file is the **lookup table** for translating user intent.
+
+## Decision Table
+
+| User says (roughly)... | Mode | Example config | Modalities | Conditions |
+|------------------------|------|----------------|------------|------------|
+| "generate videos from text", "T2V LoRA" | T2V | `configs/t2v_lora.yaml` | video gen, audio gen | none |
+| "generate videos from a starting image", "I2V" | I2V | `configs/i2v_lora.yaml` | video gen, audio gen | `first_frame` (video) |
+| **plain concept/style LoRA** ("train a LoRA on X", no specific task) | **I2V by default** (see note) | `configs/i2v_lora.yaml` | video gen, audio gen | `first_frame` (video), `probability: 0.5` |
+| "extend a video forward in time" | Video extension (prefix) | `configs/video_extend_lora.yaml` | video gen, audio gen | `prefix` (video) |
+| "extend a video backward in time" | Video extension (suffix) | `configs/video_suffix_lora.yaml` | video gen, audio gen | `suffix` (video) |
+| "fill in masked regions of a video" | Video inpainting | `configs/video_inpainting_lora.yaml` | video gen | `mask` (video) |
+| "expand a video beyond its borders" | Video outpainting | `configs/video_outpainting_lora.yaml` | video gen | `spatial_crop` (video) |
+| "style transfer from reference video", "IC-LoRA", "depth/pose/canny control" | V2V IC-LoRA | `configs/v2v_ic_lora.yaml` | video gen | `reference` (video) |
+| "generate video to match an audio track" | A2V | `configs/a2v_lora.yaml` | video gen, audio frozen | none (audio `is_generated: false`) |
+| "add sound effects to silent video", "foley", "V2A" | V2A | `configs/v2a_lora.yaml` | video frozen, audio gen | none (video `is_generated: false`) |
+| "generate audio from text", "T2A" | T2A | `configs/t2a_lora.yaml` | audio gen | none |
+| "extend audio forward / backward" | Audio extension | `configs/audio_extend_lora.yaml`, `configs/audio_suffix_lora.yaml` | audio gen | `prefix` / `suffix` (audio) |
+| "fill in masked regions of audio" | Audio inpainting | `configs/audio_inpainting_lora.yaml` | audio gen | `mask` (audio) |
+| "audio style transfer from reference", "A2A IC-LoRA" | A2A IC-LoRA | `configs/a2a_ic_lora.yaml` | audio gen | `reference` (audio) |
+| "joint video+audio reference control" | AV2AV IC-LoRA | `configs/av2av_ic_lora.yaml` | video gen, audio gen | `reference` (both) |
+| Any of the above with full fine-tune | Full FT variant | as above, set `model.training_mode: "full"` | (mode-specific) | (mode-specific) |
+
+### Why I2V is the default for a plain concept/style LoRA
+
+A "train a LoRA on X" request isn't tied to one inference mode: **LoRA weights are pipeline-agnostic** — the same checkpoint loads in both T2V and I2V inference (both use `TI2VidOneStagePipeline`/`TwoStages`). The `i2v_lora` config trains `first_frame` with **`probability: 0.5`**, so the model learns both first-frame-conditioned (I2V) and unconditioned (T2V) generation in one run, and the first frame comes from each training clip automatically (no extra data prep). That makes I2V a versatile **superset** — usable for both at inference at no extra cost — which is why it's the default for a plain LoRA. Ask the user how they'll use it (text-only / from an image / both) and only drop to `t2v_lora` if they're sure it's text-only. (See the orchestrator `SKILL.md` Phase 1.)
+
+## Disambiguation Questions
+
+When the user's first answer is ambiguous, ask **one** follow-up:
+
+- "extend a video" → forward or backward in time?
+- "control with a reference" → video reference (depth/pose/canny/etc.) or audio reference?
+- "fill in regions" → video regions (masked frames) or audio regions (masked time)?
+- "T2V" → joint video+audio (default) or video-only?
+- LoRA or full fine-tune? Default to LoRA unless the user has multi-GPU + clear reason for full.
+
+## Combining Modes
+
+The flexible strategy allows stacking conditions. Common combinations:
+
+- **I2V + V2A** (start frame + generate audio for the resulting video) — not directly expressible; would need two passes.
+- **Video extension + audio extension** — both modalities generate, both have `prefix` condition. Express in one config.
+- **IC-LoRA + I2V** — `first_frame` + `reference` conditions on the video modality.
+
+If the user asks for a combination not listed, check `packages/ltx-trainer/src/ltx_trainer/training_strategies/flexible.py` for which conditions can co-exist on a modality. Audio modality cannot use `first_frame` or `spatial_crop`.
+
+## When the Intent Doesn't Map
+
+If after disambiguation there is no entry in the table and no combination of `flexible` conditions covers the user's request, go to the **Escape Hatch** section in the orchestrator `SKILL.md`. Do not silently pick the closest mode.
+
+## LoRA Rank by Use Case
+
+Once the mode is picked, choose `lora.rank` (and matching `lora.alpha`) based on what the LoRA is supposed to capture. These are starting points; autotune doesn't sweep rank because it's a quality knob, not a step-time one.
+
+| Use case | Suggested rank | Notes |
+|----------|----------------|-------|
+| Single character, single object, single style | 32–64 | Default for most concept LoRAs. Start at 32; bump to 64 if validation samples underfit. |
+| Multi-character world, dense series, complex multi-concept | 96–128 | More capacity for distinguishing several concepts inside one LoRA. |
+| Camera move, motion, transition (i.e. behavioural, not visual) | 8–16 | Motion is a thin signal — high ranks just memorise frame content. |
+| IC-LoRA control (V2V depth/pose/Canny/etc., A2A audio reference) | 16–32 | Start at 16 for structural control (depth, pose, edges); 24–32 if the reference carries richer style/texture. Video IC-LoRA often lands lower than concept LoRAs. |
+| LTX-2 trainer's default if unsure | 32 | Safe baseline. |
+
+On the **32GB tier** (`hardware-profiles.md`), the low-VRAM config already pins `lora.rank: 16`. If the user picks a higher-rank use case on 32GB, surface the trade-off in the plan but let them decide — the autotune sweep doesn't touch rank, so a too-high rank will simply OOM at training time.
+
+Keep `alpha == rank` unless the user has a specific reason otherwise (effective scaling = `alpha / rank`).
+
+## Starting Config
+
+Copy the matching example config — the exact filename from the **Example config** column of the decision table above (e.g. T2V → `configs/t2v_lora.yaml`, I2V → `configs/i2v_lora.yaml`, A2A IC-LoRA → `configs/a2a_ic_lora.yaml`) — into `<workspace>/<run-name>/config.yaml` as the starting point. Then patch per `references/config-patching.md`.

.claude/skills/train-model/references/onboarding.md

@@ -0,0 +1,115 @@
+# First-Run Onboarding
+
+What the orchestrator's Phase 2 probe checks for, what to do when something is missing, and what the skill is allowed to set up automatically (with explicit user approval).
+
+## Prerequisites Checked in Phase 2
+
+| Prerequisite | How to detect | If missing |
+|--------------|---------------|------------|
+| CUDA GPU visible | `nvidia-smi` returns ≥1 GPU | Stop. Training requires CUDA — point the user at non-LTX-2 docs. |
+| Linux | `uname -s` returns `Linux` | Stop. Trainer uses Triton (Linux-only). |
+| `uv` installed | `command -v uv` | Offer to install (see "Auto-setup" below). |
+| Workspace synced | `[ -f uv.lock ] && uv pip list \| grep -q ltx-trainer` | Offer to run `uv sync` from repo root. |
+| LTX-2 model weights | Search `/models/`, `~/models/`, `$LTX_MODELS_DIR` for a `.safetensors` matching `*ltx*2*` | Offer to download (see "Model downloads"). |
+| Gemma text encoder dir | Search same locations for a directory containing Gemma config | Offer to download. |
+| Captioner backend | Gemini auth (`GEMINI_API_KEY`/`GOOGLE_API_KEY` or gcloud/Vertex), OR a ≥40 GiB GPU to host the Qwen3-Omni-30B vLLM server (FP8), OR captions already in the dataset | See "Captioner graceful degradation" below. **Check the HF cache for an already-downloaded Qwen model before assuming a download is needed** (see note below the table). |
+| W&B login (optional) | `uv run python -c "import wandb; print(bool(wandb.Api().api_key))"` → `True` means logged in. Uses wandb's own credential resolution (env/netrc/settings). **Don't** use `wandb status` (reports `api_key: null` even when logged in). | Not a blocker. If `False`: disabled in config + flagged in plan. If the check errors/ambiguous: ask the user, don't assume off. |
+| Disk space | `df -h $WORKSPACE` | Surface available space alongside what a run consumes (preprocessed latents, several-GB checkpoints, validation samples). Flag concerns to the user; don't enforce a hard threshold. |
+
+Surface findings as a compact table in chat. For each missing item, present the user with a concrete next step (download command, install command, or "skip this — here's the consequence").
+
+**Check the HF cache before declaring the local captioner unavailable.** The Qwen3-Omni model is served from the HuggingFace cache, not `~/models/`. Before concluding it must be downloaded (or ruling it out on free-disk grounds), check whether it's already cached:
+
+```bash
+ls -d "${HF_HOME:-$HOME/.cache/huggingface}"/hub/models--Qwen--Qwen3-Omni* 2>/dev/null \
+  && du -sh "${HF_HOME:-$HOME/.cache/huggingface}"/hub/models--Qwen--Qwen3-Omni* 2>/dev/null
+```
+
+If it's cached (~60 GiB, all shards present), no download is needed — don't rule out the local captioner because of low free disk on the *home* partition; the weights already exist. Only the GPU-VRAM constraint (≥40 GiB for FP8) then applies.
+
+## Auto-Setup (with explicit user approval)
+
+The skill may, only after the user explicitly says yes, do these setup actions. Each action is a single discrete question.
+
+### Run `uv sync`
+
+```bash
+cd <repo-root>
+uv sync
+```
+
+Ask: *"Repo not synced. Run `uv sync` now? It will download the project's Python dependencies."*
+
+### Install `uv`
+
+Ask: *"`uv` not installed. Install via the official one-liner now?"*
+
+```bash
+curl -LsSf https://astral.sh/uv/install.sh | sh
+```
+
+After install, ask the user to restart their shell or `source ~/.bashrc` before continuing.
+
+### Model downloads
+
+Use `huggingface-cli` (comes with `huggingface-hub`, transitively pulled by `uv sync`). Default destination: `$LTX_MODELS_DIR` if set, else `~/models/` (create if missing). Surface destination in the prompt — never download into the repo or into the workspace.
+
+**LTX-2 base model:**
+
+```bash
+huggingface-cli download Lightricks/LTX-2.3 \
+  ltx-2.3-22b-dev.safetensors \
+  --local-dir ~/models/ltx-2.3
+```
+
+Public reference: <https://huggingface.co/Lightricks/LTX-2.3>.
+
+**Gemma text encoder:**
+
+```bash
+huggingface-cli download google/gemma-3-12b-it-qat-q4_0-unquantized \
+  --local-dir ~/models/gemma-3-12b
+```
+
+**Qwen3-Omni captioner (only if the GPU can host it):**
+
+The local captioner is Qwen3-Omni-30B-A3B-Thinking served by a vLLM server (`scripts/serve_captioner.py`), which downloads the model (~65 GiB) on first launch via `uvx vllm` — there's no separate `huggingface-cli` step. Default **FP8** (~31 GiB weights) fits on a **40 GiB** GPU; **bf16** (~60 GiB) needs **≥66 GiB free VRAM**. On a GPU below ~40 GiB (typical consumer 24/32 GB cards), don't use it — use Gemini instead (see "Captioner graceful degradation").
+
+The base model + text encoder are large (multi-GB) downloads. Ask the user before each one — `huggingface-cli` reports the actual size at the start of the transfer. Do **not** batch them into a single "yes/no"; the user may want only what's missing.
+
+### Hugging Face login (if any downloads fail with 401)
+
+Ask: *"Hugging Face download requires login (some Lightricks models are gated). Run `huggingface-cli login` now? You'll need a token from <https://huggingface.co/settings/tokens>."*
+
+## Captioner Graceful Degradation
+
+The captioner is the trickiest prerequisite. The local backend (`qwen_omni`) is now a **30B model served by a vLLM server** — ~65 GiB download; FP8 fits on a 40 GiB GPU, bf16 needs ≥66 GiB. Typical consumer cards (24/32 GB) can't host it, so for most users **prefer Gemini**.
+
+Decision tree:
+
+1. **User already has captions in their dataset metadata** → skip the captioner entirely (Step 3 of `prepare-dataset` skips when captions are present).
+2. **GPU ≥40 GiB (FP8) / ≥66 GiB (bf16)** → `qwen_omni` is viable: launch `scripts/serve_captioner.py` first, then caption. Gemini is still fine here too.
+3. **GPU below ~40 GiB (the common consumer case)** → `qwen_omni` is not an option. Recommend **`gemini_flash`** and tell the user they need Gemini auth: a `GEMINI_API_KEY`/`GOOGLE_API_KEY` (get one at <https://aistudio.google.com/apikey>) **or** working gcloud/Vertex AI credentials.
+4. **No Gemini auth and can't run Qwen3** → the only remaining path is bringing their own captions: add a `caption` column to the dataset metadata, then re-invoke the skill.
+
+Wait for the user's choice. Don't pick one automatically — but make the hardware reality explicit so they don't try to run the server on a card that can't host it.
+
+## What Auto-Setup Does NOT Touch
+
+- Does not modify the user's shell rc files except by explicit instruction (e.g., "add `export LTX_MODELS_DIR=...` to your `.bashrc`?" with the user agreeing).
+- Does not modify `~/.gitconfig`, `~/.ssh/`, or any auth-related files.
+- Does not install GPU drivers, CUDA, or system packages.
+- Does not delete or move existing model files. If a model is found but at an unexpected path, surface and let the user decide.
+- Does not download into the repo or workspace. Models live at `~/models/` (or `$LTX_MODELS_DIR`).
+
+## Configuration Inheritance
+
+After downloads, the skill records the resolved paths into the plan's Assumptions section and into `config.yaml`:
+
+```yaml
+model:
+  model_path: "/home/<user>/models/ltx-2.3/ltx-2.3-22b-dev.safetensors"
+  text_encoder_path: "/home/<user>/models/gemma-3-12b"
+```
+
+Suggest (don't enforce) setting `LTX_MODELS_DIR=~/models` in their shell rc for future runs.

.claude/skills/train-model/references/plan-template.md

@@ -0,0 +1,107 @@
+# Plan Template
+
+Write `<workspace>/<run-name>/plan.md` following this template. The plan is the user's contract with the agent — it's the gate before any heavy work runs.
+
+Scale each section to its relevance. Skip subsections that don't apply (e.g., no captioning if data is already captioned), but never collapse "Assumptions" or "Cost/time estimate".
+
+```markdown
+# Training Plan — <run-name>
+
+## Goal
+
+<One paragraph restating the user's intent in their own terms.>
+
+## Mode
+
+**<Mode name>** — <one-line rationale linking user intent to mode>.
+
+- Config base: the concrete example config selected by the mode (e.g. `packages/ltx-trainer/configs/t2v_lora.yaml`) — use the actual filename, not a placeholder
+- Conditions: <list, or "none">
+- Training mode: <`lora` | `full`>
+
+## Dataset
+
+- Source: `<absolute path>` (<N samples>)
+- Captions: <`already present` | `will be generated with <backend>`>
+- Audio: <`present` | `absent — using --skip-audio` | `to be paired with --audio-durations`>
+- IC-LoRA references: <`present` | `to be generated via compute_reference.py` | `n/a`>
+
+## Preprocessing
+
+- Target resolution buckets: `<W>x<H>x<F>` (frames satisfy `frames % 8 == 1`; W,H divisible by 32)
+- Estimated time: ~<duration> on detected hardware
+- Output: `<workspace>/<run-name>/dataset/.precomputed/`
+
+## Training Config
+
+| Field | Value |
+|-------|-------|
+| Optimizer | <adamw / adamw8bit> |
+| Mixed precision | <bf16 / fp16> |
+| Quantization | <null / int8-quanto / ...> |
+| Gradient checkpointing | <on / off> |
+| Batch size | <N> |
+| Gradient accumulation | <N> (effective batch = <N>) |
+| Steps | <N> |
+| Learning rate | <value> |
+| LoRA rank / alpha | <N / N> (or "full FT") |
+| LoRA target modules | <list> (or "n/a") |
+| LoRA trigger word | <word> (or "n/a") |
+| Validation interval | every <N> steps |
+| Checkpoint interval | every <N> steps |
+
+## Hardware
+
+- GPU(s): <name> x <count>, <VRAM>GB per GPU
+- Launch: <`python scripts/train.py` (single) | `accelerate launch` (multi)>
+- VRAM tier: <32GB tier | 40–60GB tier | 80GB+ tier> — <low-VRAM config (`t2v_lora_low_vram.yaml`) | standard config (`t2v_lora.yaml`) | mid-range, autotuned from low-VRAM>
+
+## Sanity Check + Autotune
+
+*In plain terms: before committing to the full run, I do a quick dry run on a single clip at your target resolution to catch out-of-memory or config errors in ~2 minutes (rather than failing hours in), then try a few config variants to pick the fastest one that fits your GPU.*
+
+Mechanics:
+- 1 sample, full target resolution, 50 steps + 1 validation pass.
+- Autotune sweep: up to 5 trials varying quantization / optimizer / batch size.
+- Stops at first OOM or no-improvement.
+
+## Monitoring
+
+<One of:>
+- W&B: enabled, project `<name>`, entity `<entity-or-default>`. URL will be surfaced once training starts.
+- W&B: **not logged in** — run `wandb login` before training to enable tracking. Otherwise training proceeds without remote logging.
+
+## Outputs
+
+- Training config: `<workspace>/<run-name>/config.yaml`
+- Checkpoints: `<workspace>/<run-name>/outputs/checkpoints/`
+- Validation samples: `<workspace>/<run-name>/outputs/samples/`
+- Autotune log: `<workspace>/<run-name>/autotune.log`
+
+## Assumptions
+
+Defaults the agent chose silently. Override any by replying with the new value.
+
+- <list every non-trivial assumed value: precision, scheduler type, seed, validation prompts, checkpoint retention, etc.>
+
+## Cost / Time Estimate
+
+Give only estimates you can ground; label anything not yet measured as rough. Do **not** state a confident training duration before the sanity check has measured a real step time — say "training duration TBD until the sanity check measures step time" and fill it in afterward (per Hard Invariant #5: no fabricated predictions).
+
+- Captioning: ~<duration> (rough)
+- Preprocessing: ~<duration> (rough)
+- Sanity check + autotune: ~<duration> (rough)
+- Full training: **measured after sanity check** — then `<measured step-time> × <steps>`
+- **Total wall-clock estimate:** rough until step time is measured; refine after the sanity check.
+
+## Approve to Proceed
+
+Reply "approve" (or with edits) to start. No captioning, preprocessing, autotune, or training will run before approval.
+```
+
+## Notes on Writing the Plan
+
+- Show numbers, not adjectives. "~3 hours" beats "fairly long."
+- Surface every assumption that, if wrong, would cost the user time. Better to over-list than under-list — the user can skim.
+- If a section reveals you need to ask another question, **stop and ask** before finalizing the plan. The plan is the last gate, not the first.
+- If the user's hardware can't reasonably support the requested mode (e.g., single-GPU full FT on a 32GB consumer card), say so plainly in the plan and propose the alternative (LoRA, multi-GPU, etc.), rather than silently downgrading.

.claude/skills/train-model/references/troubleshooting.md

@@ -0,0 +1,84 @@
+# Troubleshooting
+
+Quick lookup for failures during sanity check, preprocessing, or training. For deeper coverage see `packages/ltx-trainer/docs/troubleshooting.md`.
+
+## OOM During Training Step
+
+Order of operations (cheapest first):
+
+1. `optimization.enable_gradient_checkpointing: true` (if not already on).
+2. `optimization.batch_size: 1` and increase `gradient_accumulation_steps` to preserve effective batch.
+3. `optimization.optimizer_type: "adamw8bit"`.
+4. `acceleration.quantization: "int8-quanto"`.
+5. Reduce `lora.rank` (32 → 16 → 8). Alpha follows rank.
+6. Reduce target resolution (`validation.video_dims` and re-preprocess the dataset at the new resolution).
+
+The last option is expensive — flag it clearly to the user before re-preprocessing.
+
+## OOM During Validation Sample Generation
+
+The validation pass loads decoders + runs CFG/STG inference; it can OOM even when the training step fits.
+
+1. `acceleration.load_text_encoder_in_8bit: true` (trainer config, not the dataset script).
+2. `acceleration.offload_optimizer_during_validation: true` (especially for full FT or high-rank LoRA).
+3. Reduce `validation.video_dims` (smaller validation than training is fine — it's only for visual feedback).
+4. Reduce `validation.inference_steps` (e.g. 30 → 20).
+5. Increase `validation.interval` to validate less often.
+
+## NaN Loss
+
+1. Check `acceleration.mixed_precision_mode`: prefer `"bf16"`. If `"fp16"`, switch.
+2. Verify dataset latents are well-formed: `uv run python scripts/decode_latents.py <latents-dir> <output-dir> --model-path <model>` (it decodes a whole latents directory, not a single `.pt`) should reconstruct sensibly.
+3. Lower `optimization.learning_rate` by 5x.
+4. Add `optimization.max_grad_norm: 1.0` (default; verify it's set).
+5. If using `quantization`, try `null` — INT8/INT4 quantization can interact badly with poorly-conditioned LoRA inits at high LR.
+
+## Validation Samples Look Wrong but Loss Is Fine
+
+Often not a bug — validation uses simplified inference. For real quality assessment, run a checkpoint through `packages/ltx-pipelines/` after training.
+
+## Trainer Won't Start: Config Validation Error
+
+Pydantic `extra="forbid"` means typos in field names fail loudly. Read the error carefully — it names the offending field and path. Fix and re-launch.
+
+Common offenders:
+- `latents_dir` typo or wrong relative path.
+- A required field genuinely missing after copying an example (most fields have defaults; check the error message for the exact field path).
+- `target_modules` listed at wrong nesting level (must be under `lora:`).
+
+## Trainer Won't Start: Missing Files
+
+- `model_path` not found → re-probe `/models/`, `~/models/`, `$LTX_MODELS_DIR`, or ask the user.
+- `text_encoder_path` directory missing the Gemma config → ensure the path is to the Gemma model dir, not its parent.
+- `preprocessed_data_root` doesn't contain expected subdirs → re-verify Phase 7 ran for the chosen mode (see `phases/preprocess-dataset.md`).
+
+## Resume Stops Working
+
+The trainer **does not auto-resume from `output_dir`**. Resume happens only when `model.load_checkpoint` is explicitly set to a checkpoint file; the trainer then loads those weights and looks for a `training_state_step_*.pt` next to that file to restore optimizer/scheduler/step. Common pitfalls:
+- `model.load_checkpoint` not set or set to the wrong path → fresh run from step 0 even when `outputs/checkpoints/` is full of artifacts. Patch `model.load_checkpoint` to the latest checkpoint.
+- `checkpoints.no_resume: true` is set → weights load but state is discarded. Remove the flag if you want a proper resume.
+- `training_state_step_*.pt` missing from next to the loaded checkpoint → weights load but step counter resets. Make sure the state file accompanies the checkpoint.
+- `training_state_step_*.pt` corrupted (size 0, fails `torch.load`) → trainer falls back to step 0 with a warning.
+
+## Autotune Trial Failed Mid-Sweep
+
+- If trial 2 (quantization off) OOMs: revert to trial 1 and stop the sweep. The 32GB tier baseline is correctly aggressive.
+- If trial 3 (adamw) OOMs: revert to adamw8bit. Continue with trial 4 if VRAM headroom allows.
+- If trial 4 (batch_size up) OOMs: revert and stop. We've found the ceiling.
+
+Never carry over a failing trial's deltas. Always revert to the last-known-good before the next change.
+
+## Captioning Is Slow
+
+- Local `qwen_omni` is a 30B model served by `serve_captioner.py` (vLLM). If the server won't start or OOMs on launch: keep the default `--quantization fp8` (don't use `bf16` unless ≥66 GiB free VRAM), lower `--gpu-memory-utilization`, or reduce `--max-model-len`. If the GPU can't host a 30B model at all, switch to `gemini_flash`.
+- `caption_videos.py --captioner-type qwen_omni` errors connecting → the vLLM server isn't running or `--vllm-url` doesn't match. Start `serve_captioner.py` first and confirm the port.
+- For most hardware and for larger datasets, prefer `--captioner-type gemini_flash --num-workers <N>` (needs Gemini auth: `GEMINI_API_KEY`/`GOOGLE_API_KEY` or gcloud/Vertex) — runs anywhere and parallelises; local Qwen needs a heavy GPU and a running server.
+
+## Process_Dataset Errors
+
+- "frames divisible by..." → the video doesn't have enough frames at the requested temporal resolution. Either shorten the requested frame count or use `split_scenes.py` to break long videos.
+- "shape mismatch" on existing `.precomputed/` → user requested a different resolution than the existing data. Per the invariants, **stop and ask** — do not overwrite. Offer: reuse at old resolution / re-preprocess to a new dir / abort.
+
+## When To Give Up and Ask The User
+
+If a fix isn't obvious from this file or `packages/ltx-trainer/docs/troubleshooting.md` within two attempts, stop and surface the full error + the steps already tried to the user. Don't loop indefinitely on autonomous fixes — the user has context the agent doesn't (which checkpoints are precious, what they care about preserving, etc.).