Automated PR - 2026-06-17
This commit is contained in:
github-actions[bot]
@@ -0,0 +1,277 @@
|
||||
---
|
||||
name: train-model
|
||||
description: End-to-end agent for training LTX-2 models. Probes filesystem and GPU, picks the right conditioning mode from the user's intent, prepares the dataset (scenes, captions, references), preprocesses, autotunes, launches, and monitors training. Use when the user wants to train, fine-tune, LoRA, or otherwise produce a custom LTX-2 model.
|
||||
argument-hint: [optional source path or run name]
|
||||
user-invocable: true
|
||||
allowed-tools: Bash, Read, Grep, Glob, Edit, Write, Agent, AskUserQuestion, TodoWrite
|
||||
---
|
||||
|
||||
# Train Model — Orchestrator
|
||||
|
||||
Take the user from "I want to train something" to a running, monitored training job — automating the mechanical glue (dataset layout, captioning, preprocessing, config patching, launch, monitoring) without making silent decisions on their behalf.
|
||||
|
||||
> **Source of truth for the trainer:** [`packages/ltx-trainer/docs/`](../../../packages/ltx-trainer/docs/). This skill orchestrates those scripts; it does not duplicate their reference content.
|
||||
|
||||
## Hard Invariants
|
||||
|
||||
These are non-negotiable. Re-read them before every action that touches the filesystem or starts a process.
|
||||
|
||||
1. **No file mutation outside the run workspace without explicit user approval.** The workspace is `./projects/<run-name>/`. Never overwrite, move, delete, or modify any user file or directory outside it without surfacing an explicit ask. Hours of dataset work must never be silently destroyed.
|
||||
2. **No heavy work before plan approval.** Captioning, preprocessing, training, and autotune do **not** start until the user approves `plan.md` (Phase 4). Probing the filesystem and running `nvidia-smi` is fine; encoding videos or downloading models is not.
|
||||
3. **No silent assumptions.** Every non-trivial default appears under "Assumptions" in `plan.md`.
|
||||
4. **No code changes to the trainer package without explicit consent.** If the user's intent doesn't map to a supported configuration, follow the **Escape Hatch** section below — do not unilaterally edit `packages/ltx-trainer/`.
|
||||
5. **No fabricated claims about training outcomes or data sufficiency.** Do not assert how well something *will* train, whether a dataset is "too small," how many samples/seconds of audio are "enough," which modality will "learn better," expected quality, or any similar prediction — you have no grounded basis for these, they're frequently wrong, and they mislead users. Stick to facts you can substantiate: what the trainer/docs actually say, observed numbers (loss, step time, VRAM), counts, and the user's own stated goals. If the user asks for a recommendation that depends on such judgment, you may share it **only** as an explicitly-flagged uncertainty ("I'm not sure — you'd have to try it"), never as authoritative fact. When in doubt, say less.
|
||||
|
||||
## Keep the User Informed
|
||||
|
||||
Most users don't know how this skill works under the hood — they don't know what "preprocessing," "a one-sample sanity check," or "autotune" mean or why they're happening. Narrate the run in plain language so it never feels like a black box:
|
||||
|
||||
- **Entering a phase:** one or two sentences on *what you're about to do and why* — in user terms, not jargon.
|
||||
- **Leaving a phase:** one line on *what came out of it* (e.g. "captioned 9 clips," "found the fastest stable config: batch 1, ~3s/step").
|
||||
- **Explain the non-obvious phases explicitly** — these are the ones that confuse people:
|
||||
- *Sanity check (Phase 6):* "Before the full run, I do a quick dry run on a single clip at your target resolution. It catches out-of-memory or config problems in a couple of minutes instead of failing hours into training."
|
||||
- *Autotune (Phase 6):* "Then I try a few configuration variants on that one clip to pick the fastest one that still fits your GPU — so the full run is as fast as it can be."
|
||||
- *Preprocess (Phase 7):* "I'm encoding your videos into the compressed latents the trainer reads. One-time step; the trained model never sees the raw videos directly."
|
||||
- Keep it concise — a sentence or two per transition, not walls of text or raw logs. This is running commentary, not a replacement for the upfront plan (Phase 4) or the status reports (Phase 8).
|
||||
- Long-running steps (preprocess, training): say roughly how long it'll take and that you'll report back, so silence doesn't read as "stuck."
|
||||
- **Describe what you're doing — don't editorialize about how it'll turn out.** Narration covers *what's happening*; it must not drift into unfounded predictions about training quality or data sufficiency (e.g. "26s of audio is too little," "voice won't learn well"). Those are fabricated claims — see Hard Invariant #5. State facts and the user's choices; leave the "will it be good?" judgment to the user watching the results.
|
||||
|
||||
## Phase 0 — Set Up
|
||||
|
||||
Create the workspace and todos.
|
||||
|
||||
1. Pick a workspace root in this order (use first writable):
|
||||
- `$LTX_TRAININGS_DIR`
|
||||
- `/data/ltx-trainings/`
|
||||
- `/workspace/ltx-trainings/`
|
||||
- `./projects/` (repo-relative — preferred default in this repo)
|
||||
2. Derive a tentative `<run-name>` from the user's words; finalise after Phase 1 once the mode is known. Format: `<mode>-<dataset-name>-<YYYYMMDD-HHMM>`.
|
||||
3. Create `<workspace>/<run-name>/` and seed empty subdirs: `dataset/`, `outputs/`, `overfit/`.
|
||||
4. Create a todo list covering Phases 1–9 so the user can see progress.
|
||||
|
||||
## Phase 1 — Intent
|
||||
|
||||
Ask one question, framed in user terms (not jargon):
|
||||
|
||||
> What do you want the model to learn? Examples: "generate videos from text," "make a LoRA of a specific style," "extend a video forward in time," "add sound effects to a silent video," "fill in masked regions of a video."
|
||||
|
||||
Map the answer to one or more conditioning modes via `references/mode-selector.md`. If the requested capability has no mapping, go to **Escape Hatch**.
|
||||
|
||||
### Plain concept/style LoRA → ask how it'll be used, default to I2V
|
||||
|
||||
A "train a LoRA on X" request (a character/style/concept LoRA, no specific conditioning task) maps to either T2V or I2V. These aren't locked to inference: LoRA weights are pipeline-agnostic (the same checkpoint loads in both T2V and I2V inference), and the `i2v_lora` config applies first-frame conditioning with **`probability: 0.5`** — so it learns **both** conditioned (I2V) and unconditioned (T2V) generation in one run, and the first frame is taken automatically from each training clip (no extra data prep). I2V is therefore a versatile superset.
|
||||
|
||||
Ask how they intend to use the result:
|
||||
|
||||
> Will you generate videos from **text alone** (T2V), from a **starting image** (I2V), or **both / not sure**?
|
||||
|
||||
- **Both / not sure (default):** use `i2v_lora` (probabilistic first-frame) — works for both at inference.
|
||||
- **I2V:** `i2v_lora`.
|
||||
- **Text only:** `t2v_lora`.
|
||||
|
||||
(This only applies to plain concept/style LoRAs. A specific task — extension, inpainting, foley, IC-LoRA, etc. — maps directly to its mode via `mode-selector.md`; no usage question needed.)
|
||||
|
||||
### Confirm the mode before proceeding
|
||||
|
||||
Once the mode is determined, **state it plainly and confirm it** before doing any probing or work — a wrong inference is cheap to fix here and expensive later:
|
||||
|
||||
> "Got it — I'll train an **I2V LoRA** (usable for both image-to-video and text-to-video at inference). Sound right?"
|
||||
|
||||
The mode also appears in the plan (Phase 4), but confirm it here so the rest of the flow isn't built on a wrong guess.
|
||||
|
||||
## Phase 2 — Probe
|
||||
|
||||
No questions in this phase. Inspect what's already there. Use `references/onboarding.md` as the source of truth for the prerequisite checklist and what to do when something is missing.
|
||||
|
||||
### Filesystem probe
|
||||
- If the user pointed at a path, classify: directory of raw videos, single long video, directory with a metadata file (CSV/JSON/JSONL), existing `.precomputed/`, partial outputs from a prior run.
|
||||
- For metadata files, identify columns: `video`/`media_path`, `caption`, `audio`, `reference_video`, `video_mask`, etc. (see `packages/ltx-trainer/docs/dataset-preparation.md`).
|
||||
- **Clip lengths (small datasets only):** for datasets up to a few hundred clips, `ffprobe` each clip's frame count and note the **minimum**. Clips shorter than the target frame bucket are silently skipped by `process_dataset.py`, so the shortest clip caps the achievable frame count — feed this into the Phase 3 resolution/frame-count choice (pick a bucket the clips support, or plan multi-bucket). Skip this per-clip probe for large datasets (too slow); rely instead on the post-preprocess reconciliation in Phase 7, which flags any dropped clips regardless of dataset size.
|
||||
- Check for an existing `<workspace>/<run-name>/` and whether `outputs/checkpoints/` contains a prior checkpoint (`lora_weights_step_*.safetensors` or `model_weights_step_*.safetensors`, plus a matching `training_state_step_*.pt` when resume state is enabled). This is a **resume candidate** — but note the trainer does *not* auto-resume from the output dir; resuming requires explicitly setting `model.load_checkpoint` in `config.yaml` to that checkpoint path. See `phases/launch-and-monitor.md` for the resume flow.
|
||||
- Check disk space at the workspace root. Preprocessed latents, checkpoints, and validation samples add up across a run; surface the available space alongside a rough sense of what one run consumes (one preprocessed bucket scales with sample count and resolution; each checkpoint is several GB), and warn the user if free space looks tight given their dataset size.
|
||||
|
||||
### Hardware probe
|
||||
- `nvidia-smi --query-gpu=name,memory.total,driver_version --format=csv,noheader` → GPU model, count, VRAM. Stop the run if no CUDA GPU.
|
||||
- W&B login state — use wandb's **own** credential resolution (source-agnostic: covers env var, netrc, and the wandb settings file), not a hand-rolled netrc grep and **not** `wandb status` (which misleadingly reports `api_key: null` even when logged in):
|
||||
|
||||
```bash
|
||||
uv run python -c "import wandb; print(bool(wandb.Api().api_key))" # True => logged in
|
||||
```
|
||||
|
||||
`True` → W&B is available, enable it. `False` → genuinely not logged in. If the check errors or is ambiguous, **ask the user** rather than silently disabling — a wrong "W&B off" assumption can incorrectly disable expected tracking.
|
||||
- Apply `references/hardware-profiles.md` to derive defaults (32GB / 40–60GB / 80GB+ VRAM tier).
|
||||
|
||||
### Prerequisite probe (first-run sanity)
|
||||
- `uv` installed (`command -v uv`).
|
||||
- Workspace synced (lockfile present + `ltx-trainer` import works).
|
||||
- LTX-2 `.safetensors` and Gemma text encoder dir present in `/models/`, `~/models/`, or `$LTX_MODELS_DIR`.
|
||||
- Captioner availability: Gemini auth (`GEMINI_API_KEY`/`GOOGLE_API_KEY` or gcloud/Vertex) OR a ≥40 GiB GPU to host the Qwen3-Omni-30B vLLM server (FP8; bf16 needs ≥66 GiB). On typical consumer GPUs (24/32 GB), Gemini is effectively the only local-free option — see `references/onboarding.md`.
|
||||
|
||||
For any missing prerequisite, **do not silently fail in a later phase**. Present the finding in chat with the specific next step from `references/onboarding.md`. The skill may offer to auto-install / auto-download missing pieces — but only ever after asking the user explicitly, one item at a time (model downloads are tens of GB each). Never auto-modify shell rc files or system config without consent.
|
||||
|
||||
When everything (or what the user agreed to set up) is in place, fold the resolved paths into the plan's Assumptions section.
|
||||
|
||||
### Pre-existing artifacts in the run dir
|
||||
|
||||
If the run dir (or a user-supplied path) already contains artifacts from a prior session, classify each into one of three buckets — only the third prompts the user:
|
||||
|
||||
1. **Deterministically verifiable → verify, then reuse silently (or stop).** `.precomputed/` latents: load a sample, check tensor shapes + modality coverage against the target (Phase 7). Match → reuse, no question. Mismatch/incomplete → stop and ask (reuse-at-old-spec / re-preprocess to a new dir / abort).
|
||||
2. **Cheap, fully-derived intermediates → regenerate silently.** `overfit/`, eval renders, sanity/temp configs, one-sample metadata. Delete and redo; don't ask.
|
||||
3. **Expensive AND not deterministically verifiable → ask.** Captions (`dataset.json`) and trained checkpoints/outputs. We can't programmatically decide whether existing captions or a half-finished run are what the user wants now, so surface what was found (counts, and how/when produced if knowable) and ask: reuse vs regenerate (for checkpoints: resume vs fresh).
|
||||
|
||||
Principle: only ask when reuse-vs-regenerate is a genuine judgment call with cost either way. Never silently delete user-supplied data (hard invariant #1).
|
||||
|
||||
## Phase 3 — Ask (minimum viable set)
|
||||
|
||||
Ask only what cannot be inferred. Use `AskUserQuestion` with multiple-choice when possible. Typical questions:
|
||||
|
||||
- **Target resolution / frame count** — propose a default per mode (e.g., `768x512x49` for T2V LoRA on consumer GPUs); offer overrides.
|
||||
- **Training steps** — if dataset size doesn't pin it, propose a default (e.g., 2000 for small LoRA datasets).
|
||||
- **LoRA trigger word / concept name** — only for style/concept LoRAs. Ask **only** for the word itself (or whether they want one). **Never** ask or mention *how* it's injected — it's always the `--lora-trigger` flag (passed to `process_dataset.py`, which forwards it to `process_captions.py` where the prepend happens); this is a fixed implementation detail. Presenting caption-injection as an option creates unnecessary confusion.
|
||||
- **Captioner backend** — only if more than one path is viable (e.g. a ≥40 GiB GPU can host the Qwen3-Omni-30B server *and* Gemini auth is available). On typical consumer GPUs, default to `gemini_flash` and surface that Gemini auth is required rather than asking.
|
||||
- **Model paths** — only if not found in the probe.
|
||||
|
||||
Never ask anything answerable by `ls`, `nvidia-smi`, or the W&B credential check above.
|
||||
|
||||
## Phase 4 — Plan
|
||||
|
||||
Write the plan to `<workspace>/<run-name>/plan.md` using `references/plan-template.md`. Present it to the user in chat (don't just dump the file path). Wait for explicit approval before proceeding.
|
||||
|
||||
If the user requests changes, edit the plan and re-present. Do not start Phase 5 until approval.
|
||||
|
||||
## Phase 5 — Prepare Dataset
|
||||
|
||||
If captioned metadata already exists with all required columns for the chosen mode, skip this phase. Otherwise follow `phases/prepare-dataset.md` — re-read it before acting.
|
||||
|
||||
**Captioning gate:** caption a 3-sample spot-check first, show the captions in full, and **STOP for explicit user approval** before captioning the full set. The user must approve or give tuning instructions — never auto-proceed to the full pass. (Details in `phases/prepare-dataset.md`.)
|
||||
|
||||
**Conditioning-inputs gate:** modes that need a reference (V2V/A2A/AV2AV IC-LoRA) or a mask (video/audio inpainting) require a per-sample input that encodes the user's specific idea. **Ask the user to provide it** — never invent the method (no defaulting to Canny/depth/generic masks). Only help generate it if the user explicitly asks, following *their* approach. Don't enter preprocessing for these modes without the input present. (Details in `phases/prepare-dataset.md` Step 4.)
|
||||
|
||||
## Phase 6 — Sanity Check + Autotune (always run)
|
||||
|
||||
**Tell the user what this phase is before starting it** — it's the most opaque to someone who doesn't know the design (see "Keep the User Informed"). In plain terms: a quick single-clip dry run at the target resolution to catch OOM/config errors cheaply, followed by trying a few config variants to pick the fastest stable one.
|
||||
|
||||
Run **at the full target resolution** on **one sample** before the full preprocess. Purpose:
|
||||
1. Catch OOM / config errors before paying the full preprocessing cost.
|
||||
2. Empirically pick the fastest stable config via a small sweep.
|
||||
|
||||
Steps:
|
||||
1. Pick one sample from the dataset metadata. Preprocess just that sample to `<workspace>/<run-name>/overfit/.precomputed/` (see `phases/preprocess-dataset.md` — use it in "one-sample" mode).
|
||||
2. Generate a temp config matching the planned full-run config but with `data.preprocessed_data_root: overfit/.precomputed`, `optimization.steps: 50`, `validation.interval: 50`, `checkpoints.interval: null`.
|
||||
3. Run the **baseline trial**: the conservative config from the matched VRAM tier (32GB tier = `t2v_lora_low_vram.yaml` defaults; 80GB+ tier = `t2v_lora.yaml` defaults — see `references/hardware-profiles.md`).
|
||||
4. **Success criteria** (all required):
|
||||
- No OOM, no NaN loss, no crash.
|
||||
- All 50 training steps complete.
|
||||
- Validation sample at step 50 generates successfully (validation pass is a real OOM risk — do not skip).
|
||||
- **For audio runs:** the one-sample `audio_latents/` is non-empty (the trainer log should report audio enabled). A joint/audio run that silently produced no audio latents is a failure even if steps complete — see the audio gate in `phases/preprocess-dataset.md`.
|
||||
- **Loss is NOT a success criterion.** Loss can be non-monotonic even when training is healthy.
|
||||
5. **Autotune sweep** — incremental, capped at 5 trials total. Each trial = current best + one change. Stop on OOM (revert), no step-time improvement, or 5 trials:
|
||||
- Trial 2: `quantization: null` (disable transformer quantization) if VRAM headroom allows.
|
||||
- Trial 3: `optimizer_type: adamw` (disable 8-bit optimizer) if headroom allows.
|
||||
- Trial 4: `batch_size` up (1 → 2 → 4). Adjust `gradient_accumulation_steps` proportionally to keep effective batch constant. **Note:** batch size can't be meaningfully tested on the one-sample set — defer it (test on the full set later, or just keep `batch_size: 1`, which is preferable for small concept-LoRA datasets anyway).
|
||||
- **Do not sweep:** resolution (user decision), `acceleration.load_text_encoder_in_8bit` (one-time, no step-time impact), `enable_gradient_checkpointing` (the trainer's example configs ship with it on; on the 80GB+ tier you *may* try it off, but for the 22B model it usually OOMs even with tens of GB of apparent headroom — don't expect a win; never turn it off on the 32GB tier).
|
||||
6. Collect per trial: step time and peak VRAM. **Prefer the trainer's own end-of-run stats** (it prints total time / step time and peak GPU memory) — no external timing tool is needed (`/usr/bin/time` is often not installed). Append results to `<workspace>/<run-name>/autotune.log`.
|
||||
7. Winning trial's deltas are patched into the main `config.yaml`. Summarise the sweep to the user (one line per trial + winner).
|
||||
|
||||
If the baseline trial fails, consult `references/troubleshooting.md`, propose a fix, re-run. Never push forward to the full preprocess after a failed sanity check.
|
||||
|
||||
## Phase 7 — Full Preprocess
|
||||
|
||||
Follow `phases/preprocess-dataset.md`. Re-read it before acting.
|
||||
|
||||
**If `.precomputed/` already exists at the target path** (user-supplied or prior run), the phase verifies shapes and modality coverage before reuse. On mismatch it stops and asks — never silently overwrites.
|
||||
|
||||
## Phase 8 — Launch & Monitor
|
||||
|
||||
Follow `phases/launch-and-monitor.md`. Re-read it before acting. Surface the W&B URL (if enabled) and produce periodic status reports. At completion, the phase writes `<workspace>/<run-name>/outputs/run-summary.md`.
|
||||
|
||||
## Phase 9 — Post-Train Validate
|
||||
|
||||
After training finishes, follow `phases/post-train-validate.md`. Re-read it before acting. The phase renders the final LoRA against in-distribution, out-of-distribution, and held-out prompts; surfaces the MP4 paths to the user and exits.
|
||||
|
||||
**Important constraint:** the post-train validate phase does not solicit a verdict and does not coach on causes for "soft" failures. Soft training quality has no reliable if/then rule book — the user inspects the renders and decides for themselves whether to ship, iterate, or change course. The orchestrator's job ends after Phase 9; iteration is a new invocation with a new run-name.
|
||||
|
||||
## Monitor-Only Entry
|
||||
|
||||
If invoked against an existing `<workspace>/<run-name>/` that already has a training process running or completed, **skip to Phase 8 in monitor-only mode** instead of restarting anything: report current step, recent loss, ETA, checkpoint list, W&B URL, and the resume command. Distinguish "live process" vs "stopped run with checkpoints" and offer the appropriate next action.
|
||||
|
||||
## Escape Hatch: Unsupported Modes
|
||||
|
||||
If Phase 1 intent doesn't map to any combination of supported `flexible`-strategy conditions:
|
||||
|
||||
1. Stop. Do not edit `packages/ltx-trainer/` on your own.
|
||||
2. Explain what's missing in concrete terms: "you want X. The trainer supports A, B, C via conditions D, E, F. X requires a new condition / strategy."
|
||||
3. Identify the code change needed (typically a new `Condition` subclass in `ltx_trainer/training_strategies/flexible.py` plus schema wiring in `config.py`).
|
||||
4. Ask the user explicitly: "Proceed with the code change, do it yourself, or abort?"
|
||||
5. Only on explicit consent, drop out of this skill's orchestrator and edit code as a normal agent task. After the change lands and is tested, return here.
|
||||
|
||||
## Ask-vs-Assume Cheat Sheet
|
||||
|
||||
| Decision | How |
|
||||
|----------|-----|
|
||||
| Precision, quantization, optimizer, grad checkpointing | Assume from matched VRAM tier. List in plan's "Assumptions". |
|
||||
| Checkpoint/validation interval, seed, W&B project name, output dir | Assume sensible defaults. List in "Assumptions". |
|
||||
| Target resolution / frame count | Ask if not given; propose mode-appropriate default. |
|
||||
| Step count | Ask if dataset size doesn't pin it. |
|
||||
| LoRA trigger word / concept name | Ask for the word only (style/concept LoRAs). Never ask *how* it's injected — always `--lora-trigger`. |
|
||||
| Captioner backend | Ask only if multiple backends are viable. |
|
||||
| Anything answerable by `ls`, `nvidia-smi`, or the W&B credential check | Never ask. Probe. |
|
||||
|
||||
## The Two `load_text_encoder_in_8bit` Flags
|
||||
|
||||
Same name, different layers — do not conflate:
|
||||
|
||||
| Flag | Layer | Effect |
|
||||
|------|-------|--------|
|
||||
| `process_dataset.py --load-text-encoder-in-8bit` | Preprocessing CLI | Memory during caption-embedding precompute (Phase 7). One-time per dataset. |
|
||||
| `acceleration.load_text_encoder_in_8bit` (trainer YAML) | Trainer config | Memory during validation-sample prompt-embedding caching at training start (Phase 8). One-time per run. |
|
||||
|
||||
Both are one-time costs and neither affects per-step training speed. Default per the trainer's shipped configs: **ON** on the 32GB tier (matches `t2v_lora_low_vram.yaml`), **OFF** on the 80GB+ tier (matches `t2v_lora.yaml`). No measured guidance for the 40–60GB tier beyond starting from the 32GB tier and autotuning.
|
||||
|
||||
## Workspace Layout
|
||||
|
||||
```
|
||||
<workspace>/<run-name>/
|
||||
plan.md # the approved plan
|
||||
config.yaml # generated training config (NOT in packages/ltx-trainer/configs/)
|
||||
autotune.log # per-trial sweep results
|
||||
dataset/
|
||||
dataset.json # captions + media paths (training split)
|
||||
holdout.jsonl # held-out split (if reserved)
|
||||
videos/ # source media copies (NO derived files here)
|
||||
.precomputed/ # latents/ audio_latents/ conditions/ (+ references/masks per mode)
|
||||
outputs/
|
||||
checkpoints/ # training checkpoints + states
|
||||
samples/ # in-training validation samples (step_*)
|
||||
eval/ # Phase 9: in-distribution/ out-of-distribution/ held-out/ + prompts.json
|
||||
run-summary.md # written at completion
|
||||
logs/ # all run logs
|
||||
overfit/ # Phase 6 scratch (one-sample preprocess + sanity/autotune runs)
|
||||
```
|
||||
|
||||
`<run-name>` default: `<mode>-<dataset-name>-<YYYYMMDD-HHMM>`. Surface in the plan; user may rename.
|
||||
|
||||
### Workspace hygiene (keep it clean)
|
||||
|
||||
- **Don't create undocumented directories** (e.g. an ad-hoc `scratch/`). Intermediates belong under `overfit/` (sanity/autotune scratch) or a `/tmp` tempdir — not loose in the run dir or the repo root.
|
||||
- **Never write derived files into `dataset/videos/`** (the source media dir). Latents go under `.precomputed/`; one-sample/eval metadata goes under `overfit/`, not `dataset/`.
|
||||
- **One canonical manifest per artifact** — don't leave duplicate `*_prompts.json` / metadata copies.
|
||||
- **Clean up phase byproducts:** the Phase 9 eval must delete its validate-only trainer cruft (see `phases/post-train-validate.md`); `overfit/` is scratch and may be removed after a successful run. The final tree should look like the layout above — no stray `.pt`/`.wav` files, no `eval/checkpoints/`, no duplicate manifests.
|
||||
|
||||
## References
|
||||
|
||||
- `references/mode-selector.md` — user intent → conditioning mode mapping, LoRA rank guidance (read in Phase 1).
|
||||
- `references/onboarding.md` — first-run prerequisite checklist, model download paths, captioner graceful degradation (read in Phase 2).
|
||||
- `references/hardware-profiles.md` — GPU VRAM → tier + config defaults (read in Phase 2).
|
||||
- `references/config-patching.md` — safe YAML edits + schema constraints (read whenever editing `config.yaml`).
|
||||
- `references/troubleshooting.md` — OOM, NaN, validation failures, resume (read on any failure).
|
||||
- `references/plan-template.md` — exact plan.md format (read in Phase 4).
|
||||
|
||||
## Phase Procedures
|
||||
|
||||
These are procedure documents the orchestrator reads when entering each phase. They are not standalone skills — they're never invoked by Claude's skill-discovery system. The orchestrator opens them via the `Read` tool and follows the instructions inline.
|
||||
|
||||
- `phases/prepare-dataset.md` — Phase 5: scenes, captioner iteration, IC-LoRA references, metadata, holdout split.
|
||||
- `phases/preprocess-dataset.md` — Phases 6 (one-sample) & 7 (full): `process_dataset.py` orchestration, existing-data verification.
|
||||
- `phases/launch-and-monitor.md` — Phase 8: launch command, accelerate, W&B, status reports, run-summary writing.
|
||||
- `phases/post-train-validate.md` — Phase 9: render the final LoRA against three prompt categories; surface paths only.
|
||||
@@ -0,0 +1,235 @@
|
||||
# Phase 8 — Launch & Monitor
|
||||
|
||||
Procedure document for the `train-model` orchestrator (Phase 8 + monitor-only re-entry). Read this file in full before acting on launch/monitor.
|
||||
|
||||
Goal: start the training job, surface the W&B URL, produce periodic status reports, write `run-summary.md` at completion.
|
||||
|
||||
The orchestrator's hard invariants apply (see `../SKILL.md`).
|
||||
|
||||
## Launch — Single GPU
|
||||
|
||||
```bash
|
||||
cd packages/ltx-trainer
|
||||
uv run python scripts/train.py "<workspace>/<run-name>/config.yaml"
|
||||
```
|
||||
|
||||
## Launch — Multi-GPU
|
||||
|
||||
Use Accelerate. For LoRA, DDP (default) is fine. For full FT, use FSDP.
|
||||
|
||||
```bash
|
||||
cd packages/ltx-trainer
|
||||
|
||||
# DDP (LoRA, multi-GPU)
|
||||
uv run accelerate launch scripts/train.py "<workspace>/<run-name>/config.yaml"
|
||||
|
||||
# FSDP (full FT, multi-GPU)
|
||||
uv run accelerate launch \
|
||||
--config_file configs/accelerate/fsdp.yaml \
|
||||
scripts/train.py "<workspace>/<run-name>/config.yaml"
|
||||
```
|
||||
|
||||
**Pass `--disable-progress-bars` to `train.py` whenever stdout is redirected to a log file** (every background run) or running multi-GPU. The Rich progress bar rewrites a single line with carriage returns and does **not** flush parseable newlines to a redirected log, so without this flag the log shows no step/loss lines and you're forced to poll `nvidia-smi`. With it, step/loss lines are written normally and the log is greppable.
|
||||
|
||||
## Pre-Launch Checks (every launch)
|
||||
|
||||
1. Run the self-check from `references/config-patching.md` (paths exist, frame/resolution constraints, generated modalities have matching latents dirs, references/masks dirs present for conditional modes).
|
||||
2. Confirm `nvidia-smi` shows expected GPUs available (not occupied by another process).
|
||||
3. If `wandb.enabled: true`, confirm credentials still resolve: `uv run python -c "import wandb; print(bool(wandb.Api().api_key))"` → `True`. (Don't use `wandb status`.) If `False`, surface to the user before launching — they may want to `wandb login` or run without tracking.
|
||||
|
||||
## Run In Background, Monitor Foreground
|
||||
|
||||
Long training runs should not block the agent's response loop, and they must survive past the launching turn.
|
||||
|
||||
**Prefer the agent's managed/native background-shell mechanism** (the harness facility for long-running background commands — output streaming + PID/exit tracking, survives across turns). It's the reliable way to launch training: it stays alive, streams to a log the agent can poll, and reports completion. Launch the training command through that mechanism, writing to `<workspace>/<run-name>/logs/train.log` with `--disable-progress-bars`, using **absolute paths** for the config and log (`uv run --directory packages/ltx-trainer` changes the cwd, so a relative config path won't resolve).
|
||||
|
||||
**`nohup ... &` is a last-resort fallback only.** Detached jobs can be harder to track and may not survive environment/session cleanup, so only use it if no managed background mechanism is available, and verify the PID is still alive afterward:
|
||||
|
||||
```bash
|
||||
# Fallback ONLY — prefer the managed background shell above.
|
||||
mkdir -p "<workspace>/<run-name>/logs"
|
||||
nohup uv run --directory packages/ltx-trainer python scripts/train.py \
|
||||
"<ABSOLUTE path>/<run-name>/config.yaml" --disable-progress-bars \
|
||||
> "<ABSOLUTE path>/<run-name>/logs/train.log" 2>&1 &
|
||||
echo $! > "<workspace>/<run-name>/logs/train.pid"
|
||||
```
|
||||
|
||||
## Status Report
|
||||
|
||||
Produce on user request (or every <interval> automatically). Pull from:
|
||||
|
||||
- **W&B run URL:** First lines of `train.log` after init, or `wandb.run.url` from a `wandb` Python snippet. Surface as a clickable URL.
|
||||
- **Latest step:** `tail -n 200 "<workspace>/<run-name>/logs/train.log" | grep -oE "step [0-9]+" | tail -1`.
|
||||
- **Recent loss:** `tail -n 200 "<workspace>/<run-name>/logs/train.log" | grep -oE "loss[: ]+[0-9.]+" | tail -5`.
|
||||
- **Checkpoints saved:** `ls -1t "<workspace>/<run-name>/outputs/checkpoints/" 2>/dev/null`.
|
||||
- **Validation samples:** `ls -1t "<workspace>/<run-name>/outputs/samples/" 2>/dev/null`.
|
||||
- **GPU utilization snapshot:** `nvidia-smi --query-gpu=name,utilization.gpu,memory.used,memory.total --format=csv,noheader`.
|
||||
- **ETA:** only report one **grounded in real numbers the trainer has actually emitted** — do not invent or "educated-guess" an ETA before the trainer has produced per-step timings. Compute as `(total_steps - current_step) * recent_avg_step_time`, where `recent_avg_step_time` is measured from **steady-state training steps** (the trainer's reported per-step time or log timestamps), **excluding** one-time setup that doesn't repeat per step: model loading, the step-0 validation pass, and periodic validation passes. The trainer's own early ETA projection is skewed high by the slow step-0 validation and settles after a few steps — wait for it to settle rather than quoting the inflated early figure. Until real step timings exist, say "measuring step time…" rather than guessing a duration.
|
||||
|
||||
Format the report tightly — one block, no fluff:
|
||||
|
||||
```
|
||||
Step 1240 / 2000 (62%) — loss ~0.072 (last 5: 0.071, 0.073, 0.069, 0.075, 0.072)
|
||||
ETA: ~1h 24m | GPU: 91% util, 39.2 / 48.0 GB
|
||||
Latest checkpoint: lora_weights_step_01000.safetensors
|
||||
Latest validation: samples/step_01200_*.mp4
|
||||
W&B: https://wandb.ai/<entity>/<project>/runs/<id>
|
||||
```
|
||||
|
||||
## Monitor-Only Mode
|
||||
|
||||
Invoked when the orchestrator detects an existing `<workspace>/<run-name>/` with checkpoints or a running process.
|
||||
|
||||
1. Check if the training process is live: `[ -f .../logs/train.pid ] && kill -0 $(cat .../logs/train.pid) 2>/dev/null && echo LIVE || echo STOPPED`.
|
||||
2. Produce the same status report as above.
|
||||
3. If STOPPED:
|
||||
- Compute step from last checkpoint.
|
||||
- Find the latest checkpoint pair (`lora_weights_step_*.safetensors` or `model_weights_step_*.safetensors`, plus a matching `training_state_step_*.pt` when resume state is enabled) under `<workspace>/<run-name>/outputs/checkpoints/`.
|
||||
- Patch `model.load_checkpoint` in `config.yaml` to point at that checkpoint file (this is the only way the trainer knows to resume — there's no auto-detection from `output_dir`).
|
||||
- Surface the resume command: `uv run python scripts/train.py "<workspace>/<run-name>/config.yaml"`.
|
||||
- Ask the user to confirm both the patch and the launch before applying.
|
||||
|
||||
## Resume
|
||||
|
||||
The trainer **does not auto-resume from `output_dir`**. To resume an interrupted run:
|
||||
|
||||
1. Set `model.load_checkpoint` in `config.yaml` to the latest checkpoint file (e.g. `<workspace>/<run-name>/outputs/checkpoints/lora_weights_step_02000.safetensors`).
|
||||
2. Launch normally. The trainer loads those weights, then looks for a matching `training_state_step_*.pt` **next to the loaded checkpoint** and restores optimizer/scheduler/step state from it. If the state file is missing, weights load but training starts from step 0.
|
||||
3. To load the weights but skip the state restore (e.g. for branching off into a new run from a known-good checkpoint), set `checkpoints.no_resume: true`.
|
||||
|
||||
Always ask the user before patching `model.load_checkpoint` or setting `no_resume` — checkpoints are precious.
|
||||
|
||||
## Failure During Training
|
||||
|
||||
If the training process exits non-zero:
|
||||
|
||||
1. Tail the log and identify the error type.
|
||||
2. Cross-reference `references/troubleshooting.md`.
|
||||
3. Propose a config fix (with the exact diff to `config.yaml`).
|
||||
4. Ask the user before applying. Then resume with the patched config.
|
||||
|
||||
Never silently restart a failed training run without acknowledging the failure to the user.
|
||||
|
||||
## After Training Completes
|
||||
|
||||
1. Write a **run summary** to `<workspace>/<run-name>/outputs/run-summary.md` (see below) so the user can find their bearings months later without rereading the trainer docs.
|
||||
2. Show final checkpoint path and step count.
|
||||
3. Show W&B URL if enabled.
|
||||
4. Return control to the orchestrator (Phase 9 — post-train validate runs next).
|
||||
|
||||
### Writing `run-summary.md`
|
||||
|
||||
The summary is the **landing page** for this run. Anyone (including the user months from now) should be able to read it and understand: what was trained, on what data, with what config, where everything lives, how to use the result, and how to continue. The trainer doesn't produce this — the skill does.
|
||||
|
||||
Template (fill from `plan.md`, `config.yaml`, `autotune.log`, dataset metadata, training log):
|
||||
|
||||
```markdown
|
||||
# <run-name>
|
||||
|
||||
**Trained:** <YYYY-MM-DD HH:MM> on <GPU(s)>
|
||||
**Final checkpoint:** `outputs/checkpoints/<filename>.safetensors` (step <N>)
|
||||
|
||||
## What this LoRA does
|
||||
|
||||
<One paragraph from the plan's Goal section — restating the user's intent.>
|
||||
|
||||
## Trigger word
|
||||
|
||||
`<trigger>` — include in prompts at inference time. (Omit this section if no trigger word.)
|
||||
|
||||
## Mode
|
||||
|
||||
<Mode name> (<lora|full>). Conditioning: <list of conditions, or "none">.
|
||||
|
||||
## Dataset
|
||||
|
||||
- Source: `<absolute path>`
|
||||
- Captioning: <`qwen_omni` (Qwen3-Omni-30B via vLLM) | `gemini_flash` | "user-supplied">
|
||||
- Captioner instruction used: <verbatim string, or "default">
|
||||
- Samples: <N training> + <K held-out> at <W>x<H>x<F>
|
||||
- Preprocessed to: `dataset/.precomputed/`
|
||||
|
||||
## Training config
|
||||
|
||||
Final values after autotune (deltas from baseline noted in `autotune.log`):
|
||||
|
||||
| Field | Value |
|
||||
|-------|-------|
|
||||
| Optimizer | <...> |
|
||||
| Mixed precision | <bf16/fp16> |
|
||||
| Quantization | <...> |
|
||||
| Gradient checkpointing | <on/off> |
|
||||
| Batch size × grad accum | <B> × <A> (effective <BxA>) |
|
||||
| LoRA rank / alpha | <R> / <A> (or "full FT") |
|
||||
| LoRA target modules | <list> (or "n/a") |
|
||||
| Steps | <N> |
|
||||
| Learning rate | <value> |
|
||||
| Step time (final) | ~<T>s |
|
||||
| Peak VRAM | ~<V> GB |
|
||||
|
||||
Full config: `<workspace>/<run-name>/config.yaml`
|
||||
|
||||
## Outputs
|
||||
|
||||
- Checkpoints: `outputs/checkpoints/`
|
||||
- Validation samples (during training): `outputs/samples/`
|
||||
- Post-train eval renders (if Phase 9 ran): `outputs/eval/`
|
||||
- W&B run: <url, or "(W&B not enabled)">
|
||||
|
||||
## How to use this checkpoint
|
||||
|
||||
For inference, point `packages/ltx-pipelines/` at the final checkpoint. Example invocation:
|
||||
|
||||
\`\`\`bash
|
||||
# (Minimal sketch — adapt to the pipeline you're using.)
|
||||
# load base LTX-2 model + apply this LoRA from outputs/checkpoints/<filename>.safetensors
|
||||
\`\`\`
|
||||
|
||||
## How to continue training
|
||||
|
||||
To resume from the final checkpoint (e.g. more steps, different LR), edit `config.yaml`:
|
||||
|
||||
\`\`\`yaml
|
||||
model:
|
||||
load_checkpoint: "<absolute path to outputs/checkpoints/<filename>.safetensors>"
|
||||
optimization:
|
||||
steps: <new total> # trainer resumes optimizer/scheduler/step from the training_state_step_*.pt sitting next to the checkpoint above
|
||||
\`\`\`
|
||||
|
||||
Then re-launch with the same command in the "Launched with" section below.
|
||||
|
||||
## How this was launched
|
||||
|
||||
\`\`\`
|
||||
<exact command used, with the workspace's absolute config path>
|
||||
\`\`\`
|
||||
|
||||
## Reproducibility
|
||||
|
||||
- Seed: <value>
|
||||
- Workspace: `<absolute path>`
|
||||
- Repo commit at launch: `<git rev-parse HEAD output>`
|
||||
- LTX-2 model: `<model.model_path from config>`
|
||||
- Text encoder: `<model.text_encoder_path from config>`
|
||||
```
|
||||
|
||||
Write the file using the `Write` tool. Don't embed it in a heredoc — the markdown nested in this skill is illustrative; fill the template with real values from the run's artifacts.
|
||||
|
||||
### Next steps (surface to user)
|
||||
|
||||
After writing the summary, point the user at:
|
||||
- The summary file path.
|
||||
- The final checkpoint path.
|
||||
- The W&B URL if enabled.
|
||||
- The upcoming Phase 9 (post-train validate) — the orchestrator handles the transition.
|
||||
|
||||
Suggest, but don't run:
|
||||
- Test inference with `packages/ltx-pipelines/`.
|
||||
- Push to HF Hub via the trainer's `hub.push_to_hub` config (a separate, lightweight re-launch).
|
||||
- Continue training from the final checkpoint (see summary's "How to continue training" section).
|
||||
|
||||
## Do Not
|
||||
|
||||
- Do not modify `output_dir` contents after a run completes — checkpoints belong to the user now.
|
||||
- Do not start a second training run into the same `output_dir` without explicit user approval. Resume requires patching `model.load_checkpoint` (the trainer does not auto-detect prior checkpoints); a true fresh-start from the same dir additionally needs `checkpoints.no_resume: true`.
|
||||
- Do not auto-restart a failed run without diagnosis and user approval.
|
||||
@@ -0,0 +1,171 @@
|
||||
# Phase 9 — Post-Train Validate
|
||||
|
||||
Procedure document for the `train-model` orchestrator (Phase 9). Read this file in full before acting on post-train validation.
|
||||
|
||||
Goal: render the final checkpoint against three prompt categories so the user can inspect the result and form their own judgement. Save outputs in an organized layout. **Do not** prompt for pass/fail verdicts, do not infer causes for failures, do not suggest fixes — soft training failures don't have a clean if/then rule book, and pretending otherwise wastes the user's time.
|
||||
|
||||
The orchestrator's hard invariants apply (see `../SKILL.md`).
|
||||
|
||||
## What this phase does
|
||||
|
||||
1. Collect prompts for three categories.
|
||||
2. Render the final LoRA against all collected prompts.
|
||||
3. Save outputs under `<workspace>/<run-name>/outputs/eval/<category>/`.
|
||||
4. Print the paths and exit.
|
||||
|
||||
## Categories
|
||||
|
||||
### 1 — In-distribution
|
||||
|
||||
A few captions from the training set itself. Tests whether the model learned what it was shown.
|
||||
|
||||
- Default: 3 random captions from the dataset metadata (seed 42 for reproducibility).
|
||||
- Source: `<workspace>/<run-name>/dataset/dataset.json` (the captions used for training, after the held-out split).
|
||||
|
||||
### 2 — Out-of-distribution
|
||||
|
||||
Prompts the model has never seen, but in the same domain. Tests whether the model generalizes the concept beyond memorized phrasings.
|
||||
|
||||
Ask the user once:
|
||||
|
||||
> "For out-of-distribution validation, paste 2–3 prompts you'd realistically want to generate at inference time. (If you don't have any specific ones in mind, reply 'default' — I'll use a few generic prompts that include the trigger word.)"
|
||||
|
||||
- On `default`: synthesize 3 short prompts using the LoRA's trigger word and a generic scene context (e.g., "<trigger> walking in a forest at dawn"). Note these are generic — they're better than nothing, but real user-style prompts make a stronger test.
|
||||
- Otherwise: use the user's prompts verbatim.
|
||||
- **For a run with a generated audio modality, the synthesized prompts must describe the audio** — matching how the training captions describe it (inspect a few from `dataset.json` first). A prompt with no audio direction leaves the audio branch unguided and the generated audio comes out poor. E.g. for the talking-head case include spoken-voice/room-tone direction; for music/ambience/foley describe the sound character. (Categories 1 and 3 reuse the real captions verbatim, so they already carry audio description — this only applies to the synthesized Category 2 prompts.) If the user pasted their own prompts and it's an audio run, and they omitted audio direction, note that the audio may be weak without it.
|
||||
|
||||
### 3 — Held-out
|
||||
|
||||
Captions from samples that were never seen during training. Tests true generalization, not memorization.
|
||||
|
||||
- Source: `<workspace>/<run-name>/dataset/holdout.jsonl` (written by `prepare-dataset` Step 5).
|
||||
- Use all entries if there are ≤5; otherwise sample 5 with seed 42.
|
||||
- **If holdout doesn't exist** (dataset was too small, or user-skipped during prepare): print a clear note in the output summary — *"Held-out evaluation skipped: no holdout set was reserved for this run. The post-train eval only covers Categories 1 and 2."* Don't synthesize substitutes.
|
||||
|
||||
## Rendering mechanism
|
||||
|
||||
Use the trainer's existing validation infrastructure rather than wiring up `ltx-pipelines` from scratch. Create a temporary "validate-only" config and run the trainer with it.
|
||||
|
||||
### Step 1 — Build eval config
|
||||
|
||||
Copy `<workspace>/<run-name>/config.yaml` to `<workspace>/<run-name>/eval-config.yaml`. Patch:
|
||||
|
||||
```yaml
|
||||
model:
|
||||
load_checkpoint: "<absolute path to outputs/checkpoints/<final-lora>.safetensors>"
|
||||
|
||||
optimization:
|
||||
steps: 1 # we don't want to train; we want validation to fire
|
||||
# Keep batch_size/grad_accum at the run's autotuned values to match its VRAM footprint.
|
||||
|
||||
validation:
|
||||
skip_initial_validation: false
|
||||
interval: 1 # run validation at step 0 (and at the only training step)
|
||||
samples:
|
||||
# Inject all collected prompts here, tagged by category in the prompt itself
|
||||
# so the output filenames make the category obvious.
|
||||
- prompt: "[CAT1-IND] <caption from dataset.json>"
|
||||
# ... repeat for each prompt in all three categories
|
||||
# Keep video_dims, frame_rate, guidance/STG settings as the trained config.
|
||||
# Keep generate_audio consistent with the trained modality config.
|
||||
|
||||
checkpoints:
|
||||
interval: null # do not save more checkpoints
|
||||
no_resume: true # load the LoRA's weights but do not restore optimizer/scheduler/step state
|
||||
|
||||
output_dir: "<workspace>/<run-name>/outputs/eval"
|
||||
```
|
||||
|
||||
The `[CAT1-IND]`, `[CAT2-OOD]`, `[CAT3-HELDOUT]` tags in the prompt strings make the output MP4 filenames self-describing in the trainer's validation sample directory.
|
||||
|
||||
**Attach the mode's conditions to each sample.** A bare `prompt` only validates a pure text-to-X mode (T2V, T2A). For any conditioned mode, the trained model expects the same conditioning at validation time — a prompt with no conditions tests a different task than what was trained, and conditioned modes may fail outright. Add the `conditions` list that matches the run's mode (the trained `config.yaml` `training_strategy` and the example config for the mode are the reference):
|
||||
|
||||
| Mode | Add to each sample |
|
||||
|------|--------------------|
|
||||
| I2V | `conditions: [{type: first_frame, image_or_video: <frame/clip path>}]` |
|
||||
| Video extension / suffix | `conditions: [{type: prefix|suffix, ...}]` |
|
||||
| V2V / AV2AV IC-LoRA | `conditions: [{type: reference, ...}]` (point at a held-out reference) |
|
||||
| V2A (foley) | `conditions: [{type: video_to_audio, ...}]` |
|
||||
| A2V | `conditions: [{type: audio_to_video, ...}]` |
|
||||
| Inpainting (video/audio) | `conditions: [{type: mask, ...}]` |
|
||||
| Outpainting | `conditions: [{type: spatial_crop, ...}]` |
|
||||
| A2A IC-LoRA | `conditions: [{type: reference, ...}]` (held-out reference audio) |
|
||||
| T2V, T2A | none — a bare `prompt` is correct |
|
||||
|
||||
Mirror the condition shapes used in the mode's example config under `packages/ltx-trainer/configs/`. For held-out (Category 3) and OOD (Category 2) samples on conditioned modes, draw the conditioning media from the held-out set so the eval stays out-of-distribution.
|
||||
|
||||
### Step 2 — Run the trainer in validate-only mode
|
||||
|
||||
```bash
|
||||
cd packages/ltx-trainer
|
||||
uv run python scripts/train.py "<workspace>/<run-name>/eval-config.yaml"
|
||||
```
|
||||
|
||||
The trainer will load the LoRA, run initial validation against all the prompts, do one trivial training step (which we discard), and exit. Validation samples land in `<workspace>/<run-name>/outputs/eval/samples/`.
|
||||
|
||||
### Step 3 — Organize outputs and clean up trainer cruft
|
||||
|
||||
The validate-only run is a trainer run, so it inevitably writes throwaway artifacts: an indexed `samples/` dir, a forced final checkpoint (the trainer **always** saves one at the end, regardless of `checkpoints.interval`), and a `training_config.yaml`. Don't leave these around or duplicate the renders.
|
||||
|
||||
1. **Move** (don't copy) each generated MP4 from the trainer's indexed `samples/` dir into the category layout, naming by category + a slug of the prompt. Use the index→category mapping you built when constructing `validation.samples`.
|
||||
2. **Delete the trainer cruft** from the eval dir once the renders are moved: the indexed `samples/` dir, the forced `checkpoints/` dir, and `training_config.yaml`. (These are byproducts of the validate-only hack — there's no config flag to suppress the final-checkpoint save, so clean it up here.)
|
||||
3. Write **one** manifest, `outputs/eval/prompts.json` (filename → full prompt + category). Don't leave a second copy elsewhere.
|
||||
4. Remove the temporary `eval-config.yaml` (or keep it under the run's scratch, not in `outputs/`).
|
||||
|
||||
Final eval layout — exactly this, nothing else:
|
||||
|
||||
```
|
||||
<workspace>/<run-name>/outputs/eval/
|
||||
in-distribution/ <NN>_<prompt-slug>.mp4 ...
|
||||
out-of-distribution/ <NN>_<prompt-slug>.mp4 ...
|
||||
held-out/ <NN>_<prompt-slug>.mp4 ... # only if a holdout set existed
|
||||
prompts.json # filename -> full prompt + category (single manifest)
|
||||
```
|
||||
|
||||
No `eval/samples/`, no `eval/checkpoints/`, no `eval/training_config.yaml`, no duplicate manifest.
|
||||
|
||||
### Step 4 — Surface paths
|
||||
|
||||
Print a tight block, no judgement, no follow-up question:
|
||||
|
||||
```
|
||||
Post-train evaluation complete.
|
||||
|
||||
In-distribution renders (<K> samples):
|
||||
<workspace>/<run-name>/outputs/eval/in-distribution/
|
||||
|
||||
Out-of-distribution renders (<M> samples):
|
||||
<workspace>/<run-name>/outputs/eval/out-of-distribution/
|
||||
|
||||
Held-out renders (<N> samples):
|
||||
<workspace>/<run-name>/outputs/eval/held-out/ # or: "(skipped — no holdout set)"
|
||||
|
||||
Open the MP4s and decide for yourself whether the model is good. Soft
|
||||
training quality is judged by watching the videos, not by a checklist —
|
||||
there's no substitute for your own eyes here.
|
||||
```
|
||||
|
||||
Return control to the orchestrator. The orchestrator's run is now complete.
|
||||
|
||||
## What this phase does NOT do
|
||||
|
||||
- Does not ask "is this good?" / "pass / partial / fail?".
|
||||
- Does not infer failure causes.
|
||||
- Does not suggest fixes, follow-up runs, hyperparameter changes, dataset changes.
|
||||
- Does not write any verdict to `run-summary.md` or elsewhere.
|
||||
- Does not delete or modify training checkpoints.
|
||||
- Does not push to any remote / cloud / registry.
|
||||
|
||||
The user looks at the videos and makes their own call. If they want to iterate, they re-invoke the orchestrator with a new run-name.
|
||||
|
||||
## Failure modes
|
||||
|
||||
- **Final checkpoint missing.** Surface and stop. Don't render against an intermediate checkpoint without explicit user consent.
|
||||
- **`load_checkpoint` OOM at inference time.** Lower `validation.video_dims` in the eval config (smaller renders are still useful for a sanity look). Retry once. If still OOM, surface the failure and let the user run inference manually via `packages/ltx-pipelines/`.
|
||||
- **All renders look broken/black.** May be an inference-pipeline-side issue rather than a training failure. Mention in the output block: *"If renders look broken across all categories, try `packages/ltx-pipelines/` directly to rule out a pipeline issue."* Then exit. Do not investigate further.
|
||||
|
||||
## Do not
|
||||
|
||||
- Do not skip Category 1 or 2. They're cheap and informative.
|
||||
- Do not invent a held-out set if `holdout.jsonl` is missing — the prepare-dataset step decides that.
|
||||
- Do not coach the user on what "good" means for their use case.
|
||||
@@ -0,0 +1,245 @@
|
||||
# Phase 5 — Prepare Dataset
|
||||
|
||||
Procedure document for the `train-model` orchestrator (Phase 5). Read this file in full before acting on the prepare-dataset phase.
|
||||
|
||||
Goal: produce a captioned, complete dataset metadata file at `<workspace>/<run-name>/dataset/dataset.json` consumable by `process_dataset.py`. Idempotent — re-runs skip work already done.
|
||||
|
||||
The orchestrator's hard invariants apply (see `../SKILL.md`), especially: **no file mutation outside the workspace without explicit user approval.**
|
||||
|
||||
## Inputs
|
||||
|
||||
The orchestrator passes:
|
||||
- Source path (directory of videos / single video / pre-existing metadata file).
|
||||
- Target mode (T2V, I2V, V2V IC-LoRA, V2A, etc.) — determines which columns are required.
|
||||
- Captioner backend choice (Qwen3-Omni local vLLM server / Gemini Flash cloud / skip).
|
||||
- Workspace path `<workspace>/<run-name>/`.
|
||||
|
||||
## Required Columns by Mode
|
||||
|
||||
`process_dataset.py` detects columns by convention and resolves each to a role. The media column may be `video` **or** `audio`. When a dataset has a `video` column with an audio track and no separate `audio` column, audio is **auto-extracted** from the video (unless `--skip-audio`), so an explicit `audio` column is only needed when the audio lives in separate files.
|
||||
|
||||
**Video-generating modes** (need a `video` column):
|
||||
|
||||
| Mode | Required | Optional |
|
||||
|------|----------|----------|
|
||||
| T2V, I2V, video extension/suffix | `video`, `caption` | `audio` (else auto-extracted) |
|
||||
| Video outpainting | `video`, `caption` | |
|
||||
| Video inpainting | `video`, `caption`, `video_mask` | |
|
||||
| V2A (foley) | `video`, `caption` | `audio` (target; else auto-extracted) |
|
||||
| A2V | `video`, `caption` | `audio` (else auto-extracted from video) |
|
||||
| V2V IC-LoRA | `video`, `caption`, `reference_video` | |
|
||||
| AV2AV IC-LoRA | `video`, `caption`, `reference_video`, `reference_audio` | `audio` (else auto-extracted) |
|
||||
|
||||
**Audio-only modes** (no `video` column — the media column is `audio`):
|
||||
|
||||
| Mode | Required | Optional |
|
||||
|------|----------|----------|
|
||||
| T2A | `audio`, `caption` | |
|
||||
| Audio extension/suffix | `audio`, `caption` | |
|
||||
| Audio inpainting | `audio`, `caption`, `audio_mask` | |
|
||||
| A2A IC-LoRA | `audio`, `caption`, `reference_audio` | |
|
||||
|
||||
Aliases: `media_path` for `video`, `ref_media_path` for `reference_video`.
|
||||
|
||||
## Workflow
|
||||
|
||||
### Step 1 — Classify source
|
||||
|
||||
```bash
|
||||
# If source is a file, identify type:
|
||||
file "<source>"
|
||||
# If source is a directory, count media:
|
||||
find "<source>" -maxdepth 1 -type f \( -name "*.mp4" -o -name "*.mov" -o -name "*.webm" \) | wc -l
|
||||
```
|
||||
|
||||
Cases:
|
||||
- **Pre-existing metadata file** (CSV/JSON/JSONL) → copy to `<workspace>/<run-name>/dataset/dataset.json`, audit columns. Skip to Step 4.
|
||||
- **Directory of short scenes** → skip Step 2, go to Step 3.
|
||||
- **Directory containing long videos** → run Step 2.
|
||||
- **Single long video** → run Step 2.
|
||||
|
||||
**Stage the media under `dataset/` before captioning — don't discover this by failing.** Both `caption_videos.py` and `process_dataset.py` reference media by paths **relative to the metadata file's own directory**, so the media must live under `<workspace>/<run-name>/dataset/`. Stage it up front into `dataset/videos/` (and write metadata paths relative to `dataset/`, e.g. `videos/1.mp4`):
|
||||
- Prefer **symlinks** (instant, no disk cost): `ln -s <abs-source>/<clip> <workspace>/<run-name>/dataset/videos/<clip>`. Symlinks pointing at the original source location work correctly.
|
||||
- Use a **copy** instead if the workspace and source are on different filesystems or the source may move/change.
|
||||
- Never caption or preprocess directly against an out-of-tree source path (e.g. `/path/to/source-videos`) — it will fail the relative-path resolution. The original source is left untouched either way.
|
||||
(Scene-split output in Step 2 already lands under `dataset/scenes/`, which satisfies this.)
|
||||
|
||||
### Step 2 — Scene splitting (only for long videos)
|
||||
|
||||
`split_scenes.py` takes **one video file** at a time (`video_path` and `output_dir` are both positional arguments). When the source is a directory of long videos, iterate over each file. To drop scenes shorter than 2 seconds, use `--filter-shorter-than 2s` (the `--min-scene-length` option is an integer **frame** count, not seconds — don't pass a float).
|
||||
|
||||
```bash
|
||||
cd packages/ltx-trainer
|
||||
# Single file:
|
||||
uv run python scripts/split_scenes.py \
|
||||
"<video-file>" \
|
||||
"<workspace>/<run-name>/dataset/scenes" \
|
||||
--filter-shorter-than 2s
|
||||
|
||||
# Directory of long videos — iterate:
|
||||
for f in "<source>"/*.mp4 "<source>"/*.mov "<source>"/*.webm; do
|
||||
[ -e "$f" ] || continue
|
||||
uv run python scripts/split_scenes.py "$f" \
|
||||
"<workspace>/<run-name>/dataset/scenes" \
|
||||
--filter-shorter-than 2s
|
||||
done
|
||||
```
|
||||
|
||||
Result: scenes saved to `<workspace>/<run-name>/dataset/scenes/`. Pass that directory to Step 3.
|
||||
|
||||
### Step 3 — Captioning
|
||||
|
||||
Skip entirely if a metadata file with all required `caption` entries already exists.
|
||||
|
||||
**Use the captioner's default instruction.** `caption_videos.py` ships a well-tuned default caption prompt — use it as-is (do **not** pass `--instruction`). Captioning runs in two phases: a small **spot-check pass** so the user can confirm the captions look sane, then a **full pass** on the rest.
|
||||
|
||||
A custom `--instruction` is the exception, not the norm. Only use one when:
|
||||
- the **nature of the dataset genuinely demands it** (e.g. a narrow domain the default prompt won't describe well), or
|
||||
- the **user, after seeing the spot-check captions, explicitly asks** for a change (e.g. "too much background detail").
|
||||
|
||||
Do not invent a custom instruction pre-emptively, and in particular **do not bake a subject name / trigger word into the captions via `--instruction`** — the trigger word is handled separately at preprocessing (see "Trigger word" below).
|
||||
|
||||
#### Choosing a backend
|
||||
|
||||
Two backends, with very different hardware needs:
|
||||
|
||||
- **`qwen_omni` (local, default):** Qwen3-Omni-30B-A3B-Thinking served by a local vLLM HTTP server (`serve_captioner.py`). ~65 GiB model download. Default **FP8** quantization uses ~31 GiB of weights and **fits on a 40 GiB GPU** (plus KV cache); **bf16** uses ~60 GiB and needs **≥66 GiB free VRAM**.
|
||||
- **`gemini_flash` (cloud):** Google `gemini-3.5-flash`. No local model, runs anywhere, parallelisable with `--num-workers`.
|
||||
|
||||
**Steer modest hardware to Gemini.** If the GPU is below ~40 GiB (i.e. typical consumer cards — 24 GB / 32 GB), it can't host even the FP8 server, so the local captioner isn't an option — recommend `gemini_flash` and tell the user they'll need Gemini auth: either a `GEMINI_API_KEY`/`GOOGLE_API_KEY` (get one at <https://aistudio.google.com/apikey>) or working gcloud/Vertex AI credentials. If they can't or won't set that up and the hardware can't run Qwen3, the only remaining path is bringing their own captions in the dataset metadata (skip captioning entirely). On a 40 GiB+ GPU the local server is viable (FP8); bf16 needs an 80GB-class card.
|
||||
|
||||
#### Qwen server prerequisite (qwen_omni only)
|
||||
|
||||
The local backend talks to a vLLM server that must already be running. Launch it once in a **separate terminal** (it stays loaded across captioning runs):
|
||||
|
||||
```bash
|
||||
cd packages/ltx-trainer
|
||||
uv run python scripts/serve_captioner.py # FP8 by default, serves on http://127.0.0.1:8001/v1
|
||||
# bf16 (needs >= 66 GiB free VRAM): --quantization bf16
|
||||
# different port/interface: --port 9000 --host 0.0.0.0
|
||||
```
|
||||
|
||||
First launch downloads the model (~65 GiB). `caption_videos.py` reaches it via `--vllm-url` (default `http://127.0.0.1:8001/v1`). Skip this entirely when using `gemini_flash`.
|
||||
|
||||
#### 3a — Spot-check pass (3 samples)
|
||||
|
||||
Caption 3 samples with the **default prompt** (no `--instruction`) to confirm the captioner is producing sane output before committing to the whole set.
|
||||
|
||||
```bash
|
||||
cd packages/ltx-trainer
|
||||
# qwen_omni (server from the previous step must be running):
|
||||
uv run python scripts/caption_videos.py \
|
||||
"<workspace>/<run-name>/dataset/videos/<one-staged-clip>" \
|
||||
--output "<workspace>/<run-name>/dataset/preview-captions.json" \
|
||||
--captioner-type qwen_omni
|
||||
# Point at 3 staged clips under dataset/videos/ (a small subdir or 3 explicit files) — not the out-of-tree source.
|
||||
# Optional: --vllm-url http://127.0.0.1:9000/v1 (if the server uses a non-default port)
|
||||
```
|
||||
|
||||
Print the 3 captions **in full** to the user, then **STOP and wait** for their explicit verdict:
|
||||
|
||||
> "Here are sample captions from the default prompt. Please review them — reply 'good' to caption the rest, or tell me what to change."
|
||||
|
||||
**This is a hard gate. Do NOT caption the full set until the user explicitly approves the samples.** Do not auto-proceed, do not assume "looks fine," do not batch this with other questions. The user must either approve or give tuning instructions first — the whole point of the spot-check is to let them judge caption quality and content before paying for the full pass.
|
||||
|
||||
If the user requests changes, introduce a custom `--instruction` (or switch backend), re-run the spot-check on the same 3 samples, show the new captions, and **stop for approval again**. Loop until the user approves. If a custom instruction still isn't converging after a few rounds, switch captioner backend or have the user supply a few manual captions as examples — but still don't proceed to the full set without their OK.
|
||||
|
||||
#### 3b — Full pass
|
||||
|
||||
Run on the **staged media dir** (`dataset/videos/` from Step 1, or `dataset/scenes/` from Step 2) with the **default prompt** (or the same `--instruction` only if one was explicitly agreed in 3a). Caption the staged in-tree media — not the original out-of-tree source path.
|
||||
|
||||
**Qwen3-Omni (local — server must be running):**
|
||||
|
||||
```bash
|
||||
cd packages/ltx-trainer
|
||||
uv run python scripts/caption_videos.py \
|
||||
"<workspace>/<run-name>/dataset/videos" \
|
||||
--output "<workspace>/<run-name>/dataset/dataset.json" \
|
||||
--captioner-type qwen_omni
|
||||
```
|
||||
|
||||
**Gemini Flash (cloud — runs anywhere, parallelisable):**
|
||||
|
||||
```bash
|
||||
# Auth: GEMINI_API_KEY / GOOGLE_API_KEY env var, or gcloud / Vertex AI credentials.
|
||||
cd packages/ltx-trainer
|
||||
uv run python scripts/caption_videos.py \
|
||||
"<workspace>/<run-name>/dataset/videos" \
|
||||
--output "<workspace>/<run-name>/dataset/dataset.json" \
|
||||
--captioner-type gemini_flash \
|
||||
--num-workers 5
|
||||
```
|
||||
|
||||
Output: JSON list of `{caption, media_path}` with paths **relative to the output file location**. The 3 spot-check captions can be merged in to avoid re-captioning them.
|
||||
|
||||
#### Trigger word (handled at preprocessing, not in captions)
|
||||
|
||||
For style/concept LoRAs the trigger word is **not** written into the captions here. It is prepended to every caption at preprocessing: pass `--lora-trigger "<word>"` to `process_dataset.py` in Phase 7, which forwards it to the caption-processing step (`process_captions.py`, where the prepend actually happens). That is the canonical mechanism — keep the captions describing what's actually on screen (via the default prompt), and let the trigger flag bind the concept to the token. Record the chosen trigger word in the plan so Phase 7 passes it through. Do not also bake the word into captions (it would double up).
|
||||
|
||||
**The injection mechanism is a fixed implementation detail — never make it a user-facing question.** The *only* trigger-word thing to ask the user is the **word itself** (or whether they want a trigger word at all). Do **not** ask, mention, or present as an option *how* it gets injected (e.g. "inject into the caption vs via `process_dataset`") — it is always `--lora-trigger`, full stop. Surfacing the method as a choice creates unnecessary confusion.
|
||||
|
||||
### Step 4 — Conditioning inputs (modes that need references or masks)
|
||||
|
||||
Some modes need a per-sample conditioning input beyond the video/audio and caption:
|
||||
|
||||
| Mode | Required extra input | Column |
|
||||
|------|----------------------|--------|
|
||||
| V2V IC-LoRA | reference video | `reference_video` (alias `ref_media_path`) |
|
||||
| AV2AV IC-LoRA | reference video + reference audio | `reference_video`, `reference_audio` |
|
||||
| A2A IC-LoRA | reference audio | `reference_audio` |
|
||||
| Video inpainting | per-frame video mask | `video_mask` |
|
||||
| Audio inpainting | audio mask | `audio_mask` |
|
||||
|
||||
These inputs encode **the user's specific idea** for the LoRA (what the reference represents, which regions the mask covers). There is no universal recipe, so **do not invent or default to a particular method** (e.g. don't assume Canny edges, depth, pose, or some generic box/border mask). The agent must not pick the conditioning semantics for the user.
|
||||
|
||||
Workflow when the chosen mode needs one of these and the dataset doesn't already provide it:
|
||||
|
||||
1. **Check first** — if the user already supplied the column (and the files exist), use it as-is and move on.
|
||||
2. **Otherwise, ask the user to provide it**, explaining concretely what's needed: the column name, that it's one file per sample aligned to each clip, and that the *content/semantics are their call* (what the reference should depict, what the mask should cover). Make clear this reflects their specific use-case — you won't guess it.
|
||||
3. **Help generate only if the user asks.** If they say "can you generate the references/masks by doing X" (X = their described method), then help: write or run a small script for *their* approach, or use a repo tool if it fits. One such tool exists — `scripts/compute_reference.py` generates **Canny edge** reference videos — but only mention/use it if the user specifically wants Canny; never offer it as the default.
|
||||
4. **Hard gate:** do not proceed to preprocessing for a conditioning mode until the required column is present with real files. Surface clearly if it's missing.
|
||||
|
||||
**Column-naming note (if references are generated):** `compute_reference.py` writes a `reference_video` field, which
|
||||
`process_dataset.py` detects automatically. Legacy datasets using `ref_media_path` also work.
|
||||
|
||||
### Step 5 — Holdout split
|
||||
|
||||
Reserve a subset of samples as a **held-out set** never seen during training. This is what Phase 9 (post-train validate) renders against to test true generalization rather than memorization.
|
||||
|
||||
Decision tree:
|
||||
|
||||
1. **User already supplied a held-out set** (separate file or directory they explicitly nominated): do not split. Copy/reference their file to `<workspace>/<run-name>/dataset/holdout.jsonl` and leave `dataset.json` as-is.
|
||||
2. **Small dataset** (judge qualitatively; tens of samples or fewer): holding samples out meaningfully reduces training capacity. Ask the user:
|
||||
> "Dataset has <N> samples. Reserving a holdout meaningfully reduces what's available for training. Options: (a) reserve 1–2 for holdout, (b) skip holdout — post-train eval will only test in-distribution. Your call."
|
||||
3. **Otherwise:** auto-split. Reserve a small fraction (this skill's default: roughly 10% of samples, bounded so the holdout doesn't grow huge — a handful of held-out samples is usually enough). Use seed 42 for the split so it's reproducible. Surface the count and the picked IDs in the plan.
|
||||
|
||||
After splitting, write `<workspace>/<run-name>/dataset/holdout.jsonl` (one JSON object per line with the same columns as `dataset.json`). **Remove the held-out entries from `dataset.json`** so they don't enter preprocessing or training.
|
||||
|
||||
Always print:
|
||||
> "Held out <K> of <N> samples for post-train evaluation. Held-out IDs: <list>."
|
||||
|
||||
If holdout is skipped, surface in the plan: *"Skipping holdout — dataset is too small. Post-train eval will only render in-distribution prompts; true generalization isn't testable for this run."*
|
||||
|
||||
### Step 6 — Audit
|
||||
|
||||
Before returning to the orchestrator, verify the metadata file has all required columns for the chosen mode. Print a one-line summary:
|
||||
|
||||
> "Prepared <N> training samples (+ <K> held out) for <mode>. Columns: <list>. Saved to `<workspace>/<run-name>/dataset/dataset.json` (+ `holdout.jsonl`)."
|
||||
|
||||
## Idempotency
|
||||
|
||||
- If `dataset.json` already exists and all required columns are present: skip captioning. Confirm reuse with the user only if the file was supplied by them outside the workspace (per the orchestrator's file-safety invariant).
|
||||
- If captioning was partial (some entries missing `caption`), re-run captioning only on the missing entries by filtering the metadata file before passing to `caption_videos.py`.
|
||||
|
||||
## Failure Modes
|
||||
|
||||
- Qwen server won't start / OOMs on launch → use the default `--quantization fp8` (not `bf16`), lower `--gpu-memory-utilization`, or reduce `--max-model-len` on `serve_captioner.py`. If the GPU simply can't host a 30B model, switch to `gemini_flash`.
|
||||
- `caption_videos.py` can't connect (qwen_omni) → the vLLM server isn't running or `--vllm-url` is wrong. Start `serve_captioner.py` first and confirm the URL/port match.
|
||||
- Gemini rate-limit → reduce `--num-workers`, retry.
|
||||
- Scene splitter produces 0 scenes → the detector found no cuts, or `--filter-shorter-than` removed everything. Lower/remove `--filter-shorter-than`, or adjust the detector threshold. (`--min-scene-length` is an integer minimum-frames-per-scene for the detector, not a short-scene filter.)
|
||||
- IC-LoRA reference compute fails on some frames → script logs the failures; report counts to the user and ask whether to proceed with the remaining samples or stop.
|
||||
|
||||
## Do Not
|
||||
|
||||
- Do not move or rename the user's source files. The skill reads them in place; the workspace contains only **derived** artifacts.
|
||||
- Do not delete `scenes/` or partial captioning outputs without approval — they may be expensive to regenerate.
|
||||
@@ -0,0 +1,130 @@
|
||||
# Phases 6 (one-sample) & 7 (full) — Preprocess Dataset
|
||||
|
||||
Procedure document for the `train-model` orchestrator. Read this file in full before acting on the preprocess phase.
|
||||
|
||||
Goal: run `process_dataset.py` to produce VAE latents, audio latents, and text embeddings. Two modes:
|
||||
1. **One-sample** (Phase 6 sanity check) — preprocess a single sample to `<workspace>/<run-name>/overfit/.precomputed/`.
|
||||
2. **Full** (Phase 7) — preprocess the whole dataset to `<workspace>/<run-name>/dataset/.precomputed/`.
|
||||
|
||||
The orchestrator's hard invariants apply (see `../SKILL.md`), especially: **never silently overwrite existing user data.**
|
||||
|
||||
## Required Subdirectories by Mode
|
||||
|
||||
Under `.precomputed/`:
|
||||
|
||||
| Subdir | Required for |
|
||||
|--------|--------------|
|
||||
| `latents/` | Video-bearing modes only (T2V, I2V, video extend/inpaint/outpaint, V2V/AV2AV IC-LoRA, A2V, V2A). **Not** produced for audio-only modes. |
|
||||
| `conditions/` | Always (text embeddings) |
|
||||
| `audio_latents/` | Any mode with audio: video modes carrying audio, plus all audio-only modes (T2A, audio extend/suffix/inpaint, A2A IC-LoRA) |
|
||||
| `reference_latents/` | V2V IC-LoRA, AV2AV IC-LoRA |
|
||||
| `reference_audio_latents/` | A2A IC-LoRA, AV2AV IC-LoRA |
|
||||
| `video_masks/` | Video inpainting |
|
||||
| `audio_masks/` | Audio inpainting |
|
||||
|
||||
**Audio-only modes** (T2A, audio extend/suffix, audio inpainting, A2A IC-LoRA) produce `audio_latents/` + `conditions/` (plus `audio_masks/` or `reference_audio_latents/` as applicable) and **no `latents/`**. Do not flag a missing `latents/` as incomplete for these modes.
|
||||
|
||||
## Workflow — Full Preprocess
|
||||
|
||||
### Step 1 — Verify existing `.precomputed/` (if present)
|
||||
|
||||
If `<workspace>/<run-name>/dataset/.precomputed/` already exists:
|
||||
|
||||
1. List subdirectories present. Confirm all required for the chosen mode are present.
|
||||
2. Load one sample per modality and check tensor shapes:
|
||||
```bash
|
||||
uv run python -c "import torch; t = torch.load('<path>'); print(t.shape if hasattr(t, 'shape') else {k: v.shape for k, v in t.items()})"
|
||||
```
|
||||
3. Compare shapes against the target resolution from the plan.
|
||||
|
||||
**On any mismatch or missing subdirectory: STOP. Do not run `process_dataset.py`.** Ask the user via `AskUserQuestion`:
|
||||
- Reuse the existing data at its current resolution (update plan + config accordingly).
|
||||
- Re-preprocess to a new directory (`<workspace>/<run-name>/dataset/.precomputed-v2/` etc.) — preserves the existing data.
|
||||
- Abort.
|
||||
|
||||
**Never pass `--overwrite` without explicit user approval** for this exact action.
|
||||
|
||||
### Step 2 — Invoke `process_dataset.py`
|
||||
|
||||
```bash
|
||||
cd packages/ltx-trainer
|
||||
uv run python scripts/process_dataset.py \
|
||||
"<workspace>/<run-name>/dataset/dataset.json" \
|
||||
--resolution-buckets "<W>x<H>x<F>" \
|
||||
--model-path "<absolute-model-path>" \
|
||||
--text-encoder-path "<absolute-gemma-path>" \
|
||||
--output-dir "<workspace>/<run-name>/dataset/.precomputed" \
|
||||
--load-text-encoder-in-8bit # on 32GB tier (low-VRAM config), per t2v_lora_low_vram.yaml
|
||||
```
|
||||
|
||||
Add as needed:
|
||||
- `--skip-audio` — if mode doesn't use audio (T2V video-only variants).
|
||||
- `--audio-durations "<list>"` — for T2A from a captions-only file.
|
||||
- `--lora-trigger "<trigger>"` — for style/concept LoRAs.
|
||||
- `--reference-downscale-factor <N>` — for IC-LoRA modes if downscaled references are desired.
|
||||
- `--video-column`, `--caption-column` — only if the metadata file uses non-standard column names.
|
||||
|
||||
**Do not pass `--overwrite`** unless re-preprocessing was explicitly approved in Step 1.
|
||||
|
||||
### Step 3 — Audit output
|
||||
|
||||
After completion, verify:
|
||||
|
||||
```bash
|
||||
ls "<workspace>/<run-name>/dataset/.precomputed/"
|
||||
# Expected subdirs per the mode table above.
|
||||
|
||||
# Count files in each:
|
||||
for d in latents conditions audio_latents reference_latents video_masks audio_masks; do
|
||||
if [ -d "<workspace>/<run-name>/dataset/.precomputed/$d" ]; then
|
||||
echo "$d: $(ls "<workspace>/<run-name>/dataset/.precomputed/$d" | wc -l)"
|
||||
fi
|
||||
done
|
||||
```
|
||||
|
||||
Counts in each required subdir should equal the dataset sample count.
|
||||
|
||||
**Reconcile counts — do this for every run, any dataset size.** Compare the `latents/` (and `audio_latents/`) count against the caption/sample count. If fewer latents were produced, `process_dataset.py` **silently skipped** clips — most commonly because they were **shorter than the target frame bucket** (it logs each skip). When counts don't match:
|
||||
|
||||
1. Identify which clips were dropped (grep the preprocess log for skip/"fewer frames" lines, or diff the produced `.pt` stems against the metadata).
|
||||
2. **Surface it to the user** with the count and the specific clips — never silently proceed on a shrunk dataset.
|
||||
3. Offer options: re-preprocess at a **smaller frame bucket** the clips support, add a **second (shorter) bucket** to keep the short clips (multi-bucket requires `batch_size: 1`), or accept the loss. Let the user decide.
|
||||
|
||||
**Audio gate (hard stop for audio runs).** For any run with an audio modality (joint audio+video, A2V, V2A, T2A, audio-only modes), verify `audio_latents/` is **present and non-empty** with one `.pt` per sample. `process_dataset.py` **swallows audio-decode errors and continues** — it logs "0 videos with audio" and produces empty `audio_latents/` rather than failing. If an audio run produced no audio latents, **stop** — do not proceed to training (it would silently train audio-free). The usual cause is a broken audio decode path (e.g. missing/incompatible `torchcodec`); confirm `uv run python -c "import torchaudio; torchaudio.load('<a clip>')"` works (see `references/troubleshooting.md`), fix it, then re-preprocess with `--overwrite`.
|
||||
|
||||
## Workflow — One-Sample (Phase 6)
|
||||
|
||||
Same as full preprocess, but operate on a single-sample metadata file:
|
||||
|
||||
1. Pick the first sample from `<workspace>/<run-name>/dataset/dataset.json` and write a one-sample metadata file **inside the dataset dir** — e.g. `<workspace>/<run-name>/dataset/_one_sample.json` — copying the entry **verbatim, keeping its relative `media_path`**. `process_dataset.py` resolves media paths relative to the metadata file's own directory, so the one-sample file must sit beside the real media (i.e. in `dataset/`, the same dir as `dataset.json`). **Do not** place it in `overfit/` and **do not** rewrite the path to an absolute one — an absolute path produces mirrored nested output dirs (`.precomputed/latents/absolute/path/.../x.pt`) instead of a clean `latents/x.pt`.
|
||||
2. Run `process_dataset.py` on that file with `--output-dir "<workspace>/<run-name>/overfit/.precomputed"` (output still goes to `overfit/`, only the metadata lives in `dataset/`).
|
||||
3. Use the **same `--resolution-buckets`** as the planned full run. Critical: a small-shape sanity check is misleading because resolution is the dominant memory factor.
|
||||
4. Clean up the temporary `dataset/_one_sample.json` afterward (it's scratch; don't leave it in the dataset dir).
|
||||
|
||||
## Decode-and-Verify (optional debug aid)
|
||||
|
||||
If the user reports validation samples look wrong or training diverges, decode one preprocessed sample back to media:
|
||||
|
||||
`decode_latents.py` takes the **latents directory** and an **output directory** as positional arguments (it decodes the whole directory, not a single `.pt` file). Add `--with-audio` and `--audio-latents-dir` if the dataset has audio.
|
||||
|
||||
```bash
|
||||
cd packages/ltx-trainer
|
||||
uv run python scripts/decode_latents.py \
|
||||
"<workspace>/<run-name>/dataset/.precomputed/latents" \
|
||||
"<workspace>/<run-name>/dataset/.precomputed/decoded_check" \
|
||||
--model-path "<absolute-model-path>"
|
||||
```
|
||||
|
||||
If decoded output is garbled, preprocessing itself is suspect (wrong model, wrong VAE).
|
||||
|
||||
## Failure Modes
|
||||
|
||||
- **"shape mismatch" on resume:** Step 1's verification check. Ask user before any mutation.
|
||||
- **`frames % 8 != 1`** error from process_dataset.py: the requested frame count is invalid; correct in the plan and re-launch.
|
||||
- **VRAM OOM during preprocessing:** add `--load-text-encoder-in-8bit`. If still OOM, reduce `--batch-size`.
|
||||
- **Disk full:** preprocessed latents can be large (especially audio). Surface to user with a `du -sh` summary of `.precomputed/`.
|
||||
|
||||
## Do Not
|
||||
|
||||
- Do not delete or overwrite existing `.precomputed/` data without explicit user approval for that exact action.
|
||||
- Do not preprocess at a smaller resolution to "save time" — the sanity check exists specifically to validate the planned resolution.
|
||||
@@ -0,0 +1,73 @@
|
||||
# Config Patching
|
||||
|
||||
How to safely produce `<workspace>/<run-name>/config.yaml` from an example in `packages/ltx-trainer/configs/`. The trainer's config schema is Pydantic with `extra="forbid"` — unknown fields are rejected. Full field reference: [`packages/ltx-trainer/docs/configuration-reference.md`](../../../../packages/ltx-trainer/docs/configuration-reference.md).
|
||||
|
||||
## Workflow
|
||||
|
||||
1. Copy the example config matching the selected mode (see `mode-selector.md`) to `<workspace>/<run-name>/config.yaml`.
|
||||
2. Patch fields as described below. Preserve YAML comments where possible — they help the user audit the run later.
|
||||
3. **Never** edit the example config in `packages/ltx-trainer/configs/`. That's the user's reference library.
|
||||
|
||||
## Required Patches (every run)
|
||||
|
||||
| Field | Value |
|
||||
|-------|-------|
|
||||
| `model.model_path` | Absolute path to local `.safetensors` (from probe or user). |
|
||||
| `model.text_encoder_path` | Absolute path to local Gemma directory (from probe or user). |
|
||||
| `data.preprocessed_data_root` | `<workspace>/<run-name>/dataset/.precomputed` (absolute). |
|
||||
| `output_dir` | `<workspace>/<run-name>/outputs` (absolute). |
|
||||
|
||||
## Hardware-Driven Patches
|
||||
|
||||
Apply per the matched VRAM tier in `references/hardware-profiles.md`. After autotune (Phase 6), patch the winning trial's deltas in.
|
||||
|
||||
## Schema Constraints (validate before launch)
|
||||
|
||||
These will cause Pydantic errors or runtime failures; check before invoking the trainer.
|
||||
|
||||
- **Frame count:** `validation.video_dims[2]` must satisfy `frames % 8 == 1` (1, 9, 17, 25, 33, 41, 49, 57, 65, 73, 81, 89, ...).
|
||||
- **Resolution:** `validation.video_dims[0]` and `[1]` must be divisible by 32.
|
||||
- **Multi-bucket training:** if dataset uses multiple resolution buckets, set `optimization.batch_size: 1`.
|
||||
- **At least one generated modality:** `training_strategy` must have at least one of `video.is_generated` or `audio.is_generated` set to `true`.
|
||||
- **Audio condition restrictions:** the audio modality cannot use `first_frame` or `spatial_crop` conditions.
|
||||
- **Strategy name:** prefer `training_strategy.name: "flexible"`. `text_to_video` and `video_to_video` still work but emit deprecation warnings.
|
||||
|
||||
## LoRA Patches
|
||||
|
||||
For style/concept LoRAs:
|
||||
|
||||
- `lora.rank` and `lora.alpha`: set from the matched VRAM tier and use case. **32GB tier** pins rank 16 per `t2v_lora_low_vram.yaml`; **80GB+ tier** uses rank 32 per `t2v_lora.yaml`. Keep `alpha == rank`. See `mode-selector.md` for use-case-driven rank guidance.
|
||||
- `lora.target_modules`: short patterns like `"to_k"`, `"to_q"`, `"to_v"`, `"to_out.0"` match all attention modules (video + audio + cross-modal). Add `"ff.net.0.proj"`, `"ff.net.2"` only if user explicitly wants higher capacity.
|
||||
- **Audio-only LoRA targets** (T2A, audio inpainting): use `"audio_attn1.to_*"`, `"audio_attn2.to_*"` patterns to avoid touching video weights. See `configs/t2a_lora.yaml` for the exact list.
|
||||
|
||||
## Validation Sample Prompts
|
||||
|
||||
The example configs ship with placeholder validation prompts. Validation condition fields are documented in
|
||||
[`configuration-reference.md#validation-condition-types`](../../../../packages/ltx-trainer/docs/configuration-reference.md#validation-condition-types).
|
||||
For style/concept LoRAs:
|
||||
- Replace at least one `validation.samples[].prompt` with a prompt that uses the user's trigger word or describes the target concept. Tells the user something useful at the first validation interval.
|
||||
- Keep `validation.video_dims` consistent with the training resolution to make samples comparable.
|
||||
- **Describe the audio, for any run with a generated audio modality** (joint audio+video, T2A, V2A, etc.). The validation prompts must describe the audio the **same way the training captions do** — if the training captions transcribe speech or characterise sound (e.g. *"he says: ‘…’"*, *"calm spoken voice, quiet room tone"*, *"upbeat acoustic guitar"*), the validation prompts must include comparable audio direction. A prompt with no audio description gives the model no guidance for the audio branch and the generated audio comes out poor. This is not speech-specific — any audio (music, ambience, foley) needs describing. Mirror the structure/level of audio detail found in the dataset captions (inspect a few before writing the prompts).
|
||||
|
||||
## W&B Patches
|
||||
|
||||
- If the W&B credential check passes (`uv run python -c "import wandb; print(bool(wandb.Api().api_key))"` → `True`): `wandb.enabled: true`, `wandb.project` = `ltx2-<mode>`, `wandb.tags` includes the mode. (Do not use `wandb status` — it falsely reports `api_key: null` when logged in via netrc.)
|
||||
- If `False`: `wandb.enabled: false`. Surface in plan: *"Not logged in to W&B — run `wandb login` before training to enable tracking."* If the check errored/was ambiguous, ask the user rather than assuming off.
|
||||
|
||||
## Output Dir Behaviour
|
||||
|
||||
The trainer resumes optimizer/scheduler/step state **only when `model.load_checkpoint` is set** to a checkpoint file; it then looks for a matching `training_state_step_*.pt` next to that file. It does **not** auto-detect prior checkpoints in `output_dir/checkpoints/`. To resume, patch `model.load_checkpoint` to the latest checkpoint. To load weights but skip state restore: `checkpoints.no_resume: true`.
|
||||
|
||||
For the orchestrator's resume flow: when resuming an interrupted run, patch `model.load_checkpoint` to the latest checkpoint under `output_dir/checkpoints/`. Leaving it unset starts a fresh run from step 0 even if checkpoints exist on disk.
|
||||
|
||||
## Self-Check Before Launch
|
||||
|
||||
Before any `python scripts/train.py` invocation:
|
||||
|
||||
1. All `model_path`, `text_encoder_path`, `preprocessed_data_root` exist on disk.
|
||||
2. Frame and resolution constraints satisfied (see above).
|
||||
3. Generated modalities have matching latents directories under `.precomputed/`.
|
||||
4. For modes with `reference` condition: `reference_latents/` (and/or `reference_audio_latents/`) exists.
|
||||
5. For modes with `mask` condition: `video_masks/` (and/or `audio_masks/`) exists.
|
||||
|
||||
A failed check at this point is much cheaper than a failed training start.
|
||||
@@ -0,0 +1,98 @@
|
||||
# VRAM Tiers
|
||||
|
||||
Map probed GPU(s) to a starting training config. The autotune sweep in Phase 6 then empirically improves on this baseline. Use these **tier names** in `plan.md` and user-facing chat — not letter codes.
|
||||
|
||||
Source of truth: the two configs shipped in the trainer repo —
|
||||
`packages/ltx-trainer/configs/t2v_lora.yaml` (standard) and
|
||||
`packages/ltx-trainer/configs/t2v_lora_low_vram.yaml` (low VRAM).
|
||||
Per `packages/ltx-trainer/docs/quick-start.md`, the trainer documents
|
||||
**80GB recommended** and **32GB minimum**. Anything below 32GB is
|
||||
unsupported by the project.
|
||||
|
||||
## Probe
|
||||
|
||||
```bash
|
||||
nvidia-smi --query-gpu=name,memory.total --format=csv,noheader
|
||||
```
|
||||
|
||||
Pick the **smallest** VRAM tier across the visible GPUs. Multi-GPU only adds throughput at the same per-GPU memory budget — it doesn't relax per-GPU limits.
|
||||
|
||||
## Minimum Gate
|
||||
|
||||
If per-GPU VRAM is **< 32 GB**, stop the run. Surface to the user:
|
||||
|
||||
> "This GPU has <N>GB VRAM. The LTX-2 trainer requires a minimum of 32GB (see `packages/ltx-trainer/docs/quick-start.md`). Training is unlikely to fit even with maximum memory savings, and we don't ship a tested config below 32GB. Options: (a) abort, (b) try anyway with the low-VRAM config and accept it may OOM — purely at your own risk."
|
||||
|
||||
Do not invent a sub-32GB tier. The trainer team doesn't ship one.
|
||||
|
||||
## 32GB tier — low-VRAM config
|
||||
|
||||
**VRAM range:** 32 GB per GPU (trainer minimum).
|
||||
|
||||
**Typical GPUs:** RTX 5090, V100 32GB.
|
||||
|
||||
Start from `packages/ltx-trainer/configs/t2v_lora_low_vram.yaml` verbatim. Key choices already in that file (do not re-specify in `<workspace>/<run-name>/config.yaml` — copy the file and patch only the paths from `references/config-patching.md`):
|
||||
|
||||
- `optimizer_type: "adamw8bit"`
|
||||
- `enable_gradient_checkpointing: true`
|
||||
- `batch_size: 1`, `gradient_accumulation_steps: 1`
|
||||
- `quantization: "int8-quanto"`
|
||||
- `load_text_encoder_in_8bit: true`
|
||||
- `offload_optimizer_during_validation: true`
|
||||
- `lora.rank: 16`, `lora.alpha: 16`
|
||||
|
||||
Autotune (Phase 6) will sweep `quantization` off, `optimizer_type` → adamw, and `batch_size` up — but at 32GB the sweep often hits OOM on trial 2 or 3. That's fine; the conservative baseline still works.
|
||||
|
||||
## 80GB+ tier — standard config
|
||||
|
||||
**VRAM range:** 80 GB per GPU and above (trainer recommended).
|
||||
|
||||
**Typical GPUs:** A100 80GB, H100 80GB, H200, B200.
|
||||
|
||||
Start from `packages/ltx-trainer/configs/t2v_lora.yaml` verbatim. Key choices already in that file:
|
||||
|
||||
- `optimizer_type: "adamw"`
|
||||
- `enable_gradient_checkpointing: true` (autotune may turn it off if headroom allows)
|
||||
- `batch_size: 1`, `gradient_accumulation_steps: 1`
|
||||
- `quantization: null`
|
||||
- `load_text_encoder_in_8bit: false`
|
||||
- `lora.rank: 32`, `lora.alpha: 32`
|
||||
|
||||
On the 80GB+ tier, the autotune baseline already equals this config (adamw, no quantization), so the quantization/optimizer trials are no-ops; the only real lever is gradient checkpointing off — but for the 22B model that **usually OOMs even with tens of GB of apparent headroom**, so treat a win there as unlikely. The trainer reports its own step-time and peak-VRAM at the end of each run — use those rather than an external timer.
|
||||
|
||||
For ≥140GB GPUs (H200, B200), the same 80GB+ tier baseline applies. FA3/FA4 attention backends are viable on Hopper/Blackwell and can speed up training, but they're optional — the trainer's defaults work on PyTorch SDPA without extra setup.
|
||||
|
||||
## 40–60GB tier — mid-range (autotune from low-VRAM)
|
||||
|
||||
**VRAM range:** 40–60 GB per GPU. The trainer doesn't ship a tested config for this range.
|
||||
|
||||
**Typical GPUs:** A40, A6000 48GB, L40, RTX 6000 Ada.
|
||||
|
||||
Start from the **32GB tier** (low-VRAM config) and let autotune relax `quantization`, `optimizer_type`, and `batch_size` based on actual headroom. Don't pre-bake intermediate YAML values that haven't been measured. Surface this as **40–60GB tier** in the plan.
|
||||
|
||||
## Multi-GPU
|
||||
|
||||
If `nvidia-smi` reports N ≥ 2 GPUs of the same model:
|
||||
|
||||
- Launch with `uv run accelerate launch scripts/train.py <config>`.
|
||||
- Use `packages/ltx-trainer/configs/accelerate/fsdp.yaml` for full fine-tune.
|
||||
- DDP (default `accelerate launch` without a config file) is fine for LoRA.
|
||||
- Effective batch = `batch_size * gradient_accumulation_steps * num_gpus`. Reduce `gradient_accumulation_steps` proportionally to keep the effective batch consistent with the plan.
|
||||
|
||||
## Full Fine-Tune
|
||||
|
||||
If the user chose full fine-tune (`model.training_mode: "full"`):
|
||||
|
||||
- Require multi-GPU + FSDP on 80GB+ tier GPUs. Otherwise warn in the plan that single-GPU full FT is unlikely to fit and propose LoRA instead.
|
||||
- Set `acceleration.offload_optimizer_during_validation: true` always (optimizer state is huge under full FT).
|
||||
|
||||
## Model Path Constraints
|
||||
|
||||
- `model.model_path`: local `.safetensors` only. No URLs.
|
||||
- `model.text_encoder_path`: local Gemma model directory. No URLs.
|
||||
|
||||
If probe didn't find these in conventional locations (`/models/`, `~/models/`, `$LTX_MODELS_DIR`), ask the user in Phase 3 (or offer to download per `references/onboarding.md`).
|
||||
|
||||
## Notes on Loss-of-Generality
|
||||
|
||||
The two anchor tiers (32GB and 80GB+) correspond directly to the two configs the trainer ships. The autotune sweep is the empirical layer — if a particular GPU consistently lands on a different stable config, **update the relevant trainer config first**, not this file. This skill follows the trainer's choices, not the other way around.
|
||||
@@ -0,0 +1,74 @@
|
||||
# Mode Selector
|
||||
|
||||
Map the user's stated intent to a `flexible`-strategy configuration. All modes are supported via a single strategy (`training_strategy.name: "flexible"`); the difference is which modality is generated and which `conditions` are attached.
|
||||
|
||||
> Full reference: [`packages/ltx-trainer/docs/training-modes.md`](../../../../packages/ltx-trainer/docs/training-modes.md). This file is the **lookup table** for translating user intent.
|
||||
|
||||
## Decision Table
|
||||
|
||||
| User says (roughly)... | Mode | Example config | Modalities | Conditions |
|
||||
|------------------------|------|----------------|------------|------------|
|
||||
| "generate videos from text", "T2V LoRA" | T2V | `configs/t2v_lora.yaml` | video gen, audio gen | none |
|
||||
| "generate videos from a starting image", "I2V" | I2V | `configs/i2v_lora.yaml` | video gen, audio gen | `first_frame` (video) |
|
||||
| **plain concept/style LoRA** ("train a LoRA on X", no specific task) | **I2V by default** (see note) | `configs/i2v_lora.yaml` | video gen, audio gen | `first_frame` (video), `probability: 0.5` |
|
||||
| "extend a video forward in time" | Video extension (prefix) | `configs/video_extend_lora.yaml` | video gen, audio gen | `prefix` (video) |
|
||||
| "extend a video backward in time" | Video extension (suffix) | `configs/video_suffix_lora.yaml` | video gen, audio gen | `suffix` (video) |
|
||||
| "fill in masked regions of a video" | Video inpainting | `configs/video_inpainting_lora.yaml` | video gen | `mask` (video) |
|
||||
| "expand a video beyond its borders" | Video outpainting | `configs/video_outpainting_lora.yaml` | video gen | `spatial_crop` (video) |
|
||||
| "style transfer from reference video", "IC-LoRA", "depth/pose/canny control" | V2V IC-LoRA | `configs/v2v_ic_lora.yaml` | video gen | `reference` (video) |
|
||||
| "generate video to match an audio track" | A2V | `configs/a2v_lora.yaml` | video gen, audio frozen | none (audio `is_generated: false`) |
|
||||
| "add sound effects to silent video", "foley", "V2A" | V2A | `configs/v2a_lora.yaml` | video frozen, audio gen | none (video `is_generated: false`) |
|
||||
| "generate audio from text", "T2A" | T2A | `configs/t2a_lora.yaml` | audio gen | none |
|
||||
| "extend audio forward / backward" | Audio extension | `configs/audio_extend_lora.yaml`, `configs/audio_suffix_lora.yaml` | audio gen | `prefix` / `suffix` (audio) |
|
||||
| "fill in masked regions of audio" | Audio inpainting | `configs/audio_inpainting_lora.yaml` | audio gen | `mask` (audio) |
|
||||
| "audio style transfer from reference", "A2A IC-LoRA" | A2A IC-LoRA | `configs/a2a_ic_lora.yaml` | audio gen | `reference` (audio) |
|
||||
| "joint video+audio reference control" | AV2AV IC-LoRA | `configs/av2av_ic_lora.yaml` | video gen, audio gen | `reference` (both) |
|
||||
| Any of the above with full fine-tune | Full FT variant | as above, set `model.training_mode: "full"` | (mode-specific) | (mode-specific) |
|
||||
|
||||
### Why I2V is the default for a plain concept/style LoRA
|
||||
|
||||
A "train a LoRA on X" request isn't tied to one inference mode: **LoRA weights are pipeline-agnostic** — the same checkpoint loads in both T2V and I2V inference (both use `TI2VidOneStagePipeline`/`TwoStages`). The `i2v_lora` config trains `first_frame` with **`probability: 0.5`**, so the model learns both first-frame-conditioned (I2V) and unconditioned (T2V) generation in one run, and the first frame comes from each training clip automatically (no extra data prep). That makes I2V a versatile **superset** — usable for both at inference at no extra cost — which is why it's the default for a plain LoRA. Ask the user how they'll use it (text-only / from an image / both) and only drop to `t2v_lora` if they're sure it's text-only. (See the orchestrator `SKILL.md` Phase 1.)
|
||||
|
||||
## Disambiguation Questions
|
||||
|
||||
When the user's first answer is ambiguous, ask **one** follow-up:
|
||||
|
||||
- "extend a video" → forward or backward in time?
|
||||
- "control with a reference" → video reference (depth/pose/canny/etc.) or audio reference?
|
||||
- "fill in regions" → video regions (masked frames) or audio regions (masked time)?
|
||||
- "T2V" → joint video+audio (default) or video-only?
|
||||
- LoRA or full fine-tune? Default to LoRA unless the user has multi-GPU + clear reason for full.
|
||||
|
||||
## Combining Modes
|
||||
|
||||
The flexible strategy allows stacking conditions. Common combinations:
|
||||
|
||||
- **I2V + V2A** (start frame + generate audio for the resulting video) — not directly expressible; would need two passes.
|
||||
- **Video extension + audio extension** — both modalities generate, both have `prefix` condition. Express in one config.
|
||||
- **IC-LoRA + I2V** — `first_frame` + `reference` conditions on the video modality.
|
||||
|
||||
If the user asks for a combination not listed, check `packages/ltx-trainer/src/ltx_trainer/training_strategies/flexible.py` for which conditions can co-exist on a modality. Audio modality cannot use `first_frame` or `spatial_crop`.
|
||||
|
||||
## When the Intent Doesn't Map
|
||||
|
||||
If after disambiguation there is no entry in the table and no combination of `flexible` conditions covers the user's request, go to the **Escape Hatch** section in the orchestrator `SKILL.md`. Do not silently pick the closest mode.
|
||||
|
||||
## LoRA Rank by Use Case
|
||||
|
||||
Once the mode is picked, choose `lora.rank` (and matching `lora.alpha`) based on what the LoRA is supposed to capture. These are starting points; autotune doesn't sweep rank because it's a quality knob, not a step-time one.
|
||||
|
||||
| Use case | Suggested rank | Notes |
|
||||
|----------|----------------|-------|
|
||||
| Single character, single object, single style | 32–64 | Default for most concept LoRAs. Start at 32; bump to 64 if validation samples underfit. |
|
||||
| Multi-character world, dense series, complex multi-concept | 96–128 | More capacity for distinguishing several concepts inside one LoRA. |
|
||||
| Camera move, motion, transition (i.e. behavioural, not visual) | 8–16 | Motion is a thin signal — high ranks just memorise frame content. |
|
||||
| IC-LoRA control (V2V depth/pose/Canny/etc., A2A audio reference) | 16–32 | Start at 16 for structural control (depth, pose, edges); 24–32 if the reference carries richer style/texture. Video IC-LoRA often lands lower than concept LoRAs. |
|
||||
| LTX-2 trainer's default if unsure | 32 | Safe baseline. |
|
||||
|
||||
On the **32GB tier** (`hardware-profiles.md`), the low-VRAM config already pins `lora.rank: 16`. If the user picks a higher-rank use case on 32GB, surface the trade-off in the plan but let them decide — the autotune sweep doesn't touch rank, so a too-high rank will simply OOM at training time.
|
||||
|
||||
Keep `alpha == rank` unless the user has a specific reason otherwise (effective scaling = `alpha / rank`).
|
||||
|
||||
## Starting Config
|
||||
|
||||
Copy the matching example config — the exact filename from the **Example config** column of the decision table above (e.g. T2V → `configs/t2v_lora.yaml`, I2V → `configs/i2v_lora.yaml`, A2A IC-LoRA → `configs/a2a_ic_lora.yaml`) — into `<workspace>/<run-name>/config.yaml` as the starting point. Then patch per `references/config-patching.md`.
|
||||
@@ -0,0 +1,115 @@
|
||||
# First-Run Onboarding
|
||||
|
||||
What the orchestrator's Phase 2 probe checks for, what to do when something is missing, and what the skill is allowed to set up automatically (with explicit user approval).
|
||||
|
||||
## Prerequisites Checked in Phase 2
|
||||
|
||||
| Prerequisite | How to detect | If missing |
|
||||
|--------------|---------------|------------|
|
||||
| CUDA GPU visible | `nvidia-smi` returns ≥1 GPU | Stop. Training requires CUDA — point the user at non-LTX-2 docs. |
|
||||
| Linux | `uname -s` returns `Linux` | Stop. Trainer uses Triton (Linux-only). |
|
||||
| `uv` installed | `command -v uv` | Offer to install (see "Auto-setup" below). |
|
||||
| Workspace synced | `[ -f uv.lock ] && uv pip list \| grep -q ltx-trainer` | Offer to run `uv sync` from repo root. |
|
||||
| LTX-2 model weights | Search `/models/`, `~/models/`, `$LTX_MODELS_DIR` for a `.safetensors` matching `*ltx*2*` | Offer to download (see "Model downloads"). |
|
||||
| Gemma text encoder dir | Search same locations for a directory containing Gemma config | Offer to download. |
|
||||
| Captioner backend | Gemini auth (`GEMINI_API_KEY`/`GOOGLE_API_KEY` or gcloud/Vertex), OR a ≥40 GiB GPU to host the Qwen3-Omni-30B vLLM server (FP8), OR captions already in the dataset | See "Captioner graceful degradation" below. **Check the HF cache for an already-downloaded Qwen model before assuming a download is needed** (see note below the table). |
|
||||
| W&B login (optional) | `uv run python -c "import wandb; print(bool(wandb.Api().api_key))"` → `True` means logged in. Uses wandb's own credential resolution (env/netrc/settings). **Don't** use `wandb status` (reports `api_key: null` even when logged in). | Not a blocker. If `False`: disabled in config + flagged in plan. If the check errors/ambiguous: ask the user, don't assume off. |
|
||||
| Disk space | `df -h $WORKSPACE` | Surface available space alongside what a run consumes (preprocessed latents, several-GB checkpoints, validation samples). Flag concerns to the user; don't enforce a hard threshold. |
|
||||
|
||||
Surface findings as a compact table in chat. For each missing item, present the user with a concrete next step (download command, install command, or "skip this — here's the consequence").
|
||||
|
||||
**Check the HF cache before declaring the local captioner unavailable.** The Qwen3-Omni model is served from the HuggingFace cache, not `~/models/`. Before concluding it must be downloaded (or ruling it out on free-disk grounds), check whether it's already cached:
|
||||
|
||||
```bash
|
||||
ls -d "${HF_HOME:-$HOME/.cache/huggingface}"/hub/models--Qwen--Qwen3-Omni* 2>/dev/null \
|
||||
&& du -sh "${HF_HOME:-$HOME/.cache/huggingface}"/hub/models--Qwen--Qwen3-Omni* 2>/dev/null
|
||||
```
|
||||
|
||||
If it's cached (~60 GiB, all shards present), no download is needed — don't rule out the local captioner because of low free disk on the *home* partition; the weights already exist. Only the GPU-VRAM constraint (≥40 GiB for FP8) then applies.
|
||||
|
||||
## Auto-Setup (with explicit user approval)
|
||||
|
||||
The skill may, only after the user explicitly says yes, do these setup actions. Each action is a single discrete question.
|
||||
|
||||
### Run `uv sync`
|
||||
|
||||
```bash
|
||||
cd <repo-root>
|
||||
uv sync
|
||||
```
|
||||
|
||||
Ask: *"Repo not synced. Run `uv sync` now? It will download the project's Python dependencies."*
|
||||
|
||||
### Install `uv`
|
||||
|
||||
Ask: *"`uv` not installed. Install via the official one-liner now?"*
|
||||
|
||||
```bash
|
||||
curl -LsSf https://astral.sh/uv/install.sh | sh
|
||||
```
|
||||
|
||||
After install, ask the user to restart their shell or `source ~/.bashrc` before continuing.
|
||||
|
||||
### Model downloads
|
||||
|
||||
Use `huggingface-cli` (comes with `huggingface-hub`, transitively pulled by `uv sync`). Default destination: `$LTX_MODELS_DIR` if set, else `~/models/` (create if missing). Surface destination in the prompt — never download into the repo or into the workspace.
|
||||
|
||||
**LTX-2 base model:**
|
||||
|
||||
```bash
|
||||
huggingface-cli download Lightricks/LTX-2.3 \
|
||||
ltx-2.3-22b-dev.safetensors \
|
||||
--local-dir ~/models/ltx-2.3
|
||||
```
|
||||
|
||||
Public reference: <https://huggingface.co/Lightricks/LTX-2.3>.
|
||||
|
||||
**Gemma text encoder:**
|
||||
|
||||
```bash
|
||||
huggingface-cli download google/gemma-3-12b-it-qat-q4_0-unquantized \
|
||||
--local-dir ~/models/gemma-3-12b
|
||||
```
|
||||
|
||||
**Qwen3-Omni captioner (only if the GPU can host it):**
|
||||
|
||||
The local captioner is Qwen3-Omni-30B-A3B-Thinking served by a vLLM server (`scripts/serve_captioner.py`), which downloads the model (~65 GiB) on first launch via `uvx vllm` — there's no separate `huggingface-cli` step. Default **FP8** (~31 GiB weights) fits on a **40 GiB** GPU; **bf16** (~60 GiB) needs **≥66 GiB free VRAM**. On a GPU below ~40 GiB (typical consumer 24/32 GB cards), don't use it — use Gemini instead (see "Captioner graceful degradation").
|
||||
|
||||
The base model + text encoder are large (multi-GB) downloads. Ask the user before each one — `huggingface-cli` reports the actual size at the start of the transfer. Do **not** batch them into a single "yes/no"; the user may want only what's missing.
|
||||
|
||||
### Hugging Face login (if any downloads fail with 401)
|
||||
|
||||
Ask: *"Hugging Face download requires login (some Lightricks models are gated). Run `huggingface-cli login` now? You'll need a token from <https://huggingface.co/settings/tokens>."*
|
||||
|
||||
## Captioner Graceful Degradation
|
||||
|
||||
The captioner is the trickiest prerequisite. The local backend (`qwen_omni`) is now a **30B model served by a vLLM server** — ~65 GiB download; FP8 fits on a 40 GiB GPU, bf16 needs ≥66 GiB. Typical consumer cards (24/32 GB) can't host it, so for most users **prefer Gemini**.
|
||||
|
||||
Decision tree:
|
||||
|
||||
1. **User already has captions in their dataset metadata** → skip the captioner entirely (Step 3 of `prepare-dataset` skips when captions are present).
|
||||
2. **GPU ≥40 GiB (FP8) / ≥66 GiB (bf16)** → `qwen_omni` is viable: launch `scripts/serve_captioner.py` first, then caption. Gemini is still fine here too.
|
||||
3. **GPU below ~40 GiB (the common consumer case)** → `qwen_omni` is not an option. Recommend **`gemini_flash`** and tell the user they need Gemini auth: a `GEMINI_API_KEY`/`GOOGLE_API_KEY` (get one at <https://aistudio.google.com/apikey>) **or** working gcloud/Vertex AI credentials.
|
||||
4. **No Gemini auth and can't run Qwen3** → the only remaining path is bringing their own captions: add a `caption` column to the dataset metadata, then re-invoke the skill.
|
||||
|
||||
Wait for the user's choice. Don't pick one automatically — but make the hardware reality explicit so they don't try to run the server on a card that can't host it.
|
||||
|
||||
## What Auto-Setup Does NOT Touch
|
||||
|
||||
- Does not modify the user's shell rc files except by explicit instruction (e.g., "add `export LTX_MODELS_DIR=...` to your `.bashrc`?" with the user agreeing).
|
||||
- Does not modify `~/.gitconfig`, `~/.ssh/`, or any auth-related files.
|
||||
- Does not install GPU drivers, CUDA, or system packages.
|
||||
- Does not delete or move existing model files. If a model is found but at an unexpected path, surface and let the user decide.
|
||||
- Does not download into the repo or workspace. Models live at `~/models/` (or `$LTX_MODELS_DIR`).
|
||||
|
||||
## Configuration Inheritance
|
||||
|
||||
After downloads, the skill records the resolved paths into the plan's Assumptions section and into `config.yaml`:
|
||||
|
||||
```yaml
|
||||
model:
|
||||
model_path: "/home/<user>/models/ltx-2.3/ltx-2.3-22b-dev.safetensors"
|
||||
text_encoder_path: "/home/<user>/models/gemma-3-12b"
|
||||
```
|
||||
|
||||
Suggest (don't enforce) setting `LTX_MODELS_DIR=~/models` in their shell rc for future runs.
|
||||
@@ -0,0 +1,107 @@
|
||||
# Plan Template
|
||||
|
||||
Write `<workspace>/<run-name>/plan.md` following this template. The plan is the user's contract with the agent — it's the gate before any heavy work runs.
|
||||
|
||||
Scale each section to its relevance. Skip subsections that don't apply (e.g., no captioning if data is already captioned), but never collapse "Assumptions" or "Cost/time estimate".
|
||||
|
||||
```markdown
|
||||
# Training Plan — <run-name>
|
||||
|
||||
## Goal
|
||||
|
||||
<One paragraph restating the user's intent in their own terms.>
|
||||
|
||||
## Mode
|
||||
|
||||
**<Mode name>** — <one-line rationale linking user intent to mode>.
|
||||
|
||||
- Config base: the concrete example config selected by the mode (e.g. `packages/ltx-trainer/configs/t2v_lora.yaml`) — use the actual filename, not a placeholder
|
||||
- Conditions: <list, or "none">
|
||||
- Training mode: <`lora` | `full`>
|
||||
|
||||
## Dataset
|
||||
|
||||
- Source: `<absolute path>` (<N samples>)
|
||||
- Captions: <`already present` | `will be generated with <backend>`>
|
||||
- Audio: <`present` | `absent — using --skip-audio` | `to be paired with --audio-durations`>
|
||||
- IC-LoRA references: <`present` | `to be generated via compute_reference.py` | `n/a`>
|
||||
|
||||
## Preprocessing
|
||||
|
||||
- Target resolution buckets: `<W>x<H>x<F>` (frames satisfy `frames % 8 == 1`; W,H divisible by 32)
|
||||
- Estimated time: ~<duration> on detected hardware
|
||||
- Output: `<workspace>/<run-name>/dataset/.precomputed/`
|
||||
|
||||
## Training Config
|
||||
|
||||
| Field | Value |
|
||||
|-------|-------|
|
||||
| Optimizer | <adamw / adamw8bit> |
|
||||
| Mixed precision | <bf16 / fp16> |
|
||||
| Quantization | <null / int8-quanto / ...> |
|
||||
| Gradient checkpointing | <on / off> |
|
||||
| Batch size | <N> |
|
||||
| Gradient accumulation | <N> (effective batch = <N>) |
|
||||
| Steps | <N> |
|
||||
| Learning rate | <value> |
|
||||
| LoRA rank / alpha | <N / N> (or "full FT") |
|
||||
| LoRA target modules | <list> (or "n/a") |
|
||||
| LoRA trigger word | <word> (or "n/a") |
|
||||
| Validation interval | every <N> steps |
|
||||
| Checkpoint interval | every <N> steps |
|
||||
|
||||
## Hardware
|
||||
|
||||
- GPU(s): <name> x <count>, <VRAM>GB per GPU
|
||||
- Launch: <`python scripts/train.py` (single) | `accelerate launch` (multi)>
|
||||
- VRAM tier: <32GB tier | 40–60GB tier | 80GB+ tier> — <low-VRAM config (`t2v_lora_low_vram.yaml`) | standard config (`t2v_lora.yaml`) | mid-range, autotuned from low-VRAM>
|
||||
|
||||
## Sanity Check + Autotune
|
||||
|
||||
*In plain terms: before committing to the full run, I do a quick dry run on a single clip at your target resolution to catch out-of-memory or config errors in ~2 minutes (rather than failing hours in), then try a few config variants to pick the fastest one that fits your GPU.*
|
||||
|
||||
Mechanics:
|
||||
- 1 sample, full target resolution, 50 steps + 1 validation pass.
|
||||
- Autotune sweep: up to 5 trials varying quantization / optimizer / batch size.
|
||||
- Stops at first OOM or no-improvement.
|
||||
|
||||
## Monitoring
|
||||
|
||||
<One of:>
|
||||
- W&B: enabled, project `<name>`, entity `<entity-or-default>`. URL will be surfaced once training starts.
|
||||
- W&B: **not logged in** — run `wandb login` before training to enable tracking. Otherwise training proceeds without remote logging.
|
||||
|
||||
## Outputs
|
||||
|
||||
- Training config: `<workspace>/<run-name>/config.yaml`
|
||||
- Checkpoints: `<workspace>/<run-name>/outputs/checkpoints/`
|
||||
- Validation samples: `<workspace>/<run-name>/outputs/samples/`
|
||||
- Autotune log: `<workspace>/<run-name>/autotune.log`
|
||||
|
||||
## Assumptions
|
||||
|
||||
Defaults the agent chose silently. Override any by replying with the new value.
|
||||
|
||||
- <list every non-trivial assumed value: precision, scheduler type, seed, validation prompts, checkpoint retention, etc.>
|
||||
|
||||
## Cost / Time Estimate
|
||||
|
||||
Give only estimates you can ground; label anything not yet measured as rough. Do **not** state a confident training duration before the sanity check has measured a real step time — say "training duration TBD until the sanity check measures step time" and fill it in afterward (per Hard Invariant #5: no fabricated predictions).
|
||||
|
||||
- Captioning: ~<duration> (rough)
|
||||
- Preprocessing: ~<duration> (rough)
|
||||
- Sanity check + autotune: ~<duration> (rough)
|
||||
- Full training: **measured after sanity check** — then `<measured step-time> × <steps>`
|
||||
- **Total wall-clock estimate:** rough until step time is measured; refine after the sanity check.
|
||||
|
||||
## Approve to Proceed
|
||||
|
||||
Reply "approve" (or with edits) to start. No captioning, preprocessing, autotune, or training will run before approval.
|
||||
```
|
||||
|
||||
## Notes on Writing the Plan
|
||||
|
||||
- Show numbers, not adjectives. "~3 hours" beats "fairly long."
|
||||
- Surface every assumption that, if wrong, would cost the user time. Better to over-list than under-list — the user can skim.
|
||||
- If a section reveals you need to ask another question, **stop and ask** before finalizing the plan. The plan is the last gate, not the first.
|
||||
- If the user's hardware can't reasonably support the requested mode (e.g., single-GPU full FT on a 32GB consumer card), say so plainly in the plan and propose the alternative (LoRA, multi-GPU, etc.), rather than silently downgrading.
|
||||
@@ -0,0 +1,84 @@
|
||||
# Troubleshooting
|
||||
|
||||
Quick lookup for failures during sanity check, preprocessing, or training. For deeper coverage see `packages/ltx-trainer/docs/troubleshooting.md`.
|
||||
|
||||
## OOM During Training Step
|
||||
|
||||
Order of operations (cheapest first):
|
||||
|
||||
1. `optimization.enable_gradient_checkpointing: true` (if not already on).
|
||||
2. `optimization.batch_size: 1` and increase `gradient_accumulation_steps` to preserve effective batch.
|
||||
3. `optimization.optimizer_type: "adamw8bit"`.
|
||||
4. `acceleration.quantization: "int8-quanto"`.
|
||||
5. Reduce `lora.rank` (32 → 16 → 8). Alpha follows rank.
|
||||
6. Reduce target resolution (`validation.video_dims` and re-preprocess the dataset at the new resolution).
|
||||
|
||||
The last option is expensive — flag it clearly to the user before re-preprocessing.
|
||||
|
||||
## OOM During Validation Sample Generation
|
||||
|
||||
The validation pass loads decoders + runs CFG/STG inference; it can OOM even when the training step fits.
|
||||
|
||||
1. `acceleration.load_text_encoder_in_8bit: true` (trainer config, not the dataset script).
|
||||
2. `acceleration.offload_optimizer_during_validation: true` (especially for full FT or high-rank LoRA).
|
||||
3. Reduce `validation.video_dims` (smaller validation than training is fine — it's only for visual feedback).
|
||||
4. Reduce `validation.inference_steps` (e.g. 30 → 20).
|
||||
5. Increase `validation.interval` to validate less often.
|
||||
|
||||
## NaN Loss
|
||||
|
||||
1. Check `acceleration.mixed_precision_mode`: prefer `"bf16"`. If `"fp16"`, switch.
|
||||
2. Verify dataset latents are well-formed: `uv run python scripts/decode_latents.py <latents-dir> <output-dir> --model-path <model>` (it decodes a whole latents directory, not a single `.pt`) should reconstruct sensibly.
|
||||
3. Lower `optimization.learning_rate` by 5x.
|
||||
4. Add `optimization.max_grad_norm: 1.0` (default; verify it's set).
|
||||
5. If using `quantization`, try `null` — INT8/INT4 quantization can interact badly with poorly-conditioned LoRA inits at high LR.
|
||||
|
||||
## Validation Samples Look Wrong but Loss Is Fine
|
||||
|
||||
Often not a bug — validation uses simplified inference. For real quality assessment, run a checkpoint through `packages/ltx-pipelines/` after training.
|
||||
|
||||
## Trainer Won't Start: Config Validation Error
|
||||
|
||||
Pydantic `extra="forbid"` means typos in field names fail loudly. Read the error carefully — it names the offending field and path. Fix and re-launch.
|
||||
|
||||
Common offenders:
|
||||
- `latents_dir` typo or wrong relative path.
|
||||
- A required field genuinely missing after copying an example (most fields have defaults; check the error message for the exact field path).
|
||||
- `target_modules` listed at wrong nesting level (must be under `lora:`).
|
||||
|
||||
## Trainer Won't Start: Missing Files
|
||||
|
||||
- `model_path` not found → re-probe `/models/`, `~/models/`, `$LTX_MODELS_DIR`, or ask the user.
|
||||
- `text_encoder_path` directory missing the Gemma config → ensure the path is to the Gemma model dir, not its parent.
|
||||
- `preprocessed_data_root` doesn't contain expected subdirs → re-verify Phase 7 ran for the chosen mode (see `phases/preprocess-dataset.md`).
|
||||
|
||||
## Resume Stops Working
|
||||
|
||||
The trainer **does not auto-resume from `output_dir`**. Resume happens only when `model.load_checkpoint` is explicitly set to a checkpoint file; the trainer then loads those weights and looks for a `training_state_step_*.pt` next to that file to restore optimizer/scheduler/step. Common pitfalls:
|
||||
- `model.load_checkpoint` not set or set to the wrong path → fresh run from step 0 even when `outputs/checkpoints/` is full of artifacts. Patch `model.load_checkpoint` to the latest checkpoint.
|
||||
- `checkpoints.no_resume: true` is set → weights load but state is discarded. Remove the flag if you want a proper resume.
|
||||
- `training_state_step_*.pt` missing from next to the loaded checkpoint → weights load but step counter resets. Make sure the state file accompanies the checkpoint.
|
||||
- `training_state_step_*.pt` corrupted (size 0, fails `torch.load`) → trainer falls back to step 0 with a warning.
|
||||
|
||||
## Autotune Trial Failed Mid-Sweep
|
||||
|
||||
- If trial 2 (quantization off) OOMs: revert to trial 1 and stop the sweep. The 32GB tier baseline is correctly aggressive.
|
||||
- If trial 3 (adamw) OOMs: revert to adamw8bit. Continue with trial 4 if VRAM headroom allows.
|
||||
- If trial 4 (batch_size up) OOMs: revert and stop. We've found the ceiling.
|
||||
|
||||
Never carry over a failing trial's deltas. Always revert to the last-known-good before the next change.
|
||||
|
||||
## Captioning Is Slow
|
||||
|
||||
- Local `qwen_omni` is a 30B model served by `serve_captioner.py` (vLLM). If the server won't start or OOMs on launch: keep the default `--quantization fp8` (don't use `bf16` unless ≥66 GiB free VRAM), lower `--gpu-memory-utilization`, or reduce `--max-model-len`. If the GPU can't host a 30B model at all, switch to `gemini_flash`.
|
||||
- `caption_videos.py --captioner-type qwen_omni` errors connecting → the vLLM server isn't running or `--vllm-url` doesn't match. Start `serve_captioner.py` first and confirm the port.
|
||||
- For most hardware and for larger datasets, prefer `--captioner-type gemini_flash --num-workers <N>` (needs Gemini auth: `GEMINI_API_KEY`/`GOOGLE_API_KEY` or gcloud/Vertex) — runs anywhere and parallelises; local Qwen needs a heavy GPU and a running server.
|
||||
|
||||
## Process_Dataset Errors
|
||||
|
||||
- "frames divisible by..." → the video doesn't have enough frames at the requested temporal resolution. Either shorten the requested frame count or use `split_scenes.py` to break long videos.
|
||||
- "shape mismatch" on existing `.precomputed/` → user requested a different resolution than the existing data. Per the invariants, **stop and ask** — do not overwrite. Offer: reuse at old resolution / re-preprocess to a new dir / abort.
|
||||
|
||||
## When To Give Up and Ask The User
|
||||
|
||||
If a fix isn't obvious from this file or `packages/ltx-trainer/docs/troubleshooting.md` within two attempts, stop and surface the full error + the steps already tried to the user. Don't loop indefinitely on autonomous fixes — the user has context the agent doesn't (which checkpoints are precious, what they care about preserving, etc.).
|
||||
Reference in New Issue
Block a user