indigo/LTX-2
1
0
Fork 0
Code Issues Pull Requests 1 Actions Packages Projects Releases Wiki Activity

Automated PR - 2026-06-17

Browse Source
This commit is contained in:
github-actions[bot]
2026-06-17 14:06:32 +00:00
parent d6053703e0
commit 0d3d3a3855
90 changed files with 8265 additions and 4511 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.claude/skills/train-model/phases/launch-and-monitor.md
+235
View File
@@ -0,0 +1,235 @@
# Phase 8 — Launch & Monitor
Procedure document for the `train-model` orchestrator (Phase 8 + monitor-only re-entry). Read this file in full before acting on launch/monitor.
Goal: start the training job, surface the W&B URL, produce periodic status reports, write `run-summary.md` at completion.
The orchestrator's hard invariants apply (see `../SKILL.md`).
## Launch — Single GPU
```bash
cd packages/ltx-trainer
uv run python scripts/train.py "<workspace>/<run-name>/config.yaml"
```
## Launch — Multi-GPU
Use Accelerate. For LoRA, DDP (default) is fine. For full FT, use FSDP.
```bash
cd packages/ltx-trainer
# DDP (LoRA, multi-GPU)
uv run accelerate launch scripts/train.py "<workspace>/<run-name>/config.yaml"
# FSDP (full FT, multi-GPU)
uv run accelerate launch \
--config_file configs/accelerate/fsdp.yaml \
scripts/train.py "<workspace>/<run-name>/config.yaml"
```
**Pass `--disable-progress-bars` to `train.py` whenever stdout is redirected to a log file** (every background run) or running multi-GPU. The Rich progress bar rewrites a single line with carriage returns and does **not** flush parseable newlines to a redirected log, so without this flag the log shows no step/loss lines and you're forced to poll `nvidia-smi`. With it, step/loss lines are written normally and the log is greppable.
## Pre-Launch Checks (every launch)
1. Run the self-check from `references/config-patching.md` (paths exist, frame/resolution constraints, generated modalities have matching latents dirs, references/masks dirs present for conditional modes).
2. Confirm `nvidia-smi` shows expected GPUs available (not occupied by another process).
3. If `wandb.enabled: true`, confirm credentials still resolve: `uv run python -c "import wandb; print(bool(wandb.Api().api_key))"` → `True`. (Don't use `wandb status`.) If `False`, surface to the user before launching — they may want to `wandb login` or run without tracking.
## Run In Background, Monitor Foreground
Long training runs should not block the agent's response loop, and they must survive past the launching turn.
**Prefer the agent's managed/native background-shell mechanism** (the harness facility for long-running background commands — output streaming + PID/exit tracking, survives across turns). It's the reliable way to launch training: it stays alive, streams to a log the agent can poll, and reports completion. Launch the training command through that mechanism, writing to `<workspace>/<run-name>/logs/train.log` with `--disable-progress-bars`, using **absolute paths** for the config and log (`uv run --directory packages/ltx-trainer` changes the cwd, so a relative config path won't resolve).
**`nohup ... &` is a last-resort fallback only.** Detached jobs can be harder to track and may not survive environment/session cleanup, so only use it if no managed background mechanism is available, and verify the PID is still alive afterward:
```bash
# Fallback ONLY — prefer the managed background shell above.
mkdir -p "<workspace>/<run-name>/logs"
nohup uv run --directory packages/ltx-trainer python scripts/train.py \
"<ABSOLUTE path>/<run-name>/config.yaml" --disable-progress-bars \
> "<ABSOLUTE path>/<run-name>/logs/train.log" 2>&1 &
echo $! > "<workspace>/<run-name>/logs/train.pid"
```
## Status Report
Produce on user request (or every <interval> automatically). Pull from:
- **W&B run URL:** First lines of `train.log` after init, or `wandb.run.url` from a `wandb` Python snippet. Surface as a clickable URL.
- **Latest step:** `tail -n 200 "<workspace>/<run-name>/logs/train.log" | grep -oE "step [0-9]+" | tail -1`.
- **Recent loss:** `tail -n 200 "<workspace>/<run-name>/logs/train.log" | grep -oE "loss[: ]+[0-9.]+" | tail -5`.
- **Checkpoints saved:** `ls -1t "<workspace>/<run-name>/outputs/checkpoints/" 2>/dev/null`.
- **Validation samples:** `ls -1t "<workspace>/<run-name>/outputs/samples/" 2>/dev/null`.
- **GPU utilization snapshot:** `nvidia-smi --query-gpu=name,utilization.gpu,memory.used,memory.total --format=csv,noheader`.
- **ETA:** only report one **grounded in real numbers the trainer has actually emitted** — do not invent or "educated-guess" an ETA before the trainer has produced per-step timings. Compute as `(total_steps - current_step) * recent_avg_step_time`, where `recent_avg_step_time` is measured from **steady-state training steps** (the trainer's reported per-step time or log timestamps), **excluding** one-time setup that doesn't repeat per step: model loading, the step-0 validation pass, and periodic validation passes. The trainer's own early ETA projection is skewed high by the slow step-0 validation and settles after a few steps — wait for it to settle rather than quoting the inflated early figure. Until real step timings exist, say "measuring step time…" rather than guessing a duration.
Format the report tightly — one block, no fluff:
```
Step 1240 / 2000 (62%) — loss ~0.072 (last 5: 0.071, 0.073, 0.069, 0.075, 0.072)
ETA: ~1h 24m | GPU: 91% util, 39.2 / 48.0 GB
Latest checkpoint: lora_weights_step_01000.safetensors
Latest validation: samples/step_01200_*.mp4
W&B: https://wandb.ai/<entity>/<project>/runs/<id>
```
## Monitor-Only Mode
Invoked when the orchestrator detects an existing `<workspace>/<run-name>/` with checkpoints or a running process.
1. Check if the training process is live: `[ -f .../logs/train.pid ] && kill -0 $(cat .../logs/train.pid) 2>/dev/null && echo LIVE || echo STOPPED`.
2. Produce the same status report as above.
3. If STOPPED:
- Compute step from last checkpoint.
- Find the latest checkpoint pair (`lora_weights_step_*.safetensors` or `model_weights_step_*.safetensors`, plus a matching `training_state_step_*.pt` when resume state is enabled) under `<workspace>/<run-name>/outputs/checkpoints/`.
- Patch `model.load_checkpoint` in `config.yaml` to point at that checkpoint file (this is the only way the trainer knows to resume — there's no auto-detection from `output_dir`).
- Surface the resume command: `uv run python scripts/train.py "<workspace>/<run-name>/config.yaml"`.
- Ask the user to confirm both the patch and the launch before applying.
## Resume
The trainer **does not auto-resume from `output_dir`**. To resume an interrupted run:
1. Set `model.load_checkpoint` in `config.yaml` to the latest checkpoint file (e.g. `<workspace>/<run-name>/outputs/checkpoints/lora_weights_step_02000.safetensors`).
2. Launch normally. The trainer loads those weights, then looks for a matching `training_state_step_*.pt` **next to the loaded checkpoint** and restores optimizer/scheduler/step state from it. If the state file is missing, weights load but training starts from step 0.
3. To load the weights but skip the state restore (e.g. for branching off into a new run from a known-good checkpoint), set `checkpoints.no_resume: true`.
Always ask the user before patching `model.load_checkpoint` or setting `no_resume` — checkpoints are precious.
## Failure During Training
If the training process exits non-zero:
1. Tail the log and identify the error type.
2. Cross-reference `references/troubleshooting.md`.
3. Propose a config fix (with the exact diff to `config.yaml`).
4. Ask the user before applying. Then resume with the patched config.
Never silently restart a failed training run without acknowledging the failure to the user.
## After Training Completes
1. Write a **run summary** to `<workspace>/<run-name>/outputs/run-summary.md` (see below) so the user can find their bearings months later without rereading the trainer docs.
2. Show final checkpoint path and step count.
3. Show W&B URL if enabled.
4. Return control to the orchestrator (Phase 9 — post-train validate runs next).
### Writing `run-summary.md`
The summary is the **landing page** for this run. Anyone (including the user months from now) should be able to read it and understand: what was trained, on what data, with what config, where everything lives, how to use the result, and how to continue. The trainer doesn't produce this — the skill does.
Template (fill from `plan.md`, `config.yaml`, `autotune.log`, dataset metadata, training log):
```markdown
# <run-name>
**Trained:** <YYYY-MM-DD HH:MM> on <GPU(s)>
**Final checkpoint:** `outputs/checkpoints/<filename>.safetensors` (step <N>)
## What this LoRA does
<One paragraph from the plan's Goal section — restating the user's intent.>
## Trigger word
`<trigger>` — include in prompts at inference time. (Omit this section if no trigger word.)
## Mode
<Mode name> (<lora|full>). Conditioning: <list of conditions, or "none">.
## Dataset
- Source: `<absolute path>`
- Captioning: <`qwen_omni` (Qwen3-Omni-30B via vLLM) | `gemini_flash` | "user-supplied">
- Captioner instruction used: <verbatim string, or "default">
- Samples: <N training> + <K held-out> at <W>x<H>x<F>
- Preprocessed to: `dataset/.precomputed/`
## Training config
Final values after autotune (deltas from baseline noted in `autotune.log`):
| Field | Value |
|-------|-------|
| Optimizer | <...> |
| Mixed precision | <bf16/fp16> |
| Quantization | <...> |
| Gradient checkpointing | <on/off> |
| Batch size × grad accum | <B> × <A> (effective <BxA>) |
| LoRA rank / alpha | <R> / <A> (or "full FT") |
| LoRA target modules | <list> (or "n/a") |
| Steps | <N> |
| Learning rate | <value> |
| Step time (final) | ~<T>s |
| Peak VRAM | ~<V> GB |
Full config: `<workspace>/<run-name>/config.yaml`
## Outputs
- Checkpoints: `outputs/checkpoints/`
- Validation samples (during training): `outputs/samples/`
- Post-train eval renders (if Phase 9 ran): `outputs/eval/`
- W&B run: <url, or "(W&B not enabled)">
## How to use this checkpoint
For inference, point `packages/ltx-pipelines/` at the final checkpoint. Example invocation:
\`\`\`bash
# (Minimal sketch — adapt to the pipeline you're using.)
# load base LTX-2 model + apply this LoRA from outputs/checkpoints/<filename>.safetensors
\`\`\`
## How to continue training
To resume from the final checkpoint (e.g. more steps, different LR), edit `config.yaml`:
\`\`\`yaml
model:
load_checkpoint: "<absolute path to outputs/checkpoints/<filename>.safetensors>"
optimization:
steps: <new total> # trainer resumes optimizer/scheduler/step from the training_state_step_*.pt sitting next to the checkpoint above
\`\`\`
Then re-launch with the same command in the "Launched with" section below.
## How this was launched
\`\`\`
<exact command used, with the workspace's absolute config path>
\`\`\`
## Reproducibility
- Seed: <value>
- Workspace: `<absolute path>`
- Repo commit at launch: `<git rev-parse HEAD output>`
- LTX-2 model: `<model.model_path from config>`
- Text encoder: `<model.text_encoder_path from config>`
```
Write the file using the `Write` tool. Don't embed it in a heredoc — the markdown nested in this skill is illustrative; fill the template with real values from the run's artifacts.
### Next steps (surface to user)
After writing the summary, point the user at:
- The summary file path.
- The final checkpoint path.
- The W&B URL if enabled.
- The upcoming Phase 9 (post-train validate) — the orchestrator handles the transition.
Suggest, but don't run:
- Test inference with `packages/ltx-pipelines/`.
- Push to HF Hub via the trainer's `hub.push_to_hub` config (a separate, lightweight re-launch).
- Continue training from the final checkpoint (see summary's "How to continue training" section).
## Do Not
- Do not modify `output_dir` contents after a run completes — checkpoints belong to the user now.
- Do not start a second training run into the same `output_dir` without explicit user approval. Resume requires patching `model.load_checkpoint` (the trainer does not auto-detect prior checkpoints); a true fresh-start from the same dir additionally needs `checkpoints.no_resume: true`.
- Do not auto-restart a failed run without diagnosis and user approval.
.claude/skills/train-model/phases/post-train-validate.md
+171
View File
@@ -0,0 +1,171 @@
# Phase 9 — Post-Train Validate
Procedure document for the `train-model` orchestrator (Phase 9). Read this file in full before acting on post-train validation.
Goal: render the final checkpoint against three prompt categories so the user can inspect the result and form their own judgement. Save outputs in an organized layout. **Do not** prompt for pass/fail verdicts, do not infer causes for failures, do not suggest fixes — soft training failures don't have a clean if/then rule book, and pretending otherwise wastes the user's time.
The orchestrator's hard invariants apply (see `../SKILL.md`).
## What this phase does
1. Collect prompts for three categories.
2. Render the final LoRA against all collected prompts.
3. Save outputs under `<workspace>/<run-name>/outputs/eval/<category>/`.
4. Print the paths and exit.
## Categories
### 1 — In-distribution
A few captions from the training set itself. Tests whether the model learned what it was shown.
- Default: 3 random captions from the dataset metadata (seed 42 for reproducibility).
- Source: `<workspace>/<run-name>/dataset/dataset.json` (the captions used for training, after the held-out split).
### 2 — Out-of-distribution
Prompts the model has never seen, but in the same domain. Tests whether the model generalizes the concept beyond memorized phrasings.
Ask the user once:
> "For out-of-distribution validation, paste 2–3 prompts you'd realistically want to generate at inference time. (If you don't have any specific ones in mind, reply 'default' — I'll use a few generic prompts that include the trigger word.)"
- On `default`: synthesize 3 short prompts using the LoRA's trigger word and a generic scene context (e.g., "<trigger> walking in a forest at dawn"). Note these are generic — they're better than nothing, but real user-style prompts make a stronger test.
- Otherwise: use the user's prompts verbatim.
- **For a run with a generated audio modality, the synthesized prompts must describe the audio** — matching how the training captions describe it (inspect a few from `dataset.json` first). A prompt with no audio direction leaves the audio branch unguided and the generated audio comes out poor. E.g. for the talking-head case include spoken-voice/room-tone direction; for music/ambience/foley describe the sound character. (Categories 1 and 3 reuse the real captions verbatim, so they already carry audio description — this only applies to the synthesized Category 2 prompts.) If the user pasted their own prompts and it's an audio run, and they omitted audio direction, note that the audio may be weak without it.
### 3 — Held-out
Captions from samples that were never seen during training. Tests true generalization, not memorization.
- Source: `<workspace>/<run-name>/dataset/holdout.jsonl` (written by `prepare-dataset` Step 5).
- Use all entries if there are ≤5; otherwise sample 5 with seed 42.
- **If holdout doesn't exist** (dataset was too small, or user-skipped during prepare): print a clear note in the output summary — *"Held-out evaluation skipped: no holdout set was reserved for this run. The post-train eval only covers Categories 1 and 2."* Don't synthesize substitutes.
## Rendering mechanism
Use the trainer's existing validation infrastructure rather than wiring up `ltx-pipelines` from scratch. Create a temporary "validate-only" config and run the trainer with it.
### Step 1 — Build eval config
Copy `<workspace>/<run-name>/config.yaml` to `<workspace>/<run-name>/eval-config.yaml`. Patch:
```yaml
model:
load_checkpoint: "<absolute path to outputs/checkpoints/<final-lora>.safetensors>"
optimization:
steps: 1 # we don't want to train; we want validation to fire
# Keep batch_size/grad_accum at the run's autotuned values to match its VRAM footprint.
validation:
skip_initial_validation: false
interval: 1 # run validation at step 0 (and at the only training step)
samples:
# Inject all collected prompts here, tagged by category in the prompt itself
# so the output filenames make the category obvious.
- prompt: "[CAT1-IND] <caption from dataset.json>"
# ... repeat for each prompt in all three categories
# Keep video_dims, frame_rate, guidance/STG settings as the trained config.
# Keep generate_audio consistent with the trained modality config.
checkpoints:
interval: null # do not save more checkpoints
no_resume: true # load the LoRA's weights but do not restore optimizer/scheduler/step state
output_dir: "<workspace>/<run-name>/outputs/eval"
```
The `[CAT1-IND]`, `[CAT2-OOD]`, `[CAT3-HELDOUT]` tags in the prompt strings make the output MP4 filenames self-describing in the trainer's validation sample directory.
**Attach the mode's conditions to each sample.** A bare `prompt` only validates a pure text-to-X mode (T2V, T2A). For any conditioned mode, the trained model expects the same conditioning at validation time — a prompt with no conditions tests a different task than what was trained, and conditioned modes may fail outright. Add the `conditions` list that matches the run's mode (the trained `config.yaml` `training_strategy` and the example config for the mode are the reference):
| Mode | Add to each sample |
|------|--------------------|
| I2V | `conditions: [{type: first_frame, image_or_video: <frame/clip path>}]` |
| Video extension / suffix | `conditions: [{type: prefix|suffix, ...}]` |
| V2V / AV2AV IC-LoRA | `conditions: [{type: reference, ...}]` (point at a held-out reference) |
| V2A (foley) | `conditions: [{type: video_to_audio, ...}]` |
| A2V | `conditions: [{type: audio_to_video, ...}]` |
| Inpainting (video/audio) | `conditions: [{type: mask, ...}]` |
| Outpainting | `conditions: [{type: spatial_crop, ...}]` |
| A2A IC-LoRA | `conditions: [{type: reference, ...}]` (held-out reference audio) |
| T2V, T2A | none — a bare `prompt` is correct |
Mirror the condition shapes used in the mode's example config under `packages/ltx-trainer/configs/`. For held-out (Category 3) and OOD (Category 2) samples on conditioned modes, draw the conditioning media from the held-out set so the eval stays out-of-distribution.
### Step 2 — Run the trainer in validate-only mode
```bash
cd packages/ltx-trainer
uv run python scripts/train.py "<workspace>/<run-name>/eval-config.yaml"
```
The trainer will load the LoRA, run initial validation against all the prompts, do one trivial training step (which we discard), and exit. Validation samples land in `<workspace>/<run-name>/outputs/eval/samples/`.
### Step 3 — Organize outputs and clean up trainer cruft
The validate-only run is a trainer run, so it inevitably writes throwaway artifacts: an indexed `samples/` dir, a forced final checkpoint (the trainer **always** saves one at the end, regardless of `checkpoints.interval`), and a `training_config.yaml`. Don't leave these around or duplicate the renders.
1. **Move** (don't copy) each generated MP4 from the trainer's indexed `samples/` dir into the category layout, naming by category + a slug of the prompt. Use the index→category mapping you built when constructing `validation.samples`.
2. **Delete the trainer cruft** from the eval dir once the renders are moved: the indexed `samples/` dir, the forced `checkpoints/` dir, and `training_config.yaml`. (These are byproducts of the validate-only hack — there's no config flag to suppress the final-checkpoint save, so clean it up here.)
3. Write **one** manifest, `outputs/eval/prompts.json` (filename → full prompt + category). Don't leave a second copy elsewhere.
4. Remove the temporary `eval-config.yaml` (or keep it under the run's scratch, not in `outputs/`).
Final eval layout — exactly this, nothing else:
```
<workspace>/<run-name>/outputs/eval/
in-distribution/ <NN>_<prompt-slug>.mp4 ...
out-of-distribution/ <NN>_<prompt-slug>.mp4 ...
held-out/ <NN>_<prompt-slug>.mp4 ... # only if a holdout set existed
prompts.json # filename -> full prompt + category (single manifest)
```
No `eval/samples/`, no `eval/checkpoints/`, no `eval/training_config.yaml`, no duplicate manifest.
### Step 4 — Surface paths
Print a tight block, no judgement, no follow-up question:
```
Post-train evaluation complete.
In-distribution renders (<K> samples):
<workspace>/<run-name>/outputs/eval/in-distribution/
Out-of-distribution renders (<M> samples):
<workspace>/<run-name>/outputs/eval/out-of-distribution/
Held-out renders (<N> samples):
<workspace>/<run-name>/outputs/eval/held-out/ # or: "(skipped — no holdout set)"
Open the MP4s and decide for yourself whether the model is good. Soft
training quality is judged by watching the videos, not by a checklist —
there's no substitute for your own eyes here.
```
Return control to the orchestrator. The orchestrator's run is now complete.
## What this phase does NOT do
- Does not ask "is this good?" / "pass / partial / fail?".
- Does not infer failure causes.
- Does not suggest fixes, follow-up runs, hyperparameter changes, dataset changes.
- Does not write any verdict to `run-summary.md` or elsewhere.
- Does not delete or modify training checkpoints.
- Does not push to any remote / cloud / registry.
The user looks at the videos and makes their own call. If they want to iterate, they re-invoke the orchestrator with a new run-name.
## Failure modes
- **Final checkpoint missing.** Surface and stop. Don't render against an intermediate checkpoint without explicit user consent.
- **`load_checkpoint` OOM at inference time.** Lower `validation.video_dims` in the eval config (smaller renders are still useful for a sanity look). Retry once. If still OOM, surface the failure and let the user run inference manually via `packages/ltx-pipelines/`.
- **All renders look broken/black.** May be an inference-pipeline-side issue rather than a training failure. Mention in the output block: *"If renders look broken across all categories, try `packages/ltx-pipelines/` directly to rule out a pipeline issue."* Then exit. Do not investigate further.
## Do not
- Do not skip Category 1 or 2. They're cheap and informative.
- Do not invent a held-out set if `holdout.jsonl` is missing — the prepare-dataset step decides that.
- Do not coach the user on what "good" means for their use case.
.claude/skills/train-model/phases/prepare-dataset.md
+245
View File
@@ -0,0 +1,245 @@
# Phase 5 — Prepare Dataset
Procedure document for the `train-model` orchestrator (Phase 5). Read this file in full before acting on the prepare-dataset phase.
Goal: produce a captioned, complete dataset metadata file at `<workspace>/<run-name>/dataset/dataset.json` consumable by `process_dataset.py`. Idempotent — re-runs skip work already done.
The orchestrator's hard invariants apply (see `../SKILL.md`), especially: **no file mutation outside the workspace without explicit user approval.**
## Inputs
The orchestrator passes:
- Source path (directory of videos / single video / pre-existing metadata file).
- Target mode (T2V, I2V, V2V IC-LoRA, V2A, etc.) — determines which columns are required.
- Captioner backend choice (Qwen3-Omni local vLLM server / Gemini Flash cloud / skip).
- Workspace path `<workspace>/<run-name>/`.
## Required Columns by Mode
`process_dataset.py` detects columns by convention and resolves each to a role. The media column may be `video` **or** `audio`. When a dataset has a `video` column with an audio track and no separate `audio` column, audio is **auto-extracted** from the video (unless `--skip-audio`), so an explicit `audio` column is only needed when the audio lives in separate files.
**Video-generating modes** (need a `video` column):
| Mode | Required | Optional |
|------|----------|----------|
| T2V, I2V, video extension/suffix | `video`, `caption` | `audio` (else auto-extracted) |
| Video outpainting | `video`, `caption` | |
| Video inpainting | `video`, `caption`, `video_mask` | |
| V2A (foley) | `video`, `caption` | `audio` (target; else auto-extracted) |
| A2V | `video`, `caption` | `audio` (else auto-extracted from video) |
| V2V IC-LoRA | `video`, `caption`, `reference_video` | |
| AV2AV IC-LoRA | `video`, `caption`, `reference_video`, `reference_audio` | `audio` (else auto-extracted) |
**Audio-only modes** (no `video` column — the media column is `audio`):
| Mode | Required | Optional |
|------|----------|----------|
| T2A | `audio`, `caption` | |
| Audio extension/suffix | `audio`, `caption` | |
| Audio inpainting | `audio`, `caption`, `audio_mask` | |
| A2A IC-LoRA | `audio`, `caption`, `reference_audio` | |
Aliases: `media_path` for `video`, `ref_media_path` for `reference_video`.
## Workflow
### Step 1 — Classify source
```bash
# If source is a file, identify type:
file "<source>"
# If source is a directory, count media:
find "<source>" -maxdepth 1 -type f \( -name "*.mp4" -o -name "*.mov" -o -name "*.webm" \) | wc -l
```
Cases:
- **Pre-existing metadata file** (CSV/JSON/JSONL) → copy to `<workspace>/<run-name>/dataset/dataset.json`, audit columns. Skip to Step 4.
- **Directory of short scenes** → skip Step 2, go to Step 3.
- **Directory containing long videos** → run Step 2.
- **Single long video** → run Step 2.
**Stage the media under `dataset/` before captioning — don't discover this by failing.** Both `caption_videos.py` and `process_dataset.py` reference media by paths **relative to the metadata file's own directory**, so the media must live under `<workspace>/<run-name>/dataset/`. Stage it up front into `dataset/videos/` (and write metadata paths relative to `dataset/`, e.g. `videos/1.mp4`):
- Prefer **symlinks** (instant, no disk cost): `ln -s <abs-source>/<clip> <workspace>/<run-name>/dataset/videos/<clip>`. Symlinks pointing at the original source location work correctly.
- Use a **copy** instead if the workspace and source are on different filesystems or the source may move/change.
- Never caption or preprocess directly against an out-of-tree source path (e.g. `/path/to/source-videos`) — it will fail the relative-path resolution. The original source is left untouched either way.
(Scene-split output in Step 2 already lands under `dataset/scenes/`, which satisfies this.)
### Step 2 — Scene splitting (only for long videos)
`split_scenes.py` takes **one video file** at a time (`video_path` and `output_dir` are both positional arguments). When the source is a directory of long videos, iterate over each file. To drop scenes shorter than 2 seconds, use `--filter-shorter-than 2s` (the `--min-scene-length` option is an integer **frame** count, not seconds — don't pass a float).
```bash
cd packages/ltx-trainer
# Single file:
uv run python scripts/split_scenes.py \
"<video-file>" \
"<workspace>/<run-name>/dataset/scenes" \
--filter-shorter-than 2s
# Directory of long videos — iterate:
for f in "<source>"/*.mp4 "<source>"/*.mov "<source>"/*.webm; do
[ -e "$f" ] || continue
uv run python scripts/split_scenes.py "$f" \
"<workspace>/<run-name>/dataset/scenes" \
--filter-shorter-than 2s
done
```
Result: scenes saved to `<workspace>/<run-name>/dataset/scenes/`. Pass that directory to Step 3.
### Step 3 — Captioning
Skip entirely if a metadata file with all required `caption` entries already exists.
**Use the captioner's default instruction.** `caption_videos.py` ships a well-tuned default caption prompt — use it as-is (do **not** pass `--instruction`). Captioning runs in two phases: a small **spot-check pass** so the user can confirm the captions look sane, then a **full pass** on the rest.
A custom `--instruction` is the exception, not the norm. Only use one when:
- the **nature of the dataset genuinely demands it** (e.g. a narrow domain the default prompt won't describe well), or
- the **user, after seeing the spot-check captions, explicitly asks** for a change (e.g. "too much background detail").
Do not invent a custom instruction pre-emptively, and in particular **do not bake a subject name / trigger word into the captions via `--instruction`** — the trigger word is handled separately at preprocessing (see "Trigger word" below).
#### Choosing a backend
Two backends, with very different hardware needs:
- **`qwen_omni` (local, default):** Qwen3-Omni-30B-A3B-Thinking served by a local vLLM HTTP server (`serve_captioner.py`). ~65 GiB model download. Default **FP8** quantization uses ~31 GiB of weights and **fits on a 40 GiB GPU** (plus KV cache); **bf16** uses ~60 GiB and needs **≥66 GiB free VRAM**.
- **`gemini_flash` (cloud):** Google `gemini-3.5-flash`. No local model, runs anywhere, parallelisable with `--num-workers`.
**Steer modest hardware to Gemini.** If the GPU is below ~40 GiB (i.e. typical consumer cards — 24 GB / 32 GB), it can't host even the FP8 server, so the local captioner isn't an option — recommend `gemini_flash` and tell the user they'll need Gemini auth: either a `GEMINI_API_KEY`/`GOOGLE_API_KEY` (get one at <https://aistudio.google.com/apikey>) or working gcloud/Vertex AI credentials. If they can't or won't set that up and the hardware can't run Qwen3, the only remaining path is bringing their own captions in the dataset metadata (skip captioning entirely). On a 40 GiB+ GPU the local server is viable (FP8); bf16 needs an 80GB-class card.
#### Qwen server prerequisite (qwen_omni only)
The local backend talks to a vLLM server that must already be running. Launch it once in a **separate terminal** (it stays loaded across captioning runs):
```bash
cd packages/ltx-trainer
uv run python scripts/serve_captioner.py # FP8 by default, serves on http://127.0.0.1:8001/v1
# bf16 (needs >= 66 GiB free VRAM): --quantization bf16
# different port/interface: --port 9000 --host 0.0.0.0
```
First launch downloads the model (~65 GiB). `caption_videos.py` reaches it via `--vllm-url` (default `http://127.0.0.1:8001/v1`). Skip this entirely when using `gemini_flash`.
#### 3a — Spot-check pass (3 samples)
Caption 3 samples with the **default prompt** (no `--instruction`) to confirm the captioner is producing sane output before committing to the whole set.
```bash
cd packages/ltx-trainer
# qwen_omni (server from the previous step must be running):
uv run python scripts/caption_videos.py \
"<workspace>/<run-name>/dataset/videos/<one-staged-clip>" \
--output "<workspace>/<run-name>/dataset/preview-captions.json" \
--captioner-type qwen_omni
# Point at 3 staged clips under dataset/videos/ (a small subdir or 3 explicit files) — not the out-of-tree source.
# Optional: --vllm-url http://127.0.0.1:9000/v1 (if the server uses a non-default port)
```
Print the 3 captions **in full** to the user, then **STOP and wait** for their explicit verdict:
> "Here are sample captions from the default prompt. Please review them — reply 'good' to caption the rest, or tell me what to change."
**This is a hard gate. Do NOT caption the full set until the user explicitly approves the samples.** Do not auto-proceed, do not assume "looks fine," do not batch this with other questions. The user must either approve or give tuning instructions first — the whole point of the spot-check is to let them judge caption quality and content before paying for the full pass.
If the user requests changes, introduce a custom `--instruction` (or switch backend), re-run the spot-check on the same 3 samples, show the new captions, and **stop for approval again**. Loop until the user approves. If a custom instruction still isn't converging after a few rounds, switch captioner backend or have the user supply a few manual captions as examples — but still don't proceed to the full set without their OK.
#### 3b — Full pass
Run on the **staged media dir** (`dataset/videos/` from Step 1, or `dataset/scenes/` from Step 2) with the **default prompt** (or the same `--instruction` only if one was explicitly agreed in 3a). Caption the staged in-tree media — not the original out-of-tree source path.
**Qwen3-Omni (local — server must be running):**
```bash
cd packages/ltx-trainer
uv run python scripts/caption_videos.py \
"<workspace>/<run-name>/dataset/videos" \
--output "<workspace>/<run-name>/dataset/dataset.json" \
--captioner-type qwen_omni
```
**Gemini Flash (cloud — runs anywhere, parallelisable):**
```bash
# Auth: GEMINI_API_KEY / GOOGLE_API_KEY env var, or gcloud / Vertex AI credentials.
cd packages/ltx-trainer
uv run python scripts/caption_videos.py \
"<workspace>/<run-name>/dataset/videos" \
--output "<workspace>/<run-name>/dataset/dataset.json" \
--captioner-type gemini_flash \
--num-workers 5
```
Output: JSON list of `{caption, media_path}` with paths **relative to the output file location**. The 3 spot-check captions can be merged in to avoid re-captioning them.
#### Trigger word (handled at preprocessing, not in captions)
For style/concept LoRAs the trigger word is **not** written into the captions here. It is prepended to every caption at preprocessing: pass `--lora-trigger "<word>"` to `process_dataset.py` in Phase 7, which forwards it to the caption-processing step (`process_captions.py`, where the prepend actually happens). That is the canonical mechanism — keep the captions describing what's actually on screen (via the default prompt), and let the trigger flag bind the concept to the token. Record the chosen trigger word in the plan so Phase 7 passes it through. Do not also bake the word into captions (it would double up).
**The injection mechanism is a fixed implementation detail — never make it a user-facing question.** The *only* trigger-word thing to ask the user is the **word itself** (or whether they want a trigger word at all). Do **not** ask, mention, or present as an option *how* it gets injected (e.g. "inject into the caption vs via `process_dataset`") — it is always `--lora-trigger`, full stop. Surfacing the method as a choice creates unnecessary confusion.
### Step 4 — Conditioning inputs (modes that need references or masks)
Some modes need a per-sample conditioning input beyond the video/audio and caption:
| Mode | Required extra input | Column |
|------|----------------------|--------|
| V2V IC-LoRA | reference video | `reference_video` (alias `ref_media_path`) |
| AV2AV IC-LoRA | reference video + reference audio | `reference_video`, `reference_audio` |
| A2A IC-LoRA | reference audio | `reference_audio` |
| Video inpainting | per-frame video mask | `video_mask` |
| Audio inpainting | audio mask | `audio_mask` |
These inputs encode **the user's specific idea** for the LoRA (what the reference represents, which regions the mask covers). There is no universal recipe, so **do not invent or default to a particular method** (e.g. don't assume Canny edges, depth, pose, or some generic box/border mask). The agent must not pick the conditioning semantics for the user.
Workflow when the chosen mode needs one of these and the dataset doesn't already provide it:
1. **Check first** — if the user already supplied the column (and the files exist), use it as-is and move on.
2. **Otherwise, ask the user to provide it**, explaining concretely what's needed: the column name, that it's one file per sample aligned to each clip, and that the *content/semantics are their call* (what the reference should depict, what the mask should cover). Make clear this reflects their specific use-case — you won't guess it.
3. **Help generate only if the user asks.** If they say "can you generate the references/masks by doing X" (X = their described method), then help: write or run a small script for *their* approach, or use a repo tool if it fits. One such tool exists — `scripts/compute_reference.py` generates **Canny edge** reference videos — but only mention/use it if the user specifically wants Canny; never offer it as the default.
4. **Hard gate:** do not proceed to preprocessing for a conditioning mode until the required column is present with real files. Surface clearly if it's missing.
**Column-naming note (if references are generated):** `compute_reference.py` writes a `reference_video` field, which
`process_dataset.py` detects automatically. Legacy datasets using `ref_media_path` also work.
### Step 5 — Holdout split
Reserve a subset of samples as a **held-out set** never seen during training. This is what Phase 9 (post-train validate) renders against to test true generalization rather than memorization.
Decision tree:
1. **User already supplied a held-out set** (separate file or directory they explicitly nominated): do not split. Copy/reference their file to `<workspace>/<run-name>/dataset/holdout.jsonl` and leave `dataset.json` as-is.
2. **Small dataset** (judge qualitatively; tens of samples or fewer): holding samples out meaningfully reduces training capacity. Ask the user:
> "Dataset has <N> samples. Reserving a holdout meaningfully reduces what's available for training. Options: (a) reserve 1–2 for holdout, (b) skip holdout — post-train eval will only test in-distribution. Your call."
3. **Otherwise:** auto-split. Reserve a small fraction (this skill's default: roughly 10% of samples, bounded so the holdout doesn't grow huge — a handful of held-out samples is usually enough). Use seed 42 for the split so it's reproducible. Surface the count and the picked IDs in the plan.
After splitting, write `<workspace>/<run-name>/dataset/holdout.jsonl` (one JSON object per line with the same columns as `dataset.json`). **Remove the held-out entries from `dataset.json`** so they don't enter preprocessing or training.
Always print:
> "Held out <K> of <N> samples for post-train evaluation. Held-out IDs: <list>."
If holdout is skipped, surface in the plan: *"Skipping holdout — dataset is too small. Post-train eval will only render in-distribution prompts; true generalization isn't testable for this run."*
### Step 6 — Audit
Before returning to the orchestrator, verify the metadata file has all required columns for the chosen mode. Print a one-line summary:
> "Prepared <N> training samples (+ <K> held out) for <mode>. Columns: <list>. Saved to `<workspace>/<run-name>/dataset/dataset.json` (+ `holdout.jsonl`)."
## Idempotency
- If `dataset.json` already exists and all required columns are present: skip captioning. Confirm reuse with the user only if the file was supplied by them outside the workspace (per the orchestrator's file-safety invariant).
- If captioning was partial (some entries missing `caption`), re-run captioning only on the missing entries by filtering the metadata file before passing to `caption_videos.py`.
## Failure Modes
- Qwen server won't start / OOMs on launch → use the default `--quantization fp8` (not `bf16`), lower `--gpu-memory-utilization`, or reduce `--max-model-len` on `serve_captioner.py`. If the GPU simply can't host a 30B model, switch to `gemini_flash`.
- `caption_videos.py` can't connect (qwen_omni) → the vLLM server isn't running or `--vllm-url` is wrong. Start `serve_captioner.py` first and confirm the URL/port match.
- Gemini rate-limit → reduce `--num-workers`, retry.
- Scene splitter produces 0 scenes → the detector found no cuts, or `--filter-shorter-than` removed everything. Lower/remove `--filter-shorter-than`, or adjust the detector threshold. (`--min-scene-length` is an integer minimum-frames-per-scene for the detector, not a short-scene filter.)
- IC-LoRA reference compute fails on some frames → script logs the failures; report counts to the user and ask whether to proceed with the remaining samples or stop.
## Do Not
- Do not move or rename the user's source files. The skill reads them in place; the workspace contains only **derived** artifacts.
- Do not delete `scenes/` or partial captioning outputs without approval — they may be expensive to regenerate.
.claude/skills/train-model/phases/preprocess-dataset.md
+130
View File
@@ -0,0 +1,130 @@
# Phases 6 (one-sample) & 7 (full) — Preprocess Dataset
Procedure document for the `train-model` orchestrator. Read this file in full before acting on the preprocess phase.
Goal: run `process_dataset.py` to produce VAE latents, audio latents, and text embeddings. Two modes:
1. **One-sample** (Phase 6 sanity check) — preprocess a single sample to `<workspace>/<run-name>/overfit/.precomputed/`.
2. **Full** (Phase 7) — preprocess the whole dataset to `<workspace>/<run-name>/dataset/.precomputed/`.
The orchestrator's hard invariants apply (see `../SKILL.md`), especially: **never silently overwrite existing user data.**
## Required Subdirectories by Mode
Under `.precomputed/`:
| Subdir | Required for |
|--------|--------------|
| `latents/` | Video-bearing modes only (T2V, I2V, video extend/inpaint/outpaint, V2V/AV2AV IC-LoRA, A2V, V2A). **Not** produced for audio-only modes. |
| `conditions/` | Always (text embeddings) |
| `audio_latents/` | Any mode with audio: video modes carrying audio, plus all audio-only modes (T2A, audio extend/suffix/inpaint, A2A IC-LoRA) |
| `reference_latents/` | V2V IC-LoRA, AV2AV IC-LoRA |
| `reference_audio_latents/` | A2A IC-LoRA, AV2AV IC-LoRA |
| `video_masks/` | Video inpainting |
| `audio_masks/` | Audio inpainting |
**Audio-only modes** (T2A, audio extend/suffix, audio inpainting, A2A IC-LoRA) produce `audio_latents/` + `conditions/` (plus `audio_masks/` or `reference_audio_latents/` as applicable) and **no `latents/`**. Do not flag a missing `latents/` as incomplete for these modes.
## Workflow — Full Preprocess
### Step 1 — Verify existing `.precomputed/` (if present)
If `<workspace>/<run-name>/dataset/.precomputed/` already exists:
1. List subdirectories present. Confirm all required for the chosen mode are present.
2. Load one sample per modality and check tensor shapes:
```bash
uv run python -c "import torch; t = torch.load('<path>'); print(t.shape if hasattr(t, 'shape') else {k: v.shape for k, v in t.items()})"
```
3. Compare shapes against the target resolution from the plan.
**On any mismatch or missing subdirectory: STOP. Do not run `process_dataset.py`.** Ask the user via `AskUserQuestion`:
- Reuse the existing data at its current resolution (update plan + config accordingly).
- Re-preprocess to a new directory (`<workspace>/<run-name>/dataset/.precomputed-v2/` etc.) — preserves the existing data.
- Abort.
**Never pass `--overwrite` without explicit user approval** for this exact action.
### Step 2 — Invoke `process_dataset.py`
```bash
cd packages/ltx-trainer
uv run python scripts/process_dataset.py \
"<workspace>/<run-name>/dataset/dataset.json" \
--resolution-buckets "<W>x<H>x<F>" \
--model-path "<absolute-model-path>" \
--text-encoder-path "<absolute-gemma-path>" \
--output-dir "<workspace>/<run-name>/dataset/.precomputed" \
--load-text-encoder-in-8bit # on 32GB tier (low-VRAM config), per t2v_lora_low_vram.yaml
```
Add as needed:
- `--skip-audio` — if mode doesn't use audio (T2V video-only variants).
- `--audio-durations "<list>"` — for T2A from a captions-only file.
- `--lora-trigger "<trigger>"` — for style/concept LoRAs.
- `--reference-downscale-factor <N>` — for IC-LoRA modes if downscaled references are desired.
- `--video-column`, `--caption-column` — only if the metadata file uses non-standard column names.
**Do not pass `--overwrite`** unless re-preprocessing was explicitly approved in Step 1.
### Step 3 — Audit output
After completion, verify:
```bash
ls "<workspace>/<run-name>/dataset/.precomputed/"
# Expected subdirs per the mode table above.
# Count files in each:
for d in latents conditions audio_latents reference_latents video_masks audio_masks; do
if [ -d "<workspace>/<run-name>/dataset/.precomputed/$d" ]; then
echo "$d: $(ls "<workspace>/<run-name>/dataset/.precomputed/$d" | wc -l)"
fi
done
```
Counts in each required subdir should equal the dataset sample count.
**Reconcile counts — do this for every run, any dataset size.** Compare the `latents/` (and `audio_latents/`) count against the caption/sample count. If fewer latents were produced, `process_dataset.py` **silently skipped** clips — most commonly because they were **shorter than the target frame bucket** (it logs each skip). When counts don't match:
1. Identify which clips were dropped (grep the preprocess log for skip/"fewer frames" lines, or diff the produced `.pt` stems against the metadata).
2. **Surface it to the user** with the count and the specific clips — never silently proceed on a shrunk dataset.
3. Offer options: re-preprocess at a **smaller frame bucket** the clips support, add a **second (shorter) bucket** to keep the short clips (multi-bucket requires `batch_size: 1`), or accept the loss. Let the user decide.
**Audio gate (hard stop for audio runs).** For any run with an audio modality (joint audio+video, A2V, V2A, T2A, audio-only modes), verify `audio_latents/` is **present and non-empty** with one `.pt` per sample. `process_dataset.py` **swallows audio-decode errors and continues** — it logs "0 videos with audio" and produces empty `audio_latents/` rather than failing. If an audio run produced no audio latents, **stop** — do not proceed to training (it would silently train audio-free). The usual cause is a broken audio decode path (e.g. missing/incompatible `torchcodec`); confirm `uv run python -c "import torchaudio; torchaudio.load('<a clip>')"` works (see `references/troubleshooting.md`), fix it, then re-preprocess with `--overwrite`.
## Workflow — One-Sample (Phase 6)
Same as full preprocess, but operate on a single-sample metadata file:
1. Pick the first sample from `<workspace>/<run-name>/dataset/dataset.json` and write a one-sample metadata file **inside the dataset dir** — e.g. `<workspace>/<run-name>/dataset/_one_sample.json` — copying the entry **verbatim, keeping its relative `media_path`**. `process_dataset.py` resolves media paths relative to the metadata file's own directory, so the one-sample file must sit beside the real media (i.e. in `dataset/`, the same dir as `dataset.json`). **Do not** place it in `overfit/` and **do not** rewrite the path to an absolute one — an absolute path produces mirrored nested output dirs (`.precomputed/latents/absolute/path/.../x.pt`) instead of a clean `latents/x.pt`.
2. Run `process_dataset.py` on that file with `--output-dir "<workspace>/<run-name>/overfit/.precomputed"` (output still goes to `overfit/`, only the metadata lives in `dataset/`).
3. Use the **same `--resolution-buckets`** as the planned full run. Critical: a small-shape sanity check is misleading because resolution is the dominant memory factor.
4. Clean up the temporary `dataset/_one_sample.json` afterward (it's scratch; don't leave it in the dataset dir).
## Decode-and-Verify (optional debug aid)
If the user reports validation samples look wrong or training diverges, decode one preprocessed sample back to media:
`decode_latents.py` takes the **latents directory** and an **output directory** as positional arguments (it decodes the whole directory, not a single `.pt` file). Add `--with-audio` and `--audio-latents-dir` if the dataset has audio.
```bash
cd packages/ltx-trainer
uv run python scripts/decode_latents.py \
"<workspace>/<run-name>/dataset/.precomputed/latents" \
"<workspace>/<run-name>/dataset/.precomputed/decoded_check" \
--model-path "<absolute-model-path>"
```
If decoded output is garbled, preprocessing itself is suspect (wrong model, wrong VAE).
## Failure Modes
- **"shape mismatch" on resume:** Step 1's verification check. Ask user before any mutation.
- **`frames % 8 != 1`** error from process_dataset.py: the requested frame count is invalid; correct in the plan and re-launch.
- **VRAM OOM during preprocessing:** add `--load-text-encoder-in-8bit`. If still OOM, reduce `--batch-size`.
- **Disk full:** preprocessed latents can be large (especially audio). Surface to user with a `du -sh` summary of `.precomputed/`.
## Do Not
- Do not delete or overwrite existing `.precomputed/` data without explicit user approval for that exact action.
- Do not preprocess at a smaller resolution to "save time" — the sanity check exists specifically to validate the planned resolution.