LTX-2/.claude/skills/train-model/references/troubleshooting.md

# Troubleshooting

Quick lookup for failures during sanity check, preprocessing, or training. For deeper coverage see `packages/ltx-trainer/docs/troubleshooting.md`.

## OOM During Training Step

Order of operations (cheapest first):

1. `optimization.enable_gradient_checkpointing: true` (if not already on).
2. `optimization.batch_size: 1` and increase `gradient_accumulation_steps` to preserve effective batch.
3. `optimization.optimizer_type: "adamw8bit"`.
4. `acceleration.quantization: "int8-quanto"`.
5. Reduce `lora.rank` (32 → 16 → 8). Alpha follows rank.
6. Reduce target resolution (`validation.video_dims` and re-preprocess the dataset at the new resolution).

The last option is expensive — flag it clearly to the user before re-preprocessing.

## OOM During Validation Sample Generation

The validation pass loads decoders + runs CFG/STG inference; it can OOM even when the training step fits.

1. `acceleration.load_text_encoder_in_8bit: true` (trainer config, not the dataset script).
2. `acceleration.offload_optimizer_during_validation: true` (especially for full FT or high-rank LoRA).
3. Reduce `validation.video_dims` (smaller validation than training is fine — it's only for visual feedback).
4. Reduce `validation.inference_steps` (e.g. 30 → 20).
5. Increase `validation.interval` to validate less often.

## NaN Loss

1. Check `acceleration.mixed_precision_mode`: prefer `"bf16"`. If `"fp16"`, switch.
2. Verify dataset latents are well-formed: `uv run python scripts/decode_latents.py <latents-dir> <output-dir> --model-path <model>` (it decodes a whole latents directory, not a single `.pt`) should reconstruct sensibly.
3. Lower `optimization.learning_rate` by 5x.
4. Add `optimization.max_grad_norm: 1.0` (default; verify it's set).
5. If using `quantization`, try `null` — INT8/INT4 quantization can interact badly with poorly-conditioned LoRA inits at high LR.

## Validation Samples Look Wrong but Loss Is Fine

Often not a bug — validation uses simplified inference. For real quality assessment, run a checkpoint through `packages/ltx-pipelines/` after training.

## Trainer Won't Start: Config Validation Error

Pydantic `extra="forbid"` means typos in field names fail loudly. Read the error carefully — it names the offending field and path. Fix and re-launch.

Common offenders:
- `latents_dir` typo or wrong relative path.
- A required field genuinely missing after copying an example (most fields have defaults; check the error message for the exact field path).
- `target_modules` listed at wrong nesting level (must be under `lora:`).

## Trainer Won't Start: Missing Files

- `model_path` not found → re-probe `/models/`, `~/models/`, `$LTX_MODELS_DIR`, or ask the user.
- `text_encoder_path` directory missing the Gemma config → ensure the path is to the Gemma model dir, not its parent.
- `preprocessed_data_root` doesn't contain expected subdirs → re-verify Phase 7 ran for the chosen mode (see `phases/preprocess-dataset.md`).

## Resume Stops Working

The trainer **does not auto-resume from `output_dir`**. Resume happens only when `model.load_checkpoint` is explicitly set to a checkpoint file; the trainer then loads those weights and looks for a `training_state_step_*.pt` next to that file to restore optimizer/scheduler/step. Common pitfalls:
- `model.load_checkpoint` not set or set to the wrong path → fresh run from step 0 even when `outputs/checkpoints/` is full of artifacts. Patch `model.load_checkpoint` to the latest checkpoint.
- `checkpoints.no_resume: true` is set → weights load but state is discarded. Remove the flag if you want a proper resume.
- `training_state_step_*.pt` missing from next to the loaded checkpoint → weights load but step counter resets. Make sure the state file accompanies the checkpoint.
- `training_state_step_*.pt` corrupted (size 0, fails `torch.load`) → trainer falls back to step 0 with a warning.

## Autotune Trial Failed Mid-Sweep

- If trial 2 (quantization off) OOMs: revert to trial 1 and stop the sweep. The 32GB tier baseline is correctly aggressive.
- If trial 3 (adamw) OOMs: revert to adamw8bit. Continue with trial 4 if VRAM headroom allows.
- If trial 4 (batch_size up) OOMs: revert and stop. We've found the ceiling.

Never carry over a failing trial's deltas. Always revert to the last-known-good before the next change.

## Captioning Is Slow

- Local `qwen_omni` is a 30B model served by `serve_captioner.py` (vLLM). If the server won't start or OOMs on launch: keep the default `--quantization fp8` (don't use `bf16` unless ≥66 GiB free VRAM), lower `--gpu-memory-utilization`, or reduce `--max-model-len`. If the GPU can't host a 30B model at all, switch to `gemini_flash`.
- `caption_videos.py --captioner-type qwen_omni` errors connecting → the vLLM server isn't running or `--vllm-url` doesn't match. Start `serve_captioner.py` first and confirm the port.
- For most hardware and for larger datasets, prefer `--captioner-type gemini_flash --num-workers <N>` (needs Gemini auth: `GEMINI_API_KEY`/`GOOGLE_API_KEY` or gcloud/Vertex) — runs anywhere and parallelises; local Qwen needs a heavy GPU and a running server.

## Process_Dataset Errors

- "frames divisible by..." → the video doesn't have enough frames at the requested temporal resolution. Either shorten the requested frame count or use `split_scenes.py` to break long videos.
- "shape mismatch" on existing `.precomputed/` → user requested a different resolution than the existing data. Per the invariants, **stop and ask** — do not overwrite. Offer: reuse at old resolution / re-preprocess to a new dir / abort.

## When To Give Up and Ask The User

If a fix isn't obvious from this file or `packages/ltx-trainer/docs/troubleshooting.md` within two attempts, stop and surface the full error + the steps already tried to the user. Don't loop indefinitely on autonomous fixes — the user has context the agent doesn't (which checkpoints are precious, what they care about preserving, etc.).