# Phases 6 (one-sample) & 7 (full) — Preprocess Dataset Procedure document for the `train-model` orchestrator. Read this file in full before acting on the preprocess phase. Goal: run `process_dataset.py` to produce VAE latents, audio latents, and text embeddings. Two modes: 1. **One-sample** (Phase 6 sanity check) — preprocess a single sample to `//overfit/.precomputed/`. 2. **Full** (Phase 7) — preprocess the whole dataset to `//dataset/.precomputed/`. The orchestrator's hard invariants apply (see `../SKILL.md`), especially: **never silently overwrite existing user data.** ## Required Subdirectories by Mode Under `.precomputed/`: | Subdir | Required for | |--------|--------------| | `latents/` | Video-bearing modes only (T2V, I2V, video extend/inpaint/outpaint, V2V/AV2AV IC-LoRA, A2V, V2A). **Not** produced for audio-only modes. | | `conditions/` | Always (text embeddings) | | `audio_latents/` | Any mode with audio: video modes carrying audio, plus all audio-only modes (T2A, audio extend/suffix/inpaint, A2A IC-LoRA) | | `reference_latents/` | V2V IC-LoRA, AV2AV IC-LoRA | | `reference_audio_latents/` | A2A IC-LoRA, AV2AV IC-LoRA | | `video_masks/` | Video inpainting | | `audio_masks/` | Audio inpainting | **Audio-only modes** (T2A, audio extend/suffix, audio inpainting, A2A IC-LoRA) produce `audio_latents/` + `conditions/` (plus `audio_masks/` or `reference_audio_latents/` as applicable) and **no `latents/`**. Do not flag a missing `latents/` as incomplete for these modes. ## Workflow — Full Preprocess ### Step 1 — Verify existing `.precomputed/` (if present) If `//dataset/.precomputed/` already exists: 1. List subdirectories present. Confirm all required for the chosen mode are present. 2. Load one sample per modality and check tensor shapes: ```bash uv run python -c "import torch; t = torch.load(''); print(t.shape if hasattr(t, 'shape') else {k: v.shape for k, v in t.items()})" ``` 3. Compare shapes against the target resolution from the plan. **On any mismatch or missing subdirectory: STOP. Do not run `process_dataset.py`.** Ask the user via `AskUserQuestion`: - Reuse the existing data at its current resolution (update plan + config accordingly). - Re-preprocess to a new directory (`//dataset/.precomputed-v2/` etc.) — preserves the existing data. - Abort. **Never pass `--overwrite` without explicit user approval** for this exact action. ### Step 2 — Invoke `process_dataset.py` ```bash cd packages/ltx-trainer uv run python scripts/process_dataset.py \ "//dataset/dataset.json" \ --resolution-buckets "xx" \ --model-path "" \ --text-encoder-path "" \ --output-dir "//dataset/.precomputed" \ --load-text-encoder-in-8bit # on 32GB tier (low-VRAM config), per t2v_lora_low_vram.yaml ``` Add as needed: - `--skip-audio` — if mode doesn't use audio (T2V video-only variants). - `--audio-durations ""` — for T2A from a captions-only file. - `--lora-trigger ""` — for style/concept LoRAs. - `--reference-downscale-factor ` — for IC-LoRA modes if downscaled references are desired. - `--video-column`, `--caption-column` — only if the metadata file uses non-standard column names. **Do not pass `--overwrite`** unless re-preprocessing was explicitly approved in Step 1. ### Step 3 — Audit output After completion, verify: ```bash ls "//dataset/.precomputed/" # Expected subdirs per the mode table above. # Count files in each: for d in latents conditions audio_latents reference_latents video_masks audio_masks; do if [ -d "//dataset/.precomputed/$d" ]; then echo "$d: $(ls "//dataset/.precomputed/$d" | wc -l)" fi done ``` Counts in each required subdir should equal the dataset sample count. **Reconcile counts — do this for every run, any dataset size.** Compare the `latents/` (and `audio_latents/`) count against the caption/sample count. If fewer latents were produced, `process_dataset.py` **silently skipped** clips — most commonly because they were **shorter than the target frame bucket** (it logs each skip). When counts don't match: 1. Identify which clips were dropped (grep the preprocess log for skip/"fewer frames" lines, or diff the produced `.pt` stems against the metadata). 2. **Surface it to the user** with the count and the specific clips — never silently proceed on a shrunk dataset. 3. Offer options: re-preprocess at a **smaller frame bucket** the clips support, add a **second (shorter) bucket** to keep the short clips (multi-bucket requires `batch_size: 1`), or accept the loss. Let the user decide. **Audio gate (hard stop for audio runs).** For any run with an audio modality (joint audio+video, A2V, V2A, T2A, audio-only modes), verify `audio_latents/` is **present and non-empty** with one `.pt` per sample. `process_dataset.py` **swallows audio-decode errors and continues** — it logs "0 videos with audio" and produces empty `audio_latents/` rather than failing. If an audio run produced no audio latents, **stop** — do not proceed to training (it would silently train audio-free). The usual cause is a broken audio decode path (e.g. missing/incompatible `torchcodec`); confirm `uv run python -c "import torchaudio; torchaudio.load('')"` works (see `references/troubleshooting.md`), fix it, then re-preprocess with `--overwrite`. ## Workflow — One-Sample (Phase 6) Same as full preprocess, but operate on a single-sample metadata file: 1. Pick the first sample from `//dataset/dataset.json` and write a one-sample metadata file **inside the dataset dir** — e.g. `//dataset/_one_sample.json` — copying the entry **verbatim, keeping its relative `media_path`**. `process_dataset.py` resolves media paths relative to the metadata file's own directory, so the one-sample file must sit beside the real media (i.e. in `dataset/`, the same dir as `dataset.json`). **Do not** place it in `overfit/` and **do not** rewrite the path to an absolute one — an absolute path produces mirrored nested output dirs (`.precomputed/latents/absolute/path/.../x.pt`) instead of a clean `latents/x.pt`. 2. Run `process_dataset.py` on that file with `--output-dir "//overfit/.precomputed"` (output still goes to `overfit/`, only the metadata lives in `dataset/`). 3. Use the **same `--resolution-buckets`** as the planned full run. Critical: a small-shape sanity check is misleading because resolution is the dominant memory factor. 4. Clean up the temporary `dataset/_one_sample.json` afterward (it's scratch; don't leave it in the dataset dir). ## Decode-and-Verify (optional debug aid) If the user reports validation samples look wrong or training diverges, decode one preprocessed sample back to media: `decode_latents.py` takes the **latents directory** and an **output directory** as positional arguments (it decodes the whole directory, not a single `.pt` file). Add `--with-audio` and `--audio-latents-dir` if the dataset has audio. ```bash cd packages/ltx-trainer uv run python scripts/decode_latents.py \ "//dataset/.precomputed/latents" \ "//dataset/.precomputed/decoded_check" \ --model-path "" ``` If decoded output is garbled, preprocessing itself is suspect (wrong model, wrong VAE). ## Failure Modes - **"shape mismatch" on resume:** Step 1's verification check. Ask user before any mutation. - **`frames % 8 != 1`** error from process_dataset.py: the requested frame count is invalid; correct in the plan and re-launch. - **VRAM OOM during preprocessing:** add `--load-text-encoder-in-8bit`. If still OOM, reduce `--batch-size`. - **Disk full:** preprocessed latents can be large (especially audio). Surface to user with a `du -sh` summary of `.precomputed/`. ## Do Not - Do not delete or overwrite existing `.precomputed/` data without explicit user approval for that exact action. - Do not preprocess at a smaller resolution to "save time" — the sanity check exists specifically to validate the planned resolution.