PoseRenderer/README.md

# OpenPose Renderer for Maya

Turns an animated Maya skeleton into OpenPose-style stick-figure images (up
to 25 BODY_25 body/foot + 21+21 hand + 70 face = 137 keypoints), for use
with ControlNet and similar tools. Built entirely on Maya's Python API 2.0.
Targets Maya 2022-2024 (Maya 2022 = Python 3.7, 2023 = Python 3.9, 2024 =
Python 3.10; PySide2 -- PySide6 also detected automatically if present).

Body output supports two OpenPose skeleton formats, selectable per
character (see *Body model* below): **BODY_25** (25 points, includes
MidHip + feet) and **COCO** (18 points, no MidHip/feet, Neck connects
directly to each hip -- OpenPose's original/legacy body model). Both use
the exact same joint mapping; switching formats does not require remapping.

## Install

**Option A -- Maya module (recommended):** copy (or symlink) this whole
`PoseRenderer` folder somewhere on disk, then add its *parent* directory to
`MAYA_MODULE_PATH` (e.g. in `Maya.env`):
```
MAYA_MODULE_PATH = C:\path\to\parent\folder
```
`PoseRenderer.mod` (at the repo root) handles the rest: it registers the
module as `PoseRenderer`, adds `plug-ins/` to `MAYA_PLUG_IN_PATH` and
`python/` to `PYTHONPATH` automatically. In Maya: **Windows >
Settings/Preferences > Plug-in Manager**, find `openposeRenderer.py`, and
check *Loaded* (and *Auto load* if desired) -- it's now loadable by its
short name (`cmds.loadPlugin("openposeRenderer")`) from anywhere.

**Option B -- plug-in path only:** skip the module system and add
`plug-ins/` directly to `MAYA_PLUG_IN_PATH` instead:
```
MAYA_PLUG_IN_PATH = C:\path\to\PoseRenderer\plug-ins
```
The plugin adds its own `python/` folder to `sys.path` automatically at
load time either way (via `MFnPlugin.loadPath()`), so `PYTHONPATH` never
needs setting by hand for the plugin itself to work -- Option A's
`PYTHONPATH` entry only matters if you want `python/openpose_renderer` (or
the bundled `debugpy`) importable outside of loading the plugin, e.g. from
a shelf script.

## Usage

### 1. Map a character

- Run `cmds.openposeCreateCharacter(name="myCharacter")` (or use the
  mapping editor's *Create Character* button) to create one
  `openposeCharacter` node per person you want to output. Scenes with
  multiple `openposeCharacter` nodes render as multi-person OpenPose output.
- Open the mapping editor: `cmds.openposeShowMappingEditor()`.
- Select the character node in the dropdown, select a joint in the
  viewport, click *Set from Selected* next to the OpenPose_full slot it
  corresponds to. Repeat for as many of the 137 slots as your rig has
  joints for -- unmapped slots simply render with zero confidence (no
  point/limb drawn), matching how OpenPose itself represents undetected
  keypoints. Face and hand slots are optional; body-only rigs work fine.
- Mappings are plain Maya connections (`joint.message ->
  openposeCharacter.targetJoint[i]`), so they save with the scene and
  survive joint renames/reparenting.

### 2. Viewport preview

With an `openposeCharacter` node in the scene, a colored OpenPose-style
skeleton overlay is drawn live over the mapped joints in every viewport
(toggle per-character via the node's `enableDisplay` attribute; tune with
`jointRadiusScale` / `limbThicknessScale`). This is for QA of the mapping,
independent of rendering.

### Body model (BODY_25 / COCO)

Each `openposeCharacter` node has a `bodyModel` enum attribute (`BODY_25`,
the default, or `COCO`), settable in the mapping editor's *Body Model*
dropdown or directly via `cmds.setAttr("myCharacter.bodyModel", 1)` (0 =
BODY_25, 1 = COCO). It controls both the live viewport overlay and the
rendered output for that character:

- **BODY_25**: all 25 body/foot points -- MidHip, hips/knees/ankles, eyes,
  ears, and the 6 foot points (heels + big/small toes).
- **COCO**: the original 18-point OpenPose body model -- no MidHip, no
  feet, and the neck connects directly to each hip instead of through a
  pelvis point. Slots you mapped for MidHip/feet are simply not drawn in
  this mode; nothing needs to be remapped to switch.

Hands and face are unaffected by this setting either way. Rendered from the
same mapped HumanIK-named test rig (`test/openpose_humanik_test.ma`):

| BODY_25 | COCO |
| --- | --- |
| ![BODY_25 output](test/openpose_humanik_test_preview.png) | ![COCO output](test/openpose_humanik_test_preview_coco.png) |

### Face source: facial joints, or ARKit 52 blendshapes

Most facial rigs have no joints at all -- they're driven entirely by
blendShape targets, often named per Apple's ARKit convention (from an
iPhone/TrueDepth mocap pipeline, Live Link Face, or a hand-keyed rig built
to match it). Each `openposeCharacter` has a `faceSource` enum (mapping
editor's *Face Source* dropdown, or `cmds.setAttr("myCharacter.faceSource",
1)`; 0 = `Joints`, 1 = `ARKitBlendshapes`) controlling where its 70 face
keypoints come from:

- **Joints** (default): unchanged from above -- map `Face_*` slots to
  facial joints if your rig has them.
- **ARKitBlendshapes**: the 70 face keypoints are instead computed
  procedurally from 52 connectable `blendshapeWeight[0..51]` float slots
  (in Apple's canonical `ARFaceAnchor.BlendShapeLocation` order/naming --
  `eyeBlinkLeft`, `jawOpen`, `mouthSmileLeft`, etc.). Select your
  blendShape node (or an ARKit-mocap-driven control) and click *Auto-Connect
  ARKit Blendshapes from Selected* in the mapping editor -- it connects
  every one of the 52 canonical names that exists as an attribute on the
  selected node. The resulting face is anchored at the character's mapped
  `Body_Nose` joint and billboards to always face whichever camera is
  drawing (viewport or render camera), sized by the `faceScale` attribute
  (world units; default 15 -- tune to your scene's unit scale).

**Accuracy caveat:** there is no published, authoritative "which ARKit
weight moves which OpenPose landmark, by how much" table anywhere (unlike
BODY_25/COCO, which OpenPose's own source defines exactly). The neutral
face template and per-blendshape displacement rules in
`face_blendshapes.py` are a hand-authored, geometrically-reasoned
approximation -- each shape nudges only the anatomically obvious nearby
landmarks in an intuitive direction. It produces a recognizable, responsive
OpenPose-style face suitable for ControlNet conditioning, but is not a
biomechanically precise simulation. `jawForward` and `tongueOut` have no
representable effect on a frontal 2D landmark set and are no-ops.

Rendered from `test/openpose_arkit_face_test.ma` (a control node with all 52
ARKit-named attributes, auto-connected -- see that scene for the pattern):

| Neutral | `jawOpen` + smile + raised brows |
| --- | --- |
| ![Neutral face](test/openpose_arkit_face_test_neutral.png) | ![Expression face](test/openpose_arkit_face_test_expression.png) |

### 3. Render OpenPose images

- Switch a viewport panel's **Renderer** menu to **OpenPose** to make it
  the active/selectable renderer for that panel (this is the Viewport 2.0
  render override integration point).
- To actually generate the OpenPose PNG sequence, run:
  ```python
  cmds.openposeRenderSequence(
      startFrame=1, endFrame=48,
      camera="renderCam",              # optional; defaults to the active view's camera
      outputDirectory="C:/out/openpose",  # optional; defaults to <workspace>/images/openpose
      showInRenderView=True,
  )
  ```
  Each frame is projected through the given camera at the scene's render
  resolution (`defaultResolution` node), rasterized as a black-background,
  OpenPose-colored image (limbs under joints, canonical BODY_25 palette,
  per-finger rainbow hand colors, white face dots), written to disk, and
  pushed into Maya's Render View so you can watch it render like any other
  renderer.

## Debugging (VS Code)

`.vscode/launch.json` provides:

- **`Maya: Attach (debugpy)`** -- attaches to a running, interactive Maya.
  `debugpy` is bundled in `python/` (no separate pip install needed --
  verified importable under all three target Maya Python versions,
  3.7/3.9/3.10). Each session, run `scripts/start_debug_server.py` inside
  Maya's Script Editor first (see that file's docstring), then launch this
  config from VS Code to hit breakpoints anywhere in `plug-ins/` or
  `python/openpose_renderer/` -- including the draw override and render
  override callbacks, which only run inside a real Maya viewport.
- **`mayapy 2022/2023/2024: Run current file`** -- runs the file you have
  open through that Maya version's own `mayapy.exe` interpreter (with
  `python/` on `PYTHONPATH`), for headless scripts that call
  `maya.standalone.initialize()` themselves. Useful for quick iteration on
  `constants.py`/`projector.py`/`rasterizer.py` without opening Maya's UI.

## Notes / known limitations

- Colors/connectivity: BODY_25 and COCO keypoint layout and limb
  connectivity are taken verbatim from OpenPose's own
  `POSE_BODY_25_BODY_PART_PAIRS` / COCO section of `POSE_BODY_PART_PAIRS`
  (`src/openpose/pose/poseParameters.cpp`). BODY_25's colors are likewise
  verbatim from `POSE_BODY_25_COLORS_RENDER_GPU`. COCO limb colors, and all
  hand/face colors, are generated from an HSV hue wheel (rainbow per limb
  or finger, white face dots) reproducing OpenPose's own visual scheme
  rather than a hardcoded transcription of its longer, generated gradient
  tables.
- Projection does not attempt to replicate Maya's film-gate/overscan/fit
  reconciliation pixel-for-pixel -- for correct results, set the render
  camera's film aspect ratio to match the scene's render resolution, as you
  would for any Maya render.
- The "OpenPose" Renderer-menu entry uses a standard scene/HUD/present
  Viewport 2.0 operation chain (so switching to it always leaves the panel
  usable); the actual OpenPose image generation happens via
  `openposeRenderSequence`, not via GPU render-target pixel substitution
  inside the viewport override.