PoseRenderer/README.md

# OpenPose Renderer for Maya

Turns an animated Maya skeleton into OpenPose-style stick-figure images (up
to 25 BODY_25 body/foot + 21+21 hand + 70 face = 137 keypoints), for use
with ControlNet and similar tools. Built entirely on Maya's Python API 2.0.
Targets Maya 2022-2024 (Maya 2022 = Python 3.7, 2023 = Python 3.9, 2024 =
Python 3.10; PySide2 -- PySide6 also detected automatically if present).

Body output supports two OpenPose skeleton formats, selectable per
character (see *Body model* below): **BODY_25** (25 points, includes
MidHip + feet) and **COCO** (18 points, no MidHip/feet, Neck connects
directly to each hip -- OpenPose's original/legacy body model). Both use
the exact same joint mapping; switching formats does not require remapping.

## Install

**Option A -- Maya module (recommended):** copy (or symlink) this whole
`PoseRenderer` folder somewhere on disk, then add its *parent* directory to
`MAYA_MODULE_PATH` (e.g. in `Maya.env`):
```
MAYA_MODULE_PATH = C:\path\to\parent\folder
```
`PoseRenderer.mod` (at the repo root) handles the rest: it registers the
module as `PoseRenderer`, adds `plug-ins/` to `MAYA_PLUG_IN_PATH` and
`python/` to `PYTHONPATH` automatically. In Maya: **Windows >
Settings/Preferences > Plug-in Manager**, find `openposeRenderer.py`, and
check *Loaded* (and *Auto load* if desired) -- it's now loadable by its
short name (`cmds.loadPlugin("openposeRenderer")`) from anywhere.

**Option B -- plug-in path only:** skip the module system and add
`plug-ins/` directly to `MAYA_PLUG_IN_PATH` instead:
```
MAYA_PLUG_IN_PATH = C:\path\to\PoseRenderer\plug-ins
```
The plugin adds its own `python/` folder to `sys.path` automatically at
load time either way (via `MFnPlugin.loadPath()`), so `PYTHONPATH` never
needs setting by hand for the plugin itself to work -- Option A's
`PYTHONPATH` entry only matters if you want `python/openpose_renderer` (or
the bundled `debugpy`) importable outside of loading the plugin, e.g. from
a shelf script.

## Usage

### 1. Map a character

- Run `cmds.openposeCreateCharacter(name="myCharacter")` (or use the
  mapping editor's *Create Character* button) to create one
  `openposeCharacter` node per person you want to output. Scenes with
  multiple `openposeCharacter` nodes render as multi-person OpenPose output.
- Open the mapping editor: `cmds.openposeShowMappingEditor()`.
- Select the character node in the dropdown, select a joint in the
  viewport, click *Set from Selected* next to the OpenPose_full slot it
  corresponds to. Repeat for as many of the 137 slots as your rig has
  joints for -- unmapped slots simply render with zero confidence (no
  point/limb drawn), matching how OpenPose itself represents undetected
  keypoints. Face and hand slots are optional; body-only rigs work fine.
- Mappings are plain Maya connections (`joint.message ->
  openposeCharacter.targetJoint[i]`), so they save with the scene and
  survive joint renames/reparenting.

### 2. Viewport preview

With an `openposeCharacter` node in the scene, a colored OpenPose-style
skeleton overlay is drawn live over the mapped joints in every viewport
(toggle per-character via the node's `enableDisplay` attribute; tune with
`jointRadiusScale` / `limbThicknessScale`). This is for QA of the mapping,
independent of rendering.

### Body model (BODY_25 / COCO)

Each `openposeCharacter` node has a `bodyModel` enum attribute (`BODY_25`,
the default, or `COCO`), settable in the mapping editor's *Body Model*
dropdown or directly via `cmds.setAttr("myCharacter.bodyModel", 1)` (0 =
BODY_25, 1 = COCO). It controls both the live viewport overlay and the
rendered output for that character:

- **BODY_25**: all 25 body/foot points -- MidHip, hips/knees/ankles, eyes,
  ears, and the 6 foot points (heels + big/small toes).
- **COCO**: the original 18-point OpenPose body model -- no MidHip, no
  feet, and the neck connects directly to each hip instead of through a
  pelvis point. Slots you mapped for MidHip/feet are simply not drawn in
  this mode; nothing needs to be remapped to switch.

Hands and face are unaffected by this setting either way.

### Default eyes/ears

Most rigs (including HumanIK) have no `REye`/`LEye`/`REar`/`LEar` joints --
only a Head/Nose. By default (`useDefaultEyesEars`, on unless turned off),
whenever any of those 4 slots is unmapped but `Body_Nose` *is* mapped, it
gets a synthetic head-relative position instead of being invisible --
anchored at and rotating with the joint mapped to `Body_Nose` itself (sized
by `faceScale`), so they turn with the head's actual rotation rather than
billboarding toward the camera. An explicit joint mapped to any of the 4
slots always overrides this fallback. The offsets themselves are a
reasonable approximation of human proportions, not from a published
source.

Note also: BODY_25/COCO's `RShoulder->REar` / `LShoulder->LEar` limb lines
(present in OpenPose's own spec -- a detection-robustness feature of the
original network, not a real anatomical link) are intentionally **not**
drawn here; ears connect only via eye->ear, staying leaf points. This is a
deliberate deviation from upstream, by request -- see the note in
`constants.py`.

Rendered from the
same mapped HumanIK-named test rig (`test/openpose_humanik_test.ma`):

| BODY_25 | COCO |
| --- | --- |
| ![BODY_25 output](test/openpose_humanik_test_preview.png) | ![COCO output](test/openpose_humanik_test_preview_coco.png) |

### Face source: facial joints, or ARKit 52 blendshapes

Most facial rigs have no joints at all -- they're driven entirely by
blendShape targets, often named per Apple's ARKit convention (from an
iPhone/TrueDepth mocap pipeline, Live Link Face, or a hand-keyed rig built
to match it). Each `openposeCharacter` has a `faceSource` enum (mapping
editor's *Face Source* dropdown, or `cmds.setAttr("myCharacter.faceSource",
1)`; 0 = `Joints`, 1 = `ARKitBlendshapes`) controlling where its 70 face
keypoints come from:

- **Joints** (default): unchanged from above -- map `Face_*` slots to
  facial joints if your rig has them.
- **ARKitBlendshapes**: the 70 face keypoints are instead computed
  procedurally from 52 connectable `blendshapeWeight[0..51]` float slots
  (in Apple's canonical `ARFaceAnchor.BlendShapeLocation` order/naming --
  `eyeBlinkLeft`, `jawOpen`, `mouthSmileLeft`, etc.). Select your
  blendShape node (or an ARKit-mocap-driven control) and click *Auto-Connect
  ARKit Blendshapes from Selected* in the mapping editor -- it connects
  every one of the 52 canonical names that exists as an attribute on the
  selected node. The resulting face is anchored at the character's mapped
  `Body_Nose` joint, sized by the `faceScale` attribute (world units;
  default 15 -- tune to your scene's unit scale).

  A `faceFollow` enum (mapping editor's *Face Follow* dropdown, or
  `cmds.setAttr("myCharacter.faceFollow", 1)`; 0 = `Camera`, 1 = `Neck`)
  controls the face plane's orientation:
  - **Camera** (default): always faces whichever camera is drawing
    (viewport or render camera) -- works regardless of rig conventions,
    since it never has to guess which way a joint's local axes point.
  - **Neck**: rotates *with* the character's mapped `Body_Neck` joint
    instead, using that joint's own local X/Y axes as the face plane's
    right/up -- so the face turns with the head. This is rig-dependent (it
    assumes local X = right, local Y = up, which is common but not
    universal); if the face looks rotated relative to the head, adjust the
    neck joint's rotate axis, or use `Camera` instead. Falls back to
    `Camera` if no joint is mapped to `Body_Neck`.

**Accuracy caveat:** there is no published, authoritative "which ARKit
weight moves which OpenPose landmark, by how much" table anywhere (unlike
BODY_25/COCO, which OpenPose's own source defines exactly). The neutral
face template and per-blendshape displacement rules in
`face_blendshapes.py` are a hand-authored, geometrically-reasoned
approximation -- each shape nudges only the anatomically obvious nearby
landmarks in an intuitive direction. It produces a recognizable, responsive
OpenPose-style face suitable for ControlNet conditioning, but is not a
biomechanically precise simulation. `jawForward` and `tongueOut` have no
representable effect on a frontal 2D landmark set and are no-ops.

Rendered from `test/openpose_arkit_face_test.ma` (a control node with all 52
ARKit-named attributes, auto-connected -- see that scene for the pattern):

| Neutral | `jawOpen` + smile + raised brows | `faceFollow=Neck`, neck tilted 25° |
| --- | --- | --- |
| ![Neutral face](test/openpose_arkit_face_test_neutral.png) | ![Expression face](test/openpose_arkit_face_test_expression.png) | ![Neck-follow face](test/openpose_arkit_face_test_neck_follow.png) |

### 3. Render OpenPose images

- Switch a viewport panel's **Renderer** menu to **OpenPose** to make it
  the active/selectable renderer for that panel (this is the Viewport 2.0
  render override integration point).
- To actually generate the OpenPose PNG sequence, run:
  ```python
  cmds.openposeRenderSequence(
      startFrame=1, endFrame=48,
      camera="renderCam",              # optional; defaults to the active view's camera
      outputDirectory="C:/out/openpose",  # optional; defaults to <workspace>/images/openpose
      showInRenderView=True,
  )
  ```
  Each frame is projected through the given camera at the scene's render
  resolution (`defaultResolution` node), rasterized as a black-background,
  OpenPose-colored image (limbs under joints, canonical BODY_25 palette,
  per-finger rainbow hand colors, white face contour lines), written to
  disk, and pushed into Maya's Render View so you can watch it render like
  any other renderer.

## Debugging (VS Code)

`.vscode/launch.json` provides:

- **`Maya: Attach (debugpy)`** -- attaches to a running, interactive Maya.
  `debugpy` is bundled in `python/` (no separate pip install needed --
  verified importable under all three target Maya Python versions,
  3.7/3.9/3.10). Each session, run `scripts/start_debug_server.py` inside
  Maya's Script Editor first (see that file's docstring), then launch this
  config from VS Code to hit breakpoints anywhere in `plug-ins/` or
  `python/openpose_renderer/` -- including the draw override and render
  override callbacks, which only run inside a real Maya viewport.
- **`mayapy 2022/2023/2024: Run current file`** -- runs the file you have
  open through that Maya version's own `mayapy.exe` interpreter (with
  `python/` on `PYTHONPATH`), for headless scripts that call
  `maya.standalone.initialize()` themselves. Useful for quick iteration on
  `constants.py`/`projector.py`/`rasterizer.py` without opening Maya's UI.

## Notes / known limitations

- Colors/connectivity: BODY_25 and COCO keypoint layout and limb
  connectivity are taken from OpenPose's own
  `POSE_BODY_25_BODY_PART_PAIRS` / COCO section of `POSE_BODY_PART_PAIRS`
  (`src/openpose/pose/poseParameters.cpp`), with BODY_25's colors likewise
  verbatim from `POSE_BODY_25_COLORS_RENDER_GPU` -- with one deliberate,
  by-request deviation: the `RShoulder->REar` / `LShoulder->LEar` lines
  upstream includes are dropped, so ears stay leaf points connected only
  via eye->ear (see "Default eyes/ears" above). The face's 63 contour
  lines (jaw/eyebrows/nose/eyes/mouth) and solid-white color are equally
  verbatim, from `FACE_PAIRS_RENDER_GPU` / `FACE_COLORS_RENDER_GPU`
  (`include/openpose/face/faceParameters.hpp`) -- independently confirmed
  against OpenPose's own published keypoint diagrams. COCO limb colors and
  all hand colors are generated from an HSV hue wheel (rainbow per limb or
  finger) reproducing OpenPose's own visual scheme rather than a hardcoded
  transcription of its longer, generated gradient tables.
- Projection does not attempt to replicate Maya's film-gate/overscan/fit
  reconciliation pixel-for-pixel -- for correct results, set the render
  camera's film aspect ratio to match the scene's render resolution, as you
  would for any Maya render.
- The "OpenPose" Renderer-menu entry uses a standard scene/HUD/present
  Viewport 2.0 operation chain (so switching to it always leaves the panel
  usable); the actual OpenPose image generation happens via
  `openposeRenderSequence`, not via GPU render-target pixel substitution
  inside the viewport override.