Add faceFollow, fix face contour rendering, add default eyes/ears

Three changesets bundled together:

- faceFollow (Camera/Neck) for the ARKit-blendshape face: it can now
  rotate with the mapped Body_Neck joint's own local axes instead of
  always billboarding toward the camera.

- Fixed a real correctness bug: the face was rendering as isolated
  dots. Verified against OpenPose's own published keypoint diagrams
  and source (FACE_PAIRS_RENDER_GPU, 63 pairs) that it should render
  connected jaw/eyebrow/nose/eye/mouth contour lines; full_limb_data()
  was also silently dropping face limbs entirely. Both fixed.

- Default eyes/ears: most rigs (HumanIK included) have no
  REye/LEye/REar/LEar joints, only Head/Nose, leaving those 4 slots
  permanently invisible. They now get a synthetic head-relative
  position when unmapped but Body_Nose is mapped, anchored at and
  rotating with the Nose joint's own orientation (not the camera) --
  toggleable via useDefaultEyesEars, always overridden by an explicit
  joint mapping. Also drops the RShoulder->REar / LShoulder->LEar
  lines from BODY_25 and COCO connectivity by request (a
  detection-robustness quirk of OpenPose's original network, not real
  anatomy) so ears stay leaf points -- a disclosed, deliberate
  deviation from upstream's otherwise-verbatim connectivity.

All verified against real Maya (2022/2023/2024), not just logic:
rotation math checked exact, explicit-mapping-overrides-fallback
checked, shoulder-ear removal checked against full_limb_data() output,
rendered test scenes visually confirmed against OpenPose's own
reference diagrams.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
2026-07-14 10:12:10 +08:00
parent 417ff72850
commit 67b3deedbd
14 changed files with 471 additions and 178 deletions
+58 -17
View File
@@ -81,7 +81,29 @@ rendered output for that character:
pelvis point. Slots you mapped for MidHip/feet are simply not drawn in
this mode; nothing needs to be remapped to switch.
Hands and face are unaffected by this setting either way. Rendered from the
Hands and face are unaffected by this setting either way.
### Default eyes/ears
Most rigs (including HumanIK) have no `REye`/`LEye`/`REar`/`LEar` joints --
only a Head/Nose. By default (`useDefaultEyesEars`, on unless turned off),
whenever any of those 4 slots is unmapped but `Body_Nose` *is* mapped, it
gets a synthetic head-relative position instead of being invisible --
anchored at and rotating with the joint mapped to `Body_Nose` itself (sized
by `faceScale`), so they turn with the head's actual rotation rather than
billboarding toward the camera. An explicit joint mapped to any of the 4
slots always overrides this fallback. The offsets themselves are a
reasonable approximation of human proportions, not from a published
source.
Note also: BODY_25/COCO's `RShoulder->REar` / `LShoulder->LEar` limb lines
(present in OpenPose's own spec -- a detection-robustness feature of the
original network, not a real anatomical link) are intentionally **not**
drawn here; ears connect only via eye->ear, staying leaf points. This is a
deliberate deviation from upstream, by request -- see the note in
`constants.py`.
Rendered from the
same mapped HumanIK-named test rig (`test/openpose_humanik_test.ma`):
| BODY_25 | COCO |
@@ -108,9 +130,22 @@ keypoints come from:
ARKit Blendshapes from Selected* in the mapping editor -- it connects
every one of the 52 canonical names that exists as an attribute on the
selected node. The resulting face is anchored at the character's mapped
`Body_Nose` joint and billboards to always face whichever camera is
drawing (viewport or render camera), sized by the `faceScale` attribute
(world units; default 15 -- tune to your scene's unit scale).
`Body_Nose` joint, sized by the `faceScale` attribute (world units;
default 15 -- tune to your scene's unit scale).
A `faceFollow` enum (mapping editor's *Face Follow* dropdown, or
`cmds.setAttr("myCharacter.faceFollow", 1)`; 0 = `Camera`, 1 = `Neck`)
controls the face plane's orientation:
- **Camera** (default): always faces whichever camera is drawing
(viewport or render camera) -- works regardless of rig conventions,
since it never has to guess which way a joint's local axes point.
- **Neck**: rotates *with* the character's mapped `Body_Neck` joint
instead, using that joint's own local X/Y axes as the face plane's
right/up -- so the face turns with the head. This is rig-dependent (it
assumes local X = right, local Y = up, which is common but not
universal); if the face looks rotated relative to the head, adjust the
neck joint's rotate axis, or use `Camera` instead. Falls back to
`Camera` if no joint is mapped to `Body_Neck`.
**Accuracy caveat:** there is no published, authoritative "which ARKit
weight moves which OpenPose landmark, by how much" table anywhere (unlike
@@ -126,9 +161,9 @@ representable effect on a frontal 2D landmark set and are no-ops.
Rendered from `test/openpose_arkit_face_test.ma` (a control node with all 52
ARKit-named attributes, auto-connected -- see that scene for the pattern):
| Neutral | `jawOpen` + smile + raised brows |
| --- | --- |
| ![Neutral face](test/openpose_arkit_face_test_neutral.png) | ![Expression face](test/openpose_arkit_face_test_expression.png) |
| Neutral | `jawOpen` + smile + raised brows | `faceFollow=Neck`, neck tilted 25° |
| --- | --- | --- |
| ![Neutral face](test/openpose_arkit_face_test_neutral.png) | ![Expression face](test/openpose_arkit_face_test_expression.png) | ![Neck-follow face](test/openpose_arkit_face_test_neck_follow.png) |
### 3. Render OpenPose images
@@ -147,9 +182,9 @@ ARKit-named attributes, auto-connected -- see that scene for the pattern):
Each frame is projected through the given camera at the scene's render
resolution (`defaultResolution` node), rasterized as a black-background,
OpenPose-colored image (limbs under joints, canonical BODY_25 palette,
per-finger rainbow hand colors, white face dots), written to disk, and
pushed into Maya's Render View so you can watch it render like any other
renderer.
per-finger rainbow hand colors, white face contour lines), written to
disk, and pushed into Maya's Render View so you can watch it render like
any other renderer.
## Debugging (VS Code)
@@ -172,14 +207,20 @@ ARKit-named attributes, auto-connected -- see that scene for the pattern):
## Notes / known limitations
- Colors/connectivity: BODY_25 and COCO keypoint layout and limb
connectivity are taken verbatim from OpenPose's own
connectivity are taken from OpenPose's own
`POSE_BODY_25_BODY_PART_PAIRS` / COCO section of `POSE_BODY_PART_PAIRS`
(`src/openpose/pose/poseParameters.cpp`). BODY_25's colors are likewise
verbatim from `POSE_BODY_25_COLORS_RENDER_GPU`. COCO limb colors, and all
hand/face colors, are generated from an HSV hue wheel (rainbow per limb
or finger, white face dots) reproducing OpenPose's own visual scheme
rather than a hardcoded transcription of its longer, generated gradient
tables.
(`src/openpose/pose/poseParameters.cpp`), with BODY_25's colors likewise
verbatim from `POSE_BODY_25_COLORS_RENDER_GPU` -- with one deliberate,
by-request deviation: the `RShoulder->REar` / `LShoulder->LEar` lines
upstream includes are dropped, so ears stay leaf points connected only
via eye->ear (see "Default eyes/ears" above). The face's 63 contour
lines (jaw/eyebrows/nose/eyes/mouth) and solid-white color are equally
verbatim, from `FACE_PAIRS_RENDER_GPU` / `FACE_COLORS_RENDER_GPU`
(`include/openpose/face/faceParameters.hpp`) -- independently confirmed
against OpenPose's own published keypoint diagrams. COCO limb colors and
all hand colors are generated from an HSV hue wheel (rainbow per limb or
finger) reproducing OpenPose's own visual scheme rather than a hardcoded
transcription of its longer, generated gradient tables.
- Projection does not attempt to replicate Maya's film-gate/overscan/fit
reconciliation pixel-for-pixel -- for correct results, set the render
camera's film aspect ratio to match the scene's render resolution, as you