Add faceFollow, fix face contour rendering, add default eyes/ears

Three changesets bundled together:

- faceFollow (Camera/Neck) for the ARKit-blendshape face: it can now
  rotate with the mapped Body_Neck joint's own local axes instead of
  always billboarding toward the camera.

- Fixed a real correctness bug: the face was rendering as isolated
  dots. Verified against OpenPose's own published keypoint diagrams
  and source (FACE_PAIRS_RENDER_GPU, 63 pairs) that it should render
  connected jaw/eyebrow/nose/eye/mouth contour lines; full_limb_data()
  was also silently dropping face limbs entirely. Both fixed.

- Default eyes/ears: most rigs (HumanIK included) have no
  REye/LEye/REar/LEar joints, only Head/Nose, leaving those 4 slots
  permanently invisible. They now get a synthetic head-relative
  position when unmapped but Body_Nose is mapped, anchored at and
  rotating with the Nose joint's own orientation (not the camera) --
  toggleable via useDefaultEyesEars, always overridden by an explicit
  joint mapping. Also drops the RShoulder->REar / LShoulder->LEar
  lines from BODY_25 and COCO connectivity by request (a
  detection-robustness quirk of OpenPose's original network, not real
  anatomy) so ears stay leaf points -- a disclosed, deliberate
  deviation from upstream's otherwise-verbatim connectivity.

All verified against real Maya (2022/2023/2024), not just logic:
rotation math checked exact, explicit-mapping-overrides-fallback
checked, shoulder-ear removal checked against full_limb_data() output,
rendered test scenes visually confirmed against OpenPose's own
reference diagrams.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
2026-07-14 10:12:10 +08:00
parent 417ff72850
commit 67b3deedbd
14 changed files with 471 additions and 178 deletions
+55 -12
View File
@@ -55,20 +55,18 @@ def project_point(world_point, camera_path, width, height):
return pixel_x, pixel_y, in_front
def billboard_world_points(anchor_world_point, camera_path, local_points_xy, scale):
"""Place 2D local (x, y) points on a plane centered at ``anchor_world_point``
that always faces ``camera_path`` (camera-space X/Y used as the plane's
right/up axes), scaled by ``scale``.
Used to draw/render the procedural ARKit-blendshape face without needing
any facial joint orientation -- it billboards toward whichever camera is
currently drawing (the active viewport camera for the live overlay, the
render camera for output), sidestepping the rig-specific, unanswerable
question of "which way does this head joint's local frame face".
"""
world_matrix = camera_path.inclusiveMatrix()
def local_axes_world(dag_path):
"""World-space (right, up) unit vectors from a DAG node's local X/Y axes."""
world_matrix = dag_path.inclusiveMatrix()
right = (om.MVector(1.0, 0.0, 0.0) * world_matrix).normal()
up = (om.MVector(0.0, 1.0, 0.0) * world_matrix).normal()
return right, up
def plane_world_points(anchor_world_point, right, up, local_points_xy, scale):
"""Place 2D local (x, y) points on a plane centered at ``anchor_world_point``
using the given world-space ``right``/``up`` axes, scaled by ``scale``.
"""
anchor = om.MPoint(anchor_world_point)
return [
om.MPoint(anchor + right * (x * scale) + up * (y * scale))
@@ -76,6 +74,51 @@ def billboard_world_points(anchor_world_point, camera_path, local_points_xy, sca
]
def billboard_world_points(anchor_world_point, camera_path, local_points_xy, scale):
"""``plane_world_points`` using ``camera_path``'s own local X/Y as
right/up, so the plane always faces whichever camera is drawing.
Used to draw/render the procedural ARKit-blendshape face without needing
any facial joint orientation -- it billboards toward whichever camera is
currently drawing (the active viewport camera for the live overlay, the
render camera for output), sidestepping the rig-specific, unanswerable
question of "which way does this head joint's local frame face". This is
the "Camera" faceFollow mode.
"""
right, up = local_axes_world(camera_path)
return plane_world_points(anchor_world_point, right, up, local_points_xy, scale)
def oriented_world_points(anchor_world_point, orientation_dag_path, local_points_xy, scale):
"""``plane_world_points`` using ``orientation_dag_path``'s own local X/Y
as right/up, so the plane rotates *with* that joint (e.g. the mapped
Neck) instead of always facing the camera. This is the "Neck" faceFollow
mode -- unlike "Camera", it is rig-dependent: it assumes the joint's
local X is the character's right and local Y is up, which matches
common convention but is not guaranteed for every rig. If the face
appears rotated relative to the head, adjust the neck joint's rotate
axis/orient, or use "Camera" mode instead.
"""
right, up = local_axes_world(orientation_dag_path)
return plane_world_points(anchor_world_point, right, up, local_points_xy, scale)
def default_head_relative_points(anchor_world_point, orientation_dag_path, scale, local_offsets):
"""Compute head-relative fallback world points (e.g. the default
REye/LEye/REar/LEar positions in constants.DEFAULT_EYE_EAR_LOCAL_OFFSETS)
anchored at ``anchor_world_point`` (the mapped Nose), oriented by
``orientation_dag_path``'s own local X/Y axes -- always the joint
mapped to Nose itself (the head), so these points turn with the head's
actual rotation rather than billboarding toward the camera.
``local_offsets``: {index: (x, y)}. Returns {index: MPoint}.
"""
indices = list(local_offsets.keys())
offsets = [local_offsets[i] for i in indices]
world_points = oriented_world_points(anchor_world_point, orientation_dag_path, offsets, scale)
return dict(zip(indices, world_points))
def project_character(mapped_joint_paths, camera_path, width, height):
"""Project one character's mapping to OpenPose_full pixel keypoints.