MediaPlane/docs/Viewport_2_0_Integration_Spec.md

Viewport 2.0 Integration Specification

Overview

This document specifies the implementation details for integrating video frame display into Maya's Viewport 2.0 using MHWRender::MPxDrawOverride.

Requirements

• Display video frames in Viewport 2.0 as an image plane
• Support for RGB and RGBA frame formats
• Proper handling of frame rate and playback speed
• Integration with Maya's viewport rendering pipeline
• Support for post-effects (crop, resize, flip) applied to displayed frames
• Efficient texture updates to minimize performance impact
• Compatibility with Maya 2023's Viewport 2.0 API

Implementation Details

Core Components

VideoFrameDrawOverride Class

Derived from MHWRender::MPxDrawOverride, responsible for rendering video frames in the viewport.

Key methods:

• bool isBounded(const MDagPath& objPath) const - Returns whether the object has a bounding box
• MBoundingBox boundingBox(const MDagPath& objPath) const - Returns bounding box of the object
• uint32_t drawFlags() const - Returns draw flags for the override
• void prepareForDraw(...) - Prepares resources needed for drawing
• void addUIDrawables(...) - Adds UI drawables (if needed)
• void hasUIDrawables() const - Returns whether UI drawables are present
• void draw(...) - Main drawing method where video frame is rendered

Texture Management

• OpenGL texture object to hold current video frame
• Texture updates when new frame is available
• Proper texture format matching (GL_RGB, GL_RGBA, etc.)
• Efficient texture sub-data updates when possible
• Texture cleanup when node is destroyed

Frame Data Handling

• Receive frame data from Maya node via output attributes
• Convert frame data to appropriate OpenGL format if needed
• Update texture with new frame data
• Maintain frame timestamp for synchronization

Integration Points

• Maya Node Integration: Connect to MPxNode's output attributes to receive frame data
• Frame Rate Synchronization: Use frame timestamp to synchronize with Maya's timeline
• Post-effects: Apply transformations to frame data before texture upload
• Caching: Interface with frame caching mechanism to get frames efficiently

Rendering Approach

1. Receive new frame data from MPxNode (via attribute change notification)
2. Convert frame data to OpenGL-compatible format if necessary
3. Update OpenGL texture with new frame data
4. In viewport draw call:
   • Set up appropriate shader program (simple texture shader)
   • Bind texture
   • Draw quad covering the image plane area
   • Apply any viewport-specific transformations

Shader Program

• Simple vertex and fragment shaders for texture rendering
• Vertex shader: Pass through texture coordinates
• Fragment shader: Sample texture and output color
• Optional: Support for color correction or other viewport-specific effects

Texture Updates

• Use GL_PIXEL_UNPACK_BUFFER for efficient updates if supported
• Fallback to glTexSubImage2D for broader compatibility
• Handle different pixel formats (RGB, RGBA, BGRA, etc.)
• Flip Y-axis if necessary (OpenGL vs image coordinate systems)

Bounding Box and Selection

• Return appropriate bounding box based on frame dimensions and aspect ratio
• Support for viewport selection of the image plane
• Handle transform nodes (position, rotation, scale) if applicable

Threading Considerations

• Viewport drawing occurs on main thread
• Texture updates must happen on main thread (OpenGL context)
• Use thread-safe queue to transfer frame data from decoding thread to main thread
• Synchronize access to shared frame data

Error Handling

• Check for OpenGL errors after texture operations
• Handle texture creation failures gracefully
• Provide fallback rendering (e.g., colored quad) when frame data unavailable
• Log errors to Maya's script editor using MGlobal::displayError

Performance Considerations

• Minimize texture format conversions
• Update only changed portions of texture when possible
• Use appropriate texture filtering (GL_LINEAR for smooth playback)
• Avoid expensive operations during viewport drawing
• Consider using persistent mapped buffers for frequent updates

Maya API Specifics

• Use MHWRender::MPxDrawOverride as base class
• Register draw override with MHWRender::MRenderer
• Use MImage class for image format conversions if needed
• Leverage MHWRender::MShaderManager for shader programs
• Follow Viewport 2.0 API guidelines for draw overrides

Dependencies

• Maya Node implementation (for frame data)
• Frame caching mechanism
• OpenGL 3.2+ (required for Viewport 2.0)
• Maya API 2023 (MHWRender module)

Overview

This document specifies the implementation details for integrating video frame display into Maya's Viewport 2.0 using MHWRender::MPxDrawOverride.

Requirements

• Display video frames in Viewport 2.0 as an image plane
• Support for RGB and RGBA frame formats
• Proper handling of frame rate and playback speed
• Integration with Maya's viewport rendering pipeline
• Support for post-effects (crop, resize, flip) applied to displayed frames
• Efficient texture updates to minimize performance impact
• Compatibility with Maya 2023's Viewport 2.0 API

Implementation Details

Core Components

VideoFrameDrawOverride Class

Derived from MHWRender::MPxDrawOverride, responsible for rendering video frames in the viewport.

Key methods:

• bool isBounded(const MDagPath& objPath) const - Returns whether the object has a bounding box
• MBoundingBox boundingBox(const MDagPath& objPath) const - Returns bounding box of the object
• uint32_t drawFlags() const - Returns draw flags for the override
• void prepareForDraw(...) - Prepares resources needed for drawing
• void addUIDrawables(...) - Adds UI drawables (if needed)
• void hasUIDrawables() const - Returns whether UI drawables are present
• void draw(...) - Main drawing method where video frame is rendered

Texture Management

• OpenGL texture object to hold current video frame
• Texture updates when new frame is available
• Proper texture format matching (GL_RGB, GL_RGBA, etc.)
• Efficient texture sub-data updates when possible
• Texture cleanup when node is destroyed

Frame Data Handling

• Receive frame data from Maya node via output attributes
• Convert frame data to appropriate OpenGL format if needed
• Update texture with new frame data
• Maintain frame timestamp for synchronization

Integration Points

• Maya Node Integration: Connect to MPxNode's output attributes to receive frame data
• Frame Rate Synchronization: Use frame timestamp to synchronize with Maya's timeline
• Post-effects: Apply transformations to frame data before texture upload
• Caching: Interface with frame caching mechanism to get frames efficiently

Rendering Approach

1. Receive new frame data from MPxNode (via attribute change notification)
2. Convert frame data to OpenGL-compatible format if necessary
3. Update OpenGL texture with new frame data
4. In viewport draw call:
   • Set up appropriate shader program (simple texture shader)
   • Bind texture
   • Draw quad covering the image plane area
   • Apply any viewport-specific transformations

Shader Program

• Simple vertex and fragment shaders for texture rendering
• Vertex shader: Pass through texture coordinates
• Fragment shader: Sample texture and output color
• Optional: Support for color correction or other viewport-specific effects

Texture Updates

• Use GL_PIXEL_UNPACK_BUFFER for efficient updates if supported
• Fallback to glTexSubImage2D for broader compatibility
• Handle different pixel formats (RGB, RGBA, BGRA, etc.)
• Flip Y-axis if necessary (OpenGL vs image coordinate systems)

Bounding Box and Selection

• Return appropriate bounding box based on frame dimensions and aspect ratio
• Support for viewport selection of the image plane
• Handle transform nodes (position, rotation, scale) if applicable

Threading Considerations

• Viewport drawing occurs on main thread
• Texture updates must happen on main thread (OpenGL context)
• Use thread-safe queue to transfer frame data from decoding thread to main thread
• Synchronize access to shared frame data

Error Handling

• Check for OpenGL errors after texture operations
• Handle texture creation failures gracefully
• Provide fallback rendering (e.g., colored quad) when frame data unavailable
• Log errors to Maya's script editor using MGlobal::displayError

Performance Considerations

• Minimize texture format conversions
• Update only changed portions of texture when possible
• Use appropriate texture filtering (GL_LINEAR for smooth playback)
• Avoid expensive operations during viewport drawing
• Consider using persistent mapped buffers for frequent updates

Maya API Specifics

• Use MHWRender::MPxDrawOverride as base class
• Register draw override with MHWRender::MRenderer
• Use MImage class for image format conversions if needed
• Leverage MHWRender::MShaderManager for shader programs
• Follow Viewport 2.0 API guidelines for draw overrides

Dependencies

• Maya Node implementation (for frame data)
• Frame caching mechanism
• OpenGL 3.2+ (required for Viewport 2.0)
• Maya API 2023 (MHWRender module)