# Viewport 2.0 Integration Specification ## Overview This document specifies the implementation details for integrating video frame display into Maya's Viewport 2.0 using MHWRender::MPxDrawOverride. ## Requirements - Display video frames in Viewport 2.0 as an image plane - Support for RGB and RGBA frame formats - Proper handling of frame rate and playback speed - Integration with Maya's viewport rendering pipeline - Support for post-effects (crop, resize, flip) applied to displayed frames - Efficient texture updates to minimize performance impact - Compatibility with Maya 2023's Viewport 2.0 API ## Implementation Details ### Core Components #### VideoFrameDrawOverride Class Derived from MHWRender::MPxDrawOverride, responsible for rendering video frames in the viewport. Key methods: - `bool isBounded(const MDagPath& objPath) const` - Returns whether the object has a bounding box - `MBoundingBox boundingBox(const MDagPath& objPath) const` - Returns bounding box of the object - `uint32_t drawFlags() const` - Returns draw flags for the override - `void prepareForDraw(...)` - Prepares resources needed for drawing - `void addUIDrawables(...)` - Adds UI drawables (if needed) - `void hasUIDrawables() const` - Returns whether UI drawables are present - `void draw(...)` - Main drawing method where video frame is rendered #### Texture Management - OpenGL texture object to hold current video frame - Texture updates when new frame is available - Proper texture format matching (GL_RGB, GL_RGBA, etc.) - Efficient texture sub-data updates when possible - Texture cleanup when node is destroyed #### Frame Data Handling - Receive frame data from Maya node via output attributes - Convert frame data to appropriate OpenGL format if needed - Update texture with new frame data - Maintain frame timestamp for synchronization ### Integration Points - Maya Node Integration: Connect to MPxNode's output attributes to receive frame data - Frame Rate Synchronization: Use frame timestamp to synchronize with Maya's timeline - Post-effects: Apply transformations to frame data before texture upload - Caching: Interface with frame caching mechanism to get frames efficiently ### Rendering Approach 1. Receive new frame data from MPxNode (via attribute change notification) 2. Convert frame data to OpenGL-compatible format if necessary 3. Update OpenGL texture with new frame data 4. In viewport draw call: - Set up appropriate shader program (simple texture shader) - Bind texture - Draw quad covering the image plane area - Apply any viewport-specific transformations ### Shader Program - Simple vertex and fragment shaders for texture rendering - Vertex shader: Pass through texture coordinates - Fragment shader: Sample texture and output color - Optional: Support for color correction or other viewport-specific effects ### Texture Updates - Use GL_PIXEL_UNPACK_BUFFER for efficient updates if supported - Fallback to glTexSubImage2D for broader compatibility - Handle different pixel formats (RGB, RGBA, BGRA, etc.) - Flip Y-axis if necessary (OpenGL vs image coordinate systems) ### Bounding Box and Selection - Return appropriate bounding box based on frame dimensions and aspect ratio - Support for viewport selection of the image plane - Handle transform nodes (position, rotation, scale) if applicable ### Threading Considerations - Viewport drawing occurs on main thread - Texture updates must happen on main thread (OpenGL context) - Use thread-safe queue to transfer frame data from decoding thread to main thread - Synchronize access to shared frame data ### Error Handling - Check for OpenGL errors after texture operations - Handle texture creation failures gracefully - Provide fallback rendering (e.g., colored quad) when frame data unavailable - Log errors to Maya's script editor using MGlobal::displayError ### Performance Considerations - Minimize texture format conversions - Update only changed portions of texture when possible - Use appropriate texture filtering (GL_LINEAR for smooth playback) - Avoid expensive operations during viewport drawing - Consider using persistent mapped buffers for frequent updates ### Maya API Specifics - Use MHWRender::MPxDrawOverride as base class - Register draw override with MHWRender::MRenderer - Use MImage class for image format conversions if needed - Leverage MHWRender::MShaderManager for shader programs - Follow Viewport 2.0 API guidelines for draw overrides ## Dependencies - Maya Node implementation (for frame data) - Frame caching mechanism - OpenGL 3.2+ (required for Viewport 2.0) - Maya API 2023 (MHWRender module) ## Overview This document specifies the implementation details for integrating video frame display into Maya's Viewport 2.0 using MHWRender::MPxDrawOverride. ## Requirements - Display video frames in Viewport 2.0 as an image plane - Support for RGB and RGBA frame formats - Proper handling of frame rate and playback speed - Integration with Maya's viewport rendering pipeline - Support for post-effects (crop, resize, flip) applied to displayed frames - Efficient texture updates to minimize performance impact - Compatibility with Maya 2023's Viewport 2.0 API ## Implementation Details ### Core Components #### VideoFrameDrawOverride Class Derived from MHWRender::MPxDrawOverride, responsible for rendering video frames in the viewport. Key methods: - `bool isBounded(const MDagPath& objPath) const` - Returns whether the object has a bounding box - `MBoundingBox boundingBox(const MDagPath& objPath) const` - Returns bounding box of the object - `uint32_t drawFlags() const` - Returns draw flags for the override - `void prepareForDraw(...)` - Prepares resources needed for drawing - `void addUIDrawables(...)` - Adds UI drawables (if needed) - `void hasUIDrawables() const` - Returns whether UI drawables are present - `void draw(...)` - Main drawing method where video frame is rendered #### Texture Management - OpenGL texture object to hold current video frame - Texture updates when new frame is available - Proper texture format matching (GL_RGB, GL_RGBA, etc.) - Efficient texture sub-data updates when possible - Texture cleanup when node is destroyed #### Frame Data Handling - Receive frame data from Maya node via output attributes - Convert frame data to appropriate OpenGL format if needed - Update texture with new frame data - Maintain frame timestamp for synchronization ### Integration Points - Maya Node Integration: Connect to MPxNode's output attributes to receive frame data - Frame Rate Synchronization: Use frame timestamp to synchronize with Maya's timeline - Post-effects: Apply transformations to frame data before texture upload - Caching: Interface with frame caching mechanism to get frames efficiently ### Rendering Approach 1. Receive new frame data from MPxNode (via attribute change notification) 2. Convert frame data to OpenGL-compatible format if necessary 3. Update OpenGL texture with new frame data 4. In viewport draw call: - Set up appropriate shader program (simple texture shader) - Bind texture - Draw quad covering the image plane area - Apply any viewport-specific transformations ### Shader Program - Simple vertex and fragment shaders for texture rendering - Vertex shader: Pass through texture coordinates - Fragment shader: Sample texture and output color - Optional: Support for color correction or other viewport-specific effects ### Texture Updates - Use GL_PIXEL_UNPACK_BUFFER for efficient updates if supported - Fallback to glTexSubImage2D for broader compatibility - Handle different pixel formats (RGB, RGBA, BGRA, etc.) - Flip Y-axis if necessary (OpenGL vs image coordinate systems) ### Bounding Box and Selection - Return appropriate bounding box based on frame dimensions and aspect ratio - Support for viewport selection of the image plane - Handle transform nodes (position, rotation, scale) if applicable ### Threading Considerations - Viewport drawing occurs on main thread - Texture updates must happen on main thread (OpenGL context) - Use thread-safe queue to transfer frame data from decoding thread to main thread - Synchronize access to shared frame data ### Error Handling - Check for OpenGL errors after texture operations - Handle texture creation failures gracefully - Provide fallback rendering (e.g., colored quad) when frame data unavailable - Log errors to Maya's script editor using MGlobal::displayError ### Performance Considerations - Minimize texture format conversions - Update only changed portions of texture when possible - Use appropriate texture filtering (GL_LINEAR for smooth playback) - Avoid expensive operations during viewport drawing - Consider using persistent mapped buffers for frequent updates ### Maya API Specifics - Use MHWRender::MPxDrawOverride as base class - Register draw override with MHWRender::MRenderer - Use MImage class for image format conversions if needed - Leverage MHWRender::MShaderManager for shader programs - Follow Viewport 2.0 API guidelines for draw overrides ## Dependencies - Maya Node implementation (for frame data) - Frame caching mechanism - OpenGL 3.2+ (required for Viewport 2.0) - Maya API 2023 (MHWRender module)