indigo/MediaPlane
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Init Repo

Browse Source
This commit is contained in:
indigo
2026-03-19 09:34:11 +08:00
commit 6fcf135d6a
263 changed files with 304424 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/FFmpeg_Integration_Spec.md
+142
View File
@@ -0,0 +1,142 @@
# FFmpeg Integration Specification
## Overview
This document specifies the implementation details for integrating FFmpeg into the Maya Image Plane Node plugin for video frame decoding.
## Requirements
- Decode MP4 video files using FFmpeg libraries
- Extract video frames as raw pixel data (RGB or RGBA format)
- Provide frame-by-frame access for synchronization with Maya's timeline
- Handle various video codecs commonly used in MP4 containers
- Error handling for corrupted or unsupported video files
## Implementation Details
### Library Linking
- Link against FFmpeg libraries: libavcodec, libavformat, libavutil, libswscale
- Use pre-built FFmpeg binaries or build from source as needed
- Ensure compatibility with Maya 2023's compiler toolchain (Visual Studio 2017)
### Core Components
#### VideoDecoder Class
Responsible for initializing FFmpeg, opening video files, and decoding frames.
Key methods:
- `bool open(const std::string& filename)` - Opens video file and initializes FFmpeg context
- `void close()` - Releases FFmpeg resources
- `bool readFrame(AVFrame* frame)` - Decodes next video frame into provided AVFrame
- `int getWidth() const` - Returns video width
- `int getHeight() const` - Returns video height
- `double getFrameRate() const` - Returns video frame rate
- `int64_t getFrameCount() const` - Returns total number of frames
- `int64_t getCurrentFrame() const` - Returns current frame position
#### Pixel Format Conversion
- Convert decoded frames to RGB or RGBA format using libswscale
- Store converted frames in CPU memory for further processing
- Provide access to raw pixel data for viewport rendering
### Threading Considerations
- FFmpeg decoding should occur on a separate thread to avoid blocking Maya's main thread
- Use thread-safe queue for frame delivery to main thread
- Implement proper synchronization mechanisms (mutexes, condition variables)
### Error Handling
- Check return values from all FFmpeg functions
- Provide meaningful error messages for common issues:
- File not found
- Unsupported codec
- Corrupted file
- Memory allocation failures
- Graceful degradation when FFmpeg is unavailable
### Integration with Maya Node
- VideoDecoder instance managed by the MPxNode implementation
- Frame requests triggered by Maya's time changes
- Caching mechanism to avoid re-decoding frames
## Configuration
- FFmpeg library paths configurable via CMake
- Option to use system-installed FFmpeg or bundled binaries
- Build-time option to enable/disable FFmpeg integration (for testing)
## Testing
- Unit tests for VideoDecoder class
- Integration tests with sample MP4 files
- Performance testing for frame decoding speed
- Memory leak detection
## Dependencies
- FFmpeg 4.0 or later (tested with 4.4)
- libavcodec, libavformat, libavutil, libswscale
## Overview
This document specifies the implementation details for integrating FFmpeg into the Maya Image Plane Node plugin for video frame decoding.
## Requirements
- Decode MP4 video files using FFmpeg libraries
- Extract video frames as raw pixel data (RGB or RGBA format)
- Provide frame-by-frame access for synchronization with Maya's timeline
- Handle various video codecs commonly used in MP4 containers
- Error handling for corrupted or unsupported video files
## Implementation Details
### Library Linking
- Link against FFmpeg libraries: libavcodec, libavformat, libavutil, libswscale
- Use pre-built FFmpeg binaries or build from source as needed
- Ensure compatibility with Maya 2023's compiler toolchain (Visual Studio 2017)
### Core Components
#### VideoDecoder Class
Responsible for initializing FFmpeg, opening video files, and decoding frames.
Key methods:
- `bool open(const std::string& filename)` - Opens video file and initializes FFmpeg context
- `void close()` - Releases FFmpeg resources
- `bool readFrame(AVFrame* frame)` - Decodes next video frame into provided AVFrame
- `int getWidth() const` - Returns video width
- `int getHeight() const` - Returns video height
- `double getFrameRate() const` - Returns video frame rate
- `int64_t getFrameCount() const` - Returns total number of frames
- `int64_t getCurrentFrame() const` - Returns current frame position
#### Pixel Format Conversion
- Convert decoded frames to RGB or RGBA format using libswscale
- Store converted frames in CPU memory for further processing
- Provide access to raw pixel data for viewport rendering
### Threading Considerations
- FFmpeg decoding should occur on a separate thread to avoid blocking Maya's main thread
- Use thread-safe queue for frame delivery to main thread
- Implement proper synchronization mechanisms (mutexes, condition variables)
### Error Handling
- Check return values from all FFmpeg functions
- Provide meaningful error messages for common issues:
- File not found
- Unsupported codec
- Corrupted file
- Memory allocation failures
- Graceful degradation when FFmpeg is unavailable
### Integration with Maya Node
- VideoDecoder instance managed by the MPxNode implementation
- Frame requests triggered by Maya's time changes
- Caching mechanism to avoid re-decoding frames
## Configuration
- FFmpeg library paths configurable via CMake
- Option to use system-installed FFmpeg or bundled binaries
- Build-time option to enable/disable FFmpeg integration (for testing)
## Testing
- Unit tests for VideoDecoder class
- Integration tests with sample MP4 files
- Performance testing for frame decoding speed
- Memory leak detection
## Dependencies
- FFmpeg 4.0 or later (tested with 4.4)
- libavcodec, libavformat, libavutil, libswscale
docs/Frame_Rate_Synchronization_Spec.md
+192
View File
@@ -0,0 +1,192 @@
# Frame Rate Synchronization Specification
## Overview
This document specifies the implementation details for synchronizing video playback with Maya's timeline frame rate and providing user-adjustable playback rate control.
## Requirements
- Read Maya's current frame rate from the timeline
- Synchronize video frame advancement with Maya's timeline when enabled
- Allow user to override playback speed with a multiplier
- Support looping and non-looping playback modes
- Handle frame rate changes in Maya's timeline dynamically
- Provide smooth playback even when Maya's frame rate differs from video's native frame rate
## Implementation Details
### Frame Rate Sources
1. **Maya's Timeline Frame Rate**: Obtained from `MTime::uiUnit()` or `MTime::getFrameRate()`
2. **Video's Native Frame Rate**: Obtained from FFmpeg decoder (`getFrameRate()` method)
3. **User Playback Rate Multiplier**: Custom attribute on the MPxNode
### Core Logic
#### Frame Calculation Algorithm
When `useMayaFrameRate` is enabled:
```
effectiveFrameRate = mayaFrameRate * userPlaybackRate
targetFrame = (currentTime - startTime) * effectiveFrameRate
```
When `useMayaFrameRate` is disabled:
```
effectiveFrameRate = videoFrameRate * userPlaybackRate
targetFrame = (currentTime - startTime) * effectiveFrameRate
```
Where:
- `currentTime`: Maya's current time in seconds
- `startTime`: Time when video playback started or was reset
- `targetFrame`: Frame number to display (0-based)
- `mayaFrameRate`: Frames per second from Maya's timeline
- `videoFrameRate`: Native frames per second from video file
- `userPlaybackRate`: User-defined multiplier (1.0 = normal speed)
### Implementation Components
#### Time Management Class
Handles time calculations and frame targeting:
- Store start time when playback begins or resets
- Calculate target frame based on current time and settings
- Handle looping by wrapping target frame within video duration
- Provide methods to get current frame with sub-frame precision for interpolation
#### Attribute Definitions
In the MPxNode:
- `useMayaFrameRate` (bool): Toggle between Maya-synced and video-native frame rates
- `playbackRate` (double): User-adjustable playback rate multiplier (default 1.0)
- `startTime` (double): Internal attribute to track when playback started
- `isPlaying` (bool): Internal attribute to track playback state
#### Integration with Maya Timeline
- Connect to Maya's timeChanged event via node attributes
- Use `currentTime` input attribute driven by Maya's timeline
- Update node computation when timeline changes
- Handle scrubbing (jumping to different times) correctly
### Threading Considerations
- Time calculations occur on Maya's main thread during node computation
- No shared state with decoding thread except for frame requests
- Frame requests to decoder should be thread-safe
### Error Handling
- Handle invalid frame rates (zero or negative)
- Clamp playback rate to reasonable range (e.g., 0.01 to 10.0)
- Handle case where video frame rate is unavailable
- Graceful degradation to approximate synchronization
### Performance Considerations
- Minimize calculations in node compute method
- Cache Maya's frame rate when possible (update only when changed)
- Avoid expensive trigonometric or transcendental functions in real-time paths
### Integration Points
- MPxNode: Provides frame calculation logic and attributes
- FFmpeg Decoder: Receives frame requests based on calculated target frame
- Viewport 2.0: Displays the frame requested by the synchronization logic
- Caching: Requests frames from cache based on target frame
### Maya API Specifics
- Use MTime class for time manipulations
- Use MAnimControl to get Maya's current time if not using attribute connection
- Use MTime::uiUnit() to get current UI time unit
- Register time changed callbacks if needed for more responsive updates
## Dependencies
- Maya API 2023 (MTime, MAnimControl)
- MPxNode implementation
- FFmpeg decoder interface
## Overview
This document specifies the implementation details for synchronizing video playback with Maya's timeline frame rate and providing user-adjustable playback rate control.
## Requirements
- Read Maya's current frame rate from the timeline
- Synchronize video frame advancement with Maya's timeline when enabled
- Allow user to override playback speed with a multiplier
- Support looping and non-looping playback modes
- Handle frame rate changes in Maya's timeline dynamically
- Provide smooth playback even when Maya's frame rate differs from video's native frame rate
## Implementation Details
### Frame Rate Sources
1. **Maya's Timeline Frame Rate**: Obtained from `MTime::uiUnit()` or `MTime::getFrameRate()`
2. **Video's Native Frame Rate**: Obtained from FFmpeg decoder (`getFrameRate()` method)
3. **User Playback Rate Multiplier**: Custom attribute on the MPxNode
### Core Logic
#### Frame Calculation Algorithm
When `useMayaFrameRate` is enabled:
```
effectiveFrameRate = mayaFrameRate * userPlaybackRate
targetFrame = (currentTime - startTime) * effectiveFrameRate
```
When `useMayaFrameRate` is disabled:
```
effectiveFrameRate = videoFrameRate * userPlaybackRate
targetFrame = (currentTime - startTime) * effectiveFrameRate
```
Where:
- `currentTime`: Maya's current time in seconds
- `startTime`: Time when video playback started or was reset
- `targetFrame`: Frame number to display (0-based)
- `mayaFrameRate`: Frames per second from Maya's timeline
- `videoFrameRate`: Native frames per second from video file
- `userPlaybackRate`: User-defined multiplier (1.0 = normal speed)
### Implementation Components
#### Time Management Class
Handles time calculations and frame targeting:
- Store start time when playback begins or resets
- Calculate target frame based on current time and settings
- Handle looping by wrapping target frame within video duration
- Provide methods to get current frame with sub-frame precision for interpolation
#### Attribute Definitions
In the MPxNode:
- `useMayaFrameRate` (bool): Toggle between Maya-synced and video-native frame rates
- `playbackRate` (double): User-adjustable playback rate multiplier (default 1.0)
- `startTime` (double): Internal attribute to track when playback started
- `isPlaying` (bool): Internal attribute to track playback state
#### Integration with Maya Timeline
- Connect to Maya's timeChanged event via node attributes
- Use `currentTime` input attribute driven by Maya's timeline
- Update node computation when timeline changes
- Handle scrubbing (jumping to different times) correctly
### Threading Considerations
- Time calculations occur on Maya's main thread during node computation
- No shared state with decoding thread except for frame requests
- Frame requests to decoder should be thread-safe
### Error Handling
- Handle invalid frame rates (zero or negative)
- Clamp playback rate to reasonable range (e.g., 0.01 to 10.0)
- Handle case where video frame rate is unavailable
- Graceful degradation to approximate synchronization
### Performance Considerations
- Minimize calculations in node compute method
- Cache Maya's frame rate when possible (update only when changed)
- Avoid expensive trigonometric or transcendental functions in real-time paths
### Integration Points
- MPxNode: Provides frame calculation logic and attributes
- FFmpeg Decoder: Receives frame requests based on calculated target frame
- Viewport 2.0: Displays the frame requested by the synchronization logic
- Caching: Requests frames from cache based on target frame
### Maya API Specifics
- Use MTime class for time manipulations
- Use MAnimControl to get Maya's current time if not using attribute connection
- Use MTime::uiUnit() to get current UI time unit
- Register time changed callbacks if needed for more responsive updates
## Dependencies
- Maya API 2023 (MTime, MAnimControl)
- MPxNode implementation
- FFmpeg decoder interface
docs/Maya_Node_Implementation_Spec.md
+228
View File
@@ -0,0 +1,228 @@
# Maya Node Implementation Specification
## Overview
This document specifies the implementation details for the Maya MPxNode plugin that handles video file input and frame output for the Image Plane Node.
## Requirements
- Create a custom MPxNode that accepts video file path as input
- Output video frame data that can be consumed by viewport rendering
- Support attributes for frame rate control and playback rate adjustment
- Integrate with FFmpeg video decoding component
- Provide frame caching interface
- Follow Maya API best practices for node creation
## Implementation Details
### Node Definition
- Node Name: `imagePlaneVideo`
- Node ID: Unique ID obtained from Autodesk
- Classification: `filter` or `image` (to be determined based on Maya's categorization)
- Register as a Maya plugin using MFnPlugin
### Attributes
#### Input Attributes
1. `videoFile` (message or string) - Path to the video file
2. `frameRate` (double) - Playback rate multiplier (1.0 = normal speed)
3. `useMayaFrameRate` (bool) - Whether to synchronize with Maya's timeline frame rate
4. `currentTime` (double) - Connection to Maya's timeline (for frame calculation)
5. `loop` (bool) - Whether to loop the video playback
6. `postEffectCrop` (double4) - Crop parameters (x, y, width, height)
7. `postEffectResize` (double2) - Resize parameters (width, height)
8. `postEffectFlip` (bool2) - Flip parameters (flipX, flipY)
#### Output Attributes
1. `outFrameData` (message or custom data type) - Pointer to video frame data
2. `outFrameWidth` (int) - Width of current frame
3. `outFrameHeight` (int) - Height of current frame
4. `outFrameFormat` (int) - Pixel format of current frame (RGB, RGBA, etc.)
5. `outFrameTimestamp` (double) - Timestamp of current frame
6. `isValid` (bool) - Whether the current frame data is valid
### Core Components
#### VideoFrameData Class
Custom data class to hold video frame information:
- Pointer to pixel data (RGB/RGBA buffer)
- Width and height
- Pixel format
- Timestamp
- Reference counting for proper cleanup
#### Node Computation
In the `compute()` method:
1. Check if video file attribute has changed
2. If changed, initialize FFmpeg decoder with new file
3. Calculate target frame based on:
- Maya's current time (if useMayaFrameRate is true)
- Frame rate multiplier attribute
- Loop settings
4. Request frame from FFmpeg decoder (or cache)
5. Apply post-effects if specified
6. Update output attributes with frame data
7. Mark node as clean
### Integration Points
- FFmpeg Integration: Use VideoDecoder class from FFmpeg integration spec
- Viewport 2.0 Integration: Provide frame data to MPxDrawOverride via output attributes
- Caching: Interface with frame caching mechanism (to be specified separately)
- Post-effects: Apply transformations after decoding but before output
### Threading Considerations
- Node computation occurs on Maya's main thread
- FFmpeg decoding should happen on background thread
- Use thread-safe mechanisms to transfer decoded frames to main thread
- Cache access must be thread-safe
### Error Handling and Validation
- Validate video file path exists and is readable
- Check FFmpeg initialization success
- Handle end-of-file conditions (loop or stop)
- Provide error states via output attributes
- Log errors to Maya's script editor using MGlobal::displayError
### Node Initialization and Cleanup
- `initialize()`: Define all attributes and set up attribute affects
- `constructor()`: Initialize member variables
- `destructor()`: Clean up FFmpeg decoder and frame data
- `postConstructor()`: Set node to be internally managed if needed
### Attribute Affects
Specify which attributes affect which outputs:
- videoFile -> outFrameData, outFrameWidth, outFrameHeight, outFrameFormat, isValid
- frameRate -> outFrameData, outFrameTimestamp
- useMayaFrameRate -> outFrameData, outFrameTimestamp
- currentTime -> outFrameData, outFrameTimestamp
- loop -> outFrameData, outFrameTimestamp
- postEffect* -> outFrameData (if implemented as part of node computation)
### Performance Considerations
- Minimize computation in `compute()` method
- Cache frequently accessed values
- Avoid expensive operations during node evaluation
- Use dirty propagation to minimize unnecessary recomputation
## Maya API Specifics
- Use MFnTypedAttribute for message/string attributes
- Use MFnNumericAttribute for double/int/bool attributes
- Use MFnEnumAttribute for format options if needed
- Set appropriate attribute properties (keyable, storable, readable, writable)
- Use MTypeId for unique node identification
- Register node with MFnPlugin::registerNode
## Dependencies
- FFmpeg integration component
- Frame caching mechanism
- Maya API 2023 (OpenMaya, OpenMayaFX)
## Overview
This document specifies the implementation details for the Maya MPxNode plugin that handles video file input and frame output for the Image Plane Node.
## Requirements
- Create a custom MPxNode that accepts video file path as input
- Output video frame data that can be consumed by viewport rendering
- Support attributes for frame rate control and playback rate adjustment
- Integrate with FFmpeg video decoding component
- Provide frame caching interface
- Follow Maya API best practices for node creation
## Implementation Details
### Node Definition
- Node Name: `imagePlaneVideo`
- Node ID: Unique ID obtained from Autodesk
- Classification: `filter` or `image` (to be determined based on Maya's categorization)
- Register as a Maya plugin using MFnPlugin
### Attributes
#### Input Attributes
1. `videoFile` (message or string) - Path to the video file
2. `frameRate` (double) - Playback rate multiplier (1.0 = normal speed)
3. `useMayaFrameRate` (bool) - Whether to synchronize with Maya's timeline frame rate
4. `currentTime` (double) - Connection to Maya's timeline (for frame calculation)
5. `loop` (bool) - Whether to loop the video playback
6. `postEffectCrop` (double4) - Crop parameters (x, y, width, height)
7. `postEffectResize` (double2) - Resize parameters (width, height)
8. `postEffectFlip` (bool2) - Flip parameters (flipX, flipY)
#### Output Attributes
1. `outFrameData` (message or custom data type) - Pointer to video frame data
2. `outFrameWidth` (int) - Width of current frame
3. `outFrameHeight` (int) - Height of current frame
4. `outFrameFormat` (int) - Pixel format of current frame (RGB, RGBA, etc.)
5. `outFrameTimestamp` (double) - Timestamp of current frame
6. `isValid` (bool) - Whether the current frame data is valid
### Core Components
#### VideoFrameData Class
Custom data class to hold video frame information:
- Pointer to pixel data (RGB/RGBA buffer)
- Width and height
- Pixel format
- Timestamp
- Reference counting for proper cleanup
#### Node Computation
In the `compute()` method:
1. Check if video file attribute has changed
2. If changed, initialize FFmpeg decoder with new file
3. Calculate target frame based on:
- Maya's current time (if useMayaFrameRate is true)
- Frame rate multiplier attribute
- Loop settings
4. Request frame from FFmpeg decoder (or cache)
5. Apply post-effects if specified
6. Update output attributes with frame data
7. Mark node as clean
### Integration Points
- FFmpeg Integration: Use VideoDecoder class from FFmpeg integration spec
- Viewport 2.0 Integration: Provide frame data to MPxDrawOverride via output attributes
- Caching: Interface with frame caching mechanism (to be specified separately)
- Post-effects: Apply transformations after decoding but before output
### Threading Considerations
- Node computation occurs on Maya's main thread
- FFmpeg decoding should happen on background thread
- Use thread-safe mechanisms to transfer decoded frames to main thread
- Cache access must be thread-safe
### Error Handling and Validation
- Validate video file path exists and is readable
- Check FFmpeg initialization success
- Handle end-of-file conditions (loop or stop)
- Provide error states via output attributes
- Log errors to Maya's script editor using MGlobal::displayError
### Node Initialization and Cleanup
- `initialize()`: Define all attributes and set up attribute affects
- `constructor()`: Initialize member variables
- `destructor()`: Clean up FFmpeg decoder and frame data
- `postConstructor()`: Set node to be internally managed if needed
### Attribute Affects
Specify which attributes affect which outputs:
- videoFile -> outFrameData, outFrameWidth, outFrameHeight, outFrameFormat, isValid
- frameRate -> outFrameData, outFrameTimestamp
- useMayaFrameRate -> outFrameData, outFrameTimestamp
- currentTime -> outFrameData, outFrameTimestamp
- loop -> outFrameData, outFrameTimestamp
- postEffect* -> outFrameData (if implemented as part of node computation)
### Performance Considerations
- Minimize computation in `compute()` method
- Cache frequently accessed values
- Avoid expensive operations during node evaluation
- Use dirty propagation to minimize unnecessary recomputation
## Maya API Specifics
- Use MFnTypedAttribute for message/string attributes
- Use MFnNumericAttribute for double/int/bool attributes
- Use MFnEnumAttribute for format options if needed
- Set appropriate attribute properties (keyable, storable, readable, writable)
- Use MTypeId for unique node identification
- Register node with MFnPlugin::registerNode
## Dependencies
- FFmpeg integration component
- Frame caching mechanism
- Maya API 2023 (OpenMaya, OpenMayaFX)
docs/Module_Packaging_Spec.md
+288
View File
@@ -0,0 +1,288 @@
# Module Packaging Specification
## Overview
This document specifies the implementation details for packaging the Maya Image Plane Node plugin as a Maya module for easy installation and distribution.
## Requirements
- Create a proper Maya module structure
- Ensure the plugin loads correctly when placed in Maya's module path
- Support multiple versions of Maya (specifically targeting Maya 2023)
- Include all necessary dependencies (FFmpeg binaries if not system-installed)
- Provide version information for the plugin
- Support both debug and release builds
- Allow easy uninstallation and upgrading
## Implementation Details
### Maya Module Structure
A Maya module consists of:
1. A module definition file (.mod)
2. The plugin binary (.mll for Windows)
3. Optional: scripts, icons, help files, etc.
### Module Definition File
Create `MayaImagePlaneNode.mod` with contents:
```
+ MayaImagePlaneNode 1.0 ${MAYA_IMAGE_PLANE_NODE_PATH}
PLUG-INS:
```
Where:
- `MayaImagePlaneNode` is the module name
- `1.0` is the version
- `${MAYA_IMAGE_PLANE_NODE_PATH}` is the environment variable pointing to the module root
- `PLUG-INS:` specifies where to find plugin files
### Directory Structure
```
MayaImagePlaneNode/
├── MayaImagePlaneNode.mod
├── bin/
│ └── MayaImagePlaneNode.mll
├── lib/
│ ├── avcodec-58.dll
│ ├── avformat-58.dll
│ ├── avutil-56.dll
│ └── swscale-5.dll
├── scripts/
│ └── (optional MEL/Python scripts)
├── icons/
│ └── (optional node icons)
└── doc/
└── (optional documentation)
```
### Build System Integration
Modify CMake to:
1. Install plugin to appropriate bin directory
2. Copy FFmpeg DLLs to bin directory (if bundling)
3. Generate module file during build
4. Provide option to create package for distribution
### Versioning
- Use semantic versioning (major.minor.patch)
- Version information accessible via plugin attributes
- Update module file version when plugin version changes
### Dependencies Handling
Option 1: System FFmpeg
- Rely on FFmpeg being installed in system PATH
- Simpler deployment but requires user to install FFmpeg separately
Option 2: Bundled FFmpeg
- Include FFmpeg DLLs with the plugin
- Larger distribution but guaranteed to work
- Must comply with FFmpeg licensing (LGPL/GPL)
Option 3: Hybrid
- Use system FFmpeg if available, fallback to bundled
- Best of both approaches
### Installation Process
1. User extracts module to a directory
2. Sets MAYA_IMAGE_PLANE_NODE_PATH environment variable to module root
OR places module in standard Maya module locations:
- `<user>/Documents/maya/<version>/modules/`
- `<Maya installation>/modules/`
3. Maya automatically detects and loads the module on startup
4. Plugin becomes available in Maya's Plugin Manager
### Uninstallation
1. Remove module directory
2. Remove environment variable (if set)
3. Restart Maya
### Build Configurations
- Debug build: MayaImagePlaneNode_debug.mll
- Release build: MayaImagePlaneNode.mll
- Consider using different module files or paths for debug vs release
### Maya API Version Compatibility
- Compiled against Maya 2023 API
- May work with other versions but not guaranteed
- Consider using version-specific module files if needed
### Loading Verification
- Plugin should register successfully with Maya's plugin system
- Node type should be available in Node Editor
- Attributes should be accessible and editable
- Viewport display should work when node is created
### Dependencies
- Maya 2023 development kit
- FFmpeg libraries (matching Maya's compiler version)
- CMake 3.14+
- Visual Studio 2017 (matching Maya 2023's compiler)
## Implementation Plan
### CMake Modifications
1. Add install() commands for plugin and dependencies
2. Generate module file using configure_file()
3. Optionally create package() target for CPack
4. Set up version numbers
### Module File Template
Create `MayaImagePlaneNode.mod.in`:
```
+ MayaImagePlaneNode @PROJECT_VERSION@ @CMAKE_INSTALL_PREFIX@
PLUG-INS:
```
### Environment Variables
- MAYA_IMAGE_PLANE_NODE_PATH: Points to module root
- Alternatively, rely on standard Maya module discovery
### Testing
- Verify module loads in clean Maya installation
- Test with both debug and release builds
- Test FFmpeg dependency resolution
- Verify plugin appears in Plugin Manager
- Test node creation and attribute editing
## Dependencies
- CMake
- Maya 2023 SDK
## Overview
This document specifies the implementation details for packaging the Maya Image Plane Node plugin as a Maya module for easy installation and distribution.
## Requirements
- Create a proper Maya module structure
- Ensure the plugin loads correctly when placed in Maya's module path
- Support multiple versions of Maya (specifically targeting Maya 2023)
- Include all necessary dependencies (FFmpeg binaries if not system-installed)
- Provide version information for the plugin
- Support both debug and release builds
- Allow easy uninstallation and upgrading
## Implementation Details
### Maya Module Structure
A Maya module consists of:
1. A module definition file (.mod)
2. The plugin binary (.mll for Windows)
3. Optional: scripts, icons, help files, etc.
### Module Definition File
Create `MayaImagePlaneNode.mod` with contents:
```
+ MayaImagePlaneNode 1.0 ${MAYA_IMAGE_PLANE_NODE_PATH}
PLUG-INS:
```
Where:
- `MayaImagePlaneNode` is the module name
- `1.0` is the version
- `${MAYA_IMAGE_PLANE_NODE_PATH}` is the environment variable pointing to the module root
- `PLUG-INS:` specifies where to find plugin files
### Directory Structure
```
MayaImagePlaneNode/
├── MayaImagePlaneNode.mod
├── bin/
│ └── MayaImagePlaneNode.mll
├── lib/
│ ├── avcodec-58.dll
│ ├── avformat-58.dll
│ ├── avutil-56.dll
│ └── swscale-5.dll
├── scripts/
│ └── (optional MEL/Python scripts)
├── icons/
│ └── (optional node icons)
└── doc/
└── (optional documentation)
```
### Build System Integration
Modify CMake to:
1. Install plugin to appropriate bin directory
2. Copy FFmpeg DLLs to bin directory (if bundling)
3. Generate module file during build
4. Provide option to create package for distribution
### Versioning
- Use semantic versioning (major.minor.patch)
- Version information accessible via plugin attributes
- Update module file version when plugin version changes
### Dependencies Handling
Option 1: System FFmpeg
- Rely on FFmpeg being installed in system PATH
- Simpler deployment but requires user to install FFmpeg separately
Option 2: Bundled FFmpeg
- Include FFmpeg DLLs with the plugin
- Larger distribution but guaranteed to work
- Must comply with FFmpeg licensing (LGPL/GPL)
Option 3: Hybrid
- Use system FFmpeg if available, fallback to bundled
- Best of both approaches
### Installation Process
1. User extracts module to a directory
2. Sets MAYA_IMAGE_PLANE_NODE_PATH environment variable to module root
OR places module in standard Maya module locations:
- `<user>/Documents/maya/<version>/modules/`
- `<Maya installation>/modules/`
3. Maya automatically detects and loads the module on startup
4. Plugin becomes available in Maya's Plugin Manager
### Uninstallation
1. Remove module directory
2. Remove environment variable (if set)
3. Restart Maya
### Build Configurations
- Debug build: MayaImagePlaneNode_debug.mll
- Release build: MayaImagePlaneNode.mll
- Consider using different module files or paths for debug vs release
### Maya API Version Compatibility
- Compiled against Maya 2023 API
- May work with other versions but not guaranteed
- Consider using version-specific module files if needed
### Loading Verification
- Plugin should register successfully with Maya's plugin system
- Node type should be available in Node Editor
- Attributes should be accessible and editable
- Viewport display should work when node is created
### Dependencies
- Maya 2023 development kit
- FFmpeg libraries (matching Maya's compiler version)
- CMake 3.14+
- Visual Studio 2017 (matching Maya 2023's compiler)
## Implementation Plan
### CMake Modifications
1. Add install() commands for plugin and dependencies
2. Generate module file using configure_file()
3. Optionally create package() target for CPack
4. Set up version numbers
### Module File Template
Create `MayaImagePlaneNode.mod.in`:
```
+ MayaImagePlaneNode @PROJECT_VERSION@ @CMAKE_INSTALL_PREFIX@
PLUG-INS:
```
### Environment Variables
- MAYA_IMAGE_PLANE_NODE_PATH: Points to module root
- Alternatively, rely on standard Maya module discovery
### Testing
- Verify module loads in clean Maya installation
- Test with both debug and release builds
- Test FFmpeg dependency resolution
- Verify plugin appears in Plugin Manager
- Test node creation and attribute editing
## Dependencies
- CMake
- Maya 2023 SDK
docs/Post_Effects_Spec.md
+232
View File
@@ -0,0 +1,232 @@
# Post-Effects Specification
## Overview
This document specifies the implementation details for post-effects processing on video frames, including crop, resize, and flip operations.
## Requirements
- Implement crop functionality to select a region of interest from the video frame
- Implement resize functionality to scale the video frame to different dimensions
- Implement flip functionality to mirror the video frame horizontally and/or vertically
- Allow combining multiple post-effects in any order
- Maintain high performance for real-time playback
- Support various pixel formats (RGB, RGBA, etc.)
- Provide intuitive user controls for each effect
## Implementation Details
### Effect Definitions
#### Crop
- Parameters: x, y, width, height (in pixels)
- Defines rectangular region to extract from source frame
- Coordinates relative to top-left corner of source frame
- Width and height must be within source frame bounds
- If crop region extends beyond source frame, clamp to valid region
#### Resize
- Parameters: width, height (in pixels)
- Target dimensions for output frame
- Maintain aspect ratio option (to be determined)
- Use appropriate filtering algorithm (bilinear recommended for balance of quality/performance)
#### Flip
- Parameters: flipX (bool), flipY (bool)
- flipX: Mirror frame horizontally (left-right)
- flipY: Mirror frame vertically (top-bottom)
- Can be applied independently or together
### Processing Order
The recommended order of operations is:
1. Crop (select region of interest)
2. Resize (scale to desired dimensions)
3. Flip (apply mirroring)
This order ensures that flipping operates on the final composed image.
### Implementation Components
#### FrameProcessor Class
Responsible for applying post-effects to decoded video frames.
Key methods:
- `bool processFrame(const AVFrame* srcFrame, AVFrame* dstFrame)` - Apply all enabled effects
- `void setCropParameters(int x, int y, int width, int height)` - Configure crop
- `void setResizeParameters(int width, int height)` - Configure resize
- `void setFlipParameters(bool flipX, bool flipY)` - Configure flip
- `void enableCrop(bool enabled)` - Enable/disable crop effect
- `void enableResize(bool enabled)` - Enable/disable resize effect
- `void enableFlip(bool enabled)` - Enable/disable flip effect
- `bool isCropEnabled() const` - Check if crop is enabled
- `bool isResizeEnabled() const` - Check if resize is enabled
- `bool isFlipEnabled() const` - Check if flip is enabled
#### Internal Processing Steps
1. Validate input frame
2. Apply crop if enabled:
- Calculate source rectangle
- Extract region from source frame
- Handle format conversion if needed
3. Apply resize if enabled:
- Use libswscale for scaling with appropriate filters
- Allocate temporary frame if needed
4. Apply flip if enabled:
- Perform in-place mirroring of frame data
- Handle different pixel formats correctly
5. Output processed frame
### Integration Points
- FFmpeg Decoder: Receives raw frames from decoder
- Maya Node: Applies post-effects before outputting frame data
- Viewport 2.0: Receives post-processed frames for display
- Caching: Option to cache frames before or after post-effects (to be determined)
### Threading Considerations
- Post-processing should occur on the same thread as frame retrieval from decoder
- Can be done either in background decoding thread or main thread
- If done in background thread, ensure thread-safe delivery to main thread
- Minimize processing time to avoid blocking pipeline
### Error Handling
- Validate effect parameters (non-negative dimensions, etc.)
- Handle memory allocation failures gracefully
- Provide fallback to unprocessed frame if processing fails
- Log errors to Maya's script editor using MGlobal::displayError
### Performance Considerations
- Use hardware-accelerated operations where possible (though limited in CPU-based processing)
- Minimize memory allocations and copies
- Use efficient scaling algorithms (bilinear as good compromise)
- Consider processing only when parameters change
- Reuse buffers when possible to avoid reallocations
### Pixel Format Support
- Primary support for RGB24 and RGBA32 formats
- Convert other formats to supported ones if needed
- Maintain format consistency through processing chain
- Handle format conversion with libswscale when necessary
### Maya API Specifics
- Store effect parameters as node attributes
- Use MFnNumericAttribute for numeric parameters (x, y, width, height, flip bools)
- Use MFnBooleanAttribute for enable/disable toggles
- Implement attribute validation in node compute method
- Provide default values that result in no-op when effects disabled
## Dependencies
- FFmpeg libraries (libswscale for scaling and format conversion)
- MPxNode implementation
- Frame data structures
## Overview
This document specifies the implementation details for post-effects processing on video frames, including crop, resize, and flip operations.
## Requirements
- Implement crop functionality to select a region of interest from the video frame
- Implement resize functionality to scale the video frame to different dimensions
- Implement flip functionality to mirror the video frame horizontally and/or vertically
- Allow combining multiple post-effects in any order
- Maintain high performance for real-time playback
- Support various pixel formats (RGB, RGBA, etc.)
- Provide intuitive user controls for each effect
## Implementation Details
### Effect Definitions
#### Crop
- Parameters: x, y, width, height (in pixels)
- Defines rectangular region to extract from source frame
- Coordinates relative to top-left corner of source frame
- Width and height must be within source frame bounds
- If crop region extends beyond source frame, clamp to valid region
#### Resize
- Parameters: width, height (in pixels)
- Target dimensions for output frame
- Maintain aspect ratio option (to be determined)
- Use appropriate filtering algorithm (bilinear recommended for balance of quality/performance)
#### Flip
- Parameters: flipX (bool), flipY (bool)
- flipX: Mirror frame horizontally (left-right)
- flipY: Mirror frame vertically (top-bottom)
- Can be applied independently or together
### Processing Order
The recommended order of operations is:
1. Crop (select region of interest)
2. Resize (scale to desired dimensions)
3. Flip (apply mirroring)
This order ensures that flipping operates on the final composed image.
### Implementation Components
#### FrameProcessor Class
Responsible for applying post-effects to decoded video frames.
Key methods:
- `bool processFrame(const AVFrame* srcFrame, AVFrame* dstFrame)` - Apply all enabled effects
- `void setCropParameters(int x, int y, int width, int height)` - Configure crop
- `void setResizeParameters(int width, int height)` - Configure resize
- `void setFlipParameters(bool flipX, bool flipY)` - Configure flip
- `void enableCrop(bool enabled)` - Enable/disable crop effect
- `void enableResize(bool enabled)` - Enable/disable resize effect
- `void enableFlip(bool enabled)` - Enable/disable flip effect
- `bool isCropEnabled() const` - Check if crop is enabled
- `bool isResizeEnabled() const` - Check if resize is enabled
- `bool isFlipEnabled() const` - Check if flip is enabled
#### Internal Processing Steps
1. Validate input frame
2. Apply crop if enabled:
- Calculate source rectangle
- Extract region from source frame
- Handle format conversion if needed
3. Apply resize if enabled:
- Use libswscale for scaling with appropriate filters
- Allocate temporary frame if needed
4. Apply flip if enabled:
- Perform in-place mirroring of frame data
- Handle different pixel formats correctly
5. Output processed frame
### Integration Points
- FFmpeg Decoder: Receives raw frames from decoder
- Maya Node: Applies post-effects before outputting frame data
- Viewport 2.0: Receives post-processed frames for display
- Caching: Option to cache frames before or after post-effects (to be determined)
### Threading Considerations
- Post-processing should occur on the same thread as frame retrieval from decoder
- Can be done either in background decoding thread or main thread
- If done in background thread, ensure thread-safe delivery to main thread
- Minimize processing time to avoid blocking pipeline
### Error Handling
- Validate effect parameters (non-negative dimensions, etc.)
- Handle memory allocation failures gracefully
- Provide fallback to unprocessed frame if processing fails
- Log errors to Maya's script editor using MGlobal::displayError
### Performance Considerations
- Use hardware-accelerated operations where possible (though limited in CPU-based processing)
- Minimize memory allocations and copies
- Use efficient scaling algorithms (bilinear as good compromise)
- Consider processing only when parameters change
- Reuse buffers when possible to avoid reallocations
### Pixel Format Support
- Primary support for RGB24 and RGBA32 formats
- Convert other formats to supported ones if needed
- Maintain format consistency through processing chain
- Handle format conversion with libswscale when necessary
### Maya API Specifics
- Store effect parameters as node attributes
- Use MFnNumericAttribute for numeric parameters (x, y, width, height, flip bools)
- Use MFnBooleanAttribute for enable/disable toggles
- Implement attribute validation in node compute method
- Provide default values that result in no-op when effects disabled
## Dependencies
- FFmpeg libraries (libswscale for scaling and format conversion)
- MPxNode implementation
- Frame data structures
docs/User_Adjustable_Playback_Rate_Spec.md
+146
View File
@@ -0,0 +1,146 @@
# User-Adjustable Playback Rate Specification
## Overview
This document specifies the implementation details for allowing users to adjust video playback speed independently of Maya's timeline frame rate.
## Requirements
- Provide a user-controllable playback rate multiplier attribute
- Allow playback rates from slow motion (0.1x) to fast motion (10.0x)
- Support reverse playback (negative rates) - optional
- Integrate with frame rate synchronization logic
- Maintain audio-video sync if audio is ever added (for future expansion)
- Provide smooth acceleration/deceleration when rate changes
## Implementation Details
### Attribute Definition
- `playbackRate` (double): User-adjustable playback rate multiplier
- Default value: 1.0 (normal speed)
- Minimum value: 0.01 (1% speed)
- Maximum value: 10.0 (10x speed)
- Keyable: Yes (can be animated)
- Storable: Yes (saved with scene)
- Readable/Writable: Yes
### Integration with Frame Rate Synchronization
The playback rate multiplies the effective frame rate in the synchronization logic:
When `useMayaFrameRate` is true:
```
effectiveFrameRate = mayaFrameRate * playbackRate
```
When `useMayaFrameRate` is false:
```
effectiveFrameRate = videoFrameRate * playbackRate
```
### Implementation Components
#### UI Considerations
- The attribute should appear in the node's attribute editor
- Consider adding a slider widget for intuitive control (via MPxNodeUI or similar)
- Provide visual feedback when rate is not 1.0
#### Behavior
- When playbackRate changes, recalculate target frame immediately
- Maintain synchronization with Maya's timeline when useMayaFrameRate is true
- Allow creative effects like slow motion, fast motion, and reverse playback
- Handle rate changes smoothly to avoid jumps in playback
### Error Handling
- Clamp playbackRate to valid range in node compute method
- Handle extreme values gracefully
- Provide warnings in script editor for clamped values
### Performance Considerations
- Simple multiplication operation, minimal performance impact
- No additional memory allocation required
- Attribute change notifications are lightweight
### Integration Points
- MPxNode: Contains the playbackRate attribute
- Frame Rate Synchronization: Uses playbackRate in calculations
- Developer 2: Implements the attribute and integrates with synchronization logic
### Maya API Specifics
- Use MFnNumericAttribute to create the attribute
- Set attribute properties (min, max, default, keyable, etc.)
- Handle attribute changes in compute method
- Use MFnDoubleData if storing as internal data (not needed for simple double attribute)
## Dependencies
- MPxNode implementation
- Frame rate synchronization logic
## Overview
This document specifies the implementation details for allowing users to adjust video playback speed independently of Maya's timeline frame rate.
## Requirements
- Provide a user-controllable playback rate multiplier attribute
- Allow playback rates from slow motion (0.1x) to fast motion (10.0x)
- Support reverse playback (negative rates) - optional
- Integrate with frame rate synchronization logic
- Maintain audio-video sync if audio is ever added (for future expansion)
- Provide smooth acceleration/deceleration when rate changes
## Implementation Details
### Attribute Definition
- `playbackRate` (double): User-adjustable playback rate multiplier
- Default value: 1.0 (normal speed)
- Minimum value: 0.01 (1% speed)
- Maximum value: 10.0 (10x speed)
- Keyable: Yes (can be animated)
- Storable: Yes (saved with scene)
- Readable/Writable: Yes
### Integration with Frame Rate Synchronization
The playback rate multiplies the effective frame rate in the synchronization logic:
When `useMayaFrameRate` is true:
```
effectiveFrameRate = mayaFrameRate * playbackRate
```
When `useMayaFrameRate` is false:
```
effectiveFrameRate = videoFrameRate * playbackRate
```
### Implementation Components
#### UI Considerations
- The attribute should appear in the node's attribute editor
- Consider adding a slider widget for intuitive control (via MPxNodeUI or similar)
- Provide visual feedback when rate is not 1.0
#### Behavior
- When playbackRate changes, recalculate target frame immediately
- Maintain synchronization with Maya's timeline when useMayaFrameRate is true
- Allow creative effects like slow motion, fast motion, and reverse playback
- Handle rate changes smoothly to avoid jumps in playback
### Error Handling
- Clamp playbackRate to valid range in node compute method
- Handle extreme values gracefully
- Provide warnings in script editor for clamped values
### Performance Considerations
- Simple multiplication operation, minimal performance impact
- No additional memory allocation required
- Attribute change notifications are lightweight
### Integration Points
- MPxNode: Contains the playbackRate attribute
- Frame Rate Synchronization: Uses playbackRate in calculations
- Developer 2: Implements the attribute and integrates with synchronization logic
### Maya API Specifics
- Use MFnNumericAttribute to create the attribute
- Set attribute properties (min, max, default, keyable, etc.)
- Handle attribute changes in compute method
- Use MFnDoubleData if storing as internal data (not needed for simple double attribute)
## Dependencies
- MPxNode implementation
- Frame rate synchronization logic
docs/Video_Frame_Caching_Spec.md
+220
View File
@@ -0,0 +1,220 @@
# Video Frame Caching Specification
## Overview
This document specifies the implementation details for video frame caching to enable smooth playback and random access to video frames.
## Requirements
- Cache decoded video frames to avoid re-decoding during playback
- Support random access to any frame in the video
- Implement intelligent cache replacement policy (LRU or similar)
- Handle memory limits to prevent excessive RAM usage
- Provide thread-safe access for concurrent reading and writing
- Support cache invalidation when video file changes
- Enable prefetching of future frames for smooth playback
- Support different cache policies (cache all, cache recent, cache keyframes only)
## Implementation Details
### Cache Design
#### FrameCache Class
Manages storage and retrieval of decoded video frames.
Key methods:
- `bool getFrame(int64_t frameIndex, AVFrame*& frame)` - Retrieve frame from cache
- `bool putFrame(int64_t frameIndex, const AVFrame* frame)` - Store frame in cache
- `void clear()` - Remove all frames from cache
- `bool contains(int64_t frameIndex) const` - Check if frame is cached
- `size_t size() const` - Number of frames currently cached
- `size_t memoryUsage() const` - Total memory used by cached frames
- `void setMaxMemorySize(size_t maxBytes)` - Set memory limit for cache
- `void setMaxFrameCount(size_t maxFrames)` - Set maximum number of frames to cache
#### Cache Entry
Each cache entry contains:
- Frame index (int64_t)
- AVFrame pointer (with proper reference counting)
- Timestamp of when frame was cached (for LRU)
- Access count (for LFU variants)
- Dirty flag (if frame has been modified by post-effects)
### Cache Algorithms
#### LRU (Least Recently Used)
- Remove least recently accessed frame when cache is full
- Implement using doubly-linked list + hash map for O(1) operations
- Update access time on both get and put operations
#### LFU (Least Frequently Used) - Optional
- Remove least frequently accessed frame when cache is full
- May provide better performance for certain access patterns
- More complex to implement than LRU
#### FIFO (First In, First Out) - Optional
- Remove oldest frame (by insertion time) when cache is full
- Simple to implement but may not optimal for playback patterns
### Implementation Components
#### Frame Data Management
- Proper reference counting for AVFrame objects (av_frame_ref/av_frame_unref)
- Deep copy of frame data when needed for post-processing
- Handle different pixel formats correctly
- Manage allocation and deallocation of frame buffers
#### Thread Safety
- Mutex protection for cache operations
- Read-write lock for better concurrency (multiple readers, single writer)
- Atomic operations for reference counting where possible
- Lock-free techniques for high-performance scenarios (if needed)
#### Cache Policies
- **Cache All**: Attempt to cache every decoded frame (memory permitting)
- **Cache Recent**: Cache only the most recently accessed frames (sliding window)
- **Cache Keyframes Only**: Cache only I-frames for faster seeking
- **Predictive Caching**: Prefetch frames ahead of current playback position
### Integration Points
- FFmpeg Decoder: Stores decoded frames in cache, retrieves from cache when available
- Maya Node: Requests frames from cache based on synchronization logic
- Post-effects: Can operate on cached frames or request uncached frames
- Viewport 2.0: Displays frames retrieved from cache
### Memory Management
- Calculate memory usage based on frame dimensions and pixel format
- Implement automatic eviction when memory limit exceeded
- Provide statistics on cache hit/miss ratios
- Allow configuration of memory limits via attributes or configuration file
### Error Handling
- Handle out-of-memory conditions gracefully
- Validate frame indices (negative, beyond video duration)
- Handle cache corruption (though unlikely in memory-only cache)
- Provide fallback to direct decoding when cache operations fail
### Performance Considerations
- Minimize lock contention in multithreaded scenarios
- Use efficient hash functions for frame index lookup
- Consider using memory pools for frame allocations
- Optimize for sequential access patterns (common in video playback)
- Implement asynchronous prefetching to hide latency
### Maya API Specifics
- Expose cache statistics as node attributes for debugging
- Provide attributes to control cache behavior (size limit, policy)
- Implement cache clearing when video file attribute changes
- Use MGlobal::displayInfo for cache statistics reporting
## Dependencies
- FFmpeg libraries (for AVFrame management)
- MPxNode implementation
- Standard C++ library (unordered_map, mutex, etc.)
## Overview
This document specifies the implementation details for video frame caching to enable smooth playback and random access to video frames.
## Requirements
- Cache decoded video frames to avoid re-decoding during playback
- Support random access to any frame in the video
- Implement intelligent cache replacement policy (LRU or similar)
- Handle memory limits to prevent excessive RAM usage
- Provide thread-safe access for concurrent reading and writing
- Support cache invalidation when video file changes
- Enable prefetching of future frames for smooth playback
- Support different cache policies (cache all, cache recent, cache keyframes only)
## Implementation Details
### Cache Design
#### FrameCache Class
Manages storage and retrieval of decoded video frames.
Key methods:
- `bool getFrame(int64_t frameIndex, AVFrame*& frame)` - Retrieve frame from cache
- `bool putFrame(int64_t frameIndex, const AVFrame* frame)` - Store frame in cache
- `void clear()` - Remove all frames from cache
- `bool contains(int64_t frameIndex) const` - Check if frame is cached
- `size_t size() const` - Number of frames currently cached
- `size_t memoryUsage() const` - Total memory used by cached frames
- `void setMaxMemorySize(size_t maxBytes)` - Set memory limit for cache
- `void setMaxFrameCount(size_t maxFrames)` - Set maximum number of frames to cache
#### Cache Entry
Each cache entry contains:
- Frame index (int64_t)
- AVFrame pointer (with proper reference counting)
- Timestamp of when frame was cached (for LRU)
- Access count (for LFU variants)
- Dirty flag (if frame has been modified by post-effects)
### Cache Algorithms
#### LRU (Least Recently Used)
- Remove least recently accessed frame when cache is full
- Implement using doubly-linked list + hash map for O(1) operations
- Update access time on both get and put operations
#### LFU (Least Frequently Used) - Optional
- Remove least frequently accessed frame when cache is full
- May provide better performance for certain access patterns
- More complex to implement than LRU
#### FIFO (First In, First Out) - Optional
- Remove oldest frame (by insertion time) when cache is full
- Simple to implement but may not optimal for playback patterns
### Implementation Components
#### Frame Data Management
- Proper reference counting for AVFrame objects (av_frame_ref/av_frame_unref)
- Deep copy of frame data when needed for post-processing
- Handle different pixel formats correctly
- Manage allocation and deallocation of frame buffers
#### Thread Safety
- Mutex protection for cache operations
- Read-write lock for better concurrency (multiple readers, single writer)
- Atomic operations for reference counting where possible
- Lock-free techniques for high-performance scenarios (if needed)
#### Cache Policies
- **Cache All**: Attempt to cache every decoded frame (memory permitting)
- **Cache Recent**: Cache only the most recently accessed frames (sliding window)
- **Cache Keyframes Only**: Cache only I-frames for faster seeking
- **Predictive Caching**: Prefetch frames ahead of current playback position
### Integration Points
- FFmpeg Decoder: Stores decoded frames in cache, retrieves from cache when available
- Maya Node: Requests frames from cache based on synchronization logic
- Post-effects: Can operate on cached frames or request uncached frames
- Viewport 2.0: Displays frames retrieved from cache
### Memory Management
- Calculate memory usage based on frame dimensions and pixel format
- Implement automatic eviction when memory limit exceeded
- Provide statistics on cache hit/miss ratios
- Allow configuration of memory limits via attributes or configuration file
### Error Handling
- Handle out-of-memory conditions gracefully
- Validate frame indices (negative, beyond video duration)
- Handle cache corruption (though unlikely in memory-only cache)
- Provide fallback to direct decoding when cache operations fail
### Performance Considerations
- Minimize lock contention in multithreaded scenarios
- Use efficient hash functions for frame index lookup
- Consider using memory pools for frame allocations
- Optimize for sequential access patterns (common in video playback)
- Implement asynchronous prefetching to hide latency
### Maya API Specifics
- Expose cache statistics as node attributes for debugging
- Provide attributes to control cache behavior (size limit, policy)
- Implement cache clearing when video file attribute changes
- Use MGlobal::displayInfo for cache statistics reporting
## Dependencies
- FFmpeg libraries (for AVFrame management)
- MPxNode implementation
- Standard C++ library (unordered_map, mutex, etc.)
docs/Viewport_2_0_Integration_Spec.md
+212
View File
@@ -0,0 +1,212 @@
# Viewport 2.0 Integration Specification
## Overview
This document specifies the implementation details for integrating video frame display into Maya's Viewport 2.0 using MHWRender::MPxDrawOverride.
## Requirements
- Display video frames in Viewport 2.0 as an image plane
- Support for RGB and RGBA frame formats
- Proper handling of frame rate and playback speed
- Integration with Maya's viewport rendering pipeline
- Support for post-effects (crop, resize, flip) applied to displayed frames
- Efficient texture updates to minimize performance impact
- Compatibility with Maya 2023's Viewport 2.0 API
## Implementation Details
### Core Components
#### VideoFrameDrawOverride Class
Derived from MHWRender::MPxDrawOverride, responsible for rendering video frames in the viewport.
Key methods:
- `bool isBounded(const MDagPath& objPath) const` - Returns whether the object has a bounding box
- `MBoundingBox boundingBox(const MDagPath& objPath) const` - Returns bounding box of the object
- `uint32_t drawFlags() const` - Returns draw flags for the override
- `void prepareForDraw(...)` - Prepares resources needed for drawing
- `void addUIDrawables(...)` - Adds UI drawables (if needed)
- `void hasUIDrawables() const` - Returns whether UI drawables are present
- `void draw(...)` - Main drawing method where video frame is rendered
#### Texture Management
- OpenGL texture object to hold current video frame
- Texture updates when new frame is available
- Proper texture format matching (GL_RGB, GL_RGBA, etc.)
- Efficient texture sub-data updates when possible
- Texture cleanup when node is destroyed
#### Frame Data Handling
- Receive frame data from Maya node via output attributes
- Convert frame data to appropriate OpenGL format if needed
- Update texture with new frame data
- Maintain frame timestamp for synchronization
### Integration Points
- Maya Node Integration: Connect to MPxNode's output attributes to receive frame data
- Frame Rate Synchronization: Use frame timestamp to synchronize with Maya's timeline
- Post-effects: Apply transformations to frame data before texture upload
- Caching: Interface with frame caching mechanism to get frames efficiently
### Rendering Approach
1. Receive new frame data from MPxNode (via attribute change notification)
2. Convert frame data to OpenGL-compatible format if necessary
3. Update OpenGL texture with new frame data
4. In viewport draw call:
- Set up appropriate shader program (simple texture shader)
- Bind texture
- Draw quad covering the image plane area
- Apply any viewport-specific transformations
### Shader Program
- Simple vertex and fragment shaders for texture rendering
- Vertex shader: Pass through texture coordinates
- Fragment shader: Sample texture and output color
- Optional: Support for color correction or other viewport-specific effects
### Texture Updates
- Use GL_PIXEL_UNPACK_BUFFER for efficient updates if supported
- Fallback to glTexSubImage2D for broader compatibility
- Handle different pixel formats (RGB, RGBA, BGRA, etc.)
- Flip Y-axis if necessary (OpenGL vs image coordinate systems)
### Bounding Box and Selection
- Return appropriate bounding box based on frame dimensions and aspect ratio
- Support for viewport selection of the image plane
- Handle transform nodes (position, rotation, scale) if applicable
### Threading Considerations
- Viewport drawing occurs on main thread
- Texture updates must happen on main thread (OpenGL context)
- Use thread-safe queue to transfer frame data from decoding thread to main thread
- Synchronize access to shared frame data
### Error Handling
- Check for OpenGL errors after texture operations
- Handle texture creation failures gracefully
- Provide fallback rendering (e.g., colored quad) when frame data unavailable
- Log errors to Maya's script editor using MGlobal::displayError
### Performance Considerations
- Minimize texture format conversions
- Update only changed portions of texture when possible
- Use appropriate texture filtering (GL_LINEAR for smooth playback)
- Avoid expensive operations during viewport drawing
- Consider using persistent mapped buffers for frequent updates
### Maya API Specifics
- Use MHWRender::MPxDrawOverride as base class
- Register draw override with MHWRender::MRenderer
- Use MImage class for image format conversions if needed
- Leverage MHWRender::MShaderManager for shader programs
- Follow Viewport 2.0 API guidelines for draw overrides
## Dependencies
- Maya Node implementation (for frame data)
- Frame caching mechanism
- OpenGL 3.2+ (required for Viewport 2.0)
- Maya API 2023 (MHWRender module)
## Overview
This document specifies the implementation details for integrating video frame display into Maya's Viewport 2.0 using MHWRender::MPxDrawOverride.
## Requirements
- Display video frames in Viewport 2.0 as an image plane
- Support for RGB and RGBA frame formats
- Proper handling of frame rate and playback speed
- Integration with Maya's viewport rendering pipeline
- Support for post-effects (crop, resize, flip) applied to displayed frames
- Efficient texture updates to minimize performance impact
- Compatibility with Maya 2023's Viewport 2.0 API
## Implementation Details
### Core Components
#### VideoFrameDrawOverride Class
Derived from MHWRender::MPxDrawOverride, responsible for rendering video frames in the viewport.
Key methods:
- `bool isBounded(const MDagPath& objPath) const` - Returns whether the object has a bounding box
- `MBoundingBox boundingBox(const MDagPath& objPath) const` - Returns bounding box of the object
- `uint32_t drawFlags() const` - Returns draw flags for the override
- `void prepareForDraw(...)` - Prepares resources needed for drawing
- `void addUIDrawables(...)` - Adds UI drawables (if needed)
- `void hasUIDrawables() const` - Returns whether UI drawables are present
- `void draw(...)` - Main drawing method where video frame is rendered
#### Texture Management
- OpenGL texture object to hold current video frame
- Texture updates when new frame is available
- Proper texture format matching (GL_RGB, GL_RGBA, etc.)
- Efficient texture sub-data updates when possible
- Texture cleanup when node is destroyed
#### Frame Data Handling
- Receive frame data from Maya node via output attributes
- Convert frame data to appropriate OpenGL format if needed
- Update texture with new frame data
- Maintain frame timestamp for synchronization
### Integration Points
- Maya Node Integration: Connect to MPxNode's output attributes to receive frame data
- Frame Rate Synchronization: Use frame timestamp to synchronize with Maya's timeline
- Post-effects: Apply transformations to frame data before texture upload
- Caching: Interface with frame caching mechanism to get frames efficiently
### Rendering Approach
1. Receive new frame data from MPxNode (via attribute change notification)
2. Convert frame data to OpenGL-compatible format if necessary
3. Update OpenGL texture with new frame data
4. In viewport draw call:
- Set up appropriate shader program (simple texture shader)
- Bind texture
- Draw quad covering the image plane area
- Apply any viewport-specific transformations
### Shader Program
- Simple vertex and fragment shaders for texture rendering
- Vertex shader: Pass through texture coordinates
- Fragment shader: Sample texture and output color
- Optional: Support for color correction or other viewport-specific effects
### Texture Updates
- Use GL_PIXEL_UNPACK_BUFFER for efficient updates if supported
- Fallback to glTexSubImage2D for broader compatibility
- Handle different pixel formats (RGB, RGBA, BGRA, etc.)
- Flip Y-axis if necessary (OpenGL vs image coordinate systems)
### Bounding Box and Selection
- Return appropriate bounding box based on frame dimensions and aspect ratio
- Support for viewport selection of the image plane
- Handle transform nodes (position, rotation, scale) if applicable
### Threading Considerations
- Viewport drawing occurs on main thread
- Texture updates must happen on main thread (OpenGL context)
- Use thread-safe queue to transfer frame data from decoding thread to main thread
- Synchronize access to shared frame data
### Error Handling
- Check for OpenGL errors after texture operations
- Handle texture creation failures gracefully
- Provide fallback rendering (e.g., colored quad) when frame data unavailable
- Log errors to Maya's script editor using MGlobal::displayError
### Performance Considerations
- Minimize texture format conversions
- Update only changed portions of texture when possible
- Use appropriate texture filtering (GL_LINEAR for smooth playback)
- Avoid expensive operations during viewport drawing
- Consider using persistent mapped buffers for frequent updates
### Maya API Specifics
- Use MHWRender::MPxDrawOverride as base class
- Register draw override with MHWRender::MRenderer
- Use MImage class for image format conversions if needed
- Leverage MHWRender::MShaderManager for shader programs
- Follow Viewport 2.0 API guidelines for draw overrides
## Dependencies
- Maya Node implementation (for frame data)
- Frame caching mechanism
- OpenGL 3.2+ (required for Viewport 2.0)
- Maya API 2023 (MHWRender module)