MediaPlane/docs/Post_Effects_Spec.md

Post-Effects Specification

Overview

This document specifies the implementation details for post-effects processing on video frames, including crop, resize, and flip operations.

Requirements

• Implement crop functionality to select a region of interest from the video frame
• Implement resize functionality to scale the video frame to different dimensions
• Implement flip functionality to mirror the video frame horizontally and/or vertically
• Allow combining multiple post-effects in any order
• Maintain high performance for real-time playback
• Support various pixel formats (RGB, RGBA, etc.)
• Provide intuitive user controls for each effect

Implementation Details

Effect Definitions

Crop

• Parameters: x, y, width, height (in pixels)
• Defines rectangular region to extract from source frame
• Coordinates relative to top-left corner of source frame
• Width and height must be within source frame bounds
• If crop region extends beyond source frame, clamp to valid region

Resize

• Parameters: width, height (in pixels)
• Target dimensions for output frame
• Maintain aspect ratio option (to be determined)
• Use appropriate filtering algorithm (bilinear recommended for balance of quality/performance)

Flip

• Parameters: flipX (bool), flipY (bool)
• flipX: Mirror frame horizontally (left-right)
• flipY: Mirror frame vertically (top-bottom)
• Can be applied independently or together

Processing Order

The recommended order of operations is:

1. Crop (select region of interest)
2. Resize (scale to desired dimensions)
3. Flip (apply mirroring) This order ensures that flipping operates on the final composed image.

Implementation Components

FrameProcessor Class

Responsible for applying post-effects to decoded video frames.

Key methods:

• bool processFrame(const AVFrame* srcFrame, AVFrame* dstFrame) - Apply all enabled effects
• void setCropParameters(int x, int y, int width, int height) - Configure crop
• void setResizeParameters(int width, int height) - Configure resize
• void setFlipParameters(bool flipX, bool flipY) - Configure flip
• void enableCrop(bool enabled) - Enable/disable crop effect
• void enableResize(bool enabled) - Enable/disable resize effect
• void enableFlip(bool enabled) - Enable/disable flip effect
• bool isCropEnabled() const - Check if crop is enabled
• bool isResizeEnabled() const - Check if resize is enabled
• bool isFlipEnabled() const - Check if flip is enabled

Internal Processing Steps

1. Validate input frame
2. Apply crop if enabled:
   • Calculate source rectangle
   • Extract region from source frame
   • Handle format conversion if needed
3. Apply resize if enabled:
   • Use libswscale for scaling with appropriate filters
   • Allocate temporary frame if needed
4. Apply flip if enabled:
   • Perform in-place mirroring of frame data
   • Handle different pixel formats correctly
5. Output processed frame

Integration Points

• FFmpeg Decoder: Receives raw frames from decoder
• Maya Node: Applies post-effects before outputting frame data
• Viewport 2.0: Receives post-processed frames for display
• Caching: Option to cache frames before or after post-effects (to be determined)

Threading Considerations

• Post-processing should occur on the same thread as frame retrieval from decoder
• Can be done either in background decoding thread or main thread
• If done in background thread, ensure thread-safe delivery to main thread
• Minimize processing time to avoid blocking pipeline

Error Handling

• Validate effect parameters (non-negative dimensions, etc.)
• Handle memory allocation failures gracefully
• Provide fallback to unprocessed frame if processing fails
• Log errors to Maya's script editor using MGlobal::displayError

Performance Considerations

• Use hardware-accelerated operations where possible (though limited in CPU-based processing)
• Minimize memory allocations and copies
• Use efficient scaling algorithms (bilinear as good compromise)
• Consider processing only when parameters change
• Reuse buffers when possible to avoid reallocations

Pixel Format Support

• Primary support for RGB24 and RGBA32 formats
• Convert other formats to supported ones if needed
• Maintain format consistency through processing chain
• Handle format conversion with libswscale when necessary

Maya API Specifics

• Store effect parameters as node attributes
• Use MFnNumericAttribute for numeric parameters (x, y, width, height, flip bools)
• Use MFnBooleanAttribute for enable/disable toggles
• Implement attribute validation in node compute method
• Provide default values that result in no-op when effects disabled

Dependencies

• FFmpeg libraries (libswscale for scaling and format conversion)
• MPxNode implementation
• Frame data structures

Overview

This document specifies the implementation details for post-effects processing on video frames, including crop, resize, and flip operations.

Requirements

• Implement crop functionality to select a region of interest from the video frame
• Implement resize functionality to scale the video frame to different dimensions
• Implement flip functionality to mirror the video frame horizontally and/or vertically
• Allow combining multiple post-effects in any order
• Maintain high performance for real-time playback
• Support various pixel formats (RGB, RGBA, etc.)
• Provide intuitive user controls for each effect

Implementation Details

Effect Definitions

Crop

• Parameters: x, y, width, height (in pixels)
• Defines rectangular region to extract from source frame
• Coordinates relative to top-left corner of source frame
• Width and height must be within source frame bounds
• If crop region extends beyond source frame, clamp to valid region

Resize

• Parameters: width, height (in pixels)
• Target dimensions for output frame
• Maintain aspect ratio option (to be determined)
• Use appropriate filtering algorithm (bilinear recommended for balance of quality/performance)

Flip

• Parameters: flipX (bool), flipY (bool)
• flipX: Mirror frame horizontally (left-right)
• flipY: Mirror frame vertically (top-bottom)
• Can be applied independently or together

Processing Order

The recommended order of operations is:

1. Crop (select region of interest)
2. Resize (scale to desired dimensions)
3. Flip (apply mirroring) This order ensures that flipping operates on the final composed image.

Implementation Components

FrameProcessor Class

Responsible for applying post-effects to decoded video frames.

Key methods:

• bool processFrame(const AVFrame* srcFrame, AVFrame* dstFrame) - Apply all enabled effects
• void setCropParameters(int x, int y, int width, int height) - Configure crop
• void setResizeParameters(int width, int height) - Configure resize
• void setFlipParameters(bool flipX, bool flipY) - Configure flip
• void enableCrop(bool enabled) - Enable/disable crop effect
• void enableResize(bool enabled) - Enable/disable resize effect
• void enableFlip(bool enabled) - Enable/disable flip effect
• bool isCropEnabled() const - Check if crop is enabled
• bool isResizeEnabled() const - Check if resize is enabled
• bool isFlipEnabled() const - Check if flip is enabled

Internal Processing Steps

1. Validate input frame
2. Apply crop if enabled:
   • Calculate source rectangle
   • Extract region from source frame
   • Handle format conversion if needed
3. Apply resize if enabled:
   • Use libswscale for scaling with appropriate filters
   • Allocate temporary frame if needed
4. Apply flip if enabled:
   • Perform in-place mirroring of frame data
   • Handle different pixel formats correctly
5. Output processed frame

Integration Points

• FFmpeg Decoder: Receives raw frames from decoder
• Maya Node: Applies post-effects before outputting frame data
• Viewport 2.0: Receives post-processed frames for display
• Caching: Option to cache frames before or after post-effects (to be determined)

Threading Considerations

• Post-processing should occur on the same thread as frame retrieval from decoder
• Can be done either in background decoding thread or main thread
• If done in background thread, ensure thread-safe delivery to main thread
• Minimize processing time to avoid blocking pipeline

Error Handling

• Validate effect parameters (non-negative dimensions, etc.)
• Handle memory allocation failures gracefully
• Provide fallback to unprocessed frame if processing fails
• Log errors to Maya's script editor using MGlobal::displayError

Performance Considerations

• Use hardware-accelerated operations where possible (though limited in CPU-based processing)
• Minimize memory allocations and copies
• Use efficient scaling algorithms (bilinear as good compromise)
• Consider processing only when parameters change
• Reuse buffers when possible to avoid reallocations

Pixel Format Support

• Primary support for RGB24 and RGBA32 formats
• Convert other formats to supported ones if needed
• Maintain format consistency through processing chain
• Handle format conversion with libswscale when necessary

Maya API Specifics

• Store effect parameters as node attributes
• Use MFnNumericAttribute for numeric parameters (x, y, width, height, flip bools)
• Use MFnBooleanAttribute for enable/disable toggles
• Implement attribute validation in node compute method
• Provide default values that result in no-op when effects disabled

Dependencies

• FFmpeg libraries (libswscale for scaling and format conversion)
• MPxNode implementation
• Frame data structures