MediaPlane/docs/FFmpeg_Integration_Spec.md

FFmpeg Integration Specification

Overview

This document specifies the implementation details for integrating FFmpeg into the Maya Image Plane Node plugin for video frame decoding.

Requirements

• Decode MP4 video files using FFmpeg libraries
• Extract video frames as raw pixel data (RGB or RGBA format)
• Provide frame-by-frame access for synchronization with Maya's timeline
• Handle various video codecs commonly used in MP4 containers
• Error handling for corrupted or unsupported video files

Implementation Details

Library Linking

• Link against FFmpeg libraries: libavcodec, libavformat, libavutil, libswscale
• Use pre-built FFmpeg binaries or build from source as needed
• Ensure compatibility with Maya 2023's compiler toolchain (Visual Studio 2017)

Core Components

VideoDecoder Class

Responsible for initializing FFmpeg, opening video files, and decoding frames.

Key methods:

• bool open(const std::string& filename) - Opens video file and initializes FFmpeg context
• void close() - Releases FFmpeg resources
• bool readFrame(AVFrame* frame) - Decodes next video frame into provided AVFrame
• int getWidth() const - Returns video width
• int getHeight() const - Returns video height
• double getFrameRate() const - Returns video frame rate
• int64_t getFrameCount() const - Returns total number of frames
• int64_t getCurrentFrame() const - Returns current frame position

Pixel Format Conversion

• Convert decoded frames to RGB or RGBA format using libswscale
• Store converted frames in CPU memory for further processing
• Provide access to raw pixel data for viewport rendering

Threading Considerations

• FFmpeg decoding should occur on a separate thread to avoid blocking Maya's main thread
• Use thread-safe queue for frame delivery to main thread
• Implement proper synchronization mechanisms (mutexes, condition variables)

Error Handling

• Check return values from all FFmpeg functions
• Provide meaningful error messages for common issues:
  • File not found
  • Unsupported codec
  • Corrupted file
  • Memory allocation failures
• Graceful degradation when FFmpeg is unavailable

Integration with Maya Node

• VideoDecoder instance managed by the MPxNode implementation
• Frame requests triggered by Maya's time changes
• Caching mechanism to avoid re-decoding frames

Configuration

• FFmpeg library paths configurable via CMake
• Option to use system-installed FFmpeg or bundled binaries
• Build-time option to enable/disable FFmpeg integration (for testing)

Testing

• Unit tests for VideoDecoder class
• Integration tests with sample MP4 files
• Performance testing for frame decoding speed
• Memory leak detection

Dependencies

• FFmpeg 4.0 or later (tested with 4.4)
• libavcodec, libavformat, libavutil, libswscale

Overview

This document specifies the implementation details for integrating FFmpeg into the Maya Image Plane Node plugin for video frame decoding.

Requirements

• Decode MP4 video files using FFmpeg libraries
• Extract video frames as raw pixel data (RGB or RGBA format)
• Provide frame-by-frame access for synchronization with Maya's timeline
• Handle various video codecs commonly used in MP4 containers
• Error handling for corrupted or unsupported video files

Implementation Details

Library Linking

• Link against FFmpeg libraries: libavcodec, libavformat, libavutil, libswscale
• Use pre-built FFmpeg binaries or build from source as needed
• Ensure compatibility with Maya 2023's compiler toolchain (Visual Studio 2017)

Core Components

VideoDecoder Class

Responsible for initializing FFmpeg, opening video files, and decoding frames.

Key methods:

• bool open(const std::string& filename) - Opens video file and initializes FFmpeg context
• void close() - Releases FFmpeg resources
• bool readFrame(AVFrame* frame) - Decodes next video frame into provided AVFrame
• int getWidth() const - Returns video width
• int getHeight() const - Returns video height
• double getFrameRate() const - Returns video frame rate
• int64_t getFrameCount() const - Returns total number of frames
• int64_t getCurrentFrame() const - Returns current frame position

Pixel Format Conversion

• Convert decoded frames to RGB or RGBA format using libswscale
• Store converted frames in CPU memory for further processing
• Provide access to raw pixel data for viewport rendering

Threading Considerations

• FFmpeg decoding should occur on a separate thread to avoid blocking Maya's main thread
• Use thread-safe queue for frame delivery to main thread
• Implement proper synchronization mechanisms (mutexes, condition variables)

Error Handling

• Check return values from all FFmpeg functions
• Provide meaningful error messages for common issues:
  • File not found
  • Unsupported codec
  • Corrupted file
  • Memory allocation failures
• Graceful degradation when FFmpeg is unavailable

Integration with Maya Node

• VideoDecoder instance managed by the MPxNode implementation
• Frame requests triggered by Maya's time changes
• Caching mechanism to avoid re-decoding frames

Configuration

• FFmpeg library paths configurable via CMake
• Option to use system-installed FFmpeg or bundled binaries
• Build-time option to enable/disable FFmpeg integration (for testing)

Testing

• Unit tests for VideoDecoder class
• Integration tests with sample MP4 files
• Performance testing for frame decoding speed
• Memory leak detection

Dependencies

• FFmpeg 4.0 or later (tested with 4.4)
• libavcodec, libavformat, libavutil, libswscale