MediaPlane/docs/Video_Frame_Caching_Spec.md

Video Frame Caching Specification

Overview

This document specifies the implementation details for video frame caching to enable smooth playback and random access to video frames.

Requirements

• Cache decoded video frames to avoid re-decoding during playback
• Support random access to any frame in the video
• Implement intelligent cache replacement policy (LRU or similar)
• Handle memory limits to prevent excessive RAM usage
• Provide thread-safe access for concurrent reading and writing
• Support cache invalidation when video file changes
• Enable prefetching of future frames for smooth playback
• Support different cache policies (cache all, cache recent, cache keyframes only)

Implementation Details

Cache Design

FrameCache Class

Manages storage and retrieval of decoded video frames.

Key methods:

• bool getFrame(int64_t frameIndex, AVFrame*& frame) - Retrieve frame from cache
• bool putFrame(int64_t frameIndex, const AVFrame* frame) - Store frame in cache
• void clear() - Remove all frames from cache
• bool contains(int64_t frameIndex) const - Check if frame is cached
• size_t size() const - Number of frames currently cached
• size_t memoryUsage() const - Total memory used by cached frames
• void setMaxMemorySize(size_t maxBytes) - Set memory limit for cache
• void setMaxFrameCount(size_t maxFrames) - Set maximum number of frames to cache

Cache Entry

Each cache entry contains:

• Frame index (int64_t)
• AVFrame pointer (with proper reference counting)
• Timestamp of when frame was cached (for LRU)
• Access count (for LFU variants)
• Dirty flag (if frame has been modified by post-effects)

Cache Algorithms

LRU (Least Recently Used)

• Remove least recently accessed frame when cache is full
• Implement using doubly-linked list + hash map for O(1) operations
• Update access time on both get and put operations

LFU (Least Frequently Used) - Optional

• Remove least frequently accessed frame when cache is full
• May provide better performance for certain access patterns
• More complex to implement than LRU

FIFO (First In, First Out) - Optional

• Remove oldest frame (by insertion time) when cache is full
• Simple to implement but may not optimal for playback patterns

Implementation Components

Frame Data Management

• Proper reference counting for AVFrame objects (av_frame_ref/av_frame_unref)
• Deep copy of frame data when needed for post-processing
• Handle different pixel formats correctly
• Manage allocation and deallocation of frame buffers

Thread Safety

• Mutex protection for cache operations
• Read-write lock for better concurrency (multiple readers, single writer)
• Atomic operations for reference counting where possible
• Lock-free techniques for high-performance scenarios (if needed)

Cache Policies

• Cache All: Attempt to cache every decoded frame (memory permitting)
• Cache Recent: Cache only the most recently accessed frames (sliding window)
• Cache Keyframes Only: Cache only I-frames for faster seeking
• Predictive Caching: Prefetch frames ahead of current playback position

Integration Points

• FFmpeg Decoder: Stores decoded frames in cache, retrieves from cache when available
• Maya Node: Requests frames from cache based on synchronization logic
• Post-effects: Can operate on cached frames or request uncached frames
• Viewport 2.0: Displays frames retrieved from cache

Memory Management

• Calculate memory usage based on frame dimensions and pixel format
• Implement automatic eviction when memory limit exceeded
• Provide statistics on cache hit/miss ratios
• Allow configuration of memory limits via attributes or configuration file

Error Handling

• Handle out-of-memory conditions gracefully
• Validate frame indices (negative, beyond video duration)
• Handle cache corruption (though unlikely in memory-only cache)
• Provide fallback to direct decoding when cache operations fail

Performance Considerations

• Minimize lock contention in multithreaded scenarios
• Use efficient hash functions for frame index lookup
• Consider using memory pools for frame allocations
• Optimize for sequential access patterns (common in video playback)
• Implement asynchronous prefetching to hide latency

Maya API Specifics

• Expose cache statistics as node attributes for debugging
• Provide attributes to control cache behavior (size limit, policy)
• Implement cache clearing when video file attribute changes
• Use MGlobal::displayInfo for cache statistics reporting

Dependencies

• FFmpeg libraries (for AVFrame management)
• MPxNode implementation
• Standard C++ library (unordered_map, mutex, etc.)

Overview

This document specifies the implementation details for video frame caching to enable smooth playback and random access to video frames.

Requirements

• Cache decoded video frames to avoid re-decoding during playback
• Support random access to any frame in the video
• Implement intelligent cache replacement policy (LRU or similar)
• Handle memory limits to prevent excessive RAM usage
• Provide thread-safe access for concurrent reading and writing
• Support cache invalidation when video file changes
• Enable prefetching of future frames for smooth playback
• Support different cache policies (cache all, cache recent, cache keyframes only)

Implementation Details

Cache Design

FrameCache Class

Manages storage and retrieval of decoded video frames.

Key methods:

• bool getFrame(int64_t frameIndex, AVFrame*& frame) - Retrieve frame from cache
• bool putFrame(int64_t frameIndex, const AVFrame* frame) - Store frame in cache
• void clear() - Remove all frames from cache
• bool contains(int64_t frameIndex) const - Check if frame is cached
• size_t size() const - Number of frames currently cached
• size_t memoryUsage() const - Total memory used by cached frames
• void setMaxMemorySize(size_t maxBytes) - Set memory limit for cache
• void setMaxFrameCount(size_t maxFrames) - Set maximum number of frames to cache

Cache Entry

Each cache entry contains:

• Frame index (int64_t)
• AVFrame pointer (with proper reference counting)
• Timestamp of when frame was cached (for LRU)
• Access count (for LFU variants)
• Dirty flag (if frame has been modified by post-effects)

Cache Algorithms

LRU (Least Recently Used)

• Remove least recently accessed frame when cache is full
• Implement using doubly-linked list + hash map for O(1) operations
• Update access time on both get and put operations

LFU (Least Frequently Used) - Optional

• Remove least frequently accessed frame when cache is full
• May provide better performance for certain access patterns
• More complex to implement than LRU

FIFO (First In, First Out) - Optional

• Remove oldest frame (by insertion time) when cache is full
• Simple to implement but may not optimal for playback patterns

Implementation Components

Frame Data Management

• Proper reference counting for AVFrame objects (av_frame_ref/av_frame_unref)
• Deep copy of frame data when needed for post-processing
• Handle different pixel formats correctly
• Manage allocation and deallocation of frame buffers

Thread Safety

• Mutex protection for cache operations
• Read-write lock for better concurrency (multiple readers, single writer)
• Atomic operations for reference counting where possible
• Lock-free techniques for high-performance scenarios (if needed)

Cache Policies

• Cache All: Attempt to cache every decoded frame (memory permitting)
• Cache Recent: Cache only the most recently accessed frames (sliding window)
• Cache Keyframes Only: Cache only I-frames for faster seeking
• Predictive Caching: Prefetch frames ahead of current playback position

Integration Points

• FFmpeg Decoder: Stores decoded frames in cache, retrieves from cache when available
• Maya Node: Requests frames from cache based on synchronization logic
• Post-effects: Can operate on cached frames or request uncached frames
• Viewport 2.0: Displays frames retrieved from cache

Memory Management

• Calculate memory usage based on frame dimensions and pixel format
• Implement automatic eviction when memory limit exceeded
• Provide statistics on cache hit/miss ratios
• Allow configuration of memory limits via attributes or configuration file

Error Handling

• Handle out-of-memory conditions gracefully
• Validate frame indices (negative, beyond video duration)
• Handle cache corruption (though unlikely in memory-only cache)
• Provide fallback to direct decoding when cache operations fail

Performance Considerations

• Minimize lock contention in multithreaded scenarios
• Use efficient hash functions for frame index lookup
• Consider using memory pools for frame allocations
• Optimize for sequential access patterns (common in video playback)
• Implement asynchronous prefetching to hide latency

Maya API Specifics

• Expose cache statistics as node attributes for debugging
• Provide attributes to control cache behavior (size limit, policy)
• Implement cache clearing when video file attribute changes
• Use MGlobal::displayInfo for cache statistics reporting

Dependencies

• FFmpeg libraries (for AVFrame management)
• MPxNode implementation
• Standard C++ library (unordered_map, mutex, etc.)