LinkDesk/.kiro/specs/shot-cascading-deletion/design.md

# Shot Soft Deletion Design

## Overview

This design document outlines the implementation of comprehensive soft deletion for shots in the VFX Project Management System. The solution marks shots and all related data as deleted without removing them from the database, ensuring data preservation for audit and recovery purposes while hiding deleted content from normal operations.

The design follows a single-phase approach: immediate database updates within a transaction to mark all related records as deleted. Physical files are preserved on the file system for potential recovery. This ensures data consistency, maintains audit trails, and provides recovery capabilities.

## Architecture

### High-Level Flow

```mermaid
sequenceDiagram
    participant UI as Frontend UI
    participant API as Backend API
    participant DB as Database

    UI->>API: GET /shots/{id}/deletion-info
    API->>DB: Query shot, tasks, submissions, attachments (non-deleted only)
    DB-->>API: Return deletion summary
    API-->>UI: Deletion info with counts and affected users

    UI->>UI: Show confirmation dialog
    UI->>API: DELETE /shots/{id} (soft delete)
    
    API->>DB: Begin transaction
    API->>DB: UPDATE shot SET deleted_at = NOW(), deleted_by = user_id
    API->>DB: UPDATE tasks SET deleted_at = NOW(), deleted_by = user_id WHERE shot_id = ?
    API->>DB: UPDATE submissions SET deleted_at = NOW(), deleted_by = user_id WHERE task_id IN (...)
    API->>DB: UPDATE attachments SET deleted_at = NOW(), deleted_by = user_id WHERE task_id IN (...)
    API->>DB: UPDATE production_notes SET deleted_at = NOW(), deleted_by = user_id WHERE task_id IN (...)
    API->>DB: UPDATE reviews SET deleted_at = NOW(), deleted_by = user_id WHERE submission_id IN (...)
    API->>DB: INSERT INTO activities (type='shot_deleted', ...)
    API->>DB: Commit transaction
    
    API-->>UI: Success response with summary
    UI->>UI: Remove shot from UI immediately
```

### Component Architecture

```mermaid
graph TB
    subgraph "Frontend Components"
        SDC[ShotDeleteConfirmDialog]
        DIS[DeletionInfoService]
        SS[ShotService]
    end
    
    subgraph "Backend Services"
        SDS[ShotSoftDeletionService]
        AS[ActivityService]
        RS[RecoveryService]
    end
    
    subgraph "Data Layer"
        DB[(Database)]
    end
    
    SDC --> DIS
    DIS --> SS
    SS --> SDS
    SDS --> AS
    SDS --> RS
    SDS --> DB
    AS --> DB
    RS --> DB
```

## Components and Interfaces

### Backend Components

#### ShotSoftDeletionService

**Purpose**: Orchestrates the complete shot soft deletion process including database updates and audit logging.

**Key Methods**:
- `get_deletion_info(shot_id: int, db: Session) -> DeletionInfo`
- `soft_delete_shot_cascade(shot_id: int, db: Session, current_user: User) -> DeletionResult`
- `mark_related_data_deleted(shot_id: int, db: Session, current_user: User, deleted_at: datetime)`

#### RecoveryService

**Purpose**: Handles recovery of soft-deleted shots and related data.

**Key Methods**:
- `get_deleted_shots(project_id: int, db: Session) -> List[DeletedShot]`
- `recover_shot(shot_id: int, db: Session, current_user: User) -> RecoveryResult`
- `preview_recovery(shot_id: int, db: Session) -> RecoveryInfo`

#### ActivityService (Enhanced)

**Purpose**: Manages activity logging for deletion and recovery operations.

**Key Methods**:
- `log_shot_soft_deletion(shot: Shot, user: User, deletion_info: DeletionInfo)`
- `log_shot_recovery(shot: Shot, user: User, recovery_info: RecoveryInfo)`
- `get_activities_including_deleted(filters: ActivityFilters) -> List[Activity]`

### Frontend Components

#### ShotDeleteConfirmDialog

**Purpose**: Provides comprehensive deletion confirmation with impact summary.

**Props**:
- `shot: Shot` - The shot to be deleted
- `isOpen: boolean` - Dialog visibility state
- `onConfirm: (shotId: number) => void` - Deletion confirmation callback
- `onCancel: () => void` - Cancellation callback

**Features**:
- Displays deletion impact summary (task count, file count, affected users)
- Shows loading state during deletion
- Provides clear cancel option
- Displays success/error messages

#### DeletionInfoService

**Purpose**: Fetches and formats shot deletion impact information.

**Key Methods**:
- `getDeletionInfo(shotId: number): Promise<DeletionInfo>`
- `formatDeletionSummary(info: DeletionInfo): string`
- `getAffectedUsers(info: DeletionInfo): User[]`

## Data Models

### Database Schema Changes

All relevant tables need to be updated with soft deletion fields:

```sql
-- Add soft deletion columns to all relevant tables
ALTER TABLE shots ADD COLUMN deleted_at TIMESTAMP NULL;
ALTER TABLE shots ADD COLUMN deleted_by INTEGER NULL REFERENCES users(id);

ALTER TABLE tasks ADD COLUMN deleted_at TIMESTAMP NULL;
ALTER TABLE tasks ADD COLUMN deleted_by INTEGER NULL REFERENCES users(id);

ALTER TABLE submissions ADD COLUMN deleted_at TIMESTAMP NULL;
ALTER TABLE submissions ADD COLUMN deleted_by INTEGER NULL REFERENCES users(id);

ALTER TABLE task_attachments ADD COLUMN deleted_at TIMESTAMP NULL;
ALTER TABLE task_attachments ADD COLUMN deleted_by INTEGER NULL REFERENCES users(id);

ALTER TABLE production_notes ADD COLUMN deleted_at TIMESTAMP NULL;
ALTER TABLE production_notes ADD COLUMN deleted_by INTEGER NULL REFERENCES users(id);

ALTER TABLE reviews ADD COLUMN deleted_at TIMESTAMP NULL;
ALTER TABLE reviews ADD COLUMN deleted_by INTEGER NULL REFERENCES users(id);

-- Add indexes for efficient querying of non-deleted records
CREATE INDEX idx_shots_not_deleted ON shots (id) WHERE deleted_at IS NULL;
CREATE INDEX idx_tasks_not_deleted ON tasks (shot_id) WHERE deleted_at IS NULL;
CREATE INDEX idx_submissions_not_deleted ON submissions (task_id) WHERE deleted_at IS NULL;
CREATE INDEX idx_attachments_not_deleted ON task_attachments (task_id) WHERE deleted_at IS NULL;
CREATE INDEX idx_notes_not_deleted ON production_notes (task_id) WHERE deleted_at IS NULL;
CREATE INDEX idx_reviews_not_deleted ON reviews (submission_id) WHERE deleted_at IS NULL;
```

### DeletionInfo Schema

```typescript
interface DeletionInfo {
  shot_id: number
  shot_name: string
  episode_name: string
  project_name: string
  
  // Counts of items that will be marked as deleted
  task_count: number
  submission_count: number
  attachment_count: number
  note_count: number
  review_count: number
  
  // File information (preserved, not deleted)
  total_file_size: number
  file_count: number
  
  // Affected users
  affected_users: AffectedUser[]
  
  // Timestamps
  last_activity_date?: string
  created_at: string
}

interface AffectedUser {
  id: number
  name: string
  email: string
  role: string
  task_count: number
  submission_count: number
  note_count: number
  last_activity_date?: string
}
```

### DeletionResult Schema

```typescript
interface DeletionResult {
  success: boolean
  shot_id: number
  shot_name: string
  
  // Database update results
  marked_deleted_tasks: number
  marked_deleted_submissions: number
  marked_deleted_attachments: number
  marked_deleted_notes: number
  marked_deleted_reviews: number
  
  // Timing
  operation_duration: number
  deleted_at: string
  deleted_by: number
  
  // Errors
  errors: string[]
  warnings: string[]
}
```

### RecoveryInfo Schema

```typescript
interface RecoveryInfo {
  shot_id: number
  shot_name: string
  episode_name: string
  project_name: string
  
  // Counts of items that will be recovered
  task_count: number
  submission_count: number
  attachment_count: number
  note_count: number
  review_count: number
  
  // Deletion information
  deleted_at: string
  deleted_by: number
  deleted_by_name: string
  
  // File status
  files_preserved: boolean
  file_count: number
}

interface RecoveryResult {
  success: boolean
  shot_id: number
  shot_name: string
  
  // Recovery results
  recovered_tasks: number
  recovered_submissions: number
  recovered_attachments: number
  recovered_notes: number
  recovered_reviews: number
  
  // Timing
  operation_duration: number
  recovered_at: string
  recovered_by: number
  
  // Errors
  errors: string[]
  warnings: string[]
}
```
```

## Correctness Properties

*A property is a characteristic or behavior that should hold true across all valid executions of a system-essentially, a formal statement about what the system should do. Properties serve as the bridge between human-readable specifications and machine-verifiable correctness guarantees.*

<function_calls>
<invoke name="prework">
<parameter name="featureName">shot-cascading-deletion

### Property 1: Complete task cascade soft deletion
*For any* shot with associated tasks, soft deleting the shot should result in all tasks being marked as deleted with the same timestamp
**Validates: Requirements 1.2**

### Property 2: Complete submission cascade soft deletion  
*For any* set of tasks with associated submissions, marking tasks as deleted should result in all submissions being marked as deleted
**Validates: Requirements 1.3**

### Property 3: Complete production notes cascade soft deletion
*For any* set of tasks with associated production notes, marking tasks as deleted should result in all production notes being marked as deleted
**Validates: Requirements 1.4**

### Property 4: Complete attachment cascade soft deletion
*For any* set of tasks with associated attachments, marking tasks as deleted should result in all attachments being marked as deleted
**Validates: Requirements 1.5**

### Property 5: Complete review cascade soft deletion
*For any* set of submissions with associated reviews, marking submissions as deleted should result in all reviews being marked as deleted
**Validates: Requirements 1.6**

### Property 6: Shot query exclusion
*For any* query for shots, results should exclude all shots where deleted_at is not null
**Validates: Requirements 2.1**

### Property 7: Task query exclusion
*For any* query for tasks, results should exclude all tasks where deleted_at is not null
**Validates: Requirements 2.2**

### Property 8: Submission query exclusion
*For any* query for submissions, results should exclude all submissions where deleted_at is not null
**Validates: Requirements 2.3**

### Property 9: Attachment query exclusion
*For any* query for attachments, results should exclude all attachments where deleted_at is not null
**Validates: Requirements 2.4**

### Property 10: Production notes query exclusion
*For any* query for production notes, results should exclude all notes where deleted_at is not null
**Validates: Requirements 2.5**

### Property 11: Deletion count accuracy
*For any* shot deletion info request, the returned counts should exactly match the actual number of records that would be marked as deleted
**Validates: Requirements 3.2, 3.3, 3.4, 3.5**

### Property 12: Affected user identification
*For any* shot with tasks assigned to users, the deletion info should include all users who have assigned tasks, submissions, or notes
**Validates: Requirements 4.1, 4.2, 4.3**

### Property 13: Activity date calculation
*For any* affected user, the most recent activity date should be the latest timestamp among their tasks, submissions, and notes for that shot
**Validates: Requirements 4.4**

### Property 14: Activity query exclusion
*For any* activity feed query, results should exclude activities related to deleted shots, tasks, and submissions
**Validates: Requirements 5.2**

### Property 15: Deletion audit logging
*For any* successful shot soft deletion, a new activity record should be created documenting the deletion with shot name, timestamp, and user
**Validates: Requirements 5.3, 5.4**

### Property 16: Transaction atomicity
*For any* shot soft deletion, either all database updates succeed and are committed, or all changes are rolled back on any failure
**Validates: Requirements 6.1, 6.2, 6.4**

### Property 17: Audit trail completeness
*For any* shot deletion operation, all significant events should be logged with complete context information including user and timestamp
**Validates: Requirements 8.1, 8.2, 8.3, 8.5**

### Property 18: Recovery completeness
*For any* shot recovery operation, all related records that were marked as deleted should be restored to active status
**Validates: Requirements 11.3**

### Property 19: Recovery audit logging
*For any* successful shot recovery, a new activity record should be created documenting the recovery with user and timestamp information
**Validates: Requirements 11.4**

### Property 20: Data preservation
*For any* soft deleted shot, all original data including files should remain unchanged and accessible for recovery
**Validates: Requirements 11.1**

## Error Handling

### Database Error Handling

1. **Transaction Rollback**: Any database error during soft deletion triggers complete rollback
2. **Constraint Violations**: Database constraints are handled gracefully with clear error messages
3. **Concurrent Access**: Database locks prevent concurrent deletion attempts on the same shot
4. **Connection Failures**: Database connection issues are retried with exponential backoff
5. **Already Deleted**: Attempts to delete already deleted shots return appropriate error messages

### Data Consistency Error Handling

1. **Missing Related Records**: Missing related records are handled gracefully without failing the operation
2. **Orphaned Records**: Orphaned records are identified and handled appropriately
3. **Timestamp Consistency**: All related records receive the same deletion timestamp
4. **User Reference Integrity**: Deleted_by references are validated before updates

### Recovery Error Handling

1. **Already Active**: Attempts to recover already active shots return appropriate error messages
2. **Missing Dependencies**: Recovery validates that parent records (episode, project) are still active
3. **Partial Recovery Failures**: Failed recovery of individual records doesn't prevent recovery of others
4. **User Permission Validation**: Recovery operations validate user permissions before proceeding

## Testing Strategy

### Unit Testing Approach

Unit tests will focus on individual components and their specific responsibilities:

- **ShotDeletionService**: Test deletion logic, error handling, and transaction management
- **FileCleanuService**: Test file operations, batch processing, and error recovery
- **ActivityService**: Test logging functionality and audit trail creation
- **Frontend Components**: Test UI behavior, confirmation dialogs, and user interactions

### Property-Based Testing Approach

Property-based tests will verify the universal properties across all valid inputs using **Hypothesis** for Python backend testing and **fast-check** for TypeScript frontend testing. Each property-based test will run a minimum of 100 iterations to ensure comprehensive coverage.

**Backend Property Tests** (using Hypothesis):
- Generate random shots with varying numbers of tasks, submissions, and attachments
- Test cascading soft deletion properties across different data configurations
- Verify query exclusion properties with various deleted/active data combinations
- Test recovery properties with different deletion scenarios
- Test error handling properties with simulated database failures

**Frontend Property Tests** (using fast-check):
- Generate random deletion info objects and verify UI calculations
- Test dialog behavior with various user interaction patterns
- Verify state management across different component configurations
- Test recovery UI with various deleted shot configurations

**Property Test Tagging**: Each property-based test will include a comment with the format:
`# Feature: shot-soft-deletion, Property {number}: {property_text}`

### Integration Testing

Integration tests will verify the complete deletion workflow:

- End-to-end deletion scenarios with real database and file system operations
- Cross-component communication and data flow validation
- Error propagation and recovery across system boundaries
- Performance testing with large datasets

### Test Data Management

- **Database Fixtures**: Standardized test data sets with known relationships
- **File System Mocking**: Controlled file system environments for testing
- **Error Simulation**: Configurable failure injection for error path testing
- **Performance Datasets**: Large-scale test data for performance validation

## Implementation Notes

### Database Considerations

1. **Cascade Configuration**: Leverage existing SQLAlchemy cascade relationships where possible
2. **Index Optimization**: Ensure proper indexing on foreign key relationships for efficient deletion
3. **Batch Operations**: Use bulk delete operations for large datasets
4. **Connection Pooling**: Manage database connections efficiently during long operations

### File System Considerations

1. **Path Validation**: Validate all file paths before attempting deletion
2. **Atomic Operations**: Use atomic file operations where possible
3. **Cleanup Ordering**: Delete files before directories to avoid permission issues
4. **Storage Monitoring**: Monitor disk space during cleanup operations

### Performance Considerations

1. **Lazy Loading**: Avoid loading unnecessary data during deletion operations
2. **Batch Processing**: Process files and records in configurable batch sizes
3. **Background Tasks**: Use background task queues for file cleanup operations
4. **Progress Tracking**: Provide progress feedback for long-running operations

### Security Considerations

1. **Authorization**: Verify user permissions before allowing deletion
2. **Path Traversal**: Prevent directory traversal attacks in file paths
3. **Audit Logging**: Log all deletion attempts for security auditing
4. **Data Sanitization**: Ensure complete data removal for sensitive information
## Implementation Notes

### Database Considerations

1. **Soft Delete Columns**: Add deleted_at and deleted_by columns to all relevant tables
2. **Index Optimization**: Create partial indexes on non-deleted records for efficient querying
3. **Query Modification**: Update all existing queries to exclude deleted records by default
4. **Migration Strategy**: Implement database migrations to add soft delete columns safely

### Query Pattern Updates

1. **Default Filtering**: All model queries should include `WHERE deleted_at IS NULL` by default
2. **Admin Queries**: Provide special query methods for administrators to include deleted records
3. **Recovery Queries**: Implement queries to find and recover deleted records
4. **Performance Optimization**: Use database indexes to ensure efficient filtering of deleted records

### Performance Considerations

1. **Index Strategy**: Create partial indexes on active records to maintain query performance
2. **Batch Updates**: Use bulk update operations for marking large numbers of records as deleted
3. **Query Optimization**: Ensure all queries efficiently exclude deleted records
4. **Memory Management**: Process large datasets in batches to avoid memory issues

### Security Considerations

1. **Authorization**: Verify user permissions before allowing deletion or recovery
2. **Audit Logging**: Log all deletion and recovery attempts for security auditing
3. **Data Access**: Ensure deleted data is only accessible to authorized administrators
4. **Recovery Permissions**: Implement strict permissions for data recovery operations

### Migration Strategy

1. **Schema Updates**: Add soft delete columns to existing tables without downtime
2. **Data Integrity**: Ensure existing data remains unaffected during migration
3. **Query Updates**: Gradually update application queries to use soft delete filtering
4. **Rollback Plan**: Provide rollback procedures in case of migration issues