[GH-ISSUE #3609] [Feature] Video Reverse Engineering: Deconstruct rendered video artifacts into editing timelines, visual effects, motion tracking, and audio stems #1339

New issue

Open

opened 2026-10-16 21:04:28 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-16 21:04:28 +00:00

Owner

Originally created by @matbrgz on GitHub (Oct 14, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/3609

Summary

Rendered video files (.mp4, .mov, .webm, .mkv) are the flattened, compiled outputs of non-linear video editing (NLE) and compositing workflows. All original multi-track source layers—scene cuts, transitions, color grading, motion graphics, composite overlays, camera tracking, and mixed audio—are merged into a flat grid of pixels and a unified audio stream.

This feature proposal introduces Video Reverse Engineering capabilities to REA: decomposing a rendered video artifact back into its structural editing timeline, parameterizing visual effects and color grades, tracking camera and object motion, isolating audio stems, and exporting an interchangeable timeline (such as OpenTimelineIO or FCPXML).

Use Cases

Digital Media Forensics & Integrity: Detect frame splices, re-compression artifacts, localized tampering, deepfake face-swaps, and identify NLE software signatures (Premiere, DaVinci Resolve, Final Cut, FFmpeg).
VFX & Animation Reverse Engineering: Recover keyframe parameters, camera trajectory, color grading LUTs, and animated graphic overlays from reference video assets.
Timeline Reconstruction: Invert flattened videos back into editable multi-track timelines for analysis in NLEs (DaVinci Resolve, Premiere Pro, Blender).
UI/UX & Mobile App Interaction Analysis: Extract exact animation curves, transition timings, easing functions, and screen recording states from captured mobile or desktop app videos.

Proposed Architecture: Video Reverse Engineering Engine

                                    +-----------------------------------------+
                                    | Input: Video Artifact (.mp4, .mov, etc.)|
                                    +--------------------+--------------------+
                                                         |
                         +-------------------------------+-------------------------------+
                         |                                                               |
                         v                                                               v
           +---------------------------+                                   +---------------------------+
           | Bitstream & Container     |                                   | Audio Demixing & Event    |
           | Forensics (GOP, QP, atoms)|                                   | Analysis (Stems, ASR, BPM)|
           +-------------+-------------+                                   +-------------+-------------+
                         |                                                               |
                         +-------------------------------+-------------------------------+
                                                         |
                                                         v
                                     +---------------------------------------+
                                     | Shot Boundary & Transition Parser     |
                                     | (Hard cuts, dissolves, wipes, zooms)  |
                                     +-------------------+-------------------+
                                                         |
                                                         v
                                     +---------------------------------------+
                                     | VFX, Grade & Motion Inversion         |
                                     | (Color LUTs, optical flow, camera,    |
                                     |  lens distortion, motion graphics OCR)|
                                     +-------------------+-------------------+
                                                         |
                                                         v
                                     +---------------------------------------+
                                     | Timeline Synthesis & Export           |
                                     | (OpenTimelineIO .otio, FCPXML, EDL)   |
                                     +---------------------------------------+

1. Bitstream, Container & Codec Forensics

Container Atoms / Metadata: Parse ISO-BMFF (MP4, QuickTime MOV) and Matroska (MKV, WebM) atoms (moov, mvhd, trak, udta, XMP_). Identify encoder toolchains, creation dates, camera profiles, and device hardware tags.
Bitstream Structure (H.264, H.265/HEVC, AV1, VP9):
- Analyze GOP (Group of Pictures) cadence, I/P/B frame intervals, and closed vs. open GOP structures.
- Map Quantization Parameter (QP) distributions across frames to detect compression anomalies, re-encoding boundaries, or spliced visual elements.

2. Shot Boundary Detection & Transition Decomposition

Cut Point Detection: Frame-accurate identification of hard cuts with SMPTE timecodes (HH:MM:SS:FF) and frame numbers.
Gradual Transition Parameterization:
- Identify transition types: cross-dissolve, dip to black/white, additive dissolve, linear/directional wipes, and push/slide moves.
- Extract transition curves, start/end frames, and overlap duration.

3. Visual Effects (VFX), Color Grade & Camera Trajectory Inversion

Color Grading & 3D LUT Inversion:
- Estimate color space transfer characteristics (Rec.709, Rec.2020, Log curves).
- Approximate color adjustments (lift, gamma, gain, contrast, saturation) and synthesize a corrective 3D .cube LUT for each shot.
Camera Motion Estimation:
- Dense optical flow and feature point tracking to estimate 2D camera motion (pan, tilt, zoom, roll) and compute camera trajectory curves.
- Detect artificial camera shake vs. stabilized footage.
Optical & Lens Distortion Parameters:
- Estimate radial and tangential lens distortion coefficients (k_1, k_2).
- Detect synthetic post-processing effects: chromatic aberration, vignetting, bloom/glow, and film grain noise profiles.
Overlays, Motion Graphics & OCR:
- Extract bounding boxes, typography, and text content from lower-thirds, titles, and subtitles with frame-accurate keyframes.
- Alpha matte / foreground segmentation for picture-in-picture, logos, and graphic overlays.

4. Audio Stem Separation & Acoustic Event Analysis

Stem Demixing: Separate the composite audio track into 4 discrete channels:
1. Dialogue / Vocals: Isolated speech track.
2. Music: Soundtrack and background score.
3. Sound Effects (SFX): Foley, ambient impacts, and user interface sounds.
4. Ambient: Background atmospheric room tone/noise.
Speech Recognition (ASR): Extract automated transcription with word-level timecodes for subtitle track generation.
Beat & Tempo Detection: Measure BPM and beat grid alignment, correlating cut decisions with musical rhythms.

5. Timeline Synthesis & NLE Export

OpenTimelineIO (.otio) Generation: Assemble all recovered shots, transition durations, and separated audio stems into an OpenTimelineIO schema.
NLE Interchange Formats: Export .fcpxml (Final Cut Pro / DaVinci Resolve) and .edl (Edit Decision List) for direct import into professional video editing suites.

Safety, Resource Budgets & Error Handling

Inert Offline Processing: Analysis runs strictly offline using local toolchains (such as FFmpeg/FFprobe libraries or headless neural models); no media is uploaded externally.
Adaptive Resolution Downsampling & Decimation: Video frames can be processed at configurable proxy resolutions (e.g. 720p or keyframes only) to bound memory and CPU/GPU resource consumption.
Memory & Time Budgeting: Hard caps on analyzed video duration, track count, and export file sizes, emitting partial evidence if budget limits are reached.
Corrupted / Unsupported Media: Emit structured, typed diagnostics (corrupted_container, unsupported_codec, drm_protected) with byte-offset details.

Acceptance Criteria

Introduce analyze_video_artifact (CLI and MCP tool) accepting standard container formats (.mp4, .mov, .mkv, .webm).
Accurately identify hard cuts and gradual transitions with verifiable frame timecodes on synthetic test videos.
Output separated audio stem metadata and word-level speech transcripts.
Generate valid OpenTimelineIO (.otio) and FCPXML files that cleanly load recovered cuts and audio tracks in external video editors.
Enforce bounded resource limits and provide comprehensive test fixtures.

Originally created by @matbrgz on GitHub (Oct 14, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/3609 [![Type: Feature](https://img.shields.io/badge/type-feature-blue.svg?style=flat-square)](https://github.com/morluto/rea) [![Domain: Game & Engine RE](https://img.shields.io/badge/domain-game_&_engine_RE-green.svg?style=flat-square)](https://github.com/morluto/rea) [![Status: Proposed](https://img.shields.io/badge/status-proposed-blue.svg?style=flat-square)](https://github.com/morluto/rea) [![Ecosystem: REA](https://img.shields.io/badge/ecosystem-morluto%2Frea-8A2BE2.svg?style=flat-square)](https://github.com/morluto/rea) ## Summary Rendered video files (`.mp4`, `.mov`, `.webm`, `.mkv`) are the flattened, compiled outputs of non-linear video editing (NLE) and compositing workflows. All original multi-track source layers—scene cuts, transitions, color grading, motion graphics, composite overlays, camera tracking, and mixed audio—are merged into a flat grid of pixels and a unified audio stream. This feature proposal introduces **Video Reverse Engineering** capabilities to REA: decomposing a rendered video artifact back into its structural editing timeline, parameterizing visual effects and color grades, tracking camera and object motion, isolating audio stems, and exporting an interchangeable timeline (such as OpenTimelineIO or FCPXML). --- ## Use Cases - **Digital Media Forensics & Integrity:** Detect frame splices, re-compression artifacts, localized tampering, deepfake face-swaps, and identify NLE software signatures (Premiere, DaVinci Resolve, Final Cut, FFmpeg). - **VFX & Animation Reverse Engineering:** Recover keyframe parameters, camera trajectory, color grading LUTs, and animated graphic overlays from reference video assets. - **Timeline Reconstruction:** Invert flattened videos back into editable multi-track timelines for analysis in NLEs (DaVinci Resolve, Premiere Pro, Blender). - **UI/UX & Mobile App Interaction Analysis:** Extract exact animation curves, transition timings, easing functions, and screen recording states from captured mobile or desktop app videos. --- ## Proposed Architecture: Video Reverse Engineering Engine ``` +-----------------------------------------+ | Input: Video Artifact (.mp4, .mov, etc.)| +--------------------+--------------------+ | +-------------------------------+-------------------------------+ | | v v +---------------------------+ +---------------------------+ | Bitstream & Container | | Audio Demixing & Event | | Forensics (GOP, QP, atoms)| | Analysis (Stems, ASR, BPM)| +-------------+-------------+ +-------------+-------------+ | | +-------------------------------+-------------------------------+ | v +---------------------------------------+ | Shot Boundary & Transition Parser | | (Hard cuts, dissolves, wipes, zooms) | +-------------------+-------------------+ | v +---------------------------------------+ | VFX, Grade & Motion Inversion | | (Color LUTs, optical flow, camera, | | lens distortion, motion graphics OCR)| +-------------------+-------------------+ | v +---------------------------------------+ | Timeline Synthesis & Export | | (OpenTimelineIO .otio, FCPXML, EDL) | +---------------------------------------+ ``` ### 1. Bitstream, Container & Codec Forensics - **Container Atoms / Metadata:** Parse ISO-BMFF (MP4, QuickTime MOV) and Matroska (MKV, WebM) atoms (`moov`, `mvhd`, `trak`, `udta`, `XMP_`). Identify encoder toolchains, creation dates, camera profiles, and device hardware tags. - **Bitstream Structure (H.264, H.265/HEVC, AV1, VP9):** - Analyze GOP (Group of Pictures) cadence, I/P/B frame intervals, and closed vs. open GOP structures. - Map Quantization Parameter (QP) distributions across frames to detect compression anomalies, re-encoding boundaries, or spliced visual elements. ### 2. Shot Boundary Detection & Transition Decomposition - **Cut Point Detection:** Frame-accurate identification of hard cuts with SMPTE timecodes (`HH:MM:SS:FF`) and frame numbers. - **Gradual Transition Parameterization:** - Identify transition types: cross-dissolve, dip to black/white, additive dissolve, linear/directional wipes, and push/slide moves. - Extract transition curves, start/end frames, and overlap duration. ### 3. Visual Effects (VFX), Color Grade & Camera Trajectory Inversion - **Color Grading & 3D LUT Inversion:** - Estimate color space transfer characteristics (Rec.709, Rec.2020, Log curves). - Approximate color adjustments (lift, gamma, gain, contrast, saturation) and synthesize a corrective 3D `.cube` LUT for each shot. - **Camera Motion Estimation:** - Dense optical flow and feature point tracking to estimate 2D camera motion (pan, tilt, zoom, roll) and compute camera trajectory curves. - Detect artificial camera shake vs. stabilized footage. - **Optical & Lens Distortion Parameters:** - Estimate radial and tangential lens distortion coefficients ($k_1, k_2$). - Detect synthetic post-processing effects: chromatic aberration, vignetting, bloom/glow, and film grain noise profiles. - **Overlays, Motion Graphics & OCR:** - Extract bounding boxes, typography, and text content from lower-thirds, titles, and subtitles with frame-accurate keyframes. - Alpha matte / foreground segmentation for picture-in-picture, logos, and graphic overlays. ### 4. Audio Stem Separation & Acoustic Event Analysis - **Stem Demixing:** Separate the composite audio track into 4 discrete channels: 1. **Dialogue / Vocals:** Isolated speech track. 2. **Music:** Soundtrack and background score. 3. **Sound Effects (SFX):** Foley, ambient impacts, and user interface sounds. 4. **Ambient:** Background atmospheric room tone/noise. - **Speech Recognition (ASR):** Extract automated transcription with word-level timecodes for subtitle track generation. - **Beat & Tempo Detection:** Measure BPM and beat grid alignment, correlating cut decisions with musical rhythms. ### 5. Timeline Synthesis & NLE Export - **OpenTimelineIO (.otio) Generation:** Assemble all recovered shots, transition durations, and separated audio stems into an OpenTimelineIO schema. - **NLE Interchange Formats:** Export `.fcpxml` (Final Cut Pro / DaVinci Resolve) and `.edl` (Edit Decision List) for direct import into professional video editing suites. --- ## Safety, Resource Budgets & Error Handling - **Inert Offline Processing:** Analysis runs strictly offline using local toolchains (such as FFmpeg/FFprobe libraries or headless neural models); no media is uploaded externally. - **Adaptive Resolution Downsampling & Decimation:** Video frames can be processed at configurable proxy resolutions (e.g. 720p or keyframes only) to bound memory and CPU/GPU resource consumption. - **Memory & Time Budgeting:** Hard caps on analyzed video duration, track count, and export file sizes, emitting partial evidence if budget limits are reached. - **Corrupted / Unsupported Media:** Emit structured, typed diagnostics (`corrupted_container`, `unsupported_codec`, `drm_protected`) with byte-offset details. --- ## Acceptance Criteria 1. Introduce `analyze_video_artifact` (CLI and MCP tool) accepting standard container formats (`.mp4`, `.mov`, `.mkv`, `.webm`). 2. Accurately identify hard cuts and gradual transitions with verifiable frame timecodes on synthetic test videos. 3. Output separated audio stem metadata and word-level speech transcripts. 4. Generate valid OpenTimelineIO (`.otio`) and FCPXML files that cleanly load recovered cuts and audio tracks in external video editors. 5. Enforce bounded resource limits and provide comprehensive test fixtures.

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

morluto/rea#1339

No description provided.

Rows
Columns