morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 07:47:29 +00:00

[GH-ISSUE #3611] [Feature] Music and Sound Effects Reverse Engineering: Deconstruct audio into multitrack stems, polyphonic MIDI, synth patches, and DSP effect chains #1341

New issue

Open

opened 2026-10-16 21:04:28 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-16 21:04:28 +00:00

Owner

Originally created by @matbrgz on GitHub (Oct 14, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/3611

Summary

Rendered audio files (.wav, .flac, .mp3, .ogg, .opus) and compiled game audio banks (FMOD .bank/.fsb, Wwise .bnk/.pck) represent the flattened, compiled outputs of music production, sound design, and acoustic engineering. The constituent multitrack elements—individual instrument layers, musical notes/scores, synthesis parameters (oscillators, filters, ADSR envelopes), and DSP effect chains (EQ, reverb, delay, distortion)—are collapsed into stereo or multichannel waveforms.

This feature proposal introduces Music and Sound Effects Reverse Engineering to REA: decomposing mixed audio into isolated instrument stems, transcribing polyphonic audio into MIDI, parameterizing synthesizer patches and SFX synthesis recipes, estimating room acoustics and DSP effect chains, and unpacking proprietary game audio banks.

Use Cases

Acoustic & Game Audio Forensics: Unpack compiled game audio banks (FMOD, Wwise, Unity, Unreal) to extract cues, loop markers, sound event hierarchies, and raw sample streams without running the game engine.
Music Transcription & Analysis: Recover polyphonic MIDI scores, chord progressions, tempo maps (BPM), and musical keys from rendered music tracks.
Sound Design & Synth Patch Inversion: Reverse engineer how a synthesizer lead, bass patch, or sound effect (impact, riser, laser, explosion) was synthesized, estimating oscillator types, ADSR envelope timings, filter cutoff/resonance, and LFO modulation.
DSP Effect Chain Recovery: Estimate reverb room impulse responses (RT_{60}), delay feedback timings, compression ratios, and harmonic distortion profiles from recorded audio.

Proposed Architecture: Audio Reverse Engineering Engine

                                    +-----------------------------------------+
                                    | Input: Audio File (.wav, .mp3, .ogg) or |
                                    |        Game Bank (FMOD .bank, Wwise .bnk)|
                                    +--------------------+--------------------+
                                                         |
                         +-------------------------------+-------------------------------+
                         |                                                               |
                         v                                                               v
           +---------------------------+                                   +---------------------------+
           | Game Audio Bank Unpacker  |                                   | Multitrack Source         |
           | (FMOD .bank/.fsb, Wwise)  |                                   | Separation (Stems/Demix)  |
           +-------------+-------------+                                   +-------------+-------------+
                         |                                                               |
                         +-------------------------------+-------------------------------+
                                                         |
                                                         v
                                     +---------------------------------------+
                                     | Polyphonic Note & Music Transcription |
                                     | (MIDI, Chords, Key, Tempo Map BPM)    |
                                     +-------------------+-------------------+
                                                         |
                                                         v
                                     +---------------------------------------+
                                     | Sound Design & Synth Patch Inversion  |
                                     | (Oscillators, ADSR, Filters, SFX Type)|
                                     +-------------------+-------------------+
                                                         |
                                                         v
                                     +---------------------------------------+
                                     | DSP Effect Chain Parameter Recovery   |
                                     | (Reverb RT60, Delay, Modulation, Sat) |
                                     +-------------------+-------------------+
                                                         |
                                                         v
                                     +---------------------------------------+
                                     | Structured Evidence & DAW Interchange |
                                     | (MIDI .mid, Stem files, JSON recipe)  |
                                     +---------------------------------------+

1. Game Audio Bank Unpacking

FMOD Studio & FSB5: Parse FMOD .bank and .fsb (FSB5) containers. Extract audio streams (Vorbis, PCM, FADPCM), loop point markers, event metadata, and sound tables.
Audiokinetic Wwise: Parse Wwise SoundBank (.bnk) and package (.pck) files, mapping Bank IDs, Event IDs, and embedded sound objects.
Engine Containers: Support extraction of Unity AudioClip serialized assets and Unreal Engine .uasset SoundWave audio streams.

2. Source Separation & Multitrack Demixing

Instrument Demixing: Separate composite stereo tracks into discrete, high-fidelity stems:
- Vocals
- Drums & Percussion
- Bass
- Guitars / Acoustic Instruments
- Synthesizers, Keys & Pads
- Sound Effects / Foley
Transient / Harmonic Decomposition: Deconstruct audio signals into transient (percussive strike/attack) and stationary harmonic (sustained tonal content) components via spectral phase/magnitude separation.

3. Polyphonic Music Transcription & Harmony Extraction

Polyphonic Audio-to-MIDI: Transcribe melodic and polyphonic harmonic parts into standard MIDI (.mid) files, preserving pitch, microtonal pitch bends, onset velocities, and exact note durations.
Chord & Harmony Recognition: Detect chord sequences, tonal root, scale mode (Major, Minor, Dorian, etc.), and functional harmonic progression.
Tempo & Beat Grid Analysis: Frame-accurate BPM calculation, dynamic tempo map recovery (accelerando/ritardando), and beat grid alignment (downbeat and bar markers).

4. Synthesizer Patch & Sound Effect (SFX) Inversion

Oscillator & Waveform Estimation: Perform spectral harmonic analysis (STFT) to estimate oscillator types (sine, sawtooth, square, pulse width, triangle, or FM operator ratios).
Envelope Parameterization (ADSR): Measure amplitude envelope slopes to recover Attack, Decay, Sustain, and Release milliseconds.
Filter Inversion: Detect filter topology (low-pass, high-pass, band-pass), cutoff frequency trajectory (f_c), resonance (Q), and filter envelope modulation depth.
SFX Categorization & Synthesis Recipe: Classify sound effects into structured ontologies (Impacts, Whooshes/Risers, Lasers, Explosions, Mechanical Clicks, Foley) and generate reproducible synthesis recipes (JSON patch definitions, Pure Data, or Faust DSP code).

5. DSP Effect Chain Parameter Recovery

Acoustic Reverb Estimation: Estimate room impulse response (IR) via blind deconvolution, calculating reverberation decay time (RT_{60}), early reflection delays, and dry/wet ratio.
Delay & Echo: Detect discrete delay repeats, tempo-synced subdivisions (e.g. 1/8 dotted, 1/4 note), feedback percentage, and ping-pong stereo widening.
Modulation Effects: Measure LFO frequency rate (Hz) and depth for chorus, flanger, phaser, and tremolo.
Nonlinear Distortion & Saturation: Analyze harmonic distortion spectra to identify even vs. odd harmonic content (e.g., tube saturation vs. hard-clipping fuzz).

Safety, Offline Boundaries & Resource Budgets

Inert Offline Execution: All DSP and machine learning inference runs completely offline and locally without sending audio bytes to cloud APIs.
Resource Safeguards: Enforce strict time and sample-count bounds on audio files, support downsampled proxy analysis for long files, and set strict memory limits during demixing and spectrogram generation.
Unsupported Formats: Emit typed, actionable diagnostics (corrupted_audio, unsupported_sample_format, encrypted_bank) with byte offsets and header inspection details.

Acceptance Criteria

Introduce analyze_audio_artifact in REA CLI and MCP contracts.
Extract embedded audio streams and loop points from synthetic FMOD .bank and Wwise .bnk test fixtures.
Transcribe monophonic and polyphonic synthetic test files into valid MIDI (.mid) files matching ground truth note events.
Parameterize ADSR envelopes and filter cutoff frequencies on synthetic synthesizer audio patches.
Export structured JSON evidence documenting BPM, key, chord progressions, and recovered DSP effect parameters.

Originally created by @matbrgz on GitHub (Oct 14, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/3611 [![Type: Feature](https://img.shields.io/badge/type-feature-blue.svg?style=flat-square)](https://github.com/morluto/rea) [![Domain: Game & Engine RE](https://img.shields.io/badge/domain-game_&_engine_RE-green.svg?style=flat-square)](https://github.com/morluto/rea) [![Status: Proposed](https://img.shields.io/badge/status-proposed-blue.svg?style=flat-square)](https://github.com/morluto/rea) [![Ecosystem: REA](https://img.shields.io/badge/ecosystem-morluto%2Frea-8A2BE2.svg?style=flat-square)](https://github.com/morluto/rea) ## Summary Rendered audio files (`.wav`, `.flac`, `.mp3`, `.ogg`, `.opus`) and compiled game audio banks (FMOD `.bank`/`.fsb`, Wwise `.bnk`/`.pck`) represent the flattened, compiled outputs of music production, sound design, and acoustic engineering. The constituent multitrack elements—individual instrument layers, musical notes/scores, synthesis parameters (oscillators, filters, ADSR envelopes), and DSP effect chains (EQ, reverb, delay, distortion)—are collapsed into stereo or multichannel waveforms. This feature proposal introduces **Music and Sound Effects Reverse Engineering** to REA: decomposing mixed audio into isolated instrument stems, transcribing polyphonic audio into MIDI, parameterizing synthesizer patches and SFX synthesis recipes, estimating room acoustics and DSP effect chains, and unpacking proprietary game audio banks. --- ## Use Cases - **Acoustic & Game Audio Forensics:** Unpack compiled game audio banks (FMOD, Wwise, Unity, Unreal) to extract cues, loop markers, sound event hierarchies, and raw sample streams without running the game engine. - **Music Transcription & Analysis:** Recover polyphonic MIDI scores, chord progressions, tempo maps (BPM), and musical keys from rendered music tracks. - **Sound Design & Synth Patch Inversion:** Reverse engineer how a synthesizer lead, bass patch, or sound effect (impact, riser, laser, explosion) was synthesized, estimating oscillator types, ADSR envelope timings, filter cutoff/resonance, and LFO modulation. - **DSP Effect Chain Recovery:** Estimate reverb room impulse responses ($RT_{60}$), delay feedback timings, compression ratios, and harmonic distortion profiles from recorded audio. --- ## Proposed Architecture: Audio Reverse Engineering Engine ``` +-----------------------------------------+ | Input: Audio File (.wav, .mp3, .ogg) or | | Game Bank (FMOD .bank, Wwise .bnk)| +--------------------+--------------------+ | +-------------------------------+-------------------------------+ | | v v +---------------------------+ +---------------------------+ | Game Audio Bank Unpacker | | Multitrack Source | | (FMOD .bank/.fsb, Wwise) | | Separation (Stems/Demix) | +-------------+-------------+ +-------------+-------------+ | | +-------------------------------+-------------------------------+ | v +---------------------------------------+ | Polyphonic Note & Music Transcription | | (MIDI, Chords, Key, Tempo Map BPM) | +-------------------+-------------------+ | v +---------------------------------------+ | Sound Design & Synth Patch Inversion | | (Oscillators, ADSR, Filters, SFX Type)| +-------------------+-------------------+ | v +---------------------------------------+ | DSP Effect Chain Parameter Recovery | | (Reverb RT60, Delay, Modulation, Sat) | +-------------------+-------------------+ | v +---------------------------------------+ | Structured Evidence & DAW Interchange | | (MIDI .mid, Stem files, JSON recipe) | +---------------------------------------+ ``` ### 1. Game Audio Bank Unpacking - **FMOD Studio & FSB5:** Parse FMOD `.bank` and `.fsb` (FSB5) containers. Extract audio streams (Vorbis, PCM, FADPCM), loop point markers, event metadata, and sound tables. - **Audiokinetic Wwise:** Parse Wwise SoundBank (`.bnk`) and package (`.pck`) files, mapping Bank IDs, Event IDs, and embedded sound objects. - **Engine Containers:** Support extraction of Unity `AudioClip` serialized assets and Unreal Engine `.uasset` SoundWave audio streams. ### 2. Source Separation & Multitrack Demixing - **Instrument Demixing:** Separate composite stereo tracks into discrete, high-fidelity stems: - Vocals - Drums & Percussion - Bass - Guitars / Acoustic Instruments - Synthesizers, Keys & Pads - Sound Effects / Foley - **Transient / Harmonic Decomposition:** Deconstruct audio signals into transient (percussive strike/attack) and stationary harmonic (sustained tonal content) components via spectral phase/magnitude separation. ### 3. Polyphonic Music Transcription & Harmony Extraction - **Polyphonic Audio-to-MIDI:** Transcribe melodic and polyphonic harmonic parts into standard **MIDI (`.mid`)** files, preserving pitch, microtonal pitch bends, onset velocities, and exact note durations. - **Chord & Harmony Recognition:** Detect chord sequences, tonal root, scale mode (Major, Minor, Dorian, etc.), and functional harmonic progression. - **Tempo & Beat Grid Analysis:** Frame-accurate BPM calculation, dynamic tempo map recovery (accelerando/ritardando), and beat grid alignment (downbeat and bar markers). ### 4. Synthesizer Patch & Sound Effect (SFX) Inversion - **Oscillator & Waveform Estimation:** Perform spectral harmonic analysis (STFT) to estimate oscillator types (sine, sawtooth, square, pulse width, triangle, or FM operator ratios). - **Envelope Parameterization (ADSR):** Measure amplitude envelope slopes to recover Attack, Decay, Sustain, and Release milliseconds. - **Filter Inversion:** Detect filter topology (low-pass, high-pass, band-pass), cutoff frequency trajectory ($f_c$), resonance ($Q$), and filter envelope modulation depth. - **SFX Categorization & Synthesis Recipe:** Classify sound effects into structured ontologies (Impacts, Whooshes/Risers, Lasers, Explosions, Mechanical Clicks, Foley) and generate reproducible synthesis recipes (JSON patch definitions, Pure Data, or Faust DSP code). ### 5. DSP Effect Chain Parameter Recovery - **Acoustic Reverb Estimation:** Estimate room impulse response (IR) via blind deconvolution, calculating reverberation decay time ($RT_{60}$), early reflection delays, and dry/wet ratio. - **Delay & Echo:** Detect discrete delay repeats, tempo-synced subdivisions (e.g. 1/8 dotted, 1/4 note), feedback percentage, and ping-pong stereo widening. - **Modulation Effects:** Measure LFO frequency rate (Hz) and depth for chorus, flanger, phaser, and tremolo. - **Nonlinear Distortion & Saturation:** Analyze harmonic distortion spectra to identify even vs. odd harmonic content (e.g., tube saturation vs. hard-clipping fuzz). --- ## Safety, Offline Boundaries & Resource Budgets - **Inert Offline Execution:** All DSP and machine learning inference runs completely offline and locally without sending audio bytes to cloud APIs. - **Resource Safeguards:** Enforce strict time and sample-count bounds on audio files, support downsampled proxy analysis for long files, and set strict memory limits during demixing and spectrogram generation. - **Unsupported Formats:** Emit typed, actionable diagnostics (`corrupted_audio`, `unsupported_sample_format`, `encrypted_bank`) with byte offsets and header inspection details. --- ## Acceptance Criteria 1. Introduce `analyze_audio_artifact` in REA CLI and MCP contracts. 2. Extract embedded audio streams and loop points from synthetic FMOD `.bank` and Wwise `.bnk` test fixtures. 3. Transcribe monophonic and polyphonic synthetic test files into valid MIDI (`.mid`) files matching ground truth note events. 4. Parameterize ADSR envelopes and filter cutoff frequencies on synthetic synthesizer audio patches. 5. Export structured JSON evidence documenting BPM, key, chord progressions, and recovered DSP effect parameters.

bsakaguchi referenced this issue

2026-10-17 21:10:30 +00:00

[PR #1341] [MERGED] chore(main): release rea-agents 6.3.0 #2809

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

morluto/rea#1341

No description provided.

Rows
Columns