[GH-ISSUE #3691] [Feature] Chromium LevelDB reverse engineering: SSTables, WAL/MANIFEST history, IndexedDB keyspaces and deleted-state reconstruction #1403

New issue

Open

opened 2026-10-16 21:05:08 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-16 21:05:08 +00:00

Owner

Originally created by @matbrgz on GitHub (Oct 14, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/3691

Problem Statement & Context

LevelDB is Google's key-value storage library based on Log-Structured Merge-trees (LSM-trees). It serves as the standard local persistence backing store for Chromium-based browsers, Electron desktop applications, and embedded WebView environments—storing IndexedDB (*.indexeddb.leveldb), local storage, service worker registrations, and internal browser state.

While Chromium has initiated a multi-year migration to SQLite for IndexedDB, LevelDB remains the active default backing store across millions of deployed web and desktop applications. In incident response, security audits, and offline reverse engineering, examining LevelDB storage directly is critical. However, standard LevelDB APIs only expose active, current keys via db->Get(), obscuring:

LSM-Tree Metadata & History (MANIFEST-*, CURRENT): Version edits, added/deleted SSTables across compaction levels (0 through 6), and sequence numbers.
Write-Ahead Logs (*.log): Uncommitted or recently written records not yet flushed to SSTable blocks.
Internal Key Structure & Sequence Numbers: 8-byte trailer packing sequence numbers and value types (TypeDeletion / tombstones vs TypeValue).
Deleted State & Historical Versions: Old record versions retained in uncompacted SSTables or slack space before minor/major compactions.
Chromium IndexedDB Keyspace Encoding: Chromium serializes database IDs, object store IDs, index IDs, and key types using variable-length integer and byte prefix encodings (e.g. kDatabaseNameType, kObjectStoreMetadataType, kIndexDataValueType).

REA requires a native LevelDB parser and forensic analyzer capable of parsing raw LevelDB directories without invoking C++ native LevelDB repair binaries, reconstructing both current and deleted records, and decoding Chromium IndexedDB schemas.

Proposed Architecture

flowchart TD
    A["LevelDB Directory (*.indexeddb.leveldb / LevelDB store)"] --> B["Chromium LevelDB Forensics & RE Engine"]
    B --> C["Manifest & LSM-Tree Version Reconstructor"]
    B --> D["SSTable & WAL Low-Level Block Parser"]
    B --> E["Chromium IndexedDB Schema Decoder"]
    
    C --> C1["CURRENT & MANIFEST-xxxxxx Parser"]
    C --> C2["VersionEdit History & Compaction Level Hierarchy (L0-L6)"]
    
    D --> D1["Write-Ahead Log (*.log) Record Slicing"]
    D --> D2["SSTable (*.ldb, *.sst) Data Block & Filter Block Parsing"]
    D --> D3["Internal Keys: Sequence Numbers & Tombstone Recovery"]
    
    E --> E1["IndexedDB Key Prefix & Varint Decoding"]
    E --> E2["Object Store Schema & Record Deserialization (V8 Serialized Script Value)"]
    
    C & D & E --> F["Unified Storage State Manifest (Active, Deleted, Historical Records)"]

Technical Capabilities & Modules

LSM-Tree Hierarchy & Version History Recovery:
- Parse CURRENT pointer and sequence of MANIFEST-* files containing VersionEdit records.
- Reconstruct the active SSTable tree per level (Level 0 through Level 6) and historical compaction events.
- Track chronological sequence number progression across records.
Low-Level Block & WAL Parsing:
- Parse LevelDB Write-Ahead Log (*.log) chunk types: FULL, FIRST, MIDDLE, LAST.
- Parse Table format (.ldb / .sst): Trailer, Footer (metaindex handle, index handle), Index Blocks, and Data Blocks with Snappy decompression support.
- Read Bloom filter blocks to verify key presence without full table scans.
Internal Key Decoding & Deleted Record Recovery:
- Dissect 8-byte internal key tag into 56-bit sequence number and 8-bit value type.
- Detect TypeDeletion (tombstones) and retrieve overwritten or soft-deleted historical record values still present in higher-level SSTables prior to compaction.
Chromium IndexedDB Keyspace De-serialization:
- Decode Chromium IndexedDB binary key encodings:
  - kDatabaseNameType: Database names per origin.
  - kObjectStoreMetadataType: Object store names, key paths, auto-increment flags.
  - kIndexMetadataType: Secondary index definitions and uniqueness constraints.
  - kIndexDataValueType / kObjectStoreDataValueType: Serialized record data.
- Parse V8 SerializedScriptValue payloads storing JavaScript objects and arrays.
Integrity & Offline Export:
- Generate a structured JSON/SQLite export of the entire database state.
- Flag corrupted blocks or truncated WAL segments.

Acceptance Criteria

Static parsing of LevelDB directories without relying on native LevelDB C++ shared libraries.
Successful extraction of active key-value pairs from .ldb, .sst, and .log files.
Identification and recovery of deleted or superseded historical records using sequence numbers.
Complete decoding of Chromium IndexedDB metadata (databases, object stores, indexes).
Deserialization of V8 structured clone values into standard JSON representations.

Complements #3621 (Unknown binary file-format inference)
Complements #117 (Full-stack reconstruction)

Originally created by @matbrgz on GitHub (Oct 14, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/3691 [![Type: Feature](https://img.shields.io/badge/type-feature-blue.svg?style=flat-square)](https://github.com/morluto/rea) [![Domain: Game & Engine RE](https://img.shields.io/badge/domain-game_&_engine_RE-green.svg?style=flat-square)](https://github.com/morluto/rea) [![Status: Proposed](https://img.shields.io/badge/status-proposed-blue.svg?style=flat-square)](https://github.com/morluto/rea) [![Ecosystem: REA](https://img.shields.io/badge/ecosystem-morluto%2Frea-8A2BE2.svg?style=flat-square)](https://github.com/morluto/rea) ### Problem Statement & Context LevelDB is Google's key-value storage library based on Log-Structured Merge-trees (LSM-trees). It serves as the standard local persistence backing store for Chromium-based browsers, Electron desktop applications, and embedded WebView environments—storing IndexedDB (`*.indexeddb.leveldb`), local storage, service worker registrations, and internal browser state. While Chromium has initiated a multi-year migration to SQLite for IndexedDB, LevelDB remains the active default backing store across millions of deployed web and desktop applications. In incident response, security audits, and offline reverse engineering, examining LevelDB storage directly is critical. However, standard LevelDB APIs only expose active, current keys via `db->Get()`, obscuring: 1. **LSM-Tree Metadata & History (`MANIFEST-*`, `CURRENT`)**: Version edits, added/deleted SSTables across compaction levels (0 through 6), and sequence numbers. 2. **Write-Ahead Logs (`*.log`)**: Uncommitted or recently written records not yet flushed to SSTable blocks. 3. **Internal Key Structure & Sequence Numbers**: 8-byte trailer packing sequence numbers and value types (`TypeDeletion` / tombstones vs `TypeValue`). 4. **Deleted State & Historical Versions**: Old record versions retained in uncompacted SSTables or slack space before minor/major compactions. 5. **Chromium IndexedDB Keyspace Encoding**: Chromium serializes database IDs, object store IDs, index IDs, and key types using variable-length integer and byte prefix encodings (e.g. `kDatabaseNameType`, `kObjectStoreMetadataType`, `kIndexDataValueType`). REA requires a native LevelDB parser and forensic analyzer capable of parsing raw LevelDB directories without invoking C++ native LevelDB repair binaries, reconstructing both current and deleted records, and decoding Chromium IndexedDB schemas. ### Proposed Architecture ```mermaid flowchart TD A["LevelDB Directory (*.indexeddb.leveldb / LevelDB store)"] --> B["Chromium LevelDB Forensics & RE Engine"] B --> C["Manifest & LSM-Tree Version Reconstructor"] B --> D["SSTable & WAL Low-Level Block Parser"] B --> E["Chromium IndexedDB Schema Decoder"] C --> C1["CURRENT & MANIFEST-xxxxxx Parser"] C --> C2["VersionEdit History & Compaction Level Hierarchy (L0-L6)"] D --> D1["Write-Ahead Log (*.log) Record Slicing"] D --> D2["SSTable (*.ldb, *.sst) Data Block & Filter Block Parsing"] D --> D3["Internal Keys: Sequence Numbers & Tombstone Recovery"] E --> E1["IndexedDB Key Prefix & Varint Decoding"] E --> E2["Object Store Schema & Record Deserialization (V8 Serialized Script Value)"] C & D & E --> F["Unified Storage State Manifest (Active, Deleted, Historical Records)"] ``` ### Technical Capabilities & Modules 1. **LSM-Tree Hierarchy & Version History Recovery:** - Parse `CURRENT` pointer and sequence of `MANIFEST-*` files containing `VersionEdit` records. - Reconstruct the active SSTable tree per level (Level 0 through Level 6) and historical compaction events. - Track chronological sequence number progression across records. 2. **Low-Level Block & WAL Parsing:** - Parse LevelDB Write-Ahead Log (`*.log`) chunk types: `FULL`, `FIRST`, `MIDDLE`, `LAST`. - Parse Table format (`.ldb` / `.sst`): Trailer, Footer (metaindex handle, index handle), Index Blocks, and Data Blocks with Snappy decompression support. - Read Bloom filter blocks to verify key presence without full table scans. 3. **Internal Key Decoding & Deleted Record Recovery:** - Dissect 8-byte internal key tag into 56-bit sequence number and 8-bit value type. - Detect `TypeDeletion` (tombstones) and retrieve overwritten or soft-deleted historical record values still present in higher-level SSTables prior to compaction. 4. **Chromium IndexedDB Keyspace De-serialization:** - Decode Chromium IndexedDB binary key encodings: - `kDatabaseNameType`: Database names per origin. - `kObjectStoreMetadataType`: Object store names, key paths, auto-increment flags. - `kIndexMetadataType`: Secondary index definitions and uniqueness constraints. - `kIndexDataValueType` / `kObjectStoreDataValueType`: Serialized record data. - Parse V8 `SerializedScriptValue` payloads storing JavaScript objects and arrays. 5. **Integrity & Offline Export:** - Generate a structured JSON/SQLite export of the entire database state. - Flag corrupted blocks or truncated WAL segments. ### Acceptance Criteria - [ ] Static parsing of LevelDB directories without relying on native LevelDB C++ shared libraries. - [ ] Successful extraction of active key-value pairs from `.ldb`, `.sst`, and `.log` files. - [ ] Identification and recovery of deleted or superseded historical records using sequence numbers. - [ ] Complete decoding of Chromium IndexedDB metadata (databases, object stores, indexes). - [ ] Deserialization of V8 structured clone values into standard JSON representations. ### Related Issues & Dependencies - Complements #3621 (Unknown binary file-format inference) - Complements #117 (Full-stack reconstruction)

bsakaguchi referenced this issue

2026-10-17 21:11:06 +00:00

[PR #1403] [MERGED] feat: add Hermes as a supported MCP client #2859

bsakaguchi referenced this issue

2026-10-17 21:11:58 +00:00

[PR #1511] [MERGED] test(clients): verify native Qwen Pi and Hermes workflows #2931

bsakaguchi referenced this issue

2026-10-31 00:55:08 +00:00

[PR #1754] [MERGED] fix(setup): preserve unrelated Hermes YAML line formatting #4795