[GH-ISSUE #3702] [Feature] Anvil world container and NBT reverse engineering: .mca chunk regions, level.dat, compound tags, datapacks and modloader metadata graphs #1412

New issue

Open

opened 2026-10-16 21:05:13 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-16 21:05:13 +00:00

Owner

Originally created by @matbrgz on GitHub (Oct 14, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/3702

Problem Statement & Context

Voxel-based sandbox game worlds store massive, procedurally generated multi-dimensional terrains, player inventories, entity data, and custom logic across structured binary file formats:

Named Binary Tag (NBT) Protocol: A hierarchical binary serialization format composed of typed payload tags: TAG_End (0), TAG_Byte (1), TAG_Short (2), TAG_Int (3), TAG_Long (4), TAG_Float (5), TAG_Double (6), TAG_Byte_Array (7), TAG_String (8), TAG_List (9), TAG_Compound (10), TAG_Int_Array (11), TAG_Long_Array (12).
Anvil Region Containers (.mca): Regional terrain files storing 32x32 (1024) chunks. Each .mca file begins with an 8 KB header consisting of:
- Chunk location table: 1024 4-byte entries (3 bytes sector offset, 1 byte sector count).
- Chunk timestamp table: 1024 4-byte integer modification timestamps.
- Sector-aligned chunk data compressed via Zlib (type 2) or LZ4 (type 3).
World Configuration & Registry State (level.dat): Gzip-compressed NBT compound storing world seed, spawn coordinates, gamerules, data version, and custom dimension registries.
Data Packs: Zip or directory archives containing JSON-defined recipes, advancements, loot tables, world generation biomes, and procedural command functions (.mcfunction).
Modloader Ecosystems & Dependency Manifests: Metadata descriptors (fabric.mod.json, mods.toml, quilt.mod.json, plugin.yml) declaring mod ID, entrypoints, mixin configurations, access wideners, dependencies, and listener events.

REA needs a dedicated static analyzer capable of parsing NBT trees, extracting Anvil chunks without game execution, and modeling datapack/modloader dependency graphs.

Proposed Architecture

flowchart TD
    A["World Directory (.mca, level.dat) / Datapack / Mod Archive"] --> B["Voxel World & NBT Semantic Engine"]
    B --> C["NBT Deserializer (Gzip / Zlib / Uncompressed)"]
    B --> D["Anvil Region (.mca) Slicer & Chunk Extractor"]
    B --> E["Datapack & Modloader Metadata Reconstructor"]
    
    C --> C1["TAG_Compound & Recursive Tag Parser"]
    C --> C2["level.dat & Player / Entity State Deserialization"]
    
    D --> D1["8 KB Header Location & Timestamp Tables"]
    D --> D2["Sector Slicing & Chunk Decompression"]
    D --> D3["Section Palette & Block State Extraction"]
    
    E --> E1["Datapack Recipes, Loot Tables & Functions"]
    E --> E2["Mod Manifests (fabric.mod.json, mods.toml, plugin.yml)"]
    E --> E3["Mixin Configurations & Event Listener Graphs"]
    
    C & D & E --> F["World Manifest, Chunk Inspector & Mod Dependency Graph"]

Technical Capabilities & Modules

Universal NBT Deserializer:
- Parse all 13 standard NBT tag types from raw, Gzip-compressed, or Zlib-compressed binary streams.
- Robustly export NBT trees to structured JSON or readable SNBT (Stringified NBT) formats.
- Parse root level.dat files, extracting world seed, difficulty, game rules, and versioning tags.
Anvil Region Container Parsing (.mca):
- Parse 8 KB location and timestamp headers for regional files (r.X.Z.mca).
- Extract individual chunk sectors, validating sector lengths and compression types (Zlib, Gzip, LZ4).
- Decompress chunk NBT data and index chunk coordinate positions (xPos, zPos, yPos).
Chunk Structure & Palette Extraction:
- Dissect 16x16x16 chunk sections, reading palette tables and bit-packed block state arrays (BlockStates).
- Extract Block Entity compound states (chests, signs, furnaces, custom inventory items).
- Extract Entity arrays (mobs, items, vehicles) with positional coordinates and NBT attributes.
Datapack Logic & Recipe Inspection:
- Traverse data pack directory structures (data/<namespace>/...).
- Parse JSON recipes, advancement triggers, and loot table probabilities.
- Disassemble .mcfunction text files into structured command flowcharts.
Modloader Ecosystem & Manifest Modeling:
- Parse mod descriptors: fabric.mod.json, META-INF/mods.toml, quilt.mod.json, plugin.yml.
- Reconstruct mod dependency trees (depends, recommends, conflicts).
- Identify declared Mixin configuration JSONs (mixins.json) and access widener files (.accesswidener).

Acceptance Criteria

Static, safe deserialization of any arbitrary NBT stream (compressed or uncompressed).
Parsing of .mca Anvil region files and extraction of individual chunk NBT compounds.
Accurate decoding of palette-based block states and tile entity records.
Ingestion and validation of datapack recipes, loot tables, and function files.
Parsing of modern modloader manifests and reconstruction of mod dependency graphs.

Complements #3621 (Unknown binary file-format inference)
Complements #1558 (Audit analysis and reverse engineering capabilities across JVM bytecode)

Originally created by @matbrgz on GitHub (Oct 14, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/3702 [![Type: Feature](https://img.shields.io/badge/type-feature-blue.svg?style=flat-square)](https://github.com/morluto/rea) [![Domain: Game & Engine RE](https://img.shields.io/badge/domain-game_&_engine_RE-green.svg?style=flat-square)](https://github.com/morluto/rea) [![Status: Proposed](https://img.shields.io/badge/status-proposed-blue.svg?style=flat-square)](https://github.com/morluto/rea) [![Ecosystem: REA](https://img.shields.io/badge/ecosystem-morluto%2Frea-8A2BE2.svg?style=flat-square)](https://github.com/morluto/rea) ### Problem Statement & Context Voxel-based sandbox game worlds store massive, procedurally generated multi-dimensional terrains, player inventories, entity data, and custom logic across structured binary file formats: 1. **Named Binary Tag (NBT) Protocol**: A hierarchical binary serialization format composed of typed payload tags: `TAG_End` (0), `TAG_Byte` (1), `TAG_Short` (2), `TAG_Int` (3), `TAG_Long` (4), `TAG_Float` (5), `TAG_Double` (6), `TAG_Byte_Array` (7), `TAG_String` (8), `TAG_List` (9), `TAG_Compound` (10), `TAG_Int_Array` (11), `TAG_Long_Array` (12). 2. **Anvil Region Containers (`.mca`)**: Regional terrain files storing 32x32 (1024) chunks. Each `.mca` file begins with an 8 KB header consisting of: - Chunk location table: 1024 4-byte entries (3 bytes sector offset, 1 byte sector count). - Chunk timestamp table: 1024 4-byte integer modification timestamps. - Sector-aligned chunk data compressed via Zlib (type 2) or LZ4 (type 3). 3. **World Configuration & Registry State (`level.dat`)**: Gzip-compressed NBT compound storing world seed, spawn coordinates, gamerules, data version, and custom dimension registries. 4. **Data Packs**: Zip or directory archives containing JSON-defined recipes, advancements, loot tables, world generation biomes, and procedural command functions (`.mcfunction`). 5. **Modloader Ecosystems & Dependency Manifests**: Metadata descriptors (`fabric.mod.json`, `mods.toml`, `quilt.mod.json`, `plugin.yml`) declaring mod ID, entrypoints, mixin configurations, access wideners, dependencies, and listener events. REA needs a dedicated static analyzer capable of parsing NBT trees, extracting Anvil chunks without game execution, and modeling datapack/modloader dependency graphs. ### Proposed Architecture ```mermaid flowchart TD A["World Directory (.mca, level.dat) / Datapack / Mod Archive"] --> B["Voxel World & NBT Semantic Engine"] B --> C["NBT Deserializer (Gzip / Zlib / Uncompressed)"] B --> D["Anvil Region (.mca) Slicer & Chunk Extractor"] B --> E["Datapack & Modloader Metadata Reconstructor"] C --> C1["TAG_Compound & Recursive Tag Parser"] C --> C2["level.dat & Player / Entity State Deserialization"] D --> D1["8 KB Header Location & Timestamp Tables"] D --> D2["Sector Slicing & Chunk Decompression"] D --> D3["Section Palette & Block State Extraction"] E --> E1["Datapack Recipes, Loot Tables & Functions"] E --> E2["Mod Manifests (fabric.mod.json, mods.toml, plugin.yml)"] E --> E3["Mixin Configurations & Event Listener Graphs"] C & D & E --> F["World Manifest, Chunk Inspector & Mod Dependency Graph"] ``` ### Technical Capabilities & Modules 1. **Universal NBT Deserializer:** - Parse all 13 standard NBT tag types from raw, Gzip-compressed, or Zlib-compressed binary streams. - Robustly export NBT trees to structured JSON or readable SNBT (Stringified NBT) formats. - Parse root `level.dat` files, extracting world seed, difficulty, game rules, and versioning tags. 2. **Anvil Region Container Parsing (`.mca`):** - Parse 8 KB location and timestamp headers for regional files (`r.X.Z.mca`). - Extract individual chunk sectors, validating sector lengths and compression types (Zlib, Gzip, LZ4). - Decompress chunk NBT data and index chunk coordinate positions (`xPos`, `zPos`, `yPos`). 3. **Chunk Structure & Palette Extraction:** - Dissect 16x16x16 chunk sections, reading palette tables and bit-packed block state arrays (`BlockStates`). - Extract Block Entity compound states (chests, signs, furnaces, custom inventory items). - Extract Entity arrays (mobs, items, vehicles) with positional coordinates and NBT attributes. 4. **Datapack Logic & Recipe Inspection:** - Traverse data pack directory structures (`data/<namespace>/...`). - Parse JSON recipes, advancement triggers, and loot table probabilities. - Disassemble `.mcfunction` text files into structured command flowcharts. 5. **Modloader Ecosystem & Manifest Modeling:** - Parse mod descriptors: `fabric.mod.json`, `META-INF/mods.toml`, `quilt.mod.json`, `plugin.yml`. - Reconstruct mod dependency trees (`depends`, `recommends`, `conflicts`). - Identify declared Mixin configuration JSONs (`mixins.json`) and access widener files (`.accesswidener`). ### Acceptance Criteria - [ ] Static, safe deserialization of any arbitrary NBT stream (compressed or uncompressed). - [ ] Parsing of `.mca` Anvil region files and extraction of individual chunk NBT compounds. - [ ] Accurate decoding of palette-based block states and tile entity records. - [ ] Ingestion and validation of datapack recipes, loot tables, and function files. - [ ] Parsing of modern modloader manifests and reconstruction of mod dependency graphs. ### Related Issues & Dependencies - Complements #3621 (Unknown binary file-format inference) - Complements #1558 (Audit analysis and reverse engineering capabilities across JVM bytecode)

bsakaguchi referenced this issue

2026-10-17 21:11:12 +00:00

[PR #1412] [CLOSED] fix: report oversized JSON input as a runtime string limit #2866

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

morluto/rea#1412

No description provided.

Rows
Columns