morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[GH-ISSUE #3667] [Feature] RAG/vector-store reverse engineering: FAISS, Chroma, LanceDB, embeddings, chunk provenance and retriever reconstruction #1378

New issue

Open

opened 2026-10-16 21:04:52 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-16 21:04:52 +00:00

Owner

Originally created by @matbrgz on GitHub (Oct 14, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/3667

Summary

Retrieval-Augmented Generation (RAG) is the dominant architecture for grounding LLMs in proprietary enterprise data, codebases, legal documents, and personal memory. RAG applications index unstructured knowledge into vector databases and search indices:

FAISS (Facebook AI Similarity Search): Binary index files (.faiss, .index) storing compressed vector quantizations (IVF, HNSW, PQ - Product Quantization);
ChromaDB / SQLite: SQLite collections (chroma.sqlite3) and Parquet / binary vector segment files;
LanceDB / Lance Format: Columnar vector storage with zero-copy disk-backed indices;
Document Chunk Registries: Stored document chunks, source file paths, token spans, and metadata attributes.

Currently, REA does not cover vector databases or RAG indices. When analyzing an AI application, security researchers and analysts need to inspect what proprietary knowledge is indexed, what embedding models were used, and how chunks are retrieved.

This feature proposal introduces dedicated RAG and Vector-Store Reverse Engineering to REA, enabling automated inspection of FAISS, ChromaDB, and LanceDB indices, recovering vector dimensions, similarity metrics, chunk provenance, and retriever configurations.

Technical Capabilities & Architecture

                                    +-----------------------------------------+
                                    | Input: Vector Store Directory or Index  |
                                    |        (index.faiss, chroma.sqlite3)    |
                                    +--------------------+--------------------+
                                                         |
                         +-------------------------------+-------------------------------+
                         |                                                               |
                         v                                                               v
           +---------------------------+                                   +---------------------------+
           | FAISS Index Binary Parser |                                   | Chroma / LanceDB Columnar |
           | (IVF, HNSW, PQ, Flat)     |                                   | Segment Reader            |
           +-------------+-------------+                                   +-------------+-------------+
                         |                                                               |
                         +-------------------------------+-------------------------------+
                                                         |
                                                         v
                                     +---------------------------------------+
                                     | Vector Dimensionality & Metric Engine |
                                     | (Dimensions, Cosine, L2, Dot Product) |
                                     +-------------------+-------------------+
                                                         |
                         +-------------------------------+-------------------------------+
                         |                                                               |
                         v                                                               v
           +---------------------------+                                   +---------------------------+
           | Chunk Provenance Carving  |                                   | Retriever Parameter       |
           | (Document IDs, File Paths)|                                   | Inversion (top-k, filters)|
           +-------------+-------------+                                   +-------------+-------------+
                                                         |
                                                         v
                                     +---------------------------------------+
                                     | REA Evidence: Knowledge Base Dossier, |
                                     | Indexed Documents & Retriever Graph   |
                                     +---------------------------------------+

1. Vector Store Format Ingestion

FAISS Binary Index Parser: Parse raw FAISS index serialization streams without requiring native faiss binary installations:
- Header magic words (IxFl, IxF2, IxHn, IxPq).
- Index types: IndexFlatL2, IndexFlatIP, IndexIVFFlat, IndexHNSWFlat, IndexPQ.
- Number of indexed vectors (N) and vector dimension (d).
ChromaDB Ingestion: Parse chroma.sqlite3 databases and vector segment files, extracting collection names, embedding metadata, and document records.
LanceDB Ingestion: Parse Lance columnar layout directories, reading vector column definitions and IVF-PQ inverted indices.

2. Dimensionality, Metric & Quantization Analysis

Vector Parameters: Extract embedding vector dimension (d, e.g. 384, 768, 1536, 3072), matching against known embedding models (e.g. OpenAI text-embedding-3-small, BGE, E5, Cohere).
Distance Metric: Identify the configured distance metric: Inner Product (Cosine similarity), Euclidean (L_2), or Manhattan (L_1).
Product Quantization (PQ): For compressed indices, extract codebook centroids (M sub-vectors, k centroids per sub-space).

3. Chunk Provenance & Metadata Extraction

Document Source Mapping: Carve document metadata stored alongside vectors:
- Original file names, URLs, and local file paths.
- Chunk IDs and parent document relationships.
- Text snippet samples and token length distributions.
Metadata Filters: Catalog indexed metadata fields (e.g. user_id, department, date, category, access_level) used for filtered retrieval.

4. Retriever Workflow & LLM Integration Inversion

Retrieval Hyperparameters: Extract configured retrieval values:
- top\_k candidate count (e.g. 3, 5, 10).
- Score threshold cutoffs and minimum similarity scores.
- Hybrid search weights (dense semantic vector score vs. sparse BM25 keyword score).
- Reranker model configurations (e.g. Cohere Rerank, BGE-Reranker).
RAG Dataflow Graph: Build a directed flow graph: User Query -> Embedding Model -> Vector Search Index -> Chunk Filtering -> Context Window Assembly -> LLM Prompt.

Acceptance Criteria

Introduce analyze_vector_store in REA CLI and MCP contracts, accepting FAISS index files, ChromaDB directories, or LanceDB stores.
Ingest synthetic FAISS (IndexFlat and IndexIVF) files and extract vector counts, dimensions, and metric types.
Extract indexed document metadata and text chunk provenance from ChromaDB SQLite fixtures.
Identify probable embedding model families based on dimensionality and normalization metrics.
Output retriever configurations (top\_k, hybrid search parameters) in structured JSON evidence.

#3323 [Feature] Inspect GGUF and safetensors model files without loading them
#3651 [Feature] LLM application reverse engineering
#3621 [Feature] Unknown binary file-format inference

Originally created by @matbrgz on GitHub (Oct 14, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/3667 [![Type: Feature](https://img.shields.io/badge/type-feature-blue.svg?style=flat-square)](https://github.com/morluto/rea) [![Domain: Game & Engine RE](https://img.shields.io/badge/domain-game_&_engine_RE-green.svg?style=flat-square)](https://github.com/morluto/rea) [![Status: Proposed](https://img.shields.io/badge/status-proposed-blue.svg?style=flat-square)](https://github.com/morluto/rea) [![Ecosystem: REA](https://img.shields.io/badge/ecosystem-morluto%2Frea-8A2BE2.svg?style=flat-square)](https://github.com/morluto/rea) ## Summary Retrieval-Augmented Generation (RAG) is the dominant architecture for grounding LLMs in proprietary enterprise data, codebases, legal documents, and personal memory. RAG applications index unstructured knowledge into vector databases and search indices: - **FAISS (Facebook AI Similarity Search):** Binary index files (`.faiss`, `.index`) storing compressed vector quantizations (IVF, HNSW, PQ - Product Quantization); - **ChromaDB / SQLite:** SQLite collections (`chroma.sqlite3`) and Parquet / binary vector segment files; - **LanceDB / Lance Format:** Columnar vector storage with zero-copy disk-backed indices; - **Document Chunk Registries:** Stored document chunks, source file paths, token spans, and metadata attributes. Currently, REA does not cover vector databases or RAG indices. When analyzing an AI application, security researchers and analysts need to inspect what proprietary knowledge is indexed, what embedding models were used, and how chunks are retrieved. This feature proposal introduces dedicated **RAG and Vector-Store Reverse Engineering** to REA, enabling automated inspection of FAISS, ChromaDB, and LanceDB indices, recovering vector dimensions, similarity metrics, chunk provenance, and retriever configurations. --- ## Technical Capabilities & Architecture ``` +-----------------------------------------+ | Input: Vector Store Directory or Index | | (index.faiss, chroma.sqlite3) | +--------------------+--------------------+ | +-------------------------------+-------------------------------+ | | v v +---------------------------+ +---------------------------+ | FAISS Index Binary Parser | | Chroma / LanceDB Columnar | | (IVF, HNSW, PQ, Flat) | | Segment Reader | +-------------+-------------+ +-------------+-------------+ | | +-------------------------------+-------------------------------+ | v +---------------------------------------+ | Vector Dimensionality & Metric Engine | | (Dimensions, Cosine, L2, Dot Product) | +-------------------+-------------------+ | +-------------------------------+-------------------------------+ | | v v +---------------------------+ +---------------------------+ | Chunk Provenance Carving | | Retriever Parameter | | (Document IDs, File Paths)| | Inversion (top-k, filters)| +-------------+-------------+ +-------------+-------------+ | v +---------------------------------------+ | REA Evidence: Knowledge Base Dossier, | | Indexed Documents & Retriever Graph | +---------------------------------------+ ``` ### 1. Vector Store Format Ingestion - **FAISS Binary Index Parser:** Parse raw FAISS index serialization streams without requiring native `faiss` binary installations: - Header magic words (`IxFl`, `IxF2`, `IxHn`, `IxPq`). - Index types: `IndexFlatL2`, `IndexFlatIP`, `IndexIVFFlat`, `IndexHNSWFlat`, `IndexPQ`. - Number of indexed vectors ($N$) and vector dimension ($d$). - **ChromaDB Ingestion:** Parse `chroma.sqlite3` databases and vector segment files, extracting collection names, embedding metadata, and document records. - **LanceDB Ingestion:** Parse Lance columnar layout directories, reading vector column definitions and IVF-PQ inverted indices. ### 2. Dimensionality, Metric & Quantization Analysis - **Vector Parameters:** Extract embedding vector dimension ($d$, e.g. 384, 768, 1536, 3072), matching against known embedding models (e.g. OpenAI `text-embedding-3-small`, BGE, E5, Cohere). - **Distance Metric:** Identify the configured distance metric: Inner Product (Cosine similarity), Euclidean ($L_2$), or Manhattan ($L_1$). - **Product Quantization (PQ):** For compressed indices, extract codebook centroids ($M$ sub-vectors, $k$ centroids per sub-space). ### 3. Chunk Provenance & Metadata Extraction - **Document Source Mapping:** Carve document metadata stored alongside vectors: - Original file names, URLs, and local file paths. - Chunk IDs and parent document relationships. - Text snippet samples and token length distributions. - **Metadata Filters:** Catalog indexed metadata fields (e.g. `user_id`, `department`, `date`, `category`, `access_level`) used for filtered retrieval. ### 4. Retriever Workflow & LLM Integration Inversion - **Retrieval Hyperparameters:** Extract configured retrieval values: - $top\_k$ candidate count (e.g. 3, 5, 10). - Score threshold cutoffs and minimum similarity scores. - Hybrid search weights (dense semantic vector score vs. sparse BM25 keyword score). - Reranker model configurations (e.g. Cohere Rerank, BGE-Reranker). - **RAG Dataflow Graph:** Build a directed flow graph: `User Query -> Embedding Model -> Vector Search Index -> Chunk Filtering -> Context Window Assembly -> LLM Prompt`. --- ## Acceptance Criteria 1. Introduce `analyze_vector_store` in REA CLI and MCP contracts, accepting FAISS index files, ChromaDB directories, or LanceDB stores. 2. Ingest synthetic FAISS (`IndexFlat` and `IndexIVF`) files and extract vector counts, dimensions, and metric types. 3. Extract indexed document metadata and text chunk provenance from ChromaDB SQLite fixtures. 4. Identify probable embedding model families based on dimensionality and normalization metrics. 5. Output retriever configurations ($top\_k$, hybrid search parameters) in structured JSON evidence. --- ## Related - #3323 [Feature] Inspect GGUF and safetensors model files without loading them - #3651 [Feature] LLM application reverse engineering - #3621 [Feature] Unknown binary file-format inference

bsakaguchi referenced this issue

2026-10-17 21:10:52 +00:00

[PR #1378] [MERGED] fix(apple): scope keyed archive traversal to the selection #2839

bsakaguchi commented

2026-10-31 00:30:47 +00:00

Author

Owner

@ZAABY commented on GitHub (Oct 17, 2026):

I'd like to work on a narrow first slice: read-only inspection of an offline Chroma SQLite database from one explicitly supported Chroma release.

The goal is to show what collections and document/chunk references the store contains without starting Chroma or running the application. I'd report schema/version, collection metadata, counts and references with source provenance and limits. Document text and embeddings would stay out of the default output. Unsupported schemas would be reported explicitly, not guessed.

No live server connection, network requests, writes to the original database, or inference. FAISS, LanceDB, retrieval replay and guessing the embedding model from vector dimensions would be out of scope.

Before I build the integration, does this first slice fit REA? Would you prefer a new inspect-shaped CLI/MCP operation or an extension of an existing artifact workflow? I'll propose the exact Chroma release/schema after checking it against generated fixtures.

@ZAABY commented on GitHub (Oct 17, 2026): I'd like to work on a narrow first slice: read-only inspection of an offline Chroma SQLite database from one explicitly supported Chroma release. The goal is to show what collections and document/chunk references the store contains without starting Chroma or running the application. I'd report schema/version, collection metadata, counts and references with source provenance and limits. Document text and embeddings would stay out of the default output. Unsupported schemas would be reported explicitly, not guessed. No live server connection, network requests, writes to the original database, or inference. FAISS, LanceDB, retrieval replay and guessing the embedding model from vector dimensions would be out of scope. Before I build the integration, does this first slice fit REA? Would you prefer a new inspect-shaped CLI/MCP operation or an extension of an existing artifact workflow? I'll propose the exact Chroma release/schema after checking it against generated fixtures.

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

morluto/rea#1378

No description provided.

Rows
Columns