morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[GH-ISSUE #4986] [Feature] Decode protobuf and gRPC bodies in inspect-web-network-capture #3082

New issue

Closed

opened 2026-10-31 00:36:16 +00:00 by bsakaguchi · 4 comments

bsakaguchi commented

2026-10-31 00:36:16 +00:00

Owner

Originally created by @KJLavender on GitHub (Oct 18, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/4986

Before submitting

I searched the open and closed issues and pull requests for a related proposal.
I understand the contribution scope must be reviewed before integration; open questions are welcome here.
This request does not include credentials, private samples, or sensitive target data.

Area

Evidence, artifacts, or reconstruction

Current limitation

inspect-web-network-capture keeps binary bodies as base64 plus SHA-256. That's the right thing to keep, but for protobuf traffic it's also where the trail ends, and "what does this app actually send to its server" is a pretty common question. gRPC-web and plain application/x-protobuf APIs show up a lot in web and mobile apps.

To check, I wrote a tiny HAR by hand (below, fully synthetic). It has one gRPC-web response and one raw application/x-protobuf response, both carrying the same 14-byte message: field 1 = 150, field 2 = "hello", field 3 = a nested message with field 1 = 1. For the gRPC-web record, REA 6.5.0 returns:

"binary_fields": [{"pointer": "/response/content/text", "representation": "har-base64-content",
  "state": "retained", "bytes": 19, "sha256": "ed7b12417d3b...", "content_base64": "AAAAAA4IlgESBWhlbGxvGgIIAQ=="}]

There's nothing about the 5-byte gRPC frame header or the fields inside. The raw protobuf record looks the same.

I saw #364 (gRPC/Protobuf capture) is closed, but I couldn't find protobuf decoding in the current capture tools, and the SAZ section of the capture docs says gRPC message bytes are kept opaque. What I'm asking for here is narrower than #364: decode bytes that are already in a capture, with no live capture or interception. If this is tracked somewhere else, feel free to close it as a duplicate.

Desired outcome and success criteria

For records with a protobuf content type (application/grpc, application/grpc+proto, application/grpc-web+proto, application/grpc-web-text, application/x-protobuf), or ones I select explicitly:

Split gRPC / gRPC-web framing: compressed flag, length, data frames vs trailer frames (grpc-status and so on). application/grpc-web-text bodies are base64 text, so they need one more decode before the split.
Decode the raw wire format the way protoc --decode_raw does: field numbers, wire types, varints, length-delimited values, with byte offsets back into the body.
Keep the ambiguous parts marked as inferred. A length-delimited field can be a string, bytes, a nested message or packed numbers, and you can't tell for sure without the schema.
Nice to have: decode with a caller-supplied .proto file or descriptor set when the user has one. If the schema and the raw decode disagree (the schema says string but the bytes also parse as a nested message, say), show both side by side instead of quietly picking one.

For the sample below I'd expect something like: one data frame, 14 bytes, uncompressed. Field 1 is varint 150, field 2 is 5 bytes that read as UTF-8 "hello", and field 3 is 2 bytes that parse as a nested message {1: 1}. For comparison, protoc --decode_raw should print this for the same 14 bytes:

1: 150
2: "hello"
3 {
  1: 1
}

Expected interaction with targets

Read existing files or captured data

Verification available so far

I can suggest a sample or manual check

Verification details (optional)

The HAR below plus the expected output above. The gRPC-web body is 000000000e089601120568656c6c6f1a020801 in hex.

grpc-demo.har (synthetic)

{
 "log": {
  "version": "1.2",
  "creator": {
   "name": "handmade",
   "version": "1"
  },
  "entries": [
   {
    "startedDateTime": "2026-10-18T10:00:00.000Z",
    "time": 12,
    "request": {
     "method": "POST",
     "url": "https://api.example.test/demo.v1.Greeter/SayHello",
     "httpVersion": "HTTP/1.1",
     "cookies": [],
     "headers": [
      {
       "name": "content-type",
       "value": "application/grpc-web+proto"
      }
     ],
     "queryString": [],
     "headersSize": -1,
     "bodySize": 0
    },
    "response": {
     "status": 200,
     "statusText": "OK",
     "httpVersion": "HTTP/1.1",
     "cookies": [],
     "headers": [
      {
       "name": "content-type",
       "value": "application/grpc-web+proto"
      }
     ],
     "content": {
      "size": 19,
      "mimeType": "application/grpc-web+proto",
      "text": "AAAAAA4IlgESBWhlbGxvGgIIAQ==",
      "encoding": "base64"
     },
     "redirectURL": "",
     "headersSize": -1,
     "bodySize": 19
    },
    "cache": {},
    "timings": {
     "send": 0,
     "wait": 12,
     "receive": 0
    }
   },
   {
    "startedDateTime": "2026-10-18T10:00:00.000Z",
    "time": 12,
    "request": {
     "method": "POST",
     "url": "https://api.example.test/v1/profile",
     "httpVersion": "HTTP/1.1",
     "cookies": [],
     "headers": [
      {
       "name": "content-type",
       "value": "application/x-protobuf"
      }
     ],
     "queryString": [],
     "headersSize": -1,
     "bodySize": 0
    },
    "response": {
     "status": 200,
     "statusText": "OK",
     "httpVersion": "HTTP/1.1",
     "cookies": [],
     "headers": [
      {
       "name": "content-type",
       "value": "application/x-protobuf"
      }
     ],
     "content": {
      "size": 14,
      "mimeType": "application/x-protobuf",
      "text": "CJYBEgVoZWxsbxoCCAE=",
      "encoding": "base64"
     },
     "redirectURL": "",
     "headersSize": -1,
     "bodySize": 14
    },
    "cache": {},
    "timings": {
     "send": 0,
     "wait": 12,
     "receive": 0
    }
   }
  ]
 }
}

Alternatives considered

Copying the base64 out and running protoc --decode_raw or CyberChef by hand. That works for one body, but you lose the link back to the record and the byte offsets, and you have to repeat it for every request.

Possible direction

A decoded view next to binary_fields, probably only for selected records (--record already exists) so normal output stays small.

Edit: added application/grpc-web-text (and native application/grpc) plus the raw-vs-schema point, from the comment below.

Originally created by @KJLavender on GitHub (Oct 18, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/4986 ### Before submitting - [x] I searched the open and closed issues and pull requests for a related proposal. - [x] I understand the contribution scope must be reviewed before integration; open questions are welcome here. - [x] This request does not include credentials, private samples, or sensitive target data. ### Area Evidence, artifacts, or reconstruction ### Current limitation `inspect-web-network-capture` keeps binary bodies as base64 plus SHA-256. That's the right thing to keep, but for protobuf traffic it's also where the trail ends, and "what does this app actually send to its server" is a pretty common question. gRPC-web and plain `application/x-protobuf` APIs show up a lot in web and mobile apps. To check, I wrote a tiny HAR by hand (below, fully synthetic). It has one gRPC-web response and one raw `application/x-protobuf` response, both carrying the same 14-byte message: field 1 = 150, field 2 = "hello", field 3 = a nested message with field 1 = 1. For the gRPC-web record, REA 6.5.0 returns: ```json "binary_fields": [{"pointer": "/response/content/text", "representation": "har-base64-content", "state": "retained", "bytes": 19, "sha256": "ed7b12417d3b...", "content_base64": "AAAAAA4IlgESBWhlbGxvGgIIAQ=="}] ``` There's nothing about the 5-byte gRPC frame header or the fields inside. The raw protobuf record looks the same. I saw #364 (gRPC/Protobuf capture) is closed, but I couldn't find protobuf decoding in the current capture tools, and the SAZ section of the capture docs says gRPC message bytes are kept opaque. What I'm asking for here is narrower than #364: decode bytes that are already in a capture, with no live capture or interception. If this is tracked somewhere else, feel free to close it as a duplicate. ### Desired outcome and success criteria For records with a protobuf content type (`application/grpc`, `application/grpc+proto`, `application/grpc-web+proto`, `application/grpc-web-text`, `application/x-protobuf`), or ones I select explicitly: 1. Split gRPC / gRPC-web framing: compressed flag, length, data frames vs trailer frames (`grpc-status` and so on). `application/grpc-web-text` bodies are base64 text, so they need one more decode before the split. 2. Decode the raw wire format the way `protoc --decode_raw` does: field numbers, wire types, varints, length-delimited values, with byte offsets back into the body. 3. Keep the ambiguous parts marked as inferred. A length-delimited field can be a string, bytes, a nested message or packed numbers, and you can't tell for sure without the schema. 4. Nice to have: decode with a caller-supplied `.proto` file or descriptor set when the user has one. If the schema and the raw decode disagree (the schema says string but the bytes also parse as a nested message, say), show both side by side instead of quietly picking one. For the sample below I'd expect something like: one data frame, 14 bytes, uncompressed. Field 1 is varint 150, field 2 is 5 bytes that read as UTF-8 "hello", and field 3 is 2 bytes that parse as a nested message {1: 1}. For comparison, `protoc --decode_raw` should print this for the same 14 bytes: ``` 1: 150 2: "hello" 3 { 1: 1 } ``` ### Expected interaction with targets Read existing files or captured data ### Verification available so far I can suggest a sample or manual check ### Verification details (optional) The HAR below plus the expected output above. The gRPC-web body is `000000000e089601120568656c6c6f1a020801` in hex. <details> <summary>grpc-demo.har (synthetic)</summary> ```json { "log": { "version": "1.2", "creator": { "name": "handmade", "version": "1" }, "entries": [ { "startedDateTime": "2026-10-18T10:00:00.000Z", "time": 12, "request": { "method": "POST", "url": "https://api.example.test/demo.v1.Greeter/SayHello", "httpVersion": "HTTP/1.1", "cookies": [], "headers": [ { "name": "content-type", "value": "application/grpc-web+proto" } ], "queryString": [], "headersSize": -1, "bodySize": 0 }, "response": { "status": 200, "statusText": "OK", "httpVersion": "HTTP/1.1", "cookies": [], "headers": [ { "name": "content-type", "value": "application/grpc-web+proto" } ], "content": { "size": 19, "mimeType": "application/grpc-web+proto", "text": "AAAAAA4IlgESBWhlbGxvGgIIAQ==", "encoding": "base64" }, "redirectURL": "", "headersSize": -1, "bodySize": 19 }, "cache": {}, "timings": { "send": 0, "wait": 12, "receive": 0 } }, { "startedDateTime": "2026-10-18T10:00:00.000Z", "time": 12, "request": { "method": "POST", "url": "https://api.example.test/v1/profile", "httpVersion": "HTTP/1.1", "cookies": [], "headers": [ { "name": "content-type", "value": "application/x-protobuf" } ], "queryString": [], "headersSize": -1, "bodySize": 0 }, "response": { "status": 200, "statusText": "OK", "httpVersion": "HTTP/1.1", "cookies": [], "headers": [ { "name": "content-type", "value": "application/x-protobuf" } ], "content": { "size": 14, "mimeType": "application/x-protobuf", "text": "CJYBEgVoZWxsbxoCCAE=", "encoding": "base64" }, "redirectURL": "", "headersSize": -1, "bodySize": 14 }, "cache": {}, "timings": { "send": 0, "wait": 12, "receive": 0 } } ] } } ``` </details> ### Alternatives considered Copying the base64 out and running `protoc --decode_raw` or CyberChef by hand. That works for one body, but you lose the link back to the record and the byte offsets, and you have to repeat it for every request. ### Possible direction A decoded view next to `binary_fields`, probably only for selected records (`--record` already exists) so normal output stays small. Edit: added `application/grpc-web-text` (and native `application/grpc`) plus the raw-vs-schema point, from the comment below.

bsakaguchi closed this issue

2026-10-31 00:36:16 +00:00

bsakaguchi commented

2026-10-31 00:36:17 +00:00

Author

Owner

@XinwangTao commented on GitHub (Oct 18, 2026):

Agreed this is the right slice: keep the retained base64 as the source of truth and add a decoding view on top, rather than widening #364 into live interception. The decode_raw-style output (field numbers, wire types, varints, length-delimited with byte offsets) is exactly the right level of certainty — without a schema, wire format is the only layer you can decode unambiguously, and marking length-delimited fields as inferred (string/bytes/nested/packed) is the honest thing to do. A few concrete notes: (1) gRPC-web framing needs the 1-byte compressed flag + 4-byte length stripped first, and trailer frames carry grpc-status/grpc-message — worth decoding those rather than dropping them; (2) the content-type set to probe is broader than the two mentioned: application/grpc-web+proto, application/grpc-web-text (base64-encoded), application/x-protobuf, and occasionally text proto; (3) when a caller-supplied descriptor set is used, wire-type-2 fields can disagree with user schemas (string vs nested message vs packed) — suggesting showing both raw decode and schema decode side by side, or an explicit flag to choose. Gating the decoded view behind the existing --record selector keeps default output small, which is a good default.

@XinwangTao commented on GitHub (Oct 18, 2026): Agreed this is the right slice: keep the retained base64 as the source of truth and add a decoding view on top, rather than widening #364 into live interception. The decode_raw-style output (field numbers, wire types, varints, length-delimited with byte offsets) is exactly the right level of certainty — without a schema, wire format is the only layer you can decode unambiguously, and marking length-delimited fields as inferred (string/bytes/nested/packed) is the honest thing to do. A few concrete notes: (1) gRPC-web framing needs the 1-byte compressed flag + 4-byte length stripped first, and trailer frames carry grpc-status/grpc-message — worth decoding those rather than dropping them; (2) the content-type set to probe is broader than the two mentioned: application/grpc-web+proto, application/grpc-web-text (base64-encoded), application/x-protobuf, and occasionally text proto; (3) when a caller-supplied descriptor set is used, wire-type-2 fields can disagree with user schemas (string vs nested message vs packed) — suggesting showing both raw decode and schema decode side by side, or an explicit flag to choose. Gating the decoded view behind the existing --record selector keeps default output small, which is a good default.

bsakaguchi commented

2026-10-31 00:36:18 +00:00

Author

Owner

@jimtje commented on GitHub (Oct 18, 2026):

The flow will necessarily differ depending on whether you have access to the client or not. If the client is available, the definitions (at least the types but usually more) will be either a) assets, with standard Google typing as the base, or b) embedded, in which case you can extract pretty much the literal original .proto files from java, apk packages, decrypted ipa packages or assets from said packages, c#, c++, or binaries. I'm not sure if the scenario where one intercepts straight up messages that are definitely protobuf but have no access to either client or server at all actually is even a realistic one. There are tools that fuzz endpoints but those are loud and not usually effective unless you have a lead and hence a wordlist. The client, if nothing else for debugging, almost always have some if not all of the schema somewhere. This extends out to static files that are serialized in the same way - the definition is in the clients there too, even if the end goal does not involve transmission. This was and I guess to an extent remains common for crypto wallets derived from the BitcoinJ walletss.

I would not simply guess by decoding the message beccause without context you need to at least run enum via the Google proto base schemas and unless it's embedded in the message, which are pretty rare, it's just hunches. Those can be completely off. Sometimes the messagegs are hashes, good luck figuring out those without the schema, and those are actually quite common. After that traditionally json has been the norm but python dataclasses are far easier to work with when you don't have the schema applied yet. Also the same library decode and encode, generally.

I have a written (slightly old, but it works without issue) implementation of a python decode/deserializer of static cryptocurrency files that covers All bitcoinj-derived formats and coinomi in the walletlib repo on my page. This is an IOS one-pager that extracts from .ipas: https://github.com/riyadmondol2006/iOS-Protobuf-Extractor/blob/main/ios_protobuf_extractor.py

Replacement for Google's protoc tool: https://github.com/betterproto/python-betterproto2 as it generates dataclasses from the getgo. There are other libraries but I've preferred this since using it to decode Tinder's gRPC/protobuf login scheme during a baseball rain delay (and spent 2 years fielding questions about it before finally finding someone to take it off my hands, that was bad sincce I don't use Tinder and the sheer number of people who wanted to run Tinder bots was... much higher than I could have expected).

c#: https://github.com/Xpl0itR/protodec can't really test, on a mac, but in theory it should work.

java, apk, c++: https://github.com/marin-m/pbtk CLI tools only. No clue how the GUI works but the cli does the job quite well.

Raw hex decode: https://github.com/pawitp/protobuf-decoder message only, some are obvious, some are not, some are encrypted

Blackboxproto is fairly popular but it essentially still is taking what it appears to contain + format = what it is. That works if you're getting strings and nothing but. That isn't common and without the schema as the context you literally don't need it as you can enumerate types with the stock files Google already posts and is in almost every app that uses profobuf, and the original files would be in the correct hierarchy in terms of folders/levels of granularity. That data isn't kept in messages but are sometimes very helpful and prevents you from ending up with a folder full of json files that you havve to put back together but you have neither all the king's men or all the king's horses and even if you did, do they know a 32 bit int versus a 64 bit int? The schema does. Humpty Dumpty would be something else if he is read as 32 bit instead of 64 and vice versa. You can obviously just do both but it usually doesn't do the trick since the output is valid, but still incorrect.

If it's encrypted, it's almost always RSA. No guarantees that the key would be easy to find, but I've never seen it not at least first pass RSA encrypt and then maybe something else that would require pyopenssl or the like.

I can't write as fast as Claude but I'm pretty sure that Claude trained off my code since I CC0 or public domained everything I wrote in the last 10 years and it's part of every crypto recovery and from 2021 to 2023, every Tinder bot in Python, apparently. Sorry about that if you got conned, it was supposed to be a joke to amuse a groupchat. At the very least this will save you some tokens, but really it's always the unforseen pitfalls of fuzzing blind - noisiness tends to be the first death knell to the project. Tinder literally didn't do anything for a year and didn't put in serious remediation for 18 month, and the remediation was... still the same format. They ended up ripping out their entire login about 6 months after I passed the project off. Somehow my account didn't get banned even though throwaway email with api access and it had TS Eliot's photo on it.

@jimtje commented on GitHub (Oct 18, 2026): The flow will necessarily differ depending on whether you have access to the client or not. If the client is available, the definitions (at least the types but usually more) will be either a) assets, with standard Google typing as the base, or b) embedded, in which case you can extract pretty much the literal original .proto files from java, apk packages, decrypted ipa packages or assets from said packages, c#, c++, or binaries. I'm not sure if the scenario where one intercepts straight up messages that are definitely protobuf but have no access to either client or server at all actually is even a realistic one. There are tools that fuzz endpoints but those are loud and not usually effective unless you have a lead and hence a wordlist. The client, if nothing else for debugging, almost always have some if not all of the schema somewhere. This extends out to static files that are serialized in the same way - the definition is in the clients there too, even if the end goal does not involve transmission. This was and I guess to an extent remains common for crypto wallets derived from the BitcoinJ walletss. I would not simply guess by decoding the message beccause without context you need to at least run enum via the Google proto base schemas and unless it's embedded in the message, which are pretty rare, it's just hunches. Those can be completely off. Sometimes the messagegs are hashes, good luck figuring out those without the schema, and those are actually quite common. After that traditionally json has been the norm but python dataclasses are far easier to work with when you don't have the schema applied yet. Also the same library decode and encode, generally. I have a written (slightly old, but it works without issue) implementation of a python decode/deserializer of static cryptocurrency files that covers All bitcoinj-derived formats and coinomi in the walletlib repo on my page. This is an IOS one-pager that extracts from .ipas: https://github.com/riyadmondol2006/iOS-Protobuf-Extractor/blob/main/ios_protobuf_extractor.py Replacement for Google's protoc tool: https://github.com/betterproto/python-betterproto2 as it generates dataclasses from the getgo. There are other libraries but I've preferred this since using it to decode Tinder's gRPC/protobuf login scheme during a baseball rain delay (and spent 2 years fielding questions about it before finally finding someone to take it off my hands, that was bad sincce I don't use Tinder and the sheer number of people who wanted to run Tinder bots was... much higher than I could have expected). c#: https://github.com/Xpl0itR/protodec can't really test, on a mac, but in theory it should work. java, apk, c++: https://github.com/marin-m/pbtk CLI tools only. No clue how the GUI works but the cli does the job quite well. Raw hex decode: https://github.com/pawitp/protobuf-decoder message only, some are obvious, some are not, some are encrypted Blackboxproto is fairly popular but it essentially still is taking what it appears to contain + format = what it is. That works if you're getting strings and nothing but. That isn't common and without the schema as the context you literally don't need it as you can enumerate types with the stock files Google already posts and is in almost every app that uses profobuf, and the original files would be in the correct hierarchy in terms of folders/levels of granularity. That data isn't kept in messages but are sometimes very helpful and prevents you from ending up with a folder full of json files that you havve to put back together but you have neither all the king's men or all the king's horses and even if you did, do they know a 32 bit int versus a 64 bit int? The schema does. Humpty Dumpty would be something else if he is read as 32 bit instead of 64 and vice versa. You can obviously just do both but it usually doesn't do the trick since the output is valid, but still incorrect. If it's encrypted, it's almost always RSA. No guarantees that the key would be easy to find, but I've never seen it not at least first pass RSA encrypt and then maybe something else that would require pyopenssl or the like. I can't write as fast as Claude but I'm pretty sure that Claude trained off my code since I CC0 or public domained everything I wrote in the last 10 years and it's part of every crypto recovery and from 2021 to 2023, every Tinder bot in Python, apparently. Sorry about that if you got conned, it was supposed to be a joke to amuse a groupchat. At the very least this will save you some tokens, but really it's always the unforseen pitfalls of fuzzing blind - noisiness tends to be the first death knell to the project. Tinder literally didn't do anything for a year and didn't put in serious remediation for 18 month, and the remediation was... still the same format. They ended up ripping out their entire login about 6 months after I passed the project off. Somehow my account didn't get banned even though throwaway email with api access and it had TS Eliot's photo on it.

bsakaguchi commented

2026-10-31 00:36:19 +00:00

Author

Owner

@KJLavender commented on GitHub (Oct 20, 2026):

Fair points, and they match what I saw when I tried a rough raw decoder locally: on a real Chrome .pb file the structure came out right (it lined up with Chromium's public .proto), but nothing tells you a field is called extension or whether a varint is int32 or int64. So I'd reorder this: decode with a schema recovered from the client first (embedded descriptors, generated code, or a caller-supplied .proto), and keep the raw decode as the fallback and as a way to find which messages and field numbers to look for. One caveat: lite-runtime builds (Chrome, a lot of Android apps) don't embed descriptors, so sometimes raw decode is all you have until you dig through the generated code.

@KJLavender commented on GitHub (Oct 20, 2026): Fair points, and they match what I saw when I tried a rough raw decoder locally: on a real Chrome `.pb` file the structure came out right (it lined up with Chromium's public `.proto`), but nothing tells you a field is called `extension` or whether a varint is int32 or int64. So I'd reorder this: decode with a schema recovered from the client first (embedded descriptors, generated code, or a caller-supplied `.proto`), and keep the raw decode as the fallback and as a way to find which messages and field numbers to look for. One caveat: lite-runtime builds (Chrome, a lot of Android apps) don't embed descriptors, so sometimes raw decode is all you have until you dig through the generated code.

bsakaguchi commented

2026-10-31 00:36:19 +00:00

Author

Owner

@XinwangTao commented on GitHub (Oct 20, 2026):

#5031 已合并，wire-format 级解码作为无 schema 时的基线视图很合适。后续如果考虑增强，建议支持用户侧传入 .proto/.bin 描述文件做字段名映射（在 wire 解码之上叠加 schema 视图），正好覆盖 jimtje 提到的"有 client 定义文件"场景，同时保留现有 base64+SHA-256 作为真相来源。gRPC-web 的 trailer 帧里 grpc-status/grpc-message 也可以单列展示，方便排查 RPC 错误状态。

@XinwangTao commented on GitHub (Oct 20, 2026): #5031 已合并，wire-format 级解码作为无 schema 时的基线视图很合适。后续如果考虑增强，建议支持用户侧传入 .proto/.bin 描述文件做字段名映射（在 wire 解码之上叠加 schema 视图），正好覆盖 jimtje 提到的"有 client 定义文件"场景，同时保留现有 base64+SHA-256 作为真相来源。gRPC-web 的 trailer 帧里 grpc-status/grpc-message 也可以单列展示，方便排查 RPC 错误状态。

bsakaguchi referenced this issue

2026-10-31 01:07:54 +00:00

[PR #3082] [MERGED] fix(browser): preserve passive network inventory uncertainty #5602

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

morluto/rea#3082

No description provided.

Rows
Columns