[GH-ISSUE #4414] [Bug][TransferEngine] tebench labels consistency WRITE/READ metrics as READ-only #1111

New issue

Closed

opened 2026-10-29 21:23:00 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-29 21:23:00 +00:00

Owner

Originally created by @feichai0017 on GitHub (Sep 30, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/4414

Bug report

At main c3fa13ecaf1038df933dcb784ff8ffe026381100, tebench's per-target JSON records "op_type":"read" for --op_type=read --check_consistency=true, although consistency mode executes WRITE followed by READ and pools both operations' samples. The report can therefore be mistaken for READ-only latency/throughput.

This is a benchmark labeling defect, not a transfer-engine data-integrity failure.

Environment and reproduction

Linux x86_64; unchanged upstream tebench, USE_TENT=ON, USE_CUDA=OFF; two native processes, host DRAM, real loopback standard TCP. No OrbitKV wrapper participates.

Build and create the explicit TCP configuration using the commands in #4413 (keep max_concurrent_tasks=1 on both sides). In the same Bash setup, start the bounded target on 19460, then run:

MC_TENT_CONF="$PWD/tcp-1.json" timeout -k 2s 15s "$bin" "${common[@]}" \
  --rpc_server_port=19461 --target_seg_name=127.0.0.1:19460 \
  --op_type=read --check_consistency=true \
  --result_output_jsonl="$PWD/result.jsonl"
cat result.jsonl

Use a new output file for each case; JSONL appends. A READ-only control uses --check_consistency=false.

Observed with 4 KiB blocks, batch 1, one worker, one second measurement after warmup:

{"backend":"tent","op_type":"read","block_size":4096,"batch_size":1,"num_threads":1,"aggregate_operations":15264,"aggregate_transferred_bytes":62521344}

This is an excerpt of the unmodified result row. Both processes exited 0; binary/library/configuration SHA256 checks matched before and after. The runtime observation establishes the JSON label; the deterministic source branch below establishes the effective operation sequence. No packet-level opcode trace is claimed.

Cause and expected behavior

main.cpp:59: consistency forces mixed_opcode=true.
main.cpp:172: the mixed path submits WRITE then READ; each completed batch contributes its own sample.
main.cpp:263: the report receives the original requested XferBenchConfig::op_type, which target_metrics serializes unchanged.

Expected: report effective mode mix when consistency forces the mixed path, and explain that behavior in the flag help/documentation. Preserve true READ-only and WRITE-only labels when consistency is off. No change to the consistency algorithm is needed.

Operations count completed transfer batches. The mixed latency distribution is per WRITE or READ batch, not the combined WRITE+READ pair or a business request. No byte-count error is claimed.

Version comparison and duplicate search

v0.3.13.post1 (719735896c86b56fabec6cf3e825fb2ea640597a) already forced mixed mode but did not contain the per-target JSON feature. #3779 introduced that reporting surface. Only TENT was exercised here; classic shares this code but was not runtime-tested.

Searched issues and open/closed PRs for tebench/op_type/check_consistency/JSONL. #3931 improves HP TCP patterns; merged #4260 adds submit/wait logs; neither addresses the effective-mode label. No exact fix was found in that search.

AI assistance: OpenAI Codex assisted with source inspection and the isolated reproduction.

Originally created by @feichai0017 on GitHub (Sep 30, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/4414 ## Bug report At main `c3fa13ecaf1038df933dcb784ff8ffe026381100`, tebench's per-target JSON records `"op_type":"read"` for `--op_type=read --check_consistency=true`, although consistency mode executes WRITE followed by READ and pools both operations' samples. The report can therefore be mistaken for READ-only latency/throughput. This is a benchmark labeling defect, not a transfer-engine data-integrity failure. ### Environment and reproduction Linux x86_64; unchanged upstream tebench, `USE_TENT=ON`, `USE_CUDA=OFF`; two native processes, host DRAM, real loopback standard TCP. No OrbitKV wrapper participates. Build and create the explicit TCP configuration using the commands in #4413 (keep `max_concurrent_tasks=1` on both sides). In the same Bash setup, start the bounded target on 19460, then run: ```bash MC_TENT_CONF="$PWD/tcp-1.json" timeout -k 2s 15s "$bin" "${common[@]}" \ --rpc_server_port=19461 --target_seg_name=127.0.0.1:19460 \ --op_type=read --check_consistency=true \ --result_output_jsonl="$PWD/result.jsonl" cat result.jsonl ``` Use a new output file for each case; JSONL appends. A READ-only control uses `--check_consistency=false`. Observed with 4 KiB blocks, batch 1, one worker, one second measurement after warmup: ```json {"backend":"tent","op_type":"read","block_size":4096,"batch_size":1,"num_threads":1,"aggregate_operations":15264,"aggregate_transferred_bytes":62521344} ``` This is an excerpt of the unmodified result row. Both processes exited 0; binary/library/configuration SHA256 checks matched before and after. The runtime observation establishes the JSON label; the deterministic source branch below establishes the effective operation sequence. No packet-level opcode trace is claimed. ### Cause and expected behavior - [main.cpp:59](https://github.com/kvcache-ai/Mooncake/blob/c3fa13ecaf1038df933dcb784ff8ffe026381100/mooncake-transfer-engine/benchmark/main.cpp#L59): consistency forces `mixed_opcode=true`. - [main.cpp:172](https://github.com/kvcache-ai/Mooncake/blob/c3fa13ecaf1038df933dcb784ff8ffe026381100/mooncake-transfer-engine/benchmark/main.cpp#L172): the mixed path submits WRITE then READ; each completed batch contributes its own sample. - [main.cpp:263](https://github.com/kvcache-ai/Mooncake/blob/c3fa13ecaf1038df933dcb784ff8ffe026381100/mooncake-transfer-engine/benchmark/main.cpp#L263): the report receives the original requested `XferBenchConfig::op_type`, which target_metrics serializes unchanged. Expected: report effective mode `mix` when consistency forces the mixed path, and explain that behavior in the flag help/documentation. Preserve true READ-only and WRITE-only labels when consistency is off. No change to the consistency algorithm is needed. Operations count completed transfer batches. The mixed latency distribution is per WRITE or READ batch, not the combined WRITE+READ pair or a business request. No byte-count error is claimed. ### Version comparison and duplicate search v0.3.13.post1 (`719735896c86b56fabec6cf3e825fb2ea640597a`) already forced mixed mode but did not contain the per-target JSON feature. #3779 introduced that reporting surface. Only TENT was exercised here; classic shares this code but was not runtime-tested. Searched issues and open/closed PRs for tebench/op_type/check_consistency/JSONL. #3931 improves HP TCP patterns; merged #4260 adds submit/wait logs; neither addresses the effective-mode label. No exact fix was found in that search. AI assistance: OpenAI Codex assisted with source inspection and the isolated reproduction.

bsakaguchi closed this issue

2026-10-29 21:23:00 +00:00

bsakaguchi commented

2026-10-29 21:23:01 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Sep 30, 2026):

Thanks for opening this issue, @feichai0017!

Field	Value
Issue	#4414
GitHub user ID	`132118805`
Reporter	@feichai0017

A maintainer will triage this when possible. To help us respond faster, please include:

Mooncake version or commit SHA
Environment (OS, CUDA/driver, RDMA stack if relevant)
Steps to reproduce and expected vs. actual behavior

Useful links: Documentation · Contributing guide

This message was posted automatically by the issue bot.

@github-actions[bot] commented on GitHub (Sep 30, 2026): Thanks for opening this issue, @feichai0017! | Field | Value | |-------|-------| | **Issue** | #4414 | | **GitHub user ID** | `132118805` | | **Reporter** | @feichai0017 | A maintainer will triage this when possible. To help us respond faster, please include: - Mooncake version or commit SHA - Environment (OS, CUDA/driver, RDMA stack if relevant) - Steps to reproduce and expected vs. actual behavior Useful links: [Documentation](https://kvcache-ai.github.io/Mooncake/) · [Contributing guide](https://github.com/kvcache-ai/Mooncake/blob/main/CONTRIBUTING.md) > This message was posted automatically by the issue bot.

bsakaguchi referenced this issue

2026-10-29 21:32:09 +00:00

[PR #1111] [MERGED] Update CI #2019