[GH-ISSUE #1403 ] [RFC]: Mooncake KV-Store Indexer API Standardization #449

Author

Owner

@doujiang24 commented on GitHub (Jan 21, 2026):

medium in input seems unnecessary? please show more details about the usage of the field.
keep simple as start could be better.

@doujiang24 commented on GitHub (Jan 21, 2026): `medium` in input seems unnecessary? please show more details about the usage of the field. keep simple as start could be better.

bsakaguchi commented

Author

Owner

@yejj710 commented on GitHub (Jan 21, 2026):

medium in input seems unnecessary? please show more details about the usage of the field. keep simple as start could be better.

Thanks for comment~ medium is an optional arg. I've submitted a pr that includes more detailed explanations.

@yejj710 commented on GitHub (Jan 21, 2026): > `medium` in input seems unnecessary? please show more details about the usage of the field. keep simple as start could be better. Thanks for comment~ medium is an optional arg. I've submitted a [pr](https://github.com/kvcache-ai/Mooncake/pull/1416) that includes more detailed explanations.

bsakaguchi commented

Author

Owner

@PeaBrane commented on GitHub (Jan 30, 2026):

@yejj710 several comments / questions

Should we also standardize here the hashing algo? I think each engine framework hashes their KV events differently, so we may also want to discuss a deterministic hashing algo here (sha256) as the universal contract.
If passing by raw tokens, where would the hashing happen, is it inside the KV indexer service? This may make the indexer service itself not super scalable. I wonder if we should just not support raw tokens as input altogether.
What is the mechanism here for each engine registering their engine_name to the KvIndexer service? Would there be any dynamic discovery and heartbeat mechanism here?
I think we should just return matched_blocks instead of matched_tokens and return block_size as well explicitly. Also, matched_tokens here is a bit ambiguous; is it the max matched tokens across all dp instances and memory tiers?
Should we have multi-tenant isolation, e.g. another field such as namespace or tenant_id. It also prevents a super large output, if the user is only interested in a subset of instances.

@PeaBrane commented on GitHub (Jan 30, 2026): @yejj710 several comments / questions 1. Should we also standardize here the hashing algo? I think each engine framework hashes their KV events differently, so we may also want to discuss a deterministic hashing algo here (sha256) as the universal contract. 2. If passing by raw tokens, where would the hashing happen, is it inside the KV indexer service? This may make the indexer service itself not super scalable. I wonder if we should just not support raw tokens as input altogether. 3. What is the mechanism here for each engine registering their `engine_name` to the KvIndexer service? Would there be any dynamic discovery and heartbeat mechanism here? 4. I think we should just return `matched_blocks` instead of `matched_tokens` and return `block_size` as well explicitly. Also, `matched_tokens` here is a bit ambiguous; is it the max matched tokens across all dp instances and memory tiers? 5. Should we have multi-tenant isolation, e.g. another field such as `namespace` or `tenant_id`. It also prevents a super large output, if the user is only interested in a subset of instances.

bsakaguchi commented

Author

Owner

@doujiang24 commented on GitHub (Jan 30, 2026):

Hi @PeaBrane

I think we should just return matched_blocks instead of matched_tokens and return block_size as well explicitly. Also, matched_tokens here is a bit ambiguous; is it the max matched tokens across all dp instances and memory tiers?

yeah, matched_tokens is confused, longest_matched could be better, as discussed in:
https://github.com/kvcache-ai/Mooncake/pull/1416/changes#diff-43ffc2328e105132991240dc43eb6e4186b327788430b48bf55e1b0b3d804802R61

the number of longest prefix matched token among all mediums.

Is there some benifits that router aware of the block_size? I haven't figured it out.
If there's no benefit, maybe tokens would be better?

@doujiang24 commented on GitHub (Jan 30, 2026): Hi @PeaBrane > 4. I think we should just return `matched_blocks` instead of `matched_tokens` and return `block_size` as well explicitly. Also, `matched_tokens` here is a bit ambiguous; is it the max matched tokens across all dp instances and memory tiers? yeah, matched_tokens is confused, `longest_matched` could be better, as discussed in: https://github.com/kvcache-ai/Mooncake/pull/1416/changes#diff-43ffc2328e105132991240dc43eb6e4186b327788430b48bf55e1b0b3d804802R61 > the number of longest prefix matched token among all mediums. Is there some benifits that router aware of the `block_size`? I haven't figured it out. If there's no benefit, maybe `tokens` would be better?

bsakaguchi commented

Author

Owner

@PeaBrane commented on GitHub (Jan 30, 2026):

@doujiang24 I think it's just for better observability, not strictly needed. The other points may be more important.

On an unrelated note, that may also be a need for a extra_id defaulting to None (or generally an additional optional u64) that translates to a salt hash in the indexer. That would be potentially beneficial for, say, a promise to keep hash lookups of different "customers" separate.

@PeaBrane commented on GitHub (Jan 30, 2026): @doujiang24 I think it's just for better observability, not strictly needed. The other points may be more important. On an unrelated note, that may also be a need for a extra_id defaulting to None (or generally an additional optional u64) that translates to a salt hash in the indexer. That would be potentially beneficial for, say, a promise to keep hash lookups of different "customers" separate.

bsakaguchi commented

Author

Owner

@yejj710 commented on GitHub (Jan 30, 2026):

@yejj710 several comments / questions

@PeaBrane Thank you for your comments. Here are some of my thoughts:

q1: Standardizing hashing algo brings many benefits, such as reducing the network overhead caused by passing token ids. The challenge lies in promoting different communities to adopt this practice. This involves: 1. Initializing root_hash in the inference framework, 2. Hashing algorithms, and 3. block_hash calculation in kv-events (vllm's kv-events use the maybe_convert_block_hash function by default to convert the hashed result as the block_hash value).

q2: The KV indexer will calculates the hash based on tokens (during the /query API). I think for a service, supporting more APIs might not be a bad thing?

q3: Under the current design, service discovery and heartbeat support are required, and I think that the service registration of the indexer is handled by the router. Therefore, engine_name is merely an alias for a inference service instance, which facilitates retrieval and look-up operations. I plan to first implement this through a simple /register API, with preliminary ideas like:

{
    "engine_name": "xxx",
    "engine_ip": "127.0.0.1",
    "engine_port": 8000,
    "kv_event_port": 5557,
    "service_type": "vllm/sglang..",
    "model_name": "deepseek",
    "lora_id": -1
}

Actually, I’m not familiar with this part. Do you have a better design in mind?

q4: This has been changed to longest_matched, meaning exactly as you described: the maximum number of hits across all data parallel instances and memory levels. I’ll make the updates later.

q5: The multi-tenant scenario is reasonable, and I prefer using the term tenant_id

@yejj710 commented on GitHub (Jan 30, 2026): > [@yejj710](https://github.com/yejj710) several comments / questions @PeaBrane Thank you for your comments. Here are some of my thoughts: q1: Standardizing hashing algo brings many benefits, such as reducing the network overhead caused by passing token ids. The challenge lies in promoting different communities to adopt this practice. This involves: 1. Initializing root_hash in the inference framework, 2. Hashing algorithms, and 3. block_hash calculation in kv-events (vllm's kv-events use the `maybe_convert_block_hash` function by default to convert the hashed result as the block_hash value). q2: The KV indexer will calculates the hash based on tokens (during the `/query` API). I think for a service, supporting more APIs might not be a bad thing? q3: Under the current design, service discovery and heartbeat support are required, and I think that the service registration of the indexer is handled by the router. Therefore, `engine_name` is merely an alias for a inference service instance, which facilitates retrieval and look-up operations. I plan to first implement this through a simple `/register` API, with preliminary ideas like: ```json { "engine_name": "xxx", "engine_ip": "127.0.0.1", "engine_port": 8000, "kv_event_port": 5557, "service_type": "vllm/sglang..", "model_name": "deepseek", "lora_id": -1 } ``` Actually, I’m not familiar with this part. Do you have a better design in mind? q4: This has been changed to `longest_matched`, meaning exactly as you described: the maximum number of hits across all data parallel instances and memory levels. I’ll make the updates later. q5: The multi-tenant scenario is reasonable, and I prefer using the term `tenant_id`

bsakaguchi commented

Author

Owner

@PeaBrane commented on GitHub (Jan 30, 2026):

@yejj710 Thank you.

The challenge lies in promoting different communities to adopt this practice.

Yea that is the unfortunate part, so I think this is a good opportunity to set a community contract for them to follow. At the minimum, to even use the /query-by-hash, there would be an assumption of the hashing algo used for the blocks, iiuc, and I think it would be good to define that here.

In addition, internally, I'm guessing the KvIndexer would need to rely on a unique hash for each sequence block, which I'm assuming would need to be a rolling hash of the block hashes (calling it sequence hash from here on). This is something the engines could also compute themselves and emit as KvEvents (as currently done), and we don't technically need to control that. But if we can enforce that rolling hash algo for the seq hashes as well, it would enable a lot of optimization opportunities for the indexer itself. So it would be nice to define/standardize that here as well.

For reference, dynamo uses xxh3-64 for both block hash and sequence hash computations, with a seed of 1337. We can standardize something like this.

I think for a service, supporting more APIs might not be a bad thing?

I think I'm fine with that

Do you have a better design in mind?

I'm not an expert, but can refer to the dynamo discovery and mechanism. But API wise, I think /register should be fine.

@PeaBrane commented on GitHub (Jan 30, 2026): @yejj710 Thank you. > The challenge lies in promoting different communities to adopt this practice. Yea that is the unfortunate part, so I think this is a good opportunity to set a community contract for them to follow. At the minimum, to even use the /query-by-hash, there would be an assumption of the hashing algo used for the blocks, iiuc, and I think it would be good to define that here. In addition, internally, I'm guessing the KvIndexer would need to rely on a unique hash for each sequence block, which I'm assuming would need to be a rolling hash of the block hashes (calling it sequence hash from here on). This is something the engines could also compute themselves and emit as KvEvents (as currently done), and we don't technically need to control that. But if we **can** enforce that rolling hash algo for the seq hashes as well, it would enable a lot of optimization opportunities for the indexer itself. So it would be nice to define/standardize that here as well. [For reference](https://github.com/ai-dynamo/dynamo/blob/f6d6b34e87aa391051c7740025d4a00122a8f859/lib/kv-router/src/protocols.rs#L26), dynamo uses xxh3-64 for both block hash and sequence hash computations, with a seed of 1337. We can standardize something like this. > I think for a service, supporting more APIs might not be a bad thing? I think I'm fine with that > Do you have a better design in mind? I'm not an expert, but can refer to the [dynamo discovery](https://github.com/ai-dynamo/dynamo/blob/main/lib/runtime/src/discovery/mod.rs) and mechanism. But API wise, I think `/register` should be fine.

bsakaguchi commented

Author

Owner

@doujiang24 commented on GitHub (Jan 31, 2026):

Should we also standardize here the hashing algo? I think each engine framework hashes their KV events differently, so we may also want to discuss a deterministic hashing algo here (sha256) as the universal contract.

If passing by raw tokens, where would the hashing happen, is it inside the KV indexer service? This may make the indexer service itself not super scalable. I wonder if we should just not support raw tokens as input altogether.

Agreed, a standard hashing algo could be better. xxh3-64 used in dynamo sounds like a good choice.

My thoughts:

as @PeaBrane said, standardize the hashing algo as a proposal, push sglang/vllm to following the hashing algo.
keep the token_ids in kv-events and using the standard hashing algo in kv-indexer as starting, before the engines not following the standard hashing algo.

@PeaBrane @yejj710 Thoughts?

@doujiang24 commented on GitHub (Jan 31, 2026): > 1. Should we also standardize here the hashing algo? I think each engine framework hashes their KV events differently, so we may also want to discuss a deterministic hashing algo here (sha256) as the universal contract. > 2. If passing by raw tokens, where would the hashing happen, is it inside the KV indexer service? This may make the indexer service itself not super scalable. I wonder if we should just not support raw tokens as input altogether. Agreed, a standard hashing algo could be better. xxh3-64 used in dynamo sounds like a good choice. My thoughts: 1. as @PeaBrane said, standardize the hashing algo as a proposal, push sglang/vllm to following the hashing algo. 2. keep the token_ids in kv-events and using the standard hashing algo in kv-indexer as starting, before the engines not following the standard hashing algo. @PeaBrane @yejj710 Thoughts?

bsakaguchi commented

Author

Owner

@doujiang24 commented on GitHub (Jan 31, 2026):

What is the mechanism here for each engine registering their engine_name to the KvIndexer service? Would there be any dynamic discovery and heartbeat mechanism here?

yeah, engine_name is confusing.
publisher-unique-name might be better, it is a unique name for a kv-events publisher endpoint in the engine side.

Also, router and kv-indexer shared the same dynamic discovery mechanism, so the router also know the publisher-unique-name, then it can map the publisher-unique-name to the endpoint(ip:port) for inferencing, and other metadata etc.

If I remember correctly, each dp-rank have a unique kvevents zmq publisher endpoint.

And for the dynamic discovery mechanism, It should be extensible, as there may be multiple different implementations.

@doujiang24 commented on GitHub (Jan 31, 2026): > 3. What is the mechanism here for each engine registering their `engine_name` to the KvIndexer service? Would there be any dynamic discovery and heartbeat mechanism here? yeah, `engine_name` is confusing. `publisher-unique-name` might be better, it is a unique name for a kv-events publisher endpoint in the engine side. Also, router and kv-indexer shared the same dynamic discovery mechanism, so the router also know the `publisher-unique-name`, then it can map the `publisher-unique-name` to the endpoint(`ip:port`) for inferencing, and other metadata etc. If I remember correctly, each dp-rank have a unique kvevents zmq publisher endpoint. And for the dynamic discovery mechanism, It should be extensible, as there may be multiple different implementations.

bsakaguchi commented

Author

Owner

@doujiang24 commented on GitHub (Jan 31, 2026):

Should we have multi-tenant isolation, e.g. another field such as namespace or tenant_id. It also prevents a super large output, if the user is only interested in a subset of instances.

Doe the multi-tenant means

two users shared the same engine instances, but they do not want to share the kvcache.
two users use the same model in the inference API and /query API, but they have two group of engine instances, so we need the tenant_id in the /query API?

For 1: it might not be reasonable.
For 2: it might make sense, it could be an option field, not a requirement. As a use case, we do use different model when users do not want to share engine instances.

@doujiang24 commented on GitHub (Jan 31, 2026): > 5. Should we have multi-tenant isolation, e.g. another field such as `namespace` or `tenant_id`. It also prevents a super large output, if the user is only interested in a subset of instances. Doe the multi-tenant means 1. two users shared the same engine instances, but they do not want to share the kvcache. 2. two users use the same `model` in the inference API and `/query` API, but they have two group of engine instances, so we need the `tenant_id` in the `/query` API? For 1: it might not be reasonable. For 2: it might make sense, it could be an option field, not a requirement. As a use case, we do use different `model` when users do not want to share engine instances.

bsakaguchi commented

Author

Owner

@doujiang24 commented on GitHub (Jan 31, 2026):

On an unrelated note, that may also be a need for a extra_id defaulting to None (or generally an additional optional u64) that translates to a salt hash in the indexer. That would be potentially beneficial for, say, a promise to keep hash lookups of different "customers" separate.

Does it mean: two users shared the same engine instances, but they do not want to share the kvcache?

Or is this some kind of security consideration?

@doujiang24 commented on GitHub (Jan 31, 2026): > On an unrelated note, that may also be a need for a extra_id defaulting to None (or generally an additional optional u64) that translates to a salt hash in the indexer. That would be potentially beneficial for, say, a promise to keep hash lookups of different "customers" separate. Does it mean: two users shared the same engine instances, but they do not want to share the kvcache? Or is this some kind of security consideration?

bsakaguchi commented

Author

Owner

@yejj710 commented on GitHub (Jan 31, 2026):

@PeaBrane thanks

On an unrelated note, that may also be a need for a extra_id defaulting to None that translates to a salt hash in the indexer

The idea of salt hash sounds like a very reasonable use case. I will add the cache_salt attribute for this scenario.

At the minimum, to even use the /query-by-hash, there would be an assumption of the hashing algo used for the blocks, iiuc;
I'm guessing the KvIndexer would need to rely on a unique hash for each sequence block, which I'm assuming would need to be a rolling hash of the block hashes

yeah, that's how the code is currently implemented. Maybe starting with standardization between KvIndexer and the router would be a good first step.

@yejj710 commented on GitHub (Jan 31, 2026): @PeaBrane thanks > On an unrelated note, that may also be a need for a extra_id defaulting to None that translates to a salt hash in the indexer The idea of salt hash sounds like a very reasonable use case. I will add the `cache_salt` attribute for this scenario. > At the minimum, to even use the /query-by-hash, there would be an assumption of the hashing algo used for the blocks, iiuc; > I'm guessing the KvIndexer would need to rely on a unique hash for each sequence block, which I'm assuming would need to be a rolling hash of the block hashes yeah, that's how the code is currently implemented. Maybe starting with standardization between `KvIndexer` and the `router` would be a good first step.

bsakaguchi commented

Author

Owner

@yejj710 commented on GitHub (Jan 31, 2026):

as @PeaBrane said, standardize the hashing algo as a proposal, push sglang/vllm to following the hashing algo.

keep the token_ids in kv-events and using the standard hashing algo in kv-indexer as starting, before the engines not following the standard hashing algo.

Totally agree

publisher-unique-name might be better, it is a unique name for a kv-events publisher endpoint in the engine side.
it can map the publisher-unique-name to the endpoint(ip:port) for inferencing,

This name is clearer than engine_name, but there is another issue to consider: under Hybrid DPLB and Internal DPLB deployment modes, the router cannot accurately dispatch requests to specific DP ranks, and thus cannot maintain this map. In other words, does the router need to consider both the Hybrid DPLB and Internal DPLB deployment modes?
@doujiang24

If I remember correctly, each dp-rank have a unique kvevents zmq publisher endpoint.

Yes, the specific implementation details in vLLM can be viewed at: https://github.com/vllm-project/vllm/blob/main/vllm/distributed/kv_events.py#L294

@yejj710 commented on GitHub (Jan 31, 2026): > 1. as [@PeaBrane](https://github.com/PeaBrane) said, standardize the hashing algo as a proposal, push sglang/vllm to following the hashing algo. > 2. keep the token_ids in kv-events and using the standard hashing algo in kv-indexer as starting, before the engines not following the standard hashing algo. Totally agree >`publisher-unique-name` might be better, it is a unique name for a kv-events publisher endpoint in the engine side. >it can map the publisher-unique-name to the endpoint(ip:port) for inferencing, This name is clearer than `engine_name`, but there is another issue to consider: under Hybrid DPLB and Internal DPLB deployment modes, the router cannot accurately dispatch requests to specific DP ranks, and thus cannot maintain this map. In other words, does the router need to consider both the Hybrid DPLB and Internal DPLB deployment modes? @doujiang24 >If I remember correctly, each dp-rank have a unique kvevents zmq publisher endpoint. Yes, the specific implementation details in vLLM can be viewed at: https://github.com/vllm-project/vllm/blob/main/vllm/distributed/kv_events.py#L294

bsakaguchi commented

Author

Owner

@PeaBrane commented on GitHub (Jan 31, 2026):

@doujiang24 @yejj710 Thanks all for the discussions. I think we are basically aligned now.

keep the token_ids in kv-events and using the standard hashing algo in kv-indexer as starting, before the engines not following the standard hashing algo.

Yes, exactly. We can internally compute the correct seq hash with token_ids info at current. But once the hashing algo is enforced, token_ids can be removed (which would save a lot of BW on the publisher / relay end). Additionally, could also have the engine emit block_idx as well (position of the block in the seq), but that is not strictly needed, but would make the indexer's life easier.

[for multi-tenant], two users use the same model in the inference API and /query API, but they have two group of engine instances,

Yes this is right, this would prevent the user from getting a huge output from the indexer response if they are only interested in a group of deployments.

[for the salt hash], is this some kind of security consideration?

Yes it's something like that. Technically, if the indexer is implemented correctly, we would always have logical separation of blocks, e.g. a customer cannot easily query the blocks of another customer (or get hints about them). This additional salt_cache is more as an additional guarantee that we are really "keeping the blocks of separate customers separate", so there is less surface of bugs that may introduce security vunerabilites etc. Additionally, if customer A is bloating up its radix tree really badly, it would not affect the radix traversal of customer B at all (but this gets into indexer impl details now).

@PeaBrane commented on GitHub (Jan 31, 2026): @doujiang24 @yejj710 Thanks all for the discussions. I think we are basically aligned now. > keep the token_ids in kv-events and using the standard hashing algo in kv-indexer as starting, before the engines not following the standard hashing algo. Yes, exactly. We can internally compute the correct seq hash with `token_ids` info at current. But once the hashing algo is enforced, `token_ids` can be removed (which would save a lot of BW on the publisher / relay end). Additionally, could also have the engine emit `block_idx` as well (position of the block in the seq), but that is not strictly needed, but would make the indexer's life easier. > [for multi-tenant], two users use the same model in the inference API and /query API, but they have two group of engine instances, Yes this is right, this would prevent the user from getting a huge output from the indexer response if they are only interested in a group of deployments. > [for the salt hash], is this some kind of security consideration? Yes it's something like that. Technically, if the indexer is implemented correctly, we would always have **logical** separation of blocks, e.g. a customer cannot easily query the blocks of another customer (or get hints about them). This additional salt_cache is more as an additional guarantee that we are really "keeping the blocks of separate customers separate", so there is less surface of bugs that may introduce security vunerabilites etc. Additionally, if customer A is bloating up its radix tree really badly, it would not affect the radix traversal of customer B at all (but this gets into indexer impl details now).

bsakaguchi commented

Author

Owner

@doujiang24 commented on GitHub (Jan 31, 2026):

This name is clearer than engine_name, but there is another issue to consider: under Hybrid DPLB and Internal DPLB deployment modes, the router cannot accurately dispatch requests to specific DP ranks, and thus cannot maintain this map.

@yejj710 Could you please describe the details?

In other words, does the router need to consider both the Hybrid DPLB and Internal DPLB deployment modes?

If the router could not determine the dp-rank, them the dp-rank info in kvevents is useless for router.

@doujiang24 commented on GitHub (Jan 31, 2026): > This name is clearer than `engine_name`, but there is another issue to consider: under Hybrid DPLB and Internal DPLB deployment modes, the router cannot accurately dispatch requests to specific DP ranks, and thus cannot maintain this map. @yejj710 Could you please describe the details? > In other words, does the router need to consider both the Hybrid DPLB and Internal DPLB deployment modes? If the router could not determine the dp-rank, them the dp-rank info in kvevents is useless for router.

bsakaguchi commented

Author

Owner

@doujiang24 commented on GitHub (Jan 31, 2026):

@doujiang24 @yejj710 Thanks all for the discussions. I think we are basically aligned now.

Sounds great.

[for the salt hash], is this some kind of security consideration?

Yes it's something like that. Technically, if the indexer is implemented correctly, we would always have logical separation of blocks, e.g. a customer cannot easily query the blocks of another customer (or get hints about them). This additional salt_cache is more as an additional guarantee that we are really "keeping the blocks of separate customers separate", so there is less surface of bugs that may introduce security vunerabilites etc. Additionally, if customer A is bloating up its radix tree really badly, it would not affect the radix traversal of customer B at all (but this gets into indexer impl details now).

In my opinion, each tenant_id + model + lora_name will have a seperated radix tree, customers are logical separation as expected.
But, I do agree that the salt could be a additional security improvement, and it should be optional, same as tenant_id, to make people easier to use.

@doujiang24 commented on GitHub (Jan 31, 2026): > [@doujiang24](https://github.com/doujiang24) [@yejj710](https://github.com/yejj710) Thanks all for the discussions. I think we are basically aligned now. Sounds great. > > [for the salt hash], is this some kind of security consideration? > > Yes it's something like that. Technically, if the indexer is implemented correctly, we would always have **logical** separation of blocks, e.g. a customer cannot easily query the blocks of another customer (or get hints about them). This additional salt_cache is more as an additional guarantee that we are really "keeping the blocks of separate customers separate", so there is less surface of bugs that may introduce security vunerabilites etc. Additionally, if customer A is bloating up its radix tree really badly, it would not affect the radix traversal of customer B at all (but this gets into indexer impl details now). In my opinion, each `tenant_id` + `model` + `lora_name` will have a seperated radix tree, customers are **logical** separation as expected. But, I do agree that the `salt` could be a additional security improvement, and it should be optional, same as `tenant_id`, to make people easier to use.

bsakaguchi commented

Author

Owner

@doujiang24 commented on GitHub (Jan 31, 2026):

Also, a question about the implementaion.

For a new kvevent, we are inserting it to the existing radix tree.
What should we do, when there are multiple blocks with the same hash matched the parent_block_hash in the kvevent?
Append the new block in kvevent to both of the multiple blocks?

@PeaBrane @yejj710 Thoughts?
So, the block_idx could be useful to filter the block.

@doujiang24 commented on GitHub (Jan 31, 2026): Also, a question about the implementaion. For a new kvevent, we are inserting it to the existing radix tree. What should we do, when there are multiple blocks with the same hash matched the `parent_block_hash` in the kvevent? Append the new block in kvevent to both of the multiple blocks? @PeaBrane @yejj710 Thoughts? So, the `block_idx` could be useful to filter the block.

bsakaguchi commented

Author

Owner

@PeaBrane commented on GitHub (Jan 31, 2026):

@doujiang24 I may have misunderstood the question, so please let me know.

Currently, I think for all 3 engines (vllm, sglang, trtllm), the behavior is that if there is a chain of blocks stored, there would be a kv event with a parent, then the list of blocks in sequential order. So the likely scenario here is a prefill event. And that sequential ordering is a strong assumption we make in the dynamo kv indexer. If we have the block_idx, then that sequential condition can be relaxed, and maybe the parent_block_hash info can also be unnecessary.

@PeaBrane commented on GitHub (Jan 31, 2026): @doujiang24 I may have misunderstood the question, so please let me know. Currently, I think for all 3 engines (vllm, sglang, trtllm), the behavior is that if there is a chain of blocks stored, there would be a kv event with a parent, then the list of blocks in sequential order. So the likely scenario here is a prefill event. And that sequential ordering is a strong assumption we make in the dynamo kv indexer. If we have the `block_idx`, then that sequential condition can be relaxed, and maybe the `parent_block_hash` info can also be unnecessary.

bsakaguchi commented

Author

Owner

@doujiang24 commented on GitHub (Feb 1, 2026):

@doujiang24 I may have misunderstood the question, so please let me know.

Currently, I think for all 3 engines (vllm, sglang, trtllm), the behavior is that if there is a chain of blocks stored, there would be a kv event with a parent, then the list of blocks in sequential order. So the likely scenario here is a prefill event. And that sequential ordering is a strong assumption we make in the dynamo kv indexer. If we have the block_idx, then that sequential condition can be relaxed, and maybe the parent_block_hash info can also be unnecessary.

Sorry @PeaBrane , I forgot an important detail, the question is how we handle hash collisions.

There must be a case where two different blocks have the same hash in the radix tree, as you mentioned, lookup[worker][parent_hash] in
https://github.com/ai-dynamo/dynamo/pull/5785/changes#diff-a35c64d704a831a90b2313bcc9005b21ef3f581e2b532487f2ce550959984194R121

@doujiang24 commented on GitHub (Feb 1, 2026): > [@doujiang24](https://github.com/doujiang24) I may have misunderstood the question, so please let me know. > > Currently, I think for all 3 engines (vllm, sglang, trtllm), the behavior is that if there is a chain of blocks stored, there would be a kv event with a parent, then the list of blocks in sequential order. So the likely scenario here is a prefill event. And that sequential ordering is a strong assumption we make in the dynamo kv indexer. If we have the `block_idx`, then that sequential condition can be relaxed, and maybe the `parent_block_hash` info can also be unnecessary. Sorry @PeaBrane , I forgot an important detail, the question is how we handle hash collisions. There must be a case where two different blocks have the same hash in the radix tree, as you mentioned, `lookup[worker][parent_hash]` in https://github.com/ai-dynamo/dynamo/pull/5785/changes#diff-a35c64d704a831a90b2313bcc9005b21ef3f581e2b532487f2ce550959984194R121

bsakaguchi commented

Author

Owner

@PeaBrane commented on GitHub (Feb 1, 2026):

@doujiang24 Ah, I see. I wasn't originally intending for block_idx to solve the hash collision problem (more to get rid of the radix structure). But you are right, this would indeed be an important additional benefit that I glossed over 😄

(Checking with Gemini, for a 64-bit hash, around ~5 billion blocks, hash collision prob becomes ~50%. Sounds within data-server inference scale, so in fact a legit concern. Glad you brought it up. More motivation for kv events to emit block_idx then)

@PeaBrane commented on GitHub (Feb 1, 2026): @doujiang24 Ah, I see. I wasn't originally intending for `block_idx` to solve the hash collision problem (more to get rid of the radix structure). But you are right, this would indeed be an important additional benefit that I glossed over 😄 (Checking with Gemini, for a 64-bit hash, around ~5 billion blocks, hash collision prob becomes ~50%. Sounds within data-server inference scale, so in fact a legit concern. Glad you brought it up. More motivation for kv events to emit `block_idx` then)

bsakaguchi commented

Author

Owner

@doujiang24 commented on GitHub (Feb 2, 2026):

Yep, two blocks with hash collision, will leads to two duplicated sub trees eventually (when we choose to append the new blocks to both of them).

@PeaBrane Two more questions:

Should we figure out a RFC about the kvevents, and align with Ishan?
For the implementation, seems you prefer NestedMap than RadixTree, right?

@doujiang24 commented on GitHub (Feb 2, 2026): Yep, two blocks with hash collision, will leads to two duplicated sub trees eventually (when we choose to append the new blocks to both of them). @PeaBrane Two more questions: 1. Should we figure out a RFC about the kvevents, and align with Ishan? 2. For the implementation, seems you prefer NestedMap than RadixTree, right?

bsakaguchi commented

Author

Owner

@PeaBrane commented on GitHub (Feb 2, 2026):

@doujiang24

Yes let's do that. We should continue the discussion of KvEvents elsewhere though. Do we already have an RFC page for KvEvents?
We are still prototyping and benchmarking. Not sure yet.

@PeaBrane commented on GitHub (Feb 2, 2026): @doujiang24 1. Yes let's do that. We should continue the discussion of KvEvents elsewhere though. Do we already have an RFC page for KvEvents? 2. We are still prototyping and benchmarking. Not sure yet.

bsakaguchi commented

Author

Owner

@doujiang24 commented on GitHub (Feb 2, 2026):

@PeaBrane Not entirely.
We are discussing the kvevents produced by Mooncake here: https://github.com/kvcache-ai/Mooncake/pull/1306/changes#diff-54037b18eb12d9bdb23a1116313dcef631ba2da82e33e9c02c69c47aef137ddcR107-R122

Would you like to create an RFC page, both in Mooncake, ai-dynamo or sglang/vllm is okay, we can refer it as always.

@doujiang24 commented on GitHub (Feb 2, 2026): @PeaBrane Not entirely. We are discussing the kvevents produced by Mooncake here: https://github.com/kvcache-ai/Mooncake/pull/1306/changes#diff-54037b18eb12d9bdb23a1116313dcef631ba2da82e33e9c02c69c47aef137ddcR107-R122 Would you like to create an RFC page, both in Mooncake, ai-dynamo or sglang/vllm is okay, we can refer it as always.

bsakaguchi commented

Author

Owner

@PeaBrane commented on GitHub (Feb 2, 2026):

@doujiang24 I'm ok with any as well. In mooncake is fine if it's easier. Would you like to take the lead on it?

@PeaBrane commented on GitHub (Feb 2, 2026): @doujiang24 I'm ok with any as well. In mooncake is fine if it's easier. Would you like to take the lead on it?

bsakaguchi commented

Author

Owner

@yejj710 commented on GitHub (Feb 2, 2026):

@doujiang24 Sorry for the late reply

So, the block_idx could be useful to filter the block

Thanks for the reminder. I hadn’t considered the hash collision issue before. Using block_idx to avoid collisions sounds reasonable.

@yejj710 Could you please describe the details?

we know that the llm API server is the service entry point of the engine and also the destination for router-dispatched requests. Suppose we have two nodes and want to run DP=4

Internal DPLB

# Node 0  (with ip address 10.99.48.128)
vllm serve $MODEL --data-parallel-size 4 --data-parallel-size-local 2 \
                  --data-parallel-address 10.99.48.128 --data-parallel-rpc-port 13345
# Node 1
vllm serve $MODEL --headless --data-parallel-size 4 --data-parallel-size-local 2 \
                  --data-parallel-start-rank 2 --data-parallel-address 10.99.48.128 --data-parallel-rpc-port 13345

Only Node 0 will launch the API server, so the router can only route requests to Node 0.

Hybrid DPLB

# Node 0  (with ip address 10.99.48.128)
vllm serve $MODEL --data-parallel-size 4 --data-parallel-size-local 2 \
                  --data-parallel-start-rank 0 --data-parallel-hybrid-lb \
                  --data-parallel-address 10.99.48.128 --data-parallel-rpc-port 13345
# Node 1
vllm serve $MODEL --data-parallel-size 4 --data-parallel-size-local 2 \
                  --data-parallel-start-rank 2 --data-parallel-hybrid-lb \
                  --data-parallel-address 10.99.48.128 --data-parallel-rpc-port 13345

Each node will launch an API server, and DPLB is performed within the node. In other words, the router can schedule tasks to the node but cannot specify which DP (except for nodes that deploy a single DP rank) within the node to schedule to.

Based on the above, we need to provide a unique identifier for each API server (I've tentatively named it engine_name).
Therefore, I’d like to align with everyone on whether we need to consider the two special scenarios of hybrid DPLB and internal DPLB. The foundational setup is very important. What is your opinion? @PeaBrane @doujiang24

@yejj710 commented on GitHub (Feb 2, 2026): @doujiang24 Sorry for the late reply > So, the `block_idx` could be useful to filter the block Thanks for the reminder. I hadn’t considered the hash collision issue before. Using `block_idx` to avoid collisions sounds reasonable. > @yejj710 Could you please describe the details? we know that the llm `API server` is the service entry point of the engine and also the destination for router-dispatched requests. Suppose we have two nodes and want to run DP=4 * Internal DPLB ```sh # Node 0 (with ip address 10.99.48.128) vllm serve $MODEL --data-parallel-size 4 --data-parallel-size-local 2 \ --data-parallel-address 10.99.48.128 --data-parallel-rpc-port 13345 # Node 1 vllm serve $MODEL --headless --data-parallel-size 4 --data-parallel-size-local 2 \ --data-parallel-start-rank 2 --data-parallel-address 10.99.48.128 --data-parallel-rpc-port 13345 ``` Only Node 0 will launch the API server, so the router can only route requests to Node 0. * Hybrid DPLB ```sh # Node 0 (with ip address 10.99.48.128) vllm serve $MODEL --data-parallel-size 4 --data-parallel-size-local 2 \ --data-parallel-start-rank 0 --data-parallel-hybrid-lb \ --data-parallel-address 10.99.48.128 --data-parallel-rpc-port 13345 # Node 1 vllm serve $MODEL --data-parallel-size 4 --data-parallel-size-local 2 \ --data-parallel-start-rank 2 --data-parallel-hybrid-lb \ --data-parallel-address 10.99.48.128 --data-parallel-rpc-port 13345 ``` Each node will launch an API server, and DPLB is performed within the node. In other words, the router can schedule tasks to the node but cannot specify which DP (except for nodes that deploy a single DP rank) within the node to schedule to. *** Based on the above, we need to provide a unique identifier for each API server (I've tentatively named it `engine_name`). Therefore, I’d like to align with everyone on whether we need to consider the two special scenarios of hybrid DPLB and internal DPLB. The foundational setup is very important. What is your opinion? @PeaBrane @doujiang24

bsakaguchi commented

Author

Owner

@doujiang24 commented on GitHub (Feb 2, 2026):

@PeaBrane Nope, you and Ishan could be better, you guys are the experts, I don't have too much time recently, I'm happy to contribute comments.

@doujiang24 commented on GitHub (Feb 2, 2026): @PeaBrane Nope, you and Ishan could be better, you guys are the experts, I don't have too much time recently, I'm happy to contribute comments.

bsakaguchi commented

Author

Owner

@doujiang24 commented on GitHub (Feb 2, 2026):

@yejj710 For Internal DPLB and Hybrid DPLB, the external router can not determine DP rank, right? I don't think it's a good solution.

@doujiang24 commented on GitHub (Feb 2, 2026): @yejj710 For `Internal DPLB` and `Hybrid DPLB`, the external router can not determine DP rank, right? I don't think it's a good solution.

bsakaguchi commented

Author

Owner

@yejj710 commented on GitHub (Feb 2, 2026):

For Internal DPLB and Hybrid DPLB, the external router can not determine DP rank, right?

yeah, it cannot control which specific DP rank the request is sent to.

I don't think it's a good solution.
publisher-unique-name might be better, it is a unique name for a kv-events publisher endpoint in the engine side.

I think so. Then using publisher-unique-name should be good enough. I believe we’re aligned now.

@yejj710 commented on GitHub (Feb 2, 2026): > For `Internal DPLB` and `Hybrid DPLB`, the external router can not determine DP rank, right? yeah, it cannot control which specific DP rank the request is sent to. > I don't think it's a good solution. > `publisher-unique-name` might be better, it is a unique name for a kv-events publisher endpoint in the engine side. I think so. Then using `publisher-unique-name` should be good enough. I believe we’re aligned now.

bsakaguchi commented

Author

Owner

@yejj710 commented on GitHub (Feb 2, 2026):

I think so. Then using publisher-unique-name should be good enough. I believe we’re aligned now.

Sorry, I almost forgot that mooncake also needs a corresponding publisher mechanism. Perhaps using api_server_unique_name would be more appropriate?

@yejj710 commented on GitHub (Feb 2, 2026): > I think so. Then using `publisher-unique-name` should be good enough. I believe we’re aligned now. Sorry, I almost forgot that mooncake also needs a corresponding publisher mechanism. Perhaps using `api_server_unique_name` would be more appropriate?

bsakaguchi commented

Author

Owner

@doujiang24 commented on GitHub (Feb 4, 2026):

Sorry, I almost forgot that mooncake also needs a corresponding publisher mechanism. Perhaps using api_server_unique_name would be more appropriate?

I'm not sure I understand it, do you have a whole picture to describe the api_server_unique_name, where it come from? who/how to use it?

@doujiang24 commented on GitHub (Feb 4, 2026): > Sorry, I almost forgot that mooncake also needs a corresponding publisher mechanism. Perhaps using `api_server_unique_name` would be more appropriate? I'm not sure I understand it, do you have a whole picture to describe the `api_server_unique_name`, where it come from? who/how to use it?

bsakaguchi commented

Author

Owner

@yejj710 commented on GitHub (Feb 4, 2026):

each dp-rank have a unique kvevents zmq publisher endpoint

As previously discussed, each dp-rank have a unique kvevents zmq publisher endpoint and a unique API server endpoint.
BTW, mooncake will have a unique kvevents zmq publisher endpoint.

publisher-unique-name might be better, it is a unique name for a kv-events publisher endpoint in the engine side.
Also, router and kv-indexer shared the same dynamic discovery mechanism, so the router also know the publisher-unique-name, then it can map the publisher-unique-name to the endpoint(ip:port) for inferencing, and other metadata etc.

I just thought the name api_server_unique_name might be more precise. The meaning it represents is same with the description here.

@yejj710 commented on GitHub (Feb 4, 2026): > each dp-rank have a unique kvevents zmq publisher endpoint As previously discussed, each dp-rank have a unique kvevents zmq publisher endpoint and a unique API server endpoint. BTW, mooncake will have a unique kvevents zmq publisher endpoint. > `publisher-unique-name` might be better, it is a unique name for a kv-events publisher endpoint in the engine side. > Also, router and kv-indexer shared the same dynamic discovery mechanism, so the router also know the `publisher-unique-name`, then it can map the `publisher-unique-name` to the `endpoint(ip:port)` for inferencing, and other metadata etc. I just thought the name `api_server_unique_name` might be more precise. The meaning it represents is same with the description here.

bsakaguchi commented

Author

Owner

@doujiang24 commented on GitHub (Feb 5, 2026):

@yejj710 Could you please describe a bit about the workflow, where it come from? And who/how to use it?

@doujiang24 commented on GitHub (Feb 5, 2026): @yejj710 Could you please describe a bit about the workflow, where it come from? And who/how to use it?

bsakaguchi commented

Author

Owner

@yejj710 commented on GitHub (Feb 6, 2026):

Could you please describe a bit about the workflow, where it come from? And who/how to use it?

@doujiang24 Sorry for the late reply, I've been a bit busy recently.
Regarding the shared dynamic service discovery for kv-indexer and router, I believe we have already reached a consensus, but some details do not seem to be fully aligned yet.
In current practice, the service discovery for the kv-indexer should be driven by the router (the service registration action is performed by the router). As for the registration and discovery of inference services, this should be handled by the router and the inference engine. Against this background, here is how a new inference service A is discovered by the kv-indexer and router:

A and the router completed service registration and discovery, and the router thereby learned about A's API server endpoint and the corresponding publisher endpoint
The router creates a unique ID to map the API server endpoint and other metadata (including the publisher endpoint) for inference
The router registers the information required by kv-indexer (unique ID, publisher type, publisher endpoint, etc.) with kv-indexer
The router and kv-indexer both use the unique ID to manage different LLM inference servers

As for the construction method of the unique ID, I prefer Api server endpoint related. Its meaning is more accurate and has stronger extensibility compared to publisher endpoint related:

If we need to support dispatching requests to nodes in the future, where the inference engine performs DPLB within the node, using publisher-unique-name would not meet the requirements

@yejj710 commented on GitHub (Feb 6, 2026): > Could you please describe a bit about the workflow, where it come from? And who/how to use it? @doujiang24 Sorry for the late reply, I've been a bit busy recently. Regarding the shared dynamic service discovery for kv-indexer and router, I believe we have already reached a consensus, but some details do not seem to be fully aligned yet. In current practice, the service discovery for the kv-indexer should be driven by the router (the service registration action is performed by the router). As for the registration and discovery of inference services, this should be handled by the router and the inference engine. Against this background, here is how a new inference service `A` is discovered by the kv-indexer and router: - `A` and the router completed service registration and discovery, and the router thereby learned about `A`'s `API server endpoint` and the corresponding `publisher endpoint` - The router creates a **unique ID** to map the `API server endpoint` and other metadata (including the `publisher endpoint`) for inference - The router registers the information required by kv-indexer (**unique ID**, **publisher type**, **publisher endpoint**, etc.) with kv-indexer - The router and kv-indexer both use the **unique ID** to manage different LLM inference servers As for the construction method of the **unique ID**, I prefer `Api server endpoint` related. Its meaning is more accurate and has stronger extensibility compared to `publisher endpoint` related: - If we need to support dispatching requests to nodes in the future, where the inference engine performs DPLB within the node, using `publisher-unique-name` would not meet the requirements

bsakaguchi commented

Author

Owner

@doujiang24 commented on GitHub (Feb 7, 2026):

@yejj710 Okay, I see what you mean, I don't think it works for sglang, since a single sglang instance have mutiple kvevent publishers, but a single api server endpoints, thoughts?

@doujiang24 commented on GitHub (Feb 7, 2026): @yejj710 Okay, I see what you mean, I don't think it works for sglang, since a single sglang instance have mutiple kvevent publishers, but a single api server endpoints, thoughts?

bsakaguchi commented

Author

Owner

@yejj710 commented on GitHub (Feb 9, 2026):

I don't think it works for sglang, since a single sglang instance have mutiple kvevent publishers, but a single api server endpoints, thoughts?

@doujiang24
Does multiple publishers here refer to multiple dp ranks within a single instance, or to something like multi-publisher in hierarchical storage?

We can make some small adjustments to make it compatible, for example, the information maintained on the router side is roughly based on the following structure:

engine_unique_id: {
    publisher_endpoints: [tcp://1.1.1.1:5557, tcp:1.1.1.1:6667, ..]
}

The router registers this information with the kv-indexer. And then, the kv-indexer can bind events from [tcp://1.1.1.1:5557, tcp:1.1.1.1:6667, ..] to the engine_unique_id. What's your opinion?

@yejj710 commented on GitHub (Feb 9, 2026): > I don't think it works for sglang, since a single sglang instance have mutiple kvevent publishers, but a single api server endpoints, thoughts? @doujiang24 Does `multiple publishers` here refer to multiple dp ranks within a single instance, or to something like `multi-publisher` in hierarchical storage? We can make some small adjustments to make it compatible, for example, the information maintained on the router side is roughly based on the following structure: ``` engine_unique_id: { publisher_endpoints: [tcp://1.1.1.1:5557, tcp:1.1.1.1:6667, ..] } ``` The router registers this information with the kv-indexer. And then, the kv-indexer can bind events from ` [tcp://1.1.1.1:5557, tcp:1.1.1.1:6667, ..]` to the engine_unique_id. What's your opinion?

bsakaguchi commented

Author

Owner

@PeaBrane commented on GitHub (Feb 25, 2026):

@yejj710 @doujiang24 We have an MVP of a standalone KV indexer HTTP service up in dynamo: https://github.com/ai-dynamo/dynamo/pull/6569. The API follows the conventions discussed here (/register, /unregister, /query, /query_by_hash) and works with bare vLLM and SGLang ZMQ event streams. Feedback welcome!

For the instance_id discussion — in our implementation we went with an opaque instance_id that the caller provides, with per-dp_rank ZMQ endpoints registered separately under the same instance_id. The indexer returns per-dp_rank scores nested under instance_id, and it's the router's responsibility to decide how to aggregate based on the DPLB mode. This keeps the indexer agnostic to how engines are deployed.

@PeaBrane commented on GitHub (Feb 25, 2026): @yejj710 @doujiang24 We have an MVP of a standalone KV indexer HTTP service up in dynamo: https://github.com/ai-dynamo/dynamo/pull/6569. The API follows the conventions discussed here (/register, /unregister, /query, /query_by_hash) and works with bare vLLM and SGLang ZMQ event streams. Feedback welcome! For the instance_id discussion — in our implementation we went with an opaque instance_id that the caller provides, with per-dp_rank ZMQ endpoints registered separately under the same instance_id. The indexer returns per-dp_rank scores nested under instance_id, and it's the router's responsibility to decide how to aggregate based on the DPLB mode. This keeps the indexer agnostic to how engines are deployed.

bsakaguchi commented

Author

Owner

@doujiang24 commented on GitHub (Mar 3, 2026):

@yejj710

"instance_id": "vllm-worker-1",
"block_size": 64,

these fields in query APIs, seems make not sense?
tenant_id in response does not make sense? tenant_id means a filter, there won't be other tenant_id in the response.

@doujiang24 commented on GitHub (Mar 3, 2026): @yejj710 ``` "instance_id": "vllm-worker-1", "block_size": 64, ``` 1. these fields in query APIs, seems make not sense? 2. `tenant_id` in response does not make sense? `tenant_id` means a filter, there won't be other `tenant_id` in the response.

bsakaguchi commented

Author

Owner

@doujiang24 commented on GitHub (Mar 3, 2026):

@yejj710 @PeaBrane

For /register api, we'd better register a model with multiple endpoints? i.e.

{
    "model": "deepseek",
    "endpoints": [{
        "instance_id": 1,
        "endpoint": "tcp://127.0.0.1:5557"
    }]
}

@doujiang24 commented on GitHub (Mar 3, 2026): @yejj710 @PeaBrane For `/register` api, we'd better register a model with multiple endpoints? i.e. ``` { "model": "deepseek", "endpoints": [{ "instance_id": 1, "endpoint": "tcp://127.0.0.1:5557" }] } ```

bsakaguchi commented

Author

Owner

@yejj710 commented on GitHub (Mar 4, 2026):

"instance_id": "vllm-worker-1",
"block_size": 64,
these fields in query APIs, seems make not sense?

tenant_id in response does not make sense? tenant_id means a filter, there won't be other tenant_id in the response.

@doujiang24
for q1: block_size is important, same model with different block_size will have different prefix hit;
yes, instance_id is a redundant fields, tenant_id is already sufficient

for q2: yes, I will delete this layer of nesting

For /register api, we'd better register a model with multiple endpoints?

does it mean that a single /register call registers kv-publishers for all dp_ranks? sounds reasonable

@yejj710 commented on GitHub (Mar 4, 2026): > ``` > "instance_id": "vllm-worker-1", > "block_size": 64, > ``` > 1. these fields in query APIs, seems make not sense? > 2. `tenant_id` in response does not make sense? `tenant_id` means a filter, there won't be other `tenant_id` in the response. @doujiang24 for q1: `block_size` is important, same model with different `block_size` will have different prefix hit; yes, `instance_id` is a redundant fields, `tenant_id` is already sufficient for q2: yes, I will delete this layer of nesting > For /register api, we'd better register a model with multiple endpoints? does it mean that a single /register call registers kv-publishers for all dp_ranks? sounds reasonable

bsakaguchi commented

Author

Owner

@doujiang24 commented on GitHub (Mar 5, 2026):

@yejj710
How does the kvindexer hanlde different block_size? could you please explain the workflow?

does it mean that a single /register call registers kv-publishers for all dp_ranks? sounds reasonable

Yes.

@doujiang24 commented on GitHub (Mar 5, 2026): @yejj710 How does the kvindexer hanlde different `block_size`? could you please explain the workflow? > does it mean that a single /register call registers kv-publishers for all dp_ranks? sounds reasonable Yes.

bsakaguchi commented

Author

Owner

@PeaBrane commented on GitHub (Mar 6, 2026):

@doujiang24 @yejj710
I'm guessing two radix trees with different block sizes can live in the same indexer microservice. We assume this heterogeneous design in the current dynamo impl

@PeaBrane commented on GitHub (Mar 6, 2026): @doujiang24 @yejj710 I'm guessing two radix trees with different block sizes can live in the same indexer microservice. We assume this heterogeneous design in the current dynamo impl

bsakaguchi commented

Author

Owner

@yejj710 commented on GitHub (Mar 6, 2026):

How does the kvindexer hanlde different block_size? could you please explain the workflow?

@doujiang24
As @PeaBrane mentioned, currently there are eight dimensions that affect the identifier: (model_name, block_size, additional_salt, lora_name, tenant_id, instance_id/backend_id, medium, dp_rank).

Regarding how to identify a model service instance in the indexer, I think six dimensions are sufficient (tenant_id, instance_id, model_name, lora_name, block_size, additional_salt).

Therefore, in the current implementation, a model service instance is uniquely identified using a 6-tuple. This constraint is relatively loose, allowing multiple identical model services to be managed through a single indexer; it also allows different tenants to deploy model using different block_size, and the indexer will internally maintain an independent radix-tree or nested map based on any change in one of the six dimensions.

@yejj710 commented on GitHub (Mar 6, 2026): > How does the kvindexer hanlde different `block_size`? could you please explain the workflow? @doujiang24 As @PeaBrane mentioned, currently there are eight dimensions that affect the identifier: (`model_name, block_size, additional_salt, lora_name, tenant_id, instance_id/backend_id, medium, dp_rank`). Regarding how to identify a model service instance in the indexer, I think six dimensions are sufficient (`tenant_id, instance_id, model_name, lora_name, block_size, additional_salt`). Therefore, in the current implementation, a model service instance is uniquely identified using a 6-tuple. This constraint is relatively loose, allowing multiple identical model services to be managed through a single indexer; it also allows different tenants to deploy model using different block_size, and the indexer will internally maintain an independent radix-tree or nested map based on any change in one of the six dimensions.

bsakaguchi commented