kvcache-ai/Mooncake
1
0
Fork 0
mirror of https://github.com/kvcache-ai/Mooncake.git synced 2026-11-05 01:17:55 +00:00
Code Issues 368 Projects Packages Wiki Activity

[GH-ISSUE #1207] PD mode failure #411

New issue
Closed
opened 2026-10-29 21:14:09 +00:00 by bsakaguchi · 8 comments
bsakaguchi commented 2026-10-29 21:14:09 +00:00
Owner
Copy link

Originally created by @UralmashFox on GitHub (Dec 12, 2025).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/1207

Originally assigned to: @alogfans on GitHub.

I try to follow sglang-integration-v1.md guide. Did everything like in the guide, tried to increase\dircrease size of page, max-total-tokens and chunked-prefill. Also tried to do some minor changes in docker image as far as I use Prefill, Decode and Proxy in different podes. For now the dockerfile is based on sglang:v0.5.1.post2-cu126 and looks like following:

RUN apt-get update && apt-get install -y \
    libibverbs1 libibverbs-dev \
    build-essential rdma-core \
    rdma-core infiniband-diags \
    numactl lsof net-tools iputils-ping \
    cmake g++ gcc \
    && rm -rf /var/lib/apt/lists/*

RUN pip install mooncake-transfer-engine

EXPOSE 8000

CMD ["bash"]

and the logs of prefill are the following:

W1212 12:06:07.458000 7 torch/utils/cpp_extension.py:2425] TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation.
W1212 12:06:07.458000 7 torch/utils/cpp_extension.py:2425] If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST'] to specific architectures.
WARNING:sglang.srt.server_args:Cuda graph is disabled for prefill server
[2025-12-12 12:06:07] server_args=ServerArgs(model_path='/opt/models/Qwen/Qwen3-Coder-30B-A3B-Instruct', tokenizer_path='/opt/models/Qwen/Qwen3-Coder-30B-A3B-Instruct', tokenizer_mode='auto', skip_tokenizer_init=False, load_format='auto', model_loader_extra_config='{}', trust_remote_code=False, context_length=None, is_embedding=False, enable_multimodal=None, revision=None, model_impl='auto', host='0.0.0.0', port=30000, skip_server_warmup=False, warmups=None, nccl_port=None, dtype='auto', quantization=None, quantization_param_path=None, kv_cache_dtype='auto', mem_fraction_static=0.85, max_running_requests=None, max_queued_requests=9223372036854775807, max_total_tokens=256, chunked_prefill_size=128, max_prefill_tokens=16384, schedule_policy='fcfs', schedule_conservativeness=1.0, page_size=32, hybrid_kvcache_ratio=None, swa_full_tokens_ratio=0.8, disable_hybrid_swa_memory=False, device='cuda', tp_size=1, pp_size=1, max_micro_batch_size=None, stream_interval=1, stream_output=False, random_seed=1064179575, constrained_json_whitespace_pattern=None, watchdog_timeout=300, dist_timeout=None, download_dir=None, base_gpu_id=0, gpu_id_step=1, sleep_on_idle=False, log_level='info', log_level_http=None, log_requests=False, log_requests_level=2, crash_dump_folder=None, show_time_cost=False, enable_metrics=False, enable_metrics_for_all_schedulers=False, bucket_time_to_first_token=None, bucket_inter_token_latency=None, bucket_e2e_request_latency=None, collect_tokens_histogram=False, decode_log_interval=40, enable_request_time_stats_logging=False, kv_events_config=None, gc_warning_threshold_secs=0.0, api_key=None, served_model_name='/opt/models/Qwen/Qwen3-Coder-30B-A3B-Instruct', weight_version='default', chat_template=None, completion_template=None, file_storage_path='sglang_storage', enable_cache_report=False, reasoning_parser=None, tool_call_parser='qwen3_coder', tool_server=None, dp_size=1, load_balance_method='round_robin', dist_init_addr=None, nnodes=1, node_rank=0, json_model_override_args='{}', preferred_sampling_params=None, enable_lora=None, max_lora_rank=None, lora_target_modules=None, lora_paths=None, max_loaded_loras=None, max_loras_per_batch=8, lora_backend='triton', attention_backend=None, decode_attention_backend=None, prefill_attention_backend=None, sampling_backend='flashinfer', grammar_backend='xgrammar', mm_attention_backend=None, speculative_algorithm=None, speculative_draft_model_path=None, speculative_num_steps=None, speculative_eagle_topk=None, speculative_num_draft_tokens=None, speculative_accept_threshold_single=1.0, speculative_accept_threshold_acc=1.0, speculative_token_map=None, ep_size=1, moe_a2a_backend='none', moe_runner_backend='auto', flashinfer_mxfp4_moe_precision='default', enable_flashinfer_allreduce_fusion=False, deepep_mode='auto', ep_num_redundant_experts=0, ep_dispatch_algorithm='static', init_expert_location='trivial', enable_eplb=False, eplb_algorithm='auto', eplb_rebalance_num_iterations=1000, eplb_rebalance_layers_per_chunk=None, expert_distribution_recorder_mode=None, expert_distribution_recorder_buffer_size=1000, enable_expert_distribution_metrics=False, deepep_config=None, moe_dense_tp_size=None, enable_hierarchical_cache=False, hicache_ratio=2.0, hicache_size=0, hicache_write_policy='write_through_selective', hicache_io_backend='kernel', hicache_mem_layout='layer_first', hicache_storage_backend=None, hicache_storage_prefetch_policy='best_effort', enable_double_sparsity=False, ds_channel_config_path=None, ds_heavy_channel_num=32, ds_heavy_token_num=256, ds_heavy_channel_type='qk', ds_sparse_decode_threshold=4096, cpu_offload_gb=0, offload_group_size=-1, offload_num_in_group=1, offload_prefetch_step=1, offload_mode='cpu', disable_radix_cache=False, cuda_graph_max_bs=None, cuda_graph_bs=None, disable_cuda_graph=True, disable_cuda_graph_padding=False, enable_profile_cuda_graph=False, enable_cudagraph_gc=False, enable_nccl_nvls=False, enable_symm_mem=False, disable_flashinfer_cutlass_moe_fp4_allgather=False, enable_tokenizer_batch_encode=False, disable_outlines_disk_cache=False, disable_custom_all_reduce=False, enable_mscclpp=False, disable_overlap_schedule=False, enable_mixed_chunk=False, enable_dp_attention=False, enable_dp_lm_head=False, enable_two_batch_overlap=False, tbo_token_distribution_threshold=0.48, enable_torch_compile=False, torch_compile_max_bs=32, torchao_config='', enable_nan_detection=False, enable_p2p_check=False, triton_attention_reduce_in_fp32=False, triton_attention_num_kv_splits=8, num_continuous_decode_steps=1, delete_ckpt_after_loading=False, enable_memory_saver=False, allow_auto_truncate=False, enable_custom_logit_processor=False, flashinfer_mla_disable_ragged=False, disable_shared_experts_fusion=False, disable_chunked_prefix_cache=False, disable_fast_image_processor=False, enable_return_hidden_states=False, scheduler_recv_interval=1, debug_tensor_dump_output_folder=None, debug_tensor_dump_input_file=None, debug_tensor_dump_inject=False, debug_tensor_dump_prefill_only=False, disaggregation_mode='prefill', disaggregation_transfer_backend='mooncake', disaggregation_bootstrap_port=30003, disaggregation_decode_tp=1, disaggregation_decode_dp=1, disaggregation_prefill_pp=1, disaggregation_ib_device='mlx5_12', num_reserved_decode_tokens=512, pdlb_url=None, custom_weight_loader=[], weight_loader_disable_mmap=False, enable_pdmux=False, sm_group_num=3, enable_ep_moe=False, enable_deepep_moe=False, enable_flashinfer_cutlass_moe=False, enable_flashinfer_trtllm_moe=False, enable_triton_kernel_moe=False, enable_flashinfer_mxfp4_moe=False)
[2025-12-12 12:06:08] Using default HuggingFace chat template with detected content format: string
W1212 12:06:13.575000 206 torch/utils/cpp_extension.py:2425] TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation.
W1212 12:06:13.575000 206 torch/utils/cpp_extension.py:2425] If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST'] to specific architectures.
W1212 12:06:13.977000 205 torch/utils/cpp_extension.py:2425] TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation.
W1212 12:06:13.977000 205 torch/utils/cpp_extension.py:2425] If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST'] to specific architectures.
[2025-12-12 12:06:14] Attention backend not explicitly specified. Use fa3 backend by default.
[2025-12-12 12:06:14] Init torch distributed begin.
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[2025-12-12 12:06:14] Init torch distributed ends. mem usage=0.00 GB
[2025-12-12 12:06:15] Ignore import error when loading sglang.srt.models.glm4v_moe: No module named 'transformers.models.glm4v_moe'
[2025-12-12 12:06:15] Load weight begin. avail mem=76.38 GB

Loading safetensors checkpoint shards:   0% Completed | 0/16 [00:00<?, ?it/s]

Loading safetensors checkpoint shards:   6% Completed | 1/16 [00:00<00:09,  1.53it/s]

Loading safetensors checkpoint shards:  12% Completed | 2/16 [00:01<00:09,  1.45it/s]

Loading safetensors checkpoint shards:  19% Completed | 3/16 [00:02<00:08,  1.50it/s]

Loading safetensors checkpoint shards:  25% Completed | 4/16 [00:02<00:08,  1.49it/s]

Loading safetensors checkpoint shards:  31% Completed | 5/16 [00:03<00:07,  1.51it/s]

Loading safetensors checkpoint shards:  38% Completed | 6/16 [00:03<00:06,  1.54it/s]

Loading safetensors checkpoint shards:  44% Completed | 7/16 [00:04<00:05,  1.55it/s]

Loading safetensors checkpoint shards:  50% Completed | 8/16 [00:05<00:05,  1.52it/s]

Loading safetensors checkpoint shards:  56% Completed | 9/16 [00:05<00:04,  1.48it/s]

Loading safetensors checkpoint shards:  62% Completed | 10/16 [00:06<00:04,  1.49it/s]

Loading safetensors checkpoint shards:  69% Completed | 11/16 [00:07<00:03,  1.49it/s]

Loading safetensors checkpoint shards:  75% Completed | 12/16 [00:07<00:02,  1.50it/s]

Loading safetensors checkpoint shards:  81% Completed | 13/16 [00:08<00:01,  1.53it/s]

Loading safetensors checkpoint shards:  88% Completed | 14/16 [00:09<00:01,  1.51it/s]

Loading safetensors checkpoint shards:  94% Completed | 15/16 [00:10<00:00,  1.47it/s]

Loading safetensors checkpoint shards: 100% Completed | 16/16 [00:10<00:00,  1.89it/s]

Loading safetensors checkpoint shards: 100% Completed | 16/16 [00:10<00:00,  1.57it/s]

WARNING: Logging before InitGoogleLogging() is written to STDERR
I1212 12:06:26.756870   205 transfer_engine.cpp:422] Metrics reporting is disabled (set MC_TE_METRIC=1 to enable)
I1212 12:06:26.756886   205 transfer_engine.cpp:44] Transfer Engine starting. Server: serverip, Metadata: P2PHANDSHAKE, ip_or_host_name: , rpc_port: 0
I1212 12:06:26.756888   205 transfer_engine.cpp:63] Transfer Engine parseHostNameWithPort. server_name: serverip port: 12001
I1212 12:06:26.756910   205 transfer_engine.cpp:114] Transfer Engine RPC using P2P handshake, listening on serverip:16225
I1212 12:06:26.756995   205 transfer_engine.cpp:138] Auto-discovering topology...
I1212 12:06:26.759825   205 transfer_engine.cpp:153] Topology discovery complete. Found 1 HCAs.
I1212 12:06:26.784847   205 rdma_context.cpp:423] Find best gid index: 0 on mlx5_12/
I1212 12:06:26.785706   205 rdma_context.cpp:125] RDMA device: mlx5_12, LID: 1223, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:00:00:01:26:1a:00:01:01
E1212 12:06:27.214640   865 rdma_context.cpp:205] Failed to register memory 0x7fd4ad548000: Bad address [14]
E1212 12:06:27.214716   864 rdma_context.cpp:205] Failed to register memory 0x7fd4ad500000: Bad address [14]
E1212 12:06:27.214953   869 rdma_context.cpp:205] Failed to register memory 0x7fd3e6290000: Bad address [14]
E1212 12:06:27.214982   866 rdma_context.cpp:205] Failed to register memory 0x7fd4ad590000: Bad address [14]
E1212 12:06:27.215179   863 rdma_context.cpp:205] Failed to register memory 0x7fd6bb591000: Bad address [14]
E1212 12:06:27.215278   870 rdma_context.cpp:205] Failed to register memory 0x7fd3e62d8000: Bad address [14]
E1212 12:06:27.215399   868 rdma_context.cpp:205] Failed to register memory 0x7fd3e6248000: Bad address [14]
E1212 12:06:27.215487   867 rdma_context.cpp:205] Failed to register memory 0x7fd3e6200000: Bad address [14]
E1212 12:06:27.215593   871 rdma_context.cpp:205] Failed to register memory 0x7fd3e6320000: Bad address [14]
E1212 12:06:27.215726   874 rdma_context.cpp:205] Failed to register memory 0x7fd3e6400000: Bad address [14]
E1212 12:06:27.215803   875 rdma_context.cpp:205] Failed to register memory 0x7fd3e6448000: Bad address [14]
E1212 12:06:27.215821   873 rdma_context.cpp:205] Failed to register memory 0x7fd3e63b0000: Bad address [14]
E1212 12:06:27.216099   872 rdma_context.cpp:205] Failed to register memory 0x7fd3e6368000: Bad address [14]
E1212 12:06:27.216156   878 rdma_context.cpp:205] Failed to register memory 0x7fd3e6520000: Bad address [14]
E1212 12:06:27.216158   876 rdma_context.cpp:205] Failed to register memory 0x7fd3e6490000: Bad address [14]
E1212 12:06:27.216158   877 rdma_context.cpp:205] Failed to register memory 0x7fd3e64d8000: Bad address [14]
E1212 12:06:27.216207   884 rdma_context.cpp:205] Failed to register memory 0x7fd3e66d8000: Bad address [14]
E1212 12:06:27.216306   887 rdma_context.cpp:205] Failed to register memory 0x7fd3e67b0000: Bad address [14]
E1212 12:06:27.216359   880 rdma_context.cpp:205] Failed to register memory 0x7fd3e65b0000: Bad address [14]
E1212 12:06:27.216394   890 rdma_context.cpp:205] Failed to register memory 0x7fd3e6890000: Bad address [14]
E1212 12:06:27.216470   882 rdma_context.cpp:205] Failed to register memory 0x7fd3e6648000: Bad address [14]
E1212 12:06:27.216497   892 rdma_context.cpp:205] Failed to register memory 0x7fd3e6920000: Bad address [14]
E1212 12:06:27.216533   879 rdma_context.cpp:205] Failed to register memory 0x7fd3e6568000: Bad address [14]
E1212 12:06:27.216559   885 rdma_context.cpp:205] Failed to register memory 0x7fd3e6720000: Bad address [14]
E1212 12:06:27.216602   895 rdma_context.cpp:205] Failed to register memory 0x7fd3e6a00000: Bad address [14]
E1212 12:06:27.216660   894 rdma_context.cpp:205] Failed to register memory 0x7fd3e69b0000: Bad address [14]
E1212 12:06:27.216687   893 rdma_context.cpp:205] Failed to register memory 0x7fd3e6968000: Bad address [14]
E1212 12:06:27.216733   898 rdma_context.cpp:205] Failed to register memory 0x7fd3e6ad8000: Bad address [14]
E1212 12:06:27.216763   899 rdma_context.cpp:205] Failed to register memory 0x7fd3e6b20000: Bad address [14]
E1212 12:06:27.216836   881 rdma_context.cpp:205] Failed to register memory 0x7fd3e6600000: Bad address [14]
E1212 12:06:27.216861   888 rdma_context.cpp:205] Failed to register memory 0x7fd3e6800000: Bad address [14]
E1212 12:06:27.216889   911 rdma_context.cpp:205] Failed to register memory 0x7fd3e6e90000: Bad address [14]
E1212 12:06:27.216903   905 rdma_context.cpp:205] Failed to register memory 0x7fd3e6cd8000: Bad address [14]
E1212 12:06:27.216934   897 rdma_context.cpp:205] Failed to register memory 0x7fd3e6a90000: Bad address [14]
E1212 12:06:27.216951   891 rdma_context.cpp:205] Failed to register memory 0x7fd3e68d8000: Bad address [14]
E1212 12:06:27.216984   883 rdma_context.cpp:205] Failed to register memory 0x7fd3e6690000: Bad address [14]
E1212 12:06:27.217015   900 rdma_context.cpp:205] Failed to register memory 0x7fd3e6b68000: Bad address [14]
E1212 12:06:27.217025   902 rdma_context.cpp:205] Failed to register memory 0x7fd3e6c00000: Bad address [14]
E1212 12:06:27.217056   901 rdma_context.cpp:205] Failed to register memory 0x7fd3e6bb0000: Bad address [14]
E1212 12:06:27.217085   886 rdma_context.cpp:205] Failed to register memory 0x7fd3e6768000: Bad address [14]
E1212 12:06:27.217092   903 rdma_context.cpp:205] Failed to register memory 0x7fd3e6c48000: Bad address [14]
E1212 12:06:27.217119   889 rdma_context.cpp:205] Failed to register memory 0x7fd3e6848000: Bad address [14]
E1212 12:06:27.217131   904 rdma_context.cpp:205] Failed to register memory 0x7fd3e6c90000: Bad address [14]
E1212 12:06:27.217214   907 rdma_context.cpp:205] Failed to register memory 0x7fd3e6d68000: Bad address [14]
E1212 12:06:27.217228   914 rdma_context.cpp:205] Failed to register memory 0x7fd3e6f68000: Bad address [14]
E1212 12:06:27.217242   920 rdma_context.cpp:205] Failed to register memory 0x7fd3e7120000: Bad address [14]
E1212 12:06:27.217285   916 rdma_context.cpp:205] Failed to register memory 0x7fd3e7000000: Bad address [14]
E1212 12:06:27.217309   910 rdma_context.cpp:205] Failed to register memory 0x7fd3e6e48000: Bad address [14]
E1212 12:06:27.217337   923 rdma_context.cpp:205] Failed to register memory 0x7fd3e7200000: Bad address [14]
E1212 12:06:27.217340   918 rdma_context.cpp:205] Failed to register memory 0x7fd3e7090000: Bad address [14]
E1212 12:06:27.217363   912 rdma_context.cpp:205] Failed to register memory 0x7fd3e6ed8000: Bad address [14]
E1212 12:06:27.217394   919 rdma_context.cpp:205] Failed to register memory 0x7fd3e70d8000: Bad address [14]
E1212 12:06:27.217409   908 rdma_context.cpp:205] Failed to register memory 0x7fd3e6db0000: Bad address [14]
E1212 12:06:27.217456   909 rdma_context.cpp:205] Failed to register memory 0x7fd3e6e00000: Bad address [14]
E1212 12:06:27.217484   915 rdma_context.cpp:205] Failed to register memory 0x7fd3e6fb0000: Bad address [14]
E1212 12:06:27.217499   896 rdma_context.cpp:205] Failed to register memory 0x7fd3e6a48000: Bad address [14]
E1212 12:06:27.217510   922 rdma_context.cpp:205] Failed to register memory 0x7fd3e71b0000: Bad address [14]
E1212 12:06:27.217516   906 rdma_context.cpp:205] Failed to register memory 0x7fd3e6d20000: Bad address [14]
E1212 12:06:27.217588   924 rdma_context.cpp:205] Failed to register memory 0x7fd3e7248000: Bad address [14]
E1212 12:06:27.217602   921 rdma_context.cpp:205] Failed to register memory 0x7fd3e7168000: Bad address [14]
E1212 12:06:27.217659   917 rdma_context.cpp:205] Failed to register memory 0x7fd3e7048000: Bad address [14]
E1212 12:06:27.217660   926 rdma_context.cpp:205] Failed to register memory 0x7fd3e72d8000: Bad address [14]
E1212 12:06:27.217713   928 rdma_context.cpp:205] Failed to register memory 0x7fd3e7368000: Bad address [14]
E1212 12:06:27.217716   913 rdma_context.cpp:205] Failed to register memory 0x7fd3e6f20000: Bad address [14]
E1212 12:06:27.217736   925 rdma_context.cpp:205] Failed to register memory 0x7fd3e7290000: Bad address [14]
E1212 12:06:27.217805   931 rdma_context.cpp:205] Failed to register memory 0x7fd3e7448000: Bad address [14]
E1212 12:06:27.217851   930 rdma_context.cpp:205] Failed to register memory 0x7fd3e7400000: Bad address [14]
E1212 12:06:27.217854   929 rdma_context.cpp:205] Failed to register memory 0x7fd3e73b0000: Bad address [14]
E1212 12:06:27.217870   927 rdma_context.cpp:205] Failed to register memory 0x7fd3e7320000: Bad address [14]
E1212 12:06:27.217926   932 rdma_context.cpp:205] Failed to register memory 0x7fd3e7490000: Bad address [14]
E1212 12:06:27.217938   937 rdma_context.cpp:205] Failed to register memory 0x7fd3e7600000: Bad address [14]
E1212 12:06:27.217969   936 rdma_context.cpp:205] Failed to register memory 0x7fd3e75b0000: Bad address [14]
E1212 12:06:27.218004   938 rdma_context.cpp:205] Failed to register memory 0x7fd3e7648000: Bad address [14]
E1212 12:06:27.218010   934 rdma_context.cpp:205] Failed to register memory 0x7fd3e7520000: Bad address [14]
E1212 12:06:27.218020   933 rdma_context.cpp:205] Failed to register memory 0x7fd3e74d8000: Bad address [14]
E1212 12:06:27.218029   939 rdma_context.cpp:205] Failed to register memory 0x7fd3e7690000: Bad address [14]
E1212 12:06:27.218086   940 rdma_context.cpp:205] Failed to register memory 0x7fd3e76d8000: Bad address [14]
E1212 12:06:27.218112   941 rdma_context.cpp:205] Failed to register memory 0x7fd3e7720000: Bad address [14]
E1212 12:06:27.218137   942 rdma_context.cpp:205] Failed to register memory 0x7fd3e7768000: Bad address [14]
E1212 12:06:27.218132   935 rdma_context.cpp:205] Failed to register memory 0x7fd3e7568000: Bad address [14]
E1212 12:06:27.218173   943 rdma_context.cpp:205] Failed to register memory 0x7fd3e77b0000: Bad address [14]
E1212 12:06:27.218211   944 rdma_context.cpp:205] Failed to register memory 0x7fd3e7800000: Bad address [14]
E1212 12:06:27.218248   945 rdma_context.cpp:205] Failed to register memory 0x7fd3e7848000: Bad address [14]
E1212 12:06:27.218272   946 rdma_context.cpp:205] Failed to register memory 0x7fd3e7890000: Bad address [14]
E1212 12:06:27.218309   947 rdma_context.cpp:205] Failed to register memory 0x7fd3e78d8000: Bad address [14]
E1212 12:06:27.218324   948 rdma_context.cpp:205] Failed to register memory 0x7fd3e7920000: Bad address [14]
E1212 12:06:27.218366   949 rdma_context.cpp:205] Failed to register memory 0x7fd3e7968000: Bad address [14]
E1212 12:06:27.218400   950 rdma_context.cpp:205] Failed to register memory 0x7fd3e79b0000: Bad address [14]
E1212 12:06:27.218431   951 rdma_context.cpp:205] Failed to register memory 0x7fd3e7a00000: Bad address [14]
E1212 12:06:27.218453   952 rdma_context.cpp:205] Failed to register memory 0x7fd3e7a48000: Bad address [14]
E1212 12:06:27.218487   953 rdma_context.cpp:205] Failed to register memory 0x7fd3e7a90000: Bad address [14]
E1212 12:06:27.218529   954 rdma_context.cpp:205] Failed to register memory 0x7fd3e7ad8000: Bad address [14]
E1212 12:06:27.218549   955 rdma_context.cpp:205] Failed to register memory 0x7fd3e7b20000: Bad address [14]
E1212 12:06:27.218586   956 rdma_context.cpp:205] Failed to register memory 0x7fd3e7b68000: Bad address [14]
E1212 12:06:27.218617   957 rdma_context.cpp:205] Failed to register memory 0x7fd3e7bb0000: Bad address [14]
W1212 12:06:27.218627   205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd6bb591000 length 294912
W1212 12:06:27.218652   205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd4ad500000 length 294912
E1212 12:06:27.218657   958 rdma_context.cpp:205] Failed to register memory 0x7fd3e7c00000: Bad address [14]
W1212 12:06:27.218665   205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd4ad548000 length 294912
W1212 12:06:27.218678   205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd4ad590000 length 294912
W1212 12:06:27.218688   205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e6200000 length 294912
W1212 12:06:27.218696   205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e6248000 length 294912
W1212 12:06:27.218705   205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e6290000 length 294912
W1212 12:06:27.218713   205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e62d8000 length 294912
W1212 12:06:27.218721   205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e6320000 length 294912
W1212 12:06:27.218729   205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e6368000 length 294912
W1212 12:06:27.218737   205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e63b0000 length 294912
W1212 12:06:27.218745   205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e6400000 length 294912
W1212 12:06:27.218753   205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e6448000 length 294912
W1212 12:06:27.218761   205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e6490000 length 294912
W1212 12:06:27.218770   205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e64d8000 length 294912
W1212 12:06:27.218777   205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e6520000 length 294912
W1212 12:06:27.218786   205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e6568000 length 294912
W1212 12:06:27.218793   205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e65b0000 length 294912
W1212 12:06:27.218801   205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e6600000 length 294912
W1212 12:06:27.218809   205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e6648000 length 294912
W1212 12:06:27.218816   205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e6690000 length 294912
W1212 12:06:27.218823   205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e66d8000 length 294912
W1212 12:06:27.218832   205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e6720000 length 294912
W1212 12:06:27.218840   205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e6768000 length 294912
W1212 12:06:27.218847   205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e67b0000 length 294912
W1212 12:06:27.218854   205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e6800000 length 294912
W1212 12:06:27.218863   205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e6848000 length 294912
W1212 12:06:27.218870   205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e6890000 length 294912
W1212 12:06:27.218878   205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e68d8000 length 294912
W1212 12:06:27.218885   205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e6920000 length 294912
W1212 12:06:27.218894   205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e6968000 length 294912
W1212 12:06:27.218901   205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e69b0000 length 294912
W1212 12:06:27.218907   205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e6a00000 length 294912
W1212 12:06:27.218915   205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e6a48000 length 294912
W1212 12:06:27.218922   205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e6a90000 length 294912
W1212 12:06:27.218930   205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e6ad8000 length 294912
W1212 12:06:27.218936   205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e6b20000 length 294912
W1212 12:06:27.218945   205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e6b68000 length 294912
W1212 12:06:27.218952   205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e6bb0000 length 294912
W1212 12:06:27.218959   205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e6c00000 length 294912
W1212 12:06:27.218966   205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e6c48000 length 294912
W1212 12:06:27.218973   205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e6c90000 length 294912
W1212 12:06:27.218981   205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e6cd8000 length 294912
W1212 12:06:27.218987   205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e6d20000 length 294912
W1212 12:06:27.218995   205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e6d68000 length 294912
W1212 12:06:27.219002   205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e6db0000 length 294912
W1212 12:06:27.219009   205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e6e00000 length 294912
W1212 12:06:27.219017   205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e6e48000 length 294912
W1212 12:06:27.219024   205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e6e90000 length 294912
W1212 12:06:27.219030   205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e6ed8000 length 294912
W1212 12:06:27.219038   205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e6f20000 length 294912
W1212 12:06:27.219045   205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e6f68000 length 294912
W1212 12:06:27.219053   205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e6fb0000 length 294912
W1212 12:06:27.219060   205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e7000000 length 294912
W1212 12:06:27.219069   205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e7048000 length 294912
W1212 12:06:27.219075   205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e7090000 length 294912
W1212 12:06:27.219081   205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e70d8000 length 294912
W1212 12:06:27.219089   205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e7120000 length 294912
W1212 12:06:27.219096   205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e7168000 length 294912
W1212 12:06:27.219102   205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e71b0000 length 294912
W1212 12:06:27.219110   205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e7200000 length 294912
W1212 12:06:27.219117   205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e7248000 length 294912
W1212 12:06:27.219125   205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e7290000 length 294912
W1212 12:06:27.219131   205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e72d8000 length 294912
W1212 12:06:27.219138   205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e7320000 length 294912
W1212 12:06:27.219146   205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e7368000 length 294912
W1212 12:06:27.219152   205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e73b0000 length 294912
W1212 12:06:27.219159   205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e7400000 length 294912
W1212 12:06:27.219166   205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e7448000 length 294912
W1212 12:06:27.219173   205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e7490000 length 294912
W1212 12:06:27.219179   205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e74d8000 length 294912
W1212 12:06:27.219187   205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e7520000 length 294912
W1212 12:06:27.219192   205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e7568000 length 294912
W1212 12:06:27.219199   205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e75b0000 length 294912
W1212 12:06:27.219206   205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e7600000 length 294912
W1212 12:06:27.219213   205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e7648000 length 294912
W1212 12:06:27.219219   205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e7690000 length 294912
W1212 12:06:27.219226   205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e76d8000 length 294912
W1212 12:06:27.219233   205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e7720000 length 294912
W1212 12:06:27.219239   205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e7768000 length 294912
W1212 12:06:27.219247   205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e77b0000 length 294912
W1212 12:06:27.219254   205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e7800000 length 294912
W1212 12:06:27.219262   205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e7848000 length 294912
W1212 12:06:27.219269   205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e7890000 length 294912
W1212 12:06:27.219275   205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e78d8000 length 294912
W1212 12:06:27.219282   205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e7920000 length 294912
W1212 12:06:27.219290   205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e7968000 length 294912
W1212 12:06:27.219296   205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e79b0000 length 294912
W1212 12:06:27.219302   205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e7a00000 length 294912
W1212 12:06:27.219309   205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e7a48000 length 294912
W1212 12:06:27.219316   205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e7a90000 length 294912
W1212 12:06:27.219324   205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e7ad8000 length 294912
W1212 12:06:27.219331   205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e7b20000 length 294912
W1212 12:06:27.219338   205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e7b68000 length 294912
W1212 12:06:27.219344   205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e7bb0000 length 294912
W1212 12:06:27.219354   205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e7c00000 length 294912
[2025-12-12 12:06:27] INFO:     Started server process [7]
[2025-12-12 12:06:27] INFO:     Waiting for application startup.
[2025-12-12 12:06:27] INFO:     Application startup complete.
[2025-12-12 12:06:27] INFO:     Uvicorn running on http://0.0.0.0:30000 (Press CTRL+C to quit)
[2025-12-12 12:06:28] INFO:     127.0.0.1:34916 - "GET /get_model_info HTTP/1.1" 200 OK
[2025-12-12 12:06:28] Start of pd disaggregation warmup ...
[2025-12-12 12:06:32] INFO:     127.0.0.1:34920 - "POST /generate HTTP/1.1" 200 OK
[2025-12-12 12:06:32] End of prefill disaggregation mode warmup with status 200, resp: [{'text': '%', 'output_ids': [0, 1, 2, 3, 4], 'meta_info': {'id': '2420bb71081f4e2c8d621f15c919728a', 'finish_reason': {'type': 'length', 'length': 0}, 'prompt_tokens': 4, 'weight_version': 'default', 'completion_tokens': 1, 'cached_tokens': 0, 'e2e_latency': 3.9578440189361572}}]
[2025-12-12 12:06:32] The server is fired up and ready to roll!

Despite "The server is fired up", the proxy drops on the requests the following:
KVTransferError(bootstrap_room=6677116944225248479): Could not fetch prefill parallel info from bootstrap_addr: sglang-prefill-svc:30000

which else installations should I add? Thank you in advance!

Originally created by @UralmashFox on GitHub (Dec 12, 2025). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/1207 Originally assigned to: @alogfans on GitHub. I try to follow sglang-integration-v1.md guide. Did everything like in the guide, tried to increase\dircrease size of page, max-total-tokens and chunked-prefill. Also tried to do some minor changes in docker image as far as I use Prefill, Decode and Proxy in different podes. For now the dockerfile is based on sglang:v0.5.1.post2-cu126 and looks like following: ``` RUN apt-get update && apt-get install -y \ libibverbs1 libibverbs-dev \ build-essential rdma-core \ rdma-core infiniband-diags \ numactl lsof net-tools iputils-ping \ cmake g++ gcc \ && rm -rf /var/lib/apt/lists/* RUN pip install mooncake-transfer-engine EXPOSE 8000 CMD ["bash"] ``` and the logs of prefill are the following: ``` W1212 12:06:07.458000 7 torch/utils/cpp_extension.py:2425] TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation. W1212 12:06:07.458000 7 torch/utils/cpp_extension.py:2425] If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST'] to specific architectures. WARNING:sglang.srt.server_args:Cuda graph is disabled for prefill server [2025-12-12 12:06:07] server_args=ServerArgs(model_path='/opt/models/Qwen/Qwen3-Coder-30B-A3B-Instruct', tokenizer_path='/opt/models/Qwen/Qwen3-Coder-30B-A3B-Instruct', tokenizer_mode='auto', skip_tokenizer_init=False, load_format='auto', model_loader_extra_config='{}', trust_remote_code=False, context_length=None, is_embedding=False, enable_multimodal=None, revision=None, model_impl='auto', host='0.0.0.0', port=30000, skip_server_warmup=False, warmups=None, nccl_port=None, dtype='auto', quantization=None, quantization_param_path=None, kv_cache_dtype='auto', mem_fraction_static=0.85, max_running_requests=None, max_queued_requests=9223372036854775807, max_total_tokens=256, chunked_prefill_size=128, max_prefill_tokens=16384, schedule_policy='fcfs', schedule_conservativeness=1.0, page_size=32, hybrid_kvcache_ratio=None, swa_full_tokens_ratio=0.8, disable_hybrid_swa_memory=False, device='cuda', tp_size=1, pp_size=1, max_micro_batch_size=None, stream_interval=1, stream_output=False, random_seed=1064179575, constrained_json_whitespace_pattern=None, watchdog_timeout=300, dist_timeout=None, download_dir=None, base_gpu_id=0, gpu_id_step=1, sleep_on_idle=False, log_level='info', log_level_http=None, log_requests=False, log_requests_level=2, crash_dump_folder=None, show_time_cost=False, enable_metrics=False, enable_metrics_for_all_schedulers=False, bucket_time_to_first_token=None, bucket_inter_token_latency=None, bucket_e2e_request_latency=None, collect_tokens_histogram=False, decode_log_interval=40, enable_request_time_stats_logging=False, kv_events_config=None, gc_warning_threshold_secs=0.0, api_key=None, served_model_name='/opt/models/Qwen/Qwen3-Coder-30B-A3B-Instruct', weight_version='default', chat_template=None, completion_template=None, file_storage_path='sglang_storage', enable_cache_report=False, reasoning_parser=None, tool_call_parser='qwen3_coder', tool_server=None, dp_size=1, load_balance_method='round_robin', dist_init_addr=None, nnodes=1, node_rank=0, json_model_override_args='{}', preferred_sampling_params=None, enable_lora=None, max_lora_rank=None, lora_target_modules=None, lora_paths=None, max_loaded_loras=None, max_loras_per_batch=8, lora_backend='triton', attention_backend=None, decode_attention_backend=None, prefill_attention_backend=None, sampling_backend='flashinfer', grammar_backend='xgrammar', mm_attention_backend=None, speculative_algorithm=None, speculative_draft_model_path=None, speculative_num_steps=None, speculative_eagle_topk=None, speculative_num_draft_tokens=None, speculative_accept_threshold_single=1.0, speculative_accept_threshold_acc=1.0, speculative_token_map=None, ep_size=1, moe_a2a_backend='none', moe_runner_backend='auto', flashinfer_mxfp4_moe_precision='default', enable_flashinfer_allreduce_fusion=False, deepep_mode='auto', ep_num_redundant_experts=0, ep_dispatch_algorithm='static', init_expert_location='trivial', enable_eplb=False, eplb_algorithm='auto', eplb_rebalance_num_iterations=1000, eplb_rebalance_layers_per_chunk=None, expert_distribution_recorder_mode=None, expert_distribution_recorder_buffer_size=1000, enable_expert_distribution_metrics=False, deepep_config=None, moe_dense_tp_size=None, enable_hierarchical_cache=False, hicache_ratio=2.0, hicache_size=0, hicache_write_policy='write_through_selective', hicache_io_backend='kernel', hicache_mem_layout='layer_first', hicache_storage_backend=None, hicache_storage_prefetch_policy='best_effort', enable_double_sparsity=False, ds_channel_config_path=None, ds_heavy_channel_num=32, ds_heavy_token_num=256, ds_heavy_channel_type='qk', ds_sparse_decode_threshold=4096, cpu_offload_gb=0, offload_group_size=-1, offload_num_in_group=1, offload_prefetch_step=1, offload_mode='cpu', disable_radix_cache=False, cuda_graph_max_bs=None, cuda_graph_bs=None, disable_cuda_graph=True, disable_cuda_graph_padding=False, enable_profile_cuda_graph=False, enable_cudagraph_gc=False, enable_nccl_nvls=False, enable_symm_mem=False, disable_flashinfer_cutlass_moe_fp4_allgather=False, enable_tokenizer_batch_encode=False, disable_outlines_disk_cache=False, disable_custom_all_reduce=False, enable_mscclpp=False, disable_overlap_schedule=False, enable_mixed_chunk=False, enable_dp_attention=False, enable_dp_lm_head=False, enable_two_batch_overlap=False, tbo_token_distribution_threshold=0.48, enable_torch_compile=False, torch_compile_max_bs=32, torchao_config='', enable_nan_detection=False, enable_p2p_check=False, triton_attention_reduce_in_fp32=False, triton_attention_num_kv_splits=8, num_continuous_decode_steps=1, delete_ckpt_after_loading=False, enable_memory_saver=False, allow_auto_truncate=False, enable_custom_logit_processor=False, flashinfer_mla_disable_ragged=False, disable_shared_experts_fusion=False, disable_chunked_prefix_cache=False, disable_fast_image_processor=False, enable_return_hidden_states=False, scheduler_recv_interval=1, debug_tensor_dump_output_folder=None, debug_tensor_dump_input_file=None, debug_tensor_dump_inject=False, debug_tensor_dump_prefill_only=False, disaggregation_mode='prefill', disaggregation_transfer_backend='mooncake', disaggregation_bootstrap_port=30003, disaggregation_decode_tp=1, disaggregation_decode_dp=1, disaggregation_prefill_pp=1, disaggregation_ib_device='mlx5_12', num_reserved_decode_tokens=512, pdlb_url=None, custom_weight_loader=[], weight_loader_disable_mmap=False, enable_pdmux=False, sm_group_num=3, enable_ep_moe=False, enable_deepep_moe=False, enable_flashinfer_cutlass_moe=False, enable_flashinfer_trtllm_moe=False, enable_triton_kernel_moe=False, enable_flashinfer_mxfp4_moe=False) [2025-12-12 12:06:08] Using default HuggingFace chat template with detected content format: string W1212 12:06:13.575000 206 torch/utils/cpp_extension.py:2425] TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation. W1212 12:06:13.575000 206 torch/utils/cpp_extension.py:2425] If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST'] to specific architectures. W1212 12:06:13.977000 205 torch/utils/cpp_extension.py:2425] TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation. W1212 12:06:13.977000 205 torch/utils/cpp_extension.py:2425] If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST'] to specific architectures. [2025-12-12 12:06:14] Attention backend not explicitly specified. Use fa3 backend by default. [2025-12-12 12:06:14] Init torch distributed begin. [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [2025-12-12 12:06:14] Init torch distributed ends. mem usage=0.00 GB [2025-12-12 12:06:15] Ignore import error when loading sglang.srt.models.glm4v_moe: No module named 'transformers.models.glm4v_moe' [2025-12-12 12:06:15] Load weight begin. avail mem=76.38 GB Loading safetensors checkpoint shards: 0% Completed | 0/16 [00:00<?, ?it/s] Loading safetensors checkpoint shards: 6% Completed | 1/16 [00:00<00:09, 1.53it/s] Loading safetensors checkpoint shards: 12% Completed | 2/16 [00:01<00:09, 1.45it/s] Loading safetensors checkpoint shards: 19% Completed | 3/16 [00:02<00:08, 1.50it/s] Loading safetensors checkpoint shards: 25% Completed | 4/16 [00:02<00:08, 1.49it/s] Loading safetensors checkpoint shards: 31% Completed | 5/16 [00:03<00:07, 1.51it/s] Loading safetensors checkpoint shards: 38% Completed | 6/16 [00:03<00:06, 1.54it/s] Loading safetensors checkpoint shards: 44% Completed | 7/16 [00:04<00:05, 1.55it/s] Loading safetensors checkpoint shards: 50% Completed | 8/16 [00:05<00:05, 1.52it/s] Loading safetensors checkpoint shards: 56% Completed | 9/16 [00:05<00:04, 1.48it/s] Loading safetensors checkpoint shards: 62% Completed | 10/16 [00:06<00:04, 1.49it/s] Loading safetensors checkpoint shards: 69% Completed | 11/16 [00:07<00:03, 1.49it/s] Loading safetensors checkpoint shards: 75% Completed | 12/16 [00:07<00:02, 1.50it/s] Loading safetensors checkpoint shards: 81% Completed | 13/16 [00:08<00:01, 1.53it/s] Loading safetensors checkpoint shards: 88% Completed | 14/16 [00:09<00:01, 1.51it/s] Loading safetensors checkpoint shards: 94% Completed | 15/16 [00:10<00:00, 1.47it/s] Loading safetensors checkpoint shards: 100% Completed | 16/16 [00:10<00:00, 1.89it/s] Loading safetensors checkpoint shards: 100% Completed | 16/16 [00:10<00:00, 1.57it/s] WARNING: Logging before InitGoogleLogging() is written to STDERR I1212 12:06:26.756870 205 transfer_engine.cpp:422] Metrics reporting is disabled (set MC_TE_METRIC=1 to enable) I1212 12:06:26.756886 205 transfer_engine.cpp:44] Transfer Engine starting. Server: serverip, Metadata: P2PHANDSHAKE, ip_or_host_name: , rpc_port: 0 I1212 12:06:26.756888 205 transfer_engine.cpp:63] Transfer Engine parseHostNameWithPort. server_name: serverip port: 12001 I1212 12:06:26.756910 205 transfer_engine.cpp:114] Transfer Engine RPC using P2P handshake, listening on serverip:16225 I1212 12:06:26.756995 205 transfer_engine.cpp:138] Auto-discovering topology... I1212 12:06:26.759825 205 transfer_engine.cpp:153] Topology discovery complete. Found 1 HCAs. I1212 12:06:26.784847 205 rdma_context.cpp:423] Find best gid index: 0 on mlx5_12/ I1212 12:06:26.785706 205 rdma_context.cpp:125] RDMA device: mlx5_12, LID: 1223, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:00:00:01:26:1a:00:01:01 E1212 12:06:27.214640 865 rdma_context.cpp:205] Failed to register memory 0x7fd4ad548000: Bad address [14] E1212 12:06:27.214716 864 rdma_context.cpp:205] Failed to register memory 0x7fd4ad500000: Bad address [14] E1212 12:06:27.214953 869 rdma_context.cpp:205] Failed to register memory 0x7fd3e6290000: Bad address [14] E1212 12:06:27.214982 866 rdma_context.cpp:205] Failed to register memory 0x7fd4ad590000: Bad address [14] E1212 12:06:27.215179 863 rdma_context.cpp:205] Failed to register memory 0x7fd6bb591000: Bad address [14] E1212 12:06:27.215278 870 rdma_context.cpp:205] Failed to register memory 0x7fd3e62d8000: Bad address [14] E1212 12:06:27.215399 868 rdma_context.cpp:205] Failed to register memory 0x7fd3e6248000: Bad address [14] E1212 12:06:27.215487 867 rdma_context.cpp:205] Failed to register memory 0x7fd3e6200000: Bad address [14] E1212 12:06:27.215593 871 rdma_context.cpp:205] Failed to register memory 0x7fd3e6320000: Bad address [14] E1212 12:06:27.215726 874 rdma_context.cpp:205] Failed to register memory 0x7fd3e6400000: Bad address [14] E1212 12:06:27.215803 875 rdma_context.cpp:205] Failed to register memory 0x7fd3e6448000: Bad address [14] E1212 12:06:27.215821 873 rdma_context.cpp:205] Failed to register memory 0x7fd3e63b0000: Bad address [14] E1212 12:06:27.216099 872 rdma_context.cpp:205] Failed to register memory 0x7fd3e6368000: Bad address [14] E1212 12:06:27.216156 878 rdma_context.cpp:205] Failed to register memory 0x7fd3e6520000: Bad address [14] E1212 12:06:27.216158 876 rdma_context.cpp:205] Failed to register memory 0x7fd3e6490000: Bad address [14] E1212 12:06:27.216158 877 rdma_context.cpp:205] Failed to register memory 0x7fd3e64d8000: Bad address [14] E1212 12:06:27.216207 884 rdma_context.cpp:205] Failed to register memory 0x7fd3e66d8000: Bad address [14] E1212 12:06:27.216306 887 rdma_context.cpp:205] Failed to register memory 0x7fd3e67b0000: Bad address [14] E1212 12:06:27.216359 880 rdma_context.cpp:205] Failed to register memory 0x7fd3e65b0000: Bad address [14] E1212 12:06:27.216394 890 rdma_context.cpp:205] Failed to register memory 0x7fd3e6890000: Bad address [14] E1212 12:06:27.216470 882 rdma_context.cpp:205] Failed to register memory 0x7fd3e6648000: Bad address [14] E1212 12:06:27.216497 892 rdma_context.cpp:205] Failed to register memory 0x7fd3e6920000: Bad address [14] E1212 12:06:27.216533 879 rdma_context.cpp:205] Failed to register memory 0x7fd3e6568000: Bad address [14] E1212 12:06:27.216559 885 rdma_context.cpp:205] Failed to register memory 0x7fd3e6720000: Bad address [14] E1212 12:06:27.216602 895 rdma_context.cpp:205] Failed to register memory 0x7fd3e6a00000: Bad address [14] E1212 12:06:27.216660 894 rdma_context.cpp:205] Failed to register memory 0x7fd3e69b0000: Bad address [14] E1212 12:06:27.216687 893 rdma_context.cpp:205] Failed to register memory 0x7fd3e6968000: Bad address [14] E1212 12:06:27.216733 898 rdma_context.cpp:205] Failed to register memory 0x7fd3e6ad8000: Bad address [14] E1212 12:06:27.216763 899 rdma_context.cpp:205] Failed to register memory 0x7fd3e6b20000: Bad address [14] E1212 12:06:27.216836 881 rdma_context.cpp:205] Failed to register memory 0x7fd3e6600000: Bad address [14] E1212 12:06:27.216861 888 rdma_context.cpp:205] Failed to register memory 0x7fd3e6800000: Bad address [14] E1212 12:06:27.216889 911 rdma_context.cpp:205] Failed to register memory 0x7fd3e6e90000: Bad address [14] E1212 12:06:27.216903 905 rdma_context.cpp:205] Failed to register memory 0x7fd3e6cd8000: Bad address [14] E1212 12:06:27.216934 897 rdma_context.cpp:205] Failed to register memory 0x7fd3e6a90000: Bad address [14] E1212 12:06:27.216951 891 rdma_context.cpp:205] Failed to register memory 0x7fd3e68d8000: Bad address [14] E1212 12:06:27.216984 883 rdma_context.cpp:205] Failed to register memory 0x7fd3e6690000: Bad address [14] E1212 12:06:27.217015 900 rdma_context.cpp:205] Failed to register memory 0x7fd3e6b68000: Bad address [14] E1212 12:06:27.217025 902 rdma_context.cpp:205] Failed to register memory 0x7fd3e6c00000: Bad address [14] E1212 12:06:27.217056 901 rdma_context.cpp:205] Failed to register memory 0x7fd3e6bb0000: Bad address [14] E1212 12:06:27.217085 886 rdma_context.cpp:205] Failed to register memory 0x7fd3e6768000: Bad address [14] E1212 12:06:27.217092 903 rdma_context.cpp:205] Failed to register memory 0x7fd3e6c48000: Bad address [14] E1212 12:06:27.217119 889 rdma_context.cpp:205] Failed to register memory 0x7fd3e6848000: Bad address [14] E1212 12:06:27.217131 904 rdma_context.cpp:205] Failed to register memory 0x7fd3e6c90000: Bad address [14] E1212 12:06:27.217214 907 rdma_context.cpp:205] Failed to register memory 0x7fd3e6d68000: Bad address [14] E1212 12:06:27.217228 914 rdma_context.cpp:205] Failed to register memory 0x7fd3e6f68000: Bad address [14] E1212 12:06:27.217242 920 rdma_context.cpp:205] Failed to register memory 0x7fd3e7120000: Bad address [14] E1212 12:06:27.217285 916 rdma_context.cpp:205] Failed to register memory 0x7fd3e7000000: Bad address [14] E1212 12:06:27.217309 910 rdma_context.cpp:205] Failed to register memory 0x7fd3e6e48000: Bad address [14] E1212 12:06:27.217337 923 rdma_context.cpp:205] Failed to register memory 0x7fd3e7200000: Bad address [14] E1212 12:06:27.217340 918 rdma_context.cpp:205] Failed to register memory 0x7fd3e7090000: Bad address [14] E1212 12:06:27.217363 912 rdma_context.cpp:205] Failed to register memory 0x7fd3e6ed8000: Bad address [14] E1212 12:06:27.217394 919 rdma_context.cpp:205] Failed to register memory 0x7fd3e70d8000: Bad address [14] E1212 12:06:27.217409 908 rdma_context.cpp:205] Failed to register memory 0x7fd3e6db0000: Bad address [14] E1212 12:06:27.217456 909 rdma_context.cpp:205] Failed to register memory 0x7fd3e6e00000: Bad address [14] E1212 12:06:27.217484 915 rdma_context.cpp:205] Failed to register memory 0x7fd3e6fb0000: Bad address [14] E1212 12:06:27.217499 896 rdma_context.cpp:205] Failed to register memory 0x7fd3e6a48000: Bad address [14] E1212 12:06:27.217510 922 rdma_context.cpp:205] Failed to register memory 0x7fd3e71b0000: Bad address [14] E1212 12:06:27.217516 906 rdma_context.cpp:205] Failed to register memory 0x7fd3e6d20000: Bad address [14] E1212 12:06:27.217588 924 rdma_context.cpp:205] Failed to register memory 0x7fd3e7248000: Bad address [14] E1212 12:06:27.217602 921 rdma_context.cpp:205] Failed to register memory 0x7fd3e7168000: Bad address [14] E1212 12:06:27.217659 917 rdma_context.cpp:205] Failed to register memory 0x7fd3e7048000: Bad address [14] E1212 12:06:27.217660 926 rdma_context.cpp:205] Failed to register memory 0x7fd3e72d8000: Bad address [14] E1212 12:06:27.217713 928 rdma_context.cpp:205] Failed to register memory 0x7fd3e7368000: Bad address [14] E1212 12:06:27.217716 913 rdma_context.cpp:205] Failed to register memory 0x7fd3e6f20000: Bad address [14] E1212 12:06:27.217736 925 rdma_context.cpp:205] Failed to register memory 0x7fd3e7290000: Bad address [14] E1212 12:06:27.217805 931 rdma_context.cpp:205] Failed to register memory 0x7fd3e7448000: Bad address [14] E1212 12:06:27.217851 930 rdma_context.cpp:205] Failed to register memory 0x7fd3e7400000: Bad address [14] E1212 12:06:27.217854 929 rdma_context.cpp:205] Failed to register memory 0x7fd3e73b0000: Bad address [14] E1212 12:06:27.217870 927 rdma_context.cpp:205] Failed to register memory 0x7fd3e7320000: Bad address [14] E1212 12:06:27.217926 932 rdma_context.cpp:205] Failed to register memory 0x7fd3e7490000: Bad address [14] E1212 12:06:27.217938 937 rdma_context.cpp:205] Failed to register memory 0x7fd3e7600000: Bad address [14] E1212 12:06:27.217969 936 rdma_context.cpp:205] Failed to register memory 0x7fd3e75b0000: Bad address [14] E1212 12:06:27.218004 938 rdma_context.cpp:205] Failed to register memory 0x7fd3e7648000: Bad address [14] E1212 12:06:27.218010 934 rdma_context.cpp:205] Failed to register memory 0x7fd3e7520000: Bad address [14] E1212 12:06:27.218020 933 rdma_context.cpp:205] Failed to register memory 0x7fd3e74d8000: Bad address [14] E1212 12:06:27.218029 939 rdma_context.cpp:205] Failed to register memory 0x7fd3e7690000: Bad address [14] E1212 12:06:27.218086 940 rdma_context.cpp:205] Failed to register memory 0x7fd3e76d8000: Bad address [14] E1212 12:06:27.218112 941 rdma_context.cpp:205] Failed to register memory 0x7fd3e7720000: Bad address [14] E1212 12:06:27.218137 942 rdma_context.cpp:205] Failed to register memory 0x7fd3e7768000: Bad address [14] E1212 12:06:27.218132 935 rdma_context.cpp:205] Failed to register memory 0x7fd3e7568000: Bad address [14] E1212 12:06:27.218173 943 rdma_context.cpp:205] Failed to register memory 0x7fd3e77b0000: Bad address [14] E1212 12:06:27.218211 944 rdma_context.cpp:205] Failed to register memory 0x7fd3e7800000: Bad address [14] E1212 12:06:27.218248 945 rdma_context.cpp:205] Failed to register memory 0x7fd3e7848000: Bad address [14] E1212 12:06:27.218272 946 rdma_context.cpp:205] Failed to register memory 0x7fd3e7890000: Bad address [14] E1212 12:06:27.218309 947 rdma_context.cpp:205] Failed to register memory 0x7fd3e78d8000: Bad address [14] E1212 12:06:27.218324 948 rdma_context.cpp:205] Failed to register memory 0x7fd3e7920000: Bad address [14] E1212 12:06:27.218366 949 rdma_context.cpp:205] Failed to register memory 0x7fd3e7968000: Bad address [14] E1212 12:06:27.218400 950 rdma_context.cpp:205] Failed to register memory 0x7fd3e79b0000: Bad address [14] E1212 12:06:27.218431 951 rdma_context.cpp:205] Failed to register memory 0x7fd3e7a00000: Bad address [14] E1212 12:06:27.218453 952 rdma_context.cpp:205] Failed to register memory 0x7fd3e7a48000: Bad address [14] E1212 12:06:27.218487 953 rdma_context.cpp:205] Failed to register memory 0x7fd3e7a90000: Bad address [14] E1212 12:06:27.218529 954 rdma_context.cpp:205] Failed to register memory 0x7fd3e7ad8000: Bad address [14] E1212 12:06:27.218549 955 rdma_context.cpp:205] Failed to register memory 0x7fd3e7b20000: Bad address [14] E1212 12:06:27.218586 956 rdma_context.cpp:205] Failed to register memory 0x7fd3e7b68000: Bad address [14] E1212 12:06:27.218617 957 rdma_context.cpp:205] Failed to register memory 0x7fd3e7bb0000: Bad address [14] W1212 12:06:27.218627 205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd6bb591000 length 294912 W1212 12:06:27.218652 205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd4ad500000 length 294912 E1212 12:06:27.218657 958 rdma_context.cpp:205] Failed to register memory 0x7fd3e7c00000: Bad address [14] W1212 12:06:27.218665 205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd4ad548000 length 294912 W1212 12:06:27.218678 205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd4ad590000 length 294912 W1212 12:06:27.218688 205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e6200000 length 294912 W1212 12:06:27.218696 205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e6248000 length 294912 W1212 12:06:27.218705 205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e6290000 length 294912 W1212 12:06:27.218713 205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e62d8000 length 294912 W1212 12:06:27.218721 205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e6320000 length 294912 W1212 12:06:27.218729 205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e6368000 length 294912 W1212 12:06:27.218737 205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e63b0000 length 294912 W1212 12:06:27.218745 205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e6400000 length 294912 W1212 12:06:27.218753 205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e6448000 length 294912 W1212 12:06:27.218761 205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e6490000 length 294912 W1212 12:06:27.218770 205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e64d8000 length 294912 W1212 12:06:27.218777 205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e6520000 length 294912 W1212 12:06:27.218786 205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e6568000 length 294912 W1212 12:06:27.218793 205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e65b0000 length 294912 W1212 12:06:27.218801 205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e6600000 length 294912 W1212 12:06:27.218809 205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e6648000 length 294912 W1212 12:06:27.218816 205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e6690000 length 294912 W1212 12:06:27.218823 205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e66d8000 length 294912 W1212 12:06:27.218832 205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e6720000 length 294912 W1212 12:06:27.218840 205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e6768000 length 294912 W1212 12:06:27.218847 205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e67b0000 length 294912 W1212 12:06:27.218854 205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e6800000 length 294912 W1212 12:06:27.218863 205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e6848000 length 294912 W1212 12:06:27.218870 205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e6890000 length 294912 W1212 12:06:27.218878 205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e68d8000 length 294912 W1212 12:06:27.218885 205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e6920000 length 294912 W1212 12:06:27.218894 205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e6968000 length 294912 W1212 12:06:27.218901 205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e69b0000 length 294912 W1212 12:06:27.218907 205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e6a00000 length 294912 W1212 12:06:27.218915 205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e6a48000 length 294912 W1212 12:06:27.218922 205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e6a90000 length 294912 W1212 12:06:27.218930 205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e6ad8000 length 294912 W1212 12:06:27.218936 205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e6b20000 length 294912 W1212 12:06:27.218945 205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e6b68000 length 294912 W1212 12:06:27.218952 205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e6bb0000 length 294912 W1212 12:06:27.218959 205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e6c00000 length 294912 W1212 12:06:27.218966 205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e6c48000 length 294912 W1212 12:06:27.218973 205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e6c90000 length 294912 W1212 12:06:27.218981 205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e6cd8000 length 294912 W1212 12:06:27.218987 205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e6d20000 length 294912 W1212 12:06:27.218995 205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e6d68000 length 294912 W1212 12:06:27.219002 205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e6db0000 length 294912 W1212 12:06:27.219009 205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e6e00000 length 294912 W1212 12:06:27.219017 205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e6e48000 length 294912 W1212 12:06:27.219024 205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e6e90000 length 294912 W1212 12:06:27.219030 205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e6ed8000 length 294912 W1212 12:06:27.219038 205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e6f20000 length 294912 W1212 12:06:27.219045 205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e6f68000 length 294912 W1212 12:06:27.219053 205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e6fb0000 length 294912 W1212 12:06:27.219060 205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e7000000 length 294912 W1212 12:06:27.219069 205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e7048000 length 294912 W1212 12:06:27.219075 205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e7090000 length 294912 W1212 12:06:27.219081 205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e70d8000 length 294912 W1212 12:06:27.219089 205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e7120000 length 294912 W1212 12:06:27.219096 205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e7168000 length 294912 W1212 12:06:27.219102 205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e71b0000 length 294912 W1212 12:06:27.219110 205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e7200000 length 294912 W1212 12:06:27.219117 205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e7248000 length 294912 W1212 12:06:27.219125 205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e7290000 length 294912 W1212 12:06:27.219131 205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e72d8000 length 294912 W1212 12:06:27.219138 205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e7320000 length 294912 W1212 12:06:27.219146 205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e7368000 length 294912 W1212 12:06:27.219152 205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e73b0000 length 294912 W1212 12:06:27.219159 205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e7400000 length 294912 W1212 12:06:27.219166 205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e7448000 length 294912 W1212 12:06:27.219173 205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e7490000 length 294912 W1212 12:06:27.219179 205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e74d8000 length 294912 W1212 12:06:27.219187 205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e7520000 length 294912 W1212 12:06:27.219192 205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e7568000 length 294912 W1212 12:06:27.219199 205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e75b0000 length 294912 W1212 12:06:27.219206 205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e7600000 length 294912 W1212 12:06:27.219213 205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e7648000 length 294912 W1212 12:06:27.219219 205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e7690000 length 294912 W1212 12:06:27.219226 205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e76d8000 length 294912 W1212 12:06:27.219233 205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e7720000 length 294912 W1212 12:06:27.219239 205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e7768000 length 294912 W1212 12:06:27.219247 205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e77b0000 length 294912 W1212 12:06:27.219254 205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e7800000 length 294912 W1212 12:06:27.219262 205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e7848000 length 294912 W1212 12:06:27.219269 205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e7890000 length 294912 W1212 12:06:27.219275 205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e78d8000 length 294912 W1212 12:06:27.219282 205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e7920000 length 294912 W1212 12:06:27.219290 205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e7968000 length 294912 W1212 12:06:27.219296 205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e79b0000 length 294912 W1212 12:06:27.219302 205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e7a00000 length 294912 W1212 12:06:27.219309 205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e7a48000 length 294912 W1212 12:06:27.219316 205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e7a90000 length 294912 W1212 12:06:27.219324 205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e7ad8000 length 294912 W1212 12:06:27.219331 205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e7b20000 length 294912 W1212 12:06:27.219338 205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e7b68000 length 294912 W1212 12:06:27.219344 205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e7bb0000 length 294912 W1212 12:06:27.219354 205 rdma_transport.cpp:166] RdmaTransport: Failed to register memory: addr 0x7fd3e7c00000 length 294912 [2025-12-12 12:06:27] INFO: Started server process [7] [2025-12-12 12:06:27] INFO: Waiting for application startup. [2025-12-12 12:06:27] INFO: Application startup complete. [2025-12-12 12:06:27] INFO: Uvicorn running on http://0.0.0.0:30000 (Press CTRL+C to quit) [2025-12-12 12:06:28] INFO: 127.0.0.1:34916 - "GET /get_model_info HTTP/1.1" 200 OK [2025-12-12 12:06:28] Start of pd disaggregation warmup ... [2025-12-12 12:06:32] INFO: 127.0.0.1:34920 - "POST /generate HTTP/1.1" 200 OK [2025-12-12 12:06:32] End of prefill disaggregation mode warmup with status 200, resp: [{'text': '%', 'output_ids': [0, 1, 2, 3, 4], 'meta_info': {'id': '2420bb71081f4e2c8d621f15c919728a', 'finish_reason': {'type': 'length', 'length': 0}, 'prompt_tokens': 4, 'weight_version': 'default', 'completion_tokens': 1, 'cached_tokens': 0, 'e2e_latency': 3.9578440189361572}}] [2025-12-12 12:06:32] The server is fired up and ready to roll! ``` Despite "The server is fired up", the proxy drops on the requests the following: `KVTransferError(bootstrap_room=6677116944225248479): Could not fetch prefill parallel info from bootstrap_addr: sglang-prefill-svc:30000` which else installations should I add? Thank you in advance!
bsakaguchi 2026-10-29 21:14:09 +00:00
bsakaguchi commented 2026-10-29 21:14:10 +00:00
Author
Owner
Copy link

@ykwd commented on GitHub (Dec 15, 2025):

Based on the logs, it appears that the failure occurred while registering memory with the RDMA NIC. A possible cause is a Docker configuration issue that prevents proper access to the RDMA device.

<!-- gh-comment-id:3654363734 --> @ykwd commented on GitHub (Dec 15, 2025): Based on the logs, it appears that the failure occurred while registering memory with the RDMA NIC. A possible cause is a Docker configuration issue that prevents proper access to the RDMA device.
bsakaguchi commented 2026-10-29 21:14:10 +00:00
Author
Owner
Copy link

@UralmashFox commented on GitHub (Dec 15, 2025):

Based on the logs, it appears that the failure occurred while registering memory with the RDMA NIC. A possible cause is a Docker configuration issue that prevents proper access to the RDMA device.

I’ve indeed tried quite a few things on my end - mostly through the Helm chart - such as:

  1. Adjusting memory limits and page sizes (--max-total-tokens, --page-size, etc.),
  2. Enabling privileged: true and mounting /dev/infiniband,
  3. Ensuring the DISAGG_IB_DEVICE env var points to a working HCA (mlx5_12 or any other),
  4. Using a headless service for prefill and explicitly setting the bootstrap port.
  5. In monolithic mode (without prefill/decode disaggregation), the same model and image (sglang:v0.5.1.post2-cu126) work perfectly - so the base setup is sound.

I’d really appreciate any guidance on what to check next - whether it’s a host-level RDMA setting, a missing capability, or a known issue with this SGLang version.
I can also do the same attemps I mentioned above and provide you information that you think could be useful, just tell me what exactly

<!-- gh-comment-id:3654626415 --> @UralmashFox commented on GitHub (Dec 15, 2025): > Based on the logs, it appears that the failure occurred while registering memory with the RDMA NIC. A possible cause is a Docker configuration issue that prevents proper access to the RDMA device. I’ve indeed tried quite a few things on my end - mostly through the Helm chart - such as: 1) Adjusting memory limits and page sizes (--max-total-tokens, --page-size, etc.), 2) Enabling privileged: true and mounting /dev/infiniband, 3) Ensuring the DISAGG_IB_DEVICE env var points to a working HCA (mlx5_12 or any other), 4) Using a headless service for prefill and explicitly setting the bootstrap port. 5) In monolithic mode (without prefill/decode disaggregation), the same model and image (sglang:v0.5.1.post2-cu126) work perfectly - so the base setup is sound. I’d really appreciate any guidance on what to check next - whether it’s a host-level RDMA setting, a missing capability, or a known issue with this SGLang version. I can also do the same attemps I mentioned above and provide you information that you think could be useful, just tell me what exactly
bsakaguchi commented 2026-10-29 21:14:10 +00:00
Author
Owner
Copy link

@stmatengss commented on GitHub (Dec 15, 2025):

Could you try to test the connectivity between prefill and decode?

<!-- gh-comment-id:3655343654 --> @stmatengss commented on GitHub (Dec 15, 2025): Could you try to test the connectivity between prefill and decode?
bsakaguchi commented 2026-10-29 21:14:10 +00:00
Author
Owner
Copy link

@stmatengss commented on GitHub (Dec 15, 2025):

Make sure GDR is supported. REF: https://github.com/kvcache-ai/Mooncake/blob/main/docs/source/troubleshooting/troubleshooting.md#sglang-common-questions

<!-- gh-comment-id:3655349107 --> @stmatengss commented on GitHub (Dec 15, 2025): Make sure GDR is supported. REF: https://github.com/kvcache-ai/Mooncake/blob/main/docs/source/troubleshooting/troubleshooting.md#sglang-common-questions
bsakaguchi commented 2026-10-29 21:14:10 +00:00
Author
Owner
Copy link

@UralmashFox commented on GitHub (Dec 16, 2025):

Make sure GDR is supported. REF: https://github.com/kvcache-ai/Mooncake/blob/main/docs/source/troubleshooting/troubleshooting.md#sglang-common-questions

currently (till the end of January) I switched to another environment with less gpu sources and much-much lighter llm. If the results below do not make sence with described logs abode, let me know, we will postpone the discussion to later

if to run from the prefill pod, the result is the following:

# ibv_devices
    device                 node GUID
    ------              ----------------
# lspci | grep rdma
# lsmod | grep -E 'ib_core|mlx4_core|mlx5_core|nvidia_peer_mem'
ib_core               393216  0
# lsmod | grep peer_mem
# lsmod | grep nvidia_peer_mem
#
<!-- gh-comment-id:3660055139 --> @UralmashFox commented on GitHub (Dec 16, 2025): > Make sure GDR is supported. REF: https://github.com/kvcache-ai/Mooncake/blob/main/docs/source/troubleshooting/troubleshooting.md#sglang-common-questions currently (till the end of January) I switched to another environment with less gpu sources and much-much lighter llm. If the results below do not make sence with described logs abode, let me know, we will postpone the discussion to later if to run from the prefill pod, the result is the following: ``` # ibv_devices device node GUID ------ ---------------- # lspci | grep rdma # lsmod | grep -E 'ib_core|mlx4_core|mlx5_core|nvidia_peer_mem' ib_core 393216 0 # lsmod | grep peer_mem # lsmod | grep nvidia_peer_mem # ```
bsakaguchi commented 2026-10-29 21:14:10 +00:00
Author
Owner
Copy link

@UralmashFox commented on GitHub (Dec 16, 2025):

Could you try to test the connectivity between prefill and decode?

From the prefill pod I run the following:

curl -X POST http://sglang-proxy-svc:30002/generate -H "Content-Type: application/json" -d '{
  "text": "Let me tell you a long story ",
  "sampling_params": {
    "temperature": 0
  }
}'

No new logs from prefill;
Logs from proxy:
"POST /generate HTTP/1.1" 500 Internal Server Error
Logs from decode:

[2025-12-16 11:49:05] Failed to get prefill parallel info: 503, upstream connect error or disconnect/reset before headers. retried and the latest reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused
[2025-12-16 11:49:05] Decode handshake failed for request rank=0 decode_req.req.rid='d2fecacb4504446e92c3fe5cb8fe381b' decode_req.req.bootstrap_room=4079317304456078487 with exception KVTransferError(bootstrap_room=4079317304456078487): Could not fetch prefill parallel info from bootstrap_addr: sglang-prefill-svc:30000
[2025-12-16 11:49:05] INFO:     127.0.0.6:47623 - "POST /generate HTTP/1.1" 500 Internal Server Error

How decode was run:

python -m sglang.launch_server
      --model-path=/opt/models/Qwen/Qwen2.5-Coder-0.5B-Instruct 
      --disaggregation-mode=decode
      --port=30001 
      --disaggregation-bootstrap-port=30000 
      --disaggregation-transfer-backend=mooncake
      --disaggregation-ib-device=mlx5_12
      --host=0.0.0.0
      --tp-size=1
      --trust-remote-code
     --disable-cuda-graph 
     --page-size=256
     --max-total-tokens=2048
     --chunked-prefill-size=256

How prefill was run:

python -m sglang.launch_server
      --model-path=/opt/models/Qwen/Qwen2.5-Coder-0.5B-Instruct
      --disaggregation-mode=prefill \
      --port=30000 \
      --disaggregation-transfer-backend=mooncake \
      --host=0.0.0.0 \
      --page-size=256 \
      --disaggregation-bootstrap-port=30003 \
      --max-total-tokens=1024 \
      --chunked-prefill-size=256 \
      --mem-fraction-static 0.95 \
      --tool-call-parser qwen25 \

I use --disaggregation-bootstrap-port=30000 just because I can not use 8xxx, only 3xxxxx. Actually, I'm not sure how exactly bootstrap-port should be modified - I tried to play with extra flags in params for prefill

Additionlly from decode pod I tried to do the following request:
curl http://sglang-prefill-svc:30003/route
and get Missing inputs for bootstrap server.
Then I tried to extend the request with the following:

curl -X PUT http://sglang-prefill-svc:30003/route \
  -H "Content-Type: application/json" \
  -d '{
    "dp_size": 1,
    "dp_rank": 0,
    "tp_size": 1,
    "tp_rank": 0,
    "pp_size": 1,
    "pp_rank": 0
  }'

The prefill logs:

[2025-12-16 16:23:52] Error handling request from 127.0.0.6
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/dist-packages/aiohttp/web_protocol.py", line 510, in _handle_request
    resp = await request_handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/aiohttp/web_app.py", line 569, in _handle
    return await handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/sgl-workspace/sglang/python/sglang/srt/disaggregation/mooncake/conn.py", line 1578, in _handle_route
    return await self._handle_route_put(request)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/sgl-workspace/sglang/python/sglang/srt/disaggregation/mooncake/conn.py", line 1588, in _handle_route_put
    role = data["role"]
           ~~~~^^^^^^^^
KeyError: 'role'

Based on this I make an assumption that both - prefill and bootstrap reply on decode requests.

<!-- gh-comment-id:3660147439 --> @UralmashFox commented on GitHub (Dec 16, 2025): > Could you try to test the connectivity between prefill and decode? From the prefill pod I run the following: ``` curl -X POST http://sglang-proxy-svc:30002/generate -H "Content-Type: application/json" -d '{ "text": "Let me tell you a long story ", "sampling_params": { "temperature": 0 } }' ``` No new logs from prefill; Logs from proxy: `"POST /generate HTTP/1.1" 500 Internal Server Error` Logs from decode: ``` [2025-12-16 11:49:05] Failed to get prefill parallel info: 503, upstream connect error or disconnect/reset before headers. retried and the latest reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused [2025-12-16 11:49:05] Decode handshake failed for request rank=0 decode_req.req.rid='d2fecacb4504446e92c3fe5cb8fe381b' decode_req.req.bootstrap_room=4079317304456078487 with exception KVTransferError(bootstrap_room=4079317304456078487): Could not fetch prefill parallel info from bootstrap_addr: sglang-prefill-svc:30000 [2025-12-16 11:49:05] INFO: 127.0.0.6:47623 - "POST /generate HTTP/1.1" 500 Internal Server Error ``` How decode was run: ``` python -m sglang.launch_server --model-path=/opt/models/Qwen/Qwen2.5-Coder-0.5B-Instruct --disaggregation-mode=decode --port=30001 --disaggregation-bootstrap-port=30000 --disaggregation-transfer-backend=mooncake --disaggregation-ib-device=mlx5_12 --host=0.0.0.0 --tp-size=1 --trust-remote-code --disable-cuda-graph --page-size=256 --max-total-tokens=2048 --chunked-prefill-size=256 ``` How prefill was run: ``` python -m sglang.launch_server --model-path=/opt/models/Qwen/Qwen2.5-Coder-0.5B-Instruct --disaggregation-mode=prefill \ --port=30000 \ --disaggregation-transfer-backend=mooncake \ --host=0.0.0.0 \ --page-size=256 \ --disaggregation-bootstrap-port=30003 \ --max-total-tokens=1024 \ --chunked-prefill-size=256 \ --mem-fraction-static 0.95 \ --tool-call-parser qwen25 \ ``` I use --disaggregation-bootstrap-port=30000 just because I can not use 8xxx, only 3xxxxx. Actually, I'm not sure how exactly bootstrap-port should be modified - I tried to play with extra flags in params for prefill Additionlly from decode pod I tried to do the following request: `curl http://sglang-prefill-svc:30003/route` and get **Missing inputs for bootstrap server.** Then I tried to extend the request with the following: ``` curl -X PUT http://sglang-prefill-svc:30003/route \ -H "Content-Type: application/json" \ -d '{ "dp_size": 1, "dp_rank": 0, "tp_size": 1, "tp_rank": 0, "pp_size": 1, "pp_rank": 0 }' ``` The prefill logs: ``` [2025-12-16 16:23:52] Error handling request from 127.0.0.6 Traceback (most recent call last): File "/usr/local/lib/python3.12/dist-packages/aiohttp/web_protocol.py", line 510, in _handle_request resp = await request_handler(request) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/dist-packages/aiohttp/web_app.py", line 569, in _handle return await handler(request) ^^^^^^^^^^^^^^^^^^^^^^ File "/sgl-workspace/sglang/python/sglang/srt/disaggregation/mooncake/conn.py", line 1578, in _handle_route return await self._handle_route_put(request) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/sgl-workspace/sglang/python/sglang/srt/disaggregation/mooncake/conn.py", line 1588, in _handle_route_put role = data["role"] ~~~~^^^^^^^^ KeyError: 'role' ``` Based on this I make an assumption that both - prefill and bootstrap reply on decode requests.
bsakaguchi commented 2026-10-29 21:14:11 +00:00
Author
Owner
Copy link

@github-actions[bot] commented on GitHub (Jun 10, 2026):

This issue has had no activity for 90 days and will be closed in 7 days if there is no further activity. Please comment or react if it should stay open.

<!-- gh-comment-id:4666578991 --> @github-actions[bot] commented on GitHub (Jun 10, 2026): This issue has had no activity for 90 days and will be closed in 7 days if there is no further activity. Please comment or react if it should stay open.
bsakaguchi commented 2026-10-29 21:14:11 +00:00
Author
Owner
Copy link

@github-actions[bot] commented on GitHub (Jun 18, 2026):

Closing due to 3 months of inactivity. If this is still relevant, please comment and we can reopen.

<!-- gh-comment-id:4738056592 --> @github-actions[bot] commented on GitHub (Jun 18, 2026): Closing due to 3 months of inactivity. If this is still relevant, please comment and we can reopen.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
te-coro
staryxchen-tent-capability-research
feature/musa-ep-jit
zxpdemonio/fix-tent-bootstrap-admission
add-store-rs-tone-tests
scatter-rdma-completion-pr2
P2P-Mooncake-Store
release/v0.3.14
feat/tent-rdma-adaptive-adapter
fix/ep-local-counter-gpu-scope
codex/kvcs-oplog-eviction
give-me-a-star
cruz/mooncake-kvcs-lowlevel-pr1-20261008
ylt-handshake-limits
ci/rocm-e2e-tent
sima/docs-pypi-downloads
fix-ci/hicache-mooncake-metadata-loopback
sima/cuda-driver-loading
sima/ci-performance-gate
fix/tent-rdma-unknown-topology-rail-pairing
ci/tent-e2e-dual-job
mc-patch-1
codex/p2p-merge-prerequisites
sima/store-hca-numa-placement
codex/p2p-merge-cli
codex/p2p-merge-ipv6
copilot/fix-ctests-job-failure
czh/pg-device-collectives-pr6
dependabot/cargo/mooncake-store-rs/pyo3-0.29.0
czh/pg-device-collectives-pr5
codex/segment-pool-master-cutover
sima/nightly-ci-recovery
ci/tent-params
stmatengss-patch-1
dependabot/cargo/mooncake-store-rs/rand-0.8.8
dependabot/cargo/mooncake-store-rs/opentelemetry_sdk-0.32.1
dependabot/cargo/mooncake-store-rs/xxhash-rust-0.8.16
codex/migrate-store-rest-service
fix/tone-wheel-arch-selection
czh/pg-device-collectives-pr4
czh/pg-device-collectives-pr3
ketong/fix_moe_tc
copilot/fix-test-wheel-ubuntu-job
fix/ci-wheel-torch-pg-pin
codex/migrate-structured-object-storage
codex/migrate-allocator-python
codex/migrate-vllm-python
codex/migrate-shared-segment
ci/fix_sgl_eval_install
codex/segment-pool-nof-placement-cleanup
feat/pip-native-release
copilot/bugfix-ha-standby-promotion
copilot/mooncake-store-transport-settings
copilot/fix-tensor-support-issues
copilot/optimize-rl-direction
copilot/mooncake-storeagentic-workflow
copilot/fix-client-reconnect-timeout
copilot/mooncake-error-debugging
copilot/mooncake-ipv6-support
feat/nccl-host-python-api
dependabot/go_modules/mooncake-p2p-store/src/p2pstore/google.golang.org/grpc-1.82.1
copilot/migrate-and-move-operations
copilot/migrate-and-move-operations-again
refactor/centralize-env-vars
idhanani/mooncake-retain-groups
copilot/codeownerreviewer
copilot/give-feedback-on-issue-2123
copilot/create-issue-rfc
copilot/break-cuda-monopoly-logic
copilot/compare-cudamemcpy-cu12-cu13
copilot/report-ci-error-issue
copilot/research-commit-history-analysis
copilot/explain-mooncake-store-service
copilot/analyze-project-functionality
copilot/sub-pr-1680
copilot/review-pr-1680
copilot/sub-pr-1597
copilot/fix-github-actions-job-failure
copilot/move-test-sglang-integration-job
v0.3.14
v0.3.14-pre-4
v0.3.14-pre-3
v0.3.14-pre-2
v0.3.14-pre
v0.3.14-rc1
v0.3.13.post1
v0.3.13-rc2
v0.3.13-rc1
v0.3.13
v0.3.12.post1
v0.3.12
v0.3.11-rc1
v0.3.11.post1
v0.3.11
v0.3.10.post2
v0.3.10.post1
v0.3.10
v0.3.9
v0.3.8.post1
v0.3.8
v0.3.7.post2
v0.3.7.post1
v0.3.7
v0.3.6.post1
v0.3.6
v0.3.5
v0.3.4.post2
v0.3.4.post1
v0.3.4
v0.3.3.post2
v0.3.3.post1
v0.3.3
v0.3.2.post1
v0.3.2
v0.3.1
v0.3.0
v0.3.0-beta
v0.3.0-alpha
v0.2.0
Labels
Clear labels   
ascend

   
auto-closed

   
bug

   
bug

   
CI/Build

   
Common

   
documentation

   
enhancement

   
feature request

   
good first issue

   
help wanted

   
high priority

   
MetaX

   
nightly-failure

   
pull-request
Mirrored from GitHub Pull Request

  
PyTorch Backend

   
question

   
RFC

   
Roadmap

   
stale

   
Store

   
Tests

   
Transfer Engine
No labels
ascend
auto-closed
bug
bug
CI/Build
Common
documentation
enhancement
feature request
good first issue
help wanted
high priority
MetaX
nightly-failure
pull-request
PyTorch Backend
question
RFC
Roadmap
stale
Store
Tests
Transfer Engine
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
kvcache-ai/Mooncake#411
No description provided.