[GH-ISSUE #4504] [Bug]: Ascend NPU build - loading mooncake.store aborts the process at exit (heap corruption -> SIGABRT) #1146

New issue

Closed

opened 2026-10-29 21:23:19 +00:00 by bsakaguchi · 3 comments

bsakaguchi commented 2026-10-29 21:23:19 +00:00

Owner

Copy link

Originally created by @brandneway on GitHub (Oct 8, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/4504

Describe the bug

On the Ascend NPU build, merely loading the Python extension mooncake.store (or mooncake.engine) makes any process abort during interpreter shutdown, even though nothing else is done with Mooncake. glibc reports a corrupted heap block and raises SIGABRT, so the process exits with a non-zero status.

This is not about using Mooncake: the failure fires in processes that only import it and then exit normally.

Environment

• Package: mooncake-transfer-engine-npu 0.3.11.post1 (the wheel reports itself as "Python binding of a Mooncake library using pybind11 (Ascend NPU version)")
• Hardware: Atlas 800I A3 server, 8x Ascend910 (16 dies)
• CANN: 9.1.0 (/usr/local/Ascend/cann-9.1.0)
• OS / libc: aarch64 Linux (glibc), Python 3.12

Reproduction

$ python3 -c "import mooncake.store"
corrupted size vs. prev_size
Aborted (core dumped)          # rc=134, 5/5 runs

mooncake.engine behaves identically. Two refinements that narrow it down:

# 1. No Mooncake Python code has to run -- dlopen of the extension alone is enough
$ python3 -c "import ctypes; ctypes.CDLL('<site-packages>/mooncake/store.so')"
corrupted size vs. prev_size
Aborted (core dumped)          # rc=134, 5/5 runs

# 2. It only happens on the normal-exit path
$ python3 -c "import mooncake.store, os; os._exit(0)"     # rc=0  (skips shutdown)
$ python3 -c "import mooncake"                            # rc=0  (package __init__ is empty)

Backtrace

corrupted size vs. prev_size
Program received signal SIGABRT
#0  __pthread_kill_implementation
#1  __pthread_kill_internal
#2  __GI_raise
#3  __GI_abort
#4  __libc_message
#5  malloc_printerr  <-- "corrupted size vs. prev_size"
#6  unlink_chunk          (malloc.c:1629)
#7  malloc_consolidate    (malloc.c:4779)
#8  _int_free             (malloc.c:4674)
#9  __GI___libc_free
#10 ?? () from /usr/local/Ascend/cann-9.1.0/lib64/libascend_trace.so
#11 ?? () from /usr/local/Ascend/cann-9.1.0/lib64/libascend_trace.so
#12 ?? () from /usr/local/Ascend/cann-9.1.0/lib64/libascend_trace.so
#13 _dl_fini              (elf/dl-fini.c:142)
#14 __run_exit_handlers
#15 __GI_exit
#16 __libc_start_call_main

free is called from a CANN finalizer in libascend_trace.so during _dl_fini; the block it is releasing was already corrupt by then.

What we ruled out

Hypothesis	Experiment	Result
Python object heap corruption	PYTHONMALLOC=debug	nothing reported — this is the C malloc heap
Mooncake's Python code	ctypes.CDLL(store.so) with no Python code running	still aborts
A single offending shared object	ctypes.CDLL() on libcann_hixl.so, libllm_datadist.so, libascend_trace.so individually	all exit 0
A generic "load a lot of CANN libs" issue	import torch_npu (also maps libascend_trace.so)	exits 0
The exit path itself	os._exit(0)	exits 0 — confirms shutdown-only

ldd store.so shows it links ~34 CANN libraries, among them libcann_hixl.so, libllm_datadist.so, ascend_transport.so (the latter ships inside the mooncake package itself).

What we could not determine

We did not identify which code writes the corrupted bytes. Binary-searching the CANN library set produced non-monotonic results — a 10-library subset loads cleanly while dropping one of several members makes it abort again — which is the signature of heap-layout-sensitive corruption rather than a deterministic API misuse. GLIBC_TUNABLES=glibc.malloc.check=3 also fails to catch it any earlier. The free on the stack is a victim, not the culprit.

Impact

Any short-lived process that imports mooncake.store and then exits normally is reported as failed by its parent, because the exit status is non-zero even though the work completed. Concretely, this breaks vLLM-Omni: the subprocess vLLM forks to inspect model architectures imports the whole vllm-omni package, so it dies on the way out and the parent refuses to start the deployment:

RuntimeError: Model architectures ['MiniMaxH3Encoder'] failed to be inspected

This makes every Ascend deployment containing an LLM stage fail to start. Reported upstream in vllm-project/vllm-omni#8624, which works around it by no longer importing the optional connectors eagerly — but that only removes the accidental exposure; a deployment that genuinely selects a Mooncake connector still hits this abort.

We would appreciate any pointers on whether this is a known issue with the Ascend build, whether it is fixed in a newer wheel, or which component to follow up with.

Originally created by @brandneway on GitHub (Oct 8, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/4504 ### Describe the bug On the Ascend NPU build, merely loading the Python extension `mooncake.store` (or `mooncake.engine`) makes **any process abort during interpreter shutdown**, even though nothing else is done with Mooncake. glibc reports a corrupted heap block and raises SIGABRT, so the process exits with a non-zero status. This is not about *using* Mooncake: the failure fires in processes that only import it and then exit normally. ### Environment - Package: `mooncake-transfer-engine-npu` **0.3.11.post1** (the wheel reports itself as "Python binding of a Mooncake library using pybind11 (Ascend NPU version)") - Hardware: Atlas 800I A3 server, 8x Ascend910 (16 dies) - CANN: **9.1.0** (`/usr/local/Ascend/cann-9.1.0`) - OS / libc: aarch64 Linux (glibc), Python 3.12 ### Reproduction ```bash $ python3 -c "import mooncake.store" corrupted size vs. prev_size Aborted (core dumped) # rc=134, 5/5 runs ``` `mooncake.engine` behaves identically. Two refinements that narrow it down: ```bash # 1. No Mooncake Python code has to run -- dlopen of the extension alone is enough $ python3 -c "import ctypes; ctypes.CDLL('<site-packages>/mooncake/store.so')" corrupted size vs. prev_size Aborted (core dumped) # rc=134, 5/5 runs # 2. It only happens on the normal-exit path $ python3 -c "import mooncake.store, os; os._exit(0)" # rc=0 (skips shutdown) $ python3 -c "import mooncake" # rc=0 (package __init__ is empty) ``` ### Backtrace ``` corrupted size vs. prev_size Program received signal SIGABRT #0 __pthread_kill_implementation #1 __pthread_kill_internal #2 __GI_raise #3 __GI_abort #4 __libc_message #5 malloc_printerr <-- "corrupted size vs. prev_size" #6 unlink_chunk (malloc.c:1629) #7 malloc_consolidate (malloc.c:4779) #8 _int_free (malloc.c:4674) #9 __GI___libc_free #10 ?? () from /usr/local/Ascend/cann-9.1.0/lib64/libascend_trace.so #11 ?? () from /usr/local/Ascend/cann-9.1.0/lib64/libascend_trace.so #12 ?? () from /usr/local/Ascend/cann-9.1.0/lib64/libascend_trace.so #13 _dl_fini (elf/dl-fini.c:142) #14 __run_exit_handlers #15 __GI_exit #16 __libc_start_call_main ``` `free` is called from a CANN finalizer in `libascend_trace.so` during `_dl_fini`; the block it is releasing was already corrupt by then. ### What we ruled out | Hypothesis | Experiment | Result | | --- | --- | --- | | Python object heap corruption | `PYTHONMALLOC=debug` | nothing reported — this is the C `malloc` heap | | Mooncake's Python code | `ctypes.CDLL(store.so)` with no Python code running | still aborts | | A single offending shared object | `ctypes.CDLL()` on `libcann_hixl.so`, `libllm_datadist.so`, `libascend_trace.so` individually | all exit 0 | | A generic "load a lot of CANN libs" issue | `import torch_npu` (also maps `libascend_trace.so`) | exits 0 | | The exit path itself | `os._exit(0)` | exits 0 — confirms shutdown-only | `ldd store.so` shows it links ~34 CANN libraries, among them `libcann_hixl.so`, `libllm_datadist.so`, `ascend_transport.so` (the latter ships inside the mooncake package itself). ### What we could not determine We did **not** identify which code writes the corrupted bytes. Binary-searching the CANN library set produced **non-monotonic** results — a 10-library subset loads cleanly while dropping one of several members makes it abort again — which is the signature of heap-layout-sensitive corruption rather than a deterministic API misuse. `GLIBC_TUNABLES=glibc.malloc.check=3` also fails to catch it any earlier. The `free` on the stack is a victim, not the culprit. ### Impact Any short-lived process that imports `mooncake.store` and then exits normally is reported as failed by its parent, because the exit status is non-zero even though the work completed. Concretely, this breaks vLLM-Omni: the subprocess vLLM forks to inspect model architectures imports the whole vllm-omni package, so it dies on the way out and the parent refuses to start the deployment: ``` RuntimeError: Model architectures ['MiniMaxH3Encoder'] failed to be inspected ``` This makes every Ascend deployment containing an LLM stage fail to start. Reported upstream in vllm-project/vllm-omni#8624, which works around it by no longer importing the optional connectors eagerly — but that only removes the *accidental* exposure; a deployment that genuinely selects a Mooncake connector still hits this abort. We would appreciate any pointers on whether this is a known issue with the Ascend build, whether it is fixed in a newer wheel, or which component to follow up with.

bsakaguchi closed this issue 2026-10-29 21:23:19 +00:00

bsakaguchi commented 2026-10-29 21:23:20 +00:00

Author

Owner

Copy link

@github-actions[bot] commented on GitHub (Oct 8, 2026):

Thanks for opening this issue, @brandneway!

Field	Value
Issue	#4504
GitHub user ID	53851298
Reporter	@brandneway

A maintainer will triage this when possible. To help us respond faster, please include:

• Mooncake version or commit SHA
• Environment (OS, CUDA/driver, RDMA stack if relevant)
• Steps to reproduce and expected vs. actual behavior

Useful links: Documentation · Contributing guide

This message was posted automatically by the issue bot.

<!-- gh-comment-id:6054525588 --> @github-actions[bot] commented on GitHub (Oct 8, 2026): Thanks for opening this issue, @brandneway! | Field | Value | |-------|-------| | **Issue** | #4504 | | **GitHub user ID** | `53851298` | | **Reporter** | @brandneway | A maintainer will triage this when possible. To help us respond faster, please include: - Mooncake version or commit SHA - Environment (OS, CUDA/driver, RDMA stack if relevant) - Steps to reproduce and expected vs. actual behavior Useful links: [Documentation](https://kvcache-ai.github.io/Mooncake/) · [Contributing guide](https://github.com/kvcache-ai/Mooncake/blob/main/CONTRIBUTING.md) > This message was posted automatically by the issue bot.

bsakaguchi commented 2026-10-29 21:23:20 +00:00

Author

Owner

Copy link

@ascend-direct-dev commented on GitHub (Oct 10, 2026):

更新0.3.12即可

<!-- gh-comment-id:6095513444 --> @ascend-direct-dev commented on GitHub (Oct 10, 2026): 更新0.3.12即可

bsakaguchi commented 2026-10-29 21:23:20 +00:00

Author

Owner

Copy link

@Adeshen commented on GitHub (Oct 14, 2026):

Additional evidence: vLLM Ascend v0.23.0 P/D API processes abort after SIGTERM

We observed a closely matching failure with mooncake-transfer-engine-npu==0.3.11.post1 in the vLLM Ascend v0.23.0 image, including a real model-serving shutdown failure. This appears related to this issue; we have not established that the underlying corrupting write is the same.

I saw the recommendation to upgrade to 0.3.12. We have not yet retested that version. These are observations from the original pinned image, not a claim that 0.3.12 remains affected.

Environment

• Image tag: quay.io/ascend/vllm-ascend:v0.23.0.
• Local Docker image ID: sha256:660ce23a83574fdcd28ba126b07615829c06770c06fbdf6533f03cc4d90acdb1 (not a registry manifest digest).
• vLLM commit: 0fc695fc6d1d82e9a5ac6835ac8e4e1c83703665; Ascend commit: 5cb98caaadeff42b5b62b996e34bb2aaa29d20fd, from the same-image environment record.
• Python 3.12.13. Native-library inspection resolves CANN libraries under /usr/local/Ascend/cann-9.1.0/lib64/.
• Real P/D service: Qwen3-0.6B BF16, two Ascend 910B3 devices (physical 6/7), one TP=1 API/EngineCore per device; eager, synchronous scheduling, prefix caching disabled, 64 KV blocks, max model length 512. Isolated --network none container, loopback communication.
• Import/native isolation: no NPU device passthrough, read-only host driver mount, TORCH_DEVICE_BACKEND_AUTOLOAD=0. This is a different diagnostic environment from the real-device P/D run.

Recorded installed package versions

{
  "torchaudio": "2.10.0+cpu",
  "vllm_ascend": "0.23.0",
  "torch_npu": "2.10.0.post4",
  "torchvision": "0.25.0+cpu",
  "torch": "2.10.0+cpu",
  "vllm": "0.23.0+empty",
  "mooncake-transfer-engine-npu": "0.3.11.post1"
}

Observed behavior

Run	Result before shutdown	Observed process exit
Unmodified P/D service with MooncakeConnectorV1	Three prompts: P/D output text and token strings match D-local baseline; three stream disconnect/recovery cycles complete	Both API processes: -6 (SIGABRT), corrupted size vs. prev_size
Same-image/model, two APIs without a connector	Three successful 16-token requests per API, health checks succeed	Both API processes: 0
Native isolation: load libascendcl.so or libllm_datadist.so individually	Prints BODY_OK	0
Native isolation: load libtransfer_engine.so, engine.so, or import mooncake.engine	Prints BODY_OK	-6, same heap diagnostic

The P/D driver sends SIGTERM only to each API parent process, then waits up to 45 seconds; it does not signal the entire process group in this run. Both returned -6, not the -9 code of the driver's timeout fallback. The runtime logs show the default mode=abort timeout=0s; we did not test an explicitly configured graceful-drain mode.

This primary P/D reproduction has no sitecustomize instrumentation or patched production methods. Its driver exits 1 because it asserts that all API child exit codes must be 0. A separate earlier group-SIGTERM run is not used as the main evidence here.

The no-connector control does not perform remote KV transfer or streaming cancellation; it is not a perfectly workload-matched ablation. We cannot attribute the corruption to the connector tracker, cancellation handling, or a particular CANN finalizer. No native backtrace identifying a corrupting write was captured in our runs.

Service evidence

The full source reproducer below starts P and D, compares real P/D generation against local generation, disconnects three streams, verifies recovery, and sends parent-only SIGTERM. The shutdown portions of the resulting API logs are shown first.

kv_consumer.log: shutdown excerpt (excerpt from original full log)

(APIServer pid=536) INFO 10-08 11:56:06 [launcher.py:100] [shutdown] API server: shutdown triggered
(APIServer pid=536) INFO 10-08 11:56:06 [launcher.py:116] [shutdown] API server: stopping engine client mode=abort timeout=0s
(APIServer pid=536) INFO 10-08 11:56:06 [core_client.py:652] [shutdown] MPClient: start timeout=0s
(APIServer pid=536) INFO 10-08 11:56:06 [core_client.py:654] [shutdown] MPClient: stopping engine manager
(APIServer pid=536) WARNING 10-08 11:56:06 [utils.py:607] [shutdown] Process manager: force killing remaining processes count=1
(EngineCore pid=575) INFO 10-08 11:56:06 [core.py:1178] [shutdown] EngineCore: trigger received signal=SIGTERM
(EngineCore pid=575) INFO 10-08 11:56:06 [core.py:1297] [shutdown] EngineCore: start mode=abort timeout=0s
(EngineCore pid=575) INFO 10-08 11:56:06 [core.py:1328] [shutdown] EngineCore: request processing complete; starting resource teardown
(EngineCore pid=575) INFO 10-08 11:56:06 [core.py:1191] [shutdown] EngineCore: exiting busy loop
(APIServer pid=536) INFO 10-08 11:56:06 [core_client.py:656] [shutdown] MPClient: engine manager stopped
(APIServer pid=536) INFO 10-08 11:56:06 [core_client.py:657] [shutdown] MPClient: cleaning up background resources
(APIServer pid=536) INFO 10-08 11:56:06 [core_client.py:659] [shutdown] MPClient: complete
(APIServer pid=536) ERROR 10-08 11:56:06 [async_llm.py:704] AsyncLLM output_handler failed.
(APIServer pid=536) ERROR 10-08 11:56:06 [async_llm.py:704] Traceback (most recent call last):
(APIServer pid=536) ERROR 10-08 11:56:06 [async_llm.py:704]   File "/vllm-workspace/vllm/vllm/v1/engine/async_llm.py", line 660, in output_handler
(APIServer pid=536) ERROR 10-08 11:56:06 [async_llm.py:704]     outputs = await engine_core.get_output_async()
(APIServer pid=536) ERROR 10-08 11:56:06 [async_llm.py:704]               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=536) ERROR 10-08 11:56:06 [async_llm.py:704]   File "/vllm-workspace/vllm/vllm/v1/engine/core_client.py", line 1046, in get_output_async
(APIServer pid=536) ERROR 10-08 11:56:06 [async_llm.py:704]     raise self._format_exception(outputs) from None
(APIServer pid=536) ERROR 10-08 11:56:06 [async_llm.py:704] vllm.v1.engine.exceptions.EngineDeadError: EngineCore encountered an issue. See stack trace (above) for the root cause.
(APIServer pid=536) INFO 10-08 11:56:06 [launcher.py:125] [shutdown] API server: engine client stopped
(APIServer pid=536) INFO 10-08 11:56:06 [launcher.py:128] [shutdown] API server: signalling HTTP server shutdown
(APIServer pid=536) INFO 10-08 11:56:06 [launcher.py:149] [shutdown] API server: shutting down FastAPI HTTP server
(APIServer pid=536) INFO:     Shutting down
(APIServer pid=536) INFO:     Waiting for application shutdown.
(APIServer pid=536) INFO:     Application shutdown complete.
/usr/local/python3.12.13/lib/python3.12/multiprocessing/resource_tracker.py:279: UserWarning: resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '
corrupted size vs. prev_size

kv_producer.log: shutdown excerpt (excerpt from original full log)

(APIServer pid=10) INFO 10-08 11:56:08 [launcher.py:100] [shutdown] API server: shutdown triggered
(APIServer pid=10) INFO 10-08 11:56:08 [launcher.py:116] [shutdown] API server: stopping engine client mode=abort timeout=0s
(APIServer pid=10) INFO 10-08 11:56:08 [core_client.py:652] [shutdown] MPClient: start timeout=0s
(APIServer pid=10) INFO 10-08 11:56:08 [core_client.py:654] [shutdown] MPClient: stopping engine manager
(APIServer pid=10) WARNING 10-08 11:56:08 [utils.py:607] [shutdown] Process manager: force killing remaining processes count=1
(EngineCore pid=72) INFO 10-08 11:56:08 [core.py:1178] [shutdown] EngineCore: trigger received signal=SIGTERM
(EngineCore pid=72) INFO 10-08 11:56:08 [core.py:1297] [shutdown] EngineCore: start mode=abort timeout=0s
(EngineCore pid=72) INFO 10-08 11:56:08 [core.py:1328] [shutdown] EngineCore: request processing complete; starting resource teardown
(EngineCore pid=72) INFO 10-08 11:56:08 [core.py:1191] [shutdown] EngineCore: exiting busy loop
(APIServer pid=10) INFO 10-08 11:56:08 [core_client.py:656] [shutdown] MPClient: engine manager stopped
(APIServer pid=10) INFO 10-08 11:56:08 [core_client.py:657] [shutdown] MPClient: cleaning up background resources
(APIServer pid=10) INFO 10-08 11:56:08 [core_client.py:659] [shutdown] MPClient: complete
(APIServer pid=10) ERROR 10-08 11:56:08 [async_llm.py:704] AsyncLLM output_handler failed.
(APIServer pid=10) ERROR 10-08 11:56:08 [async_llm.py:704] Traceback (most recent call last):
(APIServer pid=10) ERROR 10-08 11:56:08 [async_llm.py:704]   File "/vllm-workspace/vllm/vllm/v1/engine/async_llm.py", line 660, in output_handler
(APIServer pid=10) ERROR 10-08 11:56:08 [async_llm.py:704]     outputs = await engine_core.get_output_async()
(APIServer pid=10) ERROR 10-08 11:56:08 [async_llm.py:704]               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=10) ERROR 10-08 11:56:08 [async_llm.py:704]   File "/vllm-workspace/vllm/vllm/v1/engine/core_client.py", line 1046, in get_output_async
(APIServer pid=10) ERROR 10-08 11:56:08 [async_llm.py:704]     raise self._format_exception(outputs) from None
(APIServer pid=10) ERROR 10-08 11:56:08 [async_llm.py:704] vllm.v1.engine.exceptions.EngineDeadError: EngineCore encountered an issue. See stack trace (above) for the root cause.
(APIServer pid=10) INFO 10-08 11:56:08 [launcher.py:125] [shutdown] API server: engine client stopped
(APIServer pid=10) INFO 10-08 11:56:08 [launcher.py:128] [shutdown] API server: signalling HTTP server shutdown
(APIServer pid=10) INFO 10-08 11:56:08 [launcher.py:149] [shutdown] API server: shutting down FastAPI HTTP server
(APIServer pid=10) INFO:     Shutting down
(APIServer pid=10) INFO:     Waiting for application shutdown.
(APIServer pid=10) INFO:     Application shutdown complete.
/usr/local/python3.12.13/lib/python3.12/multiprocessing/resource_tracker.py:279: UserWarning: resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '
corrupted size vs. prev_size

Recorded process exit checks

P/D:
SERVER_EXIT kv_consumer.log -6
SERVER_EXIT kv_producer.log -6
Driver exit code: 1

No-connector control:
SERVER_EXIT control_b.log 0
SERVER_EXIT control_a.log 0
EXIT_RESULT_JSON {"server_exits": [0, 0], "connector": false, "signal": "API parent SIGTERM"}
Driver exit code: 0

pd-parent-sigterm/test_service.py — original script

import os, sys, subprocess, time, json, urllib.request, traceback
from pathlib import Path
procs=[]
def request(port,path,payload=None):
    data=None if payload is None else json.dumps(payload).encode()
    req=urllib.request.Request(f'http://127.0.0.1:{port}{path}',data=data,headers={'Content-Type':'application/json'})
    with urllib.request.urlopen(req,timeout=90) as r: return r.read().decode()
def launch(role,dev,port,kvport):
    env=os.environ.copy();env.update(ASCEND_RT_VISIBLE_DEVICES=str(dev),VLLM_HOST_IP='127.0.0.1',HCCL_IF_IP='127.0.0.1',GLOO_SOCKET_IFNAME='lo',TP_SOCKET_IFNAME='lo',HCCL_SOCKET_IFNAME='lo',VLLM_WORKER_MULTIPROC_METHOD='spawn')
    cfg={'kv_connector':'MooncakeConnectorV1','kv_role':role,'kv_port':kvport,'kv_connector_extra_config':{'prefill':{'dp_size':1,'tp_size':1},'decode':{'dp_size':1,'tp_size':1}}}
    cmd=[sys.executable,'-m','vllm.entrypoints.openai.api_server','--model','/models/qwen3-0.6b','--served-model-name','qwen','--host','127.0.0.1','--port',str(port),'--dtype','bfloat16','--enforce-eager','--no-async-scheduling','--no-enable-prefix-caching','--max-model-len','512','--max-num-batched-tokens','512','--max-num-seqs','4','--num-gpu-blocks-override','64','--gpu-memory-utilization','0.15','--additional-config','{"weight_nz_mode":0}','--kv-transfer-config',json.dumps(cfg)]
    print('COMMAND',json.dumps(cmd),flush=True)
    log=Path(f'/tmp/{role}.log');f=log.open('w');p=subprocess.Popen(cmd,env=env,stdout=f,stderr=subprocess.STDOUT,start_new_session=True);procs.append((p,f,log))
    deadline=time.monotonic()+300
    while time.monotonic()<deadline:
        if p.poll() is not None: raise RuntimeError(f'{role} exited {p.returncode}')
        try:
            request(port,'/health');print('READY',role,flush=True);return
        except Exception: time.sleep(2)
    raise TimeoutError(role+' readiness')
def main():
    launch('kv_producer',0,18080,30000);launch('kv_consumer',1,18081,30100)
    results=[]
    for i,prompt in enumerate(['The capital of France is','Write a short explanation of gravity.','A sequence of numbers: 1, 2, 3,']):
        payload={'model':'qwen','prompt':prompt,'temperature':0,'max_tokens':16,'ignore_eos':True,'logprobs':5}
        baseline=json.loads(request(18081,'/v1/completions',payload))
        pref=dict(payload,max_tokens=1,min_tokens=1,kv_transfer_params={'do_remote_decode':True,'do_remote_prefill':False,'remote_engine_id':None,'remote_block_ids':None,'remote_host':None,'remote_port':None})
        p=json.loads(request(18080,'/v1/completions',pref));params=p.get('kv_transfer_params');assert params, p
        print('PREFILL',json.dumps(p),flush=True)
        d=json.loads(request(18081,'/v1/completions',dict(payload,kv_transfer_params=params)))
        a,b=baseline['choices'][0],d['choices'][0]
        entry={'case':i,'baseline':baseline,'prefill':p,'decode':d,'text_equal':a['text']==b['text'],'tokens_equal':a['logprobs']['tokens']==b['logprobs']['tokens']}
        print('RESULT',json.dumps(entry),flush=True);results.append(entry)
        assert entry['text_equal'] and entry['tokens_equal']
    cancellations=[]
    import httpx, re
    for cycle in range(3):
        prompt='Explain the behavior of a rotating planet in detail. '+('Gravity controls planetary motion. '*24)
        payload={'model':'qwen','prompt':prompt,'temperature':0,'max_tokens':200,'ignore_eos':True}
        pref=dict(payload,max_tokens=1,min_tokens=1,kv_transfer_params={'do_remote_decode':True,'do_remote_prefill':False,'remote_engine_id':None,'remote_block_ids':None,'remote_host':None,'remote_port':None})
        p=json.loads(request(18080,'/v1/completions',pref));params=p['kv_transfer_params']
        assert params['remote_block_ids'] and params['do_remote_prefill']
        payload.update(stream=True,kv_transfer_params=params)
        first=None
        with httpx.Client(timeout=90) as client:
            with client.stream('POST','http://127.0.0.1:18081/v1/completions',json=payload) as stream:
                
                if stream.status_code!=200: print('HTTP_ERROR',stream.read().decode(),flush=True)
                stream.raise_for_status()
                for line in stream.iter_lines():
                    if not line.startswith('data: '):continue
                    assert line!='data: [DONE]', 'Finished before cancellation'
                    item=json.loads(line[6:])
                    if item['choices'] and item['choices'][0].get('text'):
                        assert item['choices'][0]['finish_reason'] is None
                        first=item;break
        assert first is not None
        print('CANCEL_CLOSED',cycle,json.dumps(first),flush=True)
        deadline=time.monotonic()+30
        while True:
            metrics={port:request(port,'/metrics') for port in (18080,18081)}
            vals=[]
            for port,m in metrics.items():
                for name in ('kv_cache_usage_perc','num_requests_running','num_requests_waiting'):
                    hits=re.findall(r'^vllm:'+name+r'\{[^\n]+\} ([0-9.e+-]+)$',m,re.M)
                    assert hits,(port,name)
                    vals.extend(float(x) for x in hits)
            if all(x==0 for x in vals):break
            if time.monotonic()>deadline:raise AssertionError(('not drained',vals))
            time.sleep(1)
        # Fresh request after cancellation checks that the service can continue.
        recovery={'model':'qwen','prompt':'The capital of France is','temperature':0,'max_tokens':16,'ignore_eos':True,'logprobs':5}
        actual=json.loads(request(18081,'/v1/completions',recovery))
        assert actual['choices'][0]['text']==results[0]['baseline']['choices'][0]['text']
        entry={'cycle':cycle,'stream_id':first['id'],'prefill':p,'first_chunk':first,'drained_metrics':metrics,'recovery':actual}
        cancellations.append(entry);print('CANCEL_RESULT',json.dumps(entry),flush=True)
    print('M04_CANCEL_RESULT_JSON='+json.dumps(cancellations),flush=True)
    for port in (18080,18081):
        print('HEALTH',port,request(port,'/health'),flush=True)
        print('METRICS',port,request(port,'/metrics'),flush=True)
    print('M04_PD_RESULT_JSON='+json.dumps({'cases':results,'passed':True}),flush=True)
if __name__=='__main__':
    try: main()
    finally:
        import signal
        for p,f,log in reversed(procs):
            if p.poll() is None:
                p.send_signal(signal.SIGTERM)
                try:p.wait(timeout=45)
                except subprocess.TimeoutExpired:os.killpg(p.pid,signal.SIGKILL);p.wait()
            f.close();print('SERVER_EXIT',log.name,p.returncode,flush=True);print('SERVER_LOG_BEGIN',log.name,flush=True);print(log.read_text(),flush=True);print('SERVER_LOG_END',log.name,flush=True)

    assert all(p.returncode==0 for p,_,_ in procs), [(log.name,p.returncode) for p,_,log in procs]

pd-parent-sigterm/run-public.sh — public adaptation of recorded invocation

#!/bin/sh
# Public adaptation: SSH wrapper/container name removed; host model path replaced.
# Run from this folder on the Ascend host; select free physical devices if needed.
# This adapted wrapper has not been rerun. The image identifier is a local image ID.
docker run --rm -i --network none --shm-size=2g --device /dev/davinci6 --device /dev/davinci7 --device /dev/davinci_manager --device /dev/devmm_svm --device /dev/hisi_hdc -e ASCEND_RT_VISIBLE_DEVICES=0,1 -e PYTHONUNBUFFERED=1 -e VLLM_WORKER_MULTIPROC_METHOD=spawn -e VLLM_ASCEND_ENABLE_NZ=0 -e GRAPH_MODE=0 -v /usr/local/Ascend/driver:/usr/local/Ascend/driver:ro -v /usr/local/sbin/npu-smi:/usr/local/sbin/npu-smi:ro -v /etc/ascend_install.info:/etc/ascend_install.info:ro -v /path/to/Qwen3-0.6B:/models/qwen3-0.6b:ro --entrypoint sh -w /vllm-workspace/vllm-ascend sha256:660ce23a83574fdcd28ba126b07615829c06770c06fbdf6533f03cc4d90acdb1 -c 'cat > /tmp/test_service.py; timeout 850 python /tmp/test_service.py' < test_service.py

no-connector-control/test_service.py — original script

import os, sys, subprocess, time, json, urllib.request, traceback
from pathlib import Path
procs=[]
def request(port,path,payload=None):
    data=None if payload is None else json.dumps(payload).encode()
    req=urllib.request.Request(f'http://127.0.0.1:{port}{path}',data=data,headers={'Content-Type':'application/json'})
    with urllib.request.urlopen(req,timeout=90) as r: return r.read().decode()
def launch(role,dev,port,kvport):
    env=os.environ.copy();env.update(ASCEND_RT_VISIBLE_DEVICES=str(dev),VLLM_HOST_IP='127.0.0.1',HCCL_IF_IP='127.0.0.1',GLOO_SOCKET_IFNAME='lo',TP_SOCKET_IFNAME='lo',HCCL_SOCKET_IFNAME='lo',VLLM_WORKER_MULTIPROC_METHOD='spawn')
    cfg={'kv_connector':'MooncakeConnectorV1','kv_role':role,'kv_port':kvport,'kv_connector_extra_config':{'prefill':{'dp_size':1,'tp_size':1},'decode':{'dp_size':1,'tp_size':1}}}
    cmd=[sys.executable,'-m','vllm.entrypoints.openai.api_server','--model','/models/qwen3-0.6b','--served-model-name','qwen','--host','127.0.0.1','--port',str(port),'--dtype','bfloat16','--enforce-eager','--no-async-scheduling','--no-enable-prefix-caching','--max-model-len','512','--max-num-batched-tokens','512','--max-num-seqs','4','--num-gpu-blocks-override','64','--gpu-memory-utilization','0.15','--additional-config','{"weight_nz_mode":0}']
    print('COMMAND',json.dumps(cmd),flush=True)
    log=Path(f'/tmp/{role}.log');f=log.open('w');p=subprocess.Popen(cmd,env=env,stdout=f,stderr=subprocess.STDOUT,start_new_session=True);procs.append((p,f,log))
    deadline=time.monotonic()+300
    while time.monotonic()<deadline:
        if p.poll() is not None: raise RuntimeError(f'{role} exited {p.returncode}')
        try:
            request(port,'/health');print('READY',role,flush=True);return
        except Exception: time.sleep(2)
    raise TimeoutError(role+' readiness')
def main():
    launch('control_a',0,18080,30000);launch('control_b',1,18081,30100)
    for port in (18080,18081):
        for prompt in ['The capital of France is','Write a short explanation of gravity.','A sequence of numbers: 1, 2, 3,']:
            payload={'model':'qwen','prompt':prompt,'temperature':0,'max_tokens':16,'ignore_eos':True,'logprobs':5}
            response=json.loads(request(port,'/v1/completions',payload))
            assert response['usage']['completion_tokens']==16
            print('CONTROL_RESPONSE',json.dumps({'port':port,'response':response}),flush=True)
        request(port,'/health')
    print('REQUEST_PHASE_PASSED',flush=True)
if __name__=='__main__':
    try: main()
    finally:
        exits=[]
        import signal
        for p,f,log in reversed(procs):
            if p.poll() is None:
                p.terminate()
                try:p.wait(timeout=60)
                except subprocess.TimeoutExpired:os.killpg(p.pid,signal.SIGKILL);p.wait()
            f.close();print('SERVER_EXIT',log.name,p.returncode,flush=True);print('SERVER_LOG_BEGIN',log.name,flush=True);print(log.read_text(),flush=True);print('SERVER_LOG_END',log.name,flush=True)

        exits=[p.returncode for p,f,log in procs]
        print('EXIT_RESULT_JSON',json.dumps({'server_exits':exits,'connector':False,'signal':'API parent SIGTERM'}),flush=True)
        assert len(exits)==2 and all(code==0 for code in exits),'API server shutdown failed'

no-connector-control/run-public.sh — public adaptation of recorded invocation

#!/bin/sh
# Public adaptation: SSH wrapper/container name removed; host model path replaced.
# Run from this folder on the Ascend host; select free physical devices if needed.
# This adapted wrapper has not been rerun. The image identifier is a local image ID.
docker run --rm -i --network none --shm-size=2g --device /dev/davinci6 --device /dev/davinci7 --device /dev/davinci_manager --device /dev/devmm_svm --device /dev/hisi_hdc -e ASCEND_RT_VISIBLE_DEVICES=0,1 -e PYTHONUNBUFFERED=1 -e VLLM_WORKER_MULTIPROC_METHOD=spawn -e VLLM_ASCEND_ENABLE_NZ=0 -e GRAPH_MODE=0 -v /usr/local/Ascend/driver:/usr/local/Ascend/driver:ro -v /usr/local/sbin/npu-smi:/usr/local/sbin/npu-smi:ro -v /etc/ascend_install.info:/etc/ascend_install.info:ro -v /path/to/Qwen3-0.6B:/models/qwen3-0.6b:ro --entrypoint sh -w /vllm-workspace/vllm-ascend sha256:660ce23a83574fdcd28ba126b07615829c06770c06fbdf6533f03cc4d90acdb1 -c 'cat > /tmp/test_service.py; timeout 850 python /tmp/test_service.py' < test_service.py

Native isolation

The diagnostic below launches separate subprocesses and checks their actual return codes. Its wrapper itself exits 0 after collecting the failures; that does not mean the child tests passed. The direct ascend_transport.so load has an unresolved-symbol error because it lacks its host library symbols; we do not treat that case as a production defect.

native-isolation/diagnose.py — original script

import subprocess,sys,json,os,importlib.util,pathlib
p=pathlib.Path(importlib.util.find_spec('mooncake').origin).parent
cases={
 'acl':"import ctypes; x=ctypes.CDLL('libascendcl.so')",
 'datadist':"import ctypes; x=ctypes.CDLL('libllm_datadist.so')",
 'transport':f"import ctypes; x=ctypes.CDLL({str(p/'ascend_transport.so')!r})",
 'engine_library':f"import ctypes; x=ctypes.CDLL({str(p/'libtransfer_engine.so')!r})",
 'engine_extension':f"import ctypes; x=ctypes.CDLL({str(p/'engine.so')!r})",
 'import':"import mooncake.engine; print([(n,getattr(mooncake.engine.TransferEngine,n).__doc__) for n in dir(mooncake.engine.TransferEngine) if not n.startswith('_')],flush=True)",
}
results=[]
for name,code in cases.items():
 r=subprocess.run([sys.executable,'-X','faulthandler','-c',code+";print('BODY_OK',flush=True)"],capture_output=True,text=True,timeout=45)
 row=dict(name=name,code=code,returncode=r.returncode,stdout=r.stdout,stderr=r.stderr)
 results.append(row); print('CASE_JSON '+json.dumps(row),flush=True)
env=os.environ.copy();env['LD_DEBUG']='libs'
r=subprocess.run([sys.executable,'-c',"import mooncake.engine;print('BODY_OK',flush=True)"],env=env,capture_output=True,text=True,timeout=45)
print('LOADER_JSON '+json.dumps(dict(returncode=r.returncode,stdout=r.stdout,stderr=r.stderr)),flush=True)
print('RESULT_JSON '+json.dumps(results),flush=True)

native-isolation/run-public.sh

#!/bin/sh
# Public adaptation: SSH wrapper/container name removed; host model path replaced.
# Run from this folder on the Ascend host; select free physical devices if needed.
# This adapted wrapper has not been rerun. The image identifier is a local image ID.
docker run --rm -i --network none -e TORCH_DEVICE_BACKEND_AUTOLOAD=0 -v /usr/local/Ascend/driver:/usr/local/Ascend/driver:ro --entrypoint python sha256:660ce23a83574fdcd28ba126b07615829c06770c06fbdf6533f03cc4d90acdb1 - < diagnose.py

Native diagnostic result summary (derived from original results.json)

[
  {
    "name": "acl",
    "returncode": 0,
    "stderr": "",
    "body_ok": true
  },
  {
    "name": "datadist",
    "returncode": 0,
    "stderr": "",
    "body_ok": true
  },
  {
    "name": "transport",
    "returncode": 1,
    "stderr": "Traceback (most recent call last):\n  File \"<string>\", line 1, in <module>\n  File \"/usr/local/python3.12.13/lib/python3.12/ctypes/__init__.py\", line 379, in __init__\n    self._handle = _dlopen(self._name, mode)\n                   ^^^^^^^^^^^^^^^^^^^^^^^^^\nOSError: /usr/local/python3.12.13/lib/python3.12/site-packages/mooncake/ascend_transport.so: undefined symbol: _ZTIN8mooncake9TransportE\n",
    "body_ok": false
  },
  {
    "name": "engine_library",
    "returncode": -6,
    "stderr": "corrupted size vs. prev_size\n",
    "body_ok": true
  },
  {
    "name": "engine_extension",
    "returncode": -6,
    "stderr": "corrupted size vs. prev_size\n",
    "body_ok": true
  },
  {
    "name": "import",
    "returncode": -6,
    "stderr": "corrupted size vs. prev_size\n",
    "body_ok": true
  }
]

The two API crashes show a deployment lifecycle impact beyond standalone imports: the recorded P/D work completes, but service shutdown is reported as a process failure. We are adding this as corroborating evidence for the existing dependency issue, not a new independently established root cause. The next check is an isolated 0.3.12 retest of both import exit and this P/D shutdown workload while keeping other conditions fixed.

<!-- gh-comment-id:6163720637 --> @Adeshen commented on GitHub (Oct 14, 2026): ## Additional evidence: vLLM Ascend v0.23.0 P/D API processes abort after SIGTERM We observed a closely matching failure with `mooncake-transfer-engine-npu==0.3.11.post1` in the vLLM Ascend v0.23.0 image, including a real model-serving shutdown failure. This appears related to this issue; we have not established that the underlying corrupting write is the same. I saw the recommendation to upgrade to 0.3.12. **We have not yet retested that version.** These are observations from the original pinned image, not a claim that 0.3.12 remains affected. ### Environment - Image tag: `quay.io/ascend/vllm-ascend:v0.23.0`. - Local Docker image ID: `sha256:660ce23a83574fdcd28ba126b07615829c06770c06fbdf6533f03cc4d90acdb1` (not a registry manifest digest). - vLLM commit: `0fc695fc6d1d82e9a5ac6835ac8e4e1c83703665`; Ascend commit: `5cb98caaadeff42b5b62b996e34bb2aaa29d20fd`, from the same-image environment record. - Python 3.12.13. Native-library inspection resolves CANN libraries under `/usr/local/Ascend/cann-9.1.0/lib64/`. - Real P/D service: Qwen3-0.6B BF16, two Ascend 910B3 devices (physical 6/7), one TP=1 API/EngineCore per device; eager, synchronous scheduling, prefix caching disabled, 64 KV blocks, max model length 512. Isolated `--network none` container, loopback communication. - Import/native isolation: no NPU device passthrough, read-only host driver mount, `TORCH_DEVICE_BACKEND_AUTOLOAD=0`. This is a different diagnostic environment from the real-device P/D run. <details> <summary>Recorded installed package versions</summary> ```json { "torchaudio": "2.10.0+cpu", "vllm_ascend": "0.23.0", "torch_npu": "2.10.0.post4", "torchvision": "0.25.0+cpu", "torch": "2.10.0+cpu", "vllm": "0.23.0+empty", "mooncake-transfer-engine-npu": "0.3.11.post1" } ``` </details> ### Observed behavior | Run | Result before shutdown | Observed process exit | | --- | --- | --- | | Unmodified P/D service with MooncakeConnectorV1 | Three prompts: P/D output text and token strings match D-local baseline; three stream disconnect/recovery cycles complete | Both API processes: `-6` (SIGABRT), `corrupted size vs. prev_size` | | Same-image/model, two APIs without a connector | Three successful 16-token requests per API, health checks succeed | Both API processes: `0` | | Native isolation: load `libascendcl.so` or `libllm_datadist.so` individually | Prints `BODY_OK` | `0` | | Native isolation: load `libtransfer_engine.so`, `engine.so`, or import `mooncake.engine` | Prints `BODY_OK` | `-6`, same heap diagnostic | The P/D driver sends **SIGTERM only to each API parent process**, then waits up to 45 seconds; it does not signal the entire process group in this run. Both returned `-6`, not the `-9` code of the driver's timeout fallback. The runtime logs show the default `mode=abort timeout=0s`; we did not test an explicitly configured graceful-drain mode. This primary P/D reproduction has no `sitecustomize` instrumentation or patched production methods. Its driver exits 1 because it asserts that all API child exit codes must be 0. A separate earlier group-SIGTERM run is not used as the main evidence here. The no-connector control does not perform remote KV transfer or streaming cancellation; it is not a perfectly workload-matched ablation. We cannot attribute the corruption to the connector tracker, cancellation handling, or a particular CANN finalizer. No native backtrace identifying a corrupting write was captured in our runs. ### Service evidence The full source reproducer below starts P and D, compares real P/D generation against local generation, disconnects three streams, verifies recovery, and sends parent-only SIGTERM. The shutdown portions of the resulting API logs are shown first. <details> <summary>kv_consumer.log: shutdown excerpt (excerpt from original full log)</summary> ```text (APIServer pid=536) INFO 10-08 11:56:06 [launcher.py:100] [shutdown] API server: shutdown triggered (APIServer pid=536) INFO 10-08 11:56:06 [launcher.py:116] [shutdown] API server: stopping engine client mode=abort timeout=0s (APIServer pid=536) INFO 10-08 11:56:06 [core_client.py:652] [shutdown] MPClient: start timeout=0s (APIServer pid=536) INFO 10-08 11:56:06 [core_client.py:654] [shutdown] MPClient: stopping engine manager (APIServer pid=536) WARNING 10-08 11:56:06 [utils.py:607] [shutdown] Process manager: force killing remaining processes count=1 (EngineCore pid=575) INFO 10-08 11:56:06 [core.py:1178] [shutdown] EngineCore: trigger received signal=SIGTERM (EngineCore pid=575) INFO 10-08 11:56:06 [core.py:1297] [shutdown] EngineCore: start mode=abort timeout=0s (EngineCore pid=575) INFO 10-08 11:56:06 [core.py:1328] [shutdown] EngineCore: request processing complete; starting resource teardown (EngineCore pid=575) INFO 10-08 11:56:06 [core.py:1191] [shutdown] EngineCore: exiting busy loop (APIServer pid=536) INFO 10-08 11:56:06 [core_client.py:656] [shutdown] MPClient: engine manager stopped (APIServer pid=536) INFO 10-08 11:56:06 [core_client.py:657] [shutdown] MPClient: cleaning up background resources (APIServer pid=536) INFO 10-08 11:56:06 [core_client.py:659] [shutdown] MPClient: complete (APIServer pid=536) ERROR 10-08 11:56:06 [async_llm.py:704] AsyncLLM output_handler failed. (APIServer pid=536) ERROR 10-08 11:56:06 [async_llm.py:704] Traceback (most recent call last): (APIServer pid=536) ERROR 10-08 11:56:06 [async_llm.py:704] File "/vllm-workspace/vllm/vllm/v1/engine/async_llm.py", line 660, in output_handler (APIServer pid=536) ERROR 10-08 11:56:06 [async_llm.py:704] outputs = await engine_core.get_output_async() (APIServer pid=536) ERROR 10-08 11:56:06 [async_llm.py:704] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (APIServer pid=536) ERROR 10-08 11:56:06 [async_llm.py:704] File "/vllm-workspace/vllm/vllm/v1/engine/core_client.py", line 1046, in get_output_async (APIServer pid=536) ERROR 10-08 11:56:06 [async_llm.py:704] raise self._format_exception(outputs) from None (APIServer pid=536) ERROR 10-08 11:56:06 [async_llm.py:704] vllm.v1.engine.exceptions.EngineDeadError: EngineCore encountered an issue. See stack trace (above) for the root cause. (APIServer pid=536) INFO 10-08 11:56:06 [launcher.py:125] [shutdown] API server: engine client stopped (APIServer pid=536) INFO 10-08 11:56:06 [launcher.py:128] [shutdown] API server: signalling HTTP server shutdown (APIServer pid=536) INFO 10-08 11:56:06 [launcher.py:149] [shutdown] API server: shutting down FastAPI HTTP server (APIServer pid=536) INFO: Shutting down (APIServer pid=536) INFO: Waiting for application shutdown. (APIServer pid=536) INFO: Application shutdown complete. /usr/local/python3.12.13/lib/python3.12/multiprocessing/resource_tracker.py:279: UserWarning: resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown warnings.warn('resource_tracker: There appear to be %d ' corrupted size vs. prev_size ``` </details> <details> <summary>kv_producer.log: shutdown excerpt (excerpt from original full log)</summary> ```text (APIServer pid=10) INFO 10-08 11:56:08 [launcher.py:100] [shutdown] API server: shutdown triggered (APIServer pid=10) INFO 10-08 11:56:08 [launcher.py:116] [shutdown] API server: stopping engine client mode=abort timeout=0s (APIServer pid=10) INFO 10-08 11:56:08 [core_client.py:652] [shutdown] MPClient: start timeout=0s (APIServer pid=10) INFO 10-08 11:56:08 [core_client.py:654] [shutdown] MPClient: stopping engine manager (APIServer pid=10) WARNING 10-08 11:56:08 [utils.py:607] [shutdown] Process manager: force killing remaining processes count=1 (EngineCore pid=72) INFO 10-08 11:56:08 [core.py:1178] [shutdown] EngineCore: trigger received signal=SIGTERM (EngineCore pid=72) INFO 10-08 11:56:08 [core.py:1297] [shutdown] EngineCore: start mode=abort timeout=0s (EngineCore pid=72) INFO 10-08 11:56:08 [core.py:1328] [shutdown] EngineCore: request processing complete; starting resource teardown (EngineCore pid=72) INFO 10-08 11:56:08 [core.py:1191] [shutdown] EngineCore: exiting busy loop (APIServer pid=10) INFO 10-08 11:56:08 [core_client.py:656] [shutdown] MPClient: engine manager stopped (APIServer pid=10) INFO 10-08 11:56:08 [core_client.py:657] [shutdown] MPClient: cleaning up background resources (APIServer pid=10) INFO 10-08 11:56:08 [core_client.py:659] [shutdown] MPClient: complete (APIServer pid=10) ERROR 10-08 11:56:08 [async_llm.py:704] AsyncLLM output_handler failed. (APIServer pid=10) ERROR 10-08 11:56:08 [async_llm.py:704] Traceback (most recent call last): (APIServer pid=10) ERROR 10-08 11:56:08 [async_llm.py:704] File "/vllm-workspace/vllm/vllm/v1/engine/async_llm.py", line 660, in output_handler (APIServer pid=10) ERROR 10-08 11:56:08 [async_llm.py:704] outputs = await engine_core.get_output_async() (APIServer pid=10) ERROR 10-08 11:56:08 [async_llm.py:704] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (APIServer pid=10) ERROR 10-08 11:56:08 [async_llm.py:704] File "/vllm-workspace/vllm/vllm/v1/engine/core_client.py", line 1046, in get_output_async (APIServer pid=10) ERROR 10-08 11:56:08 [async_llm.py:704] raise self._format_exception(outputs) from None (APIServer pid=10) ERROR 10-08 11:56:08 [async_llm.py:704] vllm.v1.engine.exceptions.EngineDeadError: EngineCore encountered an issue. See stack trace (above) for the root cause. (APIServer pid=10) INFO 10-08 11:56:08 [launcher.py:125] [shutdown] API server: engine client stopped (APIServer pid=10) INFO 10-08 11:56:08 [launcher.py:128] [shutdown] API server: signalling HTTP server shutdown (APIServer pid=10) INFO 10-08 11:56:08 [launcher.py:149] [shutdown] API server: shutting down FastAPI HTTP server (APIServer pid=10) INFO: Shutting down (APIServer pid=10) INFO: Waiting for application shutdown. (APIServer pid=10) INFO: Application shutdown complete. /usr/local/python3.12.13/lib/python3.12/multiprocessing/resource_tracker.py:279: UserWarning: resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown warnings.warn('resource_tracker: There appear to be %d ' corrupted size vs. prev_size ``` </details> <details> <summary>Recorded process exit checks</summary> ```text P/D: SERVER_EXIT kv_consumer.log -6 SERVER_EXIT kv_producer.log -6 Driver exit code: 1 No-connector control: SERVER_EXIT control_b.log 0 SERVER_EXIT control_a.log 0 EXIT_RESULT_JSON {"server_exits": [0, 0], "connector": false, "signal": "API parent SIGTERM"} Driver exit code: 0 ``` </details> <details> <summary>pd-parent-sigterm/test_service.py — original script</summary> ```python import os, sys, subprocess, time, json, urllib.request, traceback from pathlib import Path procs=[] def request(port,path,payload=None): data=None if payload is None else json.dumps(payload).encode() req=urllib.request.Request(f'http://127.0.0.1:{port}{path}',data=data,headers={'Content-Type':'application/json'}) with urllib.request.urlopen(req,timeout=90) as r: return r.read().decode() def launch(role,dev,port,kvport): env=os.environ.copy();env.update(ASCEND_RT_VISIBLE_DEVICES=str(dev),VLLM_HOST_IP='127.0.0.1',HCCL_IF_IP='127.0.0.1',GLOO_SOCKET_IFNAME='lo',TP_SOCKET_IFNAME='lo',HCCL_SOCKET_IFNAME='lo',VLLM_WORKER_MULTIPROC_METHOD='spawn') cfg={'kv_connector':'MooncakeConnectorV1','kv_role':role,'kv_port':kvport,'kv_connector_extra_config':{'prefill':{'dp_size':1,'tp_size':1},'decode':{'dp_size':1,'tp_size':1}}} cmd=[sys.executable,'-m','vllm.entrypoints.openai.api_server','--model','/models/qwen3-0.6b','--served-model-name','qwen','--host','127.0.0.1','--port',str(port),'--dtype','bfloat16','--enforce-eager','--no-async-scheduling','--no-enable-prefix-caching','--max-model-len','512','--max-num-batched-tokens','512','--max-num-seqs','4','--num-gpu-blocks-override','64','--gpu-memory-utilization','0.15','--additional-config','{"weight_nz_mode":0}','--kv-transfer-config',json.dumps(cfg)] print('COMMAND',json.dumps(cmd),flush=True) log=Path(f'/tmp/{role}.log');f=log.open('w');p=subprocess.Popen(cmd,env=env,stdout=f,stderr=subprocess.STDOUT,start_new_session=True);procs.append((p,f,log)) deadline=time.monotonic()+300 while time.monotonic()<deadline: if p.poll() is not None: raise RuntimeError(f'{role} exited {p.returncode}') try: request(port,'/health');print('READY',role,flush=True);return except Exception: time.sleep(2) raise TimeoutError(role+' readiness') def main(): launch('kv_producer',0,18080,30000);launch('kv_consumer',1,18081,30100) results=[] for i,prompt in enumerate(['The capital of France is','Write a short explanation of gravity.','A sequence of numbers: 1, 2, 3,']): payload={'model':'qwen','prompt':prompt,'temperature':0,'max_tokens':16,'ignore_eos':True,'logprobs':5} baseline=json.loads(request(18081,'/v1/completions',payload)) pref=dict(payload,max_tokens=1,min_tokens=1,kv_transfer_params={'do_remote_decode':True,'do_remote_prefill':False,'remote_engine_id':None,'remote_block_ids':None,'remote_host':None,'remote_port':None}) p=json.loads(request(18080,'/v1/completions',pref));params=p.get('kv_transfer_params');assert params, p print('PREFILL',json.dumps(p),flush=True) d=json.loads(request(18081,'/v1/completions',dict(payload,kv_transfer_params=params))) a,b=baseline['choices'][0],d['choices'][0] entry={'case':i,'baseline':baseline,'prefill':p,'decode':d,'text_equal':a['text']==b['text'],'tokens_equal':a['logprobs']['tokens']==b['logprobs']['tokens']} print('RESULT',json.dumps(entry),flush=True);results.append(entry) assert entry['text_equal'] and entry['tokens_equal'] cancellations=[] import httpx, re for cycle in range(3): prompt='Explain the behavior of a rotating planet in detail. '+('Gravity controls planetary motion. '*24) payload={'model':'qwen','prompt':prompt,'temperature':0,'max_tokens':200,'ignore_eos':True} pref=dict(payload,max_tokens=1,min_tokens=1,kv_transfer_params={'do_remote_decode':True,'do_remote_prefill':False,'remote_engine_id':None,'remote_block_ids':None,'remote_host':None,'remote_port':None}) p=json.loads(request(18080,'/v1/completions',pref));params=p['kv_transfer_params'] assert params['remote_block_ids'] and params['do_remote_prefill'] payload.update(stream=True,kv_transfer_params=params) first=None with httpx.Client(timeout=90) as client: with client.stream('POST','http://127.0.0.1:18081/v1/completions',json=payload) as stream: if stream.status_code!=200: print('HTTP_ERROR',stream.read().decode(),flush=True) stream.raise_for_status() for line in stream.iter_lines(): if not line.startswith('data: '):continue assert line!='data: [DONE]', 'Finished before cancellation' item=json.loads(line[6:]) if item['choices'] and item['choices'][0].get('text'): assert item['choices'][0]['finish_reason'] is None first=item;break assert first is not None print('CANCEL_CLOSED',cycle,json.dumps(first),flush=True) deadline=time.monotonic()+30 while True: metrics={port:request(port,'/metrics') for port in (18080,18081)} vals=[] for port,m in metrics.items(): for name in ('kv_cache_usage_perc','num_requests_running','num_requests_waiting'): hits=re.findall(r'^vllm:'+name+r'\{[^\n]+\} ([0-9.e+-]+)$',m,re.M) assert hits,(port,name) vals.extend(float(x) for x in hits) if all(x==0 for x in vals):break if time.monotonic()>deadline:raise AssertionError(('not drained',vals)) time.sleep(1) # Fresh request after cancellation checks that the service can continue. recovery={'model':'qwen','prompt':'The capital of France is','temperature':0,'max_tokens':16,'ignore_eos':True,'logprobs':5} actual=json.loads(request(18081,'/v1/completions',recovery)) assert actual['choices'][0]['text']==results[0]['baseline']['choices'][0]['text'] entry={'cycle':cycle,'stream_id':first['id'],'prefill':p,'first_chunk':first,'drained_metrics':metrics,'recovery':actual} cancellations.append(entry);print('CANCEL_RESULT',json.dumps(entry),flush=True) print('M04_CANCEL_RESULT_JSON='+json.dumps(cancellations),flush=True) for port in (18080,18081): print('HEALTH',port,request(port,'/health'),flush=True) print('METRICS',port,request(port,'/metrics'),flush=True) print('M04_PD_RESULT_JSON='+json.dumps({'cases':results,'passed':True}),flush=True) if __name__=='__main__': try: main() finally: import signal for p,f,log in reversed(procs): if p.poll() is None: p.send_signal(signal.SIGTERM) try:p.wait(timeout=45) except subprocess.TimeoutExpired:os.killpg(p.pid,signal.SIGKILL);p.wait() f.close();print('SERVER_EXIT',log.name,p.returncode,flush=True);print('SERVER_LOG_BEGIN',log.name,flush=True);print(log.read_text(),flush=True);print('SERVER_LOG_END',log.name,flush=True) assert all(p.returncode==0 for p,_,_ in procs), [(log.name,p.returncode) for p,_,log in procs] ``` </details> <details> <summary>pd-parent-sigterm/run-public.sh — public adaptation of recorded invocation</summary> ```sh #!/bin/sh # Public adaptation: SSH wrapper/container name removed; host model path replaced. # Run from this folder on the Ascend host; select free physical devices if needed. # This adapted wrapper has not been rerun. The image identifier is a local image ID. docker run --rm -i --network none --shm-size=2g --device /dev/davinci6 --device /dev/davinci7 --device /dev/davinci_manager --device /dev/devmm_svm --device /dev/hisi_hdc -e ASCEND_RT_VISIBLE_DEVICES=0,1 -e PYTHONUNBUFFERED=1 -e VLLM_WORKER_MULTIPROC_METHOD=spawn -e VLLM_ASCEND_ENABLE_NZ=0 -e GRAPH_MODE=0 -v /usr/local/Ascend/driver:/usr/local/Ascend/driver:ro -v /usr/local/sbin/npu-smi:/usr/local/sbin/npu-smi:ro -v /etc/ascend_install.info:/etc/ascend_install.info:ro -v /path/to/Qwen3-0.6B:/models/qwen3-0.6b:ro --entrypoint sh -w /vllm-workspace/vllm-ascend sha256:660ce23a83574fdcd28ba126b07615829c06770c06fbdf6533f03cc4d90acdb1 -c 'cat > /tmp/test_service.py; timeout 850 python /tmp/test_service.py' < test_service.py ``` </details> <details> <summary>no-connector-control/test_service.py — original script</summary> ```python import os, sys, subprocess, time, json, urllib.request, traceback from pathlib import Path procs=[] def request(port,path,payload=None): data=None if payload is None else json.dumps(payload).encode() req=urllib.request.Request(f'http://127.0.0.1:{port}{path}',data=data,headers={'Content-Type':'application/json'}) with urllib.request.urlopen(req,timeout=90) as r: return r.read().decode() def launch(role,dev,port,kvport): env=os.environ.copy();env.update(ASCEND_RT_VISIBLE_DEVICES=str(dev),VLLM_HOST_IP='127.0.0.1',HCCL_IF_IP='127.0.0.1',GLOO_SOCKET_IFNAME='lo',TP_SOCKET_IFNAME='lo',HCCL_SOCKET_IFNAME='lo',VLLM_WORKER_MULTIPROC_METHOD='spawn') cfg={'kv_connector':'MooncakeConnectorV1','kv_role':role,'kv_port':kvport,'kv_connector_extra_config':{'prefill':{'dp_size':1,'tp_size':1},'decode':{'dp_size':1,'tp_size':1}}} cmd=[sys.executable,'-m','vllm.entrypoints.openai.api_server','--model','/models/qwen3-0.6b','--served-model-name','qwen','--host','127.0.0.1','--port',str(port),'--dtype','bfloat16','--enforce-eager','--no-async-scheduling','--no-enable-prefix-caching','--max-model-len','512','--max-num-batched-tokens','512','--max-num-seqs','4','--num-gpu-blocks-override','64','--gpu-memory-utilization','0.15','--additional-config','{"weight_nz_mode":0}'] print('COMMAND',json.dumps(cmd),flush=True) log=Path(f'/tmp/{role}.log');f=log.open('w');p=subprocess.Popen(cmd,env=env,stdout=f,stderr=subprocess.STDOUT,start_new_session=True);procs.append((p,f,log)) deadline=time.monotonic()+300 while time.monotonic()<deadline: if p.poll() is not None: raise RuntimeError(f'{role} exited {p.returncode}') try: request(port,'/health');print('READY',role,flush=True);return except Exception: time.sleep(2) raise TimeoutError(role+' readiness') def main(): launch('control_a',0,18080,30000);launch('control_b',1,18081,30100) for port in (18080,18081): for prompt in ['The capital of France is','Write a short explanation of gravity.','A sequence of numbers: 1, 2, 3,']: payload={'model':'qwen','prompt':prompt,'temperature':0,'max_tokens':16,'ignore_eos':True,'logprobs':5} response=json.loads(request(port,'/v1/completions',payload)) assert response['usage']['completion_tokens']==16 print('CONTROL_RESPONSE',json.dumps({'port':port,'response':response}),flush=True) request(port,'/health') print('REQUEST_PHASE_PASSED',flush=True) if __name__=='__main__': try: main() finally: exits=[] import signal for p,f,log in reversed(procs): if p.poll() is None: p.terminate() try:p.wait(timeout=60) except subprocess.TimeoutExpired:os.killpg(p.pid,signal.SIGKILL);p.wait() f.close();print('SERVER_EXIT',log.name,p.returncode,flush=True);print('SERVER_LOG_BEGIN',log.name,flush=True);print(log.read_text(),flush=True);print('SERVER_LOG_END',log.name,flush=True) exits=[p.returncode for p,f,log in procs] print('EXIT_RESULT_JSON',json.dumps({'server_exits':exits,'connector':False,'signal':'API parent SIGTERM'}),flush=True) assert len(exits)==2 and all(code==0 for code in exits),'API server shutdown failed' ``` </details> <details> <summary>no-connector-control/run-public.sh — public adaptation of recorded invocation</summary> ```sh #!/bin/sh # Public adaptation: SSH wrapper/container name removed; host model path replaced. # Run from this folder on the Ascend host; select free physical devices if needed. # This adapted wrapper has not been rerun. The image identifier is a local image ID. docker run --rm -i --network none --shm-size=2g --device /dev/davinci6 --device /dev/davinci7 --device /dev/davinci_manager --device /dev/devmm_svm --device /dev/hisi_hdc -e ASCEND_RT_VISIBLE_DEVICES=0,1 -e PYTHONUNBUFFERED=1 -e VLLM_WORKER_MULTIPROC_METHOD=spawn -e VLLM_ASCEND_ENABLE_NZ=0 -e GRAPH_MODE=0 -v /usr/local/Ascend/driver:/usr/local/Ascend/driver:ro -v /usr/local/sbin/npu-smi:/usr/local/sbin/npu-smi:ro -v /etc/ascend_install.info:/etc/ascend_install.info:ro -v /path/to/Qwen3-0.6B:/models/qwen3-0.6b:ro --entrypoint sh -w /vllm-workspace/vllm-ascend sha256:660ce23a83574fdcd28ba126b07615829c06770c06fbdf6533f03cc4d90acdb1 -c 'cat > /tmp/test_service.py; timeout 850 python /tmp/test_service.py' < test_service.py ``` </details> ### Native isolation The diagnostic below launches separate subprocesses and checks their actual return codes. Its wrapper itself exits 0 after collecting the failures; that does not mean the child tests passed. The direct `ascend_transport.so` load has an unresolved-symbol error because it lacks its host library symbols; we do not treat that case as a production defect. <details> <summary>native-isolation/diagnose.py — original script</summary> ```python import subprocess,sys,json,os,importlib.util,pathlib p=pathlib.Path(importlib.util.find_spec('mooncake').origin).parent cases={ 'acl':"import ctypes; x=ctypes.CDLL('libascendcl.so')", 'datadist':"import ctypes; x=ctypes.CDLL('libllm_datadist.so')", 'transport':f"import ctypes; x=ctypes.CDLL({str(p/'ascend_transport.so')!r})", 'engine_library':f"import ctypes; x=ctypes.CDLL({str(p/'libtransfer_engine.so')!r})", 'engine_extension':f"import ctypes; x=ctypes.CDLL({str(p/'engine.so')!r})", 'import':"import mooncake.engine; print([(n,getattr(mooncake.engine.TransferEngine,n).__doc__) for n in dir(mooncake.engine.TransferEngine) if not n.startswith('_')],flush=True)", } results=[] for name,code in cases.items(): r=subprocess.run([sys.executable,'-X','faulthandler','-c',code+";print('BODY_OK',flush=True)"],capture_output=True,text=True,timeout=45) row=dict(name=name,code=code,returncode=r.returncode,stdout=r.stdout,stderr=r.stderr) results.append(row); print('CASE_JSON '+json.dumps(row),flush=True) env=os.environ.copy();env['LD_DEBUG']='libs' r=subprocess.run([sys.executable,'-c',"import mooncake.engine;print('BODY_OK',flush=True)"],env=env,capture_output=True,text=True,timeout=45) print('LOADER_JSON '+json.dumps(dict(returncode=r.returncode,stdout=r.stdout,stderr=r.stderr)),flush=True) print('RESULT_JSON '+json.dumps(results),flush=True) ``` </details> <details> <summary>native-isolation/run-public.sh</summary> ```sh #!/bin/sh # Public adaptation: SSH wrapper/container name removed; host model path replaced. # Run from this folder on the Ascend host; select free physical devices if needed. # This adapted wrapper has not been rerun. The image identifier is a local image ID. docker run --rm -i --network none -e TORCH_DEVICE_BACKEND_AUTOLOAD=0 -v /usr/local/Ascend/driver:/usr/local/Ascend/driver:ro --entrypoint python sha256:660ce23a83574fdcd28ba126b07615829c06770c06fbdf6533f03cc4d90acdb1 - < diagnose.py ``` </details> <details> <summary>Native diagnostic result summary (derived from original results.json)</summary> ```json [ { "name": "acl", "returncode": 0, "stderr": "", "body_ok": true }, { "name": "datadist", "returncode": 0, "stderr": "", "body_ok": true }, { "name": "transport", "returncode": 1, "stderr": "Traceback (most recent call last):\n File \"<string>\", line 1, in <module>\n File \"/usr/local/python3.12.13/lib/python3.12/ctypes/__init__.py\", line 379, in __init__\n self._handle = _dlopen(self._name, mode)\n ^^^^^^^^^^^^^^^^^^^^^^^^^\nOSError: /usr/local/python3.12.13/lib/python3.12/site-packages/mooncake/ascend_transport.so: undefined symbol: _ZTIN8mooncake9TransportE\n", "body_ok": false }, { "name": "engine_library", "returncode": -6, "stderr": "corrupted size vs. prev_size\n", "body_ok": true }, { "name": "engine_extension", "returncode": -6, "stderr": "corrupted size vs. prev_size\n", "body_ok": true }, { "name": "import", "returncode": -6, "stderr": "corrupted size vs. prev_size\n", "body_ok": true } ] ``` </details> The two API crashes show a deployment lifecycle impact beyond standalone imports: the recorded P/D work completes, but service shutdown is reported as a process failure. We are adding this as corroborating evidence for the existing dependency issue, not a new independently established root cause. The next check is an isolated 0.3.12 retest of both import exit and this P/D shutdown workload while keeping other conditions fixed.

bsakaguchi referenced this issue 2026-10-29 21:32:18 +00:00

[PR #1146] [MERGED] feat[Store]: Refine shm mmap logic and add reconnection for Dummy Client after the Real restarted #2038

Sign in to join this conversation.

No Branch/Tag specified

Branches Tags

main

te-coro

staryxchen-tent-capability-research

feature/musa-ep-jit

zxpdemonio/fix-tent-bootstrap-admission

add-store-rs-tone-tests

scatter-rdma-completion-pr2

P2P-Mooncake-Store

release/v0.3.14

feat/tent-rdma-adaptive-adapter

fix/ep-local-counter-gpu-scope

codex/kvcs-oplog-eviction

give-me-a-star

cruz/mooncake-kvcs-lowlevel-pr1-20261008

ylt-handshake-limits

ci/rocm-e2e-tent

sima/docs-pypi-downloads

fix-ci/hicache-mooncake-metadata-loopback

sima/cuda-driver-loading

sima/ci-performance-gate

fix/tent-rdma-unknown-topology-rail-pairing

ci/tent-e2e-dual-job

mc-patch-1

codex/p2p-merge-prerequisites

sima/store-hca-numa-placement

codex/p2p-merge-cli

codex/p2p-merge-ipv6

copilot/fix-ctests-job-failure

czh/pg-device-collectives-pr6

dependabot/cargo/mooncake-store-rs/pyo3-0.29.0

czh/pg-device-collectives-pr5

codex/segment-pool-master-cutover

sima/nightly-ci-recovery

ci/tent-params

stmatengss-patch-1

dependabot/cargo/mooncake-store-rs/rand-0.8.8

dependabot/cargo/mooncake-store-rs/opentelemetry_sdk-0.32.1

dependabot/cargo/mooncake-store-rs/xxhash-rust-0.8.16

codex/migrate-store-rest-service

fix/tone-wheel-arch-selection

czh/pg-device-collectives-pr4

czh/pg-device-collectives-pr3

ketong/fix_moe_tc

copilot/fix-test-wheel-ubuntu-job

fix/ci-wheel-torch-pg-pin

codex/migrate-structured-object-storage

codex/migrate-allocator-python

codex/migrate-vllm-python

codex/migrate-shared-segment

ci/fix_sgl_eval_install

codex/segment-pool-nof-placement-cleanup

feat/pip-native-release

copilot/bugfix-ha-standby-promotion

copilot/mooncake-store-transport-settings

copilot/fix-tensor-support-issues

copilot/optimize-rl-direction

copilot/mooncake-storeagentic-workflow

copilot/fix-client-reconnect-timeout

copilot/mooncake-error-debugging

copilot/mooncake-ipv6-support

feat/nccl-host-python-api

dependabot/go_modules/mooncake-p2p-store/src/p2pstore/google.golang.org/grpc-1.82.1

copilot/migrate-and-move-operations

copilot/migrate-and-move-operations-again

refactor/centralize-env-vars

idhanani/mooncake-retain-groups

copilot/codeownerreviewer

copilot/give-feedback-on-issue-2123

copilot/create-issue-rfc

copilot/break-cuda-monopoly-logic

copilot/compare-cudamemcpy-cu12-cu13

copilot/report-ci-error-issue

copilot/research-commit-history-analysis

copilot/explain-mooncake-store-service

copilot/analyze-project-functionality

copilot/sub-pr-1680

copilot/review-pr-1680

copilot/sub-pr-1597

copilot/fix-github-actions-job-failure

copilot/move-test-sglang-integration-job

v0.3.14

v0.3.14-pre-4

v0.3.14-pre-3

v0.3.14-pre-2

v0.3.14-pre

v0.3.14-rc1

v0.3.13.post1

v0.3.13-rc2

v0.3.13-rc1

v0.3.13

v0.3.12.post1

v0.3.12

v0.3.11-rc1

v0.3.11.post1

v0.3.11

v0.3.10.post2

v0.3.10.post1

v0.3.10

v0.3.9

v0.3.8.post1

v0.3.8

v0.3.7.post2

v0.3.7.post1

v0.3.7

v0.3.6.post1

v0.3.6

v0.3.5

v0.3.4.post2

v0.3.4.post1

v0.3.4

v0.3.3.post2

v0.3.3.post1

v0.3.3

v0.3.2.post1

v0.3.2

v0.3.1

v0.3.0

v0.3.0-beta

v0.3.0-alpha

v0.2.0

Labels

Clear labels   

ascend

  

auto-closed

  

bug

  

bug

  

CI/Build

  

Common

  

documentation

  

enhancement

  

feature request

  

good first issue

  

help wanted

  

high priority

  

MetaX

  

nightly-failure

  

pull-request

Mirrored from GitHub Pull Request

  

PyTorch Backend

  

question

  

RFC

  

Roadmap

  

stale

  

Store

  

Tests

  

Transfer Engine

No labels

ascend

auto-closed

bug

bug

CI/Build

Common

documentation

enhancement

feature request

good first issue

help wanted

high priority

MetaX

nightly-failure

pull-request

PyTorch Backend

question

RFC

Roadmap

stale

Store

Tests

Transfer Engine

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

kvcache-ai/Mooncake#1146

No description provided.