[GH-ISSUE #1008] [Bug]: core dump when compile with ascend direct #336

New issue

Closed

opened 2026-10-29 21:12:54 +00:00 by bsakaguchi · 3 comments

bsakaguchi commented

2026-10-29 21:12:54 +00:00

Owner

Originally created by @SpecterCipher on GitHub (Nov 3, 2025).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/1008

Bug Report

When I compile with -DUSE_ASCEND_DIRECT=ON, it will coredump

but stack is confused

#0  0x0000ffffb8d1fce0 in ?? () from /usr/lib64/libc.so.6
#1  0x0000ffffb8cd8efc in raise () from /usr/lib64/libc.so.6
#2  0x0000ffffb8cc6d70 in abort () from /usr/lib64/libc.so.6
#3  0x0000ffffb8d13d1c in ?? () from /usr/lib64/libc.so.6
#4  0x0000ffffb8d29dac in ?? () from /usr/lib64/libc.so.6
#5  0x0000ffffb8d2a668 in ?? () from /usr/lib64/libc.so.6
#6  0x0000ffffb8d2a7cc in ?? () from /usr/lib64/libc.so.6
#7  0x0000ffffb8d2c6f8 in ?? () from /usr/lib64/libc.so.6
#8  0x0000ffffb8d2e630 in calloc () from /usr/lib64/libc.so.6
#9  0x0000ffffb8d126dc in open_memstream () from /usr/lib64/libc.so.6
#10 0x0000ffffb8d809e4 in ?? () from /usr/lib64/libc.so.6
#11 0x0000ffffb8d80f50 in vsyslog () from /usr/lib64/libc.so.6
#12 0x0000ffffb1025528 in LogPrintSys (priority=6, format=0xffffb1035880 "(%d)%s:%d: close driver library handle succeed.\n") at ../../../../../../../../../abl/slog/utils/log_print_syslog.c:18
#13 0x0000ffffb102d848 in UnloadDriverDllFunctions () at ../../../../../../../../../abl/slog/liblog/slog/plog/plog_driver_api.c:66
#14 0x0000ffffb102e03c in DrvFunctionsUninit () at ../../../../../../../../../abl/slog/liblog/slog/plog/plog_drv.c:34
#15 0x0000ffffb1026d90 in ProcessLogFree () at ../../../../../../../../../abl/slog/liblog/slog/plog/plog_core.c:167
#16 0x0000ffffb9478918 in ?? () from /lib/ld-linux-aarch64.so.1
#17 0x0000ffffb8cdb880 in ?? () from /usr/lib64/libc.so.6
#18 0x0000ffffb8cdb98c in exit () from /usr/lib64/libc.so.6
#19 0x0000ffffb8cc7004 in ?? () from /usr/lib64/libc.so.6
#20 0x0000ffffb8cc70d8 in __libc_start_main () from /usr/lib64/libc.so.6
#21 0x00000000004007b0 in mooncake::TransferMetadata::dumpMetadataContentUnlocked (this=0xffffd5c4d21a) at Mooncake/mooncake-transfer-engine/src/transfer_metadata_dump.cpp:66
#22 0x0000ffffd5c4d1f8 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)
(gdb) f 21
#21 0x00000000004007b0 in mooncake::TransferMetadata::dumpMetadataContentUnlocked (this=0xffffd5c4d21a) at Mooncake/mooncake-transfer-engine/src/transfer_metadata_dump.cpp:66
66          LOG(INFO) << "=== Cached Segment Descriptors ===";
(gdb) p *this
$1 = {p2p_handshake_mode_ = 72, common_key_prefix_ = <error: Cannot access memory at address 0x767265737665643d>, rpc_meta_prefix_ = <error: Cannot access memory at address 0x4e555253504f5f42>, segment_lock_ = {ticket = {
      whole = 4683797751942501711, readWrite = 1414419791, {write = 21839, read = 21582, users = 12605}}, static kExclusiveLock = -4611686018427387904, lock_ = {<std::__atomic_base<long>> = {static _S_alignment = 8, 
        _M_i = 4703255907987833428}, static is_always_lock_free = true}, padding_ = {4998788115189686093, 5644508277452719446, 4774183139848373582, 4779516573612066124, 5642541195215261535, 5788625552782873157, 4998790288072852319, 
      13859641700270414, 8245940487513724752, 4913257195995427691, 8391721685706109801, 6874015182293787749, 7166178288370610022, 7957701443851677285, 4779516574504280419}}, 
  segment_id_to_desc_map_ = std::unordered_map with 6071133984153944386 elements<error reading variable: Cannot access memory at address 0x4f4c475f434f4c4c>, 
  segment_name_to_id_map_ = std::unordered_map with 5928239340454104665 elements<error reading variable: Cannot access memory at address 0x535f4d4145525453>, notify_lock_ = {ticket = {whole = 6868628584317407827, readWrite = 1314214483, {
        write = 21075, read = 20053, users = 17742}}, static kExclusiveLock = -4611686018427387904, lock_ = {<std::__atomic_base<long>> = {static _S_alignment = 8, _M_i = 4854682783879152971}, static is_always_lock_free = true}, 
    padding_ = {5714020501736145729, 5644504858759021121, 4993734732483476820, 4683871634425261885, 6864978209908146772, 8319607826889851216, 3417213296966381426, 7939675093387670337, 3414419428256538990, 7002943984556597612, 
      7016459202921980532, 5714878811961518434, 5214972970053157708, 8247064188397374799, 4697092101906590767}}, notifys = std::vector of length 15776974691121160, capacity -33886645839720416 = {<error reading variable>

Before submitting...

Ensure you searched for relevant issues and read the [documentation]

Originally created by @SpecterCipher on GitHub (Nov 3, 2025). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/1008 ### Bug Report When I compile with -DUSE_ASCEND_DIRECT=ON, it will coredump <img width="721" height="59" alt="Image" src="https://github.com/user-attachments/assets/89839f8b-bb51-4811-bfd5-82d82145d51c" /> but stack is confused ``` #0 0x0000ffffb8d1fce0 in ?? () from /usr/lib64/libc.so.6 #1 0x0000ffffb8cd8efc in raise () from /usr/lib64/libc.so.6 #2 0x0000ffffb8cc6d70 in abort () from /usr/lib64/libc.so.6 #3 0x0000ffffb8d13d1c in ?? () from /usr/lib64/libc.so.6 #4 0x0000ffffb8d29dac in ?? () from /usr/lib64/libc.so.6 #5 0x0000ffffb8d2a668 in ?? () from /usr/lib64/libc.so.6 #6 0x0000ffffb8d2a7cc in ?? () from /usr/lib64/libc.so.6 #7 0x0000ffffb8d2c6f8 in ?? () from /usr/lib64/libc.so.6 #8 0x0000ffffb8d2e630 in calloc () from /usr/lib64/libc.so.6 #9 0x0000ffffb8d126dc in open_memstream () from /usr/lib64/libc.so.6 #10 0x0000ffffb8d809e4 in ?? () from /usr/lib64/libc.so.6 #11 0x0000ffffb8d80f50 in vsyslog () from /usr/lib64/libc.so.6 #12 0x0000ffffb1025528 in LogPrintSys (priority=6, format=0xffffb1035880 "(%d)%s:%d: close driver library handle succeed.\n") at ../../../../../../../../../abl/slog/utils/log_print_syslog.c:18 #13 0x0000ffffb102d848 in UnloadDriverDllFunctions () at ../../../../../../../../../abl/slog/liblog/slog/plog/plog_driver_api.c:66 #14 0x0000ffffb102e03c in DrvFunctionsUninit () at ../../../../../../../../../abl/slog/liblog/slog/plog/plog_drv.c:34 #15 0x0000ffffb1026d90 in ProcessLogFree () at ../../../../../../../../../abl/slog/liblog/slog/plog/plog_core.c:167 #16 0x0000ffffb9478918 in ?? () from /lib/ld-linux-aarch64.so.1 #17 0x0000ffffb8cdb880 in ?? () from /usr/lib64/libc.so.6 #18 0x0000ffffb8cdb98c in exit () from /usr/lib64/libc.so.6 #19 0x0000ffffb8cc7004 in ?? () from /usr/lib64/libc.so.6 #20 0x0000ffffb8cc70d8 in __libc_start_main () from /usr/lib64/libc.so.6 #21 0x00000000004007b0 in mooncake::TransferMetadata::dumpMetadataContentUnlocked (this=0xffffd5c4d21a) at Mooncake/mooncake-transfer-engine/src/transfer_metadata_dump.cpp:66 #22 0x0000ffffd5c4d1f8 in ?? () Backtrace stopped: previous frame inner to this frame (corrupt stack?) (gdb) f 21 #21 0x00000000004007b0 in mooncake::TransferMetadata::dumpMetadataContentUnlocked (this=0xffffd5c4d21a) at Mooncake/mooncake-transfer-engine/src/transfer_metadata_dump.cpp:66 66 LOG(INFO) << "=== Cached Segment Descriptors ==="; (gdb) p *this $1 = {p2p_handshake_mode_ = 72, common_key_prefix_ = <error: Cannot access memory at address 0x767265737665643d>, rpc_meta_prefix_ = <error: Cannot access memory at address 0x4e555253504f5f42>, segment_lock_ = {ticket = { whole = 4683797751942501711, readWrite = 1414419791, {write = 21839, read = 21582, users = 12605}}, static kExclusiveLock = -4611686018427387904, lock_ = {<std::__atomic_base<long>> = {static _S_alignment = 8, _M_i = 4703255907987833428}, static is_always_lock_free = true}, padding_ = {4998788115189686093, 5644508277452719446, 4774183139848373582, 4779516573612066124, 5642541195215261535, 5788625552782873157, 4998790288072852319, 13859641700270414, 8245940487513724752, 4913257195995427691, 8391721685706109801, 6874015182293787749, 7166178288370610022, 7957701443851677285, 4779516574504280419}}, segment_id_to_desc_map_ = std::unordered_map with 6071133984153944386 elements<error reading variable: Cannot access memory at address 0x4f4c475f434f4c4c>, segment_name_to_id_map_ = std::unordered_map with 5928239340454104665 elements<error reading variable: Cannot access memory at address 0x535f4d4145525453>, notify_lock_ = {ticket = {whole = 6868628584317407827, readWrite = 1314214483, { write = 21075, read = 20053, users = 17742}}, static kExclusiveLock = -4611686018427387904, lock_ = {<std::__atomic_base<long>> = {static _S_alignment = 8, _M_i = 4854682783879152971}, static is_always_lock_free = true}, padding_ = {5714020501736145729, 5644504858759021121, 4993734732483476820, 4683871634425261885, 6864978209908146772, 8319607826889851216, 3417213296966381426, 7939675093387670337, 3414419428256538990, 7002943984556597612, 7016459202921980532, 5714878811961518434, 5214972970053157708, 8247064188397374799, 4697092101906590767}}, notifys = std::vector of length 15776974691121160, capacity -33886645839720416 = {<error reading variable> ``` ### Before submitting... - [x] Ensure you searched for relevant issues and read the [documentation]

bsakaguchi

2026-10-29 21:12:54 +00:00

closed this issue
added the
bug
label

bsakaguchi commented

2026-10-29 21:12:55 +00:00

Author

Owner

@ShangmingCai commented on GitHub (Nov 3, 2025):

CC: @ascend-direct-dev @AscendTransport @zuochunwei

@ShangmingCai commented on GitHub (Nov 3, 2025): CC: @ascend-direct-dev @AscendTransport @zuochunwei

bsakaguchi commented

2026-10-29 21:12:55 +00:00

Author

Owner

@ascend-direct-dev commented on GitHub (Nov 4, 2025):

To use Ascend Direct Transport, you should set the device firstly.

import torch
import torch_npu
torch.npu.set_device(0)
from mooncake.engine import TransferEngine
transfer_engine = TransferEngine()

@ascend-direct-dev commented on GitHub (Nov 4, 2025): To use Ascend Direct Transport, you should set the device firstly. ```python import torch import torch_npu torch.npu.set_device(0) from mooncake.engine import TransferEngine transfer_engine = TransferEngine() ```

bsakaguchi commented

2026-10-29 21:12:56 +00:00

Author

Owner

@SpecterCipher commented on GitHub (Nov 4, 2025):

To use Ascend Direct Transport, you should set the device firstly.

import torch
import torch_npu
torch.npu.set_device(0)
from mooncake.engine import TransferEngine
transfer_engine = TransferEngine()

it worked, thanks.

@SpecterCipher commented on GitHub (Nov 4, 2025): > To use Ascend Direct Transport, you should set the device firstly. > > import torch > import torch_npu > torch.npu.set_device(0) > from mooncake.engine import TransferEngine > transfer_engine = TransferEngine() it worked, thanks.

bsakaguchi referenced this issue

2026-10-29 21:27:55 +00:00

[PR #336] [MERGED] docs: add lmcache integration documentation #1491