kvcache-ai/Mooncake

Fork 0

mirror of https://github.com/kvcache-ai/Mooncake.git synced 2026-11-05 01:17:55 +00:00

[GH-ISSUE #1738] [Usage]: How to compile transfer engine wheel cp312 #503

New issue

Closed

opened 2026-10-29 21:15:33 +00:00 by bsakaguchi · 6 comments

bsakaguchi commented

2026-10-29 21:15:33 +00:00

Owner

Originally created by @easonyang1010 on GitHub (Mar 24, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/1738

Originally assigned to: @staryxchen on GitHub.

Describe your usage question

when I use this command, still got cp310 wheel
docker build -f docker/mooncake.Dockerfile \ --build-arg PYTHON_VERSION=3.12 \ --build-arg EP_TORCH_VERSIONS="2.9.1" \ -t mooncake:from-source .

Looking forward to your reply

Before submitting a new issue...

Make sure you already searched for relevant issues and read the documentation

Originally created by @easonyang1010 on GitHub (Mar 24, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/1738 Originally assigned to: @staryxchen on GitHub. ### Describe your usage question when I use this command, still got cp310 wheel `docker build -f docker/mooncake.Dockerfile \ --build-arg PYTHON_VERSION=3.12 \ --build-arg EP_TORCH_VERSIONS="2.9.1" \ -t mooncake:from-source .` Looking forward to your reply ### Before submitting a new issue... - [x] Make sure you already searched for relevant issues and read the [documentation](https://kvcache-ai.github.io/Mooncake/)

bsakaguchi closed this issue

2026-10-29 21:15:33 +00:00

bsakaguchi commented

2026-10-29 21:15:34 +00:00

Author

Owner

@staryxchen commented on GitHub (Mar 25, 2026):

Hi @easonyang1010, thanks for reporting this!

The root cause is that mooncake.Dockerfile was installing the Ubuntu system Python (python3 = 3.10 on Ubuntu 22.04) regardless of the PYTHON_VERSION build-arg, and build_wheel.sh was calling python -m build using the system default interpreter — so the wheel ABI tag was always cp310.

We've submitted a fix in #1745. Could you give it a try by building from that branch?

git fetch origin pull/1745/head:fix/dockerfile-python-version
git checkout fix/dockerfile-python-version

docker build -f docker/mooncake.Dockerfile   --build-arg PYTHON_VERSION=3.12   --build-arg EP_TORCH_VERSIONS="2.9.1"   -t mooncake:from-source .

You should now get a cp312 wheel. Please let us know if it works for you!

@staryxchen commented on GitHub (Mar 25, 2026): Hi @easonyang1010, thanks for reporting this! The root cause is that `mooncake.Dockerfile` was installing the Ubuntu system Python (`python3` = 3.10 on Ubuntu 22.04) regardless of the `PYTHON_VERSION` build-arg, and `build_wheel.sh` was calling `python -m build` using the system default interpreter — so the wheel ABI tag was always `cp310`. We've submitted a fix in #1745. Could you give it a try by building from that branch? ```bash git fetch origin pull/1745/head:fix/dockerfile-python-version git checkout fix/dockerfile-python-version docker build -f docker/mooncake.Dockerfile --build-arg PYTHON_VERSION=3.12 --build-arg EP_TORCH_VERSIONS="2.9.1" -t mooncake:from-source . ``` You should now get a `cp312` wheel. Please let us know if it works for you!

bsakaguchi commented

2026-10-29 21:15:34 +00:00

Author

Owner

@easonyang1010 commented on GitHub (Mar 25, 2026):

Sure, I would love to give it a try

@easonyang1010 commented on GitHub (Mar 25, 2026): Sure, I would love to give it a try

bsakaguchi commented

2026-10-29 21:15:34 +00:00

Author

Owner

@easonyang1010 commented on GitHub (Mar 26, 2026):

After testing, this indeed solves the problem.Thanks!

@easonyang1010 commented on GitHub (Mar 26, 2026): After testing, this indeed solves the problem.Thanks!

bsakaguchi commented

2026-10-29 21:15:34 +00:00

Author

Owner

@easonyang1010 commented on GitHub (Apr 8, 2026):

@staryxchen hi, I found your commit can produce cp312 wheel, but when using this cp312 wheel, error occurs
ImportError: Python version mismatch: module was compiled for Python 3.10, but the interpreter version is incompatible: 3.12.11
this is build log

You can now build and run Mooncake.
Note: You may need to restart your terminal or run 'source ~/.bashrc' to use Go.
 ---> Removed intermediate container a301cdfc018e
 ---> 602c160deac7
Step 15/17 : RUN mkdir -p build &&     cd build &&     cmake ..         -DBUILD_UNIT_TESTS=OFF         -DUSE_CUDA=ON         -DCMAKE_BUILD_TYPE=${CMAKE_BUILD_TYPE} &&     export LIBRARY_PATH=/usr/local/cuda/lib64/stubs:$LIBRARY_PATH &&    cmake --build . -j"$(nproc)"
 ---> Running in 7d42fe6fc530
-- The CXX compiler identification is GNU 11.4.0
-- The C compiler identification is GNU 11.4.0
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Found JsonCpp: JsonCpp::JsonCpp  
-- Found GLOG: glog::glog  
CMAKE_BUILD_TYPE: Release
-- [limit_jobs] Available memory: 30250 MB, CPU cores: 19
-- [limit_jobs] Compile jobs: 19 (1500 MB/job), Link jobs: 7 (4000 MB/job)
-- [limit_jobs] Hint: use -G Ninja for automatic compile/link parallelism separation
-- [limit_jobs] With Make, recommend: cmake --build . -j7
-- Event-driven completion is disabled
-- CUDA support is enabled
-- Http as metadata server support is enabled
-- metrics is enabled
YLT is imported by find_package
-- -------------YLT CONFIG SETTING-------------
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success
-- Found Threads: TRUE  
-- YLT_ENABLE_DL: ON
-- YLT_ENABLE_SSL: OFF
-- YLT_ENABLE_NTLS: OFF
-- YLT_ENABLE_PMR: OFF
-- YLT_ENABLE_IO_URING: OFF
-- YLT_ENABLE_STRUCT_PACK_UNPORTABLE_TYPE: OFF
-- YLT_ENABLE_STRUCT_PACK_OPTIMIZE: OFF
-- --------------------------------------------
-- pybind11 v2.13.6 
-- Found PythonInterp: /usr/bin/python3 (found suitable version "3.10.12", minimum required is "3.7") 
-- Found PythonLibs: /usr/lib/x86_64-linux-gnu/libpython3.10.so
-- Performing Test HAS_FLTO
-- Performing Test HAS_FLTO - Success
-- Found ASIO at: /usr/local/include
-- Found Python3: /usr/bin/python3.10 (found version "3.10.12") found components: Interpreter Development Development.Module Development.Embed 
-- Found CURL: /usr/lib/x86_64-linux-gnu/libcurl.so (found version "7.81.0")  
-- Mooncake Store will be built
-- Store HA backends are disabled
-- AWS SDK not found, S3 functionality will be disabled
-- Found xxHash: include=/usr/include lib=/usr/lib/x86_64-linux-gnu/libxxhash.so
-- io_uring: Enabled for mooncake_store
-- Mooncake Store Rust bindings will be built
/usr/local/lib/python3.10/dist-packages
-- Found CUDAToolkit: /usr/local/cuda/include (found version "12.8.93") 
/usr/local/lib/python3.10/dist-packages
-- Configuring done
-- Generating done
-- Build files have been written to: /workspace/build

@easonyang1010 commented on GitHub (Apr 8, 2026): @staryxchen hi, I found your commit can produce cp312 wheel, but when using this cp312 wheel, error occurs `ImportError: Python version mismatch: module was compiled for Python 3.10, but the interpreter version is incompatible: 3.12.11` this is build log ``` You can now build and run Mooncake. Note: You may need to restart your terminal or run 'source ~/.bashrc' to use Go. ---> Removed intermediate container a301cdfc018e ---> 602c160deac7 Step 15/17 : RUN mkdir -p build && cd build && cmake .. -DBUILD_UNIT_TESTS=OFF -DUSE_CUDA=ON -DCMAKE_BUILD_TYPE=${CMAKE_BUILD_TYPE} && export LIBRARY_PATH=/usr/local/cuda/lib64/stubs:$LIBRARY_PATH && cmake --build . -j"$(nproc)" ---> Running in 7d42fe6fc530 -- The CXX compiler identification is GNU 11.4.0 -- The C compiler identification is GNU 11.4.0 -- Detecting CXX compiler ABI info -- Detecting CXX compiler ABI info - done -- Check for working CXX compiler: /usr/bin/c++ - skipped -- Detecting CXX compile features -- Detecting CXX compile features - done -- Detecting C compiler ABI info -- Detecting C compiler ABI info - done -- Check for working C compiler: /usr/bin/cc - skipped -- Detecting C compile features -- Detecting C compile features - done -- Found JsonCpp: JsonCpp::JsonCpp -- Found GLOG: glog::glog CMAKE_BUILD_TYPE: Release -- [limit_jobs] Available memory: 30250 MB, CPU cores: 19 -- [limit_jobs] Compile jobs: 19 (1500 MB/job), Link jobs: 7 (4000 MB/job) -- [limit_jobs] Hint: use -G Ninja for automatic compile/link parallelism separation -- [limit_jobs] With Make, recommend: cmake --build . -j7 -- Event-driven completion is disabled -- CUDA support is enabled -- Http as metadata server support is enabled -- metrics is enabled YLT is imported by find_package -- -------------YLT CONFIG SETTING------------- -- Looking for pthread.h -- Looking for pthread.h - found -- Performing Test CMAKE_HAVE_LIBC_PTHREAD -- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success -- Found Threads: TRUE -- YLT_ENABLE_DL: ON -- YLT_ENABLE_SSL: OFF -- YLT_ENABLE_NTLS: OFF -- YLT_ENABLE_PMR: OFF -- YLT_ENABLE_IO_URING: OFF -- YLT_ENABLE_STRUCT_PACK_UNPORTABLE_TYPE: OFF -- YLT_ENABLE_STRUCT_PACK_OPTIMIZE: OFF -- -------------------------------------------- -- pybind11 v2.13.6 -- Found PythonInterp: /usr/bin/python3 (found suitable version "3.10.12", minimum required is "3.7") -- Found PythonLibs: /usr/lib/x86_64-linux-gnu/libpython3.10.so -- Performing Test HAS_FLTO -- Performing Test HAS_FLTO - Success -- Found ASIO at: /usr/local/include -- Found Python3: /usr/bin/python3.10 (found version "3.10.12") found components: Interpreter Development Development.Module Development.Embed -- Found CURL: /usr/lib/x86_64-linux-gnu/libcurl.so (found version "7.81.0") -- Mooncake Store will be built -- Store HA backends are disabled -- AWS SDK not found, S3 functionality will be disabled -- Found xxHash: include=/usr/include lib=/usr/lib/x86_64-linux-gnu/libxxhash.so -- io_uring: Enabled for mooncake_store -- Mooncake Store Rust bindings will be built /usr/local/lib/python3.10/dist-packages -- Found CUDAToolkit: /usr/local/cuda/include (found version "12.8.93") /usr/local/lib/python3.10/dist-packages -- Configuring done -- Generating done -- Build files have been written to: /workspace/build ```

bsakaguchi commented

2026-10-29 21:15:34 +00:00

Author

Owner

@staryxchen commented on GitHub (Apr 8, 2026):

Hi @easonyang1010, thanks for the follow-up — you're right, the previous fix only corrected the wheel ABI tag but the C++ extensions (pybind11 .so files) were still compiled against Python 3.10.

The root cause was that CMake's find_package(Python3) and pybind11's FindPythonInterp were still discovering the system Python 3.10 during the build, ignoring the PYTHON_VERSION build-arg.

I've pushed additional fixes to #1745:

Pass -DPython3_EXECUTABLE to CMake so find_package(Python3) picks up the correct interpreter
Set PYTHON_EXECUTABLE before add_subdirectory(pybind11) so pybind11's legacy FindPythonInterp also uses the right Python
Fix the runtime stage to install the matching Python version (instead of hardcoded system python3)

Could you give it another try?

git fetch origin pull/1745/head:fix/dockerfile-python-version
git checkout fix/dockerfile-python-version

docker build -f docker/mooncake.Dockerfile   --build-arg PYTHON_VERSION=3.12   --build-arg EP_TORCH_VERSIONS="2.9.1"   -t mooncake:from-source .

This time the CMake output should show Found Python3: /usr/bin/python3.12 (instead of 3.10). Let us know if it works!

@staryxchen commented on GitHub (Apr 8, 2026): Hi @easonyang1010, thanks for the follow-up — you're right, the previous fix only corrected the wheel ABI tag but the C++ extensions (pybind11 `.so` files) were still compiled against Python 3.10. The root cause was that CMake's `find_package(Python3)` and pybind11's `FindPythonInterp` were still discovering the system Python 3.10 during the build, ignoring the `PYTHON_VERSION` build-arg. I've pushed additional fixes to #1745: - Pass `-DPython3_EXECUTABLE` to CMake so `find_package(Python3)` picks up the correct interpreter - Set `PYTHON_EXECUTABLE` **before** `add_subdirectory(pybind11)` so pybind11's legacy `FindPythonInterp` also uses the right Python - Fix the runtime stage to install the matching Python version (instead of hardcoded system python3) Could you give it another try? ```bash git fetch origin pull/1745/head:fix/dockerfile-python-version git checkout fix/dockerfile-python-version docker build -f docker/mooncake.Dockerfile --build-arg PYTHON_VERSION=3.12 --build-arg EP_TORCH_VERSIONS="2.9.1" -t mooncake:from-source . ``` This time the CMake output should show `Found Python3: /usr/bin/python3.12` (instead of 3.10). Let us know if it works!

bsakaguchi commented

2026-10-29 21:15:35 +00:00

Author

Owner

@easonyang1010 commented on GitHub (Apr 9, 2026):

@staryxchen Hi, I have tried this solution, it does solve problems.

@easonyang1010 commented on GitHub (Apr 9, 2026): @staryxchen Hi, I have tried this solution, it does solve problems.

bsakaguchi referenced this issue

2026-10-29 21:28:53 +00:00

[PR #503] [MERGED] fix(transfer-task): fix error hanlding logic in transfer task #1611