[GH-ISSUE #5036] [RFC]: Intel XPU CI Coverage and Resource Allocation Proposal #1274

Open
opened 2026-10-29 21:24:42 +00:00 by bsakaguchi · 3 comments
Owner

Originally created by @VincyZhang on GitHub (Oct 28, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/5036

Summary

Introduce XPU CI with broad C++ and Python unit/component coverage and two model-level E2E smoke tests. Keep model execution out of PR core testing; run both application scenarios nightly.

Motivation and Goals

Mooncake already has XPU platform, transport, Store, IPC, and native EP tests, but no dedicated XPU Actions workflow at the assessment baseline. Existing tests should become repeatable CI coverage rather than relying primarily on expensive application runs.

The goals are to:

  • Detect most regressions with focused C++ and Python tests.
  • Validate Store and EP component boundaries without loading a model.
  • Retain minimal real-model coverage for KV save/load and disaggregated prefill/decode (PD).
  • Separate CPU-only builds from GPU execution.
  • Make device ownership, tested revisions, failures, and coverage explicit.

Initial non-goals are RDMA, cross-node validation, GDS, EP with more than two ranks, broad model matrices, and model-quality or performance certification. There is no RDMA hardware available.

This RFC also presents two resource proposals:

  1. Fixed runners: start with two two-cards runners, four GPUs in total, then expand to eight GPUs with runner sizes selected according to demand.
  2. Kubernetes allocation: manage an eight-GPU pool and allocate devices dynamically to test jobs submitted from GitHub Actions through kubectl.

The resource choice does not change the initial test scope or trigger policy.

Test Coverage and Frequency

GPU counts below describe test requirements, not necessarily scheduler reservations. Existing C++ binaries mix single- and dual-GPU cases, so the core job requests two GPUs and executes its phases sequentially.

Test category Coverage GPUs and supporting resources Expected frequency
Host-side C++ / Python UT Device-independent configuration, routing policy, parameter validation, and error handling; reuse or add focused cases as needed 0; hosted CPU runner Every relevant PR update and main push; nightly
XPU build and wheel checks Compilation, packaging, clean-environment installation, native extension loading, and packaged master startup 0; oneAPI build environment and matching runtime Every relevant PR update and main push; nightly
C++ platform / transport USM allocation and pointer classification, H2D/D2H/D2D, offset access, registration/lifetimes, and cross-device copies 1 for basic cases, 2 for cross-device cases Maintainer-triggered core; nightly
C++ engine / IPC Routing, TCP staging, IPC, failure fallback, io_uring file I/O, and real cross-process behavior 1 or 2 depending on the case; IPC permissions and io_uring support Maintainer-triggered core; nightly
C++ Store component integration Foreign USM buffer put/get and byte-for-byte roundtrip validation 1; lightweight local master Maintainer-triggered core; nightly
Python UT / Store component integration Binding parameters, exceptions, return values, tensor/buffer lifetimes, XPU tensor registration, and Store put/get 1 for device cases; XPU PyTorch and local master for Store Maintainer-triggered core; nightly
Python EP component integration Native dispatch/combine with synthetic activations and routing, BF16/FP8, empty tokens, overlap, and cross-stream correctness 2 same-node GPUs; XPU PyTorch and distributed execution; no model Maintainer-triggered core; nightly
Single-instance vLLM Store E2E Actual KV save/load, transfer failure counters, and generated-output checks 1; one vLLM instance, a pinned small model, and local master Once nightly; optional manual rerun; outside PR core
Dual-GPU vLLM PD E2E Separate prefill/producer and decode/consumer roles, KV handoff over TCP, key checks, and generated-output checks 2 same-node GPUs, one per role; two vLLM instances, a pinned small model, and local master Once nightly; optional manual rerun; outside PR core

C++ and Python identify implementation languages, not test levels. Store roundtrips and multi-process EP checks are component integration tests. Only the two vLLM scenarios load an actual model; these are functional smoke tests, not model accuracy benchmarks.

Existing Entry Points

Resource Proposals

Proposal 1: Fixed Runners

  • Capacity: start with two two-GPU runners (four B60s total); expand toward eight GPUs with single-, dual-, or four-GPU runners as demand grows.
  • Execution: each job owns a full runner and runs its phases sequentially. The initial pool supports two concurrent jobs; each pair is on the same host, with non-overlapping devices and per-runner isolation/locking.
  • Trade-off: close to Mooncake's existing approach and simpler to operate, but single-GPU phases leave capacity idle and busy runners still cause queueing.

Proposal 2: Kubernetes Allocation

  • Capacity: an eight-B60 pool. Actions submits Jobs through kubectl; the Kubernetes scheduler and Intel GPU device plugin allocate devices.
  • Execution: core and PD request two same-node GPUs; independent single-GPU jobs request one. The combined nightly application job reserves two GPUs throughout.
  • Trade-off: flexible allocation reduces fixed-slot waste but adds cluster, permissions, result handoff, and cleanup management. Validate B60 isolation, IPC/P2P, io_uring, and model storage; larger jobs still require suitable node topology and validated tests.
  • Scheduling: exclusive allocation prevents overlapping device ownership, not queueing. Capacity reservations or queue policies may still be needed to protect dual-GPU availability; disable preemption if running jobs must not be displaced.

Triggers and Execution

Relevant PR update / main push
  -> CPU runner: build, host-side UT, and wheel checks

Maintainer adds run-xpu-ci to a PR / manually selects a revision
  -> Resolve the exact SHA and prepare matching wheel/native test artifacts
  -> Two-GPU allocation: C++ -> Store Python -> native EP

Nightly, once hardware coverage is enabled
  -> Expanded core job
  -> Separate application job: single-instance Store -> dual-GPU PD

Core hardware testing is maintainer-triggered rather than automatic on every PR push. Results apply only to the tested head SHA; a new head needs a fresh trigger and result. Both E2E scenarios are required in an enabled nightly hardware run and may also be rerun manually.

Builds do not reserve GPUs. Hardware jobs consume matching artifacts for the selected SHA, including native C++ test binaries that are not in the wheel. Pin the oneAPI/runtime, PyTorch, vLLM image, and model versions.

Correctness and Acceptance Criteria

  • Verify actual B60 device identity/count and GPU execution; CPU fallback does not count as XPU coverage.
  • Explicitly exclude direct RDMA cases. Force TCP and prevent IPC shortcuts when testing TCP staging.
  • Require the selected test identities/counts and reject unexpected skips, missing tests, and zero-test runs.
  • Bind results to the tested SHA and record artifact hashes and runtime/image versions. Do not silently substitute another artifact.
  • Propagate test failures, cancellations, and missing required results to the CI gate; retain logs and test reports even on failure.
  • Isolate execution of PR code, clean up only the current job's resources, and verify resource release before reuse.
  • Measure B60 memory use, execution time, and queue time during rollout. Now no claim measured runtime or concurrency guarantees.

Rollout

  1. Add hosted builds, wheel checks, and native C++ test artifacts.
  2. Provision the selected resource model and validate device isolation.
  3. Enable maintainer-triggered core and nightly core with strict result gating.
  4. Enable both minimal nightly E2E scenarios.
  5. Use utilization and queue measurements to refine allocation and expansion. Any future four-GPU or larger suite requires a separate coverage decision.

Comparison with Current ROCm CI

This compares the proposed XPU design with repository behavior at the assessment baseline, not two already implemented CI systems.

Area Current ROCm CI Proposed XPU CI
Build and packaging Hosted wheel build without GPUs Same separation, plus native C++ test artifacts
C++ hardware coverage HIP transport and ROCm platform cases exist, but the dedicated build sets BUILD_UNIT_TESTS=OFF and does not execute them Include platform, transport, engine, IPC, and Store in core
Low-level Python coverage A HIP transfer test exists but is not directly invoked by the ROCm hardware workflow Include Store tensor/API and synthetic native EP checks; add focused UT as needed
Application coverage Primarily SGLang HiCache, SGLang PD/EPD, and vLLM PD Only single-instance vLLM Store and dual-GPU PD, nightly
Model dependency Hardware application suite needs model environments Core has no model dependency; only E2E loads models
Allocation Fixed core-4gpu profile; scripts set GPU indices/TP and serially reuse allocations Proposal 1: fixed runner pairs, later mixed sizes; Proposal 2: Kubernetes job-level requests
Triggers PR run-e2e-ci label/manual entry; configurable nightly hardware execution Proposed run-xpu-ci label/manual core; nightly core and both E2E scenarios

Regression Policy

When an application E2E test exposes a defect, add a focused C++ or Python regression test at the lowest practical layer rather than relying only on another expensive model scenario to prevent recurrence.

Before submitting a new issue...

  • Make sure you already searched for relevant issues and read the documentation
Originally created by @VincyZhang on GitHub (Oct 28, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/5036 ## Summary Introduce XPU CI with broad C++ and Python unit/component coverage and two model-level E2E smoke tests. Keep model execution out of PR core testing; run both application scenarios nightly. ## Motivation and Goals Mooncake already has XPU platform, transport, Store, IPC, and native EP tests, but no dedicated XPU Actions workflow at the assessment baseline. Existing tests should become repeatable CI coverage rather than relying primarily on expensive application runs. The goals are to: - Detect most regressions with focused C++ and Python tests. - Validate Store and EP component boundaries without loading a model. - Retain minimal real-model coverage for KV save/load and disaggregated prefill/decode (PD). - Separate CPU-only builds from GPU execution. - Make device ownership, tested revisions, failures, and coverage explicit. Initial non-goals are RDMA, cross-node validation, GDS, EP with more than two ranks, broad model matrices, and model-quality or performance certification. There is no RDMA hardware available. This RFC also presents two resource proposals: 1. **Fixed runners:** start with two two-cards runners, four GPUs in total, then expand to eight GPUs with runner sizes selected according to demand. 2. **Kubernetes allocation:** manage an eight-GPU pool and allocate devices dynamically to test jobs submitted from GitHub Actions through `kubectl`. The resource choice does not change the initial test scope or trigger policy. ## Test Coverage and Frequency GPU counts below describe test requirements, not necessarily scheduler reservations. Existing C++ binaries mix single- and dual-GPU cases, so the core job requests two GPUs and executes its phases sequentially. | Test category | Coverage | GPUs and supporting resources | Expected frequency | |---|---|---|---| | Host-side C++ / Python UT | Device-independent configuration, routing policy, parameter validation, and error handling; reuse or add focused cases as needed | 0; hosted CPU runner | Every relevant PR update and main push; nightly | | XPU build and wheel checks | Compilation, packaging, clean-environment installation, native extension loading, and packaged master startup | 0; oneAPI build environment and matching runtime | Every relevant PR update and main push; nightly | | C++ platform / transport | USM allocation and pointer classification, H2D/D2H/D2D, offset access, registration/lifetimes, and cross-device copies | 1 for basic cases, 2 for cross-device cases | Maintainer-triggered core; nightly | | C++ engine / IPC | Routing, TCP staging, IPC, failure fallback, io_uring file I/O, and real cross-process behavior | 1 or 2 depending on the case; IPC permissions and io_uring support | Maintainer-triggered core; nightly | | C++ Store component integration | Foreign USM buffer put/get and byte-for-byte roundtrip validation | 1; lightweight local master | Maintainer-triggered core; nightly | | Python UT / Store component integration | Binding parameters, exceptions, return values, tensor/buffer lifetimes, XPU tensor registration, and Store put/get | 1 for device cases; XPU PyTorch and local master for Store | Maintainer-triggered core; nightly | | Python EP component integration | Native dispatch/combine with synthetic activations and routing, BF16/FP8, empty tokens, overlap, and cross-stream correctness | 2 same-node GPUs; XPU PyTorch and distributed execution; no model | Maintainer-triggered core; nightly | | Single-instance vLLM Store E2E | Actual KV save/load, transfer failure counters, and generated-output checks | 1; one vLLM instance, a pinned small model, and local master | Once nightly; optional manual rerun; outside PR core | | Dual-GPU vLLM PD E2E | Separate prefill/producer and decode/consumer roles, KV handoff over TCP, key checks, and generated-output checks | 2 same-node GPUs, one per role; two vLLM instances, a pinned small model, and local master | Once nightly; optional manual rerun; outside PR core | C++ and Python identify implementation languages, not test levels. Store roundtrips and multi-process EP checks are component integration tests. Only the two vLLM scenarios load an actual model; these are functional smoke tests, not model accuracy benchmarks. ### Existing Entry Points - C++: [TENT XPU test registration](https://github.com/kvcache-ai/Mooncake/blob/main/mooncake-transfer-engine/tent/tests/CMakeLists.txt) and `RealClientTest.XpuDeviceBufferPutGetRoundTrips` in the [Store client tests](https://github.com/kvcache-ai/Mooncake/blob/main/mooncake-store/tests/pybind_client_test.cpp). - Python: `test_xpu_local_copy_paths` in the [Store tensor tests](https://github.com/kvcache-ai/Mooncake/blob/main/mooncake-wheel/tests/test_put_get_tensor.py) and the [native EP correctness script](https://github.com/kvcache-ai/Mooncake/blob/main/python/tests/ep/test_mooncake_ep_xpu.py). - Applications: [single-instance Store smoke](https://github.com/kvcache-ai/Mooncake/blob/main/scripts/xpu/vllm_store_smoke.py) and [dual-GPU PD smoke](https://github.com/kvcache-ai/Mooncake/blob/main/scripts/xpu/vllm_store_pd_smoke.py). ## Resource Proposals ### Proposal 1: Fixed Runners - **Capacity:** start with two two-GPU runners (four B60s total); expand toward eight GPUs with single-, dual-, or four-GPU runners as demand grows. - **Execution:** each job owns a full runner and runs its phases sequentially. The initial pool supports two concurrent jobs; each pair is on the same host, with non-overlapping devices and per-runner isolation/locking. - **Trade-off:** close to Mooncake's existing approach and simpler to operate, but single-GPU phases leave capacity idle and busy runners still cause queueing. ### Proposal 2: Kubernetes Allocation - **Capacity:** an eight-B60 pool. Actions submits Jobs through `kubectl`; the Kubernetes scheduler and Intel GPU device plugin allocate devices. - **Execution:** core and PD request two same-node GPUs; independent single-GPU jobs request one. The combined nightly application job reserves two GPUs throughout. - **Trade-off:** flexible allocation reduces fixed-slot waste but adds cluster, permissions, result handoff, and cleanup management. Validate B60 isolation, IPC/P2P, io_uring, and model storage; larger jobs still require suitable node topology and validated tests. - **Scheduling:** exclusive allocation prevents overlapping device ownership, not queueing. Capacity reservations or queue policies may still be needed to protect dual-GPU availability; disable preemption if running jobs must not be displaced. ## Triggers and Execution ```text Relevant PR update / main push -> CPU runner: build, host-side UT, and wheel checks Maintainer adds run-xpu-ci to a PR / manually selects a revision -> Resolve the exact SHA and prepare matching wheel/native test artifacts -> Two-GPU allocation: C++ -> Store Python -> native EP Nightly, once hardware coverage is enabled -> Expanded core job -> Separate application job: single-instance Store -> dual-GPU PD ``` Core hardware testing is maintainer-triggered rather than automatic on every PR push. Results apply only to the tested head SHA; a new head needs a fresh trigger and result. Both E2E scenarios are required in an enabled nightly hardware run and may also be rerun manually. Builds do not reserve GPUs. Hardware jobs consume matching artifacts for the selected SHA, including native C++ test binaries that are not in the wheel. Pin the oneAPI/runtime, PyTorch, vLLM image, and model versions. ## Correctness and Acceptance Criteria - Verify actual B60 device identity/count and GPU execution; CPU fallback does not count as XPU coverage. - Explicitly exclude direct RDMA cases. Force TCP and prevent IPC shortcuts when testing TCP staging. - Require the selected test identities/counts and reject unexpected skips, missing tests, and zero-test runs. - Bind results to the tested SHA and record artifact hashes and runtime/image versions. Do not silently substitute another artifact. - Propagate test failures, cancellations, and missing required results to the CI gate; retain logs and test reports even on failure. - Isolate execution of PR code, clean up only the current job's resources, and verify resource release before reuse. - Measure B60 memory use, execution time, and queue time during rollout. Now no claim measured runtime or concurrency guarantees. ## Rollout 1. Add hosted builds, wheel checks, and native C++ test artifacts. 2. Provision the selected resource model and validate device isolation. 3. Enable maintainer-triggered core and nightly core with strict result gating. 4. Enable both minimal nightly E2E scenarios. 5. Use utilization and queue measurements to refine allocation and expansion. Any future four-GPU or larger suite requires a separate coverage decision. ## Comparison with Current ROCm CI This compares the proposed XPU design with repository behavior at the assessment baseline, not two already implemented CI systems. | Area | Current ROCm CI | Proposed XPU CI | |---|---|---| | Build and packaging | Hosted wheel build without GPUs | Same separation, plus native C++ test artifacts | | C++ hardware coverage | HIP transport and ROCm platform cases exist, but the dedicated build sets `BUILD_UNIT_TESTS=OFF` and does not execute them | Include platform, transport, engine, IPC, and Store in core | | Low-level Python coverage | A HIP transfer test exists but is not directly invoked by the ROCm hardware workflow | Include Store tensor/API and synthetic native EP checks; add focused UT as needed | | Application coverage | Primarily SGLang HiCache, SGLang PD/EPD, and vLLM PD | Only single-instance vLLM Store and dual-GPU PD, nightly | | Model dependency | Hardware application suite needs model environments | Core has no model dependency; only E2E loads models | | Allocation | Fixed `core-4gpu` profile; scripts set GPU indices/TP and serially reuse allocations | Proposal 1: fixed runner pairs, later mixed sizes; Proposal 2: Kubernetes job-level requests | | Triggers | PR `run-e2e-ci` label/manual entry; configurable nightly hardware execution | Proposed `run-xpu-ci` label/manual core; nightly core and both E2E scenarios | ## Regression Policy **When an application E2E test exposes a defect, add a focused C++ or Python regression test at the lowest practical layer rather than relying only on another expensive model scenario to prevent recurrence.** ### Before submitting a new issue... - [x] Make sure you already searched for relevant issues and read the [documentation](https://kvcache-ai.github.io/Mooncake/)
Author
Owner

@github-actions[bot] commented on GitHub (Oct 28, 2026):

Thanks for opening this issue, @VincyZhang!

Field Value
Issue #5036
GitHub user ID 92291705
Reporter @VincyZhang

A maintainer will triage this when possible. To help us respond faster, please include:

  • Mooncake version or commit SHA
  • Environment (OS, CUDA/driver, RDMA stack if relevant)
  • Steps to reproduce and expected vs. actual behavior

Useful links: Documentation · Contributing guide

This message was posted automatically by the issue bot.

<!-- gh-comment-id:6464988799 --> @github-actions[bot] commented on GitHub (Oct 28, 2026): Thanks for opening this issue, @VincyZhang! | Field | Value | |-------|-------| | **Issue** | #5036 | | **GitHub user ID** | `92291705` | | **Reporter** | @VincyZhang | A maintainer will triage this when possible. To help us respond faster, please include: - Mooncake version or commit SHA - Environment (OS, CUDA/driver, RDMA stack if relevant) - Steps to reproduce and expected vs. actual behavior Useful links: [Documentation](https://kvcache-ai.github.io/Mooncake/) · [Contributing guide](https://github.com/kvcache-ai/Mooncake/blob/main/CONTRIBUTING.md) > This message was posted automatically by the issue bot.
Author
Owner

@staryxchen commented on GitHub (Oct 29, 2026):

Could we add a dedicated XPU wheel, similar to the existing EFA/AMD platform-specific wheels, to simplify installation and dependency management?

<!-- gh-comment-id:6486384377 --> @staryxchen commented on GitHub (Oct 29, 2026): Could we add a dedicated XPU wheel, similar to the existing EFA/AMD platform-specific wheels, to simplify installation and dependency management?
Author
Owner

@VincyZhang commented on GitHub (Oct 29, 2026):

3. Could we add a dedicated XPU wheel, similar to the existing EFA/AMD platform-specific wheels, to simplify installation and dependency management?

Exactly, that's what we expected too. And we also want to follow the same release workflow after we asure the quality of wheel like mooncake-transfer-engine-xpu.

<!-- gh-comment-id:6489667896 --> @VincyZhang commented on GitHub (Oct 29, 2026): > 3\. Could we add a dedicated XPU wheel, similar to the existing EFA/AMD platform-specific wheels, to simplify installation and dependency management? Exactly, that's what we expected too. And we also want to follow the same release workflow after we asure the quality of wheel like `mooncake-transfer-engine-xpu`.
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
kvcache-ai/Mooncake#1274
No description provided.