[GH-ISSUE #4565] [RFC]: [Store] NoF Storage Device Control Plane, Phase 2: Fault Isolation and Device-Level Graceful Drain #1164

New issue

Open

opened 2026-10-29 21:23:30 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented 2026-10-29 21:23:30 +00:00

Owner

Copy link

Originally created by @zxpdemonio on GitHub (Oct 10, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/4565

Changes proposed

Summary

RFC #4561 proposed a NoF storage device control plane for Mooncake Store, and Phase 1 landed in PR #4560: device metadata, a derived health state machine (UNKNOWN / HEALTHY / DEGRADED / FAILED / UNMOUNTING), a maintenance planner, and read-only admin observability.

Phase 1 made device health visible. It deliberately left the reaction of the master unchanged: after nof_heartbeat_failures_threshold consecutive probe failures the heartbeat thread unmounts the segment, and nothing happens in between. There is also no operator-facing way to stop writing to a device, or to migrate its contents off it.

This RFC covers Phase 2, which turns that health signal into action:

1. Fault isolation (cordon / uncordon) - fence a device against new allocations without unmounting it, and return it to service automatically once probes recover.
2. Device-level graceful drain - bind the existing segment DrainJob machinery to a device identity so that maintenance_plan output can actually drive evacuation.
3. Admin write endpoints - isolate, unisolate, drain, drain_status.

Reference implementation: #4566 (stacked on #4560). Related: #4431 (cross-client move/migration control plane), which owns the DrainJob lifecycle this phase builds on.

---

Motivation

The window between DEGRADED and unmount is unused

With threshold N and probe interval I, Phase 1 reports DEGRADED after N/2 failures, while the alive timeout unmounts at roughly N * I. That window currently exists only in the reported health - the master takes no action inside it. For devices backing offloaded objects this is the worst of both outcomes:

• New evictions keep landing on a device that is already failing probes, so the objects written during the window are precisely the ones most likely to be lost when the segment is finally unmounted.
• The unmount itself is a cliff. The segment disappears, its allocator is torn down, and objects whose only remaining replica was on that device become unrecoverable until they are re-fetched or re-evicted.

No operator action short of unmount

Today there is no way to express "stop writing to this target, keep its data readable, and let me migrate it". Node maintenance, NVMe target firmware upgrades, and predicted disk failures all need exactly that, and all of them currently have to be done by pulling the client out from under the master.

---

Proposed design

1. Fence state machine

Phase 2 adds one reversible action between HEALTHY and the terminal unmount:

   HEALTHY
      |  consecutive probe failures >= N/2
      v
   DEGRADED ---- auto-isolate (fence) ----+
      |                                   |
      |  probe succeeds                   |  probes keep failing
      v                                   v
   HEALTHY  <---- auto-unisolate       FAILED
                                          |  alive timeout (N * I)
                                          v
                                       UNMOUNT   (unchanged terminal rule)

Two properties matter:

• Fencing is not terminal. IsolateSegment() detaches the allocator registration of the segment and moves it to DRAINING. The segment stays mounted, its buf_allocator is untouched, and no read path in the master gates on SegmentStatus::OK, so replicas already on the device remain readable and migratable.
• The alive-timeout unmount rule is unchanged. Fencing never weakens or delays it. A device that never recovers is still unmounted at exactly the same point as before this change.

Note that fencing at DEGRADED rather than at FAILED is what gives the mechanism a window to act: FAILED is derived from the same threshold as the alive timeout, so a device that is only fenced at FAILED is unmounted in the same heartbeat iteration.

2. Fenced devices keep being probed

The heartbeat probe set is defined by one predicate, shared with on-demand probing:

// OK: serving allocations. DRAINING + isolated: fenced for maintenance.
// UNMOUNTING / GRACEFULLY_UNMOUNTING / DRAINED: leaving the cluster.
static bool IsNoFSegmentProbeCandidate(SegmentStatus status, bool isolated);

This is load-bearing for two reasons:

• Auto-recovery is only possible if a fenced device is still probed. On the first successful probe the heartbeat thread clears the fence and re-registers the allocator, unless the cordon is manual or a drain job is still active.
• Without it, a cordoned device would drop out of the probe set, its heartbeat accounting would be discarded, and its reported health would silently collapse back to UNKNOWN - exactly when an operator is watching it.

Manual cordons are never lifted automatically; only entries in auto_isolated_devices_ are.

3. Device-level drain

DrainStorageDevice(device_id, target_segments, max_concurrency):

1. Resolves the device to its mounted NoF segment and rejects devices that are already unmounting or already draining.
2. Cordons the device (manual isolation) so that no new allocation lands mid-migration.
3. Delegates to the existing CreateDrainJob() over that segment. No second migration lifecycle is introduced.
4. Records device_id -> job_id so device listings and maintenance_plan can report drain progress.

The whole operation is serialised by a dedicated mutex, so two concurrent admin requests cannot both pass the "already draining" check and orphan a job.

maintenance_plan integration:

Condition	Plan bucket	Reason
health DEGRADED / FAILED	recovery_candidates	probe_degraded / probe_failed
isolated, no active drain	recovery_candidates	device_isolated
active drain job	gc_candidates	draining
drain finished, segment DRAINED	gc_candidates	unmounting

4. Isolation and drain must not fight over segment status

Both fencing and draining are expressed as SegmentStatus::DRAINING, and the existing drain cancel and drain failure paths restore OK when they give up. Left alone, cancelling a device drain would silently put a cordoned device back into the allocation path while it was still reported as isolated.

Rather than patching three call sites, the invariant is enforced where it belongs: ScopedNoFSegmentAccess::SetSegmentStatusByName() refuses to set OK on a segment that has no allocator registration. Returning a fenced device to service is the job of UnisolateSegment(), because that is the only path that re-registers the allocator. This mirrors what the DRAM-side SetSegmentStatus() already does, and leaves non-isolated drains behaving exactly as before.

5. Admin REST endpoints

Endpoint	Method	Description
/api/v1/devices/isolate?device_id=<id>	POST	Cordon a device against new allocations
/api/v1/devices/unisolate?device_id=<id>	POST	Lift a cordon (refused while a drain job is active)
/api/v1/devices/drain	POST	Body: {device_id, target_segments?, max_concurrency?}; returns job_id
/api/v1/devices/drain_status?device_id=<id>	GET	Drain job status for a device

GET /api/v1/devices gains isolated, draining and draining_job_id alongside the Phase 1 fields.

6. Prometheus metrics export

To support cluster-wide monitoring and alerting without polling /api/v1/devices, the master exports device-level health, isolation, and drain metrics on the existing /metrics endpoint:

Metric Name	Type	Labels	Description
mooncake_master_storage_devices_total	Gauge	health	Current count of storage devices by health state (healthy, degraded, failed, unknown, unmounting)
mooncake_master_storage_devices_isolated_total	Gauge	None	Total number of storage devices currently fenced from new allocations
mooncake_master_storage_devices_auto_isolated_total	Counter	None	Cumulative count of automatic device fencing events triggered by probe degradation
mooncake_master_storage_devices_draining_total	Gauge	None	Current count of storage devices undergoing graceful drain
mooncake_master_storage_device_recovery_candidates	Gauge	None	Current count of devices recommended for data recovery / evacuation
mooncake_master_storage_device_gc_candidates	Gauge	None	Current count of devices recommended for garbage collection / space reclaim

These metrics refresh on scrape during the master storage telemetry pass, ensuring zero drift between REST query results and Prometheus timeseries.

7. Configuration

--nof_auto_isolate_degraded_devices   (default: true)

Automatic fencing is an operator-facing policy, so it has an off switch, plumbed through the config file, the gflag, and MasterServiceConfig::builder(). Turning it off restores Phase 1 behaviour exactly: health is still reported, and the alive-timeout unmount still applies.

---

Compatibility and impact

• Additive API. All endpoints and C++ methods are new. QueryDrainJob() became const, which is source-compatible for callers.
• No hot-path cost. Fencing touches the allocator registration and the heartbeat thread only. Nothing is added to the read or write data path.
• Non-NoF builds unaffected. USE_NOF=OFF still builds clean.
• Behavioural change to call out: with the default configuration, a device that reaches DEGRADED now stops receiving new allocations until it recovers or is unmounted. This is strictly narrower than the existing outcome for the same device (unmount), it is reversible, and it can be disabled with the flag above.
• Blast radius of a master-side network blip: every device would be fenced rather than every device being unmounted. Fenced devices keep serving reads and return to service on the first successful probe, which is a smaller failure than the pre-Phase-2 behaviour.

---

Testing

Unit and integration tests on a USE_NOF=ON build:

• storage_device_metadata_test - pure-function coverage of health derivation, schedulability (including the isolation and drain axes), and maintenance-plan reasons.
• nof_heartbeat_test - live heartbeat coverage: DEGRADED fencing and auto-recovery, manual cordon keeps being probed and reported, cancelled device drain keeps the device fenced, the configuration off-switch, device drain job creation and maintenance_plan reporting, and the unchanged alive-timeout unmount.
• master_admin_server_test - REST contract tests for the four new endpoints plus the Phase 1 device endpoints.
• segment_test / segment_pool_test - existing segment lifecycle coverage, unchanged by the new status guard.

---

Follow-ups (not in this phase)

• Phase 3 from #4561: feed device health and active IO load into client-side replica read balancing and write offload throttling.
• Hysteresis on the fence (for example, require K consecutive successful probes before uncordoning) to avoid flapping a device that is intermittently reachable.
• A maintenance_plan-driven controller that issues drains automatically instead of leaving the plan to an external operator.

---

Before submitting a new issue...

• Make sure you already searched for relevant issues and read the documentation

Originally created by @zxpdemonio on GitHub (Oct 10, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/4565 ### Changes proposed ## Summary RFC #4561 proposed a NoF storage device control plane for Mooncake Store, and Phase 1 landed in PR #4560: device metadata, a derived health state machine (UNKNOWN / HEALTHY / DEGRADED / FAILED / UNMOUNTING), a maintenance planner, and read-only admin observability. Phase 1 made device health **visible**. It deliberately left the reaction of the master unchanged: after `nof_heartbeat_failures_threshold` consecutive probe failures the heartbeat thread unmounts the segment, and nothing happens in between. There is also no operator-facing way to stop writing to a device, or to migrate its contents off it. This RFC covers **Phase 2**, which turns that health signal into action: 1. **Fault isolation (cordon / uncordon)** - fence a device against new allocations without unmounting it, and return it to service automatically once probes recover. 2. **Device-level graceful drain** - bind the existing segment `DrainJob` machinery to a device identity so that `maintenance_plan` output can actually drive evacuation. 3. **Admin write endpoints** - `isolate`, `unisolate`, `drain`, `drain_status`. Reference implementation: #4566 (stacked on #4560). Related: #4431 (cross-client move/migration control plane), which owns the `DrainJob` lifecycle this phase builds on. --- ## Motivation ### The window between DEGRADED and unmount is unused With threshold `N` and probe interval `I`, Phase 1 reports DEGRADED after `N/2` failures, while the alive timeout unmounts at roughly `N * I`. That window currently exists only in the reported health - the master takes no action inside it. For devices backing offloaded objects this is the worst of both outcomes: - New evictions keep landing on a device that is already failing probes, so the objects written during the window are precisely the ones most likely to be lost when the segment is finally unmounted. - The unmount itself is a cliff. The segment disappears, its allocator is torn down, and objects whose only remaining replica was on that device become unrecoverable until they are re-fetched or re-evicted. ### No operator action short of unmount Today there is no way to express "stop writing to this target, keep its data readable, and let me migrate it". Node maintenance, NVMe target firmware upgrades, and predicted disk failures all need exactly that, and all of them currently have to be done by pulling the client out from under the master. --- ## Proposed design ### 1. Fence state machine Phase 2 adds one reversible action between HEALTHY and the terminal unmount: ```text HEALTHY | consecutive probe failures >= N/2 v DEGRADED ---- auto-isolate (fence) ----+ | | | probe succeeds | probes keep failing v v HEALTHY <---- auto-unisolate FAILED | alive timeout (N * I) v UNMOUNT (unchanged terminal rule) ``` Two properties matter: - **Fencing is not terminal.** `IsolateSegment()` detaches the allocator registration of the segment and moves it to `DRAINING`. The segment stays mounted, its `buf_allocator` is untouched, and no read path in the master gates on `SegmentStatus::OK`, so replicas already on the device remain readable and migratable. - **The alive-timeout unmount rule is unchanged.** Fencing never weakens or delays it. A device that never recovers is still unmounted at exactly the same point as before this change. Note that fencing at DEGRADED rather than at FAILED is what gives the mechanism a window to act: FAILED is derived from the same threshold as the alive timeout, so a device that is only fenced at FAILED is unmounted in the same heartbeat iteration. ### 2. Fenced devices keep being probed The heartbeat probe set is defined by one predicate, shared with on-demand probing: ```cpp // OK: serving allocations. DRAINING + isolated: fenced for maintenance. // UNMOUNTING / GRACEFULLY_UNMOUNTING / DRAINED: leaving the cluster. static bool IsNoFSegmentProbeCandidate(SegmentStatus status, bool isolated); ``` This is load-bearing for two reasons: - Auto-recovery is only possible if a fenced device is still probed. On the first successful probe the heartbeat thread clears the fence and re-registers the allocator, unless the cordon is manual or a drain job is still active. - Without it, a cordoned device would drop out of the probe set, its heartbeat accounting would be discarded, and its reported health would silently collapse back to UNKNOWN - exactly when an operator is watching it. Manual cordons are never lifted automatically; only entries in `auto_isolated_devices_` are. ### 3. Device-level drain `DrainStorageDevice(device_id, target_segments, max_concurrency)`: 1. Resolves the device to its mounted NoF segment and rejects devices that are already unmounting or already draining. 2. Cordons the device (manual isolation) so that no new allocation lands mid-migration. 3. Delegates to the existing `CreateDrainJob()` over that segment. No second migration lifecycle is introduced. 4. Records `device_id -> job_id` so device listings and `maintenance_plan` can report drain progress. The whole operation is serialised by a dedicated mutex, so two concurrent admin requests cannot both pass the "already draining" check and orphan a job. `maintenance_plan` integration: | Condition | Plan bucket | Reason | | --- | --- | --- | | health DEGRADED / FAILED | recovery_candidates | `probe_degraded` / `probe_failed` | | isolated, no active drain | recovery_candidates | `device_isolated` | | active drain job | gc_candidates | `draining` | | drain finished, segment DRAINED | gc_candidates | `unmounting` | ### 4. Isolation and drain must not fight over segment status Both fencing and draining are expressed as `SegmentStatus::DRAINING`, and the existing drain cancel and drain failure paths restore `OK` when they give up. Left alone, cancelling a device drain would silently put a cordoned device back into the allocation path while it was still reported as isolated. Rather than patching three call sites, the invariant is enforced where it belongs: `ScopedNoFSegmentAccess::SetSegmentStatusByName()` refuses to set `OK` on a segment that has no allocator registration. Returning a fenced device to service is the job of `UnisolateSegment()`, because that is the only path that re-registers the allocator. This mirrors what the DRAM-side `SetSegmentStatus()` already does, and leaves non-isolated drains behaving exactly as before. ### 5. Admin REST endpoints | Endpoint | Method | Description | | --- | --- | --- | | `/api/v1/devices/isolate?device_id=<id>` | POST | Cordon a device against new allocations | | `/api/v1/devices/unisolate?device_id=<id>` | POST | Lift a cordon (refused while a drain job is active) | | `/api/v1/devices/drain` | POST | Body: `{device_id, target_segments?, max_concurrency?}`; returns `job_id` | | `/api/v1/devices/drain_status?device_id=<id>` | GET | Drain job status for a device | `GET /api/v1/devices` gains `isolated`, `draining` and `draining_job_id` alongside the Phase 1 fields. ### 6. Prometheus metrics export To support cluster-wide monitoring and alerting without polling `/api/v1/devices`, the master exports device-level health, isolation, and drain metrics on the existing `/metrics` endpoint: | Metric Name | Type | Labels | Description | | --- | --- | --- | --- | | `mooncake_master_storage_devices_total` | Gauge | `health` | Current count of storage devices by health state (`healthy`, `degraded`, `failed`, `unknown`, `unmounting`) | | `mooncake_master_storage_devices_isolated_total` | Gauge | None | Total number of storage devices currently fenced from new allocations | | `mooncake_master_storage_devices_auto_isolated_total` | Counter | None | Cumulative count of automatic device fencing events triggered by probe degradation | | `mooncake_master_storage_devices_draining_total` | Gauge | None | Current count of storage devices undergoing graceful drain | | `mooncake_master_storage_device_recovery_candidates` | Gauge | None | Current count of devices recommended for data recovery / evacuation | | `mooncake_master_storage_device_gc_candidates` | Gauge | None | Current count of devices recommended for garbage collection / space reclaim | These metrics refresh on scrape during the master storage telemetry pass, ensuring zero drift between REST query results and Prometheus timeseries. ### 7. Configuration ```text --nof_auto_isolate_degraded_devices (default: true) ``` Automatic fencing is an operator-facing policy, so it has an off switch, plumbed through the config file, the gflag, and `MasterServiceConfig::builder()`. Turning it off restores Phase 1 behaviour exactly: health is still reported, and the alive-timeout unmount still applies. --- ## Compatibility and impact - **Additive API.** All endpoints and C++ methods are new. `QueryDrainJob()` became `const`, which is source-compatible for callers. - **No hot-path cost.** Fencing touches the allocator registration and the heartbeat thread only. Nothing is added to the read or write data path. - **Non-NoF builds unaffected.** `USE_NOF=OFF` still builds clean. - **Behavioural change to call out:** with the default configuration, a device that reaches DEGRADED now stops receiving new allocations until it recovers or is unmounted. This is strictly narrower than the existing outcome for the same device (unmount), it is reversible, and it can be disabled with the flag above. - **Blast radius of a master-side network blip:** every device would be fenced rather than every device being unmounted. Fenced devices keep serving reads and return to service on the first successful probe, which is a smaller failure than the pre-Phase-2 behaviour. --- ## Testing Unit and integration tests on a `USE_NOF=ON` build: - `storage_device_metadata_test` - pure-function coverage of health derivation, schedulability (including the isolation and drain axes), and maintenance-plan reasons. - `nof_heartbeat_test` - live heartbeat coverage: DEGRADED fencing and auto-recovery, manual cordon keeps being probed and reported, cancelled device drain keeps the device fenced, the configuration off-switch, device drain job creation and `maintenance_plan` reporting, and the unchanged alive-timeout unmount. - `master_admin_server_test` - REST contract tests for the four new endpoints plus the Phase 1 device endpoints. - `segment_test` / `segment_pool_test` - existing segment lifecycle coverage, unchanged by the new status guard. --- ## Follow-ups (not in this phase) - Phase 3 from #4561: feed device health and active IO load into client-side replica read balancing and write offload throttling. - Hysteresis on the fence (for example, require K consecutive successful probes before uncordoning) to avoid flapping a device that is intermittently reachable. - A `maintenance_plan`-driven controller that issues drains automatically instead of leaving the plan to an external operator. --- ### Before submitting a new issue... - [x] Make sure you already searched for relevant issues and read the [documentation](https://kvcache-ai.github.io/Mooncake/)

bsakaguchi added the

Store

RFC

labels 2026-10-29 21:23:30 +00:00

bsakaguchi commented 2026-10-29 21:23:31 +00:00

Author

Owner

Copy link

@github-actions[bot] commented on GitHub (Oct 10, 2026):

Thanks for opening this issue, @zxpdemonio!

Field	Value
Issue	#4565
GitHub user ID	16206312
Reporter	@zxpdemonio

A maintainer will triage this when possible. To help us respond faster, please include:

• Mooncake version or commit SHA
• Environment (OS, CUDA/driver, RDMA stack if relevant)
• Steps to reproduce and expected vs. actual behavior

Useful links: Documentation · Contributing guide

This message was posted automatically by the issue bot.

<!-- gh-comment-id:6097642047 --> @github-actions[bot] commented on GitHub (Oct 10, 2026): Thanks for opening this issue, @zxpdemonio! | Field | Value | |-------|-------| | **Issue** | #4565 | | **GitHub user ID** | `16206312` | | **Reporter** | @zxpdemonio | A maintainer will triage this when possible. To help us respond faster, please include: - Mooncake version or commit SHA - Environment (OS, CUDA/driver, RDMA stack if relevant) - Steps to reproduce and expected vs. actual behavior Useful links: [Documentation](https://kvcache-ai.github.io/Mooncake/) · [Contributing guide](https://github.com/kvcache-ai/Mooncake/blob/main/CONTRIBUTING.md) > This message was posted automatically by the issue bot.

bsakaguchi referenced this issue 2026-10-29 21:32:06 +00:00

[PR #1099] [MERGED] Add a CI test for Mooncake EP Backend (CPU only) #2011

bsakaguchi referenced this issue 2026-10-29 21:32:22 +00:00

[PR #1164] [MERGED] Fix memory leak in MooncakeBackend destructor #2046

Sign in to join this conversation.

No Branch/Tag specified

Branches Tags

main

te-coro

staryxchen-tent-capability-research

feature/musa-ep-jit

zxpdemonio/fix-tent-bootstrap-admission

add-store-rs-tone-tests

scatter-rdma-completion-pr2

P2P-Mooncake-Store

release/v0.3.14

feat/tent-rdma-adaptive-adapter

fix/ep-local-counter-gpu-scope

codex/kvcs-oplog-eviction

give-me-a-star

cruz/mooncake-kvcs-lowlevel-pr1-20261008

ylt-handshake-limits

ci/rocm-e2e-tent

sima/docs-pypi-downloads

fix-ci/hicache-mooncake-metadata-loopback

sima/cuda-driver-loading

sima/ci-performance-gate

fix/tent-rdma-unknown-topology-rail-pairing

ci/tent-e2e-dual-job

mc-patch-1

codex/p2p-merge-prerequisites

sima/store-hca-numa-placement

codex/p2p-merge-cli

codex/p2p-merge-ipv6

copilot/fix-ctests-job-failure

czh/pg-device-collectives-pr6

dependabot/cargo/mooncake-store-rs/pyo3-0.29.0

czh/pg-device-collectives-pr5

codex/segment-pool-master-cutover

sima/nightly-ci-recovery

ci/tent-params

stmatengss-patch-1

dependabot/cargo/mooncake-store-rs/rand-0.8.8

dependabot/cargo/mooncake-store-rs/opentelemetry_sdk-0.32.1

dependabot/cargo/mooncake-store-rs/xxhash-rust-0.8.16

codex/migrate-store-rest-service

fix/tone-wheel-arch-selection

czh/pg-device-collectives-pr4

czh/pg-device-collectives-pr3

ketong/fix_moe_tc

copilot/fix-test-wheel-ubuntu-job

fix/ci-wheel-torch-pg-pin

codex/migrate-structured-object-storage

codex/migrate-allocator-python

codex/migrate-vllm-python

codex/migrate-shared-segment

ci/fix_sgl_eval_install

codex/segment-pool-nof-placement-cleanup

feat/pip-native-release

copilot/bugfix-ha-standby-promotion

copilot/mooncake-store-transport-settings

copilot/fix-tensor-support-issues

copilot/optimize-rl-direction

copilot/mooncake-storeagentic-workflow

copilot/fix-client-reconnect-timeout

copilot/mooncake-error-debugging

copilot/mooncake-ipv6-support

feat/nccl-host-python-api

dependabot/go_modules/mooncake-p2p-store/src/p2pstore/google.golang.org/grpc-1.82.1

copilot/migrate-and-move-operations

copilot/migrate-and-move-operations-again

refactor/centralize-env-vars

idhanani/mooncake-retain-groups

copilot/codeownerreviewer

copilot/give-feedback-on-issue-2123

copilot/create-issue-rfc

copilot/break-cuda-monopoly-logic

copilot/compare-cudamemcpy-cu12-cu13

copilot/report-ci-error-issue

copilot/research-commit-history-analysis

copilot/explain-mooncake-store-service

copilot/analyze-project-functionality

copilot/sub-pr-1680

copilot/review-pr-1680

copilot/sub-pr-1597

copilot/fix-github-actions-job-failure

copilot/move-test-sglang-integration-job

v0.3.14

v0.3.14-pre-4

v0.3.14-pre-3

v0.3.14-pre-2

v0.3.14-pre

v0.3.14-rc1

v0.3.13.post1

v0.3.13-rc2

v0.3.13-rc1

v0.3.13

v0.3.12.post1

v0.3.12

v0.3.11-rc1

v0.3.11.post1

v0.3.11

v0.3.10.post2

v0.3.10.post1

v0.3.10

v0.3.9

v0.3.8.post1

v0.3.8

v0.3.7.post2

v0.3.7.post1

v0.3.7

v0.3.6.post1

v0.3.6

v0.3.5

v0.3.4.post2

v0.3.4.post1

v0.3.4

v0.3.3.post2

v0.3.3.post1

v0.3.3

v0.3.2.post1

v0.3.2

v0.3.1

v0.3.0

v0.3.0-beta

v0.3.0-alpha

v0.2.0

Labels

Clear labels   

ascend

  

auto-closed

  

bug

  

bug

  

CI/Build

  

Common

  

documentation

  

enhancement

  

feature request

  

good first issue

  

help wanted

  

high priority

  

MetaX

  

nightly-failure

  

pull-request

Mirrored from GitHub Pull Request

  

PyTorch Backend

  

question

  

RFC

  

Roadmap

  

stale

  

Store

  

Tests

  

Transfer Engine

No labels

ascend

auto-closed

bug

bug

CI/Build

Common

documentation

enhancement

feature request

good first issue

help wanted

high priority

MetaX

nightly-failure

pull-request

PyTorch Backend

question

RFC

Roadmap

stale

Store

Tests

Transfer Engine

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

kvcache-ai/Mooncake#1164

No description provided.