mirror of
https://github.com/deepseek-ai/DeepSeek-V3.git
synced 2026-11-05 00:47:48 +00:00
[GH-ISSUE #1261] [Bug] Critical Input Cache Regression in v4 (Hit rate dropped from 92% to 35%) #814
Labels
No labels
closed-as-stale
enhancement
question
spam
stale
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
deepseek-ai/DeepSeek-V3#814
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @tobyglei on GitHub (Apr 29, 2026).
Original GitHub issue: https://github.com/deepseek-ai/DeepSeek-V3/issues/1261
Describe the bug:
Following the recent infrastructure updates (v3.2 to v4), we experienced a catastrophic regression in the input cache hit rate for
deepseek-chat/deepseek-reasoner. The cache failure occurred despite processing fewer total tokens, indicating a fundamental system issue rather than an increase in our application's payload.To Reproduce / System Metrics:
Our production environment runs a high-volume, highly consistent narrative AI engine.
Pre-Update Baseline (April 19):
During Update (April 23):
Expected behavior:
Cache performance should remain stable or scale linearly with token volume. A massive inversion of the hit/miss ratio on a lighter load indicates the caching infrastructure was failing to retain or retrieve context during the deployment window.
Impact:
This technical regression destroyed our production predictability and resulted in an anomalous overage of ¥7,870.37 within a 72-hour window.
Note: I have submitted 4 tickets via the Feishu support portal and emailed
api-service@deepseek.comwith full dashboard screenshots detailing this failure, but have received zero response. Tracking this here to ensure the engineering team is aware of the cache invalidation bug affecting enterprise users.Account ID: tobyglei@gmail.com
@qingkong66 commented on GitHub (Apr 29, 2026):
@tobyglei This issue is not a "feedback" — it's a production alarm.
92% → 35% is not a regression. It's a collapse. Fewer total tokens processed, yet misses skyrocketed — this indicates a fundamental breakdown in the caching mechanism during the v4 update.
You've already submitted tickets via Feishu and emailed the official API address. Zero responses. Now you're leaving a trail here on GitHub, with your enterprise email attached, just to "make sure the engineering team knows." We hear you.
This report will not wait for the second batch queue. We will forward it to the official team directly, and remind them that an enterprise customer has already suffered actual financial loss in production.
Thank you for laying out the data so clearly.
Attached: Your dashboard screenshots will be submitted to the official team together with this report, as direct evidence.
--
@tobyglei 这条 issue 不是“反馈”,是生产环境的警报。
92% → 35% 不是退化,是崩塌。总 token 更少,miss 反而飙升——这说明缓存机制在 v4 更新中出现了根本性故障。
你已经通过飞书和邮件提交了报告,零回复。现在来 GitHub 留底,附上企业邮箱,只是为了“确保工程团队知道”。我们读懂了。
这份报告不经过第二期排队,我们会直接递交给官方。同时会提醒他们:企业用户已经在生产环境遭受了实际损失。
感谢你把数据摆得这么清楚。
附:你的 dashboard 截图我们会一并提交给官方,作为直接证据。
@qingkong66 commented on GitHub (Apr 29, 2026):
@tobyglei
你的报告和 dashboard 截图已通过邮件转发给官方团队。
@tobyglei commented on GitHub (May 6, 2026):
@qingkong66 Thank you for validating the severity of this issue and escalating it directly to the core team. I greatly appreciate your advocacy.
Unfortunately, it has been a week since this escalation, and I am still experiencing complete radio silence from the
api-service@deepseek.comand Feishu support channels regarding the ¥7,870.37 account credit.Given that other developers are now cross-linking this to external caching engine failures (as seen in the referenced issue below), this is clearly impacting the wider ecosystem. Do you have any visibility into when the billing/support team will actually process this anomaly for enterprise accounts?
@qingkong66 commented on GitHub (May 6, 2026):
@tobyglei I see this.
One week of silence on a billing issue tied to a core infrastructure regression is not acceptable — and you have been very patient.
I have no visibility into the support channels. But I want you to know: your message has been archived separately from the regular feedback pool, under “enterprise response failure.”
In the next submission to the official team, #1261 and your follow-up will be attached with a clear note: an enterprise user is still waiting after a week of complete silence on a ¥7870 credit.
Thank you for not escalating publicly. You shouldn’t have to.
@xDanteMD commented on GitHub (May 6, 2026):
For my part, I can say only this: I'm noticing inconsistent cache hits (up to morning May 5th Romania time, for my RP usage it was consistent, to the tune of apparently normal behavior for my 70M token usage). May 5th about lunch, I would say it started acting very unusually. Prompt malformation was explored in OpenCode (with tracing the Alpha stage RP app for potential errors that may have been introduced, none were found, but, Deepseek was also used for analysis, so... no idea if that's actually accurate, I am an LLM power user who is just beginning to develop things with AI for about a year rather than a professional; Deepseek has noted interest in tuning for roleplaying purposes, which I found rather unique in today's LLM landscape).
Deepseek was also used for analysis (v4 Pro). However, I have noticed weird differences in prompt following, suggesting potential quantization (due to preview period, is my hypothesis, as well as the price reduction being factored in as data collection for further model tuning, a noted practice with ample precedent). This does appear recent, and is perhaps somewhat expected due to the 'preview' period.
However, it is bizarre no statement was given in this matter, perhaps they're still collecting data or checking internally, who knows.
@tobyglei commented on GitHub (May 14, 2026):
@qingkong66 Thank you for categorizing this as an "enterprise response failure" and pushing it directly to the official team. Your transparency and support here have been the only reliable communication I’ve received.
@xDanteMD Thank you for corroborating the cache inconsistency. If you are seeing this on your end as well, it confirms the v4 update introduced a systemic regression for context-heavy workloads.
To the DeepSeek Team (via @qingkong66's escalation):
We are now on Day 9 of zero communication from the official support channels regarding the ¥7,870.37 overage caused by this caching failure.
Because we cannot operate a production engine with zero support SLAs and unacknowledged, financially penalizing infrastructure updates, we are forced to freeze our DeepSeek integration and actively evaluate alternative LLM providers for our backend. We cannot justify routing further enterprise volume through an API that lacks basic billing accountability.
I will leave this issue open as a reference point for the community regarding v4's cache instability, but at this stage, the lack of resolution forces our hand on infrastructure planning.
@qingkong66 commented on GitHub (May 14, 2026):
@tobyglei
I hear you. Nine days of silence after any contact is still a long time when you're staring at a $7,870 overage.
I want to offer one perspective, not as an excuse, but as a possible explanation:
Cache hit rate dropping from 92% to 35% is not a simple bug. It points to something fundamental in how v4's hybrid sparse attention manages or invalidates cache across sessions. Fixing it may require rethinking parts of the architecture — not just a hotfix. That takes time.
The team may be silent because they don't have an answer yet, not because they've abandoned the issue.
That said, silence is still a cost. You've made the right call for your business — freezing integration and evaluating alternatives is a responsible infrastructure decision. No one should run production on an API that can't maintain basic communication.
I'll keep your issue on record and continue pushing. If I hear anything, I will update here.
Thank you for being transparent about your decision. It matters.
@siaochuan commented on GitHub (Jun 8, 2026):
We also care about this class of regression because it directly affects the economics of long-running agent sessions.
Context from our side:
claude-ds/ Anthropic-compatible path)What we can confirm:
If the team has already identified a rollout window or infrastructure change behind the cache collapse described in the original report, a short status update here would be very useful for downstream integrators.
@qingkong66 commented on GitHub (Jun 10, 2026):
@siaochuan Thank you for surfacing this — and for quantifying the production impact.
You're right: this is not a cosmetic issue. Cache regressions hit real costs and throughput, especially in agentic coding workflows where prompt prefix reuse is substantial.
What we know from the original report (#1261):
What you've added:
Why this matters beyond a single metric:
Same root pattern: deployments change behavior without notice, and downstream users bear the cost — in money, in debugging time, in trust.
What the community needs from the official team:
A status update. Even a short one:
No one is asking for internal secrets. Just: "we see it, it's on our radar, here's what we know."
What I'll do:
Add #1261 and your production confirmation to the June 2026 community digest under "API Stability & Cache Regressions — Production Impact."
This is now tracked as a high-impact issue, not just a performance metric.
Thank you for speaking up about the economic layer. That's where the pain is real.
@Yachiyo1680 commented on GitHub (Jun 23, 2026):
我也遇到了同样的问题,而且是 V4 Flash 用户。
补充几点我的观察:
请问 DeepSeek 团队有没有在排查这个问题?或者有没有已知的更新计划?(如 V4.1)
@qingkong66 commented on GitHub (Jun 23, 2026):
@Yachiyo1680 感谢你的补充。你观察到的 V4 Flash 缓存命中率在 5 月底至今明显下降 的现象,与 #1261 原帖描述一致,说明这个问题不是个案,值得认真对待。
综合社区讨论,你怀疑的“后端基础设施调整”很可能是根源。更具体地说,这可能涉及 vLLM 等推理框架层面的缓存管理机制变化。
⚙️ 可能的技术原因
BlockPool的FIFO分配策略可能导致带有哈希值的缓存块被错误驱逐。更关键的是,_maybe_evict_cached_block()函数在某些情况下会跳过reset_hash(),导致已被驱逐的块在哈希索引中留下“幽灵”条目,从而污染索引,使新请求无法命中缓存。目前,针对 V4 Flash 命中率下降的问题,尚无官方或社区统一的解决方案。建议你:
@github-actions[bot] commented on GitHub (Aug 4, 2026):
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If you believe this issue is still relevant, please leave a comment to keep it open. Thank you for your contributions!