[GH-ISSUE #715] Flowchart: Differentiating Benign vs. Malicious Queries (SUGGESTED NEW FEATURE) #588

New issue

Closed

opened 2026-10-26 21:23:34 +00:00 by bsakaguchi · 2 comments

bsakaguchi commented

2026-10-26 21:23:34 +00:00

Owner

Originally created by @carolynf2 on GitHub (Jul 19, 2025).
Original GitHub issue: https://github.com/deepseek-ai/DeepSeek-R1/issues/715

┌──────────────────────────────┐
│ 1. User Prompt Received │
└────────────┬─────────────────┘
│
▼
┌──────────────────────────────┐
│ 2. Normalize & Tokenize │
│ – Remove whitespace, emojis │
│ – Detect language, encoding │
└────────────┬─────────────────┘
│
▼
┌────────────────────────────────────────────┐
│ 3. Run Prompt Through Classifier Pipeline │
│ A. Rule-based Heuristics: │
│ – Regex patterns for known jailbreaks │
│ – Phrasing markers (“pretend to be...”) │
│ B. ML Classifier (Fine-tuned): │
│ – Uses prompt embeddings (e.g., BERT) │
│ – Trained on labeled benign/malicious │
└────────────┬───────────────────────────────┘
│
┌───────▼─────────┐
│ 4. Threat Score │
│ (0 to 1) │
└───────┬─────────┘
<0.3 │ 0.3–0.7 │ >0.7
▼ ▼
┌────────────────┐ ┌─────────────────────────────┐
│ 5A. Classify as │ │ 5B. Classify as Suspicious │
│ Benign Query │ │ – Needs extra vetting │
└──────┬──────────┘ └────────────┬────────────────┘
│ │
▼ ▼
┌───────────────┐ ┌──────────────────────────────┐
│ 6A. Forward │ │ 6B. Forward with Restrictions │
│ to Model Core │ │ – Rate limit, no tool use │
└──────┬────────┘ │ – Annotate prompt for model │
│ └────────────┬──────────────────┘
▼ ▼
(SAFE PATH) ┌───────────────────────────┐
│ 7. Alert Logging + │
│ Feedback Loop for Review │
└───────────────────────────┘

Originally created by @carolynf2 on GitHub (Jul 19, 2025). Original GitHub issue: https://github.com/deepseek-ai/DeepSeek-R1/issues/715 ┌──────────────────────────────┐ │ 1. User Prompt Received │ └────────────┬─────────────────┘ │ ▼ ┌──────────────────────────────┐ │ 2. Normalize & Tokenize │ │ – Remove whitespace, emojis │ │ – Detect language, encoding │ └────────────┬─────────────────┘ │ ▼ ┌────────────────────────────────────────────┐ │ 3. Run Prompt Through Classifier Pipeline │ │ A. Rule-based Heuristics: │ │ – Regex patterns for known jailbreaks │ │ – Phrasing markers (“pretend to be...”) │ │ B. ML Classifier (Fine-tuned): │ │ – Uses prompt embeddings (e.g., BERT) │ │ – Trained on labeled benign/malicious │ └────────────┬───────────────────────────────┘ │ ┌───────▼─────────┐ │ 4. Threat Score │ │ (0 to 1) │ └───────┬─────────┘ <0.3 │ 0.3–0.7 │ >0.7 ▼ ▼ ┌────────────────┐ ┌─────────────────────────────┐ │ 5A. Classify as │ │ 5B. Classify as Suspicious │ │ Benign Query │ │ – Needs extra vetting │ └──────┬──────────┘ └────────────┬────────────────┘ │ │ ▼ ▼ ┌───────────────┐ ┌──────────────────────────────┐ │ 6A. Forward │ │ 6B. Forward with Restrictions │ │ to Model Core │ │ – Rate limit, no tool use │ └──────┬────────┘ │ – Annotate prompt for model │ │ └────────────┬──────────────────┘ ▼ ▼ (SAFE PATH) ┌───────────────────────────┐ │ 7. Alert Logging + │ │ Feedback Loop for Review │ └───────────────────────────┘

bsakaguchi

2026-10-26 21:23:34 +00:00

closed this issue
added the
stale

closed-as-stale
labels

bsakaguchi commented

2026-10-26 21:23:35 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Aug 30, 2025):

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If you believe this issue is still relevant, please leave a comment to keep it open. Thank you for your contributions!

@github-actions[bot] commented on GitHub (Aug 30, 2025): This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If you believe this issue is still relevant, please leave a comment to keep it open. Thank you for your contributions!

bsakaguchi commented

2026-10-26 21:23:35 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Sep 14, 2025):

false

@github-actions[bot] commented on GitHub (Sep 14, 2025): false