[PR #8] [operator] retry stream reconciliation and fix stale session tracking #9

New issue

Open

opened 2026-10-15 12:46:37 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-15 12:46:37 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/games-on-whales/fenrir/pull/8
Author: @shrinedogg
Created: 6/13/2026
Status: 🔄 Open

Base: main ← Head: fix/session-stream-retry

📝 Commits (1)

8400e83 [operator] retry stream reconciliation and fix stale session tracking

📊 Changes

4 files changed (+36 additions, -9 deletions)

View changed files

📝 go.mod (+2 -3)
📝 go.sum (+0 -2)
📝 pkg/controllers/session.go (+25 -3)
📝 pkg/generic/controller.go (+9 -1)

📄 Description

Problem

The session controller has two issues that surface as stalled sessions and endless log spam when running under Kubernetes.

1. Stream reconciliation is never retried. Reconcile calls reconcileActiveStreams but discards the result and returns nil:

if streamError := c.reconcileActiveStreams(context.TODO(), newObj); streamError != nil {
    klog.Errorf("Failed to reconcile active streams: %s", streamError)
    // ... sets a status condition, but falls through ...
}

//!TODO: figure our retry logic. Some of these errors surely are retriable
return nil

During pod startup the session Deployment isn't ready and wolf-agent isn't listening yet, so reconcileActiveStreams fails with connection refused / "deployment not ready". Because the error is swallowed and no further informer event is guaranteed once the agent does come up, the stream is never started. The session sits idle until the 1-minute dead-session reaper deletes it, and Moonlight times out.

2. trackedSessions is never cleaned up for sessions created after startup. Reconcile inserts into trackedSessions, but only the startup path records the reverse trackedGames mapping that the deletion branch relies on. So sessions created at runtime are never removed from tracking, and every later reconcile of the same user/game iterates the stale names and logs:

Failed to get session <namespace>/<name>: ... not found

forever.

Fix

Return the stream error from Reconcile so the workqueue requeues with backoff and the stream is started as soon as wolf-agent becomes reachable.
Cap the per-item exponential backoff at 5s (custom rate limiter — same shape as workqueue.DefaultTypedControllerRateLimiter, but with the per-item max lowered from 1000s to 5s). Sessions only live ~1 minute before the reaper collects them, so retries must stay frequent to be useful.
Record the reverse trackedGames mapping on every reconcile (not just at startup), and defensively prune NotFound entries while building owner references so stale names self-heal instead of logging forever.

Testing

go build ./... (native and GOOS=linux GOARCH=amd64) and go test ./pkg/generic/... pass.
Running on my Talos cluster (NVIDIA RTX 5090, fenrir/wolf) since June 10: sessions now recover from the "agent not ready yet" startup race instead of stalling until the reaper kills them. Engineering notes / full context: https://github.com/shrinedogg/biggs.dog#patched-fork--engineering-notes

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/games-on-whales/fenrir/pull/8 **Author:** [@shrinedogg](https://github.com/shrinedogg) **Created:** 6/13/2026 **Status:** 🔄 Open **Base:** `main` ← **Head:** `fix/session-stream-retry` --- ### 📝 Commits (1) - [`8400e83`](https://github.com/games-on-whales/fenrir/commit/8400e83ab0336fc2b055191c34bfe1cce6909db3) [operator] retry stream reconciliation and fix stale session tracking ### 📊 Changes **4 files changed** (+36 additions, -9 deletions) <details> <summary>View changed files</summary> 📝 `go.mod` (+2 -3) 📝 `go.sum` (+0 -2) 📝 `pkg/controllers/session.go` (+25 -3) 📝 `pkg/generic/controller.go` (+9 -1) </details> ### 📄 Description ## Problem The session controller has two issues that surface as stalled sessions and endless log spam when running under Kubernetes. **1. Stream reconciliation is never retried.** `Reconcile` calls `reconcileActiveStreams` but discards the result and returns `nil`: ```go if streamError := c.reconcileActiveStreams(context.TODO(), newObj); streamError != nil { klog.Errorf("Failed to reconcile active streams: %s", streamError) // ... sets a status condition, but falls through ... } //!TODO: figure our retry logic. Some of these errors surely are retriable return nil ``` During pod startup the session Deployment isn't ready and `wolf-agent` isn't listening yet, so `reconcileActiveStreams` fails with `connection refused` / "deployment not ready". Because the error is swallowed and no further informer event is guaranteed once the agent does come up, the stream is never started. The session sits idle until the 1-minute dead-session reaper deletes it, and Moonlight times out. **2. `trackedSessions` is never cleaned up for sessions created after startup.** `Reconcile` inserts into `trackedSessions`, but only the startup path records the reverse `trackedGames` mapping that the deletion branch relies on. So sessions created at runtime are never removed from tracking, and every later reconcile of the same user/game iterates the stale names and logs: ``` Failed to get session <namespace>/<name>: ... not found ``` forever. ## Fix * Return the stream error from `Reconcile` so the workqueue requeues with backoff and the stream is started as soon as `wolf-agent` becomes reachable. * Cap the per-item exponential backoff at **5s** (custom rate limiter — same shape as `workqueue.DefaultTypedControllerRateLimiter`, but with the per-item max lowered from 1000s to 5s). Sessions only live ~1 minute before the reaper collects them, so retries must stay frequent to be useful. * Record the reverse `trackedGames` mapping on every reconcile (not just at startup), and defensively prune `NotFound` entries while building owner references so stale names self-heal instead of logging forever. ## Testing * `go build ./...` (native and `GOOS=linux GOARCH=amd64`) and `go test ./pkg/generic/...` pass. * Running on my Talos cluster (NVIDIA RTX 5090, fenrir/wolf) since June 10: sessions now recover from the "agent not ready yet" startup race instead of stalling until the reaper kills them. Engineering notes / full context: https://github.com/shrinedogg/biggs.dog#patched-fork--engineering-notes --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi added the

pull-request

label

2026-10-15 12:46:37 +00:00

bsakaguchi referenced this issue

2026-10-15 12:46:37 +00:00

[PR #9] [CLOSED] [moonlight-proxy] raise launch wait from 25s to 120s #11