[GH-ISSUE #82 ] Native Kubernetes Integration #59

Author

Owner

@ABeltramo commented on GitHub (Jun 29, 2024):

This is probably out of scope for Wolf, it feels to me that this should be another software that leverage Wolf inside.

Wolf currently allows multiple users to share a single host, it automatically provisions virtual desktops, devices and routes the events from one client to the right Docker container, to simplify:

flowchart TD

subgraph Wolf
 M("Moonlight server")
 uinput
 GPU
 Docker
end

 U1("User 1") --> M
 U2("User 2") --> M
 Docker --> C1("Container 1")
 Docker --> C2("Container 2")

With k8s you could scale that further so that multiple clients can be mapped to multiple machines:

flowchart TD

subgraph K8S

 M("Moonlight server")

  subgraph Pod1
    subgraph Wolf1
     uinput1
     GPU1
     Docker1
     C1("Container 1")
    end
  end

  subgraph Pod2
    subgraph Wolf2
     uinput2
     GPU2
     Docker2
     C2("Container 2")
    end
  end

end

U1("User 1") --> M
U2("User 2") --> M

M --> Wolf1
M --> Wolf2

I hope this makes sense, I don't exclude we could work on this in future once Wolf is probably more mature and stable. Controlling multiple machines is definitely outside the scope of Wolf and will be better positioned in a separate program, but, there's quite a lot of logic and stuff that could be re-used IMHO.

@ABeltramo commented on GitHub (Jun 29, 2024): This is probably out of scope for Wolf, it feels to me that this should be another software that leverage Wolf inside. Wolf currently allows multiple users to share **a single host**, it automatically provisions virtual desktops, devices and routes the events from one client to the right Docker container, to simplify: ```mermaid flowchart TD subgraph Wolf M("Moonlight server") uinput GPU Docker end U1("User 1") --> M U2("User 2") --> M Docker --> C1("Container 1") Docker --> C2("Container 2") ``` With k8s you could scale that further so that multiple clients can be mapped to **multiple machines**: ```mermaid flowchart TD subgraph K8S M("Moonlight server") subgraph Pod1 subgraph Wolf1 uinput1 GPU1 Docker1 C1("Container 1") end end subgraph Pod2 subgraph Wolf2 uinput2 GPU2 Docker2 C2("Container 2") end end end U1("User 1") --> M U2("User 2") --> M M --> Wolf1 M --> Wolf2 ``` I hope this makes sense, I don't exclude we could work on this in future once Wolf is probably more mature and stable. Controlling multiple machines is definitely outside the scope of Wolf and will be better positioned in a separate program, but, there's quite a lot of logic and stuff that could be re-used IMHO.

bsakaguchi commented

Author

Owner

@bo0tzz commented on GitHub (Jun 29, 2024):

I think the main thing about a kubernetes integration would be having Wolf talk to the kubernetes API to create pods, rather than having it use a passed-through docker socket. Would that be feasible at all, in terms of functionality & code structure?

Edit: Looking at docker.hpp, that seems to me like it's not all too far off fitting the Kubernetes API as well. I have no insight into whether all the other parts of wolf could work with Kubernetes though, in terms of passing through devices and communicating between containers and such.

@bo0tzz commented on GitHub (Jun 29, 2024): I think the main thing about a kubernetes integration would be having Wolf talk to the kubernetes API to create pods, rather than having it use a passed-through docker socket. Would that be feasible at all, in terms of functionality & code structure? Edit: Looking at docker.hpp, that seems to me like it's not all too far off fitting the Kubernetes API as well. I have no insight into whether all the other parts of wolf could work with Kubernetes though, in terms of passing through devices and communicating between containers and such.

bsakaguchi commented

Author

Owner

@ABeltramo commented on GitHub (Jun 29, 2024):

I think the main thing about a kubernetes integration would be having Wolf talk to the kubernetes API to create pods, rather than having it use a passed-through docker socket. Would that be feasible at all, in terms of functionality & code structure?

Yep that could easily be a different Runner the problem is more about the locality of things

I have no insight into whether all the other parts of wolf could work with Kubernetes though, in terms of passing through devices and communicating between containers and such.

That's exactly my point and why I've included uinput and GPU in the simple charts above (sorry, I should have been more specific). I'm not an expert on k8s, but, I know that we can't have Wolf running in one pod and an app (Steam, Firefox, ...) on another one because we can't share the local virtual input devices (uinput) and Wayland socket (GPU) between different machines.

That's why I think the proper solution would be to decouple the Moonlight protocol from the rest, this way you'll have

Users --> Moonlight server --> Multiple machines --> Wolf [Multiple desktops/instances --> Steam]

Wolf can "only" do the last step, but you need something else that coordinates the multiple Wolf instances.
I hope this clears things up; I'm very open to ideas and feedback

@ABeltramo commented on GitHub (Jun 29, 2024): > I think the main thing about a kubernetes integration would be having Wolf talk to the kubernetes API to create pods, rather than having it use a passed-through docker socket. Would that be feasible at all, in terms of functionality & code structure? Yep that could easily be a different [Runner](https://github.com/games-on-whales/wolf/blob/1d8f18198d86ecba0a849c65d65533799639e064/src/moonlight-server/state/data-structures.hpp#L42-L52) the problem is more about the _locality_ of things > I have no insight into whether all the other parts of wolf could work with Kubernetes though, in terms of passing through devices and communicating between containers and such. That's exactly my point and why I've included `uinput` and `GPU` in the simple charts above (sorry, I should have been more specific). I'm not an expert on k8s, but, I know that we can't have Wolf running in one pod and an app (Steam, Firefox, ...) on another one because we can't share the local virtual input devices (`uinput`) and Wayland socket (`GPU`) between different machines. That's why I think the proper solution would be to decouple the Moonlight protocol from the rest, this way you'll have ``` Users --> Moonlight server --> Multiple machines --> Wolf [Multiple desktops/instances --> Steam] ``` Wolf can "only" do the last step, but you need something else that coordinates the multiple Wolf instances. I hope this clears things up; I'm very open to ideas and feedback

bsakaguchi commented

Author

Owner

@bo0tzz commented on GitHub (Jun 29, 2024):

Is it 'just' unix sockets that need to be shared, or is there more to it? I think it would be possible to have separate kubernetes Pods for the different bits with a socket shared between them, with the caveat that it's a bit unusual and would only work as long as the Pods are still on the same Node (but that's honestly fine).

Wolf takes a docker socket and spins stuff up on the fly - is that required, or is it possible to have all the containers started ahead of time instead? If so, having one Pod with all of the Wolf bits inside of it would also be a reasonable solution that should work on Kubernetes without (many) changes today.

Something relevant here that isn't yet entirely clear to me - in the existing Docker setup, does a single Wolf instance support multiple clients running different things at the same time, or is it moonlight sending clients to separate Wolf instances? That would influence what the "best practice" way to fit this into Kubernetes is.

@bo0tzz commented on GitHub (Jun 29, 2024): Is it 'just' unix sockets that need to be shared, or is there more to it? I think it would be possible to have separate kubernetes Pods for the different bits with a socket shared between them, with the caveat that it's a bit unusual and would only work as long as the Pods are still on the same Node (but that's honestly fine). Wolf takes a docker socket and spins stuff up on the fly - is that required, or is it possible to have all the containers started ahead of time instead? If so, having one Pod with all of the Wolf bits inside of it would also be a reasonable solution that should work on Kubernetes without (many) changes today. Something relevant here that isn't yet entirely clear to me - in the existing Docker setup, does a single Wolf instance support multiple clients running different things at the same time, or is it moonlight sending clients to separate Wolf instances? That would influence what the "best practice" way to fit this into Kubernetes is.

bsakaguchi commented

Author

Owner

@ABeltramo commented on GitHub (Jun 29, 2024):

Is it 'just' unix sockets that need to be shared, or is there more to it? I think it would be possible to have separate kubernetes Pods for the different bits with a socket shared between them, with the caveat that it's a bit unusual and would only work as long as the Pods are still on the same Node (but that's honestly fine).

It's not just sockets, we are talking to and sharing virtual devices under /dev/input/ plus the Wayland socket that can't be shared over the network. Wolf has been developed under the assumption that you are on a single machine, lifting that assumption to manage multiple machines is out of scope for Wolf.

Wolf takes a docker socket and spins stuff up on the fly - is that required, or is it possible to have all the containers started ahead of time instead? If so, having one Pod with all of the Wolf bits inside of it would also be a reasonable solution that should work on Kubernetes without (many) changes today.

It is required because you don't know ahead of time what is going to be the client settings (resolution, FPS, how many and which devices to provision, think of joypads for example) and most importantly how many and which apps do you want to start.
Wolf automatically spins up and down containers on demand, that's the main point. You want to start Steam on your phone and another one in the household starts Retroarch on another device? Wolf will do that, automagically and transparently for the end user.

Something relevant here that isn't yet entirely clear to me - in the existing Docker setup, does a single Wolf instance support multiple clients running different things at the same time, or is it moonlight sending clients to separate Wolf instances? That would influence what the "best practice" way to fit this into Kubernetes is.

There's a single Wolf instance that will manage and control multiple containers. You can read more about how it works from a high level POW in the docs here

@ABeltramo commented on GitHub (Jun 29, 2024): > Is it 'just' unix sockets that need to be shared, or is there more to it? I think it would be possible to have separate kubernetes Pods for the different bits with a socket shared between them, with the caveat that it's a bit unusual and would only work as long as the Pods are still on the same Node (but that's honestly fine). It's not just sockets, we are talking to and sharing virtual devices under `/dev/input/` plus the Wayland socket that can't be shared over the network. Wolf has been developed under the assumption that you are on a single machine, lifting that assumption to manage multiple machines is out of scope for Wolf. > Wolf takes a docker socket and spins stuff up on the fly - is that required, or is it possible to have all the containers started ahead of time instead? If so, having one Pod with all of the Wolf bits inside of it would also be a reasonable solution that should work on Kubernetes without (many) changes today. It is required because you don't know ahead of time what is going to be the client settings (resolution, FPS, how many and which devices to provision, think of joypads for example) and most importantly how many and which apps do you want to start. Wolf automatically spins up and down containers on demand, that's the main point. You want to start Steam on your phone and another one in the household starts Retroarch on another device? Wolf will do that, automagically and transparently for the end user. > Something relevant here that isn't yet entirely clear to me - in the existing Docker setup, does a single Wolf instance support multiple clients running different things at the same time, or is it moonlight sending clients to separate Wolf instances? That would influence what the "best practice" way to fit this into Kubernetes is. There's a single Wolf instance that will manage and control multiple containers. You can read more about how it works from a high level POW in [the docs here](https://games-on-whales.github.io/wolf/stable/dev/how-it-works.html)

bsakaguchi commented

Author

Owner

@bo0tzz commented on GitHub (Jun 29, 2024):

Wolf has been developed under the assumption that you are on a single machine, lifting that assumption to manage multiple machines is out of scope for Wolf.

This is absolutely fair. I want to clarify that running in Kubernetes doesn't mean you have to split across multiple machines. You can pin things to one node, and you still get the other benefits of Kubernetes when doing that.

My suggestion for a first pass at this request would be adding a Kubernetes backend that spins up a Pod for each container that's needed, and uses pod affinity to ensure they all land on the same node, then just mounts host paths in the same way that the current Docker setup does. I expect that should Just Work without needing changes in other parts of Wolf (fingers crossed).

@bo0tzz commented on GitHub (Jun 29, 2024): > Wolf has been developed under the assumption that you are on a single machine, lifting that assumption to manage multiple machines is out of scope for Wolf. This is absolutely fair. I want to clarify that running in Kubernetes doesn't mean you have to split across multiple machines. You can pin things to one node, and you still get the other benefits of Kubernetes when doing that. My suggestion for a first pass at this request would be adding a Kubernetes backend that spins up a Pod for each container that's needed, and uses pod affinity to ensure they all land on the same node, then just mounts host paths in the same way that the current Docker setup does. I expect that should Just Work without needing changes in other parts of Wolf (fingers crossed).

bsakaguchi commented

Author

Owner

@ABeltramo commented on GitHub (Jun 29, 2024):

That doesn't sound too bad, and it'll probably fit well as a different Runner implementation. Sounds like this would give me a nice excuse to finally tip my toes into k8s.. 😅

@ABeltramo commented on GitHub (Jun 29, 2024): That doesn't sound too bad, and it'll probably fit well as a different Runner implementation. Sounds like this would give me a nice excuse to finally tip my toes into k8s.. 😅

bsakaguchi commented

Author

Owner

@bo0tzz commented on GitHub (Jun 29, 2024):

I didn't expect you'd be interested in implementing it yourself, but that would be pretty awesome :D Unfortunately I can't really write the code myself (or I'd already be doing so, lol) but if you need any help around Kube at all don't hesitate to ask!

Thank you for the great discussion btw :)

@bo0tzz commented on GitHub (Jun 29, 2024): I didn't expect you'd be interested in implementing it yourself, but that would be pretty awesome :D Unfortunately I can't really write the code myself (or I'd already be doing so, lol) but if you need any help around Kube at all don't hesitate to ask! Thank you for the great discussion btw :)

bsakaguchi commented

Author

Owner

@ABeltramo commented on GitHub (Jun 29, 2024):

Thank you for sticking around and bring up some good points! There are a few issues that I would like to tackle first, I'm not sure when I'll have time to look into this; I'll keep this open if anyone else wants to give it a shot.

@ABeltramo commented on GitHub (Jun 29, 2024): Thank you for sticking around and bring up some good points! There are a few issues that I would like to tackle first, I'm not sure when I'll have time to look into this; I'll keep this open if anyone else wants to give it a shot.

bsakaguchi commented

Author

Owner

@ABeltramo commented on GitHub (Jul 9, 2024):

So does Kasm and others, how's this relevant to the discussion?

@ABeltramo commented on GitHub (Jul 9, 2024): So does Kasm and others, how's this relevant to the discussion?

bsakaguchi commented

Author

Owner

@Relicjamin-jv commented on GitHub (Sep 27, 2024):

I wouldn't mind taking a whack at this, although I'm not too experienced at C++. I'm happy to see official development documentation, I'll go ahead and fork this project and see what I can start cooking. I agree with keeping this objectively to one node, and implementing a new runner.

@Relicjamin-jv commented on GitHub (Sep 27, 2024): I wouldn't mind taking a whack at this, although I'm not too experienced at C++. I'm happy to see official development documentation, I'll go ahead and fork this project and see what I can start cooking. I agree with keeping this objectively to one node, and implementing a new runner.

bsakaguchi commented

https://slint.dev/blog/rust-and-cpp

Author

Owner

@Relicjamin-jv commented on GitHub (Sep 27, 2024):

Looking into this a little bit further, there are no really good C++ libraries, however there is a C library that I can wrap. Having this information, would the project maintainers be against introducing another language (Rust?) to implement this feature?

@Relicjamin-jv commented on GitHub (Sep 27, 2024): Looking into this a little bit further, there are no really good C++ libraries, however there is a C library that I can wrap. Having this information, would the project maintainers be against introducing another language (Rust?) to implement this feature? https://slint.dev/blog/rust-and-cpp

bsakaguchi commented

Author

Owner

@Relicjamin-jv commented on GitHub (Sep 27, 2024):

Never mind, I'll go ahead and write a C++ HTTP client library for CRUD on pods. It will be a fun chance to learn C++.

@Relicjamin-jv commented on GitHub (Sep 27, 2024): Never mind, I'll go ahead and write a C++ HTTP client library for CRUD on pods. It will be a fun chance to learn C++.

bsakaguchi commented

2026-10-15 12:55:39 +00:00

Author

Owner

@ABeltramo commented on GitHub (Sep 27, 2024):

Nothing against Rust, in fact we use it quite extensively for our custom Wayland compositor gst-wayland-display.

My only advice would be that wrapping a C library can be done safely in C++ using smart pointers instead of pulling Rust just for implementing a wrapper. I'd be more than happy to provide more guidance, at least on set things up; feel free to give me a shout on Discord!

Also, if you want to dive into the code I really recommend using devcontainers; all dependencies will be already setup and you should have a working environment automagically in a few minutes (and I should add that to the docs..).

@ABeltramo commented on GitHub (Sep 27, 2024): Nothing against Rust, in fact we use it quite extensively for our custom Wayland compositor `gst-wayland-display`. My only advice would be that wrapping a C library can be done safely in C++ using smart pointers instead of pulling Rust just for implementing a wrapper. I'd be more than happy to provide more guidance, at least on set things up; feel free to give me a shout on Discord! Also, if you want to dive into the code I really recommend using devcontainers; all dependencies will be already setup and you should have a working environment automagically in a few minutes (and I should add that to the docs..).

bsakaguchi commented

Author

Owner

@Relicjamin-jv commented on GitHub (Sep 27, 2024):

I'll take that approach, thanks for being available, I look forward to talking with you! I will definitely use those dev containers.

@Relicjamin-jv commented on GitHub (Sep 27, 2024): I'll take that approach, thanks for being available, I look forward to talking with you! I will definitely use those dev containers.

bsakaguchi commented

2026-10-15 12:55:39 +00:00

Author

Owner

@judahrand commented on GitHub (Nov 16, 2024):

Does this issues still fall under the wontfix label? I've been playing with deploying GoW on Kubernetes with Sunshine but a lot of the containers are quite out of date and the experience isn't great. wolf behaving like kind of like a Kubernetes Operator sounds way better.

@judahrand commented on GitHub (Nov 16, 2024): Does this issues still fall under the `wontfix` label? I've been playing with deploying GoW on Kubernetes with Sunshine but a lot of the containers are quite out of date and the experience isn't great. `wolf` behaving like kind of like a Kubernetes Operator sounds way better.

bsakaguchi commented

Author

Owner

@ABeltramo commented on GitHub (Nov 16, 2024):

I don't use k8s in my homelab, so personally, I don't have much reason to work on this.
As I said before, I'd be more than happy to help anyone from the community that is willing to work on this on any language of choice!

I'll change the label accordingly 😉

@ABeltramo commented on GitHub (Nov 16, 2024): I don't use k8s in my homelab, so personally, I don't have much reason to work on this. As I said before, I'd be more than happy to help anyone from the community that is willing to work on this on any language of choice! I'll change the label accordingly 😉

bsakaguchi commented

Author

Owner

@DrummyFloyd commented on GitHub (Jan 7, 2025):

Does this issues still fall under the wontfix label? I've been playing with deploying GoW on Kubernetes with Sunshine but a lot of the containers are quite out of date and the experience isn't great. wolf behaving like kind of like a Kubernetes Operator sounds way better.

you're right wolf act an operator, seems to be a good direction for me, to look after maybe an k8s-operator sdk

do we have some good dev here ? (i'm not very good , but i can do many stuff )

is there ppl interested to make a K8S stuff with GOLANG based on the wold backend ?

@DrummyFloyd commented on GitHub (Jan 7, 2025): > Does this issues still fall under the `wontfix` label? I've been playing with deploying GoW on Kubernetes with Sunshine but a lot of the containers are quite out of date and the experience isn't great. `wolf` behaving like kind of like a Kubernetes Operator sounds way better. you're right wolf act an operator, seems to be a good direction for me, to look after maybe an k8s-operator sdk do we have some good dev here ? (i'm not very good , but i can do many stuff ) is there ppl interested to make a K8S stuff with GOLANG based on the wold backend ?

bsakaguchi commented

Author

Owner

@yitzhakbg commented on GitHub (Jan 8, 2025):

is there ppl interested to make a K8S stuff with GOLANG based on the wold backend ?
Looks promising. I'd be interested (preferably a Rust backend) but I'm not proficient.
My vision is to mass stream of Anki through Wolf in educational settings. What would be the way to go?
P.S. What is wold backend?

@yitzhakbg commented on GitHub (Jan 8, 2025): _is there ppl interested to make a K8S stuff with GOLANG based on the wold backend ?_ Looks promising. I'd be interested (preferably a Rust backend) but I'm not proficient. My vision is to mass stream of [Anki](https://apps.ankiweb.net/) through Wolf in educational settings. What would be the way to go? P.S. What is wold backend?

bsakaguchi commented

Author

Owner

@bo0tzz commented on GitHub (Jan 8, 2025):

I think a whole operator probably isn't necessary, though I guess it would work. I think the most straightforward way, like also discussed above, would be essentially to create a Kubernetes version of https://github.com/games-on-whales/wolf/blob/stable/src/moonlight-server/runners/docker.cpp (if I'm understanding the Wolf codebase correctly).

@bo0tzz commented on GitHub (Jan 8, 2025): I think a whole operator probably isn't necessary, though I guess it would _work_. I think the most straightforward way, like also discussed above, would be essentially to create a Kubernetes version of https://github.com/games-on-whales/wolf/blob/stable/src/moonlight-server/runners/docker.cpp (if I'm understanding the Wolf codebase correctly).

bsakaguchi commented

Author

Owner

@DrummyFloyd commented on GitHub (Jan 8, 2025):

I think a whole operator probably isn't necessary, though I guess it would work. I think the most straightforward way, like also discussed above, would be essentially to create a Kubernetes version of https://github.com/games-on-whales/wolf/blob/stable/src/moonlight-server/runners/docker.cpp (if I'm understanding the Wolf codebase correctly).

will try to have a look, even if i a huge 💩 with C++

@DrummyFloyd commented on GitHub (Jan 8, 2025): > I think a whole operator probably isn't necessary, though I guess it would _work_. I think the most straightforward way, like also discussed above, would be essentially to create a Kubernetes version of https://github.com/games-on-whales/wolf/blob/stable/src/moonlight-server/runners/docker.cpp (if I'm understanding the Wolf codebase correctly). will try to have a look, even if i a huge :hankey: with C++

bsakaguchi commented

Author

Owner

@yitzhakbg commented on GitHub (Jan 8, 2025):

"will try to have a look, even if i a huge 💩 with C++"
AI coding to our assistance. What does Abeltramo think?
On Wed, Jan 8, 2025 at 2:47 PM DrummyFloyd @.***> wrote:

I think a whole operator probably isn't necessary, though I guess it would
work. I think the most straightforward way, like also discussed above,
would be essentially to create a Kubernetes version of
https://github.com/games-on-whales/wolf/blob/stable/src/moonlight-server/runners/docker.cpp
(if I'm understanding the Wolf codebase correctly).

will try to have a look, even if i a huge 💩 with C++

—
Reply to this email directly, view it on GitHub
https://github.com/games-on-whales/wolf/issues/82#issuecomment-2577591819,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAADR5HNZXCNB4EXRHCTWLT2JUM63AVCNFSM6AAAAABKC3A5FCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKNZXGU4TCOBRHE
.
You are receiving this because you commented.Message ID:
@.***>

@yitzhakbg commented on GitHub (Jan 8, 2025): *"will try to have a look, even if i a huge 💩 with C++"* AI coding to our assistance. What does Abeltramo think? On Wed, Jan 8, 2025 at 2:47 PM DrummyFloyd ***@***.***> wrote: > I think a whole operator probably isn't necessary, though I guess it would > *work*. I think the most straightforward way, like also discussed above, > would be essentially to create a Kubernetes version of > https://github.com/games-on-whales/wolf/blob/stable/src/moonlight-server/runners/docker.cpp > (if I'm understanding the Wolf codebase correctly). > > will try to have a look, even if i a huge 💩 with C++ > > — > Reply to this email directly, view it on GitHub > <https://github.com/games-on-whales/wolf/issues/82#issuecomment-2577591819>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/AAADR5HNZXCNB4EXRHCTWLT2JUM63AVCNFSM6AAAAABKC3A5FCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKNZXGU4TCOBRHE> > . > You are receiving this because you commented.Message ID: > ***@***.***> >

bsakaguchi commented

Author

Owner

@anultravioletaurora commented on GitHub (Jan 8, 2025):

I've got a K8s cluster with GPUs that I can do testing on if y'all need guinea pigs!

@anultravioletaurora commented on GitHub (Jan 8, 2025): I've got a K8s cluster with GPUs that I can do testing on if y'all need guinea pigs!

bsakaguchi commented

Author

Owner

@ABeltramo commented on GitHub (Jan 8, 2025):

I think it's fairly simple to implement what @bo0tzz suggested above:

My suggestion for a first pass at this request would be adding a Kubernetes backend that spins up a Pod for each container that's needed, and uses pod affinity to ensure they all land on the same node, then just mounts host paths in the same way that the current Docker setup does. I expect that should Just Work without needing changes in other parts of Wolf (fingers crossed).

This can potentially be done in any language, compiled down to a binary and then being called by Wolf as a process runner (ex: [Test ball] ).

The main caveat to that is that Wolf wouldn't support multiple nodes as explained in detail in the previous comments.

I'd be happy to help anyone that wants to implement the little runner or a more general purpose, multi-node k8s application that coordinates multiple Wolf instances (and/or single parts of it) on each node. We have some fairly powerful APIs nowadays 😉

@ABeltramo commented on GitHub (Jan 8, 2025): I think it's fairly simple to implement what @bo0tzz suggested above: > My suggestion for a first pass at this request would be adding a Kubernetes backend that spins up a Pod for each container that's needed, and uses pod affinity to ensure they all land on the same node, then just mounts host paths in the same way that the current Docker setup does. I expect that should Just Work without needing changes in other parts of Wolf (fingers crossed). This can potentially be done **in any language**, compiled down to a binary and then being called by Wolf as a `process` runner (ex: [[Test ball]](https://github.com/games-on-whales/wolf/blob/84be69bcc5c22e4c318ba0034eaa08376ad84543/src/moonlight-server/state/default/config.v4.toml#L154-L156) ). The main caveat to that is that Wolf **wouldn't support multiple nodes** as explained in detail in the previous comments. I'd be happy to help anyone that wants to implement the little runner or a more general purpose, multi-node k8s application that coordinates multiple Wolf instances (and/or single parts of it) on each node. We have some fairly powerful APIs nowadays :wink:

bsakaguchi commented

Author

Owner

@bo0tzz commented on GitHub (Jan 8, 2025):

This can potentially be done in any language, compiled down to a binary and then being called by Wolf as a process runner

I hadn't considered that option yet, but I think with that the way to go for a first pass is pretty clearly to just call out to kubectl. That means no custom binary, not needing to code any direct HTTP interaction with the Kubernetes API, when running inside a Pod it should automatically pick up any mounted credentials, and it'll make development easier as it can just be run from a workstation rather than getting the code being worked on into Kubernetes somehow.
That should make for a good first pass that'll get things running, which means people can start using Wolf in Kube and trying stuff out, and then there's plenty of room from that point on to improve things with fancier & more elaborate setups.

As another way to simplify the first pass a bit, rather than the multiple Pods with node affinity that I suggested above, it'll probably be easier to start with just a single Pod with multiple containers for the different pieces, and the sockets mounted via emptyDir volumes. That might also play nicer if there are multiple instances running on one node.
If it'd be helpful for whoever develops this, I could potentially put together a (dummy) Pod YAML to illustrate the sort of layout I'm thinking of.

@bo0tzz commented on GitHub (Jan 8, 2025): > This can potentially be done in any language, compiled down to a binary and then being called by Wolf as a process runner I hadn't considered that option yet, but I think with that the way to go for a first pass is pretty clearly to just call out to `kubectl`. That means no custom binary, not needing to code any direct HTTP interaction with the Kubernetes API, when running inside a Pod it should automatically pick up any mounted credentials, and it'll make development easier as it can just be run from a workstation rather than getting the code being worked on into Kubernetes somehow. That should make for a good first pass that'll get things running, which means people can start using Wolf in Kube and trying stuff out, and then there's plenty of room from that point on to improve things with fancier & more elaborate setups. As another way to simplify the first pass a bit, rather than the multiple Pods with node affinity that I suggested above, it'll probably be easier to start with just a single Pod with multiple containers for the different pieces, and the sockets mounted via `emptyDir` volumes. That might also play nicer if there are multiple instances running on one node. If it'd be helpful for whoever develops this, I could potentially put together a (dummy) Pod YAML to illustrate the sort of layout I'm thinking of.

bsakaguchi commented

Author

Owner

@RyuunosukeDS3 commented on GitHub (Feb 28, 2025):

I was able to make wolf run under Kubernetes today, but only wolf. The other containers still seem to be in regular docker. For my specific use case, this will suffice. For now, I made a VERY rudimentary implementation. But I could share if there is anyone interested

@RyuunosukeDS3 commented on GitHub (Feb 28, 2025): I was able to make wolf run under Kubernetes today, but only wolf. The other containers still seem to be in regular docker. For my specific use case, this will suffice. For now, I made a VERY rudimentary implementation. But I could share if there is anyone interested

bsakaguchi commented

Author

Owner

@raiomarco commented on GitHub (Feb 28, 2025):

I was able to make wolf run under Kubernetes today, but only wolf. The other containers still seem to be in regular docker. For my specific use case, this will suffice. For now, I made a VERY rudimentary implementation. But I could share if there is anyone interested

I'm actually really interested

@raiomarco commented on GitHub (Feb 28, 2025): > I was able to make wolf run under Kubernetes today, but only wolf. The other containers still seem to be in regular docker. For my specific use case, this will suffice. For now, I made a VERY rudimentary implementation. But I could share if there is anyone interested I'm actually really interested

bsakaguchi commented

Author

Owner

@RyuunosukeDS3 commented on GitHub (Feb 28, 2025):

I will try to share as soon as I can (It COULD take some days tho, sry)

@RyuunosukeDS3 commented on GitHub (Feb 28, 2025): I will try to share as soon as I can (It COULD take some days tho, sry)

bsakaguchi commented

Author

Owner

@raiomarco commented on GitHub (Feb 28, 2025):

no problem :)

@raiomarco commented on GitHub (Feb 28, 2025): no problem :)

bsakaguchi commented

Author

Owner

@RyuunosukeDS3 commented on GitHub (Mar 1, 2025):

OK, I got lucky and got some free time now.

First of all, I ran this command in order to copy my nvidia volume to somewhere easier to mount:
docker run --rm -v nvidia-driver-vol:/volume -it -v $HOME/ubuntu-nvidia-driver:/mnt busybox sh
cp -R /volume/* /mnt

So here the manifests.

deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: wolf
  labels:
    app: wolf
spec:
  replicas: 1
  selector:
    matchLabels:
      app: wolf
  template:
    metadata:
      labels:
        app: wolf
    spec:
      hostNetwork: true
      containers:
        - name: wolf
          image: ghcr.io/games-on-whales/wolf:stable
          env:
            - name: XDG_RUNTIME_DIR
              value: "/tmp/sockets"
            - name: NVIDIA_DRIVER_VOLUME_NAME
              value: "nvidia-driver-vol"
            - name: HOST_APPS_STATE_FOLDER
              value: "/etc/wolf"
          volumeMounts:
            - name: wolf-config
              mountPath: /etc/wolf
              readOnly: false
            - name: socket-dir
              mountPath: /tmp/sockets
              readOnly: false
            - name: docker-socket
              mountPath: /var/run/docker.sock
              readOnly: false
            - name: dev-dir
              mountPath: /dev
              readOnly: false
            - name: udev-dir
              mountPath: /run/udev
              readOnly: false
            - name: nvidia-driver-vol
              mountPath: /usr/nvidia
              readOnly: false
          securityContext:
            privileged: true
          devices:
            - devicePath: /dev/dri
              type: char
            - devicePath: /dev/uinput
              type: char
            - devicePath: /dev/uhid
              type: char
            - devicePath: /dev/nvidia-uvm
              type: char
            - devicePath: /dev/nvidia-uvm-tools
              type: char
            - devicePath: /dev/nvidia-caps/nvidia-cap1
              type: char
            - devicePath: /dev/nvidia-caps/nvidia-cap2
              type: char
            - devicePath: /dev/nvidiactl
              type: char
            - devicePath: /dev/nvidia0
              type: char
            - devicePath: /dev/nvidia-modeset
              type: char
      volumes:
        - name: wolf-config
          hostPath:
            path: /etc/wolf
            type: Directory
        - name: socket-dir
          hostPath:
            path: /tmp/sockets
            type: Directory
        - name: docker-socket
          hostPath:
            path: /var/run/docker.sock
            type: Socket
        - name: dev-dir
          hostPath:
            path: /dev
            type: Directory
        - name: udev-dir
          hostPath:
            path: /run/udev
            type: Directory
        - name: nvidia-driver-vol
          persistentVolumeClaim:
            claimName: nvidia-driver-pvc

Here's a sample PV for the nvidia stuff:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: nvidia-driver-pv
spec:
  capacity:
    storage: 1Gi
  volumeMode: Filesystem
  accessModes:
    - ReadWriteMany
  persistentVolumeReclaimPolicy: Retain
  storageClassName: nvidia-driver-storage
  hostPath:
    path: /home/ryuunosuke/ubuntu-nvidia-driver
    type: Directory

Here's the PVC

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: nvidia-driver-pvc
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 1Gi
  volumeMode: Filesystem
  storageClassName: nvidia-driver-storage

And finally the service:

apiVersion: v1
kind: Service
metadata:
  name: wolf-service
spec:
  type: NodePort
  ports:
    - port: 47984
      targetPort: 47984
      nodePort: 47984
      name: sunshine-tcp-1
      protocol: TCP
    - port: 47989
      targetPort: 47989
      nodePort: 47989
      name: sunshine-tcp-2
      protocol: TCP
    - port: 48010
      targetPort: 48010
      nodePort: 48010
      name: sunshine-tcp-3
      protocol: TCP
    - port: 47998
      targetPort: 47998
      nodePort: 47998
      name: sunshine-udp-1
      protocol: UDP
    - port: 47999
      targetPort: 47999
      nodePort: 47999
      name: sunshine-udp-2
      protocol: UDP
    - port: 48000
      targetPort: 48000
      nodePort: 48000
      name: sunshine-udp-3
      protocol: UDP
    - port: 48002
      targetPort: 48002
      nodePort: 48002
      name: sunshine-udp-4
      protocol: UDP
    - port: 48010
      targetPort: 48010
      nodePort: 48010
      name: sunshine-udp-5
      protocol: UDP
  selector:
    app: wolf

In order to make this work, I had to change the default NodePorts. There's probably a more graceful way of doing it. I tried using the loadbalancer, but it didn't work. I didn't spend too much time trying to make it work tho, maybe it's something simple I haven't seen yet, maybe it is something complex. But what I did to change the default ports of my NodePort was adding this flag to the k3s systemctl: --kube-apiserver-arg=service-node-port-range=47000-49828

Like I said it is VERY rudimentary, hacky and testy, but it is something, right? As I try new stuff and evolve it, I might post an update. For now this is the best I could do.

@RyuunosukeDS3 commented on GitHub (Mar 1, 2025): OK, I got lucky and got some free time now. First of all, I ran this command in order to copy my nvidia volume to somewhere easier to mount: `docker run --rm -v nvidia-driver-vol:/volume -it -v $HOME/ubuntu-nvidia-driver:/mnt busybox sh` `cp -R /volume/* /mnt` So here the manifests. deployment.yaml ```yaml apiVersion: apps/v1 kind: Deployment metadata: name: wolf labels: app: wolf spec: replicas: 1 selector: matchLabels: app: wolf template: metadata: labels: app: wolf spec: hostNetwork: true containers: - name: wolf image: ghcr.io/games-on-whales/wolf:stable env: - name: XDG_RUNTIME_DIR value: "/tmp/sockets" - name: NVIDIA_DRIVER_VOLUME_NAME value: "nvidia-driver-vol" - name: HOST_APPS_STATE_FOLDER value: "/etc/wolf" volumeMounts: - name: wolf-config mountPath: /etc/wolf readOnly: false - name: socket-dir mountPath: /tmp/sockets readOnly: false - name: docker-socket mountPath: /var/run/docker.sock readOnly: false - name: dev-dir mountPath: /dev readOnly: false - name: udev-dir mountPath: /run/udev readOnly: false - name: nvidia-driver-vol mountPath: /usr/nvidia readOnly: false securityContext: privileged: true devices: - devicePath: /dev/dri type: char - devicePath: /dev/uinput type: char - devicePath: /dev/uhid type: char - devicePath: /dev/nvidia-uvm type: char - devicePath: /dev/nvidia-uvm-tools type: char - devicePath: /dev/nvidia-caps/nvidia-cap1 type: char - devicePath: /dev/nvidia-caps/nvidia-cap2 type: char - devicePath: /dev/nvidiactl type: char - devicePath: /dev/nvidia0 type: char - devicePath: /dev/nvidia-modeset type: char volumes: - name: wolf-config hostPath: path: /etc/wolf type: Directory - name: socket-dir hostPath: path: /tmp/sockets type: Directory - name: docker-socket hostPath: path: /var/run/docker.sock type: Socket - name: dev-dir hostPath: path: /dev type: Directory - name: udev-dir hostPath: path: /run/udev type: Directory - name: nvidia-driver-vol persistentVolumeClaim: claimName: nvidia-driver-pvc ``` Here's a sample PV for the nvidia stuff: ```yaml apiVersion: v1 kind: PersistentVolume metadata: name: nvidia-driver-pv spec: capacity: storage: 1Gi volumeMode: Filesystem accessModes: - ReadWriteMany persistentVolumeReclaimPolicy: Retain storageClassName: nvidia-driver-storage hostPath: path: /home/ryuunosuke/ubuntu-nvidia-driver type: Directory ``` Here's the PVC ```yaml apiVersion: v1 kind: PersistentVolumeClaim metadata: name: nvidia-driver-pvc spec: accessModes: - ReadWriteMany resources: requests: storage: 1Gi volumeMode: Filesystem storageClassName: nvidia-driver-storage ``` And finally the service: ```yaml apiVersion: v1 kind: Service metadata: name: wolf-service spec: type: NodePort ports: - port: 47984 targetPort: 47984 nodePort: 47984 name: sunshine-tcp-1 protocol: TCP - port: 47989 targetPort: 47989 nodePort: 47989 name: sunshine-tcp-2 protocol: TCP - port: 48010 targetPort: 48010 nodePort: 48010 name: sunshine-tcp-3 protocol: TCP - port: 47998 targetPort: 47998 nodePort: 47998 name: sunshine-udp-1 protocol: UDP - port: 47999 targetPort: 47999 nodePort: 47999 name: sunshine-udp-2 protocol: UDP - port: 48000 targetPort: 48000 nodePort: 48000 name: sunshine-udp-3 protocol: UDP - port: 48002 targetPort: 48002 nodePort: 48002 name: sunshine-udp-4 protocol: UDP - port: 48010 targetPort: 48010 nodePort: 48010 name: sunshine-udp-5 protocol: UDP selector: app: wolf ``` In order to make this work, I had to change the default NodePorts. There's probably a more graceful way of doing it. I tried using the loadbalancer, but it didn't work. I didn't spend too much time trying to make it work tho, maybe it's something simple I haven't seen yet, maybe it is something complex. But what I did to change the default ports of my NodePort was adding this flag to the k3s systemctl: `--kube-apiserver-arg=service-node-port-range=47000-49828` Like I said it is VERY rudimentary, hacky and testy, but it is something, right? As I try new stuff and evolve it, I might post an update. For now this is the best I could do.

bsakaguchi commented

Author

Owner

@ABeltramo commented on GitHub (Mar 1, 2025):

Thanks @RyuunosukeDS3 for sharing your setup.
If anyone wants to contribute it would be nice to add this in a separate page in the documentation portal. This doesn't "fix" the original issue (auto-scale on-demand to multiple nodes) but at least it would allow Wolf to run on K8S I guess.

@ABeltramo commented on GitHub (Mar 1, 2025): Thanks @RyuunosukeDS3 for sharing your setup. If anyone wants to contribute it would be nice to add this in a separate page in the documentation portal. This doesn't "fix" the original issue (auto-scale on-demand to multiple nodes) but at least it would allow Wolf to run on K8S I guess.

bsakaguchi commented

Author

Owner

@RyuunosukeDS3 commented on GitHub (Mar 1, 2025):

No prob. For me, specifically, this is nice cause i can have the configs in a gitops manner. And I also use argocd to deploy my apps.

@RyuunosukeDS3 commented on GitHub (Mar 1, 2025): No prob. For me, specifically, this is nice cause i can have the configs in a gitops manner. And I also use argocd to deploy my apps.

bsakaguchi commented

Author

Owner

@RyuunosukeDS3 commented on GitHub (Mar 19, 2025):

Jumping in to say that, for my sloppy ahh solution, if you update your nvidia drivers, you need to delete the kubernetes you need take down the manifests and uo then again. Not sure why, didn't really dig further into that but if anyone stumble upon this config and has this problem.

@RyuunosukeDS3 commented on GitHub (Mar 19, 2025): Jumping in to say that, for my sloppy ahh solution, if you update your nvidia drivers, you need to delete the kubernetes you need take down the manifests and uo then again. Not sure why, didn't really dig further into that but if anyone stumble upon this config and has this problem.

bsakaguchi commented