[GH-ISSUE #80 ] A UI for Wolf #57

Author

Owner

@brine0m commented on GitHub (Jun 24, 2024):

@ABeltramo, not sure if the above implies this but I would also suggest a rudimentary auth solution with 2 roles, admin and user. User would be able to login and pair devices and this would allow sharing of installed games between paired devices. Admin would do that and the aforementioned config / monitoring.

Also haven't read in depth of Sciter but are you intending for this to be a webapp or desktop app... just first few things I read it looks look as it uses mostly web languages but seems like its to dev desktop app

@brine0m commented on GitHub (Jun 24, 2024): @ABeltramo, not sure if the above implies this but I would also suggest a rudimentary auth solution with 2 roles, admin and user. User would be able to login and pair devices and this would allow sharing of installed games between paired devices. Admin would do that and the aforementioned config / monitoring. Also haven't read in depth of Sciter but are you intending for this to be a webapp or desktop app... just first few things I read it looks look as it uses mostly web languages but seems like its to dev desktop app

bsakaguchi commented

Author

Owner

@ABeltramo commented on GitHub (Jun 24, 2024):

not sure if the above implies this but I would also suggest a rudimentary auth solution with 2 roles, admin and user. User would be able to login and pair devices and this would allow sharing of installed games between paired devices. Admin would do that and the aforementioned config / monitoring.

Yep proper user management will definitely be added. First we'll have to plug everything together though..

Also haven't read in depth of Sciter but are you intending for this to be a webapp or desktop app... just first few things I read it looks look as it uses mostly web languages but seems like its to dev desktop app

The idea here is that Sciter will allow Wolf to render mutliple GUIs, ideally one per user/session without even having a Desktop and without requiring the user to jump over a browser (which might not be possible on some Moonlight platforms, like TVs/handelds ..).
Everything will be rendered inside Moonlight and controlled with the same mouse/kb/joypad devices.

Ideally this UI will also be easily developed by using the same frameworks as any webapp so that anyone can easily contribute!

@ABeltramo commented on GitHub (Jun 24, 2024): > not sure if the above implies this but I would also suggest a rudimentary auth solution with 2 roles, admin and user. User would be able to login and pair devices and this would allow sharing of installed games between paired devices. Admin would do that and the aforementioned config / monitoring. Yep proper user management will definitely be added. First we'll have to plug everything together though.. > Also haven't read in depth of Sciter but are you intending for this to be a webapp or desktop app... just first few things I read it looks look as it uses mostly web languages but seems like its to dev desktop app The idea here is that Sciter will allow Wolf to render mutliple GUIs, ideally one per user/session without even having a Desktop and without requiring the user to jump over a browser (which might not be possible on some Moonlight platforms, like TVs/handelds ..). Everything will be rendered inside Moonlight and controlled with the same mouse/kb/joypad devices. Ideally this UI will also be easily developed by using the same frameworks as any webapp so that anyone can easily contribute!

bsakaguchi commented

Author

Owner

@salty2011 commented on GitHub (Jun 25, 2024):

This sounds interesting, look forward to seeing it in action.

@salty2011 commented on GitHub (Jun 25, 2024): This sounds interesting, look forward to seeing it in action.

bsakaguchi commented

Author

Owner

@SamuelDudley commented on GitHub (Jul 15, 2024):

I was digging around a bit looking to see if it would be possible to use electron for the UI to simplify (maybe?) development and contributing and ran into this PR that I'm linking incase anyone finds it useful: https://github.com/electron/electron/pull/42001
Although the PR is marked as closed, it appears to be under active development by the author.

@SamuelDudley commented on GitHub (Jul 15, 2024): I was digging around a bit looking to see if it would be possible to use electron for the UI to simplify (maybe?) development and contributing and ran into this PR that I'm linking incase anyone finds it useful: https://github.com/electron/electron/pull/42001 Although the PR is marked as closed, it appears to be under active development by the author.

bsakaguchi commented

Author

Owner

@ABeltramo commented on GitHub (Jul 15, 2024):

Thanks for the pointer!
I've quickly skimmed thru it, but I can't quite figure out if electron supports running multiple instances from the same process or is it using global state like most of the other solutions? It would be nice to use a fully open source project instead of Sciter, it just seems like such a perfect fit for this use case..

@ABeltramo commented on GitHub (Jul 15, 2024): Thanks for the pointer! I've quickly skimmed thru it, but I can't quite figure out if electron supports running multiple instances from the same process or is it using global state like most of the other solutions? It would be nice to use a fully open source project instead of Sciter, it just seems like such a perfect fit for this use case..

bsakaguchi commented

Author

Owner

@SamuelDudley commented on GitHub (Jul 16, 2024):

Thanks for the pointer! I've quickly skimmed thru it, but I can't quite figure out if electron supports running multiple instances from the same process or is it using global state like most of the other solutions? It would be nice to use a fully open source project instead of Sciter, it just seems like such a perfect fit for this use case..

I agree Sciter seems to be a great technical fit and would likely be very performant.

Digging a bit deeper on the electron side it does appear you can create multiple windows from the same process: https://jasonsturges.medium.com/multiple-window-electron-app-9dbffde8ce95

I'll dig into this some more and let you know if it goes anywhere

@SamuelDudley commented on GitHub (Jul 16, 2024): > Thanks for the pointer! I've quickly skimmed thru it, but I can't quite figure out if electron supports running multiple instances from the same process or is it using global state like most of the other solutions? It would be nice to use a fully open source project instead of Sciter, it just seems like such a perfect fit for this use case.. I agree Sciter seems to be a great technical fit and would likely be very performant. Digging a bit deeper on the electron side it does appear you can create multiple windows from the same process: https://jasonsturges.medium.com/multiple-window-electron-app-9dbffde8ce95 ![1_l5OfpO58fw5qs4TI9vSX_Q](https://github.com/user-attachments/assets/6544e1a3-fe16-41a7-b936-f15992d06843) I'll dig into this some more and let you know if it goes anywhere

bsakaguchi commented

Author

Owner

@austonpramodh commented on GitHub (Aug 29, 2024):

I am curious, is the "dream solution" going to allow users to stream without moonlight?

This is where I was wondering why we would need Electron or Sciter, Why not just use raw HTML, CSS, and JS?

I might be missing something here I guess.

@austonpramodh commented on GitHub (Aug 29, 2024): I am curious, is the "dream solution" going to allow users to stream without moonlight? This is where I was wondering why we would need Electron or Sciter, Why not just use raw HTML, CSS, and JS? I might be missing something here I guess.

bsakaguchi commented

Author

Owner

@SamuelDudley commented on GitHub (Aug 29, 2024):

I am curious, is the "dream solution" going to allow users to stream without moonlight?

This is where I was wondering why we would need Electron or Sciter, Why not just use raw HTML, CSS, and JS?

I might be missing something here I guess.

Happy to be corrected, but I think it's this point which is steering the Sciter / electron discussion:

Expose direct access to the raw framebuffer where the UI is rendered so that we can feed it directly into the video
encoding pipeline.

Both Sciter and electron (maybe) are a way to generate a framebuffer that can be mixed into the existing wolf stream to overlay the UI.

@SamuelDudley commented on GitHub (Aug 29, 2024): > I am curious, is the "dream solution" going to allow users to stream without moonlight? > > This is where I was wondering why we would need Electron or Sciter, Why not just use raw HTML, CSS, and JS? > > I might be missing something here I guess. Happy to be corrected, but I think it's this point which is steering the Sciter / electron discussion: > Expose direct access to the raw framebuffer where the UI is rendered so that we can feed it directly into the video encoding pipeline. Both Sciter and electron (maybe) are a way to generate a framebuffer that can be mixed into the existing wolf stream to overlay the UI.

bsakaguchi commented

Author

Owner

@austonpramodh commented on GitHub (Aug 29, 2024):

Gotcha makes sense. But I dont understand why UI needs to be rendered(or generate raw framebuffer) instead of having traditional Rest APIs and client web pages. Unless the idea is to make streaming work on the webpage, like may be Xbox cloud gaming.

I am assuming @ABeltramo has some idea going on with this.

@austonpramodh commented on GitHub (Aug 29, 2024): Gotcha makes sense. But I dont understand why UI needs to be rendered(or generate raw framebuffer) instead of having traditional Rest APIs and client web pages. Unless the idea is to make streaming work on the webpage, like may be Xbox cloud gaming. I am assuming @ABeltramo has some idea going on with this.

bsakaguchi commented

Author

Owner

@ABeltramo commented on GitHub (Aug 29, 2024):

But I dont understand why UI needs to be rendered(or generate raw framebuffer) instead of having traditional Rest APIs and client web pages.

It's actually fairly simple, not every Moonlight client has browser support! Think of TVs, handhelds (Switch, 3DS, PSP), etc.
Having our custom UI rendering in Moonlight will allow us to instantly support every client. Also, the flow will not be very smooth right? You have to switch back and forth from the browser to Moonlight, or even worse: pickup your phone to access the browser because you can't do that from your TV!
On top of that you get joypad/remotes support for free via Moonlight because we can't assume that a mouse and keyboard will always be available to navigate the web UI.

This "UI" is still a tad up in the air, we were discussing this just earlier today with @shocklateboy92 and @salty2011 in a call; I think we are going to approach this in a series of steps:

Step 1: expose an API for Wolf

I'm working on it here: https://github.com/games-on-whales/wolf/pull/101
The idea is that we'll expose all the events that are currently powering Wolf over a Linux socket. The type of events and what would be possible it's not yet flashed out but ideally most of the Wolf components will be available over it, ex:

Moonlight pair requests
Audio/Video encoding and pipelines
Virtual devices: joypads, mouse, keyboard, pen, etc.

All this stuff would be initially just to listen for those events when Moonlight clients trigger them, but, potentially, we can also allow applications to trigger them without any Moonlight session. Ex: create a virtual joystick and send inputs from a different application; start a video pipeline that you can view from VLC...

Step 2: create an "admin" web UI

Working on it here: https://github.com/games-on-whales/wolf/pull/105 (but possibly moving it to a separate dedicated project page)

Very early day and needs to be fleshed out, but ideally this would allow you to manage and pair Moonlight clients (without going through the logs), change the settings (without manually editing the config.toml), see the logs...
This will communicate with Wolf via the API created in Step 1 and expose a web UI.

Step 3: create a "user/running" UI

This is what this issue was originally talking about and what I'd really like to build.
In my head the flow is as follows:

You connect via Moonlight to Wolf
From there you pick up a single app that it's available
Wolf starts the stream, instead of running the Docker apps straight away it instead shows a UI (you'll be able to navigate the UI both with mouse and keyboard and joypads all via Moonlight)
- It opens up as a netflix-like list of users:
  - If you pick another user that is running something you can join the session and play together (or even just watch)
  - Otherwise, you pick another user and it just show a list of apps; same as it is right now in Moonlight and it'll eventually start the app that you pick there
  - In both cases, when a selection has been made Wolf will switch the video/audio sources in the pipeline from the UI to the actual Docker container running.

It's an intermediary step compared to what we have at the moment so that we can properly support users.

Now, the dream solution is that this UI renders directly in Wolf; this way we don't have to run a Desktop, an additional Docker container and all that stuff that can fail, but we can provide feedback directly to the users.

But, that turns out to be more complicated than expected. So I'm leaning towards a more simple approach: let's just run a fullscreen browser or an Electron app that shows this UI and communicates with Wolf via the APIs. We can build all that and still re-use everything if we ever want to actually embed it in Wolf without the additional container + browser.

I hope this helps clarify it a bit, feel free to share feedback and ideas; they are very welcome!

@ABeltramo commented on GitHub (Aug 29, 2024): > But I dont understand why UI needs to be rendered(or generate raw framebuffer) instead of having traditional Rest APIs and client web pages. It's actually fairly simple, not every Moonlight client has browser support! Think of TVs, handhelds (Switch, 3DS, PSP), etc. Having our custom UI rendering in Moonlight will allow us to instantly support every client. Also, the flow will not be very smooth right? You have to switch back and forth from the browser to Moonlight, or even worse: pickup your phone to access the browser because you can't do that from your TV! On top of that you get joypad/remotes support for free via Moonlight because we can't assume that a mouse and keyboard will always be available to navigate the web UI. This "UI" is still a tad up in the air, we were discussing this just earlier today with @shocklateboy92 and @salty2011 in a call; I think we are going to approach this in a series of steps: ## Step 1: expose an API for Wolf I'm working on it here: https://github.com/games-on-whales/wolf/pull/101 The idea is that we'll expose all the events that are currently powering Wolf over a Linux socket. The type of events and what would be possible it's not yet flashed out but ideally most of the Wolf components will be available over it, ex: - Moonlight pair requests - Audio/Video encoding and pipelines - Virtual devices: joypads, mouse, keyboard, pen, etc. All this stuff would be initially just to listen for those events when Moonlight clients trigger them, but, potentially, we can also allow applications to trigger them without any Moonlight session. Ex: create a virtual joystick and send inputs from a different application; start a video pipeline that you can view from VLC... ## Step 2: create an "admin" web UI Working on it here: https://github.com/games-on-whales/wolf/pull/105 (but possibly moving it to a separate dedicated project page) Very early day and needs to be fleshed out, but ideally this would allow you to manage and pair Moonlight clients (without going through the logs), change the settings (without manually editing the `config.toml`), see the logs... This will communicate with Wolf via the API created in Step 1 and expose a web UI. ## Step 3: create a "user/running" UI This is what this issue was originally talking about and what I'd really like to build. In my head the flow is as follows: - You connect via Moonlight to Wolf - From there you pick up a single app that it's available - Wolf starts the stream, instead of running the Docker apps straight away it instead shows a UI (you'll be able to navigate the UI both with mouse and keyboard and joypads all via Moonlight) - It opens up as a netflix-like list of users: - If you pick another user that is running something you can join the session and play together (or even just watch) - Otherwise, you pick another user and it just show a list of apps; same as it is right now in Moonlight and it'll eventually start the app that you pick there - In both cases, when a selection has been made Wolf will switch the video/audio sources in the pipeline from the UI to the actual Docker container running. It's an intermediary step compared to what we have at the moment so that we can properly support users. Now, the dream solution is that this UI renders directly in Wolf; this way we don't have to run a Desktop, an additional Docker container and all that stuff that can fail, but we can provide feedback directly to the users. **But**, that turns out to be more complicated than expected. So I'm leaning towards a more simple approach: let's just run a fullscreen browser or an Electron app that shows this UI and communicates with Wolf via the APIs. We can build all that and still re-use everything if we ever want to actually embed it in Wolf without the additional container + browser. ----- I hope this helps clarify it a bit, feel free to share feedback and ideas; they are very welcome!

bsakaguchi commented

Author

Owner

@austonpramodh commented on GitHub (Aug 29, 2024):

Yup, it does clarify, Now I understand. Thank you. Super excited for this.

@austonpramodh commented on GitHub (Aug 29, 2024): Yup, it does clarify, Now I understand. Thank you. Super excited for this.

bsakaguchi commented

Author

Owner

@ABeltramo commented on GitHub (Oct 14, 2024):

Quick status update: in https://github.com/games-on-whales/wolf/pull/101 I've added a fully functional REST API, you can see the docs for it.

The plan now is to start having fun with Electron (or similar) and build the actual user UI that will then trigger stuff via the Wolf socket..

Stay tuned! 😉

@ABeltramo commented on GitHub (Oct 14, 2024): Quick status update: in https://github.com/games-on-whales/wolf/pull/101 I've added a fully functional REST API, you can see the [docs](https://games-on-whales.github.io/wolf/stable/dev/api.html) for it. The plan now is to start having fun with Electron (or similar) and build the actual user UI that will then trigger stuff via the Wolf socket.. Stay tuned! :wink:

bsakaguchi commented

Author

Owner

@SamuelDudley commented on GitHub (Oct 14, 2024):

The API looks amazing! Out of interest I spent an afternoon looking at open source lighter weight options (something similar to Sciter) and found egui (rust) https://github.com/emilk/egui/tree/master

Most of the integrations seem to support rendering to a window, which defeats the point of it being lightweight for the purposes of wolf. The skia I integrations https://github.com/lucasmerlin/egui_skia has an example that can render to an image while also being headless, which seems closer to what you are after?
Then there is the whole issue of it being written in rust rather than cpp. From an API perspective you have solved that issue :) but I'm guessing there would also need to be some gstreamer integration that would be a touch point between rust and cpp?

@SamuelDudley commented on GitHub (Oct 14, 2024): The API looks amazing! Out of interest I spent an afternoon looking at open source lighter weight options (something similar to Sciter) and found egui (rust) https://github.com/emilk/egui/tree/master Most of the integrations seem to support rendering to a window, which defeats the point of it being lightweight for the purposes of wolf. The skia I integrations https://github.com/lucasmerlin/egui_skia has an example that can render to an image while also being headless, which seems closer to what you are after? Then there is the whole issue of it being written in rust rather than cpp. From an API perspective you have solved that issue :) but I'm guessing there would also need to be some gstreamer integration that would be a touch point between rust and cpp?

bsakaguchi commented

Author

Owner

@ABeltramo commented on GitHub (Oct 14, 2024):

Thanks for looking into it, I think I'm really going to keep it simple to get the ball rolling.

We have a Wayland compositor already, I'll just install Electron in the Wolf container and fire it up when the session starts. Then, via the APIs, the Electron UI can control additional apps, and I'll just swap the "main focused" window from our comp to the newly started app.

I'm going to try to keep it simple this time, we can always overcomplicate things later (as usual).

@ABeltramo commented on GitHub (Oct 14, 2024): Thanks for looking into it, I think I'm really going to keep it simple to get the ball rolling. We have a Wayland compositor already, I'll just install Electron in the Wolf container and fire it up when the session starts. Then, via the APIs, the Electron UI can control additional apps, and I'll just swap the "main focused" window from our comp to the newly started app. I'm going to try to keep it simple this time, we can always overcomplicate things later (as usual).

bsakaguchi commented

Author

Owner

@SamuelDudley commented on GitHub (Oct 14, 2024):

Thanks for looking into it, I think I'm really going to keep it simple to get the ball rolling.

We have a Wayland compositor already, I'll just install Electron in the Wolf container and fire it up when the session starts. Then, via the APIs, the Electron UI can control additional apps, and I'll just swap the "main focused" window from our comp to the newly started app.

I'm going to try to keep it simple this time, we can always overcomplicate things later (as usual).

Yep, makes sense!

@SamuelDudley commented on GitHub (Oct 14, 2024): > Thanks for looking into it, I think I'm really going to keep it simple to get the ball rolling. > > We have a Wayland compositor already, I'll just install Electron in the Wolf container and fire it up when the session starts. Then, via the APIs, the Electron UI can control additional apps, and I'll just swap the "main focused" window from our comp to the newly started app. > > I'm going to try to keep it simple this time, we can always overcomplicate things later (as usual). Yep, makes sense!

bsakaguchi commented

Author

Owner

@salty2011 commented on GitHub (Feb 13, 2025):

Hey All,

Just wanting to reference here that there is the start of an admin web ui located in this repo
https://github.com/games-on-whales/wolfmanager

Still early days but do have client pairing / unpairing working and a means to track clients against a user. Welcome feedback and suggestions

@salty2011 commented on GitHub (Feb 13, 2025): Hey All, Just wanting to reference here that there is the start of an admin web ui located in this repo https://github.com/games-on-whales/wolfmanager Still early days but do have client pairing / unpairing working and a means to track clients against a user. Welcome feedback and suggestions

bsakaguchi commented