mirror of
https://github.com/games-on-whales/wolf.git
synced 2026-10-15 21:12:08 +00:00
[GH-ISSUE #69] Feature Idea: Deduped Steam Libraries #47
Labels
No labels
🧠 1 - 🤖 999
AMD
bug
bug
cannot reproduce
documentation
duplicate
easy
enhancement
enhancement
enhancement
fixed?
fixed?
good first issue
hard
help wanted
Intel
invalid
LXC
medium
moonlight-protocol
Multi GPU
nixos
not a bug
nvidia
pull-request
question
wontfix
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
games-on-whales/wolf#47
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @duskmoss on GitHub (May 2, 2024).
Original GitHub issue: https://github.com/games-on-whales/wolf/issues/69
It seems likely to cause problems to use the same steam library folder for multiple clients, at least at the same time.
So something that could be useful is running dedupe remove against the container folders for various clients. If mounted on BTRFS this could save a few gigabytes in the base image and many more in game downloads.
It would also be cool if there could be a main Steam container that installs all the games and is running when any steam containers for clients are open so a new client can install by steams feature to transfer from it over the "local-network". Which in this case can be a really quick bridge between containers.
Hard part on the main Steam container part might be identifying what a client steam container has installed to then installing on the "master" container. Maybe we can just look at "steamapps" grab the ids from the list of appmanifest_.acf files and then send those to steamcmd? But I'm not sure if a headless steam manage steamcmd implements the local network transfer stuff. Perhaps with a full fledged steam client we can just copy the files in? but I'm not sure if we'd need to trigger it to recognize them and offer them to other clients somehow.
@ABeltramo commented on GitHub (May 2, 2024):
Not sure if this is something that we could add to Wolf or it just needs to be properly setup on the host..
The
steamimage runs the full client, so everything should be available as is@duskmoss commented on GitHub (May 2, 2024):
It could be something that Wolf doesn't do, but its a lot of state that specifically is useful to wolf, and seems to have advantages to be tied in to its lifecycle. Integrating some sort of plan here to avoid wasted diskspace and redownloading games for each streaming device will also make users less likely to just throw a single drive in mounts and put steam library on there. But maybe that would just work well enough, it feels like it's asking for race conditions messing everything up in a multi user environment.
So for the goal of 1) reducing disk usage 2) reducing need to redownload:
This part of the feature is fairly easy to do outside, just have a cron job that dedupes the wolf folder once a day. However it makes a lot of sense to me if if Wolf schedules it when a container shuts down, to dedupe files added during that session.
| The steam image runs the full client, so everything should be available as is
Right, but the steamcmd comment wasn't about a image spun up for a client. Those do already transfer from local network, and if you have Steam running on the host they will transfer from there, though not as fast as could be done between containers on a shared internal bridge.
Building on that, if wolf can have with a container that's always on, and can be scripted to install games, it could be seamless that people don't have to download games multiple times if multiple moonlight devices play the same game on the wolf server. Another thing that could potentially be scripted externally. But here there is a risk partly installed games being transferred depending on implementation Wolf doing it when a client shuts down would make it somewhat atomic.
SteamCMD is easier to script which is why I wonder if it can do the local network transfers. Another image like is spin up for clients would for sure work to serve games to the others, but I'm not sure sosohow to script it installing games. And wolf probably would need a way to connect to it's display to administer it (set it to allow all users on the local network to transfer games at least)
Been thinking about a potentially easier path for 2) even if it does have some disadvantages:
Another approach could be adding a LanCache container to wolfs network. I stopped running one at home because I had flakiness with steam deciding to bypass the container, but I hope it would be more reliable in the virtual network.Another downside is that update churn can cause extra disk space spent on no longer relevant files. It also wouldn't dedupe as cleanly as a "main" steam container approach because it stores compressed chunks and not the specific files. On the plus side it wouldn't need any sort of system to recognize what clients installed and install them on a master container. It would be a lot less moving parts, just some docker network design.
There would still be advantages to having LanCache run by wolf and not set up externally. Mostly so it can be part of an internal bridge, because networking between containers is much faster that way than on the default bridge, DNS is handled better, and wolf can ensure DNS is set on the steam containers to use LanCache without a user having to mess with their local networks settings.
@ABeltramo commented on GitHub (May 2, 2024):
The more I read about this the more I think this is getting outside the scope of Wolf but it's a very interesting discussion and you brought up a lot of good points, there's lots to digest so here are a few random counter points:
duperemove: I don't like having to implement filesystem specific code (nor forcing such a personal choice to users), this should be scripted outside Wolf. We could easily add a pre/post session script hook to facilitate that.config.toml(you can manually overrideNetworkingConfig,NetworkModeand so on already in there).I'd be very interested in digging further into LanCache + Deduplication even if it's just going to end up being documentation and not actual code. I always wanted to checkout LanCache ony my homelab this might be the right time to actually do it!
@duskmoss commented on GitHub (May 2, 2024):
Thanks for reading and engaging!
I do like the idea of hooks and they will be useful for other stuff. I'm still tempted to say it should be something on by default for people but I can live with it not being :)
mm yeah. I suppose mostly assuming it was different client devices but same steam account, but this would be a hard part. Hmm.
I think with Lancache even if it's just publishing a docker compose example of it would massively help people out. Running it on a different machine will be much slower without multi gig networking.
I think something may still need to be done internally for dns, to make it easy to keep the traffic to the internal docker network if running one there.
I might still explore scripting the main steam container some, because I don't trust lancache to work reliably :) If it ends up feeling easy I might have you take a look and see if you want to integrate. Or just figure out how to interface it with the hooks you add.
@duskmoss commented on GitHub (May 3, 2024):
I'm realizing the thing I'd actually more want to hack on is probably something to make reusing containers easier. I don't particularly need or want each device I connect from to have its own set of containers.
Perhaps a user based collection. Or manual selection. Any thoughts for me on forking, vs preparing a PR or pointers on where I might want to start in the code?
@ABeltramo commented on GitHub (May 4, 2024):
This is a valid point and I've been thinking about it for quite some time, I believe this is part of a more general feature: share sessions (aka: co-op for multiple users or as a specific case for a single user share the container between devices).
I think there are different possible approaches to this:
Whilst the second approach might be a bit more complex it feels that it's the best way of achieving this:
Thoughts?
@vinibutturi commented on GitHub (May 31, 2024):
That would be really cool 🤩
If I may add my 2 cents, this is how I see it:
is that similar to what you're thinking or am I dreaming too much?
@ABeltramo commented on GitHub (May 31, 2024):
That's exactly what I have in mind, it's not a simple task, but it's definitely doable! Anyone wants to help out? 😅
@vinibutturi commented on GitHub (Jun 1, 2024):
I would love to help, but I'm gonna need some guidance on all the tech involved.
I got 10+ years experience with ERP development (X++ language) so I'm not starting from scratch, but this is completely new game to me.
@ABeltramo commented on GitHub (Jun 2, 2024):
The good news is that this will likely be a separate project from Wolf, so it can be done in any language (probably not C++).
It could even be just a simple GUI program that will run fullscreen, and that will communicate back with Wolf when a selection of user+app has been made.
I have to define the requirements a bit better..
@vinibutturi commented on GitHub (Jun 2, 2024):
yeah, keep it as a separated project is the way to go.
What do you think about moving this conversation to discord? More convenient and we can brainstorm it further :)
@rty813 commented on GitHub (Jun 6, 2024):
My method is to mount the .steam directory. If it's just for personal use, you can use this solution.

@a-priestley commented on GitHub (Jun 25, 2024):
my toml block for steam:
The relevant bits are in the
mountdirective./mnt/gamescontains my libraries for every platform (or no platform). The entry withshortcuts.vdfcontains configs specifying any non-steam games I've added in steam on the host machine (your steam user ID will be different), and the entry withlibraryfolders.vdfis the config for specifying extra library directories, such as when you might keep your titles installed on a separate storage drive like myself. The mount for the library itself is mapped one-to-one - it's identical on the host and client, as that's how the vdf files are configured.As an extra, I have an entry there for the
compatabilitytools.ddirectory, so that I can pass in custom tools from the host, such as ge-proton or steamtinkerlaunch.This isn't deduplication, but it works well.
EDIT:
I also have written this guide for setting up proper block-level deduplication using bees on top of btrfs, exposing storage shares over iSCSI. For this setup, the host and client are on the same machine. Potentially, each separate wolf user can own a discrete backstore pointed at the same disk image.
It's been a while since I've experimented with this, but it came with its fair share of issues (kernel I/O faults in my case), which may or may not have been due to some faulty memory I had installed in my machine at the time. I'm betting that there are improvements to be made.
@salty2011 commented on GitHub (Jan 5, 2025):
@a-priestley > thanks for the guide, will take a look, would be curious to know if you have done any changes to the various vdfs.
Just reading over this whole issue @duskmoss I dont believe there are any pending actions here today as this sorta falls outside of the scope of wolf at this stage. However there have been efforts by several others to get shared steam libraries working so that multiple copies dont exist. There are two challenges to over come, user stage not being written to the game install location. and providing a single copy of game installations. Several community members have put forward ideas that need further investigation.
I am closing this issue given no pending actions and age.
@salty2011 commented on GitHub (Jan 5, 2025):
No further action to be taken at this stage