Slow Initial Downloads (Xbox)

[GoTeamScotch]

Issue

I have steamcache and steamcache-dns running for my Xbox One downloads. The caching process is working, except I'm seeing slow download speeds on the initial download of un-cached data. I have gigabit internet from my ISP. Without caching, the download goes at 250mbps. With caching, it seems to be about 1/3 of that. Once a file is in cache, my Xbox will download it at about 250mbps once again. This result has been very consistent over the past 3 days. I understand that once the cache is primed, then the benefits become apparent. My thread here is all about the initial download though.

Read Dead Redemption 2 (80GB), for example, would take 3.5 to download through steamcache. That download would normally take under 1 hour without caching. I expected some overhead, but this seems unusual. Is this typical?

Also, I used Wireshark to inspect traffic and I don't see any obvious issues there. I saw that one of the links my Xbox was fetching was this link. I can open that in Firefox and begin downloading at 240mbps to 400mbps. That means my OS can fetch files from Xbox's servers at good speeds.

So... fetching is fast. Serving cached data is fast. But downloading fresh data is very slow.

Setup

steamcache was setup using this command:
sudo docker run --restart unless-stopped --name gamecache -v /mnt/gamecache:/data/cache -v /mnt/gamecache:/data/logs -p 192.168.1.223:80:80 steamcache/generic:latest
I am running this from a virtual machine. The VM is running Ubuntu 18 x64. Host is Windows 10. CPU i7 7700HQ - 4 cores at 2.8ghz. 16GB RAM. The VM has use of 2 cores and 4GB of RAM (will be more in production later). The VM and the cache/data folder are stored on an external SSD. I experimented with running Ubuntu natively but I didn't see a difference in speeds.
Here's how my system resources look while downloading:

Disk Z is where the VM and game cache are stored. At this same time, the Xbox itself reports about 55mbps.

DNS Configuration

sudo docker run --name steamcache-dns -p 192.168.1.223:53:53/udp -e STEAMCACHE_IP=192.168.1.223 -e USE_GENERIC_CACHE=true -e LANCACHE_IP=192.168.1.223 -e UPSTREAM_DNS=8.8.8.8 steamcache/steamcache-dns:latest

Output of container(s)

Not sure how to get this. When I run the docker commands at first, I see output from the DNS setup command, but I'm not sure how to get back to it after a reboot.

[VibroAxe]

Hi @GoTeamScotch
We've had some known issues with steam downloading fast, but never with xbox but it's possibly related.. Do you have a test setup you would be able to try the following image with: steamcache/generic:proxy_read_timeout we've had some rumours of a nginx config change making some improvements. We are unsure if these changes will make any difference but would appreciate your feedback

[GoTeamScotch]

@VibroAxe
I loaded up another docker instance using that suffix ("proxy_read_timeout") and did not see a different in initial download speeds. They're hovering around the same level, 50mbps.
I assume this docker image contains the nginx changes you were referencing? Or is there a deeper change I should make manually beyond what's included?

[VibroAxe]

@GoTeamScotch is this only affecting xbox downloads?

[GoTeamScotch]

@VibroAxe
I just loaded up Steam and I see initial downloads at about 3MB/s using steamcache, and 18MB/s without.
https://i.imgur.com/w86khZv.png (top: with, bottom: without)

On a related topic, I just noticed your pull request and am curious if that's what's affecting me as well. I believe Xboxes pipe their downloads in parallel, so that would make sense for it to resolve my issue as well. I'd definitely be willing to test!

[VibroAxe]

interesting, it might well do. The 69 PR should definitely improve your steam performance. I don't think we see enough xbox only downloads to have seen this problem elsewhere. Adding some more ip's to xbox / steamcache-dns is worth trying, let us know if you have any success?

Are you only seeing this on uncached downloads?

[VibroAxe]

also related, it looks from your docker command that you are only running a single generic instance. Generic is designed to run one container per cdn (xbox,steam,blizzard etc). If you want a single container cache solution then I'd recommend using steamcache/monolithic instead (it's specifically designed to stop collisions between game networks)

[GoTeamScotch]

Well I'll be a good test candidate then! I plan on deploying this at a LAN center that's mostly Xbox based (2 dozen Xboxes, a dozen PCs). The initial download speed issue has made me put this on the backburner since November, but if that's resolved, I can see deploying this out pretty soon. I should have feedback later this week about that.

Yes, I knew about the collisions issue, but so far I've just been focusing on the Xbox aspect of it all during testing. I'll look into Monolith before I deploy it in the real world. Thank you for the tip and for your PR.

[yuko1225]

Hi @GoTeamScotch . I think it's might caused by the "slice" mechanism. I'm not the developer here but I think you may try “steamcache/generic:noslice” and see if you can get the maximum speed. This is just to rule out the possibility of slice cause it. So don't use it on production environment.

Here is my guess: because the “slice” module breaks a big file into small segments(1M) and request these segments piece by piece(not parallel). It become a lot harder to reach the maximum bandwidth due to the "slow start" in TCP congestion control. And it also adds a lot of overhead in protocol handshake(tcp&http).
Tuning up the slice size a little bit might be a workaround even though it has some side effect in TTFB(b/c of "proxy_cache_lock_timeout 1h").

[GoTeamScotch]

I finally had some time to tinker with this again. I've switched to using Monolithic and experimented with adding IP addresses. Download speeds appeared to go up from 50mbps to about 65mbps, but it's not enough of a jump to where I'd say it was related. I tried using 2 IPs, then 4, then 10. It didn't seem to make much of a difference.

By the way, the way Xbox reports download speeds is a bit odd. In Linux I can watch network traffic going to my Xbox and see 125mbps and the Xbox itself will report much lower speeds of ~60mbps. I'm not sure why this is, but I'm going to trust Linux's measurement instead, so any speeds I reference from here on out will be based on Linux's measurements (not Xbox's).

Slicing seems to greatly impact speeds. Using the default 1MB slicing, I see around 125Mbps (same as before). If I disable slicing (or use the noslice container), the Xbox UI will appear to get stuck trying to start the download while at the same time my server shows lots of network activity and large files being downloaded. If I changed slicing size to 100m, speeds jump up to about 300mbps with spikes even higher (400mbps+), which is great! This is about as fast as I get without caching on my internet connection. Assuming there's no surprises, this may have resolved my issue. I will continue to test but so far things are looking very good.

[unspec]

The latest version of monolithic/generic now supports changing the slice size used by nginx. We've found that increasing from 1m to 8m offers a small performance boost to specific use cases (single user initial downloads of blizzard games in particular). See http://lancache.net/docs/advanced/tuning-cache/#tweaking-slice-size for information on how to make use of this.

Please note that it does come with some potential downsides (discussed in the above link) and will invalidate any already cached data on your cache if you change the value.

To tidy up the issues, if you choose to test this please post any feedback on this issue: #100

If you need any other support please see http://lancache.net/support/ or open a new issue.