Update on May 16, 2022
Manifest request codes have been enforced since 14th, which likely means they are staying for good now.
Update on April 1st, 2022
After one of the recent Steam beta client updates, download_depot command once again allows downloading older manifests without the client-side check that prevented it. This likely indicates that will be enforced by manifest request codes in the future.
Update on March 22nd, 2022
For a couple of hours, manifest request codes were enforced by the Steam CDN and were required for all requests. Requests without providing one returned 401 Unauthorized error.
However since originally writing this post, manifest request codes are granted for older manifest ids that are no longer in the appinfo, which should still allow downloading older versions of games, given that developers do not remove them.
Update on September 28th, 2021
Valve gave a statement to PC Gamer:
"We are actually not planning to disable downloading old builds. What we are working on is an approach on handling edge cases involving unowned content, and helping partners more easily take down builds that need to be removed for things like copyright issues. We’ll have more to share on that work when it’s ready to ship."
As of right now Valve's statement isn't in line with these changes as manifest request codes are still not granted for any but the most recent manifest version in the appinfo. We look forward to seeing changes to allow this in future beta builds. As a sidenote, Valve's statement goes against the fact they removed the functionality to download older depot versions from inside the Steam client last year making it only able to do so in third-party applications. We also noted that until May 2019, there already was a ownership check when downloading manifests. We hope that with this post and the press coverage it received it's been made apparent that there is additional value in allowing users to download older builds.
Original post
Steam Client Beta released on September 23, 2021 included a new method called ContentServerDirectory.GetManifestRequestCode and it is actively being used by the beta client.
This method takes app id, depot id, manifest id, branch name, and the branch password. All of these parameters have to match in the current app info for it to return a valid manifest code, if something mismatches, or you don't own the game it will return AccessDenied result. When a non-zero code is acquired, it is appended to the manifest download url. If a valid request code is provided, the manifest can be downloaded. Otherwise the content server returns an unauthorized error.
https://{cdn}/depot/{depotId}/manifest/{manifestId}/5 - before
https://{cdn}/depot/{depotId}/manifest/{manifestId}/5/{manifestRequestCode} - afterAs of now, if the manifest request code is omitted from the download url, it still works, as the Steam stable client does not have this just yet. We do not know if Steam CDN will enforce this requirement for all games (if at all), or what other requirements may be.
We believe this is only limiting access for consumers, and not actually deleting old builds from their storage. Developers can see all of their previous builds on the partner site, and rollback to any of it.
All information and conclusions in this post are subject to change as this feature is new to the Beta client and might still change.
Back in February 2020 they changed download_depot command in Steam client to perform similar checks locally, which meant you had to resort to using a third-party utility such as the Depot Downloader or steamctl to still be able to download older versions.
Before May 2019 there was a mechanism similar to this called "CDN auth tokens" which authenticated downloads by appid and depotid, but not by manifestid. This was effectively an ownership check, but still allowed downloading older game versions. After we noticed this was no longer being enforced, we submitted this to Valve's HackerOne bug tracker, and they said this behaviour was by design.
Read our previous blog post on how the Steam download system works. In short, a depot decryption key is still required to decrypt the filenames in these manifests. SteamDB does this by either owning the game or people using our token dumper software which acquires a key on behalf of them.
As for LAN caches, this should not affect them, however they might want to ignore the request code part to increase cache hits.
Some of the observations we concluded for GetManifestRequestCode
- Returns a 64-bit integer for successful requests, EResult.AccessDenied otherwise
- Checks for app ownership
- Checks for correct appid, depotid and manifestid provided
- Manifest ID has to exist in the appinfo for the requested branch
- Requesting older Manifest ID returns AccessDenied (for example, previous CS:GO update)
- Requesting Manifest ID with incorrect branch returns AccessDenied
- Codes rotate every five minutes, still valid on the CDN for ten minutes after that
- Identical codes are returned to all users
Why people use this
There are many reasons why one would want to download earlier builds of games on Steam. To name a few, modding, and speed running, preserving game versions, and rolling back broken or controversial updates. Since Steam almost never deleted old builds from their servers, it allowed customers to legally obtain these older game builds.
This "undocumented" feature of Steam may be seen as a downside from developers and publishers side, especially when it could be used to download pre-release builds if one had a Manifest ID of these builds. It is not unheard of that people were able to acquire Denuvo-free builds because a developer did not upload an executable with Denuvo before the game's release. Valve's own games have been affected by this as well, where removed content/developer files were still present in pre-release builds.
As it is possible to track Manifest IDs during preloads, they may be retroactively downloaded and decrypted after the game is fully released and the decryption key becomes available.
Some game developers, such as developers of Euro Truck Simulator 2 provide public branches containing previous game versions for modders to use. This method has a downside because there is a limit of 25 branches.
Perhaps Valve should offer game developers an option to just delete old builds if they choose to do so?
What this means for SteamDB
We will still be able to track the Manifest ID changes for all games, but accessing the file lists and tracking its changes will require our bot to actually own these games. You can donate your spare Steam keys to our bot here.
This change will impact how SteamDB operates, and we will no longer be able to fully track file lists of all games on Steam, which affects the following features:
- Depots page which shows the file list, as well as the history for any changes in it
- Patch notes pages which automatically show what files were changed in a particular update
- Technologies page which automatically detects and categorizes engines and SDKs used by all games on Steam
SteamDB does not download older manifests, these features operate entirely on the access to the latest ones (as specified by the app info).
These manifest files are raw protobufs which may be deserialized with a method in the SteamKit library. The filenames are still encrypted, which requires game ownership to obtain and decrypt.
Merely having the manifest files does not solicit piracy because you do not need any special tools to obtain them, it is enough to get the Manifest ID from app info (app_info_print command in the Steam console) and then constructing an url to download it.
In a sense, it is similar to torrent files that contain metadata about files and folders to be distributed. Manifests list the files and individual chunks to construct these files. Unlike a torrent file, after downloading said chunks, you still need a decryption key (and thus own the game to obtain the key) to reconstruct the real file.
You can backup manifest files yourself from the Steam client's depotcache folder. Depot Downloader also stores manifests as it downloads, however it is saved in its own format. The tool may be updated in the future to support Steam's manifest format.