The tale of the invisible gryph-chargers
During the development of Warhammer Age of Sigmar: Storm Ground, there was a bug where part of a unit was invisible. Specifically the Vanguard Palladors, who ride “Gryph-chargers” like horses. So these soldiers were floating around.
The game would run and function correctly, but their “mount” (the Gryph-chargers) was not visible at all. This only happened in the client build, not when playing or previewing in the editor (this was a late version of Unreal Engine 4).
No obvious error or warning messages.
These mounts were set up as part of a “wargear” system, which allowed them to be added to the unit as a component (they were equipped as a mount). This called a special codepath that overrode how the unit was displayed, since a mount was more involved than a different sword or shield. Given how specialized this was, it was a prime suspect for where the bug could be.
Unreal has a variety of ways and systems to load assets, as well as the timeline to load assets. Often they are scheduled to be loaded by other threads, and so tracing the entire code execution from load request to load fulfilled can be very tricky.
Units were preloaded at launch, traversed for default wargear, and everything loaded. There is a special set of circumstances where some assets are loaded eventually successfully, but they were flagged by some system as “known missing” (KnownMissingPackage). This “known missing” list is used later on to ignore subsequent load attempts. So if something fit this criteria, it could load once, but then if it was unloaded (no active references), it would never be able to load again. This was happening to the mount.
This bug took me days and many hours to track down and fix. My lessons were:
- Don’t underestimate the potential for engine loading code to have unexpected consequences, like can-load-once-but-never-again.
- Consider not tolerating missing assets from loading (should the game have been able to function with the invisible mount?) - this is easier said than done with complex loading patterns, but it can be done.
- Remember to zoom out - I got stuck debugging the wargear system for too long, which turned out to be not where the bug was.
- Use your debug tools - building Unreal Engine in debug mode had more debugging tools available to help track asset loads and narrow down where the failure occurred.