Modern Resource Bundling in a Game Engine

Modern Resource Bundling in a Game Engine

8/28/2026

At the end of the day, a video game is just another computer program. This class of computer program often comes with external files that are crucial to its successful execution. These external files are known as “assets,” “resources,” or “content.” I prefer the term “resource,” and I don’t really have a good reason as to why. It seems less popular in game engine design. I suppose “asset” is more appropriate. Whatever, let me be different!

The Situation

Let’s say a developer is working on a crazy new hit title, Fingle & Bingle: Partners in Crime (release year TBA). This game has sound effects. Let’s think about one specific sound effect, like a grass footstep. When the game plays this sound effect, it won’t always play the same sound sample. It’ll randomly select from 5 different samples, and maybe there’s a pitch range the game’ll randomly sample from. Let’s say that the configuration data for the sound effect itself is stored in its own JSON file.

Problem: Lots of Small Files

Okay, so here’s what that file structure might look like:

On the surface, this seems great. Don’t fix what ain’t broke, right?

… The problem is that it is kinda broke. While every operating system will face this issue, let’s strictly talk about Windows. On Windows, reading the contents of a file requires calling CreateFile, ReadFile some amount of times, and finally CloseHandle.

Interestingly, CreateFile is quite the bottleneck. Its execution is subject to interception via minifilters (as is the case with all I/O operations on Windows), and it has to perform path parsing/lookup. These things take time.

Problem: Easy “Patch-ability”

When Fingle & Bingle: Partners in Crime gets an update on its distribution platform (e.g. Steam), the client will receive a set of binary diffs. These diffs are used by the distribution client to mutate the game files that were changed in the last update. Large files might lead to unnecessarily large diffs, or worse yet, the distribution platform could fall back to sending a whole new copy of the file. Not good!

Problem: Runtime Format Resolution

Right now, when the game initializes its resources, we use its constructor directly. Like so:

AudioSample sample = new AudioSample("resources/grass_step1.wav");

This works perfectly fine. We know that grass_step1.wav is an audio sample. In fact, we expect it to be nothing but an audio sample.

What if we don’t know what grass_step1.wav is, but we still want to load it? Perhaps we call some function like this:

object someResource = Resources.Load("resources/grass_step1.wav");
// Casting someResource into AudioSample would work here.

That’s easy! We can just interrogate the file extension and dispatch the load request to some sort of handler, based on what we find. If the game is asked to initialize a resource from a .wav file, it can easily determine that it needs to be initialized as an audio sample.

Okay… what about .json? Files with that data format aren’t always encoding the same data. grass_step.json might contain sound effect data (that conforms to a struct called SoundEffect), and fingle.json might contain base stats for a character (it conforms to a struct called Character).

There are 3 yucky solutions that immediately come to mind:

  1. Load the data as an untyped data structure, like a HashMap or something.

    I would like to deal with untyped data as little as possible.

  2. Store the class name in the name of the file. (ex. grass_step.soundeffect.json, fingle.character.json)

    This feels very janky, and could add a lot of noise when browsing the game’s source files.

  3. Wrap all JSON data in a structure that declares the data type.

    grass_step.json would look something like this:

    {
    "type": "SoundEffect",
    "payload": { ... }
}

    This makes authoring a JSON file less trivial as we now have to insert magic data that the engine needs to process the file.

I don’t care for any of these options. At least option 2 and 3 allow us to somewhat enforce the struct layout. Option #1 is just absolutely abysmal.

Solution: The Resource Bundle

It’s not a new concept. Game engines will use large bundles for their resources.

Notice the italicized word: large. Let’s look at some of the file structure of a typical Unity game.

(In uncompressed Unity player builds, sharedassets* files will also have a .resS and .resource counterpart. For sake of simplicity, we don’t care.)

At build-time, Unity will figure out what assets are used in each scene and bundle them into a corresponding level* bundle. If an asset is referenced in multiple scenes, then it gets roped into the sharedassets*.asset bundle of the first scene that references it.

This is a really good system. If a game developer only references assets strictly (that is, there exists a hard, auditable reference to each used asset), and structures their scenes sensibly, they get to enjoy the benefits of this system. Unity did a really good job engineering this.

This works because Unity has very tight control over your game. The Unity player is a sandbox that takes your game as input and orchestrates everything for you. Games have to be authored via Unity’s editor. For engines that take a more hands-off approach (like a framework), this solution doesn’t work.

Alright, Unreal Engine, you’re up!

So, by default, if your game does not put any care into its bundles, this is what you will get: a single ginormous bundle. In Five Nights at Freddy’s: Security Breach, this is the exact case, and its resource bundle is 58.64 GiB in size. Unreal does let you group resources into different bundles through a numeric ID system, but that is insanely archaic. To make the best use out of it, this system might require teams to allocate numbers to specific sets of resources. Perhaps they also maintain a separate list on a Word document that might have comments like “Gary, I swear if you ∗∗∗ put Fingle’s ∗∗∗ assets in the Bingle bundle I will ∗∗∗∗∗∗∗∗∗∗∗∗∗∗!!!” … It’s also a pain to audit.

Godot is a similar story, except there is currently no user-friendly way to control how it creates bundles. By default, all your resources are scooped up into one giant bundle. You can build bundles manually and load them via scripts, but it feels like this should be a native feature of the editor.

My Resource Bundle

Here’s what I’ll do instead: we create a bunch of tiny bundles. 99% of the time that there are small resource files, they can be grouped together in a meaningful way. A bundle can be a sound effect with all of its samples collected together, or an ubershader that contains all of its possible variants, etc. Inside each bundle, we won’t have “files.” Instead, bundles will contain binary blobs that will have type information in their metadata. I will call this a “type hint.”

A type hint is like a mimetype, but there are 2-3 components. The first component is the type’s origin. The origin is usually where the resource type’s initializer lives. For example, a 2D texture would be the engine’s responsibility, so engine would be the origin for a 2D texture. Character data for Fingle & Bingle: Partners in Crime would be the game’s responsibility, so its origin would be fingleandbingle (or fnb, fingle, whatever the developer decides, really). The second and third components describe the actual type. The third component is optional, but when it’s used, it’s interpreted as a “sub-type.” Since there are 2D and 3D textures, their full type hints would be engine.texture.2d and engine.texture.3d, respectively. The game’s character data doesn’t have a sub-type, so that type hint would just be fingleandbingle.character. A type hint should be in lower-kebab-case, except for the origin, which should be simple and without spaces.

Each blob can have a name. There can be up to one blob without a name. A nameless blob is called the head. We can retrieve the head like this:

object soundEffect = Resources.Load("resources/grass_step.sfx.res");

(More on that file extension later.)

If the resource bundle doesn’t have a head, then that call will fail. We can also load named blobs. Let’s say we are interested in a blob called sample1 in resources/grass_step.sfx.res:

object sample = Resources.Load("resources/grass_step.sfx.res", "sample1");

Authoring

Alright, let’s create one of these beasts. Here’s how the API should roughly work:

  1. Construct a new bundle object.
  2. Push binary blobs to it.
  3. Flush bundle object to file.

Naively, one might store up all the binary blobs in memory up until flushing happens. I don’t like this approach. It feels like we are using more memory than required for this task. In a CI job runner environment–where memory could be constrained–this might be a problem. With that insight in mind, here’s a rough overview of my bundle format:

(This is very similar to ZIP’s structure.)

The magic sequence and version number can be written immediately upon opening the bundle. When the API consumer submits blobs to be inserted into the bundle, we can immediately write that to disk and dispose of the blob in memory. We’ll keep track of the table of contents in memory until we are ready to flush, which we can then write to disk alongside a pointer to the beginning of the table of contents. The pointer at the end is necessary for reading the bundle. If we define a fixed size for the pointer, opening the file is as simple as:

  1. Seeking a little before the end of the file.
  2. Reading the pointer.
  3. Seeking to where the pointer goes to.
  4. Read table of contents from there.

It’s just a tiny bit more complicated than if the table of contents came sequentially after the header. However, now we consume less memory during bundle creation, yippee!

There are multiple ways you can use this API at build time. Here’s what I’ve gone with.

Shader files (*.slang, for example), don’t change at all. The resource builder will parse & compile variants as it normally would, but it would output all of this information into a resource bundle. src/resources/pbr.slang would be built into bin/resources/pbr.slang.res. It is up to each individual file type’s builder to determine how the bundle should be constructed.

What about our sound effect?

Here’s our sound effect’s file structure from before:

And here’s what we’ll change it to:

Directories with an extension can be built just like a file. When the resource builder encounters a directory with an extension, it won’t traverse the directory and start building resources in there. Instead, it’ll treat the directory as one resource. This is not a new idea at all, Apple’s operating systems are notorious for this structure. When src/resources/grass_step.sfx/* gets built, it’ll become a file at bin/resources/grass_step.sfx.res. There will be 6 blobs in the bundle: 1 of type engine.audio-effect and 5 of type engine.audio-sample.wav. The .sfx bundle builder decides that definition.json will become the engine.audio-effect stored at the head of the bundle, and each sample*.wav will become engine.audio-sample.wav blobs called sample*.

Conclusion

This new paradigm requires the developer to think about how the game’s resources are built, which sucks. But, it directly addresses all of our problems, and I think it’s a good compromise. Plus, there’s plenty of room to grow. It’s trivial to add features like compression, or extra metadata suggesting how the application should process the bundle. For example, my engine allows blobs to be marked as “load eagerly,” which means that the engine should initialize the resource when opening the archive, even if it wasn’t explicitly called. This is handy in cases like sound effects, where we know that all the samples are going to be referenced by the sound effect anyway, so we can load them sooner.