🎉 Celebrating 25 Years of GameDev.net! 🎉

Not many can claim 25 years on the Internet! Join us in celebrating this milestone. Learn more about our history, and thank you for being a part of our community!

Serializing assets

Started by
12 comments, last by Zipster 3 years, 11 months ago

You'll benefit from keeping things simple and straightforward, and leaning on the existing file-system for organization during development. Along the same lines as what @Juliean suggested:

  • The asset name is its path, without extension. This takes advantage of the hierarchical file-system that all your users already understand and know how to use. If two files with the same name but different extension exist in the same folder, treat the extension as semantic information (texture, model, script, etc.). The editor/game already know from context what type of asset is expected in any given situation, as does the user.
  • Two files per asset, one the actual asset data (ex. foo.tga) and the other a metadata file with the same name + .meta (ex. foo.tga.meta). Aggregating multiple assets into a single file is less than ideal during development, as you can't easily debug or inspect the contents from the explorer. Save that for shipping or testing builds, when you want to benefit from the runtime performance improvement of loading baked assets in bulk from a handful of files.
  • Metadata file will contain the unique asset ID at the very least, along with any additional information you deem relevant. When you scan the asset directory at start-up for *.meta files, you'll be able to immediately build the repository of asset names, IDs, and paths. Asset data can be loaded into memory or created the first time it's accessed.
  • Assets reference each other by ID, so even if they move, the references don't break. The bi-directional map you built will already tell you the new name/path. Again, this allows you to manipulate assets on the file-system directly and have the infrastructure automatically adapt without breaking anything.
  • A database is useful for querying asset metadata at scale in the editor. Let's say you want to tag your textures as albedo, normal, displacement, UI, etc. and then query them by said tag, or perhaps by their disk size or dimensions, a DB is a natural way to do so. As you load the metadata at start-up, you can populate the DB and keep it in-sync with the raw metadata. But it likely won't help you much beyond this purpose at runtime.
  • Only when you “bake” the assets for shipping, do you need to worry about building the asset ledger and aggregating assets into pack files, compressing them, whatever else. Supporting loose files until then makes everyone's life easier.
Advertisement

Ah I might have presented this poorly in the first place. I wasn't going to aggregate files during development.
An example, say I want to import a skeletal model (using assimp so there will be a-lot-of-parsingTM).

Serialize textures, save their IDs.
Serialize materials for each mesh, save their IDs, hook up texture IDs into them (serialize function for the material takes std::vec<ID>).
Serialize meshes, serialization function takes material ID (one each, for now).
Serialize the skeleton, save the ID.
Serialize all animations, save the IDs.
Serialize model - this is just a bunch of ID's really, one for each mesh, one for each animation, one for skeleton. It ends up being a root of a tree pointing to other things.
These are all separate files.

I'm mostly planning to do what you posted minus the implementation details. However the scanning to get around copying files around in the filesystem is actually a great idea as it wouldn't just break if we moved assets manually, good call.

You might want to go so far as to store those references and dependencies in the metadata, instead of the raw asset itself. Then you could build your entire dependency graph from the metadata alone without needing to load any asset data, and use the aforementioned database to perform all sorts of really useful queries. The corollary to that is that if any references or dependencies change, only the metadata has to be updated and not the full asset. If some assets are only a collection of references to other assets (as it seems your models might be), they can exist are pure metadata without any payload.

This topic is closed to new replies.

Advertisement