Ninja Boss Fight said:
Any suggestions as to what I could do to make my system scalable but, still maintain efficiency?

Maybe you want to add depth to each sprite, sort by depth, and render in order. This way 3 layers might still be enough, because we can do fine grained front / back ordering by setting depth.

But not sure what problem you try to solve. And if you don't have depth yet, i wonder what happens if sprites on the same layer overlap?

Ninja Boss Fight said:
Any suggestions as to what I could do to make my system scalable but, still maintain efficiency?

In general, you don't get anything for free. Every gain comes with some pain elsewhere. The trick here is to move the pain to a spot that you don't get hurt much.

Would there ever be a need to render more than 3 layers or 9 (assuming I could implement a depth within each layer)?

Yes, someone will hit the boundaries of what you offer for some good or less-good reason.

What strategy could I use to determine where sprites should be rendered based on their sprite type without forcing users of my framework to explicitly declare?

Not sure I understand this, a user would need to specify what goes where, right?

Personally I don't see much trouble in

enum LayerId {ui, game, background};

// Giving a LayerId value with each sprite.
engine.addsprite(LayerId.ui, new Sprite("mySprite.png")); // Or store the layerid in the sprite itself.

You can also make a Layer class to store sprites of one layer, and have the user organize it:

// Storage for sprites of 1 layer.
public class Layer { ... void addSprite(Sprite); }
Layer[] layers = new Layer[3]; // Can also be a List<Layer> of course.
for (int i = 0; i < layers.length; i++) layers[i] = new Layer();
engine.setLayers(layers);  // Engine takes an array of layers, and only defines what entry is nearest and farhtest away.

// Add a sprite
final int UI_LAYER = 0;
layers[UI_LAYER].addSprite(new Sprite("mysprite.png"));

If you don't want an array, you can also do it more interactively

// Example user code creating 3 layers.
uiLayer = new Layer();
engine.renderSystem.addLayer(uiLayer);

gameLayer = new Layer();
engine.renderSystem.addLayer(gameLayer);

backgroundLayer = new Layer();
engine.renderSystem.addLayer(backgroundLayer);

// Add background sprite
backgroundLayer.addSprite(new Sprite("mysprite.png")); // probably with more parameters.

Personally I see very little difference between these approaches, but ymmv.

Ninja Boss Fight said:
the user could specify a depth in the sprite sheet file through the command line (ie when using the command line tool to convert an image in to an asset readable by the engine)

As long as I can automate building, I am fine ? Manually running commands with the right parameter values typically ends in disaster, since I make typos in everything. [EDIT: Can you query and/or modify such values?]

Specifying it at the commandline does break the “not explicit specifying” idea. Did you consider hiding it in the filename? 123_mysprite.png (or mysprite_123.png but probably the former is better as many OSes then sort on depth without doing any work for it).

Ninja Boss Fight said:
A user could move a spritesheet in to a particular layer folder and the renderer would decide where to render.

Interesting. Sometimes i did some automation for a company. E.g. after artists are done with their artwork, they save an image to a certain network folder. And a background process on a server took those images and rendered automated 3D box shots from it, useful for marketing.

It's a good form of interface i think. No need to operate on some application, so pretty fool proof and convenient to use. People really liked that they do not need to do anything else than putting a file to a folder.

Ninja Boss Fight said:
@Alberth I like some of your suggestions. Currently I have completely abstracted the need for a user to specify a layer because the engine has a specific file structure. A user could move a spritesheet in to a particular layer folder and the renderer would decide where to render. I am considering adding depth to each layer which I think would provide alot of flexibility when it comes to showing something on top or under. For example, A sprite on the background layer folder will never be rendered above a sprite on the UI layer; If a sprite is also on the background layer, the user could specify a depth in the sprite sheet file through the command line (ie when using the command line tool to convert an image in to an asset readable by the engine). This renderer could then decide how “deep” to place sprites. What do you think?

JoeJ said:
It's a good form of interface i think. No need to operate on some application, so pretty fool proof and convenient to use. People really liked that they do not need to do anything else than putting a file to a folder.

I completely disagree, this sounds horribly inflexible and cumbersome to me. For example, what if you want to use show the same texture on different layers depending on the application? Reminds me of the aweful days of the rpg-maker which had a pretty similar system, and whenever I wanted for example show a “panorama” as a “picture”, I had to duplicate the file - and when I later edited it, I had to rember to do that on all versions, yuk. I much prefer having meta-data alongside the file which you can configurate - but I'm not an artist, maybe an artist would prefer the folder-based solution but as a programmer, not that much.